Reality is different though. Many test automation approaches take a system-wide approach. Then you have lots of unwieldy test environment and test data issues to manage. You spend a lot of time trying to get all this under control. Next minute you are finding that it takes a long time to get a stable environment, automation ends up just focused on a old regression set, and there is no immediate feedback to the development team as it takes a specialist squad to decipher the results and clean up the false negatives.
In more recent times, there has been greater emphasis on "shift left". Get more of the automation at the unit level, make developers responsible for automation, make it part of the build process. In my view, this is all heading in the right direction.
Since getting involved in development in my own startup, I have been reflecting on my own experiences in this space. I am finding more of my time writing automated unit tests, as opposed to automation system tests. I am still not quite sure that Test Driven Development (TDD) delivers as much ROI as believed, but feel I am making ground in my unit tests adding value to speed up my delivery.
A big focus of my unit tests recently has been using Test Doubles to mock out underlying object behaviour. This leads to interesting object design to mock out dependency injections which would otherwise lead to external influences in the test. The mocks allow you to have good control over the piece of code you are testing. But then there remains the question, how do I know when I take away the mock and integrate with the real implementation I don't have defects?
My thoughts now are turning to the idea that we need tests of the actual behaviour that match up with the behaviour within the mock. Ultimately expected behaviour in the mock must match up with actual behaviour of the implementation. We need assertions on both side to match up. We need to ensure the contract behaviour of the mock and implementation are equivalent.
If we aren't satisfied that we have this covered we will need to invest in integration testing where we combine the two components. But then automation becomes much more challenging in this domain, coverage is much more difficult, and manual testing is frustrating.
Check out J.B. Rainsberger, Integrated Tests Are A Scam for a good overview of the challenge and the resolution. Unfortunately "jbrains" keeps the secret sauce to himself in terms of a solution, but as you can see from the presentation, collaboration/contract test automation is a key to reducing the integration challenge.
Some months ago I was at a presentation at SEEK given by REA, where they discussed Pact, which provides consumer driven contract testing. Pact addresses the challenge of how to ensure that when mocks are used to help test a consumer of an interface or service, then tests of the actual interface or service are also created to ensure that the contract is preserved. This is what I need, but Pact is in Ruby, so now I am searching for a similar process for Php without learning a new platform - doh!
In the meantime I will provide traceability between my mocks and my contract tests. I will search out other frameworks which will assist me in synchronising the mocks and contract tests.
My experiences in this space though are leading me to believe that this is an approach to address the larger scale integration challenge. If we use this approach to test service interfaces between systems, then we reduce the accumulation of system integration testing effort. This will have benefit to both product quality, as well as delivery schedules. Hopefully less time waiting for system delivery schedules to align. More ability to get the component right in isolation before integration. No doubt we will have some challenges on agreements of the service contracts between systems, getting communication between teams, etc.
Perhaps however our mocks and unit tests are the basis for specifying that behaviour and defining the contract agreement?
