I’ve been at this 30 years, and Java since it’s introduction, so some folks will feel my opinions are a bit old school. I don’t think they are, I think they’re just a bit more thorough then some folks have the patience for. Recently I butted heads with an “Agile Coach” on this, and they certainly felt I wasn’t with the program. But I’ve been a practitioner of agile methods since before they were extreme and again, I don’t think the issue is that I’m too old school, it’s just I believe the best agile methods still benefit from some traditional before and after.
My View of The Testing Stack
People get obsessed with the titles of these, but I’m not interested in that debate, I’m just enumerating the phases in a approximate chronological order:
- Unit: TDD these please!
- Scenario: Test as many scenarios as you can get your hands on. Unit testing should catch all the code paths, but a tool like Cucumber makes enumerating all the possible data sets in scenarios much easier.
- End-to-end/Integration: Put the parts together in use case sequences, using as many real bits as possible.
- Performance/Stress/Load: Does your correct code do it fast enough and resiliently enough. These can appear earlier, but it needs to happen here too.
- QA: Yup … I still feel there’s value in a separate QA phase this is what got the agile coach red in the face… more to follow on this.
- Monitoring/In Situ: Keep an eye on stuff in the wild. Keep testing them and monitoring them.
So this is a lot of testing, and plenty of folks will argue that if you get the earlier steps right, the later ones are redundant. I don’t agree obviously. I see a value to each distinct step.
- Unit: Errors cost more the further along you find them so the earlier one tests the better. So get as good coverage at the unit level as you can tolerate.
- Scenario: Using unit tests to exhaustively cover every possible scenario can be … exhausting, but it’s worth sweeping through mass combinations and permutations if you can.
- E2E/Integration: Now you know the pieces work in isolation, you need to see that they work together. This will shake out problems with complex interactions and insure that contracts where adhered to over time.
- Performance: If it’s too slow or, fragile to use, it’s useless.
- QA: Here is were a lot of folks say I’m being old school. If you’ve gotten through the prior steps isn’t this redundant? No. No matter how good your working relationship and communications with your product team and customers are, your development team always comes at their job with a technical bias. At least I hope your developers are technical enough to have a technical bias. At a bare minimum, having a separate pass of testing that is more closely aligned with the business perspectives makes sure that you avoid issues with the “any key” and the like. But this pass can help you hone and improve product goal communications, and act as feedback loop to improve the prior testing steps. In a perfect world it would become redundant.
- In Situ: Keep your eyes on the product in the wild, test them there too, you can use prior tools and scenarios here, but testing chaotic scenarios is invaluable. This is about watching the canary in the mine, and seeing if you’ve even been worried about the right canary.
From a cost perspective you always want to front load your testing as much as possible, but if your goal is quality, the old adage “measure twice, cut once” should be amended to “measure as often as you can practically tolerate, cut once”. Needless to say, automation is the key to it, tool everything, make the testing happen all over the place without human intervention.