Q: when do we prefer manual testing over automated testing?
A: I have a problem providing a simple answer to this question…
And not just because “it depends”.
Also because - “that’s not the model I use” so I don’t frame the question like that.
I don’t think in terms of “manual testing” and “automated testing”.
I think in terms of “testing”, “interacting directly with the system”, “automating predefined tasks, actions and paths”, “using tools”, etc.
- do tools already exist to help?
- do I have abstractions that cover those areas?
- do I have to create tools?
- do libraries already exist?
My testing might involve lots of automating if tools already exist.
I might not automate, if I have to create. When - if I create, the testing has to wait.
- something we need to do rarely
- usability? “look and feel”
- experimental features - surely we wouldn’t automate these
- something still under development and changes frequently
“I might not automate, if I have to create.”Usability and “look and feel” have tools to detect standards non-compliance. I might very well use those. So I might automate portions of this type of testing early.
Experimental features might fit into an existing process and so we might already have a lot of the scaffolding we need to support minimal and speedy automating - in which case I might automate those.
I have automated features that were changing frequently because we were testing the same ‘path’ through the application many, many times (every time it changed) and we often found problems (every single release) and the effort of testing it, by interacting with it, was waste. So we didn’t interact with it until the automated execution could pass reliably. But, that also depends on ‘what’ in the implementation changes, and how that change impacts the chosen approach to automate it.
Are the assumptions behind the scenarios true?
- Scenario: rare execution
- Assumption: we would spend a lot of time automating something that we would rarely use
- Scenario: experimental feature
- Assumption: we write code to automate it that we throw away
- Scenario: usability or “look and feel”
- Assumption: this is subjective
- Scenario: frequent change
- Assumption: automating a scenario always takes longer than interacting with it and maintenance in the face of change takes too long
- I have automated when other people said I should not, and it saved time.
- I have interacted manually using tools, when others said we should automate, and we found problems we would never have found if we had taken the ‘automate’ path so early.
- I have manually engaged in stress and performance testing.