Tuesday, 28 June 2011

How Can I Estimate My Testing?

Have you had anyone ask you a question about estimation? I get asked these types of questions and I suspect that the person really wants answers about how to communicate and justify their guesses.
I think they hope that some process exists which will accurately and objectively give them a set of numbers. And by using these numbers they can disavow responsibility for the production of them. And no-one will hold them responsible if the objectively produced ‘estimate’ does not meet reality.
Well in reality ‘Estimate’ really equals ‘Guess’.
So if you want some strategies:
  • Answer the question you want them to ask
  • Ask how long they want it to take
  • Trust your gut
  • Assumptions, Risks and Issues
  • Track the passage of time
I could add an etc. in there, to alert you to the incompleteness of all of this.
But I won’t.
You should assume incompleteness in my presentation. And map it on to your model to identify the blanks.

What culture do you think you work in?

Do you think you work in a culture where ‘estimates’ and ‘actual time’ have become synonyms?
Do ‘they’ berate you for continuing to work, after the ‘estimated’ length of time has passed?
Well, you have learned that regardless of where the estimate came from ‘they’ will still hold you responsible for it.
So you need to develop some beliefs. You need to believe that “estimate equals guess”.
You need to believe that any supporting documentation for your guess does not provide a justification for it, it provides a partial model of the thinking that led to the guess.
‘They’ don’t need to believe any of this.
You need to.
Then your communication will change.

Estimates equals Guesses

All estimates equate to GUESSES.
We don’t know how long things will take.
I’m a pretty good psychic. But even I don’t know.
Sometimes we put a framework around our GUESSES. And that can reveal some of our ASSUMPTIONS. It can reveal what RISKS we have considered. It can reveal what ISSUES we perceive.
The framework reveals part of the model we think we used when creating the GUESS. It helps us explain how we created the GUESS.
But even when we do that, our estimates remain GUESSES.

So what could you do?

Do you try and ‘educate’ them?
I don’t.
I’m obnoxious like that.
  • So if you get asked “How long will your testing take?”
    • Don’t say “I will provide you with an estimate for testing.”
    • Say “I don’t know.”
    • The conversation might end there.
You might respond with the question you want them to ask.
  • So if you get asked “How long will your testing take?”
    • ask “Do you mean? If I had to guess, right now, how long I think it will take for you to have enough information that you can decide whether or not you want to release this? If so then…”
    • Yup. I say stuff like that.
    • Because then we start talking about what they really want to know.
    • I don’t think I engage in an education process. I try and communicate so we all work from the same assumptions and understandings.
The test managers among you might feel a little nervous with this. As a test manager I used to feel nervous saying that. So you might ask instead:

“How long do you want it to take?”

We live in a world of deadlines. So guesses often don’t help very much.
By asking “How long do you want it to take?” you can build a plan which tries to maximise the value that you can add as quickly as possible before the deadline. Work on the high value items, risks and issues.
Don’t worry about the guesses. Think about the constraints.
When you ask “How long do you want it to take?” then you can think about how much stuff you might fit into that timescale, does it ‘feel’ right?

Trust Your Gut

What you ‘feel’ can feed into your guessing process.
You can get the whole guessing process over really quickly if you just come up with number.
You won’t have stated any assumptions. You won’t have considered any risks. you won’t have any weighted factors. But you can come up with a number quickly.
And your gut will tell you how you feel about it.
Does it feel right?
More importantly – does it feel wrong? If so, you can:
  • Ask yourself what feels wrong about it. Bam. Now you’re starting to come up with assumptions, risks and issues.
  • Come up with a new ‘better’ guess.
When you immediately decide to come up with a new ‘better’ guess. Then you trusted your gut.
If you ask your questions. You distrusted your gut. So you ask those questions to help you regain your trust in your gut.

Assumptions, Risks, Issues

Why bother with the questions?
Why bother with Assumptions, Risks and Issues?
Well, they help you communicate.
Until you develop the force of personality where you can give a number and have it accepted as a best ‘guess’. You might want to engage in a communication process.
If you do want to communicate then you can explain to people the things you have assumed. e.g.
  • I assumed we deliver a release candidate on day X
  • I assumed we have a clean build with no failing unit tests
  • etc.
You can do the same for risks and issues.

We track the passage of time

If your guess gets shot down and ‘they’ convince you into using a shorter guess. Then you can ‘manage’ the on-going process by monitoring for any:
  • negated assumptions
  • change in riskiness of the risks
  • transformation of risks into issues
This may not help in the short term as you simply point out that the correctness of the guess becomes ever less likely.
But you might help build credibility long term by communicating based on your frame of reference.
We track over time, the ‘goodness’ of our progress, and how much distance we appear to have to travel before we reach our end point.
We validate our model over time, by comparing it with the reality that we perceive.

Have you covered everything?

I just want to encourage you to keep it simple.
I think you could waste time on a complicated ‘estimation’ process using spreadsheets and lots of factors.
‘They’ might well think they need those spreadsheets. Their process may require those spreadsheets. If so, use the spreadsheets to communicate your model, don’t use them to create the guess.
Of you could:
  • guess,
  • start to do the work,
  • based on how long it currently takes extrapolate forward and…
  • guess again
If you do this then you need more courage.
Swapping a belief of “objective estimation”, for “guesses and underpinning models” takes time.
I will not try to convince you that you should.
I just want to make you aware that you could.


  1. Interesting post! It's true that in many organizations, estimation once published automatically becomes commitment and dev/test team has to work like crazy to meet it because "you said that you'll finish by this date!" For some reason, when developers/testers says "given these assumptions, my estimation is that we'll finish everything by end of April", some managers hear "blah blah blah April blah blah" and take it that the project will finish by April 1st! Of course, the date is literally a joke, but who care.

    A good approach is coming back with a range, something like "given these assumptions and the fact that there are too many unknown, we think it will take anywhere from 2 to 6 months". Of course this is not about coping with management people, but about doing the right thing. Managers too have the burden of dealing with customers, partners etc. so they need to know what to expect. Setting the right expectation to them allows them to set the right expectation to customers and partners and in the end no-one has to point fingers. And it certainly doesn't mean the team doesn't have to work hard to hit the market before competitors. It's all about having nobody mislead and have to make impossible promise.

    Having said that, I do think an estimation framework, when properly applied in a suitable context, will help developers/testers to come out with a more educated guess (or range) or at least, enable them to improve their guesses over time. In the agile crowd, velocity-based testing is the de facto method and being used with much success. I'm not aware of anything like velocity-based testing for test estimation, but my company does indeed propose a method to estimate test size and effort. If you're interested, you can read our white paper on the method at http://www.qasymphony.com/media/2012/01/Test-Case-Point-Analysis.pdf.

  2. Thanks for the comment, I'm sure people will let you know if the whitepaper helps them.