Thursday, 12 January 2017

What do you do? As a Tester, when you are asked for ROI calculation

TLDR; push back, ask questions, and if all else fails - plan it as a manageable set of tasks

At the Test Automation Guild I was asked a question about how we calculate ROI for Test Automation. And its a hard question for me to answer in 30 seconds when I don’t know enough about the situation the person asking the question finds themselves in.

Because testers are often asked by managers for information that the manager should really be dealing with and which does not seem to add value to the process and which we are concerned trivializes or views the process from the wrong perspective.

I know this is a difficult position for testers.

And I know testers are asked to do this.

I empathize. I’ve been there.

And I want you to be able to take this less seriously - consider that your only warning for what you are about to receive.

So I will attempt to provide tactics to help with the situation.

Easy Answer

And the easy answer is to just go off and do a web search for “ROI for Test Automation”.
  • You’ll find loads of papers written by consultants on how to calculate the ROI
  • And probably tools and spreadsheets to calculate ROI
And the easy way is to look through those, find one that uses numbers that you have easy access to, fill in their template or tool and give that back to your manager.

That’s still going to take time, but maybe half a day, a couple of hours if you just pick the first one you find and don’t care about the results.

Here you go:
Job Done.

But You don’t want the Easy Answer!

And if you did not go for the easy answer then it might be because:
  • that does not feel right
  • you don’t think ROI is a good metric to calculate
  • it’s not all about money
  • <insert your reason here>
Exercise: No really, ‘Insert your reason here’ 
Ask yourself enough questions so that you know
why you did not take the easy route. 
Because sometimes, the easy route, might be the right one. 
And if you choose not to take the easy road then
you are going to need some conviction to back you up
because what you have to do instead is harder. And you
need to have a solid core and belief to take you forward.
Hopefully you know why you did not go for the easy answer. And you're ready for the hard path.

Here are some options, but these all require some assertiveness, and communication skills to put into action.

Are you ready for that?

OK then:
  • try and push it back on your manager to do the ROI calculation
  • ask more questions
  • turn the ‘ask’ into a well defined project, with a set of tasks and a plan

Step one - try and push it back on your manager

This is really a management activity. Why is your manager asking you? Something is going on here that has not been fully explained to you so we need to tread carefully when we phrase our responses.
  • “That’s your job you lazy ****!”
  • “Oh! Are you resigning?”
  • “Am I getting a promotion?”
  • “Does this new responsibility mean a pay rise?”
You probably want to avoid the above.
  • “Why do you want this?”
  • “Why do you want me to do this?”
You probably want to avoid the above as well, because asking “Why?” targets someone’s belief and we are assuming that " Something is going on here that hasn’t been fully explained to you":
  • “Why…?”
    • “Because I’m your manager!”
    • “Because I do!”
    • “Because I’m telling you!”
    • “Because I’m asking nicely!”
    • “Because I’m not asking nicely!”
    • “Because!”
OK, this isn’t going well. We need a new approach.

A Quick Introduction to the Art of Verbal Fencing

  • “I don’t have time for this because I’m busy with the testing right now, can you do it?”
  • “I don’t think I’m the right person to do this because I don’t have budget responsibility, could you do it?”
I’m using a “No…Because…Redirect” template here, which is quite a sneaky attack.
  • I’ve put the negation block at the front to get it out of the way early
  • I phrased the block as an “I don’t” rather than a “No” because many people don’t like to hear “No”
  • I put a ‘because’ in, because a reason makes the early statement seem like it has more weight and justification
    • the ‘Because’ does not have to be great, it just has to be there because then we add a because and then it looks more justified because of the because.
  • The killer punch is the question at the end “Can you do it?”
    • its a question and people answer questions, so suddenly you take control of the conversation and if you have persuasion and conversation skills at this point you gain an edge.
    • Thank goodness you gain +2 on all Charisma roles because of your new haircut.
    • its a question that deals with capability, so in order to answer it they have to think about their capability and ability to do the job
    • it innocently hints at the responsibility residing with them
Exercises: Fulfill and List 
There are lots of ways of fulfilling that template, so take some time and list more.
Identify other templates you could use. 
“If…Bad…Emotion” e.g. 
“If I did that then we would lose time on the testing and we don’t want to lose any more time” 
Chain templates for killer combos: 
“I don’t think I’m the right person to do this because I don’t have budget responsibility and If I did that then we would lose time on the testing and we don’t want to lose any more time because we’re already de-prioritising parts of the testing and if something slips through we might be facing a major production outage which would be a disaster, can you do it?” 
You might get lucky and they might:
  • concentrate on the question, in which case you can keep jabbing with more questions until they give up
  • concentrate on the reason, in which case they are no longer concentrating on the ‘ask’ and you have a chance to steer the conversation until they give up
    • even if they do manage to bludgeon down your objection you can follow up with another “No…Because…Redirect”
  • concentrate on the “No” and then they have to address your statement. In the two examples that would mean - re-prioritising and re-planning the project, boosting your self-confidence, organising a training course, etc.
But all of this might not work.

Your manager might be trained in verbal fencing as well:
  • “I understand and I want you do to it anyway.”
  • “I get that and I need you do to it anyway.”
  • “I take full responsibility for that and I need to you start now.”
  • “Just do it.”
So we need to move to the next step.

Step Two - Ask More Questions

You’re not going for the easy route so you need to make sure that the activity you are about to do adds the right value so you need to get the requirements sorted out.
  • Who is the audience?
    • “Is the ROI for you or for someone else?”
    • “What format does this have to be in: spreadsheet, document, slidedeck?”
  • What does it have to support?
    • “What will this be used for?”
  • Timescales and priority?
    • “When is this supposed to be finished by?”
    • “How much time should I spend on this?”
  • Risks
    • “Has the project been told I’m spending time on this?”
    • “Who is covering my work while I’m doing this?”
  • What is the content?
    • “What information do you need?”
    • “Is there a template I should use?”
  • How often?
    • “Is this a one off or will we calculate ROI again?”
  • How can I do this?
    • “Do you have any calculations we should use?”
    • “Any links to white papers or books I should read?”
    • “Do you have the current actual measures that I should use in the calculations?”
    • “Who should I speak to? Do they know I’m doing this?”
  • etc.
With any luck your manager will give up during the questioning process:
  • because they haven’t thought this through
  • because it will be easier if they do it
But, these are serious questions. There is a difference between a 25 page ROI document and an envelope with a guess on it - granted the only difference might be the time it takes to create, but that is still a difference.

And you need information if you’re going to do it.

Which is the next step.

Step Three - Agree a Plan

Using your questions and the answers you and your manager need to agree a plan of work to create this ROI.

You will agree:
  • scope
  • format
  • contents
  • timescales
  • effort
  • relative priority
  • how often to check in
  • what do to when you get stuck
At this point:
  • you have pushed accountability for the end result back to the manager even if you are responsible for the actual tasks,
  • you’ve made it clear that you will need help and it isn’t your job alone to do it
  • you don’t have to stress about it any more

But its still wrong

I get it, you still don’t want to do it.

Depending on your relationship with your manager you might even be able to tell them that.

And I agree.

I think trying to calculate ROI for ‘Testing’ or ‘Test Automation’ does not help.

We do need to know what benefits we are expecting from the process and how much we are prepared to spend on the process. We do need to understand how much it costs to do what we are doing. And we should manage the process to achieve the benefits and change the way we work if we can’t achieve those benefits within that cost.

And that sounds like the actual work of management, rather than calculate an ROI.

At some point in the future, I’ll write up something for managers on ROI of ‘Test Automation’ or 'Testing or ‘Live Fish Regurgitation’.


  1. Hi,
    Can I suggest another alternative? First have a conversation about why you are automating. That conversation should reveal a few points that resonate with the leadership team. I could be the additional test coverage gained, like through a data-driven approach, or it could be time saved with regression testing, or supporting the ability to release more frequently (or continuous deployment).
    Once you find that important reason for automation, then find some measurements to show those benefits. This might be part of the "agree to a plan" stage, first find out what is important - then measure that.

    Good luck!


    1. Hi John, thanks for adding the comment. I agree, asking "Why" when automating is essential.

      And if people have the right relationship with their management then this could be a good question when asked about ROI. We do have to be careful with Why because it is easy for people to take the wrong way, which is why the relationship is important.

      Unfortunately "Why?" isn't mandatory for an ROI answer. We can report on ROI without knowing Why or asking Why. And I think it is one reason for people asking "What is the ROI?" rather than "Are we achieving the benefits we wanted and are we doing so with minimal waste?"

      "Why?" is one of the things I was alluding to in the 3rd last paragraph (But I didn't write "Why?" I described it in terms of the benefits we were looking to realise.)

      I hope, as you describe, that "Why?" is one of the first things that people do when they decide to commit budget and time to automating.

      Thanks for helping clarify the post. I'm sure that you are right and for some people "Why?" will be a good discussion to have first when asked for an ROI and your comment might help them realise that.


  2. Hi Alan, Yes please write something for managers on ROI of ‘Test Automation’ or 'Testing'. For when Managers get asked for this by upper/executive management.

    1. I will. But John has hinted at, and pre-empted the answer in his comment.

      We shouldn't really have to get to this stage because we should be asking "Why?" so we can work out the benefits and then continually asking - do we still care about this? are we realising this benefit? are we doing this efficiently? do we feel the benefits are worth the spend? is this still a good way to mitigate our risks? etc.?

  3. My initial way of trying to shake off such tasks is using the "yes, but..." template.
    - Sure, but can we do the first one together so that I'll learn how to do that?
    - Of course, but I will need your help quite a bit
    - Definitely, but can it wait until we're done with the release we have in a month? we're really hard pressed with this one.
    - Why, yes I can, but just take note that I don't see the entire scope, so you will have to fill in quite a lot.
    - Yes, but I think you'll get better results if you'd ask Alan, he's more familiar with the details than I am.
    - Sure, but we both know that any number I can give will be a blatant lie, right?

    Also, failing to avoid the task, there are some questions I'm missing in the post - questions that point the value of automation and the difficulty of quantifying it. With some luck, I'll give my manager the "ammo" needed to fend off the request for an ROI calculation:
    - When I got here three years ago it took us about 6 man-weeks to complete a regression cycle, in the past month we did a couple of regression cycles due to some scary changes and each took us only about 3 to 5 man-days and we were able to contribute to the new features while doing that - I can count the tester days that were gained, but how much is it worth to you to be able to make those changes a month before a major release and deliver on time for the holiday code-freeze?
    - Last feature I automated a check for I compared about 50 data points between two systems, and found out that two data points were missing in one of the 10 flows compared - Manually, I would not have time to go over the data points one by one, and I could just miss it - what's the number I should attach to such risk?
    - Remember the feature before that? when I was automating an API call and just by being lazy found that I can log in to our server using a set of credentials I have created without actually logging-in? How much did we pay those external pen-testers that found only less severe issues?

    1. Thanks for the extra info, hopefully people will make it all they way down to the comments and increase their options with your examples. There are 'lots' of questions missing in the post :) I didn't feel that I could write "etc." everywhere.

      Thanks for adding more examples and questions.


  4. Thanks for elaborating on the question Alan. I agree with the premise of your discussion. Generally putting a number on such activities is a hard one, and does more harm than good.

    The reason I keep coming back to it is, for upper management, I think at the end of the day the bottom line is Dollars and cents weather we like it or not (my MBA makes me biased here), and that's not a qualitative value. I quote to my team, if you don't put a number on your head, someone else will, and their estimate is going to be far worse.

    My take away is to stall the ROI question, if need be, just focus on the benefits, instead of cost vs benefit.

    1. Thanks for the comment.

      I think with an MBA you'll look at concepts like ROI and Cost vs Benefit differently from most managers in the the Software Development Field. I suspect you will have been exposed to more formal definitions of the terms with a formal understanding of the calculations associated with them. All of which helps add extra dimensions to your reading and contribution in the comments.

      Certainly from this post one take away is "stall the ROI question".

      This post was focused on the tactics that someone under stress can use to help themselves in the described situation, it doesn't describe all the nuances associated with ROI.

      I'm quite happy with the bottom line being Dollars and Cents and I agree with your paragraph.

      I do think we should consider cost versus benefit.

      Other nuances of ROI, I think 'invest' and 'cost' are different. I don't think we 'invest' in the tools and artifacts associated with automating. I think we spend money because those things 'cost' money and we derive benefit from them. Which might edge people over to considering cost vs benefit.

      Based on the ROI and Cost/Benefit formulas that I've seen, I think the business training and MBA books have slightly different definitions of 'invest' and 'spend' and 'cost' than dictionaries do.

      I associate invest with - put the money in, get a return, get the money back. (read small print for risks).

      I associate invest with sales linguistic trickery based on the above association 'invest $X in this training' versus 'this training costs $X'.

      Part of my avoidance of ROI is that most managers that ask for ROI don't have an MBA. But people answering the question go off and try and match ROI formulas to the request, when the manager really wants to know if the money is being spent effectively and if they still need to spend the money. And I think that is different from ROI, but is closer to cost vs benefit, but also can be answered qualitatively.

      In the post as currently written, ROI is a massive source of ambiguity, and that also contributes to the stress of the testers on the ground.

      And... perhaps all of this hints at missing questions to ask in this situation:

      - what do you mean by ROI?
      - what do you mean by Return?
      - what do you mean by Investment?
      - what would be a good ROI?
      - what is an acceptable ROI?
      - what is a bad ROI?

      In the post, we do ask how we would calculate ROI - so we could use that as a 'first question' to pursue a deeper understanding of ROI and remove the ambiguity. And it might also reveal that the manager doesn't know this, or doesn't know if their understanding matches the understanding up the management chain.

      Thanks :)

  5. I just reply back "What are you smoking, can I have some?" Then I say to them let's look at it from the perspective of what is the efficiency gain we are getting. I try to take the money aspect out of the question/equation. I'll say look at it this way; it takes 4 human testers an hour to do a smoke test of X tests, and under automation it takes 4 machines (automated testers) 15 minutes to execute the same smoke test. You now have a 4x increase in execution efficiency. BUT you need to take into account the time it took to initially build those automated tests. So the real gain is in repeated use of the automated tests later on down the line for the overall testing effort. That usually confuses them enough and they leave me alone. If it doesn't work then I just say excuse me I need to go take care of something more important.

    Jim Hazen

  6. Thanks Jim.

    Your answer has a level of assertiveness in it that takes time to build up. And sounds almost like someone with experience of management therefore it might provide options for more senior testers who are asked this question.

    The efficiency model that you describe without the dollar values, is also one that people use with dollar values to present an ROI, so a manager might respond with "Great, can you put money values on those times and the repeated usage."

    And then I assume you would expand your answer. I imagine that you would then go on to describe "and that's not all..." because the automated execution frees up the team to do something else in parallel so we have opened up more opportunities for testing and we can do more testing in areas that have no coverage from the automated execution. "and that's not all...etc."

    Exercise: For people making it this far through the comments - can you expand on Jim's list to create a list of "and that's not all...", this list will help you qualitatively describe benefits from your process and might identify opportunities that you haven't considered.

    One issue with the ROI question is it tends to ignore many of the benefits and ignores benefits that cross and intersect system boundaries e.g. if one system was "testing" then another would be "programming" and there are benefits relating to collaboration, code re-use, etc.

    Thanks for expanding the scope of the answer Jim,


    1. Alan,

      Sure, I was just trying to be smart-alecky about it. I just hate that question now, and that is because in the past I have tried to put the "money" into the equation and all it has done is paint me in an corner. I understand we need to "speak" in their language (money and time) at times, and it is key to our survival to be able to do it. I just try to diffuse that one from the start because management & C-level people tend to get "missile lock" on just the dollars. As you said they don't see the bigger picture of "qualitative" benefits. I try to steer them back to that.

      All else fails I tell them to go watch this on YouTube:
      It makes my point.


  7. I've been asked this by PHBs who have never been testers.

    Handling it is real easy:

    1. Find out what number they want.
    2. Create a way to produce the intended number.

    Most managers will have no problem letting you control the metrics and conditions.

    Remember, your job is to make the manager look good.

    I know this might sound a bit cynical or sarcastic, but this method has worked for me in a typical corporate environment with lots of layers of management.

    Happy testing!

    1. Thanks, I have seen people use this tactic successfully too.

      And by successfully I don't mean - the numbers accurately reflected the 'investment' or improved management insight or control over the process. But they did make life easier for the person producing the metrics.

      I'm glad you put in "Remember, your job is to make the manager look good." because that is a mindset that will help people reading this adopt the strategy.

      Sadly for most of my managers I didn't see that as my job, and I've never seen that as the job for people that report to me, so I've never been able to satisfactorily adopt this strategy. I've only been able to adopt supply numbers using the strategies I listed in the blog post.

      Since I've seen this strategy work in corporate environments. It might well help readers who make it this far down the page. And without any cynicism or sarcasm I can honestly thank you for adding it.

  8. The formula, which forms the basis of ROI calculation, is :

    (a) Automated test script development time = (Hourly automation time per test * Number of automated test cases) / 8

    (b) Automated test script execution time = (Automated test execution time per test * Number of automated test cases*Period of ROI) / 18

    (c)Automated test analysis time = (Test Analysis time * Period of ROI) / 8

    (d) Automated test maintenance time = (Maintenance time * Period of ROI) / 8

    (e) Manual Execution Time = (Manual test execution time * Number of manual test cases * Period of ROI) / 8

    Notes: Period of ROI is the number of weeks for which ROI is to be calculated. Divided by 8 is done wherever manual effort is needed. Divided by 18 is done wherever automation is done.

    In this method, time investment gains are considered rather than monetary gains. Once gains and investment costs are calculated, we can insert the values of these variables into the formula and get the total Efficiency ROI.

    In the Efficiency ROI calculation method in test automation, focus is on the actual efficiency of the automation and not just the money factor. This method does not require any sensitive information such as the hourly rate of tester, etc. However, this method makes several assumptions such as automated test cases fully replace the manual test cases (but in reality some automated test cases may need manual intervention) and that the manual testing counts only one manual tester.

    1. The above comment was not originally posted anonymously but since it had spammy links in, I couldn't in all conscience publish it with the spam links.

      I receive a lot of comments with links to the particular vendor linked to, but I mark all those comments as spam. And despite my writing to the vendor and asking them to stop posting spammy comments on my blogs. They still try. They are one of the reasons that no comment will every be published on my blogs without reviewing it first. I Thank them for increasing my work load.

      But fair play - this comment was actually on topic and if it had not included the links I would have been able to add it directly - Blogger doesn't allow me to edit comments from people otherwise I would have edited out the links and then it would have linked back to the profile that posted the comment.

      And now... the moment you have been waiting for... the response to the comment.

      The calculations you describe are exactly what I was saying in the post that I think do not represent ROI. It is an attempt to justify the cost by comparing one tactic with another, even though the tactics are not like for like comparable e.g. when you automate you run the automated code more often - not because of cost, but because we can and we see a benefit in doing so, and that benefit was why we automated in the first place.

      If we don't automate and we have people test the system, we don't just have them try and "Execute the Script" as often as we would if it was automated. We hopefully try and increase the benefit in using 'people' by assessing risk, going deep into the system, not repeating exactly the same path with the same data, we try and take advantage of the fact that people offer different benefits from automating.

      One benefit is that people can test the system in increasingly new and varied ways to find new information and expand our model, whereas the automated code will exercise the system and check that the coded assertions have not failed - these are two very different approaches and I find it impossible to compare them directly to build an ROI calculation.

      Thank you for the comment.

  9. Testers: There is no ROI in test automation. Here's a powerful parallel:

    Just as there is no ROI on insurance, market research, network servers, lighting, or development tools, there is no ROI on testing or test automation. The question is not being asked by a competent financial person.

    What they're ultimately asking is "why should I trust (and therefore pay for) the approach you're taking, including your use of tools?" A number won't tell them that.

    "We do need to know what benefits..." etc. (above) is the work of management, but it's also part of the self management of a responsible tester. If you can't articulate those things, it's time to learn how. Alan's post gives some valuable advice on that.

    ---Michael B.

  10. Thanks Alan for elaborating on this question, I was the one who asked about this at the Automation Guild ;)

    I was really interested in getting your opinion regarding metrics in the Agile space.

    As Automation Architect I have been challenged many times by the very well known quote "If you can't measure it, you can't improve it".

    The metrics that I usually use in Automation projects/initiatives with a decent maturity level are: Automation Coverage, Test Execution Savings in hrs or dollars, # of Defects found in a particular Sprint/Release, # of Defects found by automation Vs end user, defects by priority, Framework Maintenance Vs Automation Development, etc.

    Crunching all of these numbers is a full time job which I agree with you, it should be taken care by Managers, however, all data points should come from the Automation Team.

    1. Hi Carlos, good to have a more expansive description of the situation.

      I often measure things to improve them - but I want to measure temporarily until the situation has improved. e.g. you have a "Framework Maintenance vs Automation Development" - that might be a very useful thing to track for a while if in retrospectives people say "we are spending too much time maintaining it" - then you can measure to find out if that is true, then take action to make maintenance take less time, then measure to track if the problem has gone away, and then drop the metric. But there might also be other ways to spot this problem - stories taking too long to deliver because so much time is required to maintain the existing code base.

      I don't think all the measures you are reporting are for improvement, some seem like justification of spending time and cost.

      But, if you and the people you are reporting these to find the numbers valuable, and they are helping you improve, and the time and cost spend gathering and reviewing the numbers is seem as a useful spend then you can feel justified in continuing to do it.

      People and projects operate in different ways. Best of luck.