Monday, 21 March 2016

Behind the Scenes - "Dear Evil Tester" lessons in risk based release management

On Wednesday the 16th a paperback copy of “Dear Evil Tester” arrived through the post. This was my “final” proof copy. Was it ready for release? In this episode we learn that the “Go Live” release decision is always a business risk decision.


Errors found in the staging environment

Upon reading it, I found two tiny errors.
Did I delay publishing?
Heck no.
I’m Agile. I’m lean (with a thin layer of fat for winter).
Did I fix the errors?

Errors found late cost more to fix than early

Heck yes.
What were they?
Two typos.
  • I had written a " instead of a :
  • I missed out one character from the end of a word
I could probably have left the second error. But the first one annoyed me. Especially since I was sure I had fixed it prior to ordering the proof.
I could have hit publish, there and then, but I didn’t I fixed the error.

Integration Testing

But I don’t have any integration tests. How could I check I had fixed it?
I needed to improve my test approach. Previously I had been scanning the changed pages, and flipping through the pages to check for no unexpected changes. Then order a proof. Then wait 3 days. Then repeat.
This time I found a tool to help. diff-pdf
Perhaps, I could generate a ‘print ready pdf’ and ‘diff’ it with the previous one.
It worked. The diff showed me two pages with changes, and both exactly where I expected them.
I was pretty sure this was good enough to de-risk the release process.

Release Process

But I still had to go through a release process.
I had to resubmit the print ready proof to the CreateSpace process.
Then wait for the ‘we check your document’ process to complete.
But that would take too long. I had already started the “Coming Soon” hype machine. I couldn’t dial it back into “Coming a little bit less soon than I wanted, but still Coming Soon” mode. Or I could, but I didn’t want to.
So on Thursday morning. After the CreateSpace approval process was finished I was able to review the digital proof of the book.
I wasn’t so sure this was good enough to de-risk the release process. Because I didn’t gain any information. I didn’t notice any problems. I didn’t think there were any problems. So this just confirmed my world view.
In theory, this should mean that I found no additional risks and issues, and so should be good to go.
As a tester, I’m not used to finding no problems. I must have missed something.
I could order a print copy, wait a few days, then compare the text, and then release. If I did that I probably wouldn’t release the book for another 5 days.
What to do? What to do?

What to do?

On Thursday.
I ordered a print proof.
And I went live.
I thought - if there was a problem then I would receive the proof copy before anyone received there copy.

Friday, Saturday, Sunday

On Friday, people started tweeting that they had received their print copies from Amazon.
HUH! WHAT! My proof still hadn’t arrived!
On Saturday, people tweeted that they had received their print copies from Amazon.
WAIT! But my proof copy still hadn’t arrived!
On Sunday… etc.

Monday

And lo! The proof copy arrived.
I read through it and it looks fine.

Lessons learned

  • Release Go Live decisions are always a risk based decision.
  • Sometimes business priorities will take precedence over ‘testing’ risks and concerns.
  • Tools can help.
  • You have to know what you need a tool to do, before you find one.
  • If you self-publish, then try to order your final proof from Amazon rather than CreateSpace. It will arrive faster.

No comments:

Post a Comment