Saturday, 31 December 2016

Happy Testing New Year (for 2017)

Happy Testing New Year (for 2017)

TLDR; Happy New Year - And all the best for 2017 - resolve to make changes every day - don’t rely on a single point in the year.

I try to avoid New Year’s Resolutions.

I think we need to resolve to make changes every day.

But… The Evil Tester Sloganizer has a resolution template

Friday, 30 December 2016

How to configure Postman Native Client App to use an external proxy

TLDR; to configure Postman native client app to use a proxy start it with command line parameter --proxy-server=host:port

postman.exe --proxy-server=localhost:8888

One of things I like about Postman Chrome App is that, because it uses Chrome, it is really easy to feed through a debug proxy like Fiddler, or Zap, etc. etc.

But since Chrome Apps are going to die and Postman has a native app now, I thought I’d use that.

Uh oh! Where is the proxy setting?

Friday, 16 December 2016

Software Testing Definitions and Models - How and Why to Create your own Definition

TLDR; I avoided creating a definition of software testing for a long time because I thought it would constrain me. Instead I built a model of what I did. Eventually I generalised it to a ‘definition’ - “Testing is building and comparing models of a system to the system.”

Digital image courtesy of the Getty's Open Content Program.
[Digital image courtesy of the Getty's Open Content Program.]

From 29/11/2016 to 6/12/2016, every weekday, I used Instagram as a platform for creating a ‘daily thought’ on the topic of “Definition of Software Testing”. That was a useful process in its own, this blog post is to document the output.

What follows:
  • text description of the daily thoughts with additional commentary (keep reading)
  • the youtube video has annotated slides and the videos
  • you can find the slides on slideshare
  • raw instagram videos (see list at the end)

Monday, 12 December 2016

A testing challenge for you "I T.E.S.T with A.T.T.I.T.U.D.E"

TLDR; Rather than pursue viral marketing millions, I have a released a new application for you to use for experimenting with exploratory testing, techical web testing, and JavaScript hacking. You can find it at


I had a mad idea recently that I should start a merchandising section, for T-Shirts with slogans and images. And then I remembered that I tried that before and didn’t follow it up very well. But then I thought “This will be different, I’ll put slogans on them. People like slogans.”

Friday, 9 December 2016

Taking Software Testing More Seriously By Using Humour

TLDR; Everyone can tap into humour when they are themselves and with their friends. Can you respect your work peers enough to be yourself with them and tell hard truths using humour when necessary?

I was listening to an audio book by Grant Cardone this morning and he was talking about humour in the sales process, the author of this audio book. I was thinking, I use humour in software testing. I cover humour in the book Dear Evil Tester, and there’s a section in here in the afterword section.

This post is also available as a video - embedded below - or watch on

Thursday, 8 December 2016

Lessons learned and tips for testing from public bug bash events

TLDR; Tips for external public bug bash testers. This is not testing as you know it. Change your beliefs. Get ready in advance. Hit the system hard, hit it fast. Report quickly and well. Demoralise the other teams. “Congraturation! You Sucsess!”

Stop the Express - ZX Spectrum - from RZX Archive

Lessons learned and tips for testing from public bug bash events

There seem to be a few public ‘Software Testing Cups’ and competitions. Which all seems like jolly good fun. The closest I’ve come to taking part is when I attend public bug bashes. I’ve been to a few in the past. So I’ll list some pro-tips and experiences.

Wednesday, 7 December 2016

How to improve your CV and improve your chances of finding your next software testing job

TLDR; Write your CV as benefits, and build a portfolio, to become more than just another CV on the pile and help the busy CV reader see the value that you can bring to their team.

I’m trimming out an email backlog and came across a few “I can’t get a job - please review my CV and tell me what I’m doing wrong!” emails.

I’ve collated and distilled the advice I tend to give out, to help me trim out my email and an attempt to cut down on the number of CV’s I’m sent with people asking for “help” - although agencies and companies I’ll happily consult with you, for suitable remuneration, to help you solve your CV and recruitment issues.

Some of this may have appeared on this blog before. And if you’ve asked for my advice before don’t be alarmed. I have generalized and anonymised any personal details.

Wednesday, 30 November 2016

Notes on Leanpub podcast with Peter Yaworski - Web Hacking 101

I just finished listening to the Leanpub podcast with Peter Yaworksi.
There is a transcript with some links embedded on the podcast page

Actions Taken:
Next Actions:

Tuesday, 29 November 2016

A retrospective critique of an exploratory testing session

These are the notes for a critique of this Exploratory Testing Session on YouTube

You can watch the critique below on youtube


I picked Google Search because I thought:
  • an obvious software
  • simple to understand
  • at a high level just an input and an button
  • didn’t expect bugs to distract so could focus on thought process and execution
I was wrong.

Note: Time stamp links in the header are links to the YouTube video at the point where the retrospective notes discuss that section.

Thursday, 24 November 2016

Software Testing YouTube Channels to Subscribe to - and how to subscribe

TLDR; There are a lot of amazing YouTube channels to learn from. If you're not using subscriptions on YouTube with notifications or RSS then you are missing out.

When I started with YouTube I used it as a search engine and didn’t subscribe to any channels.

There are so many testing channels and conference channels that learning how to subscribe to YouTube channels will massively boost your learning opportunities. Use the list of recommended channels below and the instructions on how to subscribe to boost your learning opportunities.

Monday, 21 November 2016

A live exploratory testing session recorded

TLDR; Because we don't see many videos showing exploratory testing in action, where people test, and explain why they are testing. I created an up to date video. I encourage you to critique the testing.

I recorded an exploratory testing session. And its pretty raw with minimal edits.
  • I kept it to 15 mins.
  • It is painful for me to watch in places - you can probably see why.
  • I encourage you to repeat the session and see how you would approach it
  • Investigate the bug I found - is it a bug?
  • What tools would you use?
You can watch it on YouTube I'll release a full video critique of the session in a few days. 

Sunday, 13 November 2016

Announcing A New Test Tool - TestToolHub - Counterstrings, and more...

A long long time ago I started creating some coding exercises for myself - looking at the comments it was started at least at 08/12/2014 - possibly earlier.

They were:
  • How can I generate a counterstring?
  • How can I use Java to write counterstrings to the clipboard?
  • How can I generate a forward counterstring? i.e. don’t reverse a string
  • How can I type that counterstring into an input field as it is generated?
  • How can I find all the comments on an HTML page
  • How can I generate strings for unicode chars?
  • How can I create a set of clickable urls for different environments etc.
And all of this I put into

Tuesday, 25 October 2016

Q: when do we prefer manual testing over automated testing? A: Hmmm....

TLDR; As a simple rule, “I might not automate, if I have to create”. But I have done the opposite. I have automated when others would not. And I have waded in when others would have automated. To answer we need to question our situation and build a deeper model of our actual aims, options and capabilities.

Q: when do we prefer manual testing over automated testing?

A: I have a problem providing a simple answer to this question…

And not just because “it depends”.

Also because - “that’s not the model I use” so I don’t frame the question like that.

I don’t think in terms of “manual testing” and “automated testing”.

I think in terms of “testing”, “interacting directly with the system”, “automating predefined tasks, actions and paths”, “using tools”, etc.


If I’m testing then:
  • do tools already exist to help?
  • do I have abstractions that cover those areas?
  • do I have to create tools?
  • do libraries already exist?
I might automate heavily if I already have abstractions.

My testing might involve lots of automating if tools already exist.

I might not automate, if I have to create. When - if I create, the testing has to wait.

Don’t Automate Under These Circumstances…

So what about scenarios where you might expect “don’t automate” as my default answer:
  • something we need to do rarely
  • usability? “look and feel”
  • experimental features - surely we wouldn’t automate these
  • something still under development and changes frequently
Many people conduct stress and performance testing rarely. They generally automate.
“I might not automate, if I have to create.”
Usability and “look and feel” have tools to detect standards non-compliance. I might very well use those. So I might automate portions of this type of testing early.

Experimental features might fit into an existing process and so we might already have a lot of the scaffolding we need to support minimal and speedy automating - in which case I might automate those.

I have automated features that were changing frequently because we were testing the same ‘path’ through the application many, many times (every time it changed) and we often found problems (every single release) and the effort of testing it, by interacting with it, was waste. So we didn’t interact with it until the automated execution could pass reliably. But, that also depends on ‘what’ in the implementation changes, and how that change impacts the chosen approach to automate it.

What’s the real question?

If I have to ‘create’ stuff to help me automate as part of my process then I’m likely to weigh up the ‘cost’ of automating vs just getting on and interacting with it.

Are the assumptions behind the scenarios true?
  • Scenario: rare execution
  • Assumption: we would spend a lot of time automating something that we would rarely use
But perhaps we ‘rarely’ execute it because of the time and energy involved in setting up the environment and data, rather than the tooling. Perhaps we need to automate more, to allow us to ‘test’ it more frequently?
  • Scenario: experimental feature
  • Assumption: we write code to automate it that we throw away
But perhaps, the feature is so experimental that the easiest way to interact with it is by automating it, otherwise we have to spend a lot of time building the scaffolding required (GUI) for a user to interact with it.
  • Scenario: usability or “look and feel”
  • Assumption: this is subjective
But perhaps standards exist to guide us, and if we ignore those and all the existing tooling, then we create something so subjective that no-one else can use it.
  • Scenario: frequent change
  • Assumption: automating a scenario always takes longer than interacting with it and maintenance in the face of change takes too long
But perhaps, the GUI changes infrequently and the change takes place in the business rules behind the GUI. So really we are dealing with a single path, but lots of data. And perhaps we can randomize the data, and implement some generic ‘model based’ rules to check results.

Scenarios have more nuances in the real world

Very often we discuss these ‘when should we automate’ questions as hypothetical scenarios. And if we do, we need to drill into that scenario in more detail in order to come up with our answer. One of the main benefits to the exercise of ‘hypothetically…’ is the asking of questions and fleshing out a model to better explore the scenario.
  • I have automated when other people said I should not, and it saved time.
  • I have interacted manually using tools, when others said we should automate, and we found problems we would never have found if we had taken the ‘automate’ path so early.
  • I have manually engaged in stress and performance testing.
Sometimes, conducting a contrary experiment, will provide you with ‘examples’ that make it harder for you to answer these scenarios without extensive model building.

Thursday, 6 October 2016

A Case Study in Creating a Conference Presentation using Markdown

TLDR; using Marp and Pandoc I can write a report and presentation using Markdown, then add other tools to make it 'pretty'.

I’m going to show you a case study in how I’m planning a presentation for conference.

Because I want to offer you an alternative to diving straight into a powerpoint presentation.

My process for creating slides

What I’ve been doing is:
  • create an evernote
  • make all my notes in there
  • rough out some of the notes as summary info to add to a slide
  • create rough slides
  • expand my notes so they are like a ‘paper’ to support the presentation
Then I’ll:
  • move on to the slides
  • tidy up the slides
  • practice the presentation
  • keep the ‘paper’ up to date
And all of this means jumping between a few tools and I don’t really like working with presentation tools.

I’ve been looking for an alternative that fits my workflow.

An alternative that fits my workflow

And I think I might have found it. And I can:
  • write everything in markdown
  • version control my text
  • keep the ‘slides’ and the ‘paper’ together in the same file
  • create a pdf of the paper - with ‘embedded slides’ - without doing any exporting or editing
  • create a pdf of the slides - without any extra editing or maintenance
And now I think I’ve found a way of doing that - and of creating ‘nicely formatted and professional slides as well’ - all without touching a presentation tool.

So let’s have a look.

The Making Notes Phase

I’m still in the making notes phase for this presentation. This example is my presentation for TestBash Netherlands in January 2017. (official Test Bash site)

I make notes on it and restructure it when I get an idea - sometimes that means waking up at 3 in the morning and sitting in the office to type up the ideas, but that’s how I work on presentations.

In my earlier workflow, I would just have a set of notes at this point. And I would create those by running my ‘notes’ through pandoc to create a pdf.

With my new workflow. I already have:
  • a set of notes in Evernote
  • a pdf of the paper generated by Pandoc
  • a presentation (using Marp)
I could wing it tomorrow, if I had to.

This is a massive win.

I always like to be at the point that ‘I could do it tomorrow, if I had to’. But I’m never usually at that point so early in the preparation cycle.

What that means is that every iteration from now on is an improvement (hopefully).

I could wing it tomorrow!

So I have notes, and they look a bit like this:


# And I'm going to say that:

## "any software that I use to support my testing is a "Testing Tool"


So if I use Cucumber in my testing it becomes a "Testing Tool"

And you can argue about it all you want. It won't stop me using it, or misusing it (if you say its not a testing tool and I use it in my testing).



And when I say “they look a bit like this”, I mean “they look exactly like that”.

I copy and pasted the above from my notes.

And what you can see is:
  • a draft slide
  • slides a separated by ---
  • and my ‘paper’ inside the HTML comments
I would normally continue working like this, in Evernote and a text editor. Until it was time to create the slides.

I can create slides now!

By using Marp, I can save my notes to a text file.

The same text file that I would feed into pandoc.

And I can view my notes as a set of slides.

The reason I use <!-- and --> to delineate my ‘paper’ sections is because most Markdown processing tools ignore HTML comments.

Marp, ignores HTML comments and only displays the --- delineated parts as a set of slides.

Using Marp, I can:
  • view my notes as a set of slides
  • get a feel for my notes as a ‘presentation’
  • output a pdf that I could use to ‘wing it tomorrow’
The Marp slides output isn’t great, but it is certainly ‘good enough’, and I released the Java and Selenium Install Checklists to slideshare using Marp, directly from the checklist file that I maintain on github:
Same source document different rendering.

The PDF isn’t ‘perfect’ but its good enough for my purposes.

I can view the paper easily!

If I copy and paste the contents of the file or evernote into

I can see a rough version of the paper, but Dillinger shows me the <!-- and --> which isn’t great, but for review purposes this is fine and easy.

I couldn’t use Dillinger to release the paper in that condition though.

I can create a pdf paper easily!

I use pandoc to create PDFs.
If you’ve hired me for consultancy work - pretty much guaranteed that the PDF report you received at the end was generated in pandoc.
Trade Secret - I can create a report really quickly because:
  • I write it as I consult
  • When I make my notes in Evernote
  • I write in Markdown
  • I convert with Pandoc
  • I spend time at the end of the day (/trip back on the plane/journey back on the train/engagement in the airport lounge) editing and fixing spelling errors
  • generate in Pandoc - I tend not to spend much time reformatting
Shh, keep that a secret though, I don’t want my competitors to have the same advantages that I have.
Normally I just run a text file through pandoc:

pandoc mynotes.txt -f markdown -o professional_report.pdf

(pandoc can change page size and create a table of contents and output MS Word and OpenOffice files, but somethings I’ll keep as trade secrets.)

But pandoc would ignore the HTML comments as well, so I create a script:

(Get-Content slides.txt) | ForEach-Object { $_ -replace "<!--" , "-----" } | 
ForEach-Object { $_ -replace "-->","------" } | Set-Content
pandoc -f markdown -o output.pdf

Above is windows powershell which converts the HTML comments into Markdown horizontal rules.

I converted an open comment <!-- to five minus signs -----and close --> to six ------ to give me the option of ‘find and replacing’ the report back into a slide deck if I want to.

If I was on mac I’d probably create a .sh that used sed to perform the find and replace operation.

This gives me a fairly nicely formatted pdf.

Good enough.

It makes it easy for me to see:
  • where I have a lot of content,
  • where I have less
  • what is a quick slide
  • what is a slow slide
  • what is summary
  • what is detail
  • etc.

And the main point for me is that I get this with no extra effort.

I can imagine that to release the paper I would convert the comments to "" and avoid having extra horizontal lines.

But for the moment, I’m comfortable with the double lines, and the removal of comments is a one way operation.

Version control benefits

A great benefit from this is that the only things I need to version control now are:
  • a single text file
  • a conversion script
  • any images
I don’t need to add
  • a huge LibreOffice or Powerpoint presentation
And I have proper versioning so:
  • I can diff the notes and slides,
  • I can bring back lost information

But it ain’t that pretty at all

I’m not that bothered about the ‘prettyness’.

Normally I ‘prettify’ the slides close to the release date anyway.

But we can automate it all!

I’m currently evaluating Deckset.

A Mac only app which can take my ‘notes’ file with the slides and the HTML embedded ‘paper’ and can format it into something ‘pretty’:

I ‘just’ have to:
  • choose a template
  • pick the colours
  • then ‘tweak’ the Markdown to make it work better as a slide deck

Next Year…

I guess we’ll see in January 2017 how I actually create the slides.

And we’ll see which of the above slides actually make it from the ‘Notes’ phase, into the ‘Finalise Slidedeck’ phase.

Tools Mentioned
Bonus video showing the tools in action (available on youtube)

Thursday, 29 September 2016

Adhoc testing of incrementally developed HTML and CSS

TLDR; Even simple changes can use step by step risk based development to allow multiple releases, and will require multiple levels of testing.

My website is in a constant state of flux - that’s the benefit of writing it all by hand.

Today I wanted to add nifty social media icons on the page because I hear tales that there is gold to be made from the contacts that come from such things. And never one to turn my back on boundless riches, I wanted to add some social media and contact icons.

And why tell you this?

  • the testing process around this has relevance
  • I learned something about CSS and I assume other people might like to learn that too

Monday, 26 September 2016

How to Use Chrome Developer Tools JavaScript Source Snippets

TLDR; With Chrome Code Snippets we can ‘run’ small chunks of code without using the console. And using a gist we can import and export them to a file.

We already know that we can run custom JavaScript code from the console. I explained that in this blog post.
And we can use Google Chrome Code snippets to re-use that code over time.
By default we can’t import and export, but people have written helpful code to let us do that as well.

Friday, 23 September 2016

How to use the JavaScript Console for Technical Web Testing

TLDR; learn how to use the dev console and basic JavaScript commands (for, do…while, if…then, variables) to automate under the GUI and augment your interactive technical web testing.

I used to only use the JavaScript console in the Chrome Dev tools was for viewing JavaScript errors.
But I read the documentation for the JavaScript console
And realised that I could issue commands and view objects. And I slowly came to realise that I could use this to help me test better.

Thursday, 22 September 2016

How to Treat your Application as an API - "App as API"

TLDR; Look at what your GUI does. Automate that. And you’ve implemented “APP as API”

A phrase I’ve found myself using in various training sessions is “App as API”.

I think I’ve mentioned that in some previous blog posts but want to expand the concept here.
  • Some applications provide an API
  • Most applications provide a GUI
When the application provides an API, I like to take advantage of that. But…
  • Sometimes the API doesn’t do everything I want.
  • Sometimes the functionality I want to use is only available from the GUI
Do I let that stop me?


So what do I do?

Wednesday, 21 September 2016

How to write a simple random test data sentence generator

TLDR; I wrote a random test data generator, not a slogan generator

On the Importance of Test Data

We all know that test data is really really important.

If you don’t have the right data, you can’t test the functionality. You can’t check that an email is sent out on the customer’s 65th Birthday unless you have a customer who has a date of birth that will trigger that functionality.

Some data is path invariant and has to be specific to control the path.

We know this.

But we don’t always randomise enough data and our test data becomes stale and etc. etc.

Monday, 19 September 2016

Difference between hacking, cheating and automating?

TLDR; Hack for information. Cheating breaks the rules and introduces more risk. Automate the interfaces with less risk.

Previously on "Evil Tester goes Hacking JavaScript Games" I demonstrated a ‘bot’ that could play ZType

My bot cheated and automated shooting, I hacked

The bot has been through a few iterations:
  • 0 - every few seconds it uses an electro magnetic pulse and then gives itself a new emp, therefore having infinite emp devices
  • 1- every 100 milliseconds it iterates through the letters available and shoots that letter watch bot 1 beat a human
  • 2- fixes a bug in (1) so that it only shoots letters which are used on the level
  • 3- once it knows what word is currently being shot at, it uses the letters in that word, but when it doesn’t know the word it shoots all the letters on the level
  • 4 - only shoots letters on screen, every 10 milliseconds, either the start of a word then continues on the word
  • 5 - only shoots letters on screen, which are the start of a word, then focusses on that word to shoot it to bits (doesn’t wait 10 milliseconds to finish the word) - 100% efficiency
  • 6 - essentially bot 5, but only waits 2 milliseconds watch bot 6 in action
The different versions represent ‘cheating’ or ‘automating’.
I had to ‘hack’ to get the information I needed to ‘cheat’ with bot zero.

Friday, 16 September 2016

Lessons learned from Automating - Instantiated

TLDR; take small steps when automating, keep your code working at all times, automate at an appropriate interface

Working on a ZType bot reinforced a few lessons learned from automating that are important enough to draw attention to (again).
  • Your Code can gradually iterate to improve
  • You are allowed to stop improving the code
  • Start simple to add immediate value
  • Debug in small chunks
  • Readable code is understandable
  • Automate at the appropriate interface

Wednesday, 14 September 2016

Hacking JavaScript Games to improve your testing

TLDR; Learning to hack JavaScript will develop the skill of automating using the browser dev tools console.

For the last 6 months or so, I have, on and off, been learning to code JavaScript. That process has mainly involved:
  • writing some simple games
  • hacking other games to see how they were written
I’m going back to the good old days when we could ‘break’ the ZX Spectrum game and view the source, or disassemble the games on the Atari ST and hook/hack them through debuggers and monitors.
To do all of that now, JavaScript and the Browser Dev tools are perfect companions.

Tuesday, 13 September 2016

Is there a difference between "Responsive Web Testing" and "Cross Browser Testing"?

TLDR; Testing responsive web does not mean test it on lots of devices and browsers. Look at the risk associated with your technical implementation and test those. You might still have to use lots of devices and browsers.

When you test your web application, do you differentiate between “Responsive Web Testing” and “Cross Browser Testing”?

What is Responsive Web Design?

I think people still argue about “Responsive Web Design”.
I use the term to mean how the website responds to different rendering environments.
You can read more about responsive web design:
For my purposes here, I’m going to say that Responsive Web Design is responding to the size of the screen. This might be done via CSS or JavaScript or server-side based on browser headers.
So, how do we test it?

Introducing Pious Sanctimonious Standard Compliance Boy

TLDR; Browsers Lie To Us
What is worse - a sanctimonious ex-somethingorotherer or a zealous recent convert?
I guess it doesn’t really matter, I didn’t choose one role above t’other.
After my HTML Validation experience
I adopted the persona of “Pious Sanctimonious Boy” and went to work.

Friday, 9 September 2016

Batch validation of HTML as part of your web testing with Total Validator

TLDR; Total Validator Pro will spider a site and check its HTML as well as links. Free version works 1 page at a time. 30 GBP for the pro version.

I started using Total Validator Pro as well as the w3 validator (the API which powers the ‘validate’ function in Charles).
I wanted to add a ‘batch’ checking into my HTML validation process.
And Total Validator Pro seemed pretty simple, cheap and cross platform, so I tried it out.

Thursday, 8 September 2016

How to add HTML validation into your test process without even trying

TLDR; Charles proxy can use and report results in the proxy GUI

After looking at my default test process for web and realising it didn't include HTML validation by default, I decided to find the easiest way I could to add it in to my process.

I generally find that the easier I can slot something into my process, the more likely I am to use it.

We all use Dev Tools by default now

Dev tools are bundled with every browser now, so we all use them. Right?

Previously we had to download plugins and add ons. But now that it is a right click away by default, we have no excuse.

But HTML validation doesn't appear to be there. I can 'audit' the performance of the page in the Chrome dev tools, and the CSS usage checking is quite useful. But no HTML validation.

Which is the higher technical risk to your products functionality? CSS? Speed? HTML?

What could possibly go wrong? And what to do about it?

TLDR; We test based on risk. If we don’t identify risk we don’t test for it. Automated tools can reveal risk that our technical knowledge can not.

True story:

A timeline of a true story of software development:
  • 15:00 I get an idea for an ‘app’ so start making notes
  • 15:30 decide it will be faster if I just start coding it
  • 17:30 basic version of "The Evil Tester Sloganizer" done
    • it is unstyled, but generates slogans
  • 21:30 add some styling and a ‘tweet this’ slogan button, and a hacked up navigation bar
  • 10:30 what the heck, I’ll just release it
  • 10:45 what the heck, I’ll just announce it as beta, because then it doesn’t really matter if it works, right?
Whoa there!
Did you even test it?

Tuesday, 6 September 2016

Dear Evil Tester: "I think that if I 'learn automation' I can get a new job FAST"

TLDR; If you want a new job, then build a portfolio while you 'learn'

Following on from the ’How do I learn ‘automation’?" question. We had to ask questions to see what they would do with this newly learned ability to automate. If it was to find a job, then how long did they want to wait? And did they have a specific job in mind?
If there was no job in mind, and they wanted it fast then that is not a good combination for a ‘how to’ answer to the question they asked.

If you have no job in mind

Chances are, I probably won’t recommend ‘learn automation’ to you.

Dear Evil Tester: "I think that if I 'learn automation' I can get this specific type of job"

TLDR; Learn with an end goal in mind. That will keep you focussed.

Following on from the ’How do I learn ‘automation’?" question. We had to ask questions to see what they would do with this newly learned ability to automate. If it was to find a job, and they had a specific type of job in mind then I think we could probably answer the question.

If you have a better job in mind

For the marketplace, if you want to ‘learn automation’ because:
  • you want to get a new job in a specific organisation,
  • you want to work with a specific technology
You have a better idea of the job you want in mind. You aren’t just competing for a general ‘job’.

Dear Evil Tester "I want to keep my testing skills up to date otherwise I might not get another job"

Following on from the ’How do I learn ‘automation’?" question. We had to ask questions to see what they would do with this newly learned ability to automate. And because they want to perform ongoing work and this is part of their learning plan we can answer.

I want to keep my testing skills up to date

This is great. You have a job, use it.

Monday, 5 September 2016

A Model of Automating

TLDR; I automate tasks by using tools and writing code. Understand what they abstract to learn more.

A Model of Automating

What do you automate?

I automate tasks.

Sometimes I use a tool to automate tasks. Sometimes I write code.
But of course my model that underlies that statement doesn’t have as clear a separation as the words suggest.
My ‘tool’ and ‘code’ models do not have a clear separation. They overlap. I use tools when I code. I can write code to extend some tools. I can wrap tools in code.
I can model ‘Tools’ and ‘Code’ as a ‘spectrum’.
  • Tool <-> Code
    • “Easy Entry Point” <-> “Longer Learning Curve”
    • “Limited to Tool Features” <-> “Ultimate Flexibility”
    • etc.

Friday, 2 September 2016

Dear Evil Tester: “How can I learn Automation?”

TLDR; Don't just 'learn automation'. Learn it for a reason.

When I attend conferences and meetups, people often ask me how they can learn to automate stuff.
I then have to ask some follow up questions. This is not my normal approach for a "Dear Evil Tester" answer.
A "Dear Evil Tester" answer makes assumptions and takes the question and the words in the question at face value. Because that is funnier, and allows me to address generic points. But doesn’t always target a specific person’s need.
When dealing with questions from people, I try to spend a little time understanding what underpins the question, to build a better model of their world, to try and provide an answer that helps them.
I might ask a few questions before making any statements. My first statements might be to check my model of their world rather than to start answering the question, and based on their response, I might ask more questions or start to answer their question.
Follow up questions to “How can I learn Automation?” might be:
  • what do you want to automate?
  • are you trying to automate something specific at work?
  • what programming languages can you write code in?
  • how will learning automation help you?

Thursday, 18 August 2016

Remove X-Frame-Options header when testing responsive web applications and sites

TLDR; Some sites issue an X-FRAME-OPTIONS header that stops them loading in an iframe, remove it for responsive web testing with a proxy.

Does your favourite responsive testing tool fail to display the site you want to test? Have you blamed the tool and now rely on browser dev tools? Well, here’s a possible solution for you.
But before that, let’s look at that situation in more detail:
  • you want to practice your responsive web testing,
  • you have your responsive web testing tool
  • it loads your chosen web site into an iframe
  • your chosen web site doesn’t want to load in an iframe
  • your tool displays a blank image, with no error, instead of the page
  • you curse and go back to the browser dev tools

Friday, 22 July 2016

We test REST even if it isn't REST

The Microsoft REST guidelines are being critiqued for not being particularly RESTful.
Pretty much every team I’ve worked on has had arguments about their implementation of REST APIs.
  • Versioning?
  • URI formats?
  • Query strings or paths?
  • Which return code?
  • Support OPTION?
  • Custom headers?
Those discussions are important for the design, but sometimes people get hung up on disagreements about theory.

On Workarounds and Fixes

TLDR: fixes are important. when we have work to do, sometimes a workaround is more important

One of the skills I think we develop when working with software is a search for workarounds.

I was reminded of that this morning when the start button on Windows 10 refused to work.

Friday, 15 July 2016

An Open Answer to an Open Letter

TLDR; Condemn? No. Support? I like parts of the paper. Censor? No.
And now I’m doing something I don’t like: writing a blog post in response to ‘something on the internet’. I’m writing a blog post which I don’t think has any practical value. I warn you now. I don’t think you will find much humour herein either. I have added a comedy punchline at the bottom though, if you want to skip ahead.
Chris McMahon has written an open letter to named reviewers of James Bach and Michael Bolton’s paper on “Context Driven Approach to Automation in Testing”. I reviewed the paper.
In the Open Letter Chris McMahon asks the reviewers three things:
  • condemn publicly “both the tone and the substance of that paper”
  • If you do support the paper, I ask you to do so publicly.
  • “regardless of your view, I request that you ask the authors” … “to remove that paper from public view” because it “is an impediment to reasonable discussion and it has no place in the modern discourse about test automation”
I find a few things personally difficult here:
  • I don’t feel strongly about the paper. There are parts I like, parts I agree with, parts I disagree with, parts I don’t like as much. Much like any paper.
  • I don’t like censorship

Friday, 17 June 2016

Register for Free Risk/Exploratory/Technical Testing Webinar on 28th June 2016

QASymphony have kindly invited me to present a webinar for them entitled "Risk Mitigation Using Exploratory and Technical Testing".

You can register to watch the webinar live. If you can't make it, then register and you'll be sent details of the free replay.

I've talked about Technical Testing and Exploratory Testing before. This time I want to approach it from the perspective of risk.

Monday, 13 June 2016

Text Adventure Games for Testers

TL;DR Announcing RestMud, a free text adventure game designed to improve your technical testing skills.

I love text adventure games. Playing. Writing. Programming. Love 'em. And now I have created a text adventure game for testers to improve their technical testing skills.

Tuesday, 7 June 2016

Some "Dear Evil Tester" Book Reviews

After publishing “Dear Evil Tester” I forgot to setup a Google Alert, but I set one up a few days ago and this morning it picked up a ‘book review’.
As far as I know, there are two book reviews out there in the wild:
I never know quite what I'll find when I click on a book review, and having written a few reviews myself, I know how cutting they can be if the reader didn't get on with the book.
Fortunately both Jason and Mel seemed to enjoy the book.

Thursday, 19 May 2016

National Software Testing Conference 2016

On 17th May, 2016 I presented “The Art of Questioning to improve Software Testing, Agile and Automating” at the National Software Testing Conference.
It was only a 20 minute talk. So I had to focus on high level points very carefully.
You can read the slides over on my Conference Talk Page. I also wrote up a the process of pitching the talk, which might provide some value if you are curious about the process that goes on in the background for an invited talk.

Friday, 8 April 2016

How to Watch Repositories on Github via a NewsFeed

TL;DR subscribe to master commits on github with /commits/master.atom
There exist a lot of ‘lists’ and ‘notes’ on github, not just source code.
I would like to be able to be notified when these lists change.
There are official ways of watching repositories on github:
I primarily use news feeds through
The newsfeeds officially documented provides a bit too much information for me.
I really just want to know when new commits are pushed to master.

Wednesday, 6 April 2016

Behind the Scenes: Tools and workflow for blogging on blogger and writing for other reasons

TLDR; Write offline. Copy/Paste to online.

This blog is powered by blogger. I still haven’t spent a lot of time creating a template that formats it nicely. Partly because I tend to read all my blog feeds through so I really don’t know what anyone’s blog looks like. I have ‘fix blogger formatting’ on my todo list, but it never seems to rise to the top.

I don’t particularly like the way that blogger uses html for posts: it avoids paragraphs and uses span, div and br.

But, it is easy and performant, so I use it.

What I don’t do, however, is write my posts in the blogger editor.

I thought I’d give a quick overview of my publishing and writing process for this blog because this is the same process I use when I’m working on Wordpress, Most Wikis, Jira, etc.

  • write in evernote using markdown
  • copy paste markdown to
  • “export as” “HTML”
  • open downloaded .html file
  • view source
  • copy paste everything between <body></body> into the ‘HTML’ view in blogger
  • review in preview in blogger
  • publish
  • review published form


  1. Web apps crash when I use them for editing
  2. I have a record of when I wrote the blog post because it is part of my Daily Notes ‘note’
  3. Writing in markdown means I focus on the content rather than the formatting
  4. multiple review points (yes, my writing goes through multiple reviews and still ends up like this!) each one shows a slightly different ‘view’ of it, so I pick up different errors.

##Web apps crash when I use them for editing

  • I’ve lost work in Wordpress.
  • I’ve lost defects raised in Jira.
  • I’ve lost edits to wiki pages.

You name the system that allows you to ‘create’ and ‘edit’ the ‘things’ online, and I’ve lost edits to it when:

  • the browser crashed
  • the browser hung
  • the tab froze
  • I accidentally pressed some magic button on the mouse that made everything go mental
  • etc.

I don’t trust online editing, so I do most of my writing offline in evernote or a text editor.

Secondary Gain

Because I’m writing it offline I have a record of when I wrote the blog post because it is part of my Daily Notes ‘note’.

Although Evernote seems to be slowing down these days when I write long notes. I don’t think it used to do this, I may have to start moving back to a ‘Day Notes’ txt file by default and import into Evernote at the end of the day.

Content rather than format

I have no fancy icons and gimmicks to distract me from my writing. Which means you get top quality content and no padding. Actually you probably get first draft text, but at least you know I wasn’t distracted by formatting.

Multiple Review Points.

Yes, my writing goes through multiple reviews and still ends up like this!

I first review the text in Evernote. Then in Preview in the blogger editor and then on the page after publishing.

Each stage shows a slightly different ‘view’ of it, so I pick up different errors.

If I do fix for formatting it is usually after publish, when it is live.


I write this way for most of the stuff I write.

  • emails
  • tweets
  • client reports
  • birthday card greetings
  • you name it

I also do this for my testing notes and test summary reports.

Which neatly brings us back to the topic of testing.

Happy testing.

Thursday, 31 March 2016

Everyday Browsing to improve your web testing skills - Why?

Who doesn’t like looking at the innards of a web page and fiddling with it?
  • Inspect Element
  • Find the src attribute of an image on a page
  • Edit it to the url of another, different image
You could, as my son enjoys doing; visit your school website, and replace images of people with blog fish and much hilarity doth ensue.
In my Sigist slides you’ll find some ‘tips’ for improving your web technical skills which cover this type of skill.
Asking and investigating:
  • How is the site doing that?
  • What are the risks of doing that?
  • Could you test that?
  • Do you understand it?
Some people ask: Why would I need to learn this stuff? Why would I use this?
I find that interesting. They have a different core set of beliefs underpinning their approach to testing than I do. They test differently, but it means I have to explain ‘why?’ for something that I do ‘because’.
I have studied the testing domain. I’ve read a lot of ‘testing’ books and have a fairly sound grasp of the testings techniques, principles and the many, varied reasons, why we might test.
None of those books, described the technology of the system.
Very few of those books used the technology of the system as a way of identifying risk.
Risk tends to be presented as something associated with the business. Business Risk. e.g. "Risk of loss of money if the user can’t do X".
I spend a lot of time on projects understanding the technology, to identify risk in how we are using the technology and putting it together.
  • If we are using multiple databases and they are replicating information across to stay in synch, then is there a risk that a user might visit the site and see one set of data, then the next visit see a different set (because now they are connected to a database that hasn’t had the information replicated across to it?).
  • Is there a risk that something goes wrong when we visit the site and it is pulling out information from a database that is currently being synched to?
If I ask questions like that and people don’t know the answers then I think we don’t understand the technology well enough and there might be a risk of that happening. I would need to learn the technology more to find out.
If people do know, and we have ‘strategies’ for coping with it - our load balancer directs the same user to the same database. Are there any risks with our implementation of that strategy? Will our test environment have the same implementation? Could we even encounter a manifestation of this risk when we are testing?
As well as knowing the requirements. I want to understand the pieces, and how they are put together.
Because I know from putting together plastic models as a child, or flatpack furniture as an adult, that there are risks associated with putting things together.
I have to learn about technology to do this. I then have to interpret that technology with my ‘testing mind’ and in terms of the system of the project I’m working on.
I suspect that is a better ‘why?’ answer; for learning the technology, and the associated technical skills, than my more flippant:
  • Q: Why would I use this?
  • A: Well, if you don’t have the skill, you never will. If you learn it right, then you might.
You'll find some simple tasks to help expand this in my Sigist slides 

Monday, 21 March 2016

Behind the Scenes - "Dear Evil Tester" lessons in risk based release management

On Wednesday the 16th a paperback copy of “Dear Evil Tester” arrived through the post. This was my “final” proof copy. Was it ready for release? In this episode we learn that the “Go Live” release decision is always a business risk decision.

Errors found in the staging environment

Upon reading it, I found two tiny errors.
Did I delay publishing?
Heck no.
I’m Agile. I’m lean (with a thin layer of fat for winter).
Did I fix the errors?

Errors found late cost more to fix than early

Heck yes.
What were they?
Two typos.
  • I had written a " instead of a :
  • I missed out one character from the end of a word
I could probably have left the second error. But the first one annoyed me. Especially since I was sure I had fixed it prior to ordering the proof.
I could have hit publish, there and then, but I didn’t I fixed the error.

Integration Testing

But I don’t have any integration tests. How could I check I had fixed it?
I needed to improve my test approach. Previously I had been scanning the changed pages, and flipping through the pages to check for no unexpected changes. Then order a proof. Then wait 3 days. Then repeat.
This time I found a tool to help. diff-pdf
Perhaps, I could generate a ‘print ready pdf’ and ‘diff’ it with the previous one.
It worked. The diff showed me two pages with changes, and both exactly where I expected them.
I was pretty sure this was good enough to de-risk the release process.

Release Process

But I still had to go through a release process.
I had to resubmit the print ready proof to the CreateSpace process.
Then wait for the ‘we check your document’ process to complete.
But that would take too long. I had already started the “Coming Soon” hype machine. I couldn’t dial it back into “Coming a little bit less soon than I wanted, but still Coming Soon” mode. Or I could, but I didn’t want to.
So on Thursday morning. After the CreateSpace approval process was finished I was able to review the digital proof of the book.
I wasn’t so sure this was good enough to de-risk the release process. Because I didn’t gain any information. I didn’t notice any problems. I didn’t think there were any problems. So this just confirmed my world view.
In theory, this should mean that I found no additional risks and issues, and so should be good to go.
As a tester, I’m not used to finding no problems. I must have missed something.
I could order a print copy, wait a few days, then compare the text, and then release. If I did that I probably wouldn’t release the book for another 5 days.
What to do? What to do?

What to do?

On Thursday.
I ordered a print proof.
And I went live.
I thought - if there was a problem then I would receive the proof copy before anyone received there copy.

Friday, Saturday, Sunday

On Friday, people started tweeting that they had received their print copies from Amazon.
HUH! WHAT! My proof still hadn’t arrived!
On Saturday, people tweeted that they had received their print copies from Amazon.
WAIT! But my proof copy still hadn’t arrived!
On Sunday… etc.


And lo! The proof copy arrived.
I read through it and it looks fine.

Lessons learned

  • Release Go Live decisions are always a risk based decision.
  • Sometimes business priorities will take precedence over ‘testing’ risks and concerns.
  • Tools can help.
  • You have to know what you need a tool to do, before you find one.
  • If you self-publish, then try to order your final proof from Amazon rather than CreateSpace. It will arrive faster.

Friday, 18 March 2016

Behind the Scenes - The "Dear Evil Tester" Style Guide and its Impact on my Testing

During the editing stages of "Dear Evil Tester" I started to write and maintain a style guide.
This was to help me track the ‘basic’ errors I found in my writing, and as an attempt to ensure a consistent attitude and formatting in the text.
Most books, and "Dear Evil Tester" is no exception, have a ‘conventions used in this book section’. This is a mini style guide which contains a set of small templates that I can use during the writing process. This section doesn’t go so far as to list the weaknesses of the writer, which is one reason for making it public. Whereas the weaknesses can be inferred from the reading of a style guide, so we keep it secret.
I’ve embedded the style guide below.
But first… a quick reflection since I will carry this approach forward into my testing.
I generally make a lot of notes when I’m testing. I do not always summarise them into a concise guide. I will now.
I’ll write down the:
  • handy tips,
  • obvious errors I commit,
  • reminders of high level aims.
Previously I’ve relied on my archive of notes and ‘searching’ through them, which has worked well. But I think that the additional exercise of reviewing, and summarising, will help embed the knowledge in my head which might even cut down on the searching.
It should also make it possible to share with other people.
I know that we have tried to do this on projects through wiki’s but, they often fall into disuse. Partly because they are written to be ‘read’, rather than ‘remind’, so are often wordier than they need to be. Also they tend to be written for ‘others’ rather than for ‘ourselves’.
Having the guide for my testing, might also help ‘others’ spot weaknesses in my approach. For example, if I haven’t listed a shortcut/feature in the tools then I might be missing out on something that might help my testing. The guide might reveal that to someone more expert than myself.
I’ve had in mind to build these for each of my websites. When I do, this will also have the side-effect of creating a single page that I can quickly review for any obvious CSS or common JavaScript issues.
And now, the style guide…

Evil Tester Writing Style Guide

This is the ‘writing style guide’ to support editors for Dear Evil Tester.
But Alan, that’s you! Yeah, and I forget stuff. OK!

Sublime Tips:

  • ctrl + shift + F - Find across all files
  • F6 - Spellcheck

Publishing Tips

  • Remember to check in to Version Control before starting a preview
  • Remember to deploy to Dropbox before starting a preview
  • Remember to close down PDF viewer, before starting a preview

Editing reminders

If you find an error. “Find across all files” and see if you find it again - if you do, fix it!
When reading from print, to find the file, use “Find across all files”

Writing Style

Worry less about the sentence structure. Worry more about the flow. Think. What would Norvell Page write? What would Lester Dent write? Do not model H. P. Lovecraft.
Is it a sentence? Who cares. Just make sure it looks like one. And reads like one.
But we’re not supposed to start with a conjunction!
And who said that? Was it someone who started a sentence with “however”? What would the author of the Bible write? (Hint: And in paragraph two, they started with “And”! If its good enough for God…)
  • If you have (brackets) and have punctuation outside the brackets (like this), then it is only outside if it started outside. (Having a full stop outside this would be bad.)
  • Its vs it’s (remember: find across all files) - It’s wrong to write its, unless its thing is being described.
  • Don’t worry too much about capitalisation, but at least make it consistent in each letter.
  • PS. PPS. (Don’t make it a dotty abbreviation.)
  • Capitals for Things,
    • at the very least don’t mix it. “exploratory Testing” would be bad.
  • Chapter and Section Titles use Capitals
    • Letter chapters are written like sentences. Unless they have a Thing in it.
  • For example, we will not write For Example. And we will still write e.g.
But what about the readers that know grammar?
Ha. They’re going to be annoyed however you write it. Write it for the masses.

Recurring Jokes

If you ever get the opportunity to make a joke about repetition then you should take it. Copy and paste a previous section of text again. Preferably the previous sentence or paragraph. Some people will get it. Some people won’t notice. Some will call the grammar police. Either way, much hilarity shall ensue. So if you ever get the opportunity to make a joke about repetition then you should take it.

Thursday, 17 March 2016

Behind the scenes of Dear Evil Tester : Hitting Publish

Today became "Dear Evil Tester" launch day.
I suspect ‘professionals’ create a plan and stick to it. I’m more of a kanban, ship it when it is ready, kind of guy.
On the 16th March, I received a final proof copy, and I reviewed that. Then on the 17th I hit publish.
I found the free pdf comparison tool "diff-pdf" very useful for final leanpub print ready pdf comparison during my final proof phase.
The launch process proceeded as follows:
  • start at leanpub
  • check the book details
  • realise you haven’t added a price
  • add price (make it slightly cheaper than amazon kindle because of the difference in royalty rates)
  • click publish
  • check the page looks OK in incognito mode
  • tweet
  • tweet that a sample is available now so people can try before they buy
  • move on to createspace
  • click publish
  • stare at the message that says “this might take 3-4 days” and wonder how people manage to create a co-ordinated launch across all platforms
  • move on to kdp (kindle direct publishing)
  • click publish
  • again stare aghast at the “this might take 3-4 days” message
  • be amazed as you start responding to peeps tweets about buying the book. Thank you all.
  • see that have listed the book for sale
  • tweet that
  • start editing all your web pages
  • notice that the feedjs server you were relying on for your rss feeds has stopped responding, so create a new endpoint for your existing rss caching to work on different sites
  • notice that the kindle version has been published
  • tweet that
  • respond to more peeps tweets. Thank you all again.
  • Write a promotional blog post
I still have more to do in the launch.
  • make sure Amazon notice that the paperback and kindle book are actually the same book and are listed together
  • create a leanpub promo video
  • amend the sales pages
But overall, the ‘publish’ part of ‘self publishing’ has become quite smooth.
Thank you for your support.

Wednesday, 16 March 2016

Sigist Keynote March 2016 Notes

I have updated my resources page for the Sigist March 2016 Keynote to include the slide deck and the social media commentary that Lisa Crispin kindly provided during the talk. (The above photo was taken by Lisa during the talk)

The talk was possibly partially recorded and might partially make its way to YouTube. The Sigist team are experimenting with recording technology and experiencing the normal teething troubles associated with live video recording. I'll let you know if the recording makes its way to live.

During the talk the Sigist kindly provided two copies of "Java For Testers" that we gave away as prizes, and I brought along a unique 'proof' copy of "Dear Evil Tester", which I also gave away as a prize.

In the talk I was basically providing some case studies of using technical knowledge and skills to inform your testing. And gave some examples of the overlap between this style and security testing.

I also mentioned the Usborne computer books from the 80's, many of which Usborne have released as free pdfs. There are a few on the website that I do not own, so I will read those later.

These books were a major influence on my career. I learned to write adventure games using "Write your own Adventure Programs"

A book that I previously mentioned over on Selenium Simplified where I describe the relationship between Keyword Driven Test Frameworks and Text Adventure Verb Noun parsers.

I still use the lessons I learned in this book, to this day. 

I later augmented this information with another 'Dragon' book:

Monday, 14 March 2016

How to draw Evil Tester

"Dear Evil Tester" includes some Evil Tester images and comics.

In this post I will show the basic process I use to go from a scribbled, draft, pencil drawing, to an inked and digital comic.

You'll see (in super blurry vision because of the web cam autofocus) the draft A3 sheets.

Then we move to an A4 pencil sketch. And then to an A4 ink sketch.

And we'll finish with a quick demo of the tools we need to scan it in and polish it up on the computer.

To scan the drawings I use a Canon Lide 220. Small, easy to store, USB powered (minimises cable and setup hassle), portable to different machines in different rooms. Fast enough. (For document scanning with multiple pages I use a ScanSnap ix500)

I scan into Paint .Net and I use Riot to shrink the file sizes.

Friday, 11 March 2016

Behind the scenes: Dear Evil Tester Book editing continues with testing hints on Observation

A proof copy of "Dear Evil Tester" just popped through the door from CreateSpace.

So now, with a different observation tool (the book instead of the printout). I will start to spot different errors.

And we enter the next stage of editing. Hopefully after these edits and fixes, only one more proof copy will be required before we go to print.

Observation is one of the key parts of my simple testing model:

  • Modelling, Observation, Reflection, Intent, Manipulation

This model changes for technical testing to:

  • Modelling, Observation, Reflection (contains intent), Interrogation, Manipulation

Different tools change the way that we observe. Different printouts are no different. So now we enter the 'physical' proof editing stage.

Coming even sooner than before: Dear Evil Tester

Behind the scenes: publishing and editing Dear Evil Tester book on leanpub

A quick behind the scenes look at the pre-proof editing process used when I'm writing a book via

In this video you will see all the tips and tricks of the trade.

The super secret special pen that I use for editing (clue: its a "Fisher Military Space Pen Matt Black").

The hi-tech editing software in use (clue: its a text editor, in this case Sublime Text).

The hi-tech version control software in use (clue: its subversion because I'm like that at home).

The sophisticated deploy system (clue: a windows batch file).

The hi-tech local to remote environment synchronisation software (clue: Dropbox)

And the highly automated preview process (clue: I click on a link in

And other splendid Dear Evil Tester goodness.

Stay tuned for more "Behind The Scenes" videos as we approach launch of "Dear Evil Tester"

Thursday, 10 March 2016

Announcing "Dear Evil Tester" coming soon, and why I wrote it

"Dear Evil Tester" is coming soon.

It is funny.

And that's fairly revolutionary in the testing market although Rob Sabourin's "I am a bug" and Andy Glover's "Cartoon Tester" provided humour in book form before I did. And this book contains the kind of humour that I relate to most.

Huib Schoots said "Alan writes with a dark humour that I like a lot!" Thanks Huib, I like that a lot too.
Richard Bradshaw called it "a warm dark blanket of humour and wit". Thanks Richard.

I wrote "Dear Evil Tester" because I see a gap in the coverage of current testing books. I don't see many testing books which cover the 'Attitude' that we need and want testers to build up.

This is a Question and Answer book.

And the answers are written in a way that I needed/wanted someone to talk to me when I was growing up as a tester. Someone who's answers would force me to think about whether I thought what we were going was a good idea and caused me to take responsibility for my test approach. Not because it was what the process said, or because it was what the 'experts' say, instead because it was what we say.

I learned to do this.

Hopefully with enough experience we learn to do this. So testers who have more experience will be able to relate to the book on this level. And testers with less experience will hopefully have that experience as they read the book.

We can use humour to challenge how we think, just as we use humour as a defense mechanism. In some ways I've drawn upon my readings of Provocative Therapy with a "Devil's Advocate" approach to provide answers which lead to a change of attitude.

Rob Sabourin said "I laughed - then I cried - then I laughed harder - then I cried softer." Thanks Rob, it moved me too :)

I'm working through the final edits now, if you want to be informed when the book is released then you can register your interest on the page.

I hope you'll like it. And stay tuned because I'll be showing you some behind the scenes footage as I complete the editing process.

Tuesday, 8 March 2016

Video Version of Lessons Learned Testing the House of Test contest app

I know that some people will have read, and been able to implement the notes I published after testing the House of Test Data Generation contest application.

I also know that a video can help people overcome issues trying to implement the notes because you'll be able to see my testing in action and the steps I take to create the project and add the jar to the project etc.

So I have turned the case study notes into a set of free videos.

I have uploaded the first video to YouTube, if you want to watch it prior to signing up to watch the videos for free on the training site.

Monday, 29 February 2016

Lessons Learned Testing the House of Test contest app

Lessons Learned Testing the House of Test contest app

Martin Nilsson and House of Test wrote a small ‘test tool’ which they released as a competition app. I’m sure they’ll release the app later.
This is a mix of ‘testing notes’ and ‘thoughts on how to test’ this type of app. I spent about an hour or so looking at the app, so the notes are a little scrappy. I spent about 40 mins or so tidying them up for publication, in the hope that they help someone.


Martin said:
“A small challenge for all you testers out there: Test the Crappy Little DataGenerator:"


  • Download .jar
  • run .jar
  • see/use GUI
Obvious errors:
  • truncation of text
  • spelling errors in tooltips
  • spelling errors in options
    • e.g. “Captial Ch…"
I could at this point, resize, hover for tool tip text etc. And I could test it blind, but what if I get more information first?
How can I observe more about what this app is doing?
Has logging built in.
Try that first.
Hint: Always identify how to use the thing to test the thing
Turn on logging. Do something. Hunt for file. Look in running directory first. Found File.
  • tail file
tail -f DataGenerationLog.txt
Generate “Captial"
  • I see chars
  • is that random enough? don’t know - how could I test that from the GUI?
  • is auto copy into clipboard same as copy paste? (same length)
Generate Small Chars
  • I see chars
  • is that random enough? don’t know
  • is it 200? need to copy and check (into my Test Tool Hub , is 200)
International C…
  • no generation
  • no log output
  • Obvious bug
I could carry on like this, but I really want to be able to interrogate and manipulate the system effectively.

Interrogate and Manipulate by working with the source

I really want to work with the source.
Maria said:
“Give me the source code and I can assure you I’ll break it! "
But we don’t have to wait for the code. We already have the code.
Hint: Most Apps come with the source. It is hard to hide the source if it is on your machine.
  • Web Client - the source is in your browser. We need to learn to read it and debug it using the built in browser tools.
  • Java applications are pretty easy to decompile. There are tools to do this e.g. (But I use IntelliJ, as you’ll see later)
  • Ruby executables often are a self extracting zip with a ruby interpreter that runs source from a temporary directory - use a file monitor and you’ll probably find where it runs the source from.
Basically it is very hard to protect your source. Assume you always have it. Assume your users and hackers always have it.
It has ever been thus. Back in the olden days we used disassemblers and monitors with debuggers to gain access to the assembly version. Now it is just easier.
If you want to protect your code then you run it on a server. And have a client server system.

To look at the source

  • Open IntelliJ
  • Create blank maven project.
  • Add crappy_little_datagenerator_v_1.0.jar to project as a dependency using the IDE.
Now we can really see what this is about.
I can see there is a GUI by default, but also a command line tool version.
I can run that from the command line. We can have multiple main methods in a jar file. The manifest file controls which runs by default. But we can start any one of them we want.
java -cp crappy_little_datagenerator_v_1.0.jar yhtest.tool.controller.TestTool
There may well be more bugs in this interface, since it is not the one promoted but, it might make it easier to automate for someone who does not know how to code, but knows how to use the command line well.
But, we won’t do that.
We’ll work from the IDE.

Because in the IDE I can write @Test methods

I can write unit tests to explore the Data Generator library e.g.
public class QuickTest {

    public void canIRunDataGeneratorHere() throws IOException {

        DataGeneration td = new DataGeneration();

        Assert.assertEquals(3, td.generateCounterString(3).length());

I now have a way of checking if the data generation is statistically valid. I could call the data generator say 10000 times, and count each occurrence of the characters and see if they level out. If not then there might be something wrong with the generation code.
Note that:
  • we could view this as unit testing
  • or treating the app as a ‘library’
  • or view the ‘app’ as a client server with the TestDataGUI as the client and the DataGeneration class as the server.
I often do this with web applications. There are the simple models of “client -> http -> server”, and we use a proxy to mess with the messages. Or send messages direct to the server. This is typically done with an API.
But I also use the concept of App as API. Where I send through the HTTP form messages that a GUI would POST. As though it were an API, even if there is no API. And simulate GUI testing without the GUI.
Even though I’m targeting the DataGeneration class, I’m not thinking of Unit testing, I’m thinking of testing the server without going through the GUI.
This means that I will jump back and forwards between @Test code, and the GUI while I’m testing.
If I was just doing Unit testing then I probably would not have the app running at the same time.
It might look superficially like I’m Unit testing, but the thought processes are different - try it and see.
Now we know we have all the tools we need to test this thing properly, and observe how it works.

Why not just decompile

I could have just decompiled the app e.g.
Bill Mathews said:
“ just download JD-GUI and decompile the jar ;) you can see the junction ‘tests’ they ran :) "
A few people on twitter mentioned this when the competition was running.
I have decompiled the app, but since I have it as a .jar in my project, the decompiled version is essentially a source view so I can do more than if I had simply decompiled it.
From the IDE I can now:
  • run the gui
  • debug the gui so we can breakpoint it and fiddle etc.
  • run the CLI
  • debug the cli so we can breakpoint it and fiddle etc.
  • breakpoint and evaluate code while running
  • write @Test methods to explore functionality and test it

A quick investigation to reveal ‘tester thoughts’

But, what we’ll do is quickly investigate the “International C…” error we found earlier.
A quick code read and we can see:
String internationalChars = "";
Now we could have seen that in the decompiled code as well,.
While we are looking at the DataGeneration code we can see that
lowercase chars set will not generate a statistically valid set of chars since there are multiple ‘o’s. This bug would have been very hard to find from the GUI. I might have been able to find it from the CLI had I scripted something to analyse all the output.
But sometimes a code review will reveal things. Certainly I can scan the code now and find all the spelling errors quite easily without having to scan the app.
(if you are working on windows then there are usually tools to help you analyse the resource files (do windows apps still have those?)) (Or you can hex dump the binary and you’ll often find the strings tucked away at the end of the file.)
String lowercaseCharacters = "abcdefghijklmnoopqrstuvwxyz";
Now I can see what the special chars and password special chars and email friendly chars are etc.
Lookng at this list of chars I can then go and search the web to find out the rules for ‘email friendly’ and compare them to this set of chars and see if any are missing, or we have any extra.
I could probably do the same for special chars
I don’t know what password Special really means, so I assume that is a ‘valid’ set.
Since I can see more than 26 chars in the lower case set, I have to wonder if the random generation routine uses the full length of the string, or might there be an off by one error?
A quick code review later and…
I get distracted immediately because I can see that generateRandomString
uses a contains to decide what string set to use, and concatenates them together.
Which means that I wonder if all string flags are passed in by the GUI, interestingly I can see that there is an ‘email friendly’ flat in datagenerator that is not used by the GUI. Not a bug, but I can see that the GUI uses a generateEmail method, which doesn’t use the email friendly string as part of its generation, it just uses ‘l’ which is lowercase - that is an interesting choice and means that the email generation doesn’t create random emails that stress the email validation rules.
OK, get back on track.
The strings of ‘valid’ data to choose from for each of the categories are concatenated together. This will also skew the statistical ‘randomness’ of the generated output. Perhaps that was intended?
The loops all use pre increment, which confuses my code reading since I’m really not used to that.
for(int e = 0; e < length; ++e) {
    returnString = returnString + String.valueOf(stringToChooseFrom.charAt(randomGenerator.nextInt(var10)));
Is this doing what it is meant to?
Let me think it through:
  • nextInt is 0 -> var10 (not including var10)
  • charAt indexes from 0, so would end at upper level
  • e=0; while less than length, increment
OK, seems OK.
But I might have wrapped it with a unit test if I couldn’t read it. Something I would be unlikely to do if I had just decompiled it.
And I still might because I haven’t ruled out that I have read it incorrectly, but I’ll do that later.
A lot of string concatenation. I wonder if the counterstring does that or if it uses a string builder?
Note: a side thought idea because I’m in ‘tester’ mode and I’m still learning the app.
OK, it uses a string builder in
Which is good because that is more efficient.
In theory, we know exactly how long the string should be so StringBuilder could be initialised to the length, rather than 100, that would reserve enough memory for the full counterstring at the start of the process and might be faster, it might also trigger an out of memory error early in the generation rather than late.
Hmmm, jumped back to GUI and tried 20000000000 as the counter string, and it dropped to 200.
Look at code to see why. OK, code seems to use Integer, so will be restricted to max length of Integer:

Compare with an other oracle

James Bach has a counterstring app. I could compare Martin’s to that. But I have one too, so I’ll compare it to mine, since mine uses a different algorithm to James Bach’s.
Gonna try a quick counter string race with my algo.
Let us try 2000000 - that should be visible time
Mine is about 16 seconds faster.
I’ll do a manual timing, with the apps running individually so they don’t clash.
  • Martin’s 17.10
  • Mine: 1.44
Martin’s GUI controls render the text better than mine. I’m using JavaFX his is not.
But, interesting speed comparison. I wonder if it is the GUI that slows it?
Tried from the command line and Martin’s without the GUI is comparable speed to mine with GUI, that’s interesting but probably of no value. And his GUI will be released, and mine probably won’t.
There might be more to investigate there but I’ve been at this for about an hour.
OK. Time to finish and summarise some stuff.
  1. Looking at the code makes it easier to spot:
  • spelling errors
  • simple gaps in code coverage
  1. Having the option of running the app via the IDE with the decompiled code, and investigating it while running opens up more ways than just the GUI. Which opens up more testing possibilities.
  2. I can use @Test methods to explore an app, not just unit test it.

This is a practical approach

I’ve had to take this type of approach on projects before when I was working with applications where, because I’m a tester, I was not allowed to see the code. Ho Ho.
Some examples:
  • A metrics excel spreadsheet that I was sure was wrong, but it was using ‘macros’ and they wouldn’t let me see the code. It was password protected so I couldn’t access it. Did you know that open office would open a password protected spreadsheet without forcing you to use the password? They didn’t. I did.
  • Android apps are basically a zip file. Then you can decompile it. I did this to help derisk testing an app, because we were able to see how it worked, and could make a decision on what to test based on the technical risks we identified.
  • When I’m faced with a library I don’t know. I’ll do a similar thing. Look at the code. Build simple @Test methods to explore the library. These also act as ‘documentation’ for me later to refer back to. (This is basically the learning approach I recommend in Java For Testers)
This was more of a ‘how to test’ rather than ‘see the testing I did’. But I hope it is useful.
Note also, I mentioned I used my app as an oracle. I wrote a similar app. I recommend everyone who tests and who codes, to wrap up their utility code into an ‘app’. You’ll learn a lot about structuring your code, using a GUI etc. And you’ll have a project you can work on in your spare time that will grow out of control, but prove very handy.