Tuesday, 25 October 2016

Q: when do we prefer manual testing over automated testing? A: Hmmm....

TLDR; As a simple rule, “I might not automate, if I have to create”. But I have done the opposite. I have automated when others would not. And I have waded in when others would have automated. To answer we need to question our situation and build a deeper model of our actual aims, options and capabilities.

Q: when do we prefer manual testing over automated testing?

A: I have a problem providing a simple answer to this question…

And not just because “it depends”.

Also because - “that’s not the model I use” so I don’t frame the question like that.

I don’t think in terms of “manual testing” and “automated testing”.

I think in terms of “testing”, “interacting directly with the system”, “automating predefined tasks, actions and paths”, “using tools”, etc.


If I’m testing then:
  • do tools already exist to help?
  • do I have abstractions that cover those areas?
  • do I have to create tools?
  • do libraries already exist?
I might automate heavily if I already have abstractions.

My testing might involve lots of automating if tools already exist.

I might not automate, if I have to create. When - if I create, the testing has to wait.

Don’t Automate Under These Circumstances…

So what about scenarios where you might expect “don’t automate” as my default answer:
  • something we need to do rarely
  • usability? “look and feel”
  • experimental features - surely we wouldn’t automate these
  • something still under development and changes frequently
Many people conduct stress and performance testing rarely. They generally automate.
“I might not automate, if I have to create.”
Usability and “look and feel” have tools to detect standards non-compliance. I might very well use those. So I might automate portions of this type of testing early.

Experimental features might fit into an existing process and so we might already have a lot of the scaffolding we need to support minimal and speedy automating - in which case I might automate those.

I have automated features that were changing frequently because we were testing the same ‘path’ through the application many, many times (every time it changed) and we often found problems (every single release) and the effort of testing it, by interacting with it, was waste. So we didn’t interact with it until the automated execution could pass reliably. But, that also depends on ‘what’ in the implementation changes, and how that change impacts the chosen approach to automate it.

What’s the real question?

If I have to ‘create’ stuff to help me automate as part of my process then I’m likely to weigh up the ‘cost’ of automating vs just getting on and interacting with it.

Are the assumptions behind the scenarios true?
  • Scenario: rare execution
  • Assumption: we would spend a lot of time automating something that we would rarely use
But perhaps we ‘rarely’ execute it because of the time and energy involved in setting up the environment and data, rather than the tooling. Perhaps we need to automate more, to allow us to ‘test’ it more frequently?
  • Scenario: experimental feature
  • Assumption: we write code to automate it that we throw away
But perhaps, the feature is so experimental that the easiest way to interact with it is by automating it, otherwise we have to spend a lot of time building the scaffolding required (GUI) for a user to interact with it.
  • Scenario: usability or “look and feel”
  • Assumption: this is subjective
But perhaps standards exist to guide us, and if we ignore those and all the existing tooling, then we create something so subjective that no-one else can use it.
  • Scenario: frequent change
  • Assumption: automating a scenario always takes longer than interacting with it and maintenance in the face of change takes too long
But perhaps, the GUI changes infrequently and the change takes place in the business rules behind the GUI. So really we are dealing with a single path, but lots of data. And perhaps we can randomize the data, and implement some generic ‘model based’ rules to check results.

Scenarios have more nuances in the real world

Very often we discuss these ‘when should we automate’ questions as hypothetical scenarios. And if we do, we need to drill into that scenario in more detail in order to come up with our answer. One of the main benefits to the exercise of ‘hypothetically…’ is the asking of questions and fleshing out a model to better explore the scenario.
  • I have automated when other people said I should not, and it saved time.
  • I have interacted manually using tools, when others said we should automate, and we found problems we would never have found if we had taken the ‘automate’ path so early.
  • I have manually engaged in stress and performance testing.
Sometimes, conducting a contrary experiment, will provide you with ‘examples’ that make it harder for you to answer these scenarios without extensive model building.

Thursday, 6 October 2016

A Case Study in Creating a Conference Presentation using Markdown

TLDR; using Marp and Pandoc I can write a report and presentation using Markdown, then add other tools to make it 'pretty'.

I’m going to show you a case study in how I’m planning a presentation for conference.

Because I want to offer you an alternative to diving straight into a powerpoint presentation.

My process for creating slides

What I’ve been doing is:
  • create an evernote
  • make all my notes in there
  • rough out some of the notes as summary info to add to a slide
  • create rough slides
  • expand my notes so they are like a ‘paper’ to support the presentation
Then I’ll:
  • move on to the slides
  • tidy up the slides
  • practice the presentation
  • keep the ‘paper’ up to date
And all of this means jumping between a few tools and I don’t really like working with presentation tools.

I’ve been looking for an alternative that fits my workflow.

An alternative that fits my workflow

And I think I might have found it. And I can:
  • write everything in markdown
  • version control my text
  • keep the ‘slides’ and the ‘paper’ together in the same file
  • create a pdf of the paper - with ‘embedded slides’ - without doing any exporting or editing
  • create a pdf of the slides - without any extra editing or maintenance
And now I think I’ve found a way of doing that - and of creating ‘nicely formatted and professional slides as well’ - all without touching a presentation tool.

So let’s have a look.

The Making Notes Phase

I’m still in the making notes phase for this presentation. This example is my presentation for TestBash Netherlands in January 2017. (official Test Bash site)

I make notes on it and restructure it when I get an idea - sometimes that means waking up at 3 in the morning and sitting in the office to type up the ideas, but that’s how I work on presentations.

In my earlier workflow, I would just have a set of notes at this point. And I would create those by running my ‘notes’ through pandoc to create a pdf.

With my new workflow. I already have:
  • a set of notes in Evernote
  • a pdf of the paper generated by Pandoc
  • a presentation (using Marp)
I could wing it tomorrow, if I had to.

This is a massive win.

I always like to be at the point that ‘I could do it tomorrow, if I had to’. But I’m never usually at that point so early in the preparation cycle.

What that means is that every iteration from now on is an improvement (hopefully).

I could wing it tomorrow!

So I have notes, and they look a bit like this:


# And I'm going to say that:

## "any software that I use to support my testing is a "Testing Tool"


So if I use Cucumber in my testing it becomes a "Testing Tool"

And you can argue about it all you want. It won't stop me using it, or misusing it (if you say its not a testing tool and I use it in my testing).



And when I say “they look a bit like this”, I mean “they look exactly like that”.

I copy and pasted the above from my notes.

And what you can see is:
  • a draft slide
  • slides a separated by ---
  • and my ‘paper’ inside the HTML comments
I would normally continue working like this, in Evernote and a text editor. Until it was time to create the slides.

I can create slides now!

By using Marp, I can save my notes to a text file.

The same text file that I would feed into pandoc.

And I can view my notes as a set of slides.

The reason I use <!-- and --> to delineate my ‘paper’ sections is because most Markdown processing tools ignore HTML comments.

Marp, ignores HTML comments and only displays the --- delineated parts as a set of slides.

Using Marp, I can:
  • view my notes as a set of slides
  • get a feel for my notes as a ‘presentation’
  • output a pdf that I could use to ‘wing it tomorrow’
The Marp slides output isn’t great, but it is certainly ‘good enough’, and I released the Java and Selenium Install Checklists to slideshare using Marp, directly from the checklist file that I maintain on github:
Same source document different rendering.

The PDF isn’t ‘perfect’ but its good enough for my purposes.

I can view the paper easily!

If I copy and paste the contents of the file or evernote into dillinger.io

I can see a rough version of the paper, but Dillinger shows me the <!-- and --> which isn’t great, but for review purposes this is fine and easy.

I couldn’t use Dillinger to release the paper in that condition though.

I can create a pdf paper easily!

I use pandoc to create PDFs.
If you’ve hired me for consultancy work - pretty much guaranteed that the PDF report you received at the end was generated in pandoc.
Trade Secret - I can create a report really quickly because:
  • I write it as I consult
  • When I make my notes in Evernote
  • I write in Markdown
  • I convert with Pandoc
  • I spend time at the end of the day (/trip back on the plane/journey back on the train/engagement in the airport lounge) editing and fixing spelling errors
  • generate in Pandoc - I tend not to spend much time reformatting
Shh, keep that a secret though, I don’t want my competitors to have the same advantages that I have.
Normally I just run a text file through pandoc:

pandoc mynotes.txt -f markdown -o professional_report.pdf

(pandoc can change page size and create a table of contents and output MS Word and OpenOffice files, but somethings I’ll keep as trade secrets.)

But pandoc would ignore the HTML comments as well, so I create a script:

(Get-Content slides.txt) | ForEach-Object { $_ -replace "<!--" , "-----" } | 
ForEach-Object { $_ -replace "-->","------" } | Set-Content slides.md
pandoc slides.md -f markdown -o output.pdf

Above is windows powershell which converts the HTML comments into Markdown horizontal rules.

I converted an open comment <!-- to five minus signs -----and close --> to six ------ to give me the option of ‘find and replacing’ the report back into a slide deck if I want to.

If I was on mac I’d probably create a .sh that used sed to perform the find and replace operation.

This gives me a fairly nicely formatted pdf.

Good enough.

It makes it easy for me to see:
  • where I have a lot of content,
  • where I have less
  • what is a quick slide
  • what is a slow slide
  • what is summary
  • what is detail
  • etc.

And the main point for me is that I get this with no extra effort.

I can imagine that to release the paper I would convert the comments to "" and avoid having extra horizontal lines.

But for the moment, I’m comfortable with the double lines, and the removal of comments is a one way operation.

Version control benefits

A great benefit from this is that the only things I need to version control now are:
  • a single text file
  • a conversion script
  • any images
I don’t need to add
  • a huge LibreOffice or Powerpoint presentation
And I have proper versioning so:
  • I can diff the notes and slides,
  • I can bring back lost information

But it ain’t that pretty at all

I’m not that bothered about the ‘prettyness’.

Normally I ‘prettify’ the slides close to the release date anyway.

But we can automate it all!

I’m currently evaluating Deckset.

A Mac only app which can take my ‘notes’ file with the slides and the HTML embedded ‘paper’ and can format it into something ‘pretty’:

I ‘just’ have to:
  • choose a template
  • pick the colours
  • then ‘tweak’ the Markdown to make it work better as a slide deck

Next Year…

I guess we’ll see in January 2017 how I actually create the slides.

And we’ll see which of the above slides actually make it from the ‘Notes’ phase, into the ‘Finalise Slidedeck’ phase.

Tools Mentioned
Bonus video showing the tools in action (available on youtube)