Friday, 6 September 2013

10 Experiments to Improve Your Exploratory Testing Note Taking

I have some 'rules' that I apply when I take notes as I perform exploratory testing.
When I look back over how I took notes in the past I can see that I tried different experiments with my approach when building those 'rules'.
I recommend some of my experiments to you now:

  1. In Memory
  2. Only use pen and paper
  3. Only use a text editor
  4. Use a text editor and screenshot tool
  5. Record the screen and talk as you test
  6. Use a tool designed for exploratory testing
  7. Use a Mind Map
  8. Draw a diagram
  9. Automate the capture of logs
  10. Use a Spreadsheet

1 - In Memory

Only use your memory to track your exploratory testing.
This experiment mainly helps me remember that I need to do more. I now feel very uncomfortable testing with just my memory, but when I started this felt natural. I changed.

2 - Only use pen and paper

Yup, you use your computer to test, but you make notes on pen and paper.

Variants:
  • different pens, 
  • different colours, 
  • different sized paper,
  • notebooks, 
  • loose paper, 
  • text, 
  • diagrams, 
  • mind maps, 
  • scribbles.

I find this works well with a single screen, and intense moments in the testing, but I try to re-transcribe or take a photo with my phone and have the image in evernote.

3 - Only use a text editor

Experiment with different text editors, find an editor you like - I've pretty much settled on NotePad++ and Sublime Text, and I import the text files into evernote for later searching.
Different styles of note taking:

  • Prose, 
  • Notes, 
  • Time Stamped Entries, 
  • Annotations like #test #bug etc.

Can you parse your logs at a later date automatically? Would that benefit you?

Do your own search and find a text editor that works for you at the moment.
Touch typing helps. learn to touch type if you can't already.

4 - Use a text editor and a screenshot tool

Sometimes you need to capture the moment as a screenshot.
Do you just use Ctrl+Print? Do you use an image editor? Do you use a dedicated screenshot tool?
I tend to use SnagIt or Jing now. I've used lots of others in the past.
  • Where do you store the images?
  • What filename standard do you use?
  • How do you cross reference your text edit notes to the screenshot?
  • Do you think you would benefit from using a Word Processor and embedding the screenshots along side your text?


5 - Record the screen and Talk as you test

Environment can get in the way for this if you work in a shared office.
Do you have equipment that you can comfortably use for long periods of time?
Talking, and thinking and doing takes practice and time.



6 - Use a tool designed for exploratory testing

People have created a whole bunch of tools; designed, or marketed, as helping exploratory testing.
Try them. See if they work for you.
If your style clashes with the tool. consider if the tool benefits warrant a change of style from you.
Do your own search and find other tools designed for exploratory testing
If you could write your own, what features would it have? Perhaps you could use a combination of tools to gain those features now?

7 - Use a mind map 

Everyone loves creating mind maps. Few people use mind maps like Buzan suggests. Who cares, use mind maps and do it your way.
Do your own search and find a mind map tool that works for you.
What do you represent in the model?
  • Ideas?
  • Steps?
  • TimeStamps?
  • Images?
  • Screenshots?
  • Links?
  • Observations?
  • Questions?
  • To Dos?
  • ?

Over time, learn the features of the tool. consider which features you don't use. Should you? Would they help?
Perhaps you don't use enough of the features? Try a less featured tool and see if it still works for you?

8 - Draw a diagram

Pen and Paper works well for diagrams.
What do you diagram?
  • Structure? 
  • Flow? 
  • Entities? 
  • Notes? 
  • All?

GraphViz lets you write text files that it compiles into automatically positioned graphics.
You can use draw.io as an online diagrammer.
At last count there existed a Bazillion diagramming approaches and tools. Try some of them.

9 - Automate the capture of logs

You can't argue with logs right? Why bother making notes when the logs will do it for you?
  • Fiddler - for HTTPsessions
  • tail system logs (logtail, multitail, etc)
What do you make notes of when you use logs?
Do the logs capture everything you need?
How do you cross reference your notes to your logs and to your screenshots?

10 -  Use a spreadsheet

What about a grid?
Would that help?
Try it and see.
https://docs.google.com/spreadsheet

Repeat

The above covers a lot of note taking styles
  • Visual
  • Tabular
  • Outline, Tree
  • Sequential
  • Adhoc
  • Formal
Evaluate what worked and what didn't.
Take care about your judgement because some of it didn't work due to your lack of experience - try it again. Some of it didn't work because it doesn't fit you, your environment, your system, etc.
Having done them all - try them again. Some of them will seem offensive. Some will feel restrictive. You gain gain more insight when you try it again.

Summary

Even though I gave you this as a "10 experiments to improve your exploratory note taking". I have not given you a quick fix.

  • If you tried one each day, this would take you 2 working weeks.
  • If you try variants in each of the experiments ( different tool, different paper sizes, etc.) This could take a month or more.
  • If you repeat them and challenge yourself to master them, and change them, this could take up to 6 months.
  • I still experiment with my approach. I have done for years.


What experiments would you recommend" or "What experiments you have conducted?". Let me know in the comments below.

You might want to watch "What is Good Evidence" by Griffin Jones, which reminded me I needed to write about this. Griffin's talk overlaps very nicely with this post. I recommend you watch it.

13 comments:

  1. Great post of improving every-day testing.

    Regards,
    Srinivas Kadiyala
    @srinivasskc

    ReplyDelete
  2. Thanks for this. I've recently been rethinking how to record my testing sessions so this was very timely.

    ReplyDelete
    Replies
    1. Thanks Liz. I hope you find some way of sharing your lessons learned from experimenting with your approach, I find a lot of value in learning how other people record their work.

      Delete
  3. I've tried most of these, and even with Evernote search, I've still not found a method that really gets me a reliable history. It's okay for "today's testing", but it either gives me insufficient history or an unmanageable history for "last months project". I always end up falling back on TCM and issue history.

    ReplyDelete
    Replies
    1. TCM? Which One?

      I use Evernote to sync in folders into notebooks so I can limit the search. I also keep the computer folder structure so I can search and grep there too.

      I tend to use the directory for date searches and evernote for text searches.

      I sometimes remember to write with a view to future retrieval.

      Thanks Dan

      Delete
  4. This comment has been removed by the author.

    ReplyDelete
  5. As a big fan of pen and paper, this post really warms my heart. And a lot of good input besides that, so this is me taking a bow. Very nice reading.

    ReplyDelete
  6. I am new to software testing.Just a master student with no industry experience yet. Cleared ISTQB CTFL Certification and planning to pursue my master thesis in "Exploratory Testing". Please guide how it it actually performed.

    ReplyDelete
    Replies
    1. As a Masters student you will research the topic by reading all the books, journals, web sites and watching as many online videos as you can.

      To start you off:

      I've created a few videos showing how I performed some exploratory testing over on the YouTube page http://www.youtube.com/user/EviltesterVideos

      Also read the blogs written by Michael Bolton, James Bach, James Lyndsay.

      And a whole host of people - so perform some additional web searches.

      Delete
  7. Great post Alan,

    I have a style I feel comfortable with, so haven't played with different ET recording approaches for some time. Your post (and De Bono's "Think! Before It's Too Late" which I'm reading right now) has given me a bit of a push to start experimenting again, so thanks for that.

    One point I'd like to add is the fact that while my current "core" relies around a single method (spreadsheets), it is strongly supported by other methods. I'll use a notepad to record potential test ideas before a session, or a mindmap if I want a little more structure rather than just a list of points. A screenshot tool is practically mandatory (at least where I've worked) and I've used (virtual and physical) sticky notes for reminders, queries and so on.

    Good stuff!
    Dean

    ReplyDelete
    Replies
    1. Thanks Dean, and thanks for sharing notes on your approach - great to learn what you're using.

      I haven't used sticky notes for a long time. I guess sometimes I tailor my approach around the stationary cupboard :)

      Delete
  8. Very good. It`s best interesting contend for QA :)

    ReplyDelete