Sunday, 27 April 2008

Question: Which applications do you use during interviews to 'see' howcandidates do exploratory testing?

After the discussion about passion and interviewing testers I started to rethink how I conduct interviews and I think that in the future I will use MS Paint as an application to see how candidates approach testing.

A long time ago, I wrote my own little app for use during interviews. You can play with it if you like - it has many deliberately injected bugs - so no raising defect reports with me, but feel free to share any experiences that you have with it.

I mainly used this in team based exercises, and primarily to explore presupposition analysis as a method for testing.

So, now to explain the reasons why, and how, I'll use MS Paint.

I currently think I will have 2 charters:

  • A "risk identification tour" charter
  • A Scenario charter: "Function A" has changed - the existing automated tests have passed but it has a lot of internal dependencies - can you give it a quick once over and a quick regression of "sub function Y" and let us know what you find?

I picked paint for a number of reasons:

  • I class it as a ubiquitous Windows application
  • The fact that everyone has seen it should trigger some 'assumptions' about it that the process of testing can help challenge
  • Since I classed it as a 'mature' application, my immediate assumption was "will I even find any problems in this application, MS must have made it really stable by now!" (I do seem so naive and overly-optimistic sometimes). That initial moment of panic seems like a good trigger to put people in a non-complacent state and view the application hyper-critically and focus their attention on their own test approach.
  • I want to see the candidate's initial 'tour' of the application to find out what areas they identify as having high 'risk', and their rationale for that identification
  • Paint really does have bugs in it - lots of bugs - at least it does if your 'tour' brings you to the same risk area that I looked at - you can probably find bugs in the rest of the app too (my naivety got knocked out of me after 2 minutes of testing)
  • I've seen MS Word and MS Excel used in a lot of testing demos but not MS Paint, so testers may not already know its flaws (I'm trying to keep this blog post deliberately vague so that I don't highlight all the ones that I found, or even the areas I looked at)
  • It has a lot of Oracle challenges - i.e. how do I know if I saw a correct response to my action? So should make for a good question and answer session
  • It seems like a good spring board for tooling questions - e.g. what tools could have helped you during your testing?

But to help me provide some interview variety...

  • What applications do you set your candidate loose on,
  • What charters do you give them?


  1. I used to ask the candidates to test a very simple app that I got from James Bach, he used it in his tutorial "Exploratory Testing Explained" at StarEast 3 years ago (unfortunately it stayed behind when I left Sweden). I have been tested myself during a phone interview on the Calculator functionality of the Google homepage (doing the testing on the computer while on the phone, which I thought was pretty cool), and on a board describing how i would test the registration process for Gmail. I have also heard people using specific features in Word and/or Excel, as you mention. That's all I can think about for the moment!



    Thanks Marta,

    I don't think I have ever used the Google Calculator...time to investigate further.

  2. I just use the whiteboard instead of a particular application that way I can tune things depending on the role I'm trying to hire for. In general I draw a little 'screen' on the board which has three text boxes: name, state, zip (in that order) and two buttons: cancel and submit.

    I then ask the person to test it.
    - do they ask about their mission?
    - their oracle? (which in this case is me and I'll answer any question about it)
    - what sort of 'buckets' do their questions / tests fall into?
    - do they ask about the larger system this might be a part of?
    - do they draw on their experience to recommend changes?
    - etc.

    This lets me guage their clue level and where there might be gaps (as I percieve them) as well as see how they think about / approach a testing problem. To me, those two things are more important than specific test technique knowledge.

    Thanks Adam,

    That seems like a pretty re-usable method without any technology hassle and splendidly open ended.

  3. Mine is similar to Adam's. Since my company's software has a sign-up process (and it's a feature that most people are familiar with), I ask them to tell me how they'd test a simple registration form with name, password, and a submit button. It's good for making sure experienced testers really have practical experience and it gives me a good clue about people trying to enter the field.

    Until now, I've always drawn it on paper and asked them to explain how they'd test it, but I'm thinking now that I should have them go to a real site and show me how they'd test it. It would also show me how efficiently they work (on top of their testing techniques). I'm not going to create an application with built-in bugs because I think that just seeing their test cases would give me enough information.


    PS - I just came across your site today and I definitely like it.

    Thanks Scott. I paid a visit to your site. Fantastic. Now I feel inspired to get back to a decent 4 ball cascade.