Wednesday, 30 March 2011

Software Testing Club – Testing Tales released

A while back, Rob Lambert started collecting Nursery Rhymes for the Software Testing Club.
Since I enjoy Humpty Dumpty I created a version for STC. It has just become available on the Software Testing Club website, along with Rob’s entertaining take on The Gingerbread Man.
They have posted the uncensored version of Humpty so those of a weak disposition take heed of this warning before visiting said site.

Sunday, 27 March 2011

The Untold Story of the Test Plan

We all know by now the following things about test plan documents:
  • a test plan document does not substitute for the process of test planning,
  • a test plan document represents one way of communicating the results of the test planning process,
  • the process of test planning stops only when you stop testing (even then, you might obsess so much that you carry on planning)
I didn’t always know those things.

Wednesday, 23 March 2011

Selenium 2 makes automation debugging easier

One of the parts of Selenium 1.0 that I never enjoyed was debugging automation that didn’t work. I had to faff about creating custom Firefox profiles with Firebug installed and set to go through a proxy.
Selenium 2 makes all of that so much easier. With the code below, my test runs through a proxy server on port 8081 and has Firebug installed so that I can breakpoint my test and then go debugging around in the browser DOM, debug the JavaScript to make sure events get fired, etc. etc.
I found most of the information to do this on the Selenium Forums
The code should be easy enough to follow if you are already using Selenium 2

import org.junit.Test;
import org.openqa.selenium.Proxy;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.openqa.selenium.firefox.FirefoxProfile;

public void FirefoxUseExtensions() throws IOException{

  // setup the firefox proxy to point to Burpsuite
  // on port 8081

  Proxy localhostProxy = new Proxy();

  // Prior to running the test
  // Download the firebug extension file
  // to a local folder

  String extensionPath = System.getProperty("user.dir") 
      + "/"
      + "firefoxExtensions/firebug-1.6.2.xpi";

  // create a custom profile

  FirefoxProfile profile = new FirefoxProfile();

  // set the profile to use the proxy settings


  // stop firebug showing the first run
  // screen by setting the 'last version'
  // to the current one downloaded
  // try without this and see what happens!

  profile.setPreference("extensions.firebug.currentVersion", "1.6.2");

  // add the extension to firefox

  profile.addExtension(new File(extensionPath));

  // start firefox with the custom profile

  WebDriver driver = new FirefoxDriver(profile);

  // go to your favourite testing web site



Monday, 14 March 2011

Paranoia as a learning and testing strategy

Honestly who doesn’t enjoy reading conspiracy theories? Who doesn’t enjoy putting on the “Helm of Paranoia”.
I use paranoia as a learning strategy and as a testing strategy.
You can too…
For learning:
  1. Assume that the author of the book lied to you
    • This forces you to do more research
    • Read additional books
    • Read differing opinions
    • Check their facts
  2. Assume the author didn’t want you to know the truth
    • They deliberately obfuscated their writing so you have to simplify them
    • They deliberately made it overly specific so you would not learn the different ways to apply the technique, so you have to generalise it
    • They deliberately didn’t teach you the principles, so that they would retain the edge and you could never compete. So you need to take their approach apart to identify the principles that it codifies
  3. Assume the author didn’t know the truth and others misled them
    • Only you can draw out the patterns in the text to identify the underlying earth shattering secret truth that has remained hidden in the text.
For testing:
  1. Assume that the system has lied to you
    • Yes it ‘says’ it has saved the record – how do you know?
    • Yes it ‘says’ you can’t use numbers in the password – how do you know?
  2. Assume the system didn’t want you to know the truth
    • Look for the hidden buttons that others have overlain and obscured
    • Look for the hidden windows
    • Look for the hidden div elements (how can you make them appear)
    • Look for the hidden form fields – what secrets do they hold
    • Look for the secret cookies – what do they mean? What information have they tracked about you?
    • What other secrets has it hidden from you?
  3. Assume the system doesn’t know the truth
    • Can you mislead it?

The truly paranoid among you will now rightly assume that I have lied to you, that I don’t want you to know the truth, and that others have so misled me that what you have here is a pale reflection of the truth that only you can uncover.
And for your ongoing research pleasure, I heartily recommend Paranoia Magazine. I haven’t read it for a long time as I haven’t found a shop that will stock it – no doubt due to the machinations of one cabal or another. But that won’t stop you getting a good old historical dose of paranoia with the free back issues at (Just don’t tell them I sent you, they don’t need to know.)

Saturday, 12 March 2011

Sub-cultural Testing Influences gone mainstream #1 – The Assassins Creed

My testing style, attitude and approach has had many influences. I only recently realised that one of them has become very mainstream.
"nothing is true, everything is permitted"
Words attributed to Hassan-i Sabbah on his death bed. Specifically ambiguous and open to misuse. Perfect for testing.
This went all mainstream with the “Assassin’s Creed” game series
Fortunately when dealing with software systems, many of the moral dimensions of the Assassin’s phrase disappear and we no longer have to engage in the philosophical debate about boundaries of human actions. We just get on with the testing.
Objections such as “No user would ever do that” have no stopping power on the Assassin influenced tester.
If the system allows me do it, and I observe that the system allows me do it, and I figure out how to manipulate the system into doing it, I do it.
This phrase helped me through some difficult and turbulent waterfall testing times.
Try it on and see how it fits your testing style.
Just remember to treat software systems and people systems differently.

Thursday, 10 March 2011

Test Techniques Evolve

Observation –> Taxonomy Entry –> Heuristic –> Technique –> Children’s Story –> ?
  • I may have some of the arrows the wrong way around in the above diagram. After all, who really knows what influences what.
  • And as to the “?”, I hope that you decide what comes next.

Tuesday, 8 March 2011

Dangerous Test Concepts Exposed

There exist test ‘concepts’ which, while seemingly simple, have a tendency to melt my brain.
  • Black Box/White Box Testing
  • Functional/Non-Functional Testing
  • Positive/Negative Testing
Years spent studying hypnosis and revelling in the ambiguities of communication have left me with an inability to parse language the way I did as a child.
I used to have the ability to read, and use, these terms without blinking. Now I throw a ParseException.
I realise the above concepts may not seem strange to the majority of people reading this blog post. So I will attempt to describe how I respond to reading these terms, in the hope of spreading a contagion against them.

First some generalities

First some generalities across all the ‘concepts’.
I think that I think, and test, in terms of Systems. More specifically in terms of models of systems.
Yes/No models jar my sensibilities. Testers lives in the grey areas and find treasures in the shadows. Absolutes do not help me explore the System. I do not want guidance by absolutes, I want guidance towards flexibility.
All of these concepts encourage me to think and communicate in ambiguities, while pretending specificity and authority. If I say White Box testing, you still have no idea what I mean. If I describe the models that I use to guide my testing then you have a better idea of the type of tests I might create, and the type of gaps I may encounter, regardless of the colour that I paint that model.
These concepts all encourage me, as a tester, to ignore the model and think in vague conceptual terms. They categorise techniques out of context and far removed from the model that the techniques apply to.
They encourage debate about irrelevancies. e.g. Should we do White Box, or Black Box testing? Developers do White Box Testing, testers do Black Box Testing? Where does Grey box testing come in?
The amount of time wasted on projects when people ‘debate’ if the requirement should form part of the ‘functional’ or ‘nun-functional’ specification, instead of getting on with communicating what they need shocks me. (I estimate this in terms of man-years for some projects)

Black Box / White Box Testing

Ah, the most pernicious of them all.
How comfortingly I viewed these terms in my early years. I could ignore the relationships between my test models. I remember times when I made no effort to learn the technology or architecture of the system. I could simply focus on inputs and ignore flow, because I did my testing all “Black Box” like.
Then over time I grew uneasy. I started to apply White Box techniques to ‘Black Box’ingly understood systems.
This concept started to seem like the devil on my shoulder whispering “Relax, don’t do the work to view it as a system, treat it as a ‘black box’, see how much simpler it all becomes”.
I started to feel as though the constant repetition of this phrase in the testing literature stopped me hearing the siren call of the technology.
I weaned myself away from these terms by using them as insults. I engaged in a form of aversion therapy.
I labelled all testing I did without understanding the technology, or without having made an effort to observe the architecture, or without building a structural model as ‘Black Box’ testing. I imbued the term with an association of laziness.
I view all my areas of superficial modelling, as having greater risk of defects unfound.
I associated my use of ‘White Box’ testing with over-confidence. I felt nervous if I only read the code, or only based my testing on a superficial understanding of the structure. For me, the phrase warns me about my complacency. I try not to plan for complacency.
This helped me move towards a more Systems focussed approach to my software testing. I no longer feel a pull towards exclusively the Darkness or the Light.
I try to concentrate on models and identify techniques to apply to models.
I ignore the categories and partitioning of techniques presented by the Canon.

Functional/Non-functional Testing

I periodically write non-functional automated test scripts… Sometimes my scripts simply don't work and they remain non-functional until I fix them to get them functional. <insert canned laughter here>
In the past, I have communicated my testing in terms of “I built tests to cover the ‘Non-Functional Requirements’”. Which in context could have read “Speed Monster Funky Requirements” and I would have just as happily used that phrase in my test communication.
When we mean requirement based testing, then we should say “requirement based”. Removing the word “requirement” suggested to me that “Herein lies magic”. That the “Functional” tester does something different from the “Non-functional” tester. That we can phase the testing differently and do “Functional” testing and then later do “Non-functional” testing.
I removed my unease towards this phrase by always appending/inserting the word requirement.
But … I don’t like the distinction between “Functional” and “Non-Functional” because of the ineffectual processes and discussion that I have seen result from the distinction. Sadly Software development has a history of considering functionality as separate from the other “ilities” that might relate to that functionality. A tradition or an old charter, or something.
And yet I don’t see why I should let this influence my testing. I’ll do whatever ‘it’ takes, and if that means merging the functional with the non-functional to get more advantages in my testing or tooling then I will jolly well do so.

Positive and Negative Testing

I assume “positive” and “negative” as testing terms, must have come from a more objective diagnostic context than I get involved in. From a context where a negative test, rules something out completely, and a positive test rules it in.
I classify much of the testing I do as exploratory or experiment based. I do things to try and rule out certain possibilities and I could possibly count that diagnostic. But I don’t think my tests demonstrate the certainty that “positive” and “negative” testing suggests to me. I find more value in thinking in terms of a general experimental process rather than a diagnostic process.
But even the above description doesn’t seem to map to how the testers I meet describe positive and negative testing.
Testers seem to describe negative testing in terms of exercising error validation and reporting. And positive as “stuff we think will work”. I tend to view the existence of error handling functionality in applications as a positive thing.
I find it hard to use the terms –ve and +ve testing. When I try to show that the system does not work, I talk in terms of trying to ‘break’ it – clearly I don’t break it, but it works as a term for me. I also view this as a positive thing for me to do.
Yes, I freely admit to misusing words, particularly if the misuse of the word frees my thinking. I try to stop using words that constrain me.

End Notes

For all of the above ‘concepts’ I can read the definitions and almost understand why those names fit those descriptions. But I try to find other, more exact, or more contextual, or more personal, ways of describing the testing that I do.
You may not need to go as far as I did to de-programme their effect.
Younger testers may even have an immunity to them.
I still feel their pull though. My brain still finds a ‘reasonableness’ in all of them while at the same time each of the ‘concepts’ strikes me with dread. Each of them seems to proudly state that we can easily view the System as two simple partitions and we can relax about modelling or understanding the complexity.
If you speak to me using these terms, I will have to ask you what you mean. Do not assume that my testing background will allow me to understand you.
I try to stand firmly in a graduated shading testing scheme. Sometimes I even imagine colour in my test approach. Someday I will attempt to employ synesthesia.

Monday, 7 March 2011

The results are in for the Evil Tester Certification Survey

Did you enter? Probably not. And if not, you lost your chance to save the world.
The survey will remain open as I found many of the answers highly entertaining. So feel free to pop random entertaining snippets into the survey and I shall read them and chuckle.
Dare I draw conclusions from the survey, given some may have viewed it in jest?
Of course.
But first… three serious points.
  1. I take software testing incredibly seriously. I poke fun at it. Because I take it seriously.
  2. If it did not seem like a serious topic to me. I wouldn’t bother poking fun at it.
  3. I consider software testing so serious a topic that I need to think about it, not just coast along with the Status Quo. I can’t help find humour in it if I think about it. We laugh to change. If I don’t find a way to mock it then I probably haven’t thought it through from enough angles.
  4. Do you find testing serious enough to laugh at?
So, on with the statistics and numbers. The closest you will come to metrics on this web site.

We asked “Do you hold a Certificate in software testing?”
Our Survey said…
  • Yes == 56%
  • No == 44%
Proof positive then that the people who replied know what they replied about. I think.

We asked “Did your certification offer value for money?”
Our Survey Said…
  • I'm not certified, didn't you read my answer to question 1? == 41%
  • Yes == 13%
  • No == 31%
  • Definitely not, I got my money back == 0%
  • Other == 16%
Contentious. Some people didn’t think it offered value for money, BUT DIDN’T ASK FOR THEIR MONEY BACK!
Meaning – this could become a profitable business to get into. No-one asks for their money back.
Certification providers take note, you can drop your standards and it won’t make any difference.

We asked “Would you like to self certify yourself in Software Testing?”
Our Survey Said…
  • Yes, clearly that would be better == 50%
  • No, I can't possibly know better, or as much, than the official certification experts. == 3%
  • Never! What new certification scam are you trying to pull? == 47%
It seems as though some people saw through my thinly veiled attempt to see how much appetite exists for self-certification. And 50% think it sounds good. So before the certification crowd jump on this particular bandwagon. This one belongs to me.
And 3% thought they didn’t know as much as the certification experts. Which clearly (in an abuse of the survey process, but in keeping with poor statistical analysis) means that 97% of people doing the survey think they know better than the certification experts. And therefore they consider self-certification a good idea.

We asked “If a self-certification scheme did exist, how much would you pay for the privilege?”
Our Survey Said…
  • Nothing, it should be free - like all good things in life == 63%
  • < £10 == 3%
  • >= £10 && <= £50 == 16%
  • >£50 && <= £100 == 0%
  • > £100 (I only value something if I pay a lot for it) == 3%
  • Other == 16%
I don’t know what you see from these results, but I see:
  1. a free sample to hook in the 63%
  2. an immediate uptake of 35% of respondents
  3. hopeful conversion of the 63% to a total of 50%
Self-certification as a potential goldmine.

We asked “If it did cost money, what would a self-certification scheme pack have to contain so that you felt you were getting value for money?”
At this point I’m giving away the secrets of my future success. Because you too could use the information here to create a money spinning self-certification pack.
Our Survey Said…
  • Sample Certificates to fill in == 25%
  • Sample Answers to help you show off your newly certified knowledge in interviews == 25%
  • Empty promises about industry standard knowledge and future career prospects == 31%
  • Hints and tips on how to fill in your certificates == 22%
  • Affiliate links to lamination services so you could create laminated certificates == 22%
  • A direct debit form so you could continue to pay for your certification every year (and thereby 'prove' that you were still worthy of the certification) == 13%
  • A letter congratulating you on your decision == 28%
  • A document explaining example topics you could certify yourself in == 28%
  • Multiple levels of certification in one pack (e.g. novice, journeyman, master, expert, Nobel Prize Winner) == 25%
  • A Membership card == 38%
  • A Badge == 41%
  • Money off vouchers for additional certification packs == 16%
  • Other == 50%
Clearly people want to experience a sense of belonging – hence the popularity of the badge and membership card. This may explain why the certification sales pitch based on Social Proof (everyone else does it, why don’t you) does so well.
The direct debit, whilst looking good from a money angle, doesn’t seem that popular, so CAT’s compulsory re-certification scheme might not go down too well with the paying masses. The CAT folk might want to consider changing that aspect of the scheme. But since they didn’t get back to me with previous comments I sent them on their scheme, they might not take any notice of this one either.
So quick wins for all certification providers. Supply a badge and a membership card to boost sales.

We Asked “Do you wish this was a prize survey?”
Our Survey said…
  • Yes, I only fill in surveys where I can win a prize == 28%
  • No, I fill in surveys for the fun of clicking buttons and feel sure that the person on the end of the survey has read my 'other' answers carefully, so carefully in fact that I have undoubtedly succeeded in changing their mind about something or other == 50%
  • Other == 22%
Clearly people are opinionated. And want to express themselves. And you don’t need a prize to get people to fill in a survey.
In fact. Surveys have shown that if you do give a prize, you can’t trust the results.
Meaning. You can trust the results of this survey all the more.

And lastly, We Asked “Is Software Testing Certification Evil?”
  • Yes, Obviously == 22%
  • No, Of course not == 22%
  • Maybe, I'm not sure == 31%
  • Undoubtedly, my Holy Book lists certification as one of the main causes of discord in the world == 19%
  • Other (and no I won't explain why) == 6%
Pretty evenly split there. And maybe people just answered joshingly, but only 22% of our respondents claimed absolutely certainty that certification did not have the “Big E” connotation.
So time for a PR blitz from the certification providers me-thinks. And if any want to pay me for this marketing advice, please get in touch.

Favourite moments
Ah, the blooper reel. For as the credits roll, we can partake of the ‘other’ answers.
In terms of cost, there clearly exist people I could milk this scheme for all they had for they threw monetary amounts that no-one in the real world would pay for certification. Almost up to £1000 pounds or more for some people. Quite ridiculous and I’m sure none of the other schemes would charge anywhere close to that.
Some really good ‘stuff I want in the pack’ answers. Which certainly made me want to get one.
  • Hosted online certificates
  • A Propeller hat
  • Doughnuts
  • Test
  • Discounts on framing services
  • Cuddly Certification Toys
And some people saw parts of the page in Dutch which I thought splendid.

So there you have it. Self-certification clearly has legs.
And if you see any certification providers starting to give out Doughnuts on their training courses – you know where they got the idea.

Friday, 4 March 2011

The Cross-Disciple Pirates and the Canon of Test Techniques

I used to consider incorporating techniques from other disciplines into testing as something a little different. It felt right, but since the ‘industry’ didn’t do that, it seemed like a way of individually revealing our personal approach to testing.
But testing has a secret history. The building of the Traditional Testing Canon has remained shrouded in mystery until now. So for all testers following tradition, set yourself free, continue to follow Tradition, just follow the one true one. As I reveal here as “Thee True And Aythentic Historee of Software Testing - A tale of Action and Adventure”.