Monday, 31 October 2011

Build your own model of software testing – or rediscover one from several thousand years ago

I was working out the kinks in my high level software testing model, and, through a process of speed reading and stichomancy I found that I have re-created an early Buddhist doctrine.
In “The Story of Chinese Zen” by Nan Huai-Chin, I find listed the five Skandhas:
  • form
  • sensation
  • conception
  • activity
  • consciousness
I was boiling my model down to:
  • model
  • observe
  • intent
  • manipulate
  • reflect
I’ve re-ordered my list to  tie in more closely to the Skandhas.
Quite a useful coincidence.
Below I list a simple set of my correspondence ‘tween the lists.
I have ‘model’ instead of ‘form’ because our world comes to us from our perception of it, not from it itself. Perception allows us to experience bias and hallucination, and provides the scope for us to change how we perceive.
As testers ‘sensation’ comes to us through our awareness, with our observation. We have to learn how to expand our range of observations and utilise tools to help us observe. Acts of observation can help us turn noise into data and subsequently into information which we can act upon.
When we test with intent, we bring purpose into our testing. We know what we set out to do/explore/check/exploit/etc. We often move off the beaten path and open ourselves to surreptitious happenstance, but only if we observe that happening can we utilise it.
Manipulate – the favourite ‘bad’ word of the hypnotist, although even as a hypnotist I felt happy using it, and as a tester I do it frequently. Shaping the system through my action.
Reflect, if all we did was progress from our initial intent then we would not learn. We reflect, to learn, move on, ever better, and more deeply.

And so, dear reader. Did you create your model yet? If so, see where else you can find it!

Follow on Reading:

Thanks to James Lyndsay for the recent chat about correspondences between my model and his.

Sunday, 30 October 2011

Push your software testing personas to the limit

The notion of personas never really worked for me. “Bob is 35, single and likes kittens...” Blah Blah Blah.
Clearly Bob has all the characteristics of a fictional closet psychopath.
And that works better for me. “Bob is a closet psychopath”. I can use that sentence to inform my testing. I can attempt to  test like a closet psychopath.
Other personas you might want to adopt:
  • Sociopath
  • Psychopath
  • Paranoiac
Of course, those are just the most obvious examples of personas we could use in testing.
“Alan, you’ve clearly misunderstood the point of persona’s”
Well, persona’s have a rock solid history that the UX community don’t seem to promote. So its time to remind ourselves of what we’re really doing.
By looking at the true history of the persona we remind ourselves of techniques that help us build personas more effectively and quickly.
But I warn you now. Persona testing ain’t no walk in the park, princess.
And duly warned, we proceed to the history lesson…
Everyone knows the word Avatar now. Since I haven’t seen the Cameron movie I don’t immediately think of a blue alien animated thing.
I think of an Avatar as a personification. So a persona and Avatar merge and become one and the same.
Which neatly reminds  us that the Hindu mythos has an Avatar as an incarnated god.
And that Gods don’t incarnate in just one mythos.
Other mythos offer similar approaches. From the Vodoun, we can take inspiration from the Loa and from the techniques of the Houngan. We could attempt the deliberate summoning of a Loa and the subsequent possession to help us fully manifest a persona as part of our software testing process.
Imagine, in your next morning standup, if you adopt persona techniques in your testing, you could truthfully say:
“This morning I am going to eat raw chillis and cover myself in chilli rum to summon forth Baron Samedi. Much merriment will ensue as I perform some usability testing on the latest release.”
Clearly some of you reading may imagine that I mean this metaphorically. You may imagine my hinting that you should examine the Nanchons of Loa to find inspiration for your personas. Good for you.
The true testers among you will have already begun constructing your Vodoun altar.
Some of you may find your sensibilities more in keeping with a Western Magical tradition. If so, then fear not.
The western magical tradition has a rich set of rituals for the invocation of Gods. Looking up invocation in pretty much any book of ceremonial magic can kick start your use of possessional persona processes.
For those of you without a bent towards the traditional. Modern magic has taken the use of invocation ever further by drawing down fictional Godheads. As exemplified in Phil Hine’s Pseudonomicon, harnessing the Lovecraftian realm.
And I could go on, and on. Clearly I’ve barely scratched the surface of inspirational sources for your persona testing.
It would be inconscionable of me to leave you with a warning from the Pseudonomicon itself, which because of the power of reverse psychology will probably do exactly nothing of the sort:
“working with the Cthulhu Mythos is dangerous due to the high risk of obsession, personality disintegration or infestation by parasitic shells”
Yup consider yourself warned.
Treat the use of Personas in Software Testing with care.

Saturday, 29 October 2011

Build your own model of software testing – “the quotes”

Have you tried to build your own definition of Software Testing? One that you can refine as you learn more stuff and the years go by?
That never worked for me. I don’t appear to align myself well with definitions and classifications.
Building my own models however, now that works better for me.
I have started work on a new model. I want to create a simpler meta model of my testing process.
I draw inspiration for this activity from various sources, some I have listed below:
"I must create a system or be enslaved by another man’s
I will not reason and compare: my business is to create."
William Blake, Jerusalem
When we try to convey thought by writing, we are bound to sit down solidly, and construct a holy Qabalah out of nothing.
Aleister Crowley, Magick Without Tears
Tao was always nameless.
When for the first time applied to function, it was named,
Inasmuch as names are given, one should also know where to stop.
Knowing where to stop one can become imperishable.
Tao Te Ching, as translated by Ch’u Ta-Kao
It is the part of the scientist – of the intelligent and honest man of letters and of the intelligent and honest clergyman as well – to entertain heretical and forbidden opinions experimentally, even if he is finally to reject them.
Norbert Wiener, ‘God & Golem, Inc.’
You keep using that word. I do not think it means what you think it means.
Inigo Montoya, as channelled by William Goldman for “The Princess Bride”
You’re on your own. And you know what you know.
And YOU are the guy who’ll decide where to go.
Dr. Seuss, “Oh, the places you’ll go!”
Do you have any quotes that inspire you that you would care to share?
More on the model… later.

Tuesday, 20 September 2011

Running out of email addresses when you test?

I generally test web apps. And Web apps generally use an email address as the unique identifier. So by test number 2, some of you may have run out of email addresses to test with.
If this happens to you, don’t panic! Because here are the Evil Tester hints and tips for getting more email addresses than you probably ever wanted, but as a tester, have always needed.

Tuesday, 28 June 2011

How Can I Estimate My Testing?

Have you had anyone ask you a question about estimation? I get asked these types of questions and I suspect that the person really wants answers about how to communicate and justify their guesses.
I think they hope that some process exists which will accurately and objectively give them a set of numbers. And by using these numbers they can disavow responsibility for the production of them. And no-one will hold them responsible if the objectively produced ‘estimate’ does not meet reality.
Well in reality ‘Estimate’ really equals ‘Guess’.
So if you want some strategies:
  • Answer the question you want them to ask
  • Ask how long they want it to take
  • Trust your gut
  • Assumptions, Risks and Issues
  • Track the passage of time
I could add an etc. in there, to alert you to the incompleteness of all of this.
But I won’t.
You should assume incompleteness in my presentation. And map it on to your model to identify the blanks.

What culture do you think you work in?

Do you think you work in a culture where ‘estimates’ and ‘actual time’ have become synonyms?
Do ‘they’ berate you for continuing to work, after the ‘estimated’ length of time has passed?
Well, you have learned that regardless of where the estimate came from ‘they’ will still hold you responsible for it.
So you need to develop some beliefs. You need to believe that “estimate equals guess”.
You need to believe that any supporting documentation for your guess does not provide a justification for it, it provides a partial model of the thinking that led to the guess.
‘They’ don’t need to believe any of this.
You need to.
Then your communication will change.

Estimates equals Guesses

All estimates equate to GUESSES.
We don’t know how long things will take.
I’m a pretty good psychic. But even I don’t know.
Sometimes we put a framework around our GUESSES. And that can reveal some of our ASSUMPTIONS. It can reveal what RISKS we have considered. It can reveal what ISSUES we perceive.
The framework reveals part of the model we think we used when creating the GUESS. It helps us explain how we created the GUESS.
But even when we do that, our estimates remain GUESSES.

So what could you do?

Do you try and ‘educate’ them?
I don’t.
I’m obnoxious like that.
  • So if you get asked “How long will your testing take?”
    • Don’t say “I will provide you with an estimate for testing.”
    • Say “I don’t know.”
    • The conversation might end there.
You might respond with the question you want them to ask.
  • So if you get asked “How long will your testing take?”
    • ask “Do you mean? If I had to guess, right now, how long I think it will take for you to have enough information that you can decide whether or not you want to release this? If so then…”
    • Yup. I say stuff like that.
    • Because then we start talking about what they really want to know.
    • I don’t think I engage in an education process. I try and communicate so we all work from the same assumptions and understandings.
The test managers among you might feel a little nervous with this. As a test manager I used to feel nervous saying that. So you might ask instead:

“How long do you want it to take?”

We live in a world of deadlines. So guesses often don’t help very much.
By asking “How long do you want it to take?” you can build a plan which tries to maximise the value that you can add as quickly as possible before the deadline. Work on the high value items, risks and issues.
Don’t worry about the guesses. Think about the constraints.
When you ask “How long do you want it to take?” then you can think about how much stuff you might fit into that timescale, does it ‘feel’ right?

Trust Your Gut

What you ‘feel’ can feed into your guessing process.
You can get the whole guessing process over really quickly if you just come up with number.
You won’t have stated any assumptions. You won’t have considered any risks. you won’t have any weighted factors. But you can come up with a number quickly.
And your gut will tell you how you feel about it.
Does it feel right?
More importantly – does it feel wrong? If so, you can:
  • Ask yourself what feels wrong about it. Bam. Now you’re starting to come up with assumptions, risks and issues.
  • Come up with a new ‘better’ guess.
When you immediately decide to come up with a new ‘better’ guess. Then you trusted your gut.
If you ask your questions. You distrusted your gut. So you ask those questions to help you regain your trust in your gut.

Assumptions, Risks, Issues

Why bother with the questions?
Why bother with Assumptions, Risks and Issues?
Well, they help you communicate.
Until you develop the force of personality where you can give a number and have it accepted as a best ‘guess’. You might want to engage in a communication process.
If you do want to communicate then you can explain to people the things you have assumed. e.g.
  • I assumed we deliver a release candidate on day X
  • I assumed we have a clean build with no failing unit tests
  • etc.
You can do the same for risks and issues.

We track the passage of time

If your guess gets shot down and ‘they’ convince you into using a shorter guess. Then you can ‘manage’ the on-going process by monitoring for any:
  • negated assumptions
  • change in riskiness of the risks
  • transformation of risks into issues
This may not help in the short term as you simply point out that the correctness of the guess becomes ever less likely.
But you might help build credibility long term by communicating based on your frame of reference.
We track over time, the ‘goodness’ of our progress, and how much distance we appear to have to travel before we reach our end point.
We validate our model over time, by comparing it with the reality that we perceive.

Have you covered everything?

No.
I just want to encourage you to keep it simple.
I think you could waste time on a complicated ‘estimation’ process using spreadsheets and lots of factors.
‘They’ might well think they need those spreadsheets. Their process may require those spreadsheets. If so, use the spreadsheets to communicate your model, don’t use them to create the guess.
Of you could:
  • guess,
  • start to do the work,
  • based on how long it currently takes extrapolate forward and…
  • guess again
If you do this then you need more courage.
Swapping a belief of “objective estimation”, for “guesses and underpinning models” takes time.
I will not try to convince you that you should.
I just want to make you aware that you could.

Thursday, 2 June 2011

How can I learn to automate my testing using Selenium?

In this blog post, I’m going to lay out the Evil Tester free and simple 8 step guide to learning how to automate a Web Browser using Selenium.
I’ll reveal the sources of information you can use.
And yes, I will promote my book.
music by Drongomala @ flyingmountainrecords.com
I see the same questions, and variations on these questions, asked on social network forums every few days.
    • “How can I learn Selenium?”
    • “How do I use Selenium?”
    • “What do I read?”
    • “How do I get started with Selenium?”
    • “I want the answers. Please do the needful”
For some reason the previous answers don’t stick. Or people can’t find them. Or maybe they don’t look for them. Or, something… I don’t know.
But a lot of people have tried very hard to make Selenium easy to get into, and to help get you started. So if you are prepared to put the work in. The Answers are out there. Just don’t expect someone to magically drop the information into your brain, you have to put the work in to follow the learning steps.
Expect to put in some work.
If you are prepared to do that.
Then the pointers in here will help you.
First learn a bit about Selenium
Start with the basics.
Research what the tool sets out to do. Visit the official site. Have a look around.
Need/Want more than the Official Documentation
You can read two books:
I wrote a comparative review of them here:
Both David and I wrote our books when the Official documentation was harder to get into than it is now, so we aimed these books at the beginner. If you consider yourself a beginner and find the official documentation too hard. Then get hold of these books.
The official documentation has improved massively and the Selenium team have done a great job. These books go beyond the official documentation and add additional value. Sometimes people need more in depth tutorials, and more examples, more code samples, and these books provide that.
Selenium is Open-Source, Learning should be free too
OK, if you say so.
Both books have free previews. And these are big previews. Last time I checked mine was 75 pages. At the end of which you will have installed all the basic tools. Recorded a test in the IDE. Converted the test to Java. Run the test in the Eclipse IDE. Learned how to debug tests in an IDE.  Learned how to start and stop the Selenium Server programmatically. And given pointers of where else to look for more information. For some training courses, that would be day 1.
They cover:
  • installing the tools you need,
  • recording your first script in the IDE,
  • converting that into program code – because you do not want to get stuck in the IDE
And all of that in the free previews.
My book leads you through the process of automating some basic web pages and covers automating most of the common HTML features, automating AJAX, and refactoring your tests into Page Objects and building an abstraction layer. And even through the steps needed to put your tests into continuous integration. All in a consistent tutorial guide.
Read this comparative review, follow up the links, read the previews and tables of contents and see which books meets your needs.
More than enough to get you started. Enough to have you install the tools. Enough to get you moving.  In just the free preview.
Search for blog posts on how to start, here are a few searches to try:
And let’s not forget the official site http://seleniumhq.org
A lot of info out there. Some of it will be out of date. They will repeat and overlap. They will not be presented in a consistent style. But you can get all the information you need without buying the books if that is the learning style you prefer. You will learn from your mistakes as you follow the instructions. That learning will help you.
Where can I get help when I start learning?
The official site has a page all about that:
The bigger point though is not “where” but how you use that help.
  • Don’t ask generic questions.
  • Show the source code you were using when you got stuck.
  • Make sure your learning how to debug the scripts and have investigated it as far as your knowledge will take you.
People have put a lot of effort in simplifying the learning of Selenium but it still requires you to do some experimentation and make an effort.
You are learning how to code, and do some complicated stuff. Automation often requires workarounds, so if you don’t get into the habit now of trying different things, of exploring the API, of searching for tips around the area you are automating then you are doing yourself a disservice in your learning phase.
What else can I read?
I’m going to give you one link:
http://seleniumhq.wordpress.com
Adam Goucher does a great job, scanning blogs looking for material on Selenium. So follow up the links. That will take you to more blogs. Read those. Subscribe to those in your rss feed.
But I hate reading!
Fine. There are plenty of videos out there. Lots of Videos out there:

This all sounds like hard work
The learning process requires that you make an effort.
But its your choice. You do not have to learn how to use automation tools. Personally I think that will help your career and provide you with more options in your test approach, but you don’t have to learn it.
If you choose to learn it, then you have to put the effort in.
  • Learning involves Experimentation.
  • Testing involves Experimentation.
You already know how to do both those things.
Well, Automation involves experimentation too.
And remember – if you get stuck
Remember there are two books which aim to help you learn Selenium.
I wrote Selenium Simplified for the very reason that people do get stuck, so it provides step by step instructions for getting you unstuck and productive in Selenium.
Time to start now
The onus is on you. Right Now.
Just start following the 8 step process:
Step 1 - Read the official site and selenium documentation
Step 2 - Start trying to use Selenium using the documentation to help you
Step 3 - Investigate the books available, use the previews to get started
Step 4 – Auto-generate scripts from the IDE
Step 5 - Use Google to find tutorials, blogs and videos
Step 6 - Use stackExchange, linkedin and the Selenium Forums for specific help. Read the official support page
Step 7 - Read through the old issues of "A smattering of Selenium"
Step 8 - If you get stuck, and can't make progress - buy one of the books. You can apply this step at any time.
You can learn Selenium. Start Now.

Tuesday, 31 May 2011

Use Real Words to Communicate, Not Testing Phrases

I think people use ‘standard’ testing phrases too much. e.g.
  • System Testing
  • Testing
  • Performance Testing
  • Functional Testing
  • Non-functional Testing
  • Black box testing
  • Unit Testing
  • etc. etc.
The user of each phrase holding the assumption that the reader/listener of each phrase understands the phrase the same way as the user. They don’t, they have their own models that underpin that phrase.
Try using real words.
One recent example from the world in which I inhabit:
  • I could have said “we need to build automated integration tests for these stubs”.
  • Instead I said “we need write some automated scripts which check that our stub behaves the same way as the system we have stubbed”.
  • I received the answer. “Ah, I call those , <xyz> tests” (I forget the actual words of the <xyz> label)
  • I said “Fine, let’s write some <xyz> tests”
Now we have a label, for the concept. And more importantly we have agreement on what we intend to do.
I used real words to communicate.
You can do this for all aspects of your testing, resulting in contextual communication of your test planning process:
  • performance  testing might become “check that the user can use the logged-in page in under X milliseconds after logging in”
  • load testing might become “check that the server (web and app) don’t show any signs of stress under the estimate load of the first two hours after launch (we’ll monitor it on the server side”
  • any testing phrase” might become “some other set of words in your language of choice that communicates what you mean”
Feel free to inform your planning process, if you must, with concepts such as “performance testing” or “load testing” or <insert any other testing phrase here>. But do not use them to communicate.
I think that eventually you will wean yourself off those testing concepts and start to work from first principles and communicate in simpler, non-buzzword and less ambiguous terms.
I recommend you try it:
  • to communicate your test planning.
  • when asking questions on LinkedIn.
  • when thinking.
  • just try it.

Thursday, 21 April 2011

Tutorial on Burp Suite Repeater and Intruder

This tutorial on Burp Suite covers the Repeater and Intruder functionality.
  • Repeater allows you to play back a message to the server and amend it before it goes out.
  • Intruder allows you to play back messages, with various elements of the message varying with each playback e.g. a different set of parameters
In the tutorial video I explain how I use each function in my testing.
Other proxy tools do similar things. I also use JBroFuzz as it provides very similar functionality, so experiment with that too.

Did you can spot the deliberate mistake in the video? Blogging helps you make mistakes in public very easily – which helps you learn quickly. Any comments on the video will help me improve the quality of future videos.

Tuesday, 19 April 2011

No Excuses – Learn Burp Suite to aid your web testing

In March 2011 I gave a talk at the London Sigist on technical testing. I’ll make the slides for that available (…sometime soon). I didn’t want to give a ‘blaggers guide’ to technical testing. So I presented an overview of some of the thought processes and models I use.
At the end of the talk I provided a list of tools that I use. I use Burp Suite as one of my proxy servers.
I currently have a “No Excuses” hat on, so I currently try to provide as much information as I can in bite size chunks which people can pick up and move forward with. I wrote “Selenium Simplified” as a “Now you have no excuses for not learning how to program” book.
I recommend that if you want to go further with technical web testing you read the book “The Web Application Hacker’s Handbook” written by the people behind the Burp Suite tool. [amazon.com] [amazon.co.uk]
And in the same spirit I will now experiment with some videos. In the first of which I provide a simple overview of Burp Suite, in particular the Intercept and Site Map functionality. I don’t cover the nuances of usage, but I cover enough to get you started. So if you haven’t started using a proxy server as an essential part of your web testing… no excuses – start here:

I still have a lot to learn about creating video tutorials, so I appreciate all comments.

Wednesday, 30 March 2011

Software Testing Club – Testing Tales released

A while back, Rob Lambert started collecting Nursery Rhymes for the Software Testing Club.
Since I enjoy Humpty Dumpty I created a version for STC. It has just become available on the Software Testing Club website, along with Rob’s entertaining take on The Gingerbread Man.
They have posted the uncensored version of Humpty so those of a weak disposition take heed of this warning before visiting said site.

Sunday, 27 March 2011

The Untold Story of the Test Plan

We all know by now the following things about test plan documents:
  • a test plan document does not substitute for the process of test planning,
  • a test plan document represents one way of communicating the results of the test planning process,
  • the process of test planning stops only when you stop testing (even then, you might obsess so much that you carry on planning)
I didn’t always know those things.

Wednesday, 23 March 2011

Selenium 2 makes automation debugging easier

One of the parts of Selenium 1.0 that I never enjoyed was debugging automation that didn’t work. I had to faff about creating custom Firefox profiles with Firebug installed and set to go through a proxy.
Selenium 2 makes all of that so much easier. With the code below, my test runs through a proxy server on port 8081 and has Firebug installed so that I can breakpoint my test and then go debugging around in the browser DOM, debug the JavaScript to make sure events get fired, etc. etc.
I found most of the information to do this on the Selenium Forums http://groups.google.com/group/selenium-users
The code should be easy enough to follow if you are already using Selenium 2

import java.io.File;
import java.io.IOException;
import org.junit.Test;
import org.openqa.selenium.Proxy;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.openqa.selenium.firefox.FirefoxProfile;

@Test
public void FirefoxUseExtensions() throws IOException{

  // setup the firefox proxy to point to Burpsuite
  // on port 8081

  Proxy localhostProxy = new Proxy();
  localhostProxy.setProxyType(Proxy.ProxyType.MANUAL);
  localhostProxy.setHttpProxy("localhost:8081");

  // Prior to running the test
  // Download the firebug extension file
  // to a local folder

  String extensionPath = System.getProperty("user.dir") 
      + "/"
      + "firefoxExtensions/firebug-1.6.2.xpi";

  // create a custom profile

  FirefoxProfile profile = new FirefoxProfile();

  // set the profile to use the proxy settings

  profile.setProxyPreferences(localhostProxy);

  // stop firebug showing the first run
  // screen by setting the 'last version'
  // to the current one downloaded
  // try without this and see what happens!

  profile.setPreference("extensions.firebug.currentVersion", "1.6.2");

  // add the extension to firefox

  profile.addExtension(new File(extensionPath));

  // start firefox with the custom profile

  WebDriver driver = new FirefoxDriver(profile);

  // go to your favourite testing web site

  driver.get("http://www.eviltester.com");

} 

Monday, 14 March 2011

Paranoia as a learning and testing strategy

Honestly who doesn’t enjoy reading conspiracy theories? Who doesn’t enjoy putting on the “Helm of Paranoia”.
I use paranoia as a learning strategy and as a testing strategy.
You can too…
For learning:
  1. Assume that the author of the book lied to you
    • This forces you to do more research
    • Read additional books
    • Read differing opinions
    • Check their facts
  2. Assume the author didn’t want you to know the truth
    • They deliberately obfuscated their writing so you have to simplify them
    • They deliberately made it overly specific so you would not learn the different ways to apply the technique, so you have to generalise it
    • They deliberately didn’t teach you the principles, so that they would retain the edge and you could never compete. So you need to take their approach apart to identify the principles that it codifies
  3. Assume the author didn’t know the truth and others misled them
    • Only you can draw out the patterns in the text to identify the underlying earth shattering secret truth that has remained hidden in the text.
For testing:
  1. Assume that the system has lied to you
    • Yes it ‘says’ it has saved the record – how do you know?
    • Yes it ‘says’ you can’t use numbers in the password – how do you know?
  2. Assume the system didn’t want you to know the truth
    • Look for the hidden buttons that others have overlain and obscured
    • Look for the hidden windows
    • Look for the hidden div elements (how can you make them appear)
    • Look for the hidden form fields – what secrets do they hold
    • Look for the secret cookies – what do they mean? What information have they tracked about you?
    • What other secrets has it hidden from you?
  3. Assume the system doesn’t know the truth
    • Can you mislead it?

The truly paranoid among you will now rightly assume that I have lied to you, that I don’t want you to know the truth, and that others have so misled me that what you have here is a pale reflection of the truth that only you can uncover.
And for your ongoing research pleasure, I heartily recommend Paranoia Magazine. I haven’t read it for a long time as I haven’t found a shop that will stock it – no doubt due to the machinations of one cabal or another. But that won’t stop you getting a good old historical dose of paranoia with the free back issues at http://www.paranoiamagazine.com/freedownloads.html (Just don’t tell them I sent you, they don’t need to know.)

Saturday, 12 March 2011

Sub-cultural Testing Influences gone mainstream #1 – The Assassins Creed

My testing style, attitude and approach has had many influences. I only recently realised that one of them has become very mainstream.
"nothing is true, everything is permitted"
Words attributed to Hassan-i Sabbah on his death bed. Specifically ambiguous and open to misuse. Perfect for testing.
This went all mainstream with the “Assassin’s Creed” game series
Fortunately when dealing with software systems, many of the moral dimensions of the Assassin’s phrase disappear and we no longer have to engage in the philosophical debate about boundaries of human actions. We just get on with the testing.
Objections such as “No user would ever do that” have no stopping power on the Assassin influenced tester.
If the system allows me do it, and I observe that the system allows me do it, and I figure out how to manipulate the system into doing it, I do it.
This phrase helped me through some difficult and turbulent waterfall testing times.
Try it on and see how it fits your testing style.
Just remember to treat software systems and people systems differently.

Thursday, 10 March 2011

Test Techniques Evolve

Observation –> Taxonomy Entry –> Heuristic –> Technique –> Children’s Story –> ?
  • I may have some of the arrows the wrong way around in the above diagram. After all, who really knows what influences what.
  • And as to the “?”, I hope that you decide what comes next.

Tuesday, 8 March 2011

Dangerous Test Concepts Exposed

There exist test ‘concepts’ which, while seemingly simple, have a tendency to melt my brain.
  • Black Box/White Box Testing
  • Functional/Non-Functional Testing
  • Positive/Negative Testing
Years spent studying hypnosis and revelling in the ambiguities of communication have left me with an inability to parse language the way I did as a child.
I used to have the ability to read, and use, these terms without blinking. Now I throw a ParseException.
I realise the above concepts may not seem strange to the majority of people reading this blog post. So I will attempt to describe how I respond to reading these terms, in the hope of spreading a contagion against them.

First some generalities

First some generalities across all the ‘concepts’.
I think that I think, and test, in terms of Systems. More specifically in terms of models of systems.
Yes/No models jar my sensibilities. Testers lives in the grey areas and find treasures in the shadows. Absolutes do not help me explore the System. I do not want guidance by absolutes, I want guidance towards flexibility.
All of these concepts encourage me to think and communicate in ambiguities, while pretending specificity and authority. If I say White Box testing, you still have no idea what I mean. If I describe the models that I use to guide my testing then you have a better idea of the type of tests I might create, and the type of gaps I may encounter, regardless of the colour that I paint that model.
These concepts all encourage me, as a tester, to ignore the model and think in vague conceptual terms. They categorise techniques out of context and far removed from the model that the techniques apply to.
They encourage debate about irrelevancies. e.g. Should we do White Box, or Black Box testing? Developers do White Box Testing, testers do Black Box Testing? Where does Grey box testing come in?
The amount of time wasted on projects when people ‘debate’ if the requirement should form part of the ‘functional’ or ‘nun-functional’ specification, instead of getting on with communicating what they need shocks me. (I estimate this in terms of man-years for some projects)
etc.

Black Box / White Box Testing

Ah, the most pernicious of them all.
How comfortingly I viewed these terms in my early years. I could ignore the relationships between my test models. I remember times when I made no effort to learn the technology or architecture of the system. I could simply focus on inputs and ignore flow, because I did my testing all “Black Box” like.
Then over time I grew uneasy. I started to apply White Box techniques to ‘Black Box’ingly understood systems.
This concept started to seem like the devil on my shoulder whispering “Relax, don’t do the work to view it as a system, treat it as a ‘black box’, see how much simpler it all becomes”.
I started to feel as though the constant repetition of this phrase in the testing literature stopped me hearing the siren call of the technology.
I weaned myself away from these terms by using them as insults. I engaged in a form of aversion therapy.
I labelled all testing I did without understanding the technology, or without having made an effort to observe the architecture, or without building a structural model as ‘Black Box’ testing. I imbued the term with an association of laziness.
I view all my areas of superficial modelling, as having greater risk of defects unfound.
I associated my use of ‘White Box’ testing with over-confidence. I felt nervous if I only read the code, or only based my testing on a superficial understanding of the structure. For me, the phrase warns me about my complacency. I try not to plan for complacency.
This helped me move towards a more Systems focussed approach to my software testing. I no longer feel a pull towards exclusively the Darkness or the Light.
I try to concentrate on models and identify techniques to apply to models.
I ignore the categories and partitioning of techniques presented by the Canon.

Functional/Non-functional Testing

I periodically write non-functional automated test scripts… Sometimes my scripts simply don't work and they remain non-functional until I fix them to get them functional. <insert canned laughter here>
In the past, I have communicated my testing in terms of “I built tests to cover the ‘Non-Functional Requirements’”. Which in context could have read “Speed Monster Funky Requirements” and I would have just as happily used that phrase in my test communication.
When we mean requirement based testing, then we should say “requirement based”. Removing the word “requirement” suggested to me that “Herein lies magic”. That the “Functional” tester does something different from the “Non-functional” tester. That we can phase the testing differently and do “Functional” testing and then later do “Non-functional” testing.
I removed my unease towards this phrase by always appending/inserting the word requirement.
But … I don’t like the distinction between “Functional” and “Non-Functional” because of the ineffectual processes and discussion that I have seen result from the distinction. Sadly Software development has a history of considering functionality as separate from the other “ilities” that might relate to that functionality. A tradition or an old charter, or something.
And yet I don’t see why I should let this influence my testing. I’ll do whatever ‘it’ takes, and if that means merging the functional with the non-functional to get more advantages in my testing or tooling then I will jolly well do so.

Positive and Negative Testing

I assume “positive” and “negative” as testing terms, must have come from a more objective diagnostic context than I get involved in. From a context where a negative test, rules something out completely, and a positive test rules it in.
I classify much of the testing I do as exploratory or experiment based. I do things to try and rule out certain possibilities and I could possibly count that diagnostic. But I don’t think my tests demonstrate the certainty that “positive” and “negative” testing suggests to me. I find more value in thinking in terms of a general experimental process rather than a diagnostic process.
But even the above description doesn’t seem to map to how the testers I meet describe positive and negative testing.
Testers seem to describe negative testing in terms of exercising error validation and reporting. And positive as “stuff we think will work”. I tend to view the existence of error handling functionality in applications as a positive thing.
I find it hard to use the terms –ve and +ve testing. When I try to show that the system does not work, I talk in terms of trying to ‘break’ it – clearly I don’t break it, but it works as a term for me. I also view this as a positive thing for me to do.
Yes, I freely admit to misusing words, particularly if the misuse of the word frees my thinking. I try to stop using words that constrain me.

End Notes

For all of the above ‘concepts’ I can read the definitions and almost understand why those names fit those descriptions. But I try to find other, more exact, or more contextual, or more personal, ways of describing the testing that I do.
You may not need to go as far as I did to de-programme their effect.
Younger testers may even have an immunity to them.
I still feel their pull though. My brain still finds a ‘reasonableness’ in all of them while at the same time each of the ‘concepts’ strikes me with dread. Each of them seems to proudly state that we can easily view the System as two simple partitions and we can relax about modelling or understanding the complexity.
If you speak to me using these terms, I will have to ask you what you mean. Do not assume that my testing background will allow me to understand you.
I try to stand firmly in a graduated shading testing scheme. Sometimes I even imagine colour in my test approach. Someday I will attempt to employ synesthesia.

Monday, 7 March 2011

The results are in for the Evil Tester Certification Survey

Did you enter? Probably not. And if not, you lost your chance to save the world.
The survey will remain open as I found many of the answers highly entertaining. So feel free to pop random entertaining snippets into the survey and I shall read them and chuckle.
Dare I draw conclusions from the survey, given some may have viewed it in jest?
Of course.
But first… three serious points.
  1. I take software testing incredibly seriously. I poke fun at it. Because I take it seriously.
  2. If it did not seem like a serious topic to me. I wouldn’t bother poking fun at it.
  3. I consider software testing so serious a topic that I need to think about it, not just coast along with the Status Quo. I can’t help find humour in it if I think about it. We laugh to change. If I don’t find a way to mock it then I probably haven’t thought it through from enough angles.
  4. Do you find testing serious enough to laugh at?
So, on with the statistics and numbers. The closest you will come to metrics on this web site.

We asked “Do you hold a Certificate in software testing?”
Our Survey said…
  • Yes == 56%
  • No == 44%
Proof positive then that the people who replied know what they replied about. I think.

We asked “Did your certification offer value for money?”
Our Survey Said…
  • I'm not certified, didn't you read my answer to question 1? == 41%
  • Yes == 13%
  • No == 31%
  • Definitely not, I got my money back == 0%
  • Other == 16%
Contentious. Some people didn’t think it offered value for money, BUT DIDN’T ASK FOR THEIR MONEY BACK!
Meaning – this could become a profitable business to get into. No-one asks for their money back.
Certification providers take note, you can drop your standards and it won’t make any difference.

We asked “Would you like to self certify yourself in Software Testing?”
Our Survey Said…
  • Yes, clearly that would be better == 50%
  • No, I can't possibly know better, or as much, than the official certification experts. == 3%
  • Never! What new certification scam are you trying to pull? == 47%
It seems as though some people saw through my thinly veiled attempt to see how much appetite exists for self-certification. And 50% think it sounds good. So before the certification crowd jump on this particular bandwagon. This one belongs to me.
And 3% thought they didn’t know as much as the certification experts. Which clearly (in an abuse of the survey process, but in keeping with poor statistical analysis) means that 97% of people doing the survey think they know better than the certification experts. And therefore they consider self-certification a good idea.

We asked “If a self-certification scheme did exist, how much would you pay for the privilege?”
Our Survey Said…
  • Nothing, it should be free - like all good things in life == 63%
  • < £10 == 3%
  • >= £10 && <= £50 == 16%
  • >£50 && <= £100 == 0%
  • > £100 (I only value something if I pay a lot for it) == 3%
  • Other == 16%
I don’t know what you see from these results, but I see:
  1. a free sample to hook in the 63%
  2. an immediate uptake of 35% of respondents
  3. hopeful conversion of the 63% to a total of 50%
Self-certification as a potential goldmine.

We asked “If it did cost money, what would a self-certification scheme pack have to contain so that you felt you were getting value for money?”
At this point I’m giving away the secrets of my future success. Because you too could use the information here to create a money spinning self-certification pack.
Our Survey Said…
  • Sample Certificates to fill in == 25%
  • Sample Answers to help you show off your newly certified knowledge in interviews == 25%
  • Empty promises about industry standard knowledge and future career prospects == 31%
  • Hints and tips on how to fill in your certificates == 22%
  • Affiliate links to lamination services so you could create laminated certificates == 22%
  • A direct debit form so you could continue to pay for your certification every year (and thereby 'prove' that you were still worthy of the certification) == 13%
  • A letter congratulating you on your decision == 28%
  • A document explaining example topics you could certify yourself in == 28%
  • Multiple levels of certification in one pack (e.g. novice, journeyman, master, expert, Nobel Prize Winner) == 25%
  • A Membership card == 38%
  • A Badge == 41%
  • Money off vouchers for additional certification packs == 16%
  • Other == 50%
Clearly people want to experience a sense of belonging – hence the popularity of the badge and membership card. This may explain why the certification sales pitch based on Social Proof (everyone else does it, why don’t you) does so well.
The direct debit, whilst looking good from a money angle, doesn’t seem that popular, so CAT’s compulsory re-certification scheme might not go down too well with the paying masses. The CAT folk might want to consider changing that aspect of the scheme. But since they didn’t get back to me with previous comments I sent them on their scheme, they might not take any notice of this one either.
So quick wins for all certification providers. Supply a badge and a membership card to boost sales.

We Asked “Do you wish this was a prize survey?”
Our Survey said…
  • Yes, I only fill in surveys where I can win a prize == 28%
  • No, I fill in surveys for the fun of clicking buttons and feel sure that the person on the end of the survey has read my 'other' answers carefully, so carefully in fact that I have undoubtedly succeeded in changing their mind about something or other == 50%
  • Other == 22%
Clearly people are opinionated. And want to express themselves. And you don’t need a prize to get people to fill in a survey.
In fact. Surveys have shown that if you do give a prize, you can’t trust the results.
Meaning. You can trust the results of this survey all the more.

And lastly, We Asked “Is Software Testing Certification Evil?”
  • Yes, Obviously == 22%
  • No, Of course not == 22%
  • Maybe, I'm not sure == 31%
  • Undoubtedly, my Holy Book lists certification as one of the main causes of discord in the world == 19%
  • Other (and no I won't explain why) == 6%
Pretty evenly split there. And maybe people just answered joshingly, but only 22% of our respondents claimed absolutely certainty that certification did not have the “Big E” connotation.
So time for a PR blitz from the certification providers me-thinks. And if any want to pay me for this marketing advice, please get in touch.

Favourite moments
Ah, the blooper reel. For as the credits roll, we can partake of the ‘other’ answers.
In terms of cost, there clearly exist people I could milk this scheme for all they had for they threw monetary amounts that no-one in the real world would pay for certification. Almost up to £1000 pounds or more for some people. Quite ridiculous and I’m sure none of the other schemes would charge anywhere close to that.
Some really good ‘stuff I want in the pack’ answers. Which certainly made me want to get one.
  • Hosted online certificates
  • A Propeller hat
  • Doughnuts
  • Test
  • Discounts on framing services
  • Cuddly Certification Toys
And some people saw parts of the page in Dutch which I thought splendid.

Conclusion
So there you have it. Self-certification clearly has legs.
And if you see any certification providers starting to give out Doughnuts on their training courses – you know where they got the idea.

Friday, 4 March 2011

The Cross-Disciple Pirates and the Canon of Test Techniques

I used to consider incorporating techniques from other disciplines into testing as something a little different. It felt right, but since the ‘industry’ didn’t do that, it seemed like a way of individually revealing our personal approach to testing.
But testing has a secret history. The building of the Traditional Testing Canon has remained shrouded in mystery until now. So for all testers following tradition, set yourself free, continue to follow Tradition, just follow the one true one. As I reveal here as “Thee True And Aythentic Historee of Software Testing - A tale of Action and Adventure”.

Thursday, 24 February 2011

Exploratory Testing Lessons from the Tao Te Ching

Tony Bruce posted some quotes inspired by Taoist traditional writing from The Magic of Metaphor. I have a particular fondness for Taoist classics and I have a fair few translations of the Tao Te Ching on my bookshelf. I love the feeling of simplicity generated when reading.
I cherry picked the following quotes with a relevance to my testing.
If you have not before taken the following approaches in your testing then I encourage you to try them (failing this you could just read “The Berenstain Bears Inside, Outside, Upside Down” as initially recommended to me by Rob Sabourin):

On Subtle Wisdom…
  • In order to contract a thing, one should surely expand it first.
  • In order to weaken, one will surely strengthen first.
  • In order to overthrow, one will surely exalt first.
  • In order to take, one will surely give first.

What is planted by the best planter can never be removed;
What is embraced by the best embracer can never be loosened.

On Absolute Equality:
  • Blunt all that is sharp;
  • Cut all that is divisible;
  • Blur all that is brilliant;
  • Mix with all that is humble as dust;

Chapter LXIV:
  • What is motionless is easy to hold;
  • What is not yet foreshadowed is easy to form plans for;
  • What is fragile is easy to break';
  • What is minute is easy to disperse.
  • Deal with a thing before it comes into existence;
  • Regulate a thing before it gets into confusion.
  • The common people in their business often fail on the verge of succeeding.
  • Take care with the end as you do with the beginning,
  • And you will have no failure.

All the above from the 1937 Translation by Ch’u Ta-Kao
You can find various translations around the web:

Monday, 21 February 2011

Can a new certification survey change the landscape of Software TestingCertification?

I doubt it, but what the heck. Let’s try. Inspired I was by Dot Graham’s explanation of the original reasons behind certification in “Certification is evil?”. Which in turn was inspired by the Software Testing Club “Tester is for life, not just for Christmas” e-book.
My views on certification in the e-book were “You’re better off becoming an ordained minister from the Church of the Subgenius for only $30.00 than putting any money, or time, towards the Testing Certification Scam.”