Friday, 19 December 2008

How to run Selenium-RC in a virtual machine

A tale of VMWare vs VirtualPC
One issue that I have with selenium and running tests through IE is the little popup dialog saying "The web page you are viewing is trying to close the window. do you want to close this window?"
Our tests use openWindow and then close them frequently.
I wrote a little AutoIt script to sit in the background and poll for that dialog and close it when it appears.
But sometimes the AutoIt script makes it difficult for me to use the PC at the same time as the scripts run, so I decided to use VirtualPC to run the Selenium-RC server. And that would surely solve my problems? But oh no, then my problems really began...

Wednesday, 17 December 2008

Selenium and HTMLUnit - the abstraction layer

I once did an experiment to see how easy I could wrap HTMLUnit with  Selenium for automated software testing.
In my experiment I created a wrapper by extending the DefaultSelenium class and then using eclipse to create wrapper functions for all the methods. Then inject that HTMLUnitSelenium class into my abstraction layer and voila - your Selenium tests run with HTMLUnit (*cough* well, in theory).
Then I recently discovered that someone else had done the same thing. Frank Cohen, at a recent workshop, mentioned a wrapper that already existed in PushToTest.
So I hunted it out, and I tried to use it...

Friday, 25 July 2008

Noob lessons for Software Testing

Raising Defects can challenge even the best of testers, as the tester at ClicheQuest Offices found out to his peril.
Lessons this cartoon has to teach us about raising, or not raising, defects - ed: 'so we should say "lowering" then?' (sorry, I couldn't resist):
  • "All we want are the facts ma'am" as Joe Friday would prompt.
  • Don't tell people what to do.
  • Work sensible hours
  • Get your own back

My way of surviving...


Evil is my way of surviving Development Hell.

Wednesday, 23 July 2008

Software Testing, A Funny Business

Steve Martin's Autobiography starts with the following phrase:
"I did stand-up comedy for eighteen years. Ten of those years were spent learning, four years were spent refining, and four were spent in wild success... enjoyment when performing was rare - enjoyment would have been an indulgent loss of focus that comedy cannot afford. After the shows, however, I experienced long hours of elation or misery depending on how the show went..."
I spent the first ten years of my testing career learning, since then I've spent the years refining and learning. I haven't enjoyed the wild success yet. But then I test software rather than perform stand-up comedy... but on some projects- who can tell the difference?

Tuesday, 17 June 2008

Functional testing JavaScript with QUnit - initial steps

With the JavaScript I have written - I describe the tests which exercise the functions in the .js file and manipulate the variables in the .js file as Unit tests. Then when I test functions which interact with the html page that I import the .js file into - I class that as Functional testing. In this exciting episode of "Adventures in JavaScript" I describe my initial attempts at Functional testing with QUnit.

For a while I thought how could I make test the functions which interact with the html page as Unit tests? Perhaps I should learn how to use JSMock to mock out the html elements that I want to use in live? But I decided to take the easy route and in the test html file, just 'mock up' a crude version of the form I want to use when I go live. By my definition, my Unit test suite started to include some functional tests.
So how does that help me? Well, suddenly I have a suite of 'unit' tests which I can use to test the browser interaction for cross browser compatibility. And did they find cross browser bugs - yes indeedy, but if I tell you now, then I get ahead of myself so... one step at a time.
Create a test to drive the creation of the 'mock' GUI.

module("functional testing");

test("html input output elements exist",function(){

 ok(document.getElementById("inputtext").innerHTML!=undefined, "input text field exists");
 ok(document.getElementById("eprimeoutput").innerHTML!=undefined,"output text field exists");


This test should only pass when I have some elements with IDs "inputtext" and "eprimeoutput". So I create a small form at the bottom of my test page to represent my 'production' environment.


Now I did a little more than the test suggested I should, but... hey ho.

My test passed, so now I can start writing code to output the variables stored in my .js file.

But I can't do that without a test, so...

test("display output text in html element",function(){
        outputText = "<p>boo</p>"

Then I can write the code for this:

 function writeTheOutputTextToTheScreen(){
  document.getElementById("eprimeoutput").innerHTML = outputText;

And so on, until I have a nice suite of tests and covered code, all of it running in FireFox - so off I duly go to try it in Internet Explorer... wha! That test failed. OK, so now my JavaScript knowledge has let me down and I have written code which does not run on both browsers. The JavaScript Gurus among you can probably spot the error here.

IE decides to uppercase all the html I inject so that

  • <p>boo</p>

  • <P>boo</P>

Which obviously helps my functional testing no end! And worse - if I change the <p> to <P> to make the test pass in IE, then FireFox lowercases everything so the tests fail in FireFox. Sheesh!

What to do?

OK...hack around in the DOM the old fashioned way, rather than compare the innerHTML...

test("display output text in html element",function(){
        outputText = "<p>boo</p>"
 equals(document.getElementById("eprimeoutput").getElementsByTagName("p").length,1,"contains 1 para");
        equals(document.getElementById("eprimeoutput").getElementsByTagName("p")[0].innerHTML,"boo","contains boo");

Woo hoo - a cross browser functional test - however minor.

But lesson learned - I have to use the DOM functions rather than relying on innerHTML.  (don't worry - I shall get around to using JQuery eventually - I have to learn the basics the hard way first.)

This does make some of my tests rather large though...

test("check and output eprime text",function(){

 document.getElementById("inputtext").value="So Isn't this being some bad text or rather Bob's WASn't he's being good";
 equals(outputText, "<p>" + "So " + inEPrimeOutputFormat("Isn't") + 
                                   " this " + inEPrimeOutputFormat("being") +  
                                   " some bad text or rather " + 
                                   inPossibleEPrimeOutputFormat("Bob's") + " " +
                                   inEPrimeOutputFormat("WASn't") +  " " + 
                                   inPossibleEPrimeOutputFormat("he's") + " " + 
                                   inEPrimeOutputFormat("being") + " good</p>");

        para = document.getElementById("eprimeoutput").getElementsByTagName("p");
 equals(para[0].childNodes.length,13,"contains 1 elements");
        equals(para[0].childNodes[0].nodeValue,"So ");
        equals(para[0].childNodes[2].nodeValue," this ");
        equals(para[0].childNodes[4].nodeValue," some bad text or rather ");
        equals(para[0].childNodes[6].nodeValue," ");
 equals(para[0].childNodes[8].nodeValue," ");
        equals(para[0].childNodes[10].nodeValue," ");
        equals(para[0].childNodes[12].nodeValue," good");

But that will teach me for writing non-TDD tests. You can tell I did not create this using a TDD cycle because some lines of the test did not lead to the generation of lines of application code - I wrote a "well I'll just check a few conditions in the one test just to be sure" test.

Writing functional tests like these (I wrote a few more like this) allowed me to discover other differences between IE and FireFox that caused me issues. e.g. To display some stats I created another FORM with some labels to display the information:

I do not know exactly, what possessed me to write the GUI like that, but I did. And I discovered that my code to write to the innerHTML of the TEXT worked on FireFox but not in IE, so because of my failing functional tests I rewrote this to the far simpler:
  • Word Count:
  • Discouraged Words:
  • Possible Violations:

So "Hooray" for simple functional tests.

I do believe that I could simplify much of the code I have written above by using the JQuery library, and that I could also use JQuery to help me test my event code, and probably create the GUI I want from within the code itself. So I shall probably try that next time in the ever more exciting "Adventures in JavaScript TDD with QUnit".

Note: TDD does not guarantee a bug free product. I know that I have bugs in the code I have written because I find them with additional exploratory testing. But I make sure that for each bug I find - I create a failing test so that when I fix it, I know I did enough... at least until the next bug.

Also Note: I have not figured out the correct set of Wordpress plugins to use to display source code so the &gt; and &lt; in the above code looks like > and < in my code.

Sunday, 8 June 2008

Test Driven JavaScript Code Coverage using JSCoverage

Continuing on our adventures in TDD JavaScript land and we reach for the code coverage tool. The mighty Google returns JSCoverage as the first hit for JavaScript code coverage, which means it must 'be' good, right? Let's find out how well it plays with QUnit.

You can download JSCoverage as a precompiled Linux or Windows build.
So I downloaded the pre-built Windows build.
And upon opening the zip we get a jscoverage.exe and a whole bunch of documentation.
In the grand tradition associated with the previous TDD JavaScript blog post, I won't read the documentation I just downloaded. Instead I'll have a quick scan of the online manual.
JSCoverage takes two arguments, a source directory and a destination directory. Anything in the source directory gets instrumented. Since I don't want to instrument the QUnit and JQuery libraries I ...
  • move those into a folder structure of their own,
  • and put my code in its own folder structure,
  • then I run jscoverage eprimecode c_eprimecode

Err...OK, now I have a new folder called c_eprimecode... I better read the instructions...
Oh, OK, so in c_eprimecode I now have a file called jscoverage.html, and I can put the path of test_eprime.htm and when I click GO it will load that page, thereby running all the tests against the instrumented version of eprime.js

And... it worked!

100% coverage. You can see I've added a few more tests and a few more lines of code since you last saw eprime.js and I felt pleased that my TDD efforts had led to 100% coverage.
I can click on the eprime.js link and see the code coverage metrics.

All covered lines have a green count next to the line number, and the number in the green box shows how many times that line ran during the testing. If we did not cover a line then we see a read box with the ominous 0 in it. So scanning through the source for the big bad red becomes very easy.
And that only took about 5 minutes. What can I do now? Ah, Inverted mode.
If I add a line into my test_eprime.htm like the following
<button onclick='"path/to/jscoverage.html");'>Coverage report</button>

Then I can run my test_eprime.htm like normal, but click to the coverage report.

So I do that...

But now I get mixed up. I start clicking on the coverage button when in the non-instrumented version and get a 404, so I want to fix that.

I know... I'll add some JavaScript so that the button only displays on the instrumented code page...

<button id="coveragebutton"
        onclick='"jscoverage.html")'>Coverage report</button>
if (typeof(_$jscoverage)=='undefined')      

And now, just to finish off, a little cmd script to automate it all for me...

jscoverage eprimecode c_eprimecode

So now I have QUnit Unit tests and JScoverage all running together.


I do not know how well JScoverage copes with more complicated JavaScript. As my coding skills in JavaScript improve I guess I'll find out. But certainly for the moment it does the job for me.

I use code coverage as an 'after check'. So I do my TDD and get it all as good as I can, then I just double check, but examining the code coverage:

  • did I miss anything?
  • do I care about the code that got missed?
JScoverage does that for me just now.

Other coverage tools for JavaScript exist: ProtoCov, the commercial JavaScript coverage Validator, and some sort of FireBug add on. But these are noted for future investigation as I have not used these. If you have used these please comment on them below.

SideNote: not really related to the above, but I didn't want to lose it. I saw this JSMock (mocking library) that I need to try out at some point.

Friday, 6 June 2008

Test Driven JavaScript using QUnit

I have a little project that I started writing in OpenLaszlo. I chose OpenLaszlo because I thought I could script it in standard JavaScript but during the development of my minor app I found a few errors with the OpenLaszlo JavaScript implementation and the project stalled. Now, having started a new job, where the development team use splendid new technologies like QUnit, and JQuery, I resurrected my app.

So I present, to you, some notes on my experience of Test Driven JavaScript using QUnit.

Sunday, 1 June 2008

Surprise, and now... PowerShell

Well, that last post took me by surprise - I had forgotten I had set up more slogans in advance. So to quickly try and redeem the blog and add some value... some links about testing with Microsoft PowerShell.

With Practice...

Practice your evil laugh

With Practice,

you too could

have an Evil Laugh like mine.

Why not start now? Bwa Ha Ha Ha!

Sunday, 18 May 2008

Evil Tester and The Agile Team

What happens when Evil Tester and Panicky Tester see the new Agile team in action? Will they flee? Will they join in? Find out in our exciting "Evil Tester" comic book adventure.  

Sunday, 11 May 2008

Slogans! Slogans! Seriously, we don't need no stinkin' slogans.

I always thought the slogans would prove the most controversial part of - more controversial than the cartoons, and more controversial than the name Eeeevil Tester. (You do have to say it like that, otherwise you take it far too seriously).

So after sending these slogans through to you with no explanation, I now present some reasons as to why the slogans exist.

And note...I regard this set of reasons as the 'serious' set. 

Saturday, 10 May 2008

What's all this Evil stuff? Video Version

Just in case anyone thought that was getting too serious I thought I better provide a little background to the "Why Evil?" question.

The first answer seemed flippant, but all too appropriate... "Because Evil leads to comedy gold."

Allow me to explain.

Sunday, 4 May 2008

A Short History of my ISEB Software Testing Certification involvement

Back in 2001? 2002? Back whenever I noticed the ISEB certification starting, I thought "Hmm... how strange, I wonder why they would want to do this".

I read an early draft of the syllabus online and thought "Well this seems fairly simple, but misses out a lot of stuff that I do in the real world, but what harm could it cause?"

After the unleashing of the foundation certification, and training and examining began in earnest, I remember reading a trade press advert that offered a "High paid career in testing after one week of training and gaining the ISEB certification" (or words to that effect). I still had enough idealism to feel personally affronted and feeling that 'what I did' had some how become demeaned.

In that moment I vowed to 'Save the world!', or at least the small part of it relating to Software Testing Certification.

Ultimately I failed. I relate my story here.

I Care

I Care

I'm Sure it seems as though I'm Evil.

But it's only because I care.

Sunday, 27 April 2008

Question: Which applications do you use during interviews to 'see' howcandidates do exploratory testing?

After the discussion about passion and interviewing testers I started to rethink how I conduct interviews and I think that in the future I will use MS Paint as an application to see how candidates approach testing.

A long time ago, I wrote my own little app for use during interviews. You can play with it if you like - it has many deliberately injected bugs - so no raising defect reports with me, but feel free to share any experiences that you have with it.

I mainly used this in team based exercises, and primarily to explore presupposition analysis as a method for testing.

So, now to explain the reasons why, and how, I'll use MS Paint.

Saturday, 26 April 2008

Tuesday, 22 April 2008

5 books I recommend to software testers that most testers have probablynever read

"What testing books should I read?" such a hard question to answer in a land where a testing book that has value at one point in your career ceases to have value later on.

I do have some books that I recommend to testers, entirely ignoring their context - ha... see... Eeeevil...

So... 5 books, not about testing, that could  change the way you think about testing (or my taste in books). Certainly they all changed the way that I approach testing.

Friday, 18 April 2008

Challenge your assumptions and presuppositions to identify usefulvariation

Any curious tester can find a number of published heuristic documents out there on the web (James Bach, Elisabeth Hendrickson)

'Heuristics' appear regularly on blog posts. (Mike Kelly, Ainars Galvans, Scott Barber, David Gilbert)

In this post I aim to show you an easy way of identifying new test ideas without recourse to heuristics, on a case by case basis, to allow you to add further depth to your own test explorations.

Friday, 11 April 2008

Software Testing Lessons from Brief Counselling and Therapy

Brief Therapy (and other therapeutic models) provides me with some useful 'heuristics', approaches and techniques to apply during my testing.

Brief Therapy - often called Solutions Focused Therapy concentrates on moving the client towards the 'solution' that they want to achieve through the therapy process. Different from problem focused therapy which concentrate on the problems that led to whatever situation and symptoms the client currently faces.

I have read a fair few Brief Therapy books in the past, but this particular blog posting comes after reading "Brief Counselling: Narratives and Solutions" by Judith Milner and Patrick O'byrne - not a great book but a solid and 'brief' useful overview.[][]

Tuesday, 8 April 2008

Create Software Test Ideas Fast Using Presentation PreparationTechniques

image A little history... As I did my best to teach a tester how to write test ideas for an Agile story I found myself wondering why I found coming up with ideas and questions a fairly easy activity and why they seemed not to find it quite so easy. Practice would have had something to do with it, but I also suspected a slightly different mental model.

I went out to try and find 'other' models for generating ideas and structures quickly. The first thing I found in the library had the title "Blank Page to First Draft in 15 Minutes", authored by Phillip Khan-Panni (BPTFD). Obviously not considered a classic as Amazon lists it as currently out of print, but the reviewers seemed to like it. [][]

In terms of preparing for presentations it offers as much as most presentation books - but can it help me generate test ideas?...

Sunday, 30 March 2008

RegexBuddy meets RegexGuru

image Much as I appreciate the conciseness, usefulness and elegance of Regular Expressions; I just don't use them enough to have mastered them or keep the rules in my memory. I used to rely on "Mastering Regular Expressions", which I still find very useful. But I registered RegexBuddy about a year ago when I found myself struggling with a regular expression for about an hour. 2 minutes after I had bought and downloaded RegexBuddy I managed to create the desired regex.
And now Jan Goyvaerts the author of RegexBuddy has a new blog dedicated to Regexing called Regex-Guru - this quickly found its way into my RSS reader so I recommend it to all testers as it will serve as a useful reminder that the Regex exists and of its capabilities.

Friday, 28 March 2008

We don't need no stinkin' passion!

image "Me too. Me too. Me too." That sums up my reaction to Antony Marcano's TestingReflections post on interviewing testers that just don't 'come up to scratch'. Those testers we interview who claim to have a passion for testing but don't exhibit said passion.

I've gone through the same as Antony and have no words of advice to offer him, because he probably does the stuff I mention in this post already.

Wednesday, 19 March 2008

Alternative File Browser and Command Line For Windows


This evening I downloaded and started experimenting with Selenium-Grid when I looked down at my TaskBar and I got lost. I had too many File Explorer window open, and too many command line windows open. Something had to change. Step in UltraExplorer, and Console2.

Tuesday, 18 March 2008

Some Recent Software Testing Book Reviews

I put my Software Testing Book reviews directly on my Compendium Developments site.

So if you only read Evil Tester then you missed out on Reviews of:

Monday, 17 March 2008

Book Review: Beyond Bullet Points by Cliff Atkinson

image I have done a lot of public talks over the years (but never enough to consider myself 'good' at it). Over the years I have adopted numerous approaches to constructing the presentation and of constructing the slides. You can see some of my worst slides. This slide pack seems better. But overall these slides seem far too wordy and unfocussed. I wanted to see what Beyond Bullet Points (BBP) could help me do. I was pleased with the end results, even though I only partially implemented the system. Had I spent more time learning BBP I think they could have ended up far better. I have no hesitation in recommending BBP to anyone involved in constructing presentations.


Sunday, 9 March 2008

A generalised model for User Acceptance Testing (UAT)

In a previous post I discussed how I managed to do UAT badly in the past. Now I will discuss a generalised model formed from those (and other) experiences, which should allow me to make fewer UAT mistakes in the future.

Saturday, 8 March 2008

A little abstraction when testing software with Selenium-RC and Java

image The testing tool Selenium goes from strength to strength. As evidenced at the recent Selenium Users Meeting, the future plans for Selenium look really interesting and Google seem to want to continue to invest heavily in Selenium which can only mean good things for the rest of us.

Testers have learned to use abstraction over the years. Abstracting the GUI changes away from the tests with a GUI Map, abstracting the testing language away from the scripting language through 'Action Word Testing'.

The development world came up with similar levels of abstraction when they started doing testing - hence the Fit and FitNesse frameworks.

Selenium-rc presents us with an API. Around which people tend to build abstraction layers. Many of the people using Selenium-rc use abstraction automatically because they come from a development background. So the simple use of refactoring leads to a degree of abstraction.

Friday, 7 March 2008

How to do software User Acceptance Testing (UAT) really badly

I've made a lot of mistakes over the years when testing. I try not to make each mistake more than once - which usually means any future mistakes require more creativity or stupidity on my part. So I will describe some of the mistakes I (and other people I have known) have made when doing UAT so you can use more creativity (or more stupidity - you get to choose) when approaching UAT in the future.

Wednesday, 5 March 2008

Eclipse plugins to ease Test Driven Development Mistakes

Previously I described how I made some TDD mistakes and fixed them by tracking them. And now to help me maintain my good behaviour and reduce my recidivism rate, I install some eclipse plugins: Emma, FindBugs, PMD and mousefeed.

Tuesday, 4 March 2008

A simple explanation of dependency injection

For the longest time I didn't know what dependency injection meant - anytime I looked it up I glazed over thinking it really complicated.

I only recently found out while learning TDD that I can describe dependency injection in terms of - pass an object in as a parameter instead of instantiating it in the class or method.

But as soon as I learn this, I encounter a presentation that would have taught me this anyway - so I pass it on now in case you don't know what dependency injection refers to.

The following presentation on 'testability' (from a developer's perspective, i.e. how do you make your code testable) has a good summary slide that makes dependency injection seem simple.

Saturday, 1 March 2008

Tracking TDD mistakes so I can fix them

Previously I made some TDD mistakes. So to fix them. I tracked them.

I tracked the mistakes by creating an index card that listed the 'bad' things on one side, and the 'good' things on the other side. Then noted when I did the bad things, and when I did the good things. The writing of these statements seems important. So if we have a look at the card...

Thursday, 28 February 2008

My Novice Mistakes in Test Driven Development

Since I generally present myself as a tester, coupled with my relatively novice status when coding in Java using TDD. I don't mind writing up my TDD mistakes here.

Wednesday, 20 February 2008

Some Notes on Software Testing Waste - my lean perspective

One key 'lean', or 'TPS', concept covers 'muda', or waste. Over my years in Software Testing I have tried to make my testing processes more 'Agile' or 'Lean', but when I started I didn't have the concept of 'muda', I just had the knowledge that things weren't working, so I changed them and got rid of the things that I thought added no, or little value. Now that I know a little about Agile and a little about Lean, I can try to apply the concepts of 'muda' to testing.

Monday, 11 February 2008

Notes on 'abstraction' and 'equivalence classes' in Software Testing

Abstractions can put your testing in danger if you don't handle them correctly. So some hints and tips on handling the abstraction known as an 'equivalence class' may help.

I use the word 'abstraction' (wikipedia definition) a lot when I discuss Software Testing. 'Abstraction' underpins so much of my modelling approach.
I recently read Glanville Williams' concise description of abstraction, as quoted in Hayakawa's "Symbol, Status and Personality" [][][].
"Abstraction my be defined as the imaginative selection of some one characteristic of a complex situation so that it may be attended to in isolation."
" short, the perception of similarity in spite of differences."
Quoted from the Law Quarterly Review
When I read this, I thought of 'equivalence classes', since I think of 'equivalence classes' as 'abstractions' and this definition reminds me of my approach to equivalence classing.
When I model using equivalence classes I group 'things' together based on a 'perception of similarity' in spite of differences.

Equivalence Class?

e.g. Instead of dealing with, a set of 'ages' from {0-140},
I could create a set of 'male insurance rate calculation and application age' equivalence classes of {young{0-12},teen{13-19},adult{20-35},mature{36-64},retired{65-140}}.

Warnings of danger

I particularly like Glanville Williams' definition because the carefully chosen words help warn of the dangers of abstraction.
  • ...imaginative selection...
  • ...some one characteristic...
  • ...complex situation...
  • ...attended to in isolation...
  • ...perception of similarity...
  • spite of differences...
I try to remain conscious of the differences. I don't want to fool myself that I can always, under all situations, treat as equivalent, each of the 'things' that I placed into these groupings. I need the awareness that I grouped them, for the situation which I currently "attend to", and that I isolated them to aid my understanding.
I impose a grouping. I create an abstraction. I lose information ('in spite of differences') as a result of that abstraction because I want to work with a simpler model ('complex situation'). When I forget this, I may fail to spot errors in the system.

Avoid this mistake:

I have in the past made the mistake of considering an equivalence class to 'always' retain equivalence throughout all functions of the system.
i.e. any value from the 'young' set can stand for any other value - no matter what functional area or process in the system I apply to it.
I have missed errors when I treated my abstractions as 'real' instead 'imaginative'.
When I have had the error of my abstraction pointed out to me I have then made the mistake of trying to maintain a system wide equivalence class and partitioning the classes further e.g. teen{13-16}, driving age {17-19}. Rather than creating a new 'contextualised' equivalence class for the specific situation that I require the abstraction to help me with.
I have learned from these mistakes. But I will also remain mindful of Glanville Williams' definition so that I do not repeat those mistakes again.

Hints and tips:

  • I 'name' my equivalence classes e.g. young, teen, adult etc.
  • I also 'contextualise' them to avoid forgetting the 'differences'  
    • I do this by 'naming' the set, to associate it with the particular 'complex situation' which I consider the set contents will act as equivalent representatives under. e.g. 'uk male insurance rate calculation and application age'.


I don't really want to write a huge post explaining an elegant 5 line excerpt. So I will instead close by recommending that the next time you create an equivalence class; that you re-read Glanville Williams' definition, and use it as a trigger for additional thought and reflection on your modelling process.

Related Reading

  • Equivalence classes received some attention on various blogs late last year [Thinking Tester] [I.M.Testy (1)(2)(3)(4)].
  • James Bach covers some similar ground in his blog post on model-based testing.

Sunday, 10 February 2008

If a demand for testing certification exists then why don'tUniversities meet that demand?

Should ISEB even run a certification scheme?

If Industry really does have a demand for it then perhaps ISEB's role should involve collating the Industry needs and passing them on to the Universities to provide the necessary certification mechanism. Academia generally seems to 'get' certification.

The Games industry has managed to get "Games Development" recognised as a degree level subject.

If industry wants a certification then should academia meet that need?

If we want our testers educated to 'Diploma' level then push the Universities.

Get people doing MSc's in testing.

Have our consultants and trainers do part time lecturing at Universities so that the students receive teaching from Industry professionals.

I'd much rather see on someone's CV that they took a career break to study for an MSc or MPhil in Software Testing, than an "ISEB Practitioner Certificate in Software Testing".

Friday, 8 February 2008

ISEB, and the train that won't stop going

What an easy target ISEB makes, it comes in for a lot of criticism. And I think it should. To an outsider like myself the certification train looks like a money spinning exercise, why else keep cranking out certification levels? I wonder what they could do to change my perception...

ISEB currently promotes 4 testing certification levels - the Foundation, Intermediate, Practitioner Test Manager and Practitioner Test Analyst.
I find it too easy to believe that ISEB has money making intentions rather than the 'profession's best intentions to hand. Given that the names of the writers of the syllabus all supply training towards ISEB Certification. I find it too easy to perceive a conflict of interest.
I don't want to believe that 'It's all about the money' because the BCS has registered as a charitable organisation that states "Its objects are to promote the study and practice of computing and to advance knowledge of and education in IT for the benefit of the public." For the benefit of the public, not for its own monetary gain.
So I'm going to step back a little and say some good things:
  1. Syllabuses freely available on the web, this counts towards the public good.  I won't critique the content here.
  2. The foundation syllabus has matured and represents a very basic, committee produced, introduction to 'software testing'.
Encourage any students using the syllabus for self learning to look for other sources of information on the web.
Bad points:
  1. You have to chain through the exams
  2. The other syllabus do not match the maturity of the foundation and don't really add much to augment the foundation documentation.
Why do you have to chain through the exams? Why can't you just jump ahead to the practitioner if you want to? Even if you have worked as a tester for 25 years, you still have to sit and pass the foundation, and the intermediate. Why? To protect the candidate from their own risky predisposition? Let them take the risk if they choose to.
A tester, a person who deals with risk on a daily basis.
A cynic might think that the chaining related to making money or selling further training.
Suggested Fixes:
  1. Have Industry provide a syllabus which contains the things that 'it' wants the testers to know.
  2. Have ISEB ratify the syllabus and organise the creation and sitting of the exams, and the marking facilities.
  3. Remove the conflict of interest with the trainers.
    1. Do not let the trainers write the questions.
    2. Do not let the trainers define the syllabus.
    3. Do not let the trainers mark the exams.
    4. Do not let the trainers sit on the 'board'.
    5. Do not do anything that looks like a conflict of interest and makes this look like a money spinning exercise.
  4. Allow anyone who wants to, at any point in the chain take the exam. Have no dependency on a previous exam. Just charge them a fair price to cover the cost of sitting and marking the exam.
  5. Make trainer costs and pass rate/fail rates visible on the main web site to allow candidates to choose
  6. Make pass/fail rates for self study candidates visible on the main web site
  7. Provide a simple, visible, online mechanism for taking comments publicly on the syllabus and publishing a summary of disagreements. (this would help prevent duplicate comments coming in)
  8. Make any multiple choice test completely automated and as nominally priced as possible (try and make it free).
Note on the above:
On making multiple choice exams nominally priced (or free). Any fee should cover the costs of writing questions, hosting the site and database, and all related admin. Think of it like a charitable act to the testing 'profession'. This would provide a public service to the testing community. I suspect that the foundation exam has probably paid for itself by now - if so make it completely free.
Trainers can still train.
It seems entirely possible to me that when you remove conflict of interest you may lose all the people required to run the programme. This may indicate a lack of business value and business demand for the certification.
So please ISEB, and everyone involved, just stop now, add no more certifications and remove all the constraints. It really does look like you keep feeding coal into the money train.
"Old Charlie stole the handle and
The train, it won't stop going --
No way to slow down. "
Locomotive Breath, Jethro Tull
Ironically, this post will end up surrounded by Adsense adverts for ISEB certification training. Oh well, I only do this for the money anyway.

Wednesday, 6 February 2008

My notes on how to study for ISEB foundation

While trawling through my hard drive on a deletion frenzy, I found my old notes on what I did to pass the ISEB foundation exam. Bear in mind that I made these notes in 2003 and the ISEB syllabus has moved on since. But I have updated all the links and someone might find my updated notes useful.

Original Notes - with up to date links.

Hints and tips

  • The questions get written from the syllabus - know the syllabus
  • The definitions come from the glossary - forget what you think you know if you want to pass - learn the definitions in the glossary
  • The techniques come from the BS7925, read and understand this
  • At the end of your study, if you feel like doing something adventurous then read the practitioner syllabus for extra depth


  • Study the syllabus in conjunction with the Glossary of terms used in software testing and the BS7925 standard
  • Build a mind map or set of study notes for the syllabus to summarise what you think important
  • Check your understanding and notes against the definitions in the glossary and the standards
  • Learn some memory techniques and general exam techniques. The  multiple choice foundation exam tests your memory skills rather than your testing skills.

Optional with Cost

  • From the Recommended Texts: Hetzel reads at about the right level; Beizer, although admirably included in the list should count as a bit more advanced than the foundation requires (but do read it for your own education and edification); I use Roper as a good summary of test techniques at about the foundation level
  • Get the sample paper from ISEB and try it out

Additional References

Up to Date Notes

  • The BCS web page recommends that you take a course to learn this stuff - frankly I think this massive overkill for a 40 question multiple choice test. Apparently "the overall public exam pass rates are notably lower than for candidates who have attended an accredited training course", but I could not find those stats on the ISEB site. Given the cost of a training course, you could probably sit the foundation exam 6 or 7 times on your own and still not have paid the same as a course would have cost.
  • The massive BCS Reading List seems designed to intimidate you into taking a course or grabbing the "An ISEB Foundation Course"
  • Where did this Intermediate course come from! Cue a post for a later date.
  • Sunday, 3 February 2008

    How To Learn To Program

    • Make it free to Learn – open source implementations of languages.
    • Don't use a traditional programming book – too slow.
    • Learn it to do something – be focused.
    • Ask for help.
    More helpful hints and links follow...

    How To Learn

    • Make it free to Learn
      • Find some open source implementations of the language you want to learn.
      • Check out sourceforgefreecountry
    • Don't use a traditional programming book – too slow.
    • Learn it to do something
      • Find a small project to do in it
      • Find a way to generate Test Data in it
      • Parse a log file in it
      • Work on an actual system
    • Ask for help
      • Developers will help if you are sincerely interested. I've never met a developer that wasn't happy to help me understand a language in more detail if I have made an effort to learn.
    There is so much information out there on the web, all you have to do is look for it.
    Go look. Now.


    Friday, 1 February 2008

    ISEB testing certification, fear, and the hiring dilemma

    At EuroSTAR 2007 Michael Bolton started the conference off with "Why I am not Certified" - Michael had obviously kept more up to date on the ISEB progress than I. I looked at the ISEB site recently, and now I feel uneasy.

    I know that the people who started all this certification had 'industry' and possibly 'profession' best intentions in mind I assume that the people involved still do have those intentions.
    The ISEB web site quotes one of the intentions as:
    To enable software suppliers to hire certified testers and thereby gain commercial advantage over their competitors by advertising their tester recruitment policy
    At EuroSTAR I held the belief that ISEB only had "foundation" and "practitioner". But I just looked and ISEB now has "foundation", "intermediate", "practitioner test manager" and "practitioner test analyst"
    When I act fulfil my obligations as a hiring test manager. I don't think "you know, if only ISEB had more levels of certification then I could more easily distinguish between good candidates and bad candidates."
    I don't need any more levels of certification created to meet my hiring needs. ISEB certification does not currently help my hiring needs.
    Now, people obviously do get trained and certified through ISEB, and I obviously don't know their reason for doing so. This situation suggests, to me, that some sort of demand exists for the certification.
    I have my suspicions that Fear produces this demand.
    Fear from the testers that if they don't get the certification, then their application will not go forward to acceptance for jobs. That seems like a valid fear, because it happened with the foundation certificate in the UK testing market.
    As a contractor, I encountered companies who wanted ISEB certification on my CV. I encountered agents working for companies who, on the instructions from the company, filtered out CVs on the basis of "Certified? Y/N".
    I think that if job adverts start to ask for - "must have ISEB Practitioner Test Analyst" then our industry and our 'profession' will have managed to get itself in a terrible mess.
    I don't just moan though. I now present an alternative for the hiring dilemma...
    How about ISEB throw away the other certifications and create a certification for hiring managers? The "ISEB Foundation Certificate in Recruiting Testers".
    Rather than building up an industry in certification that will eventually find its way into the hiring practices of big companies that don't know any better. Train the people doing the recruiting to recognise good testers. That sounds, to me, like a good move for the industry and in line with the BCS ISEB's charitable status to act in the public good.
    But since that certificate doesn't exist - why not read...

    Wednesday, 30 January 2008

    I thought you were a tester but... you're programming

    I state no "MUST"s in here, you don't "HAVE" to know how to program to work effectively as a tester but...when you do know how to program, then you have more options. How many more? Well...

    "Reasons to be cheerful - one, two, three..."

    Reasons why I have found it useful to have programming knowledge:
    • Speak to the developers in their own language – build relationships with the development team
    • Different languages have their own problems and idioms – useful to know these
    • Different languages have to make different physical decisions when implementing logical designs
    • Implement custom test tool utilities – metric collating, defect management, data generation
    • Use existing test tool APIs and customise your test tools e.g. mercury open test architecture, TPlan API
    • Automate your test process
    • Identify new tools that you can use to help you
    • Build the application from the version control system directly
    • Learn the database schema without extensive documentation
    • Reuse development code to construct test tools
    • Get free tools working – Java & Perl
    • Regular Expressions - use and read
    • Understand the design process and artifacts (OO, UML)
    • You know what the developers mean when they say: Refactoring, polymorphism, mock objects, ...
    • Compile in Debug Mode to make applications testable
    Early in my career, I busied myself learning how to test and I worked on some projects where I didn't know the programming language in use (IBM System 360 ASM anyone?). That meant that I had fewer options in how I could approach my testing and my dealings with the development team, and sometimes I could not even understand the problems they discussed. I decided never to stay in that situation for the duration of the project again.
    "Do Testing Early" – "More Unit Testing" – Can you really promote automated unit testing if YOU don't know how to do it? What tools can you use? Why should the developers listen to your advice on coding if you can't code?
    How easily can you make the system testable? Could you help make it testable, or do the development team need to do all the work?
    Can you easily add logging in the current development language without interfering with the code? Do you know?
    I cannot claim "greatness" as a programmer, but I do learn the language for the projects that I work on - even just a little. I use the fact that I want to learn the language to strengthen my relationship with the development team by asking them for help with my learning process.
    Now. I find it interesting that I wrote the above text 4 years ago. And in that time I've worked on Agile projects, paired with developers, learned TDD, created mock objects to aid in Unit Testing. I did a lot of this through book learning and experimentation. Fortunately I have worked with developers kind enough to explain things to me, recommend books, and point out what web-sites and blogs they read.
    I appreciate that a lot of people don't want to do this. But I feel the need to reiterate - when you do, you increase the options available to you and increase your own personal flexibility.
    I highly recommend every tester learns how to program in the language used to develop the system under test.
    At least a little bit. Just a little bit? A teeny bit, just a teeny weeny bit?
    Yes, yes, dear, dear, perhaps next year
    Or maybe even never
    In which case .....
    Reasons to be Cheerful (Part 3)

    by Dury, Payne and Jankel

    Monday, 28 January 2008

    Panicky Tester meets Evil Tester

    What happens when Panicky testersmallPanickyTester meets Evil Tester smallEvilTester ? Find out in the latest episode of the exciting cartoon adventures of Evil Tester.

    Thursday, 24 January 2008

    How to find out where the time goes

    jWorkSheet is one of the secret tools that form part of my productivity armoury.
    What does it do? Why do I use it?....

    Well, I work across multiple projects and if I like to be able to see which projects are taking most of my time. If I don't track my time I get to the end of the day and go "so what did I do all day?".
    If you face a similar issue then...jWorkSheet might help you.
    jWorkSheet has gives me the ability to report on where I have been spending my time. When I need to do that formally, or informally.
    I needed a (Project -> Task -> event) model where I could track work-done-time spent on Tasks across disparate days or weeks I don't often get to work on a single task for a number of days (or hours) straight.
    I was originally using a spreadsheet that I wrote, because I would periodically browse the web but never found anything that worked the way that I did.
    But jWorkSheet uses this exact model!
    You setup the projects and tasks on one tab, then in another tab you track your day.
    I use the default reports and haven't tried customising them. The reports are all written in XSL so it should be a simple matter to write my own when I need to.
    Now for a lot of people the notion of tracking their time on an event by event basis is 'tooooo much work'. But I find it a useful practice to log what I do.
    I carry this practice through to my testing, where I log the decisions I make, the thoughts that I have, the things that I do. When I do this incrementally across the day, as it happens, it really isn't too much work, it is simply a natural part of each task. When I switch task - I spend 5 seconds making a note of it.
    The end result of all these 5 seconds? A feeling of control and the ability to see where my time goes so no more uncertainty. And with that comes the ability to use this information to plan more effectively, to make decisions about priority and optimisation more effectively.
    If you don't track your time during the day then I'd suggest as an experiment to you that you try it. And if you already do it, then see if jWorkSheet works for you.
    I subscribe to the freshmeat entry to receive update notifications.
    And so if you every find you're solidly working but end up unhalfbricking then you have no need to ask yourself "who knows where the time goes?", instead you can relax with this...
    Youtube Video

    Sunday, 20 January 2008

    Please stop the fiction

    I have an unhealthy dislike of the current trend towards "fictional" example 'based' essays.
    You know the type...
    It was Monday and I struggled into the office out of the pouring rain and before I'd even sat down, let alone think about getting a cup of coffee and reading the latest Dilbert gag, Jane walked up to me and said "We have been looking for you for half an hour, there has been a new release, we only have 20 minutes to test all the software and we have 27000 tests to run manually - what are we going to do?"
    Then the author goes on to extricate the characters from the situation gracefully and easily so that we can learn a lesson.

    I've done it myself, I'm not proud of it, probably no-one read the little bit of fictional dialogue that I put in a conference paper, but I've done it. And I think it was wrong. I'm even kind of embarrassed by the few short pieces of 'real' fiction that have been published in the 'real' world. Fortunately few people read those.
    I don't think I'd mind so much if the writer was a published fiction writer, but even then I'd have doubts. I'm currently working my way through a hefty chunk of David Morrell's back catalogue. I find much to admire in David Morrell's fiction - I have some of his books set aside as 'exemplar' action novels for study when I start writing fiction again, but I have a 50-50 hit rate with David Morrell - I like about half of the books of his I read, and half I don't and they go to the Charity Shop.
    I'm fairly picky when it comes to fiction. If I think it is bad, I stop reading, I don't like to waste my time with junk fiction. I don't read computer mags, and books, to read fiction. So how many articles in computer literature do I have to skip now? It feels like a lot.
    So aside from the 'bad' fiction what other 'reasons' can I harangue you with?
    There was one section of the Harvard Business Review that I consistently did not like. The fictional 'based on a true story' case study, and I think they use a professional writer to produce it.
    I feel like a single point is being strung out for too many pages. If you are paid by the word then pages mean money. Hey, I have written fiction - dialogue is an easy page filler and that means money spinner. If I'm trying to pad to push the page count - I default to dialogue, I can up the page count in minutes with a good stream of verbage.
    It is too easy for the protagonists - I probably read too much action fiction but there are never any twists in these articles. It always reads too much like a romance novel - where is the action? the drama? the pulp? I never get to read text like this...
    Jane hadn't noticed the tired bags under my eyes - I didn't get much sleep last night on account of the neighbours next door shooting stray cats with their M-16's. And they were lousy shots. The clatter of shells hitting a trash can sounds pretty loud at 3am - I wasn't feeling in a mood to panic. But I was grouchy. I guess I might have had a tendency to overreact. "Which environment is it loaded on?"
    "Forget it, it doesn't matter". I left her staring at the back of my head as I walked into the machine room. Let me see. P37... P37... ah heck. I started pulling the servers off the wall one by one. I'd bring down the test environment sooner or later. I wasn't thinking straight. I hadn't had my morning coffee yet. I'll call this a denial of service attack and write it up as part of the test completion report - at least this would buy me some time so I could get my coffee and read Dilbert.
    I stopped short when I pulled the last server to the ground. I slammed my hand to my sweaty forehead suddenly realising what I had done.
    No Server = No Internet. I could kiss the latest Dilbert gag goodbye - at least until the server boys managed to bring the environments back up.
    I walked off to get some coffee shaking my head and trying to catch my breath, thinking about how stupid I could be at times.
    See - fiction - bad Idea. I promise I'll stop. But to me the 'fiction' trend in the mags (and books) looks like continuing. And the worst of it?... based on the popularity of the style I think it is just me that doesn't like it.
    So some pleas, rather than a lesson...If you have to write a fiction kind of thing then:
    • could you try different genres? Horror, or Action?
    • Or try and write a comic or a cartoon? (I like comics)
    • Or use a metaphor instead
    But please no fiction.
    No servers were harmed in the making of this blog post. All names are fictional and are not intended to represent real people. This blog post was for entertainment purposes only, do not copy the actions in this blog post, they were not done in a real environment.

    Friday, 18 January 2008

    "Martial deals with surprise" - GTD related to testing

    I made some notes when watching the Dave Allen's Google Tech Talk on Getting Things Done. In order to help remember them I have mapped them on to Software Testing, with a few additional notes.

    If you haven't watched the Google Tech Talk by Dave Allen then this post may not make as much sense now, but I think it will after you come back from watching the talk. And if you have watched it, then this post may act as a small reminder for you to.
    Dave Allen said a number of statements during the talk that I made a note of. By taking tiny snippet quotes, I get the benefit of applying them to whatever I want and take them out of context. So warning... I did that below.
    Martial deals with surprise.
    So does testing. Surprise heavily influences my testing. And I currently hold the belief that the more 'surprised' we get when testing, the more information we generated with our test. A 'scripted' test which passes generated no surprises, and no real information since it 'confirmed' our model. If a test 'fails' then it expands our model since we didn't expect it to fail, we have to adjust our model to the reality of the situation and thereby generated new information that we have to report and incorporate into our model.
    Dave Allen had a few quotes which I relate to 'state' or how we feel. I view personal 'state' as an underrated concept in Software Testing and Dave Allen actually describes a very effective state for conducting exploratory testing in.
    You lose speed if you are not "present"
    Perfectly appropriate response with and engagement with what is present
    Available at present with full resources.
    If you could generate a state that you can describe using the words above, what does it feel like. Now you may find be that you have to remember a state very close in form to that description and tweak it a little, or that you can instantly adopt that state. And when you test like that, and you extend your senses, and you observe keenly and you focus your on the system and the test at hand. Do you think you can act more effectively?
    So one application for GTD would involve using it, to avoid the distractions that remove us from that state. The GTD system aims to help you create a trusted system so that we do not get distracted and we can stay on focus.
    The exploratory testing literature talks about session based testing. One tool of session based testing, the 'Charter', can act as one of the techniques to remove the 'clutter' from your mind before you start testing.
    Dave's "working hypothesis" - as soon as it in your brain and managed by your brain - it keeps getting popped back - until you put it in some trusted system that you trust you will look at at the important time in the future.
    So as an exploratory tester my 'trusted system' involves a charter and a notebook for making notes as I go along. I make notes of observations. I make notes of new ideas - I don't want to forget them, but I don't want them detracting from my focus, so I write them down, knowing that I review them after the completed testing session.
    Pay attention to where your mind is going - there is a message there - handle that otherwise you get stress
    I consider this good advice and I apply it not just to me, but also to the system under test.

    Wednesday, 16 January 2008

    Get yourself a free computer science education

    On Lambda the Ultimate I found a link to the ACM Classic Books Series. There are some great books here that I haven't read since University. So a few of my choice favourites follow...

    All the books I'm going to list below are available on the web page once you register with the ACM (and you can do this for free)

    I'm going to reread E.F Codd's "The Relational Model for Database Management" as I remember getting a little lost in there the last time I read it.

    I have read Seymour Papert's "Mindstorms" several times and my paperback copy is yellowing slightly so a pdf version will do me nicely. I'm looking forward to playing Logo with my Son as he gets older but I suspect that Squeak might be taking the place of Logo in the discussion of Children and Programming now.

    But what is really exciting me is the inclusion of Aho and Ullman's "The theory of parsing, translation, and compiling". The contents for this looks fantastic although I suspect Volume 1 parsing is going to be most interesting. I don't think I've read this one.

    The Dragon book got me through my final year university project butI still haven't properly digested Ullman's "Introduction to Automata Theory, Languages and Computation".

    ACM are trying to get pdf versions of the other books in the list, and from that list I am particularly interested in reading "Anatomy of Lisp" and Von Neumann's "The Computer and the brain" if they become available.
    So, next action: check out the list.

    Tuesday, 15 January 2008

    Some tools just get better and better

    I have been a MindManager user for a long time, but I'm not a power user. I don't script it. I don't use any plug-ins. I don't generate many documents from it. But it still seemed like one of the best mind map tools out there: fast, easy to use. Every so often it would hog the processor for no good reason. But then FreeMind suddenly got really really good. Is FreeMind now the best mind mapping tool as well as being completely free?

    For my purposes - FreeMind is my tool of choice.
    • I've been doing more and more of my writing in OpenOffice and FreeMind exports directly to that.
    • Out of the box pdf export with no additional drivers
    • A good organic layout is created automatically
    • I can script it if I want to with Groovy
    • It can support attributes on the nodes now
    • The file format is easy peasy xml for easy parsing and manipulation
    • It is fast and when not in use doesn't hog my processor
    • It can open MindManager maps directly so I have no messing about moving tool
    • Very easy to use
    • Because it is cross platform I can share my mind maps with my 'strange' Mac using colleagues
    It does support planning and calendars and schedules and all that nonsense, but I don't use a mind map tool for that.
    The fact that it is free and open source is a bonus and means that anyone can benefit from this tool with no monetary investment. I thoroughly recommend trying this tool out.

    Friday, 11 January 2008

    Music to Test By

    I just discovered my new favorite internet music site.

    I have only just dusted off my old vinyl albums from 20 years of storage, but haven't gone out and bought a turntable to re-listen to them yet so... is allowing me to re-experience the delights of Kaleidoscope (the UK band), Procol Harum, Dead Can Dance, Jade Warrior, and find some new (OK...old) music to enjoy.

    The site also has links to some online radio stations that specialise in Prog Rock - what could be better to test by than prog rock? The 10 minute French masterpiece "Depuis L'Automne" by Harmonium?
    I've just heard Gryphon for the first time. Ah, splendid, they don't make music like that anymore.