Sunday, 4 May 2008

A Short History of my ISEB Software Testing Certification involvement

Back in 2001? 2002? Back whenever I noticed the ISEB certification starting, I thought "Hmm... how strange, I wonder why they would want to do this".

I read an early draft of the syllabus online and thought "Well this seems fairly simple, but misses out a lot of stuff that I do in the real world, but what harm could it cause?"

After the unleashing of the foundation certification, and training and examining began in earnest, I remember reading a trade press advert that offered a "High paid career in testing after one week of training and gaining the ISEB certification" (or words to that effect). I still had enough idealism to feel personally affronted and feeling that 'what I did' had some how become demeaned.

In that moment I vowed to 'Save the world!', or at least the small part of it relating to Software Testing Certification.

Ultimately I failed. I relate my story here.

...But lords and knights and other noble and worthy men that con Latin but little, and have been beyond the sea, know and understand, if I say truth or no, and if I err in devising, for forgetting or else, that they may redress it and amend it. For things passed out of long time from a man's mind or from his sight, turn soon into forgetting; because that mind of man ne may not be comprehended ne withholden, for the frailty of mankind.

The Travels of Sir John Mandeville


I had vowed (and failed) to 'Save the world!' on a number of occasions before:

  • when I built my first CASE tool (a JSP diagrammer with a built in COBOL interpreter),
  • I built an ER diagram of 'Testing' (to understand and finally get a handle on Software Testing),
  • when I built a configurable entity management system (to model any testing process).
  • when I generated test scripts from graph models

Introspection: Perhaps, this idealism and failed attempts at Super-heroism led to the development of the lampooning Evil Tester comic character?

At any rate - I resolved to engage in the certification process and change it for the better.


At one of the BCS Sigist meetings (Special Interest Group In Software Testing), a call went out for people to help with the examination process (writing questions, marking, etc.). My chance to infiltrate had appeared, so I applied...

...Only to receive a rejection on the basis of not having passed the foundation certificate.

You see I had let the certification process proceed apace such that ISEB had added an additional layer of certification, called the 'Practitioner'. And now to receive consideration of competence to allow me to mark and write questions for said 'Practitioner' exam, I had to have made the grade of 'foundation'.

So I duly went out to pass the ISEB Foundation. Eschewing any form of training for such a simple and easy test. I downloaded the syllabus and began preparing for a public sitting of the exam. (see my notes on how to prepare)

Sadly this was not my only motivation. The ISEB Certification had already taken hold on the UK Job market and I had already had my CV turned away from two job applications because I did not hold the certification. So driven by the 'fear' of never working again, I sullied myself, but justified my actions on the basis of 'Save the world!'.

Making the grade

I found sitting the exam a strange experience. The courses that people sit through in preparation for the exam must train them in such a black and white manner, that some of the ambiguous and vague questions seem obvious to them.

But for me, trained in the real world, having had my black and white orientation knocked out of me, I wanted to answer with a "well, it depends, do you do X or ...?". And that led me to choosing the wrong answer on at least one occasion because, and I confess this with a shamefully lowered head, I did not achieve 100% on the 'foundation'.

But still... I made it!

I passed!

I achieved certification!

I can describe myself as a tester!

I can now wave my bit of paper and say... "I, am a tester!" (I violated the General Semantics 'is' of identify for dramatic effect)

"Nah nah, ne nah nah! I got a bit of paper, you don't. I got a bit of paper, you don't." Cough, back to the seriousness.

A second Infiltration

Armed with my certification, I wrote a second letter requesting admittance to the hallowed realms of ISEB examination... and received an acceptance in return.

The ISEB certification community contains many 'big hitters' from the UK Software Testing world so it acted as a surprisingly useful networking opportunity. Although this post may result in the 'throwing away' of all those hard won networking moments, who knows?

I made a point of attending the exam panel meetings as much as possible. While I could have involved myself more remotely, I thought that I would stand a better chance of 'changing' things if I actually attended the meetings in person.

I had never written or marked any exam papers before so I expected to receive a little tutelage to prepare me for the coming travails. But as with many things, I learned by doing, and I learned by making mistakes.

On duty my vision dissipates

My duties included at least elements:

  • I wrote exam questions - foundation and practitioner
  • I marked practitioner exam papers
  • I reviewed questions

All the judgments that we took as an exam panel used the mighty syllabus, as a constraining tool (to not go beyond it), and a driving tool (to cover it). That sanctified document which, if 'twere not in, then we could not ask thereon.

Changing the syllabus did not happen lightly and I do not remember a syllabus change driven by the exam panel. All changes occurred when the 'higher' panel decided. For two panels did exist - one which did the marking and examining, and one which engaged in more lofty pursuits; deciding what teaching should occur and who should teach it.

Very quickly my idealised vision of change dissipated. And I got down to the business of writing questions and marking exam papers.

On the art of writing exam questions

I did not master the art of writing multiple choice questions. They either read too easily, making the answer obvious to even the most unschooled and inexperienced candidate. Or required such a long description of the context underpinning the scenario I wanted the candidate to unambiguously address with a multiple choice answer.

I quickly gave up writing multiple choice questions.

I did not consider my attempts at writing the longer, essay based, 'Practitioner' questions a greater success. These questions took a long time to write, for a number of reasons:

  • Trawling the syllabus to find areas of sufficient complexity to challenge the candidates thinking process
  • Constraining the question to the limits of the syllabus
  • Writing the question, then writing an example answer, and iterating around that process took a long time

And then the question became subject to a review from the exam panel. Some questions never made it into the exam and lay strewn on the exam panel floor - more likely the foundation questions as a huge pool of these already existed.

The shortage of questions resided solely in the Practitioner exam pool. So every question brought forward got bashed into shaped, tweaked and tucked by the panel - unless withdrawn by the writer.

I think I remember the butchering of one of my questions where the judgement passed on it meant that we combined it with a subquestion on the Belbin personality types - which personally, I abhor its inclusion in the syllabus. The inclusion of Belbin represented a far worse sin than the exclusion of 'exploratory testing' or 'model based testing'.

While as, an exam panel we did our best to understand the effect the question would have on the candidates, you only saw the results when the candidates tried to answer it.

On the decipherment of chicken scrawl

Candidates please... use your best writing. But sadly time pressure, and all that, typically means that the best analogy for reading of an exam answer paper became the extraction of meaning from a paper which chickens had run over with ink covered feet.

Marking these papers took a long time. Typically two hours (sometimes more) per paper. Then we met with our exam marking partner - for each paper had two markers. To examine variance between our grading. If the variance in question seemed high then we conferred to see what one marker had seen that the other had not. For example, had one marker managed to work out why the candidate kept writing the word 'chicken' in their answer, what did they really mean?

We tried to mark fairly and consistently, using the marking guidelines, and with our knowledge of the syllabus. This did result in disappointments where people who you felt should not have passed, did manage to pass.

At the end of the whole process we collated the marks together to see if we needed to apply weightings to the final marks. For example if 'everyone' failed a question and we deemed that question at fault then we might apply weightings to 'be fair' to the candidates.

Until I learn to accept my reward

It seemed that the only 'change' I could effect involved:

  • commenting on questions that did not 'stress' the candidate well - although always constrained by the syllabus
  • writing questions to test the candidate effectively - but again constrained by the syllabus

At one point, after a long marking session I examined my vision statement.

  • Had I made any inroads into my goal of "Save the World!" - answer no.
  • Had I at least received compensation for my actions - answer no. We could not consider the monetary remuneration for all this work as 'worth it'
  • Instead, I helped to perpetuate a system I did not believe in, nor did I see it changing for the better.
  • Instead the trainers and ISEB received the monetary rewards while I had managed to voluntarily enslave myself for their benefit, to the detriment of my own time.

I made the decision to leave - I forget exactly when, I think I engaged in this process for about two years.

A Few Lessons Learned

Simple lessons. Probably already learned. But obviously easy for me to forget.

  • Do not perpetuate systems that you disagree with
    • I don't know how to stop this certification industry, but I do know that I can starve it of my involvement and my support.
  • Writing exam questions requires experience and training
    • Part of an effective certification process must involve effective questioning and evaluation. I have my doubts as to the degree that the process, as it existed when I helped with it, met that requirement. Given that we received no training in, or had no  experience of, writing or marking questions.
  • The certifications do not represent an assessment of ability
    • I read many exam papers. Very few of these 'Practitioners', exhibited in written form, a degree of knowledge sufficient that I would want to employ them.
  • Having this stuff on your CV really does help
    • Having 'past member of ISEB exam panel' on your CV helps. If you support the certification process then I encourage you to join the exam panel. That way you help to perpetuate a system that you support and you do your CV some favours.
    • My hidden agenda in the encouragement: I hope that you experience a growing unease and distaste for the process the longer that you involve yourself in it.
  • Keep your vision in mind and track progress regularly
    • In the heat of battle, you can forget the ultimate strategy all too easily so keep this in mind and track your progress towards it.
  • Always charge your full complement of expenses
    • Since I didn't join the panel for the money, I sometimes didn't claim all the expenses or follow up demanding payment of all the money after writing a question. (At least this allows me to not feel too bad about taking their money - I didn't take enough of it)


And thus, taking solace in my wretched rest, recording the time passed, I have fulfilled these things, and put them written in this book, as it would come into my mind...

The Travels of Sir John Mandeville

So, my involvement summarised, to the best I can currently remember.

If you have any reflections on your involvement then I would love to see you write about them:

  • blog them, and ping this post,
  • or leave a comment pointing at your reminiscence,
  • or write them in as a comment here.


  1. Excellent experience report.

    My involvement: I strongly criticized the current crop of certifications (not just the ISEB/ISTQB travesties, but others like them) in a keynote speech at EuroSTAR 2007. (You can see the slides here: ) The speech generated a good deal of controversy. I believe that I earned the respect of many people who had not heard much about me until then. Some of these were, are, supporters of certification and many more were not. I suspect that I offended and irritated a bunch of people whose respect wouldn't mean much to me. In this sense, these certification programs have been good for me.

    At EuroSTAR, I also earned the ISEB's first anti-certification. It says

    This is to certify that

    Michael Bolton

    having not taken any ISEB qualification, should not be disqualified from consideration or acceptance for a job in software testing

    It's signed by Simon Adams, ISEB, December 2007. It even has a drawing of the ISEB logo on it. I am proud of my anti-certification, and hope others apply for it.

    ---Michael B.

    Hi Michael,

    Thanks for the comment.

    Out of interest, how would one apply for an anti-certification certificate?

  2. Thanks for an interesting and informative read.

    You employ an interesting writing style that I find enjoyable - you must read a lot. Have you ever tried writing other than this blog?

    Hi Mark,

    All flattering comments automatically get approved - all other commentators please note.

    I do read a lot and I have tried writing other than this blog.

    I did have some fiction published in a short run mag. And I wrote the character descriptions in the computer game XS, these were subsequently turned into well acted vocal snippets for the in-game encyclopedia and which the critics duly, and universally, panned.

    Since I was paid a pittance for my contribution, and had to purchase my own copy of the game rather than receive a free one at the launch party, a launch party to which I did not even receive an invitation... I don't begrudge it that fate. I actually had to purchase my copy in the bargain bin of a computer game shop - oh the shame of it.

  3. Dear evil tester

    The truth is that the purpose of the ISTQB certification, and many other certifications is to make a lot of money. The truth is that people love to have pieces of paper that say they have achieved something. One of the most famous ISTQB teachers Dorothy Graham made a very interesting comment at EuroSTAR 2007 . She said: The foundation certificate is only there to scrape the bottom layer of ignorance away from the testers" Pretty hard words I must say. I believe she is correct. The foundation certificate exists to teach people the definitions of words and processes that a limited number of people have agreed are important to know. Any claim that anyone becomes a good tester by taking the exam is untrue. I teach a class called Software Test Design which is built on classical test techniques as well as my own experiences. No exam, no diplomas but a lot of discussions and exercises that hopefully will help students to solve real world problems. But it is not that easy to sell! Why, because whoever pays for the test classes believs that the ISTQB certificate is important to have but actually learning something useful is not!? So right now, in Sweden, the truth is that a lot of companies hiring testers require the certificate from candidates applying, consultancy companies use it as a selling argument. So while Michael Bolton actually wants to teach his students something useful, this is not the case with the certificate. However, ALL really good testers I have spoken too are totally aware of this. To really know if a tester is any good requires some knowledge and some additional work and that is obviuosly too much to ask for most...

    OK, I could go on forever but suggest that anyone that is interested to read Michaels slides that he linked to.

    "Dear Evil Tester" - I love that, but...mental note... I must make sure to personalise this blog so that people associate Evil Tester with the cartoon and not with me.

    I second the point about reading the slides. I know the slides don't convey the full spirit and drama as when Michael presented them, because I watched his talked at Eurostar. But hey ho if you missed it... you missed it. Read the slides.

    Thanks Tobbe,


  4. Compelling, organized, honest, and well-written. I confess I once had a similar notion to see what it would be like to take the exam and contribute from the inside so I, too, could have a shot at curing the world of bad certifications. I was on the fence until reading your report. I now feel cured of that desire, and consider you a tester who just ran a Build Verification Test on the ISTQB certification process and marked them with a FAIL, signalling to the rest of us reasonable, practical, heroic-minded folk that for us to try what you tried would be a colossal and disheartening waste of our time. Your blog entry here may be a force that changes the world in the way you truly intended, so thanks for your honorable service!

    Thanks for the flattering comment Jon,

    I think that if other people tried to do it the way I tried, at the level I participated, then they would fail in the same way.

  5. Alan,

    Point taken about certification ...

    How do we (people who view certification in its current form as good for the craft of testing) address folllowing social issues that arise out of our stand on certification (on the lines ..."Well, let us agree for a moment that todays certifications are bad. What is the alternative, what solutions you suggest")

    1. New testers who are intimidated by certifications by recruiters, head hunters. They simply say "If I am not certified .. I will loose few good job offers".

    2. Managers of IT services companies (especially in India) say "our customers want certfied testers. I must provide them those people. Having x number of certified testers is a big boost for me against the compitition"

    So it is new testers, customers who outsource(US and Europe) and IT test managers who are fueling the need for certified testers ... if there is money to be made in this demand - people will make it.

    How will the world of certification look from their angle?


    Hi Shrini,

    Good Comment.

    1) For the new tester. I say that they will probably lose some job offers. And they may lose 'well paid' job offers. But they will probably not lose any 'good' job offers, since those offers would come from companies that do not know how to recruit testers. I don't think the industry tackles this problem by communicating to the tester, we do it by communicating to the hiring manager. So one solution here probably requires providing more information on how to effectively recruit testers and getting that information in the hands of people who currently rely on 'lack of certification' to weed out 'bad' testers.

    2) In the short term this strategy makes sense for the offshore supplier. Having their testers certified allows them to wave that as a selling point to customers, in the same way that they used CMM Level 5. But eventually everyone has certified testers, and everyone has CMM Level 5. They very quickly lose their edge and become homogenized. A similar solution to above requires getting information out, to the people 'buying' the offshoring, on how to effectively choose an offshore supplier. The offshore suppliers themselves could help by 'educating' the purchaser on reasons why 'certified testers' do not mean 'good testers' and building a better sales pitch or value proposition.

    I think that neither of these solutions will take, as 'certification' provides an 'easy' option.

  6. Alan, thanks for that report. I applaud you for getting involved and trying to improve things. I'm curious to hear more about what kind of effort you think would be required to change the syllabus, and whether incremental changes to the syllabus would be worthwhile. Could it be that you'd have to be pretty happy with the syllabus as it is, and the certification process in general, in order to be willing to put in the effort to make any changes?

    I'm lucky in the US that I can simply ignore the certifications and not tangibly hurt my career.


    Hi Danny, you have posed a really challenging question.

    So challenging in fact, that I don't have time to answer it at the moment.

    I add this meagre excuse for an answer as a temporary measure until I can formulate a sensible reply, and so that other people can ponder your question as well.

    ... a few days later...

    Unfortunately, I do not think that changes to the syllabus alone would address the problems I have with the certification process.

    I do think the syllabus could change very quickly. If the people who set the syllabus decided to change it.

    I could guess at reasons why it does not change quickly but pure conjecture on my part would only fuel a flame session and create an unprofitable chain of debate. I can't even say that "the people who set the syllabus do not seem to believe that the certification process has flaws that require the syllabus to change", because the syllabus has changed, and the syllabus has beget yet more syllabus.

    I simply do not find that type of change appropriate nor do I find it desirable.

    In order for me to put in effort to make changes to the syllabus I would have to believe that the certification process 'worked' and really did provide its advertised benefits, and currently I don't.

    I would rather put my efforts into writing on blogs like this, clarifying my thoughts and opening them up to criticism (although everyone commenting thus far has treated me very nicely), and making the subsequent writing available to anyone who stumbles across it on whatever learning journey they choose to engage in.

    So I do not think that anything would drag me back to the certification process.

  7. Alan,

    I love your commentary on ISEB. I agree that its a a nice thing in principle but just will bring down the profession of testing. I loved your post so much so that I commented on it on my blog.

    I hope that you enjoy the read and I hope that I removed all aspects of it being a rant. I did try quite hard on remove rant type wording.


    Thanks for the comment David - I've added your blog to my rss feed. I think you managed to write a blog post that did not read like a rant. I don't know how much editing you had to do to it, but if you feel passionately about the topic then I know how easily rant-like writing flows - I do try to avoid that style too.

  8. As usual, entertaining and thought-provoking post. Reading it made me think about why I got one of those certs, and how my position about it shifted throughout the process. I also ended up with an interesting paradox.

    My initial intention with getting the ISEB certification was (naively) to see if there were any areas of testing that I was missing in order to be "a good tester". After reading the syllabus, and even more after the first day of the certification training, that objective changed to just having it on my resume as a door-opener for jobs (it was painfully obvious that in most of the subjects, my colleagues and me went much deeper than the training in our daily testing). I soon after discovered that, for the initial objective, it was much, much better to get actively involved in the testing community, which is much more productive, enriching and much more enjoyable!

    However, having the ISEB was part of the reason I was offered my first job at this company. I am now the manager of the team I started in, and although I haven't had the opportunity to hire someone, if I did, I would not make it a requirement (or even put it under "Desirable") to have an ISEB certification for candidates. I didn't either in my previous job, where I did hire people, and I was very clear about my reasons for not doing so (reasons already expressed in other comments in this blog). I have also been able to question the desirability of the ISEB certification from the day I started, in conversations with other managers responsible for hiring, and also the people in my team; I believe it has made an impact. I wouldn't have been able to do this if I hadn't go that job, which I got partially thanks to the ISEB certification.

    I tend to see this as a different (maybe more passive) kind of attempt to bring the current certification system down. I'm not sure what the point of the certification is, but I do know that I don't get better testers if they are certified by the current system, and I am not a better tester because of my ISEB. The only thing that nags at me is that this is a destructive attempt. I am not involved (yet, at any rate) in creating a certification system that does work. But I also wonder: is it needed? My ultimate certification is the quality of my work. Watch me test, and if it's good enough for you, hire me. If it's not, let me know why so I can learn, and maybe next time we'll be a good match. Unfortunately, there's still not many companies that actually ask you to test something during your interview. Maybe there should be a certification for "hiring testers"? :D

    Hi Marta,

    Thanks for describing your experiences, subversive communication tactics, and alternate strategies for learning.

    I do not know how to create a certification system that works - one reason why I still haven't written an answer to Danny's comment.

    I certainly think that a good training course in "hiring testers" could add value (I'd prefer to read the book) but then I would miss out on the certificate of completion.


  9. Funnily enough, your google ads banner pulls up lots of ISTQB certification links.. I wonder what the click-through rate is for these :)

    Hi Scott :) . Sadly not enough to compensate me for my time in the ISEB process.

  10. An excellent read!

    There is a lot of truth in what you say. BUT, when you work for a company that doesn't believe in having a test team, with many project managers who don't agree with putting aside project time for testing, you have to start to raise the profile of testing somehow.

    Qualification of any sort is usually seen as professional, and often seems the only way to raise the profile of testers. If you are qualified (as the project managers and developers are in their areas) then they may take you more seriously.

    This is the theory I am trying at the moment, and it is working. So, I have nothing better to use than what we have at the moment. If we can improve it then that would be just great...but my experience is that this is a big group who will be slow to change (and may resent any change being pushed upon them!)

    Cheers Sarah

    Hi Sarah, thanks for the comment. You have pointed out a sad truth in our UK industry.

  11. Excellent!

    I went for the same experience. My conclusions are that contents of these courses are made by people who probably were involved in testing 20 years ago. The contents are out of date and the examination is made by people with no understanding in evaluation.

    I argue more than once with the instructor about the irrelevance of the topics and the lack of realism about the real facts of our activity. The answer was always the same..."just learn exactly what it is in the handbook, because the test's question is based on it…forget about what you think!".

    I needed the certification to feed the ignorance of the contracting agencies. I feel like an apple that as been grown with chelmicals but I was given a label saying that I am organic.

    We should make a differentiation between the idea of providing testing practitioners with some sort of qualification, and the courses that currently provides these qualifications.

    I strongly believe that those today provide certification do not contribute to the standardization or improvement of the practice. They are there not because of the validity of what they do, but because they were the first in the market place.

    In sum; this is a great article. Glad to find in this activity people with their own mind. People who dare to say that the kind is naked, while most are afraid of do so.

    Hi Rene,

    Thanks for the comment.

    I think the exam puts the trainers in a difficult position. If the trainees have come to learn how to pass the exam then the trainers do a fine job. If, as it sounds you wanted, the trainees want to learn more about testing then the trainers have to keep coming back to the syllabus and the rote answers because the exam values that above 'real world'.

    One step towards differentiating between a qualification and courses that lead to the qualification would require testers to study for the exam themselves.

    I encourage every tester - if you do place a value on the certification, then study for it on your own. Learn some exam techniques. Learn the syllabus. Pass the exam - or fail the exam, learn from that, then sit the exam again - you will still save money instead of going on a course.

    Your statement about "first in the market place", rings all too true. Hence the 'need' for a 'global' standard certification like the ISTQB. I could cynically view the 'need' in terms of 'we need to compete against any local certification' but then that kind of monopoly 'is' good because then we all speak the one language. Hooray for Esperanto.

    Thank you again for commenting,


  12. Sorry it's taken me so long to come back and look at your reply - somehow, life got in the way!

    Of course I agree with your first 2 comments - I am sure there are many serious developers who have a serious interest in improving both their developing and testing skills (I hope that I was one of them, since I was the "weirdo" who made the move from development to testing...), and I would love to see more junior testers being mentored by senior testers with a view to making testing their profession.

    Maybe in the UK, these things exist to a certain extent. But unfortunately, here in Israel, too many testers have different experiences.

    The "85 grade average" students may one day become wonderful testers, but as long as the majority of the (non-testing) industry here still looks at testing PURELY as a stepping stone to development, and the job descriptions are such that you can tell that almost no-one can be bothered to write something intelligent about testing, those of us "who believe" have to try using as many tactics as possible...

    I have to agree with you also about answering real-life scenario multiple choice questions - the first set of examinees who got my exam all failed!

    Anyway, it takes me a long time till I stop bashing my head against brick walls, so thanks for your good wishes, and keep your fingers crossed :-)

  13. An informative and detailed post - it's always better to hear "here are my detailed experiences with this idea" than "I did X and didn't work, don't do X."

    On the comments, Shane wrote:
    "when you walk through that classroom door, you left the real world and entered the ISTQB world"

    This quote was disturbingly familiar. Years ago, I was a cadet in the Army Reserve Officers Training Corps (ROTC). We did modeling and simuliation of combat envrionments as leadership labortory. These labs were "canned"; there were certain prescribed things you were supposed to do, when, say, you were ambushed or you were reconnoitering an objective.

    These things were scripted; we were evaluated as leaders on how we performed.

    And, well, some of them were contrary to existing Army Combat Doctrine - so the cadets who had spent four or eight years in the "real military" and come to college had to be un-trained. We were told "there is the Army way, and the ROTC way. You'll do things the ROTC way until you get a comission, then you'll go to officers basic course and learn the right way to do things."

    The folks who stuck around (I did not) learned to "put up" with ROTC for four years; it was the junk you'd forget later in order to earn that gold bar of a second lieutenant.

    Is that what ISQTB is? The "junk" we have to "put up with" in order to get a job in software testing?

    I am not certified and I see to be doing just fine. Your Mileage May Vary.