Friday, 15 July 2016

An Open Answer to an Open Letter

TLDR; Condemn? No. Support? I like parts of the paper. Censor? No.
And now I’m doing something I don’t like: writing a blog post in response to ‘something on the internet’. I’m writing a blog post which I don’t think has any practical value. I warn you now. I don’t think you will find much humour herein either. I have added a comedy punchline at the bottom though, if you want to skip ahead.
Chris McMahon has written an open letter to named reviewers of James Bach and Michael Bolton’s paper on “Context Driven Approach to Automation in Testing”. I reviewed the paper.
In the Open Letter Chris McMahon asks the reviewers three things:
  • condemn publicly “both the tone and the substance of that paper”
  • If you do support the paper, I ask you to do so publicly.
  • “regardless of your view, I request that you ask the authors” … “to remove that paper from public view” because it “is an impediment to reasonable discussion and it has no place in the modern discourse about test automation”
I find a few things personally difficult here:
  • I don’t feel strongly about the paper. There are parts I like, parts I agree with, parts I disagree with, parts I don’t like as much. Much like any paper.
  • I don’t like censorship

I’m more annoyed by the “censor this” call to action, than the paper and the surrounding twitter storm.
I view censorship as an absolute last ditch we-have-run-out-of-all-other-options action for tackling a most heinous world threatening situation. That’s not how I view the paper.
Normally I ignore these public ‘add your voice to my voice’ calls for action. But since I was named, and since I was offended by the notion of censorship I wrote this post.

Background


Chris McMahon has written a review of the paper on his blog . He doesn’t like it.
In the paper James and Michael provide 3 case studies. 1 and 2 show some exploratory testing with tools being used. I thought these were fine.
Case 3 is an example of how not to automate something. Chris didn’t like it. I didn’t particularly like it either. But I think I had different reasons than Chris did. I think everyone can (and should) write ‘case studies’ about what they tried, what worked, and what failed. I would prefer to have seen more analysis of the reasons for the different steps in the case study and tying them back to the tool selection criteria that James and Michael write earlier in the paper. And I think some of the conclusions were overly generalised. But its not my paper and James and Michael can write what they want.
And…
If I were trying to automate the GUI of the application that James and Michael chose to automate. I would probably have started with AutoIt or AutoHotKey (Chris disagrees and seems to dislike AutoHotKey). I have used both AutoIt and AutoHotKey in the past. I have even written applications in AutoIt, and I wrote a Java Library around AutoIt DLLs.
I would have tried AutoIt or AutoHotKey first this because I was creating a short term hack to get something done.
If I were writing code to automate it longer term I would have investigated Appium Desktop drivers winappdriver or winium possibly White or Sikuli. I don’t know.
As James and Michael say in their paper automating “requires learning not only about the application and the tool, but also about how they will interact”. I have not had much success automating QT apps in the past, I try to find alternative approaches. I viewed case study 3 as an investigation approach.

Reviewer, not Editor, not Endorser


James and Michael add the names of reviewers to their paper. Regardless of whether they found the comments useful or not. Thanks James and Michael for asking me to review the paper and acknowledging the time I spent creating comments (I mean it. Thank you.).
I review a lot of material that is sent to me. I don’t always receive acknowledgment. I don’t seek acknowledgment. I review it as a favour to the person asking. They then decide what they do with the review comments. Sometimes I review things that I’m not asked to comment on, again the person receiving the comments can do what they like with them.
Some people might think that reviewing the paper means Endorsement. It doesn’t.
If I was an editor, and the paper was appearing in my publication, then it might well mean endorsement, and if I didn’t endorse it I would probably have to publish it with a caveat that ‘this paper does not represent the opinions of the publisher and blah de blah blah’. I didn’t edit it, or publish it. The final paper and all the benefits or otherwise that people derive from it are credited to James and Michael. It is their paper. They can write what they want.
You don’t get to see the comments I sent to James and Michael. And there is no reason that you should. If I lambasted the paper and provided 12 pages of hate fueled rhetoric written using cut out letters from newspapers, or had written a perfume scented love letter in my best penmanship using a quill that I handcrafted from a local peacock feather, you would never know. Either way James and Michael would thank me on their paper for the review comments.

My “Open Letter Answer”


I wrote a blog comment twice but the blogger hosted blog seems to have a system interactional issue with some comments. So I posted here. And that was the reason why I had to write this post. And I still feel sullied for having to do so. Are you enjoying this post so far? You might want to skip to the end for the comedy punchline.


My open letter answer to Chris below:


“I ask you to join me in condemning publicly both the tone and the substance of that paper.”
No.
“Condemn - to express complete disapproval of.”
I don’t think so.
James and Michael asked me to review the paper. I read it. I wrote comments. I sent the comments back to James and Michael. Some of my comments they took on board some of them they chose not to (or the paper does not exhibit all the type of changes I described). That is their right. They wrote the paper. It is theirs.
I’ll point out some features of the paper that I agree with. The simple fact that I agree with some of it, means I can’t condemn it:
  • “There are many wonderful ways tools can be used to help software testing.”
  • Tools applied incorrectly add waste
  • “Let’s break down testing further…” section
  • “Let your context drive your tooling”.
  • Case 1 and Case 2 are better than I remember them
Edit: I wrote more words here, but had to edit them out to fit the max comment length of this blog. Mainly I’m editing out longer examples of things I thought were good, and some I thought were bad, but it doesn’t actually matter what I thought. People can read it and make up their own mind.
All of the above was written to say:
  • I can’t condemn it, because there are parts I agree with
  • The parts I disagree with, I can ignore, and I don’t have to condemn
  • I gave comments to James and Michael, they may have reworked some of the text because of it, some of it they may not have. Its their paper, that’s their choice.
I don’t seem to take as much offence with the paper that you do. I don’t feel that strongly about it.
“If you do support the paper, I ask you to do so publicly.”
I think there are useful parts in this paper. I think some parts are less useful.
There are parts I liked. Some parts I didn’t
There are parts I agree with. Some parts I didn’t.
Dear blog reader: As with everything you read. Read it yourself and make up your own mind. My views of this paper are not important. Only your views of the paper are important. If you feel you are not qualified to ‘pass judgement’ on the paper. Then don’t. Read it for value.
And regardless of your view, I request that you ask the authors of the paper bearing your names to remove that paper from public view as well as to remove the copy that Keith Klain hosts here. For the reasons I pointed out, this paper is an impediment to reasonable discussion and it has no place in the modern discourse about test automation.
No. Absolutely not.
I will not knowingly engage in, or condone, censorship.
No.
The fact that I was asked this. Is the reason I responded at length.
“this paper is an impediment to reasonable discussion”.
Hardly.
You pointed out what you thought was wrong with the paper. I see nothing in your comments that demonstrate “this paper is an impediment to reasonable discussion”. On the contrary, having pointed out the flaws, you can then go on to say what you think people can do instead. That is a prompt to reasonable discussion, not an impediment.
“it has no place in the modern discourse about test automation”. People are entitled to describe their experiences of automating software. Even if they describe situations that they failed in the process. I might prefer to see some different conclusions drawn, or different generalisations made. But its not my paper.

Strong Feelings


Clearly Chris feels strongly about the paper. I don’t feel as strongly about the paper as he does.
When I do encounter ‘things on the internet’ that I feel strongly about I try to create, or link to material, that offers alternative choices and demonstrates alternative views, which might open up options for myself or my readers.
I don’t think I ‘attack’ as much as I used to. And I think I’ve taken down a lot of the old ‘book reviews’ I wrote which did that, since I don’t think they added value.
If I read a criticism, I like to read what the author thinks I should do instead. Then I can see balance.

People Read Differently


People will read the paper differently.
I read it initially to pass on comments to James and Michael, so I made the points above (in more detail) and made some other points and some very minor points. I don’t want to write them all out here because of ‘reviewer’/‘writer’ confidentiality. That sacrosanct bond between a writer and their reviewer, the comments were for their eyes only. They chose what to act on, and what to discard.
After publication, I saw the paper had changed, and I read it to see what I could take from it. And there were parts of the paper I found value in:
e.g.
  • “There are many wonderful ways tools can be used to help software testing.”
  • Tools applied incorrectly add waste
  • “Let’s break down testing further…” section
  • “Let your context drive your tooling”.
  • Case 1 and Case 2 are better than I remember them
And there are more than that, but you can read the paper for value and see what you take from it.
I did not read it to critique it or pass judgement. The paper didn’t trigger enough of a reaction in me that I thought I should massively promote the paper, or reduce the impact of the paper. I thought it would promote itself given the reputation of James and Michael. And the views in the paper are those of James and Michael. I didn’t see that I should try and enforce my views on top of that. I have blogs and conference talks and articles for that.
Some people read with pre-existing view of the authors either positive or negative. Personally, I will happily read a William Shatner novel, regardless of its perceived literary value by others. And there are some authors that I would never read again - regardless of their ability to sell millions and spawn movie after movie.
And I’m sure you can identify more ways of ‘reading’, so given the controversy that seems to have surrounded the paper I suggest you pick a reading approach prior to reading.

Related Reading


I have some papers and conference talks related to this topic:

The Paper


James and Michael have released the paper. For free. For anyone to read.
They continue to edit it. Possibly for Typos, possibly to clarify their position in the paper.
You could read it. You might disagree with it. You might find value in it. You might want to send comments to James and Michael to clarify sections. You might want to write a critque yourself, you might come out in favour with the paper, you might not.
Your choice. Your reaction. Your learning.
If you want to build a ‘movement’ against it. That’s your choice. Leave me out of it.

Humour Section - The Punchline


I will quote one item from my review, I hope James and Michael forgive me for breaking the hallowed state of “reviewer/writer” confidentiality.
I said " I don’t really see anything controversial in the paper so you could publish now if you want, and nothing bad will happen."
I honestly thought that if people read the paper, they would read it; concentrate on the parts they liked, ignore the parts they don’t.
How naive was I?

2 comments:

  1. Hi Alan,

    Thanks for the comment.

    I run into all sorts of differing perspectives about automation in software testing.

    I am a fan of using tools to get the job done.

    I use test automation to support my work.

    I try to help my customers avoid the "snarly twisted trap", which I have been caught in many times, of the job becoming to support test automation tools.

    I urge the community to share examples and case studies and open them up to constructive criticism.

    Thank you

    Rob

    ReplyDelete
    Replies
    1. Thanks Rob. I too would love to see more shared examples. :)

      Delete