Thursday, 18 December 2014

My search for easy to use, free, local HTTP servers


I have lost count of the number of times I've had to look for a local HTTP server.
  • Experimenting with an open source app
  • Writing some HTML, JavaScript, PHP
  • Testing some flash app for a client
  • Running some internal client code
  • etc. etc.
And since this isn't something I do every day. I forget how to do it, each and every time I start.

I forget:
  • Which servers I already have installed
  • Where I installed them
  • Which directory I configured them to use
  • What local names did I give them to make it 'easy' for me to work with them
  • etc. etc.
Now it might just be me that faces this problem.

If so, I expect you have already stopped reading.

So to cut to the chase, my current favourites are Mongoose (Windows, Mac, Linux) and VirtualHostX (Mac)


Other HTTP Stacks

I have used some of the biggies:
And I probably still have them installed

And some of the tinies:
And some others that I can't remember.

All have been useful at the time. Sometimes I tried to install one but couldn't get it working on client machines because of permissions etc. etc.

I started looking around for alternatives that I could use during training courses, webinars etc.

Some I have not used

Prior to writing this post I was aware that Python had the capability to start up a small http server from the command line, but I hadn't used it. After publication, Brian Goad tweeted his usage of Python to do this.

Brian continued:
could be easily used as a function that takes the dir as argument: simple-server(){ cd $1; python -m SimpleHTTPServer; }
just go to localhost:8000 and you're set!
After Brian's reminder I had a quick look to see what other languages can do this:

If you know of any more languages that have this as part of their default then leave a comment and I'll add them here.

Virtual Machine Stacks

One thing I started using were virtual machines that have software installed already and don't require a web server e.g.
These are great for getting started quickly, but require a little download overhead - which can be painful over conference internet connections.

Sometimes I set up machines in the cloud, preinstalled:
As an additional backup, I like to have a local version that I can share.

VirtualHostX for the Mac

Since I mainly travel with a Mac Laptop I started using VirtualHostX for that.

VirtualHostX is basically a GUI that helps me work with the existing Mac installed LAMP stack.

I can avoid the Mac and command line config. I can avoid installing everything else, and just use VirtualHostX to configure and start/stop everything.

This saved a massive amount of time for me and I do recommend it. But it is Mac only.

Mongoose for Mac, Windows and Linux

I recently encountered Mongoose. It works on Mac, Windows and Linux. 

I used the free version to quickly experiment with some downloaded open source libraries. 

All you do is download the small executable into the directory, run it, and you get the traditional XAMPP style taskbar tooltip icon and easy to use config. 

You can run multiple versions by having them listen on different ports.

I paid $8 for the Windows dev version which allows me to view the HTTP traffic easily as well. This $8 also gives me access to the Linux Pro version. For an extra $5 I could get access to the MacOS pro version.

Summary

I suspect that 'proper' web developers will always prefer an XAMPP installation. But they will also use it more and be completely familiar with it.

For someone like me, who jumps between apps, configs, machines, sites, etc. 

I suspect that at some point I'll probably jump back to  XAMPP due to some future client needs. But for my own work. VirtualHostX and Mongoose are my current easy to use solutions.

What do you use?

Friday, 28 November 2014

Agile Testing Days 2014 - Workshop and Tutorial

At Agile Testing Days 2014, I presented a full day workshop on "Technical Testing" in Agile and was part of the Black Ops Testing Workshop with Steve Green and Tony Bruce.

Note: there are limited spaces left on our
  Black Ops Testing Full Day tutorial
in London in January 2015

Both of these were hands on events.

In the tutorial I present examples of Technical Testing, and how to integrate Technical Testing into Agile, the participants also test a real live application and apply the techniques, mindsets and tools that I describe.

Since it describe Technical Testing in an Agile system, we also spent time discussing the injection points for the technical testing process and thought processes.

The Black Ops Testing workshop took a similar approach but with less 'talking' since it was a much shorter time period with more people.

We started with a 5 minute lightning talk from myself, Tony, and Steve. During this we emphasized something important that we hoped the participants would focus on during their testing. We then let the participants loose on the system as we mingled. We coached, asked questions and observed. Then during the debrief we extemporize on our observations and ask questions about what we saw to pull out the insights from the participants. We repeat this process over the session.

Both the Black Ops Testing Workshop and my Tutorial used Redmine as the application under test.

We picked Redmine for a number of reasons, and I'll list some below:

  • Virtual Machines are available which make it easy to install
  • The Virtual Machines can be deployed easily to Amazon cloud servers
  • It is available as an Amazon Cloud Marketplace system, making it easy to install
There are some words in there you might notice "easy to install", "deployed easily".

Wouldn't it be great if all the apps we had to test were easy to install and configure and gain access to?

Yes it would. And as testers, when we work on projects, we can stress this to the project team, or work on it ourselves so that we don't spend a lot of time messing about with environments.

I used bitnami and their dashboard to automatically deploy and configure the environment. Tony used the amazon aws marketplace and worked with their dashboard. James Lyndsay helped us out, and went all old school, and deployed the base install to the machine.

I learned from this, that my exploratory note taking approach has permeated all my work. As I was installing the environment and configuring it, I made notes of 'what' I wanted to do, 'where' I was finding the information I needed, 'how' to do the steps I took, what data I was using (usernames, passwords, environment names, etc.). And when I needed to repeat the installation (I installed to a VM in Windows, on Mac, and in the cloud), I had all my notes from the previous installations.

When something went wrong in the environment, and I didn't have access to the parts of the system I needed, I was able to look through my notes and I could see that there were activities I had not performed. I thought I had, but my notes don't lie to me as much as my memory does.

It should come as no surprise to you then, that I stress note taking in my tutorial, and in the Black Ops Testing Workshop.


You can find the slides for the Black Ops Testing Workshop on slideshare. You can find more details about Black Ops Testing over on our .com site


Agile Testing Days 2014 - Keynote

I presented a keynote at Agile Testing Days 2014, and took part in the Black Ops Testing Workshop, and presented a one day tutorial on "Technical Testing in Agile". This post covers the Keynote.

The keynote was underpinned by the notion that 'Agile' is not a 'thing', instead 'Agile' provides the context within which our project operates and therefore is part of the weltanshauung of the project. Or as I referred to it, the "System Of Development".

Because really I concentrate on 'Systems'. I think that I view 'context', as an understanding of the System. And remember that, we, the tester, form part of that system and the context as well. Therefore our 'beliefs' about testing become important as the impact how we interact with the people, and the system, and the process, in place on the project.

As ever, I made a lot of notes before the Keynote, and I present those below.

The slides are available on slideshare. The talk was recorded, but has not yet appeared online. My notes below might help you make sense of the slides.



A few things to note. Many of the keynotes overlapped: talking about role identification and beliefs, communicating from your own experience and models, building models of your process and testing, etc.

And prior to Agile Testing Days, Michael Bolton presented a Eurostar Webinar on 'Agile', which is worth watching, and James Bach presented "Skilled Testing and Agile Development" at Oredev 2014 which is worth watching. I include links to both of those for your delectation because they represent other testers applying Systems Thinking to create a model of 'Testing in Agile', watch them both.


Abstract:

Every Agile project is different, we know this, we don't do things 'by the book' on Agile projects. We learn, we interact, we change, we write the book we go along. Throughout all of this, testing needs to remain viable, and it needs to add value. Remaining viable in this kind of environment can be hard.

Fortunately, we can learn to add value. In this keynote, Alan will describe some of the approaches and models he has used to help testing remain viable. Helping testers analyze the 'system of development' so the test approach can target process risks. Helping testers harness their own unique skills and approaches. The attitudes that the testing process often needs to have driving it, and the skill sets that teams need to ensure are applied to their testing.

At a simple level, this is just Systems Thinking and Modeling. In practice this can prove highly subversive and deliberately provocative. Because we're not talking about 'fitting in', we're talking about survival.

Notes:

Warren Zevon, wrote “Ain’t that pretty at all”, in 1982

Warren Zevon wrote a song called “ain’t that pretty at all"

Warren Zevon was one of those singers who when he comes on, I pretty much have to listen, his voice and songs drag me in, rather than sitting as background music.

In this song, Mr Zevon describes a character who is pretty jaded.

     Well, I've seen all there is to see
     And I've heard all they have to say
     I've done everything I wanted to do . . .
     I've done that too

I know what that feels like. I’ve done management, performance testing, exploratory testing, agile testing, security testing, acceptance testing, UAT, automation, etc.

I’ve worked on Agile, waterfall, heavy weight documentation, lean, yada yada yada. None of it has ever fully worked or been perfect.

Feels like I’ve done everything. the danger is I become jaded or fixed in my ways.

People want Agile to be perfect.

And this Warren Zevon character And doesn’t like what he sees.

And Agile isn’t perfect.

Reality doesn’t seem to match up with his expectations, or desires, or wants.

     And it ain't that pretty at all
     Ain’t that pretty at all

Agile can be messy. It doesn’t always match the books or talks. And when you are new to it sometimes you don’t like what you see.

I went to the grand canyon, took a bus couple of hours to get there. When I got out, everyone else seemed to see a wonder example of mother nature.

I saw some cliffs.

We may not have the strategies to cope

I was back on the bus in 10 minutes. I might be jaded but I can avoid the sunk cost fallacy.

But we might not have the strategies we need to deal with the situation.

    So I'm going to hurl myself against the wall
    'Cause I'd rather feel bad than not feel anything at all

If you come from a waterfall background you might not know how to handle the interactions on an agile project. And if your prep was books and blogs, you might find they described an ideal where the strategies they used are not in place.

Some of our strategies for coping might be self destructive and we might not notice, and other people might not tell us. Because systems are self-healing and they can heal by excluding the toxic thing in the system.

Without the right strategy we make the wrong choice

And when you don’t have a lot of strategies you end up making choices and taking actions that aren’t necessarily the most appropriate for the situation.

Testers telling developers that the project would be better if they just did TDD and paired. Or if we all worked on automation acceptance tests together. Might not get the outcome that you want.

You might fall back on strategies that worked on other projects. But don’t fit in this one.

    So I'm going to hurl myself against the wall
    'Cause I'd rather feel bad than not feel anything at all

You end up wanting to write lots of documentation up front because you’ve always done it, or you want to test in places where the stories don’t go, or you want to remind everyone that ‘Agile’ isn’t done like that.

Whatever...

So very often, my job….

    I've been to Paris
    And it ain't that pretty at all
    I've been to Rome
    Guess what?

Is to help people, with their beliefs and expectations. To work with the people and system around them.

Because when I walked away from the Grand Canyon it wasn’t a reflection on the grand canyon, it was a reflection of me. My expectations were different from the reality. My belief in being wowed, because of all the things I’d heard about it stepped in the way of seeing what was on the ground in front of me. Or in the ground in front of me.

So they don’t do something stupid

   I'd like to go back to Paris someday and visit the Louvre Museum
   Get a good running start and hurl myself at the wall
   Going to hurl myself against the wall
   'Cause I'd rather feel bad than feel nothing at all
   And it ain't that pretty at all
   Ain't that pretty at all

So they don’t do something stupid on the project that stops them maximising their effectiveness, and puts blockers in the way to working effectively with the team.

I help testers survive in Agile projects

That’s never my role title. Test Manager, Test Consultant, Automation Specialist. Agile Tester. Blah Blah Blah.

But I seem to help testers survive, and help testing survive. So that we don’t fall prey to just automating acceptance criteria, we actually test the product and explore its capabilities.

And I say ’survive’, because that's what I had to learn to do.

We survive when we add value

I think we survive when we add value, learn, and make ourselves a viable part of the project in ways that are unique to us.

In “The Princess Bride”, the hero Westley is caught by The Dread Pirate Roberts, whom he offers to be his valet for 5 years.

The Dread Pirate Roberts agrees to try it but says he will most probably kill him in the morning. He’s never had a valet before so doesn’t know how a valet would add value to him, or how he could use him.

And when the morning arrives and The Dread Pirate Roberts comes to kill poor Westley. Westley thanks him for the opportunity to have learned so much about the workings of a pirate ship the previous day.

Westley explains to the dread pirate roberts how much he learned about the workings of the ship, and how he had helped the cook by pairing with him, and how he had reorganised the items in the cargo hold.

And every day Westley survives his capture by the Dread Pirate Roberts by working and adding value during the day. And every day he demonstrates the value that he can add to the operation of the pirate ship. And every day he learns more about piracy and the skills of pirates.

Until, years later, The Dread Pirate Roberts explains that The Dread Pirate Roberts is a role, not a person, and so the dock into a port, take on a new crew and Westley adopts the role of Dread Pirate Roberts.

And testers need to act such that people don't ask "What does testing do on Agile" because they know what their testers do on the project, and they see value in those activities. They know what specifically "Bob and Eris, or Dick and Jane or Janet and John" their testers, actually do.

This isn’t fluffy people stuff

When I look online and see the type of strategies that people describe for surviving and fitting in on Agile projects, it isn’t quite what I do or recommend, so I know there are alternative paths and routes in.

I’m not here to make you look good

I’m really not here to make you look good.

I mean I’m not here to make you look bad… but I might.

And I’m not here to make the product look bad… but I might.

And I’m not here to make the process, or the testing, or the ‘whatever’ look bad… but I might.

If it aint that pretty, then we can do something about it when we recognise that, and negative feedback can help.

I’m really here to help raise the bar and improve the work we all do together. If that makes you look good, then that’s great, if that makes you look bad and you improve as a result, then that’s great too. Looking good is a side-effect.

Survival does not mean 'fitting in'

I’m not talking about fitting in. Buying Lunch. Making people look good. yada yada.

I don’t think you get bonus points for doing that.

But if that’s you, don’t stop. I think its great if people want to buy doughnuts for everyone.

Its not me. So there are other ways to survive on projects.  I don’t have many strategies for ‘social success'

Personally I think team work means helping other people contribute the best stuff they can, and the stuff that only they can, and help pass that knowledge around the team. And everyone needs to to hat.

Testers survive by doing testing stuff

Its pretty much as complicated as that. Learn to test as much as you can, and drop all the bureaucracy and waste that adds no value. Then you’ve covered the basics.

Then you learn how to help others improve by passing on your knowledge and mindsets.

And we are taught how to do ‘testing stuff’ so if we do that well and improve our testing skills then we have a good chance of survival.

Testers are taught to work with the System Under Development

Essentially we are taught how to understand and work with the System Under Development

We learn techniques to analyse the system and model it and then build questions from the model which we ask of the system.

We might build a data domain model, then build questions around its boundaries and then ask the system those questions by phrasing them in form the system understands.

We learn how to do that as testers with techniques and analysis and blah de blah.. testing stuff.

We survive when we adapt to the System Of Development

Testing survives when it learns to work with the Systems in place. And two obvious systems are the System under development, and the System of development. We have to adapt to both.

And fortunately testers are often good at analysing systems. Modeling them. Viewing them from different perspectives. Breaking them into chunks. Viewing the flow of data. working out what data becomes information to subsystems. Looking for data transformations. Looking for risk in process and communication. Then figuring out different injection points.

When we work with systems under development we don't just input data at one end and see what happens out the other. Same with Systems of development, we don't just work at the stories that come in, and then check the system that comes out. We learn to look for different injection points and create feedback earlier.

So one of the main things I help testers do, is analyse the System Of Development, and adapt to it.

Many of the testing survival tricks and techniques we learn relate to Waterfall projects.

Annie Edson Taylor, thought she would become rich and famous if she could survive a trip over the niagra falls in a barrel.

She survived. She wasn’t rich and famous.

She survived by padding out her barrel and increasing the air pressure in the barrel.

You can get yourself ready before you hit Agile.

I survived waterfall by removing all padding, and taking responsibility for what I did.

I analysed the system of development and did what this specific implementation needed, so I could survive, and I took responsibility so that I could add value.

So I was in a better place than many people on waterfall projects when we started working on Agile.

Blake wrote "I must create a system..."

I must create a system or be enslaved by another man's. My business is not to reason and compare, my business is to create.

This is one of the first texts I ever read on system design and modelling, and I read it when I was about 19 or 20, and stuff stays with you when you encounter it early.

But this is my meta model. of modelling. And I know I have to own and take responsibility for my view of the world and the systems that I work with. Otherwise I’ll fall prey to ‘other’ peoples’ strategies.

I remember the first time I worked on an ‘Agile’ project

I had to learn the strategies to survive. And that’s how I can recognise the same blockers in other testers or projects new to Agile.

I fell prey to the Agile Hype

I had a whole set of beliefs about what agile was going to be like:

- the pairing
- the TDD
- the ATDD
- the BDD
- the ADBDBTBD - I might have made that one up

But there was a lot of stuff.

And it ain't that pretty at all

Reality wasn’t as pretty as I imagined.

I got stuck.

Remember I knew how to handle Waterfall. I had that down.

I could work the system of development and work around the blocks and annoyances that it threw at me.

But here was a new thing. A new system.

Stuck on...

I knew what we were ‘supposed’ to do. But people weren’t really doing that. and I didn’t know how to do this… hybrid thing.

I realised that people didn’t know what to do.

And I realised I simply didn’t have the basic skills to work in this new system of development.

I was the worst ‘pair’ in the world. You know when deer get caught in headlights. That was me the first time I was given the keyboard in a pairing session.

So I did what I always do...

Try to take over the world.

No - of course not. I’ve tried that before. Trying to impose your model on top of everyone else’s and making it work, is hard.

Its easier to treat the system as a unique system and model it. Observe it. Understand it.

Look at the parts, the subsystems, the relationships, the feedback flows, the amplifiers, the attenuators.

See I think this is what I do, I work with systems

That’s why my social strategies are… different.

I see the people systems in front of me. The processes. The politics. etc.

And that led to me changing… me

And much of what I did, I teach testers to do on projects.

I worked on my coding skills so I could pair

Every Agile Project is different. 

So we learn to analyse the system that is in place. TheWeltanschauung. The context.

System's have common concepts and elements, in general: entities, relationships, data flows, subsystems etc.

But each specific system, is different, and we can view it in different ways, with different models.

Thursday, 6 November 2014

Confessions of An Accidental Security Tester

At Oredev 2014 I presented "Confessions of an Accidental Security Tester".

The slides are on slideshare, the video below and on vimeo:


"Alan Richardson does not describe himself as a security tester. He's read the books so he knows enough to know he doesn't know or do that stuff properly. But he has found security issues, on projects, and on live sites that he depends on for his business.

You want to know user details? Yup, found those. You want to download the paid for assets from the site without paying for them? Yup, can do. You want to see the payment details for other people? OK, here they are. All of this, and more, as Alan stumbled, shocked, from one security issue to the next,

In this session Alan describes examples of security issues, and how he found them: the tools he used, why he used them, what he observed and what that triggered in his thought processes.

Perhaps most shocking, is not that the issues were live, and relatively easy to find and exploit. But that the companies were so uninterested in them. So this talk also covers how to 'advocate' for these issues. It also warns you not to expect rewards and gratitude. Companies with these type of issues typically do not have bug bounty schemes.

Nowadays, many of the tools you need to find and exploit these issues are built in to the browser. Anyone could find them. But testers have a head start. So in this session Alan shows how you can build on the knowledge and thought processes you already have, to find these types of issues.

This is a talk about pushing your functional testing further, deeper, and with more technical observation, so you too can 'accidentally' discover security issues."

CONFESSIONS OF AN ACCIDENTAL SECURITY TESTER - "I DIDN'T BREAK IN, YOU LEFT THE DOOR OPEN" from ├średev Conference on Vimeo.

Wednesday, 24 September 2014

An exploratory testing example explored: Taskwarrior

or "Why I explored Taskwarrior the way I did".

In a previous post I discussed the tooling environment that I wanted to support my testing of Taskwarrior for the Black Ops Testing webinar of 22nd September 2014.

In this post, I'll discuss the 'actual testing' that I performed, and why I think I performed it the way I did. At the end of the post I have summarised some 'principles' that I have drawn from my notes.

I didn't intend to write such a long post, but I've added sections to break it up. Best of luck if you make it all the way through.


First Some Context

First, some context:
  • 4 of us, are involved in the testing for the webinar
  • I want the webinars to have entertainment and educational value
  • I work fast to try and find 'talking points' to support the webinar

The above means that I don't always approach the activity the same way I would a commercial project, but I use the same skills and thought processes. 

Therefore to meet the context:
  • I test to 'find bugs fast' because they are 'talking points' in the webinar
  • I explore different ways of testing because we learn from that and can hopefully talk about things that are 'new' for us, and possibly also new for some people on the webinar

I make these points so that you don't view the testing in the webinar as 'this is how you should do exploratory testing' but you understand 'why' I perform the specific exploration I talk about in the webinar.

And then I started Learning


So what did I do, and what led to what?

(At this point I'm reading my notes from github to refresh my memory of the testing)

After making sure I had a good enough toolset to support my testing...

I spent a quick 20 minutes learning the basics of the application using the 30 second guide. At this point I knew 'all you need to know' to use the application and create tasks, delete tasks, complete tasks and view upcoming tasks.

Because of the tooling I set up, I can now see that there are 4 files used in the application to store data (in version 2.2)

  • pending.data
  • history
  • undo.data
  • completed.data

From seeing the files using multitail, I know:
  • the format of the data
  • items move from one file to another
  • undo.data has a 'before' and 'after' state
  • history would normally look 'empty' when viewing the file, but because I'm tailing it, I can see data added and then removed, so it acts as a temporary buffer for data

In the back of my head now, I have a model that:
  • there might be risks moving data from one file to another if a file was locked etc. 
  • text files can be amended, 
    • does the app handle malformed files? truncated files? erroneous input in files?
  • state is represented in the files, so if the app crashed midway then the files might be out of sync
  • undo would act on a task action by task action
  • as a user, the text files make it simpler for me to recover from any application errors and increase the flexibility open to me as a technical user
  • as a tester, I could create test data pretty easily by creating files from scratch or amending the files
  • as a tester, I can put the application into the state I want by amending the files and bypassing the GUI

All of these things I might want to test for if I was doing a commercial exploratory session - I don't pursue these in this session because 'yes they might make good viewing' but 'I suspect there are bugs waiting in the simpler functions'.

After 20 minutes:
  • I have a basic understanding of the application, 
  • I've read minimal documentation, 
  • I've had hands on experience with the application,
  • I have a working model of the application storage mechanism
  • I have a model of some risks to pursue

I take my initial learning and build a 'plan'


I decide to experiment with minimal data for the 'all you need to know' functionality and ask myself a series of questions, which I wrote down as a 'plan' in my notes.

  • What if I do the minimal commands incorrectly?

Which I then expanded as a set of sub questions to explore this question:
  • if I give a wrong command?
  • if I miss out an id?
  • if I repeat a command?
  • if a task id does not exist?
  • if I use a priority that does not exist?
  • if I use an attribute that does not exist?
This is not a complete scope expansion for the command and attributes I've learned about, but this is a set of questions that I can ask of the application as 'tests'.

I gain more information about the application and learn stuff, and cover some command and data scope as I do so.

I start to follow the plan


10 minutes later I start 'testing' by asking these questions to the application.

Almost immediately my 'plan' of 'work through the questions and learn stuff' goes awry.

I chose 'app' as a 'wrong command' instead of 'add'. but the application prompts me that it will modify all tasks. I didn't expect that. So I have to think:
  • the message by the application suggests that 'app' == 'append'
  • I didn't know there was an append command, I didn't spot that. A quick 'task help | grep append' tells me that there is an append command. I try to grep for 'app' but I don't see that 'app' is a synonym for append. Perhaps commands can be truncated? (A question about truncation to investigate for later)
  • I also didn't expect to see 3 tasks listed as being appended to, since I only have one pending task, one completed task, and one deleted task. (Is that normal? A question to investigate in the future)

And so, with one test I have a whole new set of areas to investigate and research. Some of this would lead to tests. Some of this would lead to reading documentation. Some of this would lead to conversations with developers and analysts and etc. if I was working on a project.

I conduct some more testing using the questions, and add the answers to my notes.

I also learn about the 'modify' command.

Now I have CRUD commands: 'add', 'task', 'modify' | 'done', 'delete'

I can see that I stop my learning session at that point - it coincidentally happens to be 90 mins. Not by design. Just, that seemed to be the right amount of time for a focused test learning session.

I break and reflect


When I come back from my break, I reflect on what I've done and decide on an approach to ask questions of:
  • how can I have the system 'show' me the data? i.e. deleted data
  • how can I add a task with a 'due' date to explore more functionality?
These questions lead me to read the documentation a little more. And I discover how to show the internal data from the system using the 'info' command. I also learn how to see 'deleted' tasks and a little more about filtering.

Seeing the data returned by 'info' makes me wonder if I can 'modify' the data shown there. 

I start testing modify


I already know that I can delete a task with the delete command. But perhaps I can modify the 'status' and create a 'deleted' task that way?

And I start to explore the modify command using the attributes that are shown to me, by the application. (you can see more detail in the report)

And I can 'delete' a task using the modify, but it does not exercise the full workflow i.e. the task is still in the pending file until I issue a 'report'. 

My model of the application changes:
  • reporting - actually does a data clean up, prior to the report and moves deleted tasks and hanging actions so that the pending file is 'clean'
  • 'modify' bypasses some of the top level application controls so might be a way of stressing the functionality a little
At this point, I start to observe unexpected side-effects in the files. And I find a bug where I can create a blank undo record that I can't undo (we demonstrate this in the webinar).

This is my first 'bug' and it is a direct result of observing a level lower than the GUI, i.e. at a technical level.

I start modifying calculated fields


I know, from viewing the storage mechanism that some fields shown on the GUI i.e. ID, and Urgency. Do not actually exist. They do not exist in the storage mechanism in the file. They are calculated fields and exist in the internal model of the data, not in the persistent model.

So I wonder if I can modify those?

The system allows me to modify them, and add them to the persistence mechanism, but ignores them for the internal model.

This doesn't seem like a major bug, but I believe it to be a bug since other attributes I try to modify, which do not exist in the internal model e.g. 'newAttrib' and 'bob' are not accepted by the modify command, but 'id' and 'urgency' are.

I start exploring the 'entry' value


The 'entry' value is the date the task was entered. This is automatically added:

  •  Should I be able to amend it?
  • What happens if I can?
So I start to experiment.

I discover that I can amend it, and that I can amend it to be in the future.

I might expect this with a 'due' date. But not a 'task creation date' i.e. I know about this task but it hasn't been created yet.

I then check if this actually matters. i.e. if tasks that didn't officially exist yet are always less urgent than tasks that do, then it probably didn't matter.

But I was able to see that a task that didn't exist yet was more urgent than a task that did.

follow on questions that I didn't pursue would then relate to other attributes on the task:
  •  if the task doesn't exist yet, should priority matter?
  • Can I have a due date before the entry exists?
  • etc.
Since I was following a defect seam, I didn't think about this at the time.

And that's why we review our testing. As I'm doing now. To reflect on what I did, and identify new ways of using the information that I found.

Playtime


I'd been testing for a while, so I wanted a 'lighter' approach.

I decided to see if I could create a recurring task, that recurred so quickly, that it would generate lots of data for me.

Recurring tasks are typically, weekly or daily. But what if a task repeated every second? in a minute I could create 60 and have more data to play with.

But I created a Sorcerers Apprentice moment. Tasks were spawning every second, but I didn't know how to stop them. The application would not allow me to mark a 'parent' recurring task as done, or delete a 'parent' recurring task. I would have to delete all the children, but they were spawning every second. What could I do?

I could probably amend the text files, but I might introduce a referential integrity or data error. I really wanted to use the system to 'fix' this.

Eventually I turned to the 'modify' command. If I can't mark the task as 'deleted' using the 'delete' command. Perhaps I can bypass the controls and modify it to 'status:deleted'

And I did. So a 'bug' that I identify about bypassing controls, was actually useful for my testing. Perhaps the 'modify' command is working as expected and is actually for admins etc.

Immediate Reflection


I decided to finish my day by looking at my logs.

I decided:

  • If only there were a better way of tracking the testing.

Which was one of the questions I identify in my 'tool setup session' but decided I had a 'good enough' approach to start.

And now, having added some value with testing, and learning even more about what I need the tooling to support. I thought I could justify some time to improve my tooling.

I had a quick search, experimented with the 'script' command, but eventually found that using a different terminal which logged input and output would give me an eve better 'good enough' environment.

Release


We held the Black Ops Testing Webinar and discussed our testing, where I learned some approaches from the rest of the team.

I released my test notes to the wild on github.

Later Reflection


This blog post represents my more detailed reflection on what I did.

This reflection was not for the purpose of scope expansion, clearly I could do 'more' testing around the areas I mentioned in my notes.

This reflection was for the purpose of thinking through my thinking, and trying to communicate it to others as well as myself. Because it is all very well seeing what I did, with the notes. And seeing the questions that I wrote allows you to build a model of my test thinking. But a meta reflection on my notes seemed like a useful activity to pursue.

If you notice any principles of approaches in my notes that I didn't document here, then please let me know in the comments.

Principles

  • Questions can drive testing...
    • You don't need the answers, you just need to know how to convert the question into a form the application can understand. The application knows all the answers, and the answers it gives you are always correct. You have to decide if those 'correct' answers were the ones you, or the spec, or the user, or the data, or etc. etc., actually wanted.
  • The answers you receive from your questions will drive your testing
    • Answers lead to more questions. More questions drive more testing.
  • The observations you make as you ask questions and review your answers, will drive your testing.
    • The level to which you can observe, will determine how far you can take this. If you only observe at the GUI layer then you are limited to a surface examination. I was able to observe at the storage layer, so I was able to pursue different approaches than someone working at the GUI.
  • Experiment with internal consistency
    • Entity level with consistency between attributes
    • Referential consistency between entities
    • Consistency between internal storage and persistent storage
    • etc.
  • Debriefs are important to create new plans of attack
    • personal debriefs allow you to learn from yourself, identify gaps and new approaches
    • your notes have to support your debriefs otherwise you will work from memory and don't give yourself all the tools you need to do a proper debrief.
  • Switch between 'serious' testing and 'playful' testing
    • It gives your brain a rest
    • It keeps you energised
    • You'll find different things.


Tuesday, 23 September 2014

Lessons learned testing Command Line Applications from Black Ops Testing Webinar

For the Black Ops Testing Webinar on 22nd September 2014 we were testing Taskwarrior, a CLI application.

I test a lot of Web software, hence the "Technical Web Testing 101" course, but this was a CLI application so I needed to get my Unix skills up to date again, and figure out what supporting infrastructure I needed.

By the way, you can see the full list of notes I made at github. In this post I'm going to explain the thought process a little more.

Before I started testing I wanted to make sure I could meet my 'technical testing' needs:
  • Observe the System
  • Control the Environment
  • Restore the App to known states
  • Take time stamp logs of my testing
  • Backup my data and logs from the VM to my main machine
And now I'll explain how I met those needs... and the principles the process served.



You will also find here a real life exploratory testing log that I wrote as I tested.

Observe the System

Part of my Technical Testing approach requires me to have the ability to Observe the system that I test.

With the web this is easy, I use proxies and developer tools.

How to do this with a CLI app? Well in this case Taskwarrior is file based. So I hunted around for some file monitoring tools.

I already knew about Multitail, so that was my default.  'Tail' allows you to monitor the changes to the end of a file. Multitail allows me to 'tail' multiple files in the same window.

I looked around for other file monitoring tools, but couldn't really find any.

With Multitail I was able to start it with a single command that was 'monitor all files in this directory'
  • multitail -Q 1 ~/.task/*

I knew that the files would grow larger than tail could display, so I really wanted a way to view the files.

James Lyndsay used Textmate to see the files changing. I didn't have time to look around for an editor that would do that (and I didn't know James was using Textmate because we test in isolation and debrief later so we can all learn from each other and ask questions). 

So I used gedit. The out of the box editing tool on Linux Ubuntu. This will reload the file if it has changed on the disk when I change tabs. And since I was monitoring the files using Multitail, I knew when to change tabs.

OK, so I'm 'good' on the monitoring front.

Control the Environment

Next thing I want the ability to do? Reset my environment to a clean state.

With Taskwarrior that simply involves deleting the data files:
  • rm ~/.task/*

Restore the App to known states

OK. So now I want the ability to backup my data, and restore it.

I found a backup link on the Taskwarrior site. But I wanted to zip up the files rather than tar them, simply because it made it easier to work cross platform.
  • zip -r ~/Dropbox/shared/taskData.zip ~/.task
The above zips up, recursively, a directory. I used recursive add, just in case Taskwarrior changed its file setup as I tested. Because my analysis of how it stored the data was based on an initial 'add a few tasks' usage I could not count on it remaining true for the life of the application usage. Perhaps as I do more complicated tasks it would change? I didn't know, so using a 'recursive add' gave me a safety net.

Same as the Multitail command, which will automatically add any new files it finds in the directory, so it added a history file after I started monitoring.

Backup my data and logs from the VM to my main machine

I would do most of my testing in a VM, and I really wanted the files to exist on my main machine because I have all the tools I need to edit and process them there. And rather than messing about with a version control system or shared folders between VM etc.

I decided the fastest way of connecting my machines was by using Dropbox. So I backup the data and my logs to Dropbox on the VM and it automatically syncs to all my other machines, including my desktop.

Take time stamp logs of my testing

I had in the back of my mind that I could probably use the 'history' function in Bash to help me track my testing.

Every command you type into the Bash shell is recorded in the history log. And you can see the commands you type if you use the 'history' command. You can even repeat them if you use '!' i.e. '!12' repeats the 12th command in the history.

I wanted to use the log as my 'these are the actual commands I typed in over this period of time.
So I had to figure out how to add time stamps to the history log
  • export HISTTIMEFORMAT='%F %T    '

I looked for simple way to extract items from the history log to a text file, but couldn't find anything in time so I eventually settled on a simple redirect e.g. redirect the output from the history command to a text file (in dropbox of course to automatically sync it)
  • history > ~/Dropbox/shared/task20140922_1125_installAndStartMultiTail.txt

And since I did not add the HISTTIMEFORMAT to anything global I tagged it on the front of my history command
  • export HISTTIMEFORMAT='%F %T    '; history > ~/Dropbox/shared/task20140922_1125_installAndStartMultiTail.txt

I added 'comments' to the history using the 'echo' command, which displays the text on screen and appears in the history as an obviously non-testing command.

I also found with history:
  • add a ' ' space before the command and it doesn't appear in the history - great for 'non core' actions
  • you can delete 'oops' actions from the history with 'history -d #number'

And that was my basic test environment setup. If I tracked this as session based testing then this would have been a test environment setup session.

All my notes are in the full testing notes I made.

I could have gone further:
  • ideally I wanted to automatically log the output of commands as well as the command input
  • I could have figured out how to make the HISTTIMEFORMAT stick
  • I could have figured out how to have the dropbox daemon run automatically rather than running in a command window
  • I could have found a tool to automatically refresh a view of the full file (i.e. as James did with textmate)
  • etc.

But I had done enough to meet my basic needs for Observation, Manipulation and Reporting.
Oh, and I decided on a Google doc for my test notes very early on, because, again, that would be real time accessible from all the machines I was working from. I exported a pdf of the docs to the gihub repo.

And what about the testing?

Yeah, I did some testing as well. You can see the notes I made for the testing in the github repo.

So how did the tools help?


As I was testing, I was able to observe the system as I entered commands.

So I 'task add' a new task and I could see an entry added to the pending.data, and the history.data, and the undo.data.

As I explored, I was able to see 'strange' data in the undo.data file that I would have missed had I not had visibility into the system.

You are able to view the data if you want, because I backed up up as I went along, and it automatically saved to dropbox, which I later committed to github.

What else?

At the end of the day, I spent a little time on the additional tooling tasks I had noted earlier.

I looked at the 'script' command which allows you to record your input and output to a file, but since it had all the display esc characters in it, it wasn't particularly useful for reviewing.

Then I thought I'd see what alternative terminal tools might be useful.

I initially thought of Terminator, since I've used it before. And when I tried it, I noticed the logger plug-in, which would have been perfect, had I had that at the start of my testing since the output is human readable and captures the command input and the system output as you test. You can see an example of the output in the github repo.

I would want to change the bash prompt so that it had the date time in it though so the log was more self documenting.

You can see the change in my environment in the two screenshots I captured.

  • The initial setup was just a bunch of terminals 
  • The final setup has Terminator for the execution, with gedit for viewing the changed files, and multitail on the right showing me the changes in the system after each command

Principles

And as I look over the preceding information I can pull out some principles that underpin what I do.
  • Good enough tooling
    • Ideally I'd love a fully featured environment and great tools, but so long as my basic needs are met, I can spend time on testing. There is no point building a great environment and not testing. But similarly, if I don' t have the tools, I can't see and do certain things that I need in my testing.
  • Learn to use the default built in tools
    • Gedit, Bash, History - none of that was new. By using the defaults I have knowledge I can transfer between environments. And can try and use the tools in more creative ways e.g. 'echo' statements in the history
  • Automated Logging doesn't trump note-taking
    • Even if I had Terminator at the start of my testing, and a full log. I'd still take notes in my exploratory log. I need to write down my questions, conclusions, thoughts, research links, etc. etc.
    • If I only had one, I'd take my exploratory log above any automated log taking tool.
  • Write the log to be read
    • When you read the exploratory log, I think it is fairly readable (you might disagree). But I did not go back and edit the content. That is all first draft writing.
      • What do I add in the edit? I answer any questions that I spot that I asked and answered. I change TODOs that I did. etc. I try not to change the substance of the text, even if my 'conclusions' as to how something are working are wrong at the start, but change at the end, I leave it as is, so I have a log of my changing model.
You might identify other principles in this text. If you do, please leave a comment and let me know.

Why didn't you use tool X?

If you have particular tools that you prefer to use when testing CLI applications then please leave a comment so I can learn from your experience.


Wednesday, 10 September 2014

Using Wireshark to observe Mobile HTTP Network traffic

You can find wireshark on line - it is a free tool.

https://www.wireshark.org/

Note that you may not be able to capture the mobile traffic on Windows because of WinPCap limitations. You may need to buy an additional adapter to do this. I'm using Mac to show you this functionality.
http://wiki.wireshark.org/CaptureSetup/WLAN#windows

What is it?

Wireshark is a tool for monitoring network traffic. Unlike an HTTP proxy server where you have to configure your machine to point to the HTTP proxy server in order to monitor the traffic. With Wireshark, you tell it to capture traffic from your network card, and it can then capture any traffic going through that network.
So if your mobile device is on the same wifi network as your Wireshark machine's wifi card. Then you can capture the wifi traffic, filter it, and then monitor the HTTP traffic from your mobile device.

Why would I want to do that?

Because sometimes the mobile app you are testing does not honour the proxy settings of the device and goes direct, so you don't see the traffic.
And because you can start learning more about the network traffic layers being used by your application and your device in general.
(It's also fun to hook into hotel wifi and airport lounge wifi - but don't tell anyone.)
But the serious point, is that we know we want to observe the http traffic. If we have the issue that we can't because we can't configure the app to point to the proxy then we need other options. We need to increase our flexibility to approaching the observation. So we have a new option - work at the network traffic level, rather than proxy.
Our aim is to keep looking for new ways of achieving our outcomes. Not finding tools, for the sake of tools. But finding new approaches.

Installing Wireshark

On Windows

The Windows install is simple. Just download and run.
https://www.wireshark.org/download.html

On Mac

Mac install was a little harder for me and it didn't work out the box so I had to do the extra steps to add the application to XQuartz
If it doesn't work then you could try, start xquartz In the Applications menu of Xquartz, customize it and "Add Item" with the command:
  • "open /Applications/Wireshark.app/Contents/MacOs/Wireshark"
  • or "open wireshark"
  • or "wireshark"
Then you could try, running wireshark from the Applications menu in XQuartz, or from the application icon directly.
You might find these links helpful if you are on a mac:

On Linux

I haven't tried the install on linux - I imagine the instructions on the Wireshark website work fine.

First Usage

Wireshark, can seem intimidating initially to work with.
It is a complicated tool and there is a lot to learn about it.

Start a Capture

On the main page, select your network card hooked to the wifi network. Then click "Capture Options".
In Capture options table. Check to see that "Mon. Mode" says enabled, for the interface you want to use. If it doesn't, you'll only see your own traffic.
To change "Mon. Mode", double click the item in the table, and choose "Capture packets in promiscuous mode" and "Capture packets in monitor mode", press [OK].
Then [Start] the capture.
If you are on an encrypted network then you might need to decrypt the traffic.
http://wiki.wireshark.org/HowToDecrypt802.11
I sometimes have to fiddle with the IEEE 802.11 preferences: changing them, hitting apply, changing them, etc. Until I see the actual http traffic.
I also have to disconnect the android device from the network, and then reconnect it, so that it sends the initial network connection and decryption packets. Feel free to test on open networks where you don't have these type of issues because they are insecure if you want to.

Filter the capture

At this point your going to start seeing a lot of traffic flowing through your network.
So you want to filter it.
in the filter text editor type "ip.addr eq 192.168.1.143" or whatever the ip of your device is, to start seeing that traffic.
then if you just want to see the http traffic you can do
"ip.addr eq 192.168.1.143 and http"
or, to see just the GET requests:
"ip.addr eq 192.168.1.143 and http.request.method eq "GET""
This is a useful tool to have in your toolbox, for those moments where you have less control over the application under test, but still want to observe the traffic in your testing.

You can see examples of Wireshark in action to help test an Android app in our Technical Web Testing 101 online course.