Thursday, 4 March 2010

Selenium as a performance and load test tool with BrowserMob

I recently had the good fortune at work to have tested an Amazon cloud hosted application. And this led to the test team having to investigate alternative performance test tools. We settled on BrowserMob and we have loved it.

I wish BrowserMob had an affiliate program, then I might get some cash if I send you off to their site. But they don’t, and I’m only writing this because I think they have a great tool and want to draw your attention to it.



We have an inhouse performance test tool. But due to our infrastructure setup, we can't use that tool to test applications outside of our test environment.

Cloud applications obviously do not get hosted in our environment so I we needed a cloud based performance test tool.

I did a bunch of web searching and reading around to try and identify options. I set some constraints:

  • I wanted something low-cost, to reduce our initial investment on this new approach - so a low cost subscription-based/pay-as-you need model seemed most suitable.
  • I needed something easy to use. Since we don't do performance testing day in and day out, I wanted something that I wouldn't forget between test runs.
  • I wanted something hosted. I didn't want to spend a lot of time setting up environments and installing software as this would increase cost and add additional risk into our evaluation and use of the tool. I rejected setting up a set of Amazon servers and installing JMeter or The Grinder on them for this reason.

I settled on BrowserMob (blog)

A company setup by Patrick Lightbody (a name you might recognise from his involvement with Selenium).

It has a very clear pricing model. In fact it has two. You can use the PAYG model (the old pricing model) and the new subscription model.

If the subscription model works out for you then it should prove incredibly cheap. If you do run a few load tests and can get them in with your allowance then you’re laughing. The first project we did on the old pricing model, we only spent $600 with BrowserMob for our entire project. Now, with the new pricing model I could have done all the testing on their free subscription model and spent about $30.

It has a great evaluation model, you sign up for the free plan and start testing. Allowing you to conduct load tests (with a limit on the number of users). This allows you to evaluate the tool under 'real' load test conditions. We did our first load test entirely within a free trial, and we received a tremendous level of support during our evaluation and this has continued throughout our use of the tool – I have never encountered support as good as this before.

If you start looking for a performance test tool for testing externally accessible web sites - check this tool out.

As for the technologies. Because it uses Amazon hosting, you can choose the geographical location to run the tests from. And you script the tests using JavaScript – which makes scripting tremendously flexible. We tend to start by creating a Selenium IDE script, adding that to BrowserMob, automatically converting it to a JavaScript test. Verifying the conversion with Fiddler, amending the script to make it a bit more data driven and add some waits and other behavioural elements. I found it really fast to write scripts.

Because it uses Selenium it can do performance tests with real browsers. But BrowserMob also uses an HTMLUnit to allow you to run the tests as virtual users (which works out cheaper) (guess which we used).

The simple reports proved and easy to read, and you get access to a MySQL database of results to do your own queries on and investigate the results in more detail.

I think every tester should try the free model and dabble in some performance testing on your own sites. A great tool, heavily recommend it, we use it for all of our external web site performance testing and it does the job quickly, easily and incredibly cost effectively.

Because of the way we work, we use the PAYG model, charge up the account with as many ‘dollars’ as we think we will need (the tool estimates how much you will need when you put your load profile together for scheduling, so if you need to charge up your account you can quickly do that with the online payment system). The PAYG model costs slightly more per test than the subscription model but we need the flexibility that the PAYG model offers and BrowserMob still works out cheaper than the competitors I looked at.

4 comments:

  1. [...] Selenium as a performance and load test tool with BrowserMob – June 11th %(postalicious-tags)( tags: selenium load performance test tool browsermob testing web )% June 12th, 2010, @ 12:00 am | Tags: links | Category: delicious links | Comments are closed | Trackback this Post | 0 views [...]

    ReplyDelete
  2. We did our first load test entirely within a free trial, and we received a tremendous level of support during our evaluation and this has continued throughout our use of the tool – I have never encountered support as good as this before.

    -- well this said me that this tool is no no. if you need support tool is ****.

    ReplyDelete
  3. Hi, thanks for leaving a humorous anonymous comment.

    I've never seen the argument that "good support" == "bad tool" before.

    I'd be interested in seeing the results of your evaluation based on more than a "Good support" argument.

    Oh by the way. You know that all the post submissions have IP addresses? And that when you post from a machine in your company network I can do a dns backtrace and see the test consultancy that you work for?

    I'll keep an eye on the white papers section of your employer's web site and see if such an evaluation appears.

    Thanks for entertaining the EvilTester.com readership.

    Alan

    ReplyDelete
  4. Browser mob is now Neustar Web Performance Management

    ReplyDelete