A Selenium CaptureNetworkTraffic Example in Java

When I learned about Selenium’s ability to capture network traffic I was really excited. This opened up a whole new world of testing possibilities. I could capture the Ajax requests made to the server and check them for validity. When I visit pages I can check the web analytics messages sent back and check their correctness. So this post explains a little about how to use CaptureNetworkTraffic.


I found two blog posts useful when learning about capturenetworktraffic:

• Testing Nexus with Selenium: A lesson in complex UI testing (Part 4) by Brian Fox
• Automated Web/HTTP Profiler with Selenium-RC and Python by Corey Goldberg

Corey has an open source tool which uses Selenium to provide some profiling stats for web visits.

---

If you would like to learn Selenium but have trouble following the examples here, then you might want to have a look at my book "Selenium Simplified" which provides a tutorial approach to learning Selenium using Java - no programming experience required.

---

A few months ago, I was looking for a proxy server that I could automate alongside my Selenium tests. I did a lot of web searches but found nothing suitable. And it never occurred to me that Selenium had this functionality out of the box until I found Corey’s post.
Basically, you start a Selenium session with:

• selenium.start("captureNetworkTraffic=true");

And then, after a few requests you can get the dump of the html traffic as a string by issuing:

• String trafficOutput = selenium.captureNetworkTraffic("json");

Simple. (you can use “xml”, “json” or “plain”)
I use the json format because when I returned it in xml some of the urls that returned were the wrong format. So in Java this means I use the gson library to parse json.

And when I run this conversion of Corey’s profiler I see the following output from visiting the EvilTester.com homepage:
Warning you may get a concurrent modification exception reported when running the test. If this happens run it again. On some machines this happens more often than others and might seriously impact your ability to use this great function :( http://jira.openqa.org/browse/SEL-713

--------------------------------
results for http://www.eviltester.com
content size: 120617 kb
http requests: 22
status 200: 21
status 403: 1

file extensions: (count, size)
jpg: 4, 9.365000
png: 2, 1.969000
js: 4, 30.733000
ico: 1, 1.244000
unknown: 8, 53.546000
gif: 2, 16.319000
css: 1, 7.441000

http timing detail: (status, method, url, size(bytes), time(ms))
200, HEAD, http://www.eviltester.com/, 0, 406
200, GET, http://www.eviltester.com/, 30052, 641
...

To learn how to use this functionality you could read the source code to Corey’s tool, in Python. Or if you prefer Java you can follow the source below. I have done a quick, partial conversion of Corey’s tool in Java to illustrate how to use Gson and captureNetworkTraffic.


You can download the full source-code for this here.


Prerequisites to using this source-code:

• You need to download Google-gson and add this lib to your project.



• Also, since I have illustrated the basic principles in a @Test you will need to add either JUnit 4 or TestNG.



• You also need to add selenium-server.jar to your project since we start Selenium automatically in this example

When you issue the selenium.captureNetworkTraffic(“json”) you receive a json string which has a collection of HTML Response Messages, e.g.

 [{
     statusCode: 200,
     method: 'HEAD',
     url: 'http://www.eviltester.com/',
     bytes: 0,
     start: '2010-05-26T13:33:37.048+0100',
     end: '2010-05-26T13:33:37.314+0100',
     timeInMillis: 266,
     requestHeaders:[{
         name: 'Host',
         value: 'www.eviltester.com'
     },{..}],
    ...
  }]

When I parse this with gson

 Gson gson = new Gson();
  
 Type collectionOfHTMLRequestsType = 
              new TypeToken<collection><htmlrequestfromselenium>>(){}.getType();
 Collection<htmlrequestfromselenium> seleniumRequests = 
              gson.fromJson(trafficOutput, collectionOfHTMLRequestsType);

Gson will automatically build a collection of objects for me.


I created two objects to help with my Gson parsing:


HTMLRequestFromSelenium.java

 package com.eviltester.captureNetworkTraffic;

 import java.util.List;

 public class HTMLRequestFromSelenium {

  public int statusCode;
  public String method;
  public String url;
  public int bytes;
  public String start;
  public String end;
  public int timeInMillis;
  public List requestHeaders;
 }

and ValuePair.java

 package com.eviltester.captureNetworkTraffic;

 public class ValuePair {

  private String name;
  private String value;
 }

Gson is well covered by the following articles

• http://sites.google.com/site/gson/gson-user-guide



• http://albertattard.blogspot.com/2009/06/practical-example-of-gson.html



• http://www.softwarepassion.com/android-series-parsing-json-data-with-gson/

So this is my SeleniumTrafficAnalyserExampleTest.java where the bulk of the work is done. Commented to explain the basics.

package com.eviltester.captureNetworkTraffic;

import static org.junit.Assert.*;
import java.lang.reflect.Type;
import java.util.ArrayList;
import java.util.Collection;
import java.util.HashMap;
import java.util.Iterator;
import java.util.Set;
import org.junit.Test;
import org.openqa.selenium.server.SeleniumServer;
import com.google.gson.Gson;
import com.google.gson.Gson.*;
import com.google.gson.reflect.*;
import com.thoughtworks.selenium.DefaultSelenium;

public class SeleniumTrafficAnalyserExampleTest {
 
 @Test
 public void testProfileEvilTester() throws Exception{
  
  // Start the Selenium Server
  SeleniumServer srvr = new SeleniumServer();
  srvr.start();
   
  // Create a Selenium Session with captureNetworkTraffic ready
  String site = "http://www.eviltester.com";
  
  DefaultSelenium selenium = new DefaultSelenium("localhost", 4444, "*firefox", site);
  selenium.start("captureNetworkTraffic=true");
  
  // open a page to get the traffic
  selenium.open("/");
   
  // dump the traffic into a variable in Json format
  String trafficOutput = selenium.captureNetworkTraffic("json");
  System.out.println(trafficOutput);

  // parse the json using Gson
  Gson gson = new Gson();
  Type collectionOfHTMLRequestsType = 
   new TypeToken<Collection<HTMLRequestFromSelenium>>(){}.getType();
  Collection<HTMLRequestFromSelenium> seleniumRequests = 
   gson.fromJson(trafficOutput, collectionOfHTMLRequestsType);
  
  // get ready to analyse the traffic
  TrafficAnalyser ta = new TrafficAnalyser(seleniumRequests);
  
  // this is pretty much copied from Corey's python example
  int num_requests = ta.get_num_requests(); 
  int total_size = ta.get_content_size(); 
  HashMap<Integer,Integer> status_map = ta.get_http_status_codes(); 
  HashMap<String, Object[]> file_extension_map = ta.get_file_extension_stats(); 
          
       System.out.println("\n\n--------------------------------"); 
       System.out.println(String.format("results for %s",site)); 
       System.out.println(String.format("content size: %d kb",total_size)); 
       System.out.println(String.format("http requests: %d",num_requests));
       
       Iterator<Integer> statusIterator = status_map.keySet().iterator() ; 
       while ( statusIterator.hasNext (  )  )  
        {  
        int key = statusIterator.next();
        System.out.println(String.format("status %d: %d", key, status_map.get(key)));
        } 
        
       System.out.println("\nfile extensions: (count, size)"); 
       Iterator<String> extensionIterator = file_extension_map.keySet().iterator() ; 
       while ( extensionIterator.hasNext (  )  )  
        {  
        String key = extensionIterator.next();
        System.out.println(String.format("%s: %d, %f", key,
     file_extension_map.get(key)[0],file_extension_map.get(key)[1]));
        } 
                     
      System.out.println("\nhttp timing detail: (status, method, url, size(bytes), time(ms))");
   for (Iterator iterator = seleniumRequests.iterator(); iterator.hasNext();) {
     HTMLRequestFromSelenium hr = (HTMLRequestFromSelenium) iterator.next();
     //totalContentSize += hr.bytes;
     System.out.println(String.format("%d, %s, %s, %d, %d",
      hr.statusCode, hr.method, hr.url, hr.bytes, hr.timeInMillis));
   }
       
  // close everything down
  selenium.close();
  selenium.stop();
  srvr.stop();
 }

}

Then the helper class TrafficAnalyser.java

package com.eviltester.captureNetworkTraffic;
import java.net.MalformedURLException;
import java.net.URL;
import java.util.Collection;
import java.util.HashMap;
import java.util.Iterator;

public class TrafficAnalyser {

 private Collection<HTMLRequestFromSelenium> seleniumRequests;
 
 public TrafficAnalyser(Collection<HTMLRequestFromSelenium> seleniumRequests) {
  this.seleniumRequests = seleniumRequests;
 }

 public int get_num_requests() {
  return seleniumRequests.size();
 }

 public int get_content_size() {
  int totalContentSize = 0;
  
  for (Iterator iterator = seleniumRequests.iterator(); iterator.hasNext();) {
   HTMLRequestFromSelenium hr = (HTMLRequestFromSelenium) iterator.next();
   totalContentSize += hr.bytes;
  }
  
  return totalContentSize;
 }

 public HashMap<Integer,Integer> get_http_status_codes() {
  HashMap<Integer,Integer> statusCodes = new HashMap<Integer,Integer>();
  
  for (Iterator iterator = seleniumRequests.iterator(); iterator.hasNext();) {
   HTMLRequestFromSelenium hr = (HTMLRequestFromSelenium) iterator.next();
   if(statusCodes.containsKey(hr.statusCode)){
    statusCodes.put(hr.statusCode, statusCodes.get(hr.statusCode)+1);
   }else{
    statusCodes.put(hr.statusCode, 1);
   }
  }
  
  return statusCodes;
 }

 public HashMap get_file_extension_stats() {
  HashMap<String,Object[]> extensions = new HashMap<String,Object[]>();
  
  for (Iterator iterator = seleniumRequests.iterator(); iterator.hasNext();) {
   HTMLRequestFromSelenium hr = (HTMLRequestFromSelenium) iterator.next();
   URL url = null;
   try {
    url = new URL(hr.url);
    String file_extension;
    
    double size = hr.bytes/1000.0;
    
    file_extension="";
    String doc = url.getPath();
    if(doc.contains("."))
     file_extension = doc.substring(doc.indexOf(".")+1).trim();

    if(file_extension.compareTo("")==0)
     file_extension = "unknown";
    
    if(extensions.containsKey(file_extension)){
     Object[] stats = extensions.get(file_extension);
     stats[0] = (Integer)stats[0] +1;
     stats[1] = (Double)stats[1] + size;
     extensions.put(file_extension, stats);
    }else{
     Object[] stats = new Object[2];
     stats[0] = 1;
     stats[1] = size;
     extensions.put(file_extension, stats);
    }
    
   } catch (MalformedURLException e) {
   }
  }
  
  return extensions;
 }

}

Hopefully this gives  a small overview of the capturenetworktraffic functionality in Selenium.





Related Reading:

• http://coreygoldberg.blogspot.com/2009/10/automated-webhttp-profiler-with.html



• http://www.sonatype.com/people/2009/10/selenium-part-4/



• http://selenium-profiler.googlecode.com/



• http://code.google.com/p/selenium-profiler/source/browse/trunk/web_profiler.py



• http://stackoverflow.com/questions/2354827/checking-http-status-code-in-selenium



• http://testautomationblog.com/tag/capture-network-traffic/



• http://groups.google.com/group/selenium-users/browse_thread/thread/85208a5db7609722/8e95d5ec813d4844?lnk=gst&q=captureNetworkTraffic#8e95d5ec813d4844

---

If you would like to learn Selenium but have trouble following the examples here, then you might want to have a look at my book "Selenium Simplified" which provides a tutorial approach to learning Selenium using Java - no programming experience required.

---