Performance-Monitoring using Watir, Meteor and Vagrant

EDIT: I was facing an issue with older PCs with turned off hardware virtualization. The testbox didn’t startup properly bewause it wasn’t able to configure the networking while booting through ‘vagrant up’, while starting the box through the VirtualBox GUI just worked fine. After turning the hardware virtualization on in the host’s BIOS the issue vanished.


We are using Planview company wide for project and portfolio management and heard more and more complains about the bad performance of the system. Especially project managers argued, that they can’t possibly work with the system because of it’s slowness. So we decided to setup a performance-monitoring system. The main requirements were:

  • We need test-satellites on the machines of the complaining project managers to objectify their observations.
  • Those test-satellites should be ready to run with minimal effort.
  • The system should continuously run without taking extra steps to actually run the performance tests.
  • The test results should be summarized in a dashboard-like overview.
  • The development effort of the test as well as the whole system should be minimal.


So I made a quick analysis of the different options available and decided to go with:


We are acting in a trusted private network and are not dealing with classified or protection-worthy data. So everything is kept simple and open and absolutely insecure.

Note: If you are behind a corporate proxy server you might need to set things up accordingly. In case it is an NTLM proxy it’s advisable to run cntlm, as an authenticating proxy, somewhere.

Setting up MongoDB

I used an old PC as the MongoDB host. I decided to go with the Minimum CD of Ubuntu 14.04 LTS as the OS. The PC needs a working internet connection to perform this type of install (remember the proxy). I installed only the bare minimum.

When dealing with MongoDB it is generally a good idea to stick to the LTS versions of Ubuntu, since they are directly supported by MongoDB. I used the default guide to install MongoDB. And everything went pretty smoothly.

After installing MongoDB through ‘apt-get’, I changed the following aspects in /etc/mongod.conf:

#this let's mongod listen on the outbound interface 
#usually this is simply the ip address of the eth0 interface

  oplogSizeMB: 2048
  replSetName: rs
#this starts the mongod deamon as part of a replication 
#set and is required to let Meteor use oplog tailing for 
#synchronizing with the database

After a first connection attempt using the CLI tool ‘mongo’ I received an error stating that ‘transparent_hugepage’ is enable and should be disabled. A quick web-search brought quite some different solutions and I decided to modify ‘/etc/rc.local’ and added the following lines before ‘exit 0’:

/bin/echo never > /sys/kernel/mm/transparent_hugepage/enabled
/bin/echo never > /sys/kernel/mm/transparent_hugepage/defrag

After a restart I connected with the database and the message was gone. The next step was initiating the replication set. In the Mongo console enter:

conf = {_id: 'rs', members: [{_id: 0, host: 'pmomongo:27017'}]}

Where ‘pmomongo’ is the hostname of the PC that hosts the MongoDB.

Creating a Vagrant Base Box

The plan was to provide a Vagrant box that can be started on any machine and then runs the tests on a regular basis and reports the results to the MongoDB.

To do this you need to install VirtualBox and Vagrant (on Windows you need additionally some Unix tools, a simple way is to install Git, that delivers everything that’s needed).

First you need to create a VirtualBox box, that later serves as the blueprint as a Vagrant base box. I created a Ubuntu box with minimal resources: 1 CPU, 512MB RAM, and a 4GB hard drive. To install the box, I used the same minimum CD image of Ubuntu I used for the MongoDB host. I consulted the official Vagrant guides (this and this) to achieve a basic box.

But before finally packaging the box as a Vagrant base box we need to install the tests and the tools required to run those tests.

The Tests

Installing Watir and Other Gems

Watir-Webdriver is a ruby tool. So you need to install ruby. The common ‘apt-get’ version is sufficient:

sudo apt-get install ruby

Afterwards install the required gems:

sudo gem install watir-webdriver phantomjs bson mongo

Installing Git

To synchronize the tests you need to install git.

sudo apt-get install git

Developing the tests

Fig. 1) Test structure

To be able to use the development tools you like and getting the tests behave predictably I strongly recommend developing the tests on your “normal” PC and not on the testbox. This way you also avoid unnecessary cluttering and blowing up the test-satellites. Of course you need the a fore mentioned tools on your PC as well. Besides using phantomjs for headless tests you might want to install chromedriver (version 2.20 worked for me, others didn’t), so that you can see how the test actually interacts with the page under test.

Later we will utilize git to synchronize the test sources between your developer machine and the test-satellites.

I structured the tests as depicted in Figure 1. On the projects root level exists the ‘runner.rb’ script that executes all the tests that are defined in the ‘tests’ folder. Inside the ‘tests’ folder are some helper scripts that encapsulate shared functionality.

I hope the code is quite self explanatory. I have added some comments were I felt them necessary.

The ‘browser.rb’ script handles the creation of browser windows. Either a headless browser with phantomjs or a normal chrome window (commented out in the gist). Planview obviously uses the ‘userAgent’ information to alter it’s functionality and appearance, which caused some tests to behave unpredictably. So I had to modify phantomjs’ capabilities, so that for the target page the client looks like a regular chrome browser.

The following gist shows a sample test. It’s quite straight forward, I guess. I think the only aspect noteworthy is the user name and password thing in the URL. Our Planview users are authenticated via AD and NTLM so I have to provide valid credentials for a known domain user.

This last gist for the tests shows the “reporter.rb” script, which sends the results to the Mongo database. I opted for ‘SecureRandom’ to generate IDs since Meteor doesn’t seem to work out-of-the-box with the regular Mongo ID mechanism.

Back to our TestBox

After developing the tests, at least an initial version, and uploading them to your favorite git repository, you clone that repository on the VirtualBox box.

git clone[USERNAME]/planview-tests.git

Note: refer to your git repository provider of choice on how to authenticate. It most certainly will involve creating ssh keys and linking the public key to your user account.


I setup cron to run the tests regularly. I have decided for five minute intervals. Additionally I added a second line to cron to pull the source code repository for changes to the test suite (every thirty minutes). This way I avoid coupling the performance test-satellites to the dashboard. And can spin up new ones, or tear down some if they are not needed any more. To modify your cron execute:

crontab -e

And after selecting you favorite editor add the following two lines:

*/5 * * * * /usr/bin/ruby ~/planview-tests/runner.rb
*/30 * * * * cd ~/planview-tests/ && git pull -q origin master

In case of an error cron tries to send local emails. To receive those error messages we need to install an MTA, easiest way is ‘postfix’. While installing, configure it to handle only local messages. To read those messages you might want to install ‘mutt’.

sudo apt-get install postfix mutt

Create a Base Box

That’s basically everything that’s needed inside the test box. So let’s create a vagrant base box from it. Shutdown the VirtualBox box, open a terminal and execute:

vagrant package --base [name of the testbox in VirtualBox]

This will create a ‘’ base box in the folder where you executed the command.

Making The Base Box Accessible

Later we want to provide a simple ‘Vagrantfile’ (a configuration file describing the box to start) and let the user just run:

vagrant up

to get everything up and running. So we have to make the base box that we just created accessible and addressable in the Vagrantfile. For this purpose I installed NGINX on another PC and configured it to serve the ‘’ file.

sudo apt-get install nginx

On this machine it created the ‘/var/www/html’ folder and put a single html file there. Which was configured as the default page in ‘/etc/nginx/sites-available/default/’. I simply added the following configuration:

    location ~ \.box$ {
        add_header Content-Type application/octet-stream;

in the main ‘server’ block and copied the file from my PC to ‘/var/www/html/vagrant/’ on the designated web server (creating the folder previously).

The corresponding Vagrantfile looks like this (I have deleted the usual comments):

I’m using the hostname of the test-host as part of the Vagrant box hostname, this way I avoid clashing names in our local network. The ‘config.vm.box_url’ should point to the webserver we just have setup.

Giving someone this Vagrantfile and installing the required tools: VirtualBox, Vagrant and Git will allow anyone to easily start a test-satellite.

Presenting the Results with Meteor and NVD3

First you need to install Meteor. Then open a terminal and execute:

meteor create plaperf

If you don’t like the name ‘plaperf’ you might want to choose a more appealing name for your Meteor project. This command creates basically a very rudimentary but functional one page Meteor app.

Go into the ‘plaperf’ directory and add the required tools. I installed coffeescript since I like the syntax better than vanilla JS. But you need additionally D3 and NVD3 to display nice graphs of the measured performance, moment.js for time formatting, lodash as the better underscore and server-autorun to make it possible for Meteor to react to changes in the MongoDB structure on the server side.

meteor add coffeescript d3js:d3 nvd3:nvd3 momentjs:moment stevezhu:lodash peerlibrary:server-autorun

You might want to remove the ‘autopublish’ package since we are doing target pub/sub to avoid loading all the data to the client, which might become quickly a lot.

meteor remove autopublish

After that you write your performance dashboard. The resulting structure looks like this.

Structure of Performance Dashboard

And here are the gists for the project’s files to follow along. Starting with the startup script, going over to the server code and finally showing the client code which is responsible for the graphs:

For completeness sake here is the CSS file:

Fig 2) Column Set Switching Measures



Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.