Make JavaScript tests part of your build: QUnit & Rhino

I know some zealous Asynchronites who don’t believe code exists unless:

• it has tests
• code and tests are in source control
• tests are running on the continuous integration (CI) build

Sound crazy?  If so, that is my kind of crazy.

Lately I have been hacking around a lot with JavaScript (HTML5, NodeJS, etc…) and was asked by another team if I had any suggestions on testing JavaScript and how they could integrate it into their build.  Most of my experience testing JavaScript has been done by executing user acceptance tests written with tools like Selenium and Cucumber (load a real browser and click/act like a real user).  Unfortunately these tests are slow and brittle compared to our unit tests.  More and more of today’s dynamic web apps have large amounts of business logic client side that doesn’t have a direct dependency on the browser.  What I want are FAST tests that I can run with every single change, before every single commit, and headless on our build server.  They might not catch everything, but they will catch a lot long before the UATs are finished.

Goal : Run JavaScript unit tests as part of our automated build

The first hurdle was finding a way to run the code headless outside of a browser.  Several of today’s embedded JavaScript interpreters are available as separate projects but for simplicity I like Rhino.  Rhino is an interpreter written entirely in Java and maintained by the Mozilla Foundation.  Running out of a single .jar file and with a nice GUI debugger, Rhino is a good place to start running JavaScript from the command line.  While not fully CommonJS compliant yet, Rhino offers a lot of nice native functions for loading other files, interacting with standard io, etc.  Also, you can easily embed Rhino in Java apps (it is now included in Java 6) and extended with Java code.

My JS testing framework of choice lately has been JSpec which I really like: nice GUI output, tons of assertions/matchers, async support, Rhino integration, and really nice r-spec-esque grammar.  Unfortunately JSpec works best on a Mac and the team in question needs to support Windows/*nix mainly.

Enter QUnit: simple, fast, and used to test JQuery.  Only one problem, QUnit is designed to run in a browser and gives all of its output in the form of DOM manipulations.  Hope was almost lost when I found a tweet by John Resig himself that suggested that QUnit could be made to work with a command line JavaScript interpreter.  Sadly despite many claims of this functionality, I couldn’t find a single good tutorial which showed me how nor a developer leveraging this approach.  A bit of hacking, a lucky catch while reading the Env.js tutorial, and twada’s qunit-tap project came together in this simple solution:

Step 1 : Write a test (myLibTest.js)

Create a simple test file for a even simpler library function.

test("Adding numbers works", function() {
    expect(3);
    ok(newAddition, "function exists");
    equals(4, newAddition(2, 2), "2 + 2 = 4");
    equals(100, newAddition(100, 0), "zero is zero");}
);

Step 2 : Create a test suite (suite.js)

This code could be included in the test file, but I like to keep them separate so that myLibTest.js could be included into a HTML page for running QUnit in normal browser mode without making any changes.

This file contains all of the Rhino specific commands as well as some formatting changes for QUnit.  After loading QUnit we need to a do a bit of house work to setup QUnit to run on the command line and override the log callback to print our test results out to standard out.  QUnit offers a number of callbacks which can be overridden to integrate w/ other testing tools (find them on the QUnit home page under “Integration into Browser Automation Tools”) Finally load our library and our tests. [EDIT: ADD]

load("../qunit/qunit/qunit.js");

QUnit.init();
QUnit.config.blocking = false;
QUnit.config.autorun = true;
QUnit.config.updateRate = 0;
QUnit.log = function(result, message) {
    print(result ? 'PASS' : 'FAIL', message);
};

load("myLib.js");
load("myLibTest.js");

Step 3 : Write the code (myLib.js)

function newAddition(x, y) {
    return x + y;
}

Step 4 : Run it

Running Rhino is piece of cake but I am VERY lazy so I created a couple of aliases to make things dead simple. (taken from the Env.js tutorial, jsd runs the debugger)

export RHINO_HOME="~/development/rhino1_7R2"
alias js="java -cp $RHINO_HOME/js.jar org.mozilla.javascript.tools.shell.Main -opt -1"
alias jsd="java -cp $RHINO_HOME/js.jar org.mozilla.javascript.tools.debugger.Main"

Once the alias are created, simple run the suite

> js suite.js
> PASS function exists
> PASS <span>2 + 2 = 4</span>, expected: <span>4</span>
> PASS <span>zero is zero</span>, expected: <span>100</span>
> 

Step 5 : Profit!

There you go, command line output of your JavaScript unit tests.  Now we can test our “pure” JavaScript which doesn’t rely on the DOM.  Using a tool like Env.js this is also possible and will be discussed in a future post. [EDIT: ADD]

Step 5+ : Notice the problem -> HTML output on the command line

You may have noticed that my output messages include HTML markup.  Sadly QUnit still has assumptions that it is running in a browser and/or reporting to an HTML file.  Over the next couple weeks I am going to work on refactoring out the output formatting code from the core unit testing logic and hopefully build out separate HTML, human readable command line, and Ant/JUnit XML output formatters so that integrating QUnit into your build process and Ant tasks is a piece of cake.

Track my progress on my GitHub fork and on the main QUnit project issue report.

[EDIT] I have posted about some of my progress and setbacks here.