Two Guys Arguing

Make JavaScript tests part of your build: QUnit & Rhino

Posted in java, javascript, software testing by benjaminplee on 11.02.10

I know some zealous Asynchronites who don’t believe code exists unless:

  • it has tests
  • code and tests are in source control
  • tests are running on the continuous integration (CI) build

Sound crazy?  If so, that is my kind of crazy.


Lately I have been hacking around a lot with JavaScript (HTML5, NodeJS, etc…) and was asked by another team if I had any suggestions on testing JavaScript and how they could integrate it into their build.  Most of my experience testing JavaScript has been done by executing user acceptance tests written with tools like Selenium and Cucumber (load a real browser and click/act like a real user).  Unfortunately these tests are slow and brittle compared to our unit tests.  More and more of today’s dynamic web apps have large amounts of business logic client side that doesn’t have a direct dependency on the browser.  What I want are FAST tests that I can run with every single change, before every single commit, and headless on our build server.  They might not catch everything, but they will catch a lot long before the UATs are finished.

Goal : Run JavaScript unit tests as part of our automated build

The first hurdle was finding a way to run the code headless outside of a browser.  Several of today’s embedded JavaScript interpreters are available as separate projects but for simplicity I like Rhino.  Rhino is an interpreter written entirely in Java and maintained by the Mozilla Foundation.  Running out of a single .jar file and with a nice GUI debugger, Rhino is a good place to start running JavaScript from the command line.  While not fully CommonJS compliant yet, Rhino offers a lot of nice native functions for loading other files, interacting with standard io, etc.  Also, you can easily embed Rhino in Java apps (it is now included in Java 6) and extended with Java code.

My JS testing framework of choice lately has been JSpec which I really like: nice GUI output, tons of assertions/matchers, async support, Rhino integration, and really nice r-spec-esque grammar.  Unfortunately JSpec works best on a Mac and the team in question needs to support Windows/*nix mainly.

Enter QUnit: simple, fast, and used to test JQuery.  Only one problem, QUnit is designed to run in a browser and gives all of its output in the form of DOM manipulations.  Hope was almost lost when I found a tweet by John Resig himself that suggested that QUnit could be made to work with a command line JavaScript interpreter.  Sadly despite many claims of this functionality, I couldn’t find a single good tutorial which showed me how nor a developer leveraging this approach.  A bit of hacking, a lucky catch while reading the Env.js tutorial, and twada’s qunit-tap project came together in this simple solution:

Step 1 : Write a test (myLibTest.js)

Create a simple test file for a even simpler library function.

test("Adding numbers works", function() {
    expect(3);
    ok(newAddition, "function exists");
    equals(4, newAddition(2, 2), "2 + 2 = 4");
    equals(100, newAddition(100, 0), "zero is zero");}
);

Step 2 : Create a test suite (suite.js)

This code could be included in the test file, but I like to keep them separate so that myLibTest.js could be included into a HTML page for running QUnit in normal browser mode without making any changes.

This file contains all of the Rhino specific commands as well as some formatting changes for QUnit.  After loading QUnit we need to a do a bit of house work to setup QUnit to run on the command line and override the log callback to print our test results out to standard out.  QUnit offers a number of callbacks which can be overridden to integrate w/ other testing tools (find them on the QUnit home page under “Integration into Browser Automation Tools”) Finally load our library and our tests. [EDIT: ADD]

load("../qunit/qunit/qunit.js");

QUnit.init();
QUnit.config.blocking = false;
QUnit.config.autorun = true;
QUnit.config.updateRate = 0;
QUnit.log = function(result, message) {
    print(result ? 'PASS' : 'FAIL', message);
};

load("myLib.js");
load("myLibTest.js");

Step 3 : Write the code (myLib.js)

function newAddition(x, y) {
    return x + y;
}

Step 4 : Run it

Running Rhino is piece of cake but I am VERY lazy so I created a couple of aliases to make things dead simple. (taken from the Env.js tutorial, jsd runs the debugger)

export RHINO_HOME="~/development/rhino1_7R2"
alias js="java -cp $RHINO_HOME/js.jar org.mozilla.javascript.tools.shell.Main -opt -1"
alias jsd="java -cp $RHINO_HOME/js.jar org.mozilla.javascript.tools.debugger.Main"

Once the alias are created, simple run the suite

> js suite.js
> PASS function exists
> PASS <span>2 + 2 = 4</span>, expected: <span>4</span>
> PASS <span>zero is zero</span>, expected: <span>100</span>
> 

Step 5 : Profit!

There you go, command line output of your JavaScript unit tests.  Now we can test our “pure” JavaScript which doesn’t rely on the DOM.  Using a tool like Env.js this is also possible and will be discussed in a future post. [EDIT: ADD]

Step 5+ : Notice the problem -> HTML output on the command line

You may have noticed that my output messages include HTML markup.  Sadly QUnit still has assumptions that it is running in a browser and/or reporting to an HTML file.  Over the next couple weeks I am going to work on refactoring out the output formatting code from the core unit testing logic and hopefully build out separate HTML, human readable command line, and Ant/JUnit XML output formatters so that integrating QUnit into your build process and Ant tasks is a piece of cake.

Track my progress on my GitHub fork and on the main QUnit project issue report.

[EDIT] I have posted about some of my progress and setbacks here.

About these ads
Tagged with: , , ,

25 Responses

Subscribe to comments with RSS.

  1. youngnh said, on 11.02.10 at 9:05 am

    I have so many questions because not having to open a browser to test client-side Javascript is a _huge_ boon to teams writing tests and expecting timely feedback.

    Last time I played around with QUnit, I had to write a webpage and include my tests via tags, I don’t see any HTML above? Is that what qunit-tap is doing for you?

    How does env.js play into this? Last time I looked, it mocked out a browser by parsing your HTML and constructing a global DOM object. I’ve had remarkable trouble parsing real-world HTML. (Webkit and Mozilla’s DOM parsers are open-source, but hairy to integrate into a Java project)

    Does your team use this only for dealing with pure Javascript, i.e. only Javascript code that is free of DOM interactions? From the dark ages of jsunit days, a dummy page to manipulate was always a core piece of client-side Javascript tests. Does that still exist?

    Great work though, man, this is really, incredibly useful (and cool). I’m looking forward to watching it grow.

    • benjaminplee said, on 11.02.10 at 2:39 pm

      Short answer:

      QUnit is designed to work with a properly formatted web page but some changes back in 2009 greatly reduced the assumptions that it was running in a browser. With a bit more effort QUnit core could be interpreter agnostic and have a default “output-er” aimed at running in a browser.

      Nothing is a complete substitute for real testing in a browser but this will get you pretty far.

  2. Amos King said, on 11.02.10 at 11:05 am

    Jasmine?

    • benjaminplee said, on 11.02.10 at 2:35 pm

      I haven’t used it, but I have heard smart people say nice things about it. Have you?

  3. [...] Line: One Step Closer Posted in javascript, software testing by benjaminplee on 11.06.10 Previously I talked about getting QUnit JavaScript tests running on the command line using a simple Rhino [...]

  4. Chilicat said, on 11.14.10 at 11:17 am

    Check out JsChilicat. It will produce JUnit output for qunit tests and it has a Intellij IDEA integartion.

    http://jschilicat.sourceforge.net

  5. [...] 这段代码它没有任何测试案例可以寻找,另外代码的抽象程度太低,读起来很费劲,为后来的将这段逻辑移植到服务器端带来很大的不便. 关于javascript测试:Make JavaScript tests part of your build: QUnit & Rhino [...]

  6. [...]  QUnit w/ Rhino posts.  At every step of the way I thought “Oh this should be simple and [...]

  7. Mat said, on 12.01.10 at 4:59 am

    A friend of mine finally managed to solve this so my qunit tests suites (which test very complex JavaScript code) run WITHOUT A BROWSER as an ant task.

    See his code http://code.google.com/p/qunit-test-runner/

    • benjaminplee said, on 12.01.10 at 7:53 am

      Thanks for the link, that is very close to what I am aiming for.

      • Java said, on 12.01.10 at 5:54 pm

        When will the binary be made available ? This is exactly what I am looking for as well. Right on!

  8. sergey said, on 12.20.10 at 6:51 pm

    Is this intended to use with stand-along javascript files? What if my javascript file references dom or jquery that requires an html page to be loaded (the html page references my js files). Is there an example on how to do this?

  9. Nickproger said, on 12.22.10 at 5:04 am

    Cool, thanks

  10. sergey said, on 12.22.10 at 11:20 am

    It’s pretty much useless. It’s rare that you would need to test stand-alone java script functions. You will need HTML file that calls out to javascript, and all these utitilities choke when you try to load html into them. qunit-test-runner have an empty download page.

  11. Percy said, on 02.01.11 at 12:26 pm

    How do we get this to print the filename and the exact line number where the error occurred. Seems like a useful bit of info thats missing from the output.
    Thanks,
    Percy

  12. Mike Koss said, on 04.07.11 at 5:06 pm

    This is very helpful. I just wrote a small wrapper for running my QUnit tests in node.js:

    https://github.com/mckoss/namespace/blob/dev/test/test-runner-node.js

    Thanks!
    Mike

  13. [...] Wanted the tests to run in a full browser environment (or as close as possible), and not just a javascript interpreter (like Rhino) [...]

  14. [...] Make JavaScript tests part of your build: QUnit & Rhino « Two Guys Arguing Testa med QUnit direkt från kommando-promten (via @madr). [...]

  15. Josh Nankin said, on 06.01.11 at 9:51 pm

    I wrote a script that will run qunit tests with phantomjs and will output junit formatted xmls. Currently it’s a symfony plugin, but there’s no reason it can’t be converted to a standalone script: http://www.symfony-project.org/plugins/phantomQunitPlugin

  16. jakewan said, on 09.16.11 at 8:50 pm

    For some reason, my output was always:

    PASS undefined
    PASS undefined
    PASS undefined

    Using the latest version of QUnit from Github, it looks like the log callback is passed an Object with result and message attributes, as opposed to separate result and message arguments. I changed the callback to this and it works:

    QUnit.log = function(result) {
    var message = result.result ? “PASS” : “FAIL”;
    message += “: ” + result.message;
    if (result.expected && result.actual) {
    message += ” (expected ” + result.expected + “; got ” + result.actual + “)”;
    }
    print(message);
    };

  17. Anthony Brew (@atbrew) said, on 09.25.11 at 4:24 pm

    Super post guys, just started playing with Javascript and trying to write a server side library, so this is invaluable.

    I added this for a bit more logging on which test had started, and how many tests passed / failed in block, some might find it handy

    QUnit.testStart = function(test){
    print(“………………………………………………………………..”);
    print(“Started:>> ” + test.name);
    };

    QUnit.testDone = function(test){
    print(“Finished:>> ” + test.name);
    print(“Passed:” + (test.passed?test.passed:0) + ” Failed:”+ (test.bad?test.bad:0) + ” Total:” + (test.total?test.total:0));

    };

  18. smiling said, on 01.13.12 at 4:34 am

    I just started using,more precisely learning, Jasmine (only its TDD aspect) to test js via browser and EnvJasmine for testing in command line.

    • smiling said, on 01.13.12 at 4:47 am

      there’re some demonstration on youtube using jasmin-dom for testing js in command line

  19. ened said, on 01.31.13 at 8:17 pm

    hi, i need to make a test , to create a program that i can use to
    calculate the average test mark for each of the students in class. Does anybody has any idea or example
    Thank u


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.