Two Guys Arguing

Make JavaScript tests part of your build: QUnit & Rhino

Posted in java, javascript, software testing by benjaminplee on 11.02.10

I know some zealous Asynchronites who don’t believe code exists unless:

  • it has tests
  • code and tests are in source control
  • tests are running on the continuous integration (CI) build

Sound crazy?  If so, that is my kind of crazy.


Lately I have been hacking around a lot with JavaScript (HTML5, NodeJS, etc…) and was asked by another team if I had any suggestions on testing JavaScript and how they could integrate it into their build.  Most of my experience testing JavaScript has been done by executing user acceptance tests written with tools like Selenium and Cucumber (load a real browser and click/act like a real user).  Unfortunately these tests are slow and brittle compared to our unit tests.  More and more of today’s dynamic web apps have large amounts of business logic client side that doesn’t have a direct dependency on the browser.  What I want are FAST tests that I can run with every single change, before every single commit, and headless on our build server.  They might not catch everything, but they will catch a lot long before the UATs are finished.

Goal : Run JavaScript unit tests as part of our automated build

The first hurdle was finding a way to run the code headless outside of a browser.  Several of today’s embedded JavaScript interpreters are available as separate projects but for simplicity I like Rhino.  Rhino is an interpreter written entirely in Java and maintained by the Mozilla Foundation.  Running out of a single .jar file and with a nice GUI debugger, Rhino is a good place to start running JavaScript from the command line.  While not fully CommonJS compliant yet, Rhino offers a lot of nice native functions for loading other files, interacting with standard io, etc.  Also, you can easily embed Rhino in Java apps (it is now included in Java 6) and extended with Java code.

My JS testing framework of choice lately has been JSpec which I really like: nice GUI output, tons of assertions/matchers, async support, Rhino integration, and really nice r-spec-esque grammar.  Unfortunately JSpec works best on a Mac and the team in question needs to support Windows/*nix mainly.

Enter QUnit: simple, fast, and used to test JQuery.  Only one problem, QUnit is designed to run in a browser and gives all of its output in the form of DOM manipulations.  Hope was almost lost when I found a tweet by John Resig himself that suggested that QUnit could be made to work with a command line JavaScript interpreter.  Sadly despite many claims of this functionality, I couldn’t find a single good tutorial which showed me how nor a developer leveraging this approach.  A bit of hacking, a lucky catch while reading the Env.js tutorial, and twada’s qunit-tap project came together in this simple solution:

Step 1 : Write a test (myLibTest.js)

Create a simple test file for a even simpler library function.

test("Adding numbers works", function() {
    expect(3);
    ok(newAddition, "function exists");
    equals(4, newAddition(2, 2), "2 + 2 = 4");
    equals(100, newAddition(100, 0), "zero is zero");}
);

Step 2 : Create a test suite (suite.js)

This code could be included in the test file, but I like to keep them separate so that myLibTest.js could be included into a HTML page for running QUnit in normal browser mode without making any changes.

This file contains all of the Rhino specific commands as well as some formatting changes for QUnit.  After loading QUnit we need to a do a bit of house work to setup QUnit to run on the command line and override the log callback to print our test results out to standard out.  QUnit offers a number of callbacks which can be overridden to integrate w/ other testing tools (find them on the QUnit home page under “Integration into Browser Automation Tools”) Finally load our library and our tests. [EDIT: ADD]

load("../qunit/qunit/qunit.js");

QUnit.init();
QUnit.config.blocking = false;
QUnit.config.autorun = true;
QUnit.config.updateRate = 0;
QUnit.log = function(result, message) {
    print(result ? 'PASS' : 'FAIL', message);
};

load("myLib.js");
load("myLibTest.js");

Step 3 : Write the code (myLib.js)

function newAddition(x, y) {
    return x + y;
}

Step 4 : Run it

Running Rhino is piece of cake but I am VERY lazy so I created a couple of aliases to make things dead simple. (taken from the Env.js tutorial, jsd runs the debugger)

export RHINO_HOME="~/development/rhino1_7R2"
alias js="java -cp $RHINO_HOME/js.jar org.mozilla.javascript.tools.shell.Main -opt -1"
alias jsd="java -cp $RHINO_HOME/js.jar org.mozilla.javascript.tools.debugger.Main"

Once the alias are created, simple run the suite

> js suite.js
> PASS function exists
> PASS <span>2 + 2 = 4</span>, expected: <span>4</span>
> PASS <span>zero is zero</span>, expected: <span>100</span>
> 

Step 5 : Profit!

There you go, command line output of your JavaScript unit tests.  Now we can test our “pure” JavaScript which doesn’t rely on the DOM.  Using a tool like Env.js this is also possible and will be discussed in a future post. [EDIT: ADD]

Step 5+ : Notice the problem -> HTML output on the command line

You may have noticed that my output messages include HTML markup.  Sadly QUnit still has assumptions that it is running in a browser and/or reporting to an HTML file.  Over the next couple weeks I am going to work on refactoring out the output formatting code from the core unit testing logic and hopefully build out separate HTML, human readable command line, and Ant/JUnit XML output formatters so that integrating QUnit into your build process and Ant tasks is a piece of cake.

Track my progress on my GitHub fork and on the main QUnit project issue report.

[EDIT] I have posted about some of my progress and setbacks here.

Tagged with: , , ,

Test-infected Marshmallows and Being a Passionate Professional

Posted in software testing by benjaminplee on 11.05.09

Alex Miller presented a talk at his recent Strange Loop conference about Walter Mischell’s marshmallow experiments.  Mischell studies children’s ability to defer gratification when they were put in a room with a marshmallow and told they would receive another one in ~20 minutes if they didn’t eat the first.  The longer term study found that kids who were able to wait, scored much higher in “life metrics” later on (e.g. work success, school studies, etc).

Alex’s talk got me thinking about deferring gratification in software development …. which imediatley made me think about testing.

In college, there were no tests.  Just me, alone, with lots of code.  And, there were lots of headaches and confusion.  Fast.

Now I write tests and consider the behavior of my system before I write a single line of code.  Now every line of code was written with a pair.  After every iteration the team gets together to discuss how we are doing and how we can do it better.  There are some headaches but a lot less confusion.  Now there is pride and quality.

Having the discipline of writing tests first, and developing software in thin vertical slices from the top down is hard.  It is hard because we desire closure, and immediate feedback.  But by focusing our energy on producing a long term quality product, we can achieve a higher level of satisfaction.

Being surrounded with that many passionate professionals makes you want to be better.

A huge thanks to Alex for putting on a great conference here in St. Louis.  I can’t wait for next year.

Cucumber: Making UATs the healthy choice

Posted in software testing, Uncategorized by benjaminplee on 11.01.09

My current project team has been eating its vegetables.  Has yours?

Selenium is a great tool for testing webapps from a true browser driven user’s perspective.  By driving most mainstream browsers via a core Javascript library, a Selenium script can easily test how your app behaves (or misbehaves) just like a real user would see.

Webrat lets you quickly write robust and thorough acceptance tests for a web application.”  Written in Ruby, Webrat allows acceptance tests to exercise an arbitrary web application through its Selenium integration, or Rails apps without starting a server.

Cucumber allows code to be written in a simple domain specific language (DSL) that anyone can read.  Layer Cucumber on top of Webrat and Selenium, and you have User Acceptance Tests (UATs) that the client can read (and ideally write), that the entire development team understands, and which can be run as BDD or regression tests.

Cucumber example taken from cuckes.info

Cucumber example taken from cuckes.info

Without going into the details of how each of these libraries work, below are some of my notes, tips, and impressions of using Cucumber to write UATs for a greenfield Java EE application over the past ~10 weeks.

  • Ideally, each test would run in its own transaction so that they wouldn’t interfere with each other and could be run in parallel.  Using Cucumber (Ruby) to test a totally separate web application running on Tomcat (Java) presents a problem.  The team debated several possible solutions using JRuby and having the Cucumber build spawn an embedded Tomcat instance but decided that the cost outweighed the benefit to the current project.
  • Since we were forced to use pure JDBC by our client, we decided to use ActiveRecord migrations to build our tables, and generate SQL which we could re-leverage from our Cucumber tests.  Custom creation Webrat steps were written to push setup test data for each model.
  • DB2 support with the Ruby ActiveRecord drivers is lacking proper install instructions on Linux (damn thing requries a full DB2 install to run). By leveraging JRuby, we were able to use the DB2 JDBC drivers instead which worked much better (without installing DB2).  Getting Cucumber, Webrat, and Selenium running in JRuby took a bit of finesse and shebang wrangling but eventually worked.
  • Outputting Cucumber results to a JUnit style XML output made our integration on our Hudson CI server simple and easy to read. (The Chuck Norris plugin helps too)
  • We tag each test with a story number (e.g. @10045) so that we can quickly run all tests for a given feature easily: cucumber –tags @10045.
  • Our QA lead works to define and write-up our Cucumber tests during pre-iteration planning and before each story enters our work queue (the team uses a modified Kanban board approach to pull work through each iteration, Cucumber is our first queue).  When she has a test written, she will tag it with @in-process to let us know that it is ready to be worked on, but not yet implemented.
  • We have custom Rake tasks to run all of our “finished” tests as well as just our “in-process” ones.  The in-process task will fail if any tests pass (they should be marked as finished).  Unfortunately this rake task doesn’t work with the Hudson/JUnit build.
  • Ongoing issues we still struggle with are:
    • making sure that small variations in DSL verbiage don’t muddy up our tests (it is easy for the team to accidentally end up with two different commands which do the same or similar actions without a clear distinction)
    • cross browser and separate environment testing happens on different dedicated servers on dependent builds so that they don’t end up stepping on each other
    • as it is with any testing library, care needs to be taken to keep only common tests grouped together and setup/background work common to all tests that NEED it.  Pushing too much work into common actions results in slow tests
    • Tests are supremely faster and more valuable than manual tests, but are not as fast as JUnit tests and will never replace manual exploratory testing.

Overall the team has been very pleased with our ability to drive our development from the tests and verify that new changes don’t break existing functionality from the users’ perspectives.

* Another Asynchrony team is working on leveraging Cuke4Duke to do similar testing of a Java thick client.  Should be interesting.

Edit: Thanks to Amos King for some proof reading help.  You can find his blog over at Dirty Information.

Follow

Get every new post delivered to your Inbox.