Two Guys Arguing

QUnit and the Command Line: One Step Closer,

Posted in javascript, software testing by benjaminplee on 11.06.10

Previously I talked about getting QUnit JavaScript tests running on the command line using a simple Rhino setup.  Hacking a few lines together meant that we could run our tests outside the constraints of a browser and potentionally as part of an automated build (e.g. Hudson).  We had just one big problem: QUnit can run outside of a browser, but it still makes LOTS of assumptions based on being in a browser.

Foremost was the fact that the QUnit.log callback passed back HTML in the message parameter.  I wasn’t the only one to catch onto this issue (GitHub issue 32).  While I was still formulating a plan of attack to refactor out all of the browser oriented code, Jörn Zaefferer was putting a much simpler fix into place.  His commit added an additional details object to all QUnit.log calls which will contains the original assertion message w/o HTML and possibly the raw expected and actual values (not yet documented on the main page).  Problem solved!

Or so it seemed.

As I tried to hack my example CLI test runner to use the quick fix I ran into several issues.

Core QUnit logic still includes browser assumptions

Even with changes to wrap browser calls and guard against assuming certain browser objects existing, qunit.js is full of browser code.  It would be great if the core unit testing code could be separated by the code necessary to execute properly within a browser page and separate from how test results should be displayed on an HTML page.  If these three responsibilities were found in 3 different objects, it would be simple to replace one or more with ones that fit the scenario much more closely without needing to resort to hacks or breaking backward compatibility.


Single Responsibility Principle

Single Responsibility Principle

Lifecycle callbacks break down outside of a browser

QUnit has some really nice lifecycle callbacks which anyone needing to integrate a testing tool can use.  They include callbacks for starting and finishing each test, individual assertions, and the whole test suite.  The first thing I wanted to add was reporting of the total number of passing and failing tests along with execution time when the tests were all done.  This looked like a simple job for QUnit.begin and QUnit.done.

It turns out that QUnit.begin won’t get called unless there is a window “load” event …. which doesn’t happen in Rhino …. so that is out.  To make matters worse, QUnit.done is getting called twice! For each test!  This means that my “final” stats are spammed with each test run.  With help of Rhino’s debugger app, I saw that the culprit were the successive “done” calls near the end of the “test” function.  Not sure how to fix that yet.

Truthiness” in JavaScript is a double edged sword

Most of the time it is great not having to worry about if a value is really “False” or if it is undefined or some other “falsey” value.  Only one problem, zero (0) is falsey too.  Going back to my C days this isn’t too big of a deal (and can be used in clever ways).  However, if you are checking for the existence of properties of an object by doing a boolean expression … don’t.  Sure, undefined is falsey and if an object doesn’t have a property it will return undefined … but what if the value of that proeperty really IS undefined or in my case zero.  No good.

var foo = {cat: "dog", count: 0 }
!!   // true
!!   // false (doesn't exist, returns undefined)
!!foo.count // false (0 is falsey)

** The double bang (!!boolean) convention is a convienent way for converting between a falsey or truthy value to the explicit values TRUE and FALSE

Unit test frameworks should standardize the order of arguments!

This is a personal pet peeve.  I wish all unit testing framework would standardize whether the EXPECTED value or the ACTUAL value should go first on equals assertions.  JUnit puts expected first.  TestNG puts it second.  QUnit puts it second.  This is damn confusing if you have to switch between these tools frequently.

I moved my current code to a new GitHub repo -> QUnit-CLI.  As I learn more and find a better solution, I will keep this repo updated.  Currently the code outputs one line for each passing test and a more detailed set of lines for each failing test (including individual assertion results).  Because of the QUnit.done problem above, the “final” test suite results are shown twice for each test (making them not very “final”) which shows the total test results and execution time. [Edited to have correct link]


Side Note: As an end goal, I would like to build a CLI for QUnit that will output Ant+JUnit style XML output which would make integrating these test results with other tools a piece of cake.  I CAN’T FIND THE XML DOCUMENTED ANYWHERE!  Lots of people have anecdotal evidence of how the XML will be produced but no one seems to have a DTD or XSD that is “official”.  If anyone knows of good documentation of the XML reports Ant generates for JUnit runs please let me know.  Thanks.


λ This post has nothing to do with Lisp or Clojure λ

9 Responses

Subscribe to comments with RSS.

  1. […] I have posted about some of my progress and setbacks here. Tagged with: javascript, qunit, […]

  2. Jörn Zaefferer said, on 11.08.10 at 4:26 am

    The browser assumptions are certainly an interesting problem to tackle. Though it looks like this is just an implementation issue, not something you really struggled with.

    QUnit.done getting called too often is a bug:

    For the truthiness problem, use strictEquals(), or just === or !== with ok().

    The argument order doesn’t really matter, its wrong in any case. Named arguments (ala Python) would solve that: equals(actual=whatever, expected=”my value”)
    In JavaScript we’d have to use an object literal, in Java it gets even worse, so we’re stuck with having to look up the argument order each time.

  3. Jörn Zaefferer said, on 11.08.10 at 4:33 am

    PS: The link to your GitHub repo links to jquery/qunit/issues…

  4. […] on 11.26.10 Previously I talked about wanting to run QUnit outside the browser and about some issues I ran into.  Finally, I have QUnit running from the command line: […]

  5. Jeff Sawatzky said, on 04.01.11 at 4:22 pm

    Have you looked at PhantomJS? Since it is a headless WebKit browser, it may solve your issues with Rhino not being a full browser?

  6. Simao Mata said, on 08.02.11 at 10:52 am


    Did you try using env.js?

    Works fine until I need to test backbone js methods. I am using localStorage for persistance but env.js doesn’t support it.

  7. […] post from 2010 about QUnit and the command line, which indicates that it’s rather difficult to do. The post mentions that the library […]

  8. […] post from 2010 about QUnit and the command line, which indicates that it’s rather difficult to do. The post mentions that the library […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s