fixies

write tests independent of their environment

As part of a larger effort at Revelytix, we’re building a Clojure lib to work with Semantic Triplestores. This post sums up my experiences testing our code with clojure.test (formerly known as clojure.test-is).

This is long. If you’re only mildly interested, skip to the good parts.

There are 5 methods I want to test: create-graph, drop-graph, insert-triple, load-file, and query.

In this post, I use two specific Semantic Web concepts, the graph and the triple. The analogy is poor, but you can think of a graph as database table, and a triple as row in a table.

My overarching concern here is that I’d like my tests to be as idempotent as possible. Once run, they should restore the state of the external triple store so that I can run any test individually or run the entire suite in any order without having to check if errors are due to incorrect preconditions. In fact, this is a requirement as clojure.test‘s run-tests function states in it’s documentation that it will run the defined tests in whatever flipping order it feels like (paraphrasing mine).

The first test I wrote was for create-graph. However, it right away violated my principle of restoring the state of the external world because it leaves a graph laying around.

(deftest test-create-graph
  (is (resultMessage= "Successfully created graph rmi://australia/server1#family"
                      (execute-command (create-graph *FAMILY*)))))

Maybe create-graph was the wrong choice for a first test. The next likely candidate is drop-graph, however, it will throw an exception if you ask it to drop a graph that doesn’t exist yet, so it’s not a perfect choice either. Really, if I’m to get away with this, I need to test both functions execution in a predefined order; my test won’t run and clean up after itself until both functions are implemented correctly.

Now, there’s 2 ways I could write a test that executes create-graph and drop-graph in a specified order and cleans up after itself:

I could write a single deftest form for each, and even though I said that run-tests does not guarantee that the tests will be run in the order they appear in the source file, the deftest form turns your test into a regular fn, callable just like any other you might create via the defn form. This allows us to then deftest a combination of both that will guarantee that they run in the right order.

(deftest test-create-graph
  ...)

(deftest test-drop-graph
  ...)

(deftest test-create-and-drop-graph
  (test-create-graph)
  (test-drop-graph))

This introduces a new problem, though. clojure.test now thinks you have 3 tests. deftest- will make the create and drop tests private, and only the composed test is now seen by clojure.test.

(deftest- test-create-graph
  ...)

(deftest- test-drop-graph
  ...)

(deftest test-create-and-drop-graph
  (test-create-graph)
  (test-drop-graph))

The second possiblity is to use the testing macro to label each part. I like this solution less since it doesn’t produce two individually runnable tests, but functionally it ensures that the tests run in order and aesthetically it describes itself quite nicely.

(deftest test-create-and-drop-graph
  (testing "Create Graph"
    (is (resultMessage= "Successfully created graph rmi://australia/server1#family"
			(execute-command (create-graph *FAMILY*)))))
  (testing "Drop Graph"
    (is (resultMessage= "Successfully dropped graph rmi://australia/server1#family"
			(execute-command (drop-graph *FAMILY*))))))

Next, I want to test the insert-triple and load-file functions. Neither make much sense to call without an existing graph to work with. I could, in each test, create a graph and then tear it down:

(deftest test-insert-triple
  (try
   (execute-command (create-graph *FAMILY*))
   (is (resultMessage= "Successfully inserted statements into rmi://australia/server1#family"
                       (execute-command (insert-triple *FAMILY* "Joe" "father_of" "Billy"))))
   (finally
    (execute-command (drop-graph *FAMILY*)))))

(deftest test-load-file
  (try
   (execute-command (create-graph *FAMILY*))
   (is (resultMessage= "Successfully inserted statements into rmi://australia/server1#family"
                       (execute-command (load-file *FAMILY* *N3_FAMILY_FILE*))))
   (finally
    (execute-command (drop-graph *FAMILY*)))))

The try/finally form is especially well-suited for testing since by contract it will ensure that despite any exceptions the code under test may throw, the graph will get dropped. It also has the benefit of returning the value of the last assertion and not the value of dropping the graph. By wrapping your assertions in a try/finally you’ve preserved the external “value” of your test while still dramatically altering what goes on when it’s called. That’s a pretty powerful encapsulation technique.

However, we’ve duplicated code setting up and tearing down the graph, so this would be a great place for a fixture. Fixtures in clojure.test are just regular functions that take a function to execute before or after the fixture’s done it’s thing:

(defn family-graph-fixture [f]
  (try
   (execute-command (create-graph *FAMILY*))
   (f)
   (finally
    (execute-command (drop-graph *FAMILY*)))))

The tests for insert-triple and load-file can now assume that their environment has been setup for them and can focus solely on the meat of their test:

(deftest test-insert-triple
   (is (resultMessage= "Successfully inserted statements into rmi://australia/server1#family"
                       (execute-command (insert-triple *FAMILY* "Joe" "father_of" "Billy")))))

(deftest test-load-file
  (is (resultMessage= "Successfully inserted statements into rmi://australia/server1#family"
                      (execute-command (load-file *FAMILY* *N3_FAMILY_FILE*)))))

Before we actually hook the fixture up to our tests, let’s take a look at the test for query. This function will blow up if it tries to select triples from a non-existent graph and we’ll want that graph to be populated with sample data. We already have a fixture for creating and dropping a graph, we can reuse that and add another for populating the graph with data.

(defn family-data-fixture [f]
  (execute-command (load-file *FAMILY* *N3_FAMILY_FILE*))
  (f))

We’re almost there. The execute-command function runs the given triplestore command on a dynamic var named *connection*. We’ll need a fixture to bind that var to a real connection.

(defn connection-fixture [f]
  (binding [*connection* (connect-to *HOST*)]
    (try
     (f)
     (finally
      (dispose *connection*)))))

We could then attach that fixture to every test we execute, but that would establish and dispose the connection on every test execution. We can save a lot of bandwidth and execution time by having every test use the same connection. Telling clojure.test to execute all tests in this fixture is simple, just put this form in your test file:

(use-fixtures :once connection-fixture)

Now we need to hook everything else up. clojure.test would suggest using (use-fixtures :each ... ) but that would establish all of the fixtures for every test. As far as I can tell, clojure.test doesn’t have a proscribed way to map individual fixtures to individual tests, so we have to apply our fixtures and compose the individual tests by hand. You override run-tests default execution by defining a test-ns-hook function with your setup-by-hand tests.

(defn test-ns-hook []
  (test-create-and-drop-graph)
  (family-graph-fixture test-insert-triple)
  (family-graph-fixture test-load-file)
  (family-graph-fixture #(family-data-fixture test-query)))

Being able to pass around fns makes the whole process quite easy. Note that I have to pass an anonymous function when nesting fixtures, but it doesn’t add too much line-noise and is written in execution-order, which increases readability. Also, even though I haven’t shown it in my above snippet, since we are defining how to run the tests with test-ns-hook we have taken responsibility for which tests get run, and no longer have to define any tests using the deftest- form.

However, now that we’ve done that, we’ve broken our (use-fixtures :once ...) statement. Calling run-tests will no longer establish the connection, and so need to add a final layer of indirection.

(deftest test-suite
  (test-graph-commands)
  (family-graph-fixture test-insert-triples)
  (family-graph-fixture test-load-file)
  (family-graph-fixture #(data-fixture test-query)))

(defn test-ns-hook []
  (connection-fixture test-suite))

Here’s the final result.

My co-worker (thats-right-I-am-dangerous-Iceman) Alex Miller points out that though it is prominent and publicly available, the front page of the clojure.test API is sometimes the last place you look to get help with this stuff, but the Overview documentation for the lib is excellent. Check it out.

tl;dr

• Fixtures as they exist today in clojure.test aren’t quite flexible enough to cover all of my testing needs, but there are not-overly-complicated ways to get around that.
• Keep tests short, testing only what concerns them and leave out all environment setup code.
• Specify setup and teardown code in their own fixture functions for maximum reuse and composability.
• Use a suite test to hook up fixtures to individual tests and test-ns-hook to run the suite