Two Guys Arguing

Fixture Macros

Posted in clojure by youngnh on 11.13.10

A few months ago, at Revelytix, we put together a pretty large test-harness at work for putting Semantic Triplestores through their paces. The effort was highly dependant on setup and teardown of the external environment.

For instance, in order to run a single test that involves making a http request to a triplestore-backed web application, I had to write code to start the virtual machine my web application and triplestore images were installed on, restore that vm to a known-state snapshot, wait for the guest operating system to come online, login to the web application and establish a session, and oh yeah, actually make the http request to see the results I was really interested in.

Ultimately, we went with a design that passed a lot of maps around, but tonight I found myself reviewing some of the other possibilities we could have pursued. Clojure maintains great separation of concerns by allowing you to pass functions around. For example, here’s the function I originally wrote to setup and teardown a virtual machine, running some arbitrary function f in-between:

(defn vm-setup [vm snaphsot f]
  (restore-snapshot vm snapshot)
  (start-vm vm)
  (try
   (f)
   (catch Exception e
     (.printStackTrace e))
   (finally
    (save-vm-state vm))))

The function knows nothing about what f does and so f can be anything.

After the call to start-vm, VirtualBox launches immediately, but it often takes 10-15 seconds for the guest OS to start running and communicating. Executing f immediately after starting the vm will often fail if you don’t first wait for the guest OS to come online. I could write that sort of functionality into the vm-setup function, but that’s mixing concerns, and I could just as easily write another setup function that tries to ping the guest OS, executing it’s payload once the machine has started communicating:

(defn host-reachable? [host timeout f]
  (if (ping host timeout)
    (f)
    (throw (Exception. (str "Could not reach host: " host " within " timeout " ms")))))

This pattern goes on. Conceptually, I ended up with a single test being expressed like this:

(vm-setup "WebAppVM1" "CleanSnapshot"
  (host-reachable? "hostname" (* 30 1000)
    (webapp-login "hostname" "username" "password"
      (times 30
        (timethis
          (http-request "/some/webapp/path"))))))

Which is pretty readable and it’s structure matches it’s meaning. The test itself is the call to

(http-request "/some/webapp/path")

and it’s nested pretty deeply in an execution context.

If we needed to resuse all of those steps, we could make it into a composite setup function:

(defn start-vm-and-login [f]
  (vm-setup "WebAppVM1" "CleanSnapshot"
    (host-reachable? "hostname" (* 30 1000)
      (webapp-login "hostname" "username" "password"
        (times 30
          (timethis
            (f))))))

The code above, however, won’t run. The forms aren’t function objects, they’re function invocations that return values. We’d have to wrap each of them in (fn [] ) in order to lambda-ize them:

(vm-setup "WebAppVM1" "CleanSnapshot"
  (fn []
    (host-reachable? "hostname" (* 30 1000)
      (fn []
        (webapp-login "hostname" "username" "password"
          (fn []
            (times 30
              (fn []
                (timethis
                  (fn [] (http-request "/some/webapp/path")))))))))))

We can write our own macro to take the first version and expand into the second, and it helps if we notice how similar it is to the -> macro. In fact, we could write:

(->> (http-request "/some/webapp/path")
     (fn [])
     (timethis)
     (fn [])
     (times 30)
     (fn [])
     (webapp-login "hostname" "username" "password")
     (fn [])
     (host-reachable? "hostname" (* 30 1000))
     (fn [])
     (vm-setup "WebAppVM1" "CleanSnapshot"))

Which reads a little bit backwards, but provides a great target for our macro to generate:

(defmacro fixture [& forms]
  `(->> ~@(interpose '(fn []) (reverse forms))))

Allowing us to write:

(fixture (vm-setup "WebAppVM1" "CleanSnapshot")
         (host-reachable? "hostname" (* 30 1000))
     (webapp-login "hostname" "username" "password")
     (times 30)
     (timethis)
     (http-request "/some/webapp/path"))

Not bad

Tagged with: , , , ,

fixies

Posted in clojure by youngnh on 03.24.10

write tests independent of their environment

As part of a larger effort at Revelytix, we’re building a Clojure lib to work with Semantic Triplestores. This post sums up my experiences testing our code with clojure.test (formerly known as clojure.test-is).

This is long. If you’re only mildly interested, skip to the good parts.

There are 5 methods I want to test: create-graph, drop-graph, insert-triple, load-file, and query.

In this post, I use two specific Semantic Web concepts, the graph and the triple. The analogy is poor, but you can think of a graph as database table, and a triple as row in a table.

My overarching concern here is that I’d like my tests to be as idempotent as possible. Once run, they should restore the state of the external triple store so that I can run any test individually or run the entire suite in any order without having to check if errors are due to incorrect preconditions. In fact, this is a requirement as clojure.test‘s run-tests function states in it’s documentation that it will run the defined tests in whatever flipping order it feels like (paraphrasing mine).

The first test I wrote was for create-graph. However, it right away violated my principle of restoring the state of the external world because it leaves a graph laying around.

(deftest test-create-graph
  (is (resultMessage= "Successfully created graph rmi://australia/server1#family"
                      (execute-command (create-graph *FAMILY*)))))

Maybe create-graph was the wrong choice for a first test. The next likely candidate is drop-graph, however, it will throw an exception if you ask it to drop a graph that doesn’t exist yet, so it’s not a perfect choice either. Really, if I’m to get away with this, I need to test both functions execution in a predefined order; my test won’t run and clean up after itself until both functions are implemented correctly.

Now, there’s 2 ways I could write a test that executes create-graph and drop-graph in a specified order and cleans up after itself:

I could write a single deftest form for each, and even though I said that run-tests does not guarantee that the tests will be run in the order they appear in the source file, the deftest form turns your test into a regular fn, callable just like any other you might create via the defn form. This allows us to then deftest a combination of both that will guarantee that they run in the right order.

(deftest test-create-graph
  ...)

(deftest test-drop-graph
  ...)

(deftest test-create-and-drop-graph
  (test-create-graph)
  (test-drop-graph))

This introduces a new problem, though. clojure.test now thinks you have 3 tests. deftest- will make the create and drop tests private, and only the composed test is now seen by clojure.test.

(deftest- test-create-graph
  ...)

(deftest- test-drop-graph
  ...)

(deftest test-create-and-drop-graph
  (test-create-graph)
  (test-drop-graph))

The second possiblity is to use the testing macro to label each part. I like this solution less since it doesn’t produce two individually runnable tests, but functionally it ensures that the tests run in order and aesthetically it describes itself quite nicely.

(deftest test-create-and-drop-graph
  (testing "Create Graph"
    (is (resultMessage= "Successfully created graph rmi://australia/server1#family"
			(execute-command (create-graph *FAMILY*)))))
  (testing "Drop Graph"
    (is (resultMessage= "Successfully dropped graph rmi://australia/server1#family"
			(execute-command (drop-graph *FAMILY*))))))

Next, I want to test the insert-triple and load-file functions. Neither make much sense to call without an existing graph to work with. I could, in each test, create a graph and then tear it down:

(deftest test-insert-triple
  (try
   (execute-command (create-graph *FAMILY*))
   (is (resultMessage= "Successfully inserted statements into rmi://australia/server1#family"
                       (execute-command (insert-triple *FAMILY* "Joe" "father_of" "Billy"))))
   (finally
    (execute-command (drop-graph *FAMILY*)))))

(deftest test-load-file
  (try
   (execute-command (create-graph *FAMILY*))
   (is (resultMessage= "Successfully inserted statements into rmi://australia/server1#family"
                       (execute-command (load-file *FAMILY* *N3_FAMILY_FILE*))))
   (finally
    (execute-command (drop-graph *FAMILY*)))))

The try/finally form is especially well-suited for testing since by contract it will ensure that despite any exceptions the code under test may throw, the graph will get dropped. It also has the benefit of returning the value of the last assertion and not the value of dropping the graph. By wrapping your assertions in a try/finally you’ve preserved the external “value” of your test while still dramatically altering what goes on when it’s called. That’s a pretty powerful encapsulation technique.

However, we’ve duplicated code setting up and tearing down the graph, so this would be a great place for a fixture. Fixtures in clojure.test are just regular functions that take a function to execute before or after the fixture’s done it’s thing:

(defn family-graph-fixture [f]
  (try
   (execute-command (create-graph *FAMILY*))
   (f)
   (finally
    (execute-command (drop-graph *FAMILY*)))))

The tests for insert-triple and load-file can now assume that their environment has been setup for them and can focus solely on the meat of their test:

(deftest test-insert-triple
   (is (resultMessage= "Successfully inserted statements into rmi://australia/server1#family"
                       (execute-command (insert-triple *FAMILY* "Joe" "father_of" "Billy")))))

(deftest test-load-file
  (is (resultMessage= "Successfully inserted statements into rmi://australia/server1#family"
                      (execute-command (load-file *FAMILY* *N3_FAMILY_FILE*)))))

Before we actually hook the fixture up to our tests, let’s take a look at the test for query. This function will blow up if it tries to select triples from a non-existent graph and we’ll want that graph to be populated with sample data. We already have a fixture for creating and dropping a graph, we can reuse that and add another for populating the graph with data.

(defn family-data-fixture [f]
  (execute-command (load-file *FAMILY* *N3_FAMILY_FILE*))
  (f))

We’re almost there. The execute-command function runs the given triplestore command on a dynamic var named *connection*. We’ll need a fixture to bind that var to a real connection.

(defn connection-fixture [f]
  (binding [*connection* (connect-to *HOST*)]
    (try
     (f)
     (finally
      (dispose *connection*)))))

We could then attach that fixture to every test we execute, but that would establish and dispose the connection on every test execution. We can save a lot of bandwidth and execution time by having every test use the same connection. Telling clojure.test to execute all tests in this fixture is simple, just put this form in your test file:

(use-fixtures :once connection-fixture)

Now we need to hook everything else up. clojure.test would suggest using (use-fixtures :each ... ) but that would establish all of the fixtures for every test. As far as I can tell, clojure.test doesn’t have a proscribed way to map individual fixtures to individual tests, so we have to apply our fixtures and compose the individual tests by hand. You override run-tests default execution by defining a test-ns-hook function with your setup-by-hand tests.

(defn test-ns-hook []
  (test-create-and-drop-graph)
  (family-graph-fixture test-insert-triple)
  (family-graph-fixture test-load-file)
  (family-graph-fixture #(family-data-fixture test-query)))

Being able to pass around fns makes the whole process quite easy. Note that I have to pass an anonymous function when nesting fixtures, but it doesn’t add too much line-noise and is written in execution-order, which increases readability. Also, even though I haven’t shown it in my above snippet, since we are defining how to run the tests with test-ns-hook we have taken responsibility for which tests get run, and no longer have to define any tests using the deftest- form.

However, now that we’ve done that, we’ve broken our (use-fixtures :once ...) statement. Calling run-tests will no longer establish the connection, and so need to add a final layer of indirection.

(deftest test-suite
  (test-graph-commands)
  (family-graph-fixture test-insert-triples)
  (family-graph-fixture test-load-file)
  (family-graph-fixture #(data-fixture test-query)))

(defn test-ns-hook []
  (connection-fixture test-suite))

Here’s the final result.


My co-worker (thats-right-I-am-dangerous-Iceman) Alex Miller points out that though it is prominent and publicly available, the front page of the clojure.test API is sometimes the last place you look to get help with this stuff, but the Overview documentation for the lib is excellent. Check it out.

tl;dr

  • Fixtures as they exist today in clojure.test aren’t quite flexible enough to cover all of my testing needs, but there are not-overly-complicated ways to get around that.
  • Keep tests short, testing only what concerns them and leave out all environment setup code.
  • Specify setup and teardown code in their own fixture functions for maximum reuse and composability.
  • Use a suite test to hook up fixtures to individual tests and test-ns-hook to run the suite
Tagged with: ,
Follow

Get every new post delivered to your Inbox.