Two Guys Arguing

symbol-macrolet and inside-out test fixtures

Posted in clojure, common lisp by youngnh on 11.15.10

Rolling right along with more ways to write fixtures, let’s say we start with a test that looks like this:

(deftest test-convoluted
  (with-connection hdb
    (try
      (do-commands "create table person (name varchar(255))")
      (insert-values :person [:name] ["bill"] ["joey"])
      (with-query-results results ["select name from person"]
    (is (= "bill" (:name (first results))))
    (is (= "joey" (:name (second results)))))
      (finally
       (do-commands "drop schema public cascade")))))

It does it’s setup and teardown inside of the deftest itself. The actual “testing” parts are the two (is) forms deeply nested inside. I’ve already written a post on how the clojure.test lib that comes with Clojure addresses this kind of complexity by providing fixtures. However, they’re a bit clumsy and aren’t particularly fine-grained. You can either run them once before (and after) all tests run, or once before and after each test runs.

The Common Lisp testing framework FiveAM has a different approach to defining and using fixtures. The FiveAM library defines two macros, def-fixture and with-fixture which define a fixture form and execute forms inside of a named fixture, respectively.

To define a fixture that takes care of the setup and teardown in the test above, we would write something like this:

(def-fixture person-schema []
  (with-connection hdb
    (try
      (do-commands "create table person (name varchar(255), age integer)")
      (insert-values :person [:name] ["bill"] ["joey"])
      (with-query-results results [query]
        (test-body))
      (finally
       (do-commands "drop schema public cascade")))))

In the above, test-body is a special captured variable that will be replaced with whatever forms you later specify. It’s where the meat of your test will go. You specify what to run there as the body of the with-fixture macro, thusly:

(deftest test-ideal
  (with-fixture person-schema []
    (is (= "bill" (:name (first results))))
    (is (= "joey" (:name (second results)))))))

The with-fixture form names the fixture we want to use, and takes care of expanding it in such a way that the variables are still checked by the compiler. If our fixture didn’t declare results in the scope that our assertions were expanded to, the compiler would complain just as if we had written the whole thing out by hand.

We can make the fixture in our example even more flexible. def-fixture can declare arguments and with-fixture can provide them. Altering our setup slightly to add a second column, we can then pass in a specific query to be run per fixture:

(def-fixture person-schema [query]
  (with-connection hdb
    (try
      (do-commands "create table person (name varchar(255), age integer)")
      (insert-values :person [:name :age] ["bill" 25] ["joey" 35])
      (with-query-results results [query]
        (test-body))
      (finally
       (do-commands "drop schema public cascade")))))

And then we can get a lot more mileage out of our solitary fixture:

(deftest test-ideal
  (testing "name column"
   (with-fixture person-schema ["select name from person"]
     (is (= "bill" (:name (first results))))
     (is (= "joey" (:name (second results))))))

  (testing "age column"
    (with-fixture person-schema ["select age from person"]
      (is (= 25 (:age (first results))))
      (is (= 35 (:age (second results)))))))

This is a trick that defmacro can’t easily perform. If we wanted to define person-schema as a macro, in order to capture results, we’d have to write put ~'results somewhere in a backtick form. It’ll work, but for any significant number of capturing symbols, there’s tildes and ticks everywhere. For a feature that would ostensibly have users writing lots and lots of their own expansions, that’s a major drawback, in my opinion. Early versions of the newly re-written ClojureQL had users do this, I believe, with little snippets of macros sprinkled throughout their code. It turns out that we can have our cake and eat it too, and you’ve probably guessed how from the title of this post.

I copped the implementation straight from FiveAM. Common Lisp has an advantage here, as the language has built-in local macros (Clojure’s are global to a namespace) and symbol macros, which FiveAM uses to great effect. However, Konrad Hinsen (of clojure.contrib.monad fame) has implemented local and symbol macros in the clojure.contrib.macro-utils lib. I used that.

My Clojure implementation consists of the two macros and a global dynamic variable that holds an atom mapping fixture names to their arguments and bodies. The def-fixture macro takes care of assoc-ing them as they are defined. The with-fixture macro pulls that fixture definition and constructs an anonymous function from it, as well as making test-body a symbol macro that expands to the given body of with-fixture.

(def *fixtures* (atom {}))

(defmacro def-fixture [name args & body]
  `(swap! *fixtures* assoc '~name (cons '~args '~body)))

(defmacro with-fixture [name args & body]
  (let [[largs lbody] (get @*fixtures* name)]
    `(symbol-macrolet [~'test-body (fn [] ~@body)]
                  ((fn ~largs ~lbody)
                   ~@args))))

I could have used a local macro, by swapping the symbol-macrolet form for a macrolet form, but then I would have had to quote the body parameter passed to with-fixture. By using a symbol macro and asking that users treat it like a fn, I can avoid that. It’s a small thing and either way works. For 8 lines of code overall, these 2 macros add a lot of flexibility to how you can define test fixtures.

Tagged with: , ,

fixies

Posted in clojure by youngnh on 03.24.10

write tests independent of their environment

As part of a larger effort at Revelytix, we’re building a Clojure lib to work with Semantic Triplestores. This post sums up my experiences testing our code with clojure.test (formerly known as clojure.test-is).

This is long. If you’re only mildly interested, skip to the good parts.

There are 5 methods I want to test: create-graph, drop-graph, insert-triple, load-file, and query.

In this post, I use two specific Semantic Web concepts, the graph and the triple. The analogy is poor, but you can think of a graph as database table, and a triple as row in a table.

My overarching concern here is that I’d like my tests to be as idempotent as possible. Once run, they should restore the state of the external triple store so that I can run any test individually or run the entire suite in any order without having to check if errors are due to incorrect preconditions. In fact, this is a requirement as clojure.test‘s run-tests function states in it’s documentation that it will run the defined tests in whatever flipping order it feels like (paraphrasing mine).

The first test I wrote was for create-graph. However, it right away violated my principle of restoring the state of the external world because it leaves a graph laying around.

(deftest test-create-graph
  (is (resultMessage= "Successfully created graph rmi://australia/server1#family"
                      (execute-command (create-graph *FAMILY*)))))

Maybe create-graph was the wrong choice for a first test. The next likely candidate is drop-graph, however, it will throw an exception if you ask it to drop a graph that doesn’t exist yet, so it’s not a perfect choice either. Really, if I’m to get away with this, I need to test both functions execution in a predefined order; my test won’t run and clean up after itself until both functions are implemented correctly.

Now, there’s 2 ways I could write a test that executes create-graph and drop-graph in a specified order and cleans up after itself:

I could write a single deftest form for each, and even though I said that run-tests does not guarantee that the tests will be run in the order they appear in the source file, the deftest form turns your test into a regular fn, callable just like any other you might create via the defn form. This allows us to then deftest a combination of both that will guarantee that they run in the right order.

(deftest test-create-graph
  ...)

(deftest test-drop-graph
  ...)

(deftest test-create-and-drop-graph
  (test-create-graph)
  (test-drop-graph))

This introduces a new problem, though. clojure.test now thinks you have 3 tests. deftest- will make the create and drop tests private, and only the composed test is now seen by clojure.test.

(deftest- test-create-graph
  ...)

(deftest- test-drop-graph
  ...)

(deftest test-create-and-drop-graph
  (test-create-graph)
  (test-drop-graph))

The second possiblity is to use the testing macro to label each part. I like this solution less since it doesn’t produce two individually runnable tests, but functionally it ensures that the tests run in order and aesthetically it describes itself quite nicely.

(deftest test-create-and-drop-graph
  (testing "Create Graph"
    (is (resultMessage= "Successfully created graph rmi://australia/server1#family"
			(execute-command (create-graph *FAMILY*)))))
  (testing "Drop Graph"
    (is (resultMessage= "Successfully dropped graph rmi://australia/server1#family"
			(execute-command (drop-graph *FAMILY*))))))

Next, I want to test the insert-triple and load-file functions. Neither make much sense to call without an existing graph to work with. I could, in each test, create a graph and then tear it down:

(deftest test-insert-triple
  (try
   (execute-command (create-graph *FAMILY*))
   (is (resultMessage= "Successfully inserted statements into rmi://australia/server1#family"
                       (execute-command (insert-triple *FAMILY* "Joe" "father_of" "Billy"))))
   (finally
    (execute-command (drop-graph *FAMILY*)))))

(deftest test-load-file
  (try
   (execute-command (create-graph *FAMILY*))
   (is (resultMessage= "Successfully inserted statements into rmi://australia/server1#family"
                       (execute-command (load-file *FAMILY* *N3_FAMILY_FILE*))))
   (finally
    (execute-command (drop-graph *FAMILY*)))))

The try/finally form is especially well-suited for testing since by contract it will ensure that despite any exceptions the code under test may throw, the graph will get dropped. It also has the benefit of returning the value of the last assertion and not the value of dropping the graph. By wrapping your assertions in a try/finally you’ve preserved the external “value” of your test while still dramatically altering what goes on when it’s called. That’s a pretty powerful encapsulation technique.

However, we’ve duplicated code setting up and tearing down the graph, so this would be a great place for a fixture. Fixtures in clojure.test are just regular functions that take a function to execute before or after the fixture’s done it’s thing:

(defn family-graph-fixture [f]
  (try
   (execute-command (create-graph *FAMILY*))
   (f)
   (finally
    (execute-command (drop-graph *FAMILY*)))))

The tests for insert-triple and load-file can now assume that their environment has been setup for them and can focus solely on the meat of their test:

(deftest test-insert-triple
   (is (resultMessage= "Successfully inserted statements into rmi://australia/server1#family"
                       (execute-command (insert-triple *FAMILY* "Joe" "father_of" "Billy")))))

(deftest test-load-file
  (is (resultMessage= "Successfully inserted statements into rmi://australia/server1#family"
                      (execute-command (load-file *FAMILY* *N3_FAMILY_FILE*)))))

Before we actually hook the fixture up to our tests, let’s take a look at the test for query. This function will blow up if it tries to select triples from a non-existent graph and we’ll want that graph to be populated with sample data. We already have a fixture for creating and dropping a graph, we can reuse that and add another for populating the graph with data.

(defn family-data-fixture [f]
  (execute-command (load-file *FAMILY* *N3_FAMILY_FILE*))
  (f))

We’re almost there. The execute-command function runs the given triplestore command on a dynamic var named *connection*. We’ll need a fixture to bind that var to a real connection.

(defn connection-fixture [f]
  (binding [*connection* (connect-to *HOST*)]
    (try
     (f)
     (finally
      (dispose *connection*)))))

We could then attach that fixture to every test we execute, but that would establish and dispose the connection on every test execution. We can save a lot of bandwidth and execution time by having every test use the same connection. Telling clojure.test to execute all tests in this fixture is simple, just put this form in your test file:

(use-fixtures :once connection-fixture)

Now we need to hook everything else up. clojure.test would suggest using (use-fixtures :each ... ) but that would establish all of the fixtures for every test. As far as I can tell, clojure.test doesn’t have a proscribed way to map individual fixtures to individual tests, so we have to apply our fixtures and compose the individual tests by hand. You override run-tests default execution by defining a test-ns-hook function with your setup-by-hand tests.

(defn test-ns-hook []
  (test-create-and-drop-graph)
  (family-graph-fixture test-insert-triple)
  (family-graph-fixture test-load-file)
  (family-graph-fixture #(family-data-fixture test-query)))

Being able to pass around fns makes the whole process quite easy. Note that I have to pass an anonymous function when nesting fixtures, but it doesn’t add too much line-noise and is written in execution-order, which increases readability. Also, even though I haven’t shown it in my above snippet, since we are defining how to run the tests with test-ns-hook we have taken responsibility for which tests get run, and no longer have to define any tests using the deftest- form.

However, now that we’ve done that, we’ve broken our (use-fixtures :once ...) statement. Calling run-tests will no longer establish the connection, and so need to add a final layer of indirection.

(deftest test-suite
  (test-graph-commands)
  (family-graph-fixture test-insert-triples)
  (family-graph-fixture test-load-file)
  (family-graph-fixture #(data-fixture test-query)))

(defn test-ns-hook []
  (connection-fixture test-suite))

Here’s the final result.


My co-worker (thats-right-I-am-dangerous-Iceman) Alex Miller points out that though it is prominent and publicly available, the front page of the clojure.test API is sometimes the last place you look to get help with this stuff, but the Overview documentation for the lib is excellent. Check it out.

tl;dr

  • Fixtures as they exist today in clojure.test aren’t quite flexible enough to cover all of my testing needs, but there are not-overly-complicated ways to get around that.
  • Keep tests short, testing only what concerns them and leave out all environment setup code.
  • Specify setup and teardown code in their own fixture functions for maximum reuse and composability.
  • Use a suite test to hook up fixtures to individual tests and test-ns-hook to run the suite
Tagged with: ,
Follow

Get every new post delivered to your Inbox.