write tests independent of their environment
As part of a larger effort at Revelytix, we’re building a Clojure lib to work with Semantic Triplestores. This post sums up my experiences testing our code with clojure.test (formerly known as clojure.test-is).
This is long. If you’re only mildly interested, skip to the good parts.
There are 5 methods I want to test:
In this post, I use two specific Semantic Web concepts, the graph and the triple. The analogy is poor, but you can think of a graph as database table, and a triple as row in a table.
My overarching concern here is that I’d like my tests to be as idempotent as possible. Once run, they should restore the state of the external triple store so that I can run any test individually or run the entire suite in any order without having to check if errors are due to incorrect preconditions. In fact, this is a requirement as clojure.test‘s
run-tests function states in it’s documentation that it will run the defined tests in whatever flipping order it feels like (paraphrasing mine).
The first test I wrote was for
create-graph. However, it right away violated my principle of restoring the state of the external world because it leaves a graph laying around.
(deftest test-create-graph (is (resultMessage= "Successfully created graph rmi://australia/server1#family" (execute-command (create-graph *FAMILY*)))))
Maybe create-graph was the wrong choice for a first test. The next likely candidate is
drop-graph, however, it will throw an exception if you ask it to drop a graph that doesn’t exist yet, so it’s not a perfect choice either. Really, if I’m to get away with this, I need to test both functions execution in a predefined order; my test won’t run and clean up after itself until both functions are implemented correctly.
Now, there’s 2 ways I could write a test that executes
drop-graph in a specified order and cleans up after itself:
I could write a single
deftest form for each, and even though I said that
run-tests does not guarantee that the tests will be run in the order they appear in the source file, the deftest form turns your test into a regular fn, callable just like any other you might create via the
defn form. This allows us to then deftest a combination of both that will guarantee that they run in the right order.
(deftest test-create-graph ...) (deftest test-drop-graph ...) (deftest test-create-and-drop-graph (test-create-graph) (test-drop-graph))
This introduces a new problem, though. clojure.test now thinks you have 3 tests.
deftest- will make the create and drop tests private, and only the composed test is now seen by clojure.test.
(deftest- test-create-graph ...) (deftest- test-drop-graph ...) (deftest test-create-and-drop-graph (test-create-graph) (test-drop-graph))
The second possiblity is to use the
testing macro to label each part. I like this solution less since it doesn’t produce two individually runnable tests, but functionally it ensures that the tests run in order and aesthetically it describes itself quite nicely.
(deftest test-create-and-drop-graph (testing "Create Graph" (is (resultMessage= "Successfully created graph rmi://australia/server1#family" (execute-command (create-graph *FAMILY*))))) (testing "Drop Graph" (is (resultMessage= "Successfully dropped graph rmi://australia/server1#family" (execute-command (drop-graph *FAMILY*))))))
Next, I want to test the
load-file functions. Neither make much sense to call without an existing graph to work with. I could, in each test, create a graph and then tear it down:
(deftest test-insert-triple (try (execute-command (create-graph *FAMILY*)) (is (resultMessage= "Successfully inserted statements into rmi://australia/server1#family" (execute-command (insert-triple *FAMILY* "Joe" "father_of" "Billy")))) (finally (execute-command (drop-graph *FAMILY*))))) (deftest test-load-file (try (execute-command (create-graph *FAMILY*)) (is (resultMessage= "Successfully inserted statements into rmi://australia/server1#family" (execute-command (load-file *FAMILY* *N3_FAMILY_FILE*)))) (finally (execute-command (drop-graph *FAMILY*)))))
The try/finally form is especially well-suited for testing since by contract it will ensure that despite any exceptions the code under test may throw, the graph will get dropped. It also has the benefit of returning the value of the last assertion and not the value of dropping the graph. By wrapping your assertions in a try/finally you’ve preserved the external “value” of your test while still dramatically altering what goes on when it’s called. That’s a pretty powerful encapsulation technique.
However, we’ve duplicated code setting up and tearing down the graph, so this would be a great place for a fixture. Fixtures in clojure.test are just regular functions that take a function to execute before or after the fixture’s done it’s thing:
(defn family-graph-fixture [f] (try (execute-command (create-graph *FAMILY*)) (f) (finally (execute-command (drop-graph *FAMILY*)))))
The tests for
load-file can now assume that their environment has been setup for them and can focus solely on the meat of their test:
(deftest test-insert-triple (is (resultMessage= "Successfully inserted statements into rmi://australia/server1#family" (execute-command (insert-triple *FAMILY* "Joe" "father_of" "Billy"))))) (deftest test-load-file (is (resultMessage= "Successfully inserted statements into rmi://australia/server1#family" (execute-command (load-file *FAMILY* *N3_FAMILY_FILE*)))))
Before we actually hook the fixture up to our tests, let’s take a look at the test for
query. This function will blow up if it tries to select triples from a non-existent graph and we’ll want that graph to be populated with sample data. We already have a fixture for creating and dropping a graph, we can reuse that and add another for populating the graph with data.
(defn family-data-fixture [f] (execute-command (load-file *FAMILY* *N3_FAMILY_FILE*)) (f))
We’re almost there. The
execute-command function runs the given triplestore command on a dynamic var named
*connection*. We’ll need a fixture to bind that var to a real connection.
(defn connection-fixture [f] (binding [*connection* (connect-to *HOST*)] (try (f) (finally (dispose *connection*)))))
We could then attach that fixture to every test we execute, but that would establish and dispose the connection on every test execution. We can save a lot of bandwidth and execution time by having every test use the same connection. Telling clojure.test to execute all tests in this fixture is simple, just put this form in your test file:
(use-fixtures :once connection-fixture)
Now we need to hook everything else up. clojure.test would suggest using
(use-fixtures :each ... ) but that would establish all of the fixtures for every test. As far as I can tell, clojure.test doesn’t have a proscribed way to map individual fixtures to individual tests, so we have to apply our fixtures and compose the individual tests by hand. You override
run-tests default execution by defining a
test-ns-hook function with your setup-by-hand tests.
(defn test-ns-hook  (test-create-and-drop-graph) (family-graph-fixture test-insert-triple) (family-graph-fixture test-load-file) (family-graph-fixture #(family-data-fixture test-query)))
Being able to pass around fns makes the whole process quite easy. Note that I have to pass an anonymous function when nesting fixtures, but it doesn’t add too much line-noise and is written in execution-order, which increases readability. Also, even though I haven’t shown it in my above snippet, since we are defining how to run the tests with
test-ns-hook we have taken responsibility for which tests get run, and no longer have to define any tests using the
However, now that we’ve done that, we’ve broken our
(use-fixtures :once ...) statement. Calling
run-tests will no longer establish the connection, and so need to add a final layer of indirection.
(deftest test-suite (test-graph-commands) (family-graph-fixture test-insert-triples) (family-graph-fixture test-load-file) (family-graph-fixture #(data-fixture test-query))) (defn test-ns-hook  (connection-fixture test-suite))
Here’s the final result.
My co-worker (thats-right-I-am-dangerous-Iceman) Alex Miller points out that though it is prominent and publicly available, the front page of the clojure.test API is sometimes the last place you look to get help with this stuff, but the Overview documentation for the lib is excellent. Check it out.
- Fixtures as they exist today in clojure.test aren’t quite flexible enough to cover all of my testing needs, but there are not-overly-complicated ways to get around that.
- Keep tests short, testing only what concerns them and leave out all environment setup code.
- Specify setup and teardown code in their own fixture functions for maximum reuse and composability.
- Use a suite test to hook up fixtures to individual tests and
test-ns-hookto run the suite
You can find the first cut of my Prezi based presentation on Piet and QuickPiet here:
I will update the link/presentation over the next week or so leading up to the next Lambda Lounge meeting.
Roll-n-rolling with the Piet ROLL command
Mr. Young and a few others questioned how QuickPiet‘s command worked …. and to be honest, at first I didn’t either. It turns out that the Piet ROLL command works fairly simply and is extremely powerful (according to npiet which I used as the basis for my testing).
The ROLL command pops two values off of the stack puts nothing back. The top value from the stack defines how many “turns” the roll executes. A turn will put the top value on the bottom and shift all other values “up” one place toward the top of the stack. The second value defines the “depth” of the roll or how many elements should be included in each turn starting at the top of the stack.
As the main Piet page suggests, a ROLL to a depth of X and a single turn will effectively bury the top of the stack down X spots. If that is as clear as mud, check out this image which shows two roll actions executed on a already populated stack.
[EDIT: Correction, the last roll in the above image shows a roll of depth 5, 2 turns. Not 3. My bad.)
I used the nPiet command line Win32 interpreter to test this and the following program shown with 1 pixel per codel** and again with 20×20 pixels per codel. nPiet is the most stable interpreter I have found and has some nice features like creating trace output images, tracing/logging statements, and automatically guessing codel sizes. Below is the 20×20 codel version and nPiet trace output.
$ ./npiet.exe -v -t ben/roll-big.gifinfo: verbose set to 1info: trace set to 1info: using file ben/roll-big.gifinfo: got gif image with 320 x 120 pixelinfo: codelsize guessed is 20 pixeltrace: step 0 (0,0/r,l nR -> 1,0/r,l dR):action: push, value 1trace: stack (1 values): 1trace: step 1 (1,0/r,l dR -> 2,0/r,l lR):action: push, value 2trace: stack (2 values): 2 1trace: step 2 (2,0/r,l lR -> 3,0/r,l nR):action: push, value 3trace: stack (3 values): 3 2 1trace: step 3 (3,0/r,l nR -> 4,0/r,l dR):action: push, value 4trace: stack (4 values): 4 3 2 1trace: step 4 (4,0/r,l dR -> 5,0/r,l lR):action: push, value 5trace: stack (5 values): 5 4 3 2 1trace: step 5 (5,0/r,l lR -> 6,0/r,l nR):action: push, value 3trace: stack (6 values): 3 5 4 3 2 1trace: step 6 (6,0/r,l nR -> 7,0/r,l dR):action: push, value 2trace: stack (7 values): 2 3 5 4 3 2 1trace: step 7 (7,0/r,l dR -> 8,0/r,l lB):action: rolltrace: stack (5 values): 3 5 4 2 1trace: step 8 (8,0/r,l lB -> 9,0/r,l nB):action: push, value 3trace: stack (6 values): 3 3 5 4 2 1trace: step 9 (9,0/r,l nB -> 10,0/r,l dB):action: push, value 2trace: stack (7 values): 2 3 3 5 4 2 1trace: step 10 (10,0/r,l dB -> 11,0/r,l lG):action: rolltrace: stack (5 values): 4 3 5 2 1trace: entering white block at 12,0 (like the perl interpreter would)...trace: step 11 (11,0/r,l lG -> 12,0/r,l WW):trace: special case: we at a white codel - continuingtrace: white cell(s) crossed - continuing with no command at 12,2...trace: step 12 (12,0/r,l WW -> 12,2/d,r lG):info: program end
** Since small Piet programs are TINY, instead of using 1×1 pixels as the unit block or codel, Piet interpreters should be able to use codels of larger sizes. Codels of larger sizes allow Piet programs to be easier to see and work with.
Create an interpreter for a new Language based on Piet: QuickPiet
To pair off of Nate’s Rock Paper Scissors challenge, here is one I am offering up to him: create an interpreter which will execute QuickPiet commands. What is QuickPiet? QuickPiet is a very basic language based off of the commands found in the esoteric Piet programming language.
Piet programmers are written as images which can be interpreted as actions based on a stack. Recently I signed up to give a talk at the April Fool’s Day meeting of the Lambda Lounge where several people will be giving short talks on esoteric and useless programming languages. Programming in Piet poses several challenges including working only with a single stack and building up your logic into pixels and colors. I should have a few posts on writing programs in Piet in the next week or so. Until then, I have been mainly struggling with how to accomplish real work with just a single stack and a very limited menu of commands to choose from. QuickPiet is an attempt to make that process easier.
- Write an interpreter for the QuickPiet language I have posted in a new GitHub Gist; http://gist.github.com/332040.
[Edit: The language spec is now in the base QuickPiet repo on GitHub]
- The interpreter should allow the user to enter in QuickPiet commands from a file or GUI text box
- The interpreter should allow the user to enter text to be handled by STDIN and should display to the user any characters sent to STDOUT
- Extra points for GUIs and/or the ability to see how the stack changes through program execution and/or at the end
The “complete” spec can be found on the Gist page, but a very quick rundown is below:
- Program execution is top down, except for GOTO commands
- Each line can be blank (ignored), a label, a comment, or a command
- Only one command per line
- There are no explicit looping or other control structures
- The interpreter maintains a single “infinitely large” stack which different commands can act on
- Example commands include (IN, OUT, PUSH, POP, ADD, SUBTRACT, GOTO, etc)
- Commands are case-insensitive, labels are not
If you have any questions or are willing to venture a solution please leave a comment. I just had this idea this morning so I am very interested in any feedback you may have. I am planning on working on my own solution and hope to have something put together today or tomorrow.