Cucumber: Making UATs the healthy choice
My current project team has been eating its vegetables. Has yours?
“Webrat lets you quickly write robust and thorough acceptance tests for a web application.” Written in Ruby, Webrat allows acceptance tests to exercise an arbitrary web application through its Selenium integration, or Rails apps without starting a server.
Cucumber allows code to be written in a simple domain specific language (DSL) that anyone can read. Layer Cucumber on top of Webrat and Selenium, and you have User Acceptance Tests (UATs) that the client can read (and ideally write), that the entire development team understands, and which can be run as BDD or regression tests.
Without going into the details of how each of these libraries work, below are some of my notes, tips, and impressions of using Cucumber to write UATs for a greenfield Java EE application over the past ~10 weeks.
- Ideally, each test would run in its own transaction so that they wouldn’t interfere with each other and could be run in parallel. Using Cucumber (Ruby) to test a totally separate web application running on Tomcat (Java) presents a problem. The team debated several possible solutions using JRuby and having the Cucumber build spawn an embedded Tomcat instance but decided that the cost outweighed the benefit to the current project.
- Since we were forced to use pure JDBC by our client, we decided to use ActiveRecord migrations to build our tables, and generate SQL which we could re-leverage from our Cucumber tests. Custom creation Webrat steps were written to push setup test data for each model.
- DB2 support with the Ruby ActiveRecord drivers is lacking proper install instructions on Linux (damn thing requries a full DB2 install to run). By leveraging JRuby, we were able to use the DB2 JDBC drivers instead which worked much better (without installing DB2). Getting Cucumber, Webrat, and Selenium running in JRuby took a bit of finesse and shebang wrangling but eventually worked.
- Outputting Cucumber results to a JUnit style XML output made our integration on our Hudson CI server simple and easy to read. (The Chuck Norris plugin helps too)
- We tag each test with a story number (e.g. @10045) so that we can quickly run all tests for a given feature easily: cucumber –tags @10045.
- Our QA lead works to define and write-up our Cucumber tests during pre-iteration planning and before each story enters our work queue (the team uses a modified Kanban board approach to pull work through each iteration, Cucumber is our first queue). When she has a test written, she will tag it with @in-process to let us know that it is ready to be worked on, but not yet implemented.
- We have custom Rake tasks to run all of our “finished” tests as well as just our “in-process” ones. The in-process task will fail if any tests pass (they should be marked as finished). Unfortunately this rake task doesn’t work with the Hudson/JUnit build.
- Ongoing issues we still struggle with are:
- making sure that small variations in DSL verbiage don’t muddy up our tests (it is easy for the team to accidentally end up with two different commands which do the same or similar actions without a clear distinction)
- cross browser and separate environment testing happens on different dedicated servers on dependent builds so that they don’t end up stepping on each other
- as it is with any testing library, care needs to be taken to keep only common tests grouped together and setup/background work common to all tests that NEED it. Pushing too much work into common actions results in slow tests
- Tests are supremely faster and more valuable than manual tests, but are not as fast as JUnit tests and will never replace manual exploratory testing.
Overall the team has been very pleased with our ability to drive our development from the tests and verify that new changes don’t break existing functionality from the users’ perspectives.
* Another Asynchrony team is working on leveraging Cuke4Duke to do similar testing of a Java thick client. Should be interesting.
Edit: Thanks to Amos King for some proof reading help. You can find his blog over at Dirty Information.