Two Guys Arguing

TestNG Group Gotcha

Posted in software testing by benjaminplee on 03.19.09

I have been forcing myself to use TestNG lately because its what my current client uses and to try something new.  Overall I have been happy with it, but I don’t think I will be switching from JUnit any time soon on projects where I have a choice.  Yes, TestNG has several features which JUnit doesn’t, but JUnit 4 made up a lot of ground from when TestNG was first released in response to the shortcomings of JUnit 3.  Plus, a lot of the features I just don’t think I would use very often.

One feature that did make a lot of sense to me was the idea of test groups.  Basically, the @Test annotation allows you to specify any number of strings as group names that the given test will belong to.  Then, when you want to execute a test suite, you can specify which groups are included or excluded from your suite.  Sounds simple enough and would allow you to divide up your test executions if you wanted to.  So I gave it a shot.

I wrote up a new test, gave it a label of “integration” (distinguishing it from my normal unit tests) and tried to run it.  ERRRR.  NullPointerException.  After several minutes verifying I had things setup correctly I realized the problem was that my @BeforeSuite and @BeforeMethod methods were not being called.  But the the test was working a minute ago ….  a quick check of the docs revealed that all of the @BeforeXXX and @AfterXXX methods ALSO can have a groups setting.  Meaning that when I told TestNG to run just the tests with a group of “integration” I was really telling it to only execute methods with that group.

This wasn’t the behavior I was expecting, but at least I understood what was going on after I read the docs.  If I added the group to my support methods, everything would be fine.  Only one problem.  The @BeforeXXX and @AfterXXX methods in question are in a base class and shouldn’t need to know about any groups a test class wants to use.

Enter the alwaysRun annotation property.  According to the TestNG docs:

For before methods (beforeSuite, beforeTest, beforeTestClass and beforeTestMethod, but not beforeGroups): If set to true, this configuration method will be run regardless of what groups it belongs to.
For after methods (afterSuite, afterClass, …): If set to true, this configuration method will be run even if one or more methods invoked previously failed or was skipped.

By adding this to the base class, the correct methods are run each time.  I am not 100% sure this won’t have other side effects that I haven’t found yet, but as it stands, it allows me to use groups in my tests AND use the base class to setup and teardown some of our common objects.  Its not a perfect solution, but it works.