Last week I had a quick and dirty discussion w/ some coworkers about Selenium, common problems they have had using it, and features they would want in a framework build on top of Selenium and/or in support of it.
Having used Selenium for user acceptance testing and regression testing for a couple projects now, it was interesting to see how many of the people in the room had had the same issues and that most of them are as of the last time I checked, still issues. I am in no way trying to imply that the developers maintaining Selenium IDE/RC/Grid are not working hard (I noticed numerous issue fixes or niceties the last few times I pulled it down), just that there seem to be some issues that are keeping it from being a whole lot easier to use.
For posterities sake, and upon request of a couple of the attendees, I present my notes on what was discussed a couple of my own thoughts. Some are more true of user acceptance testing of web-apps in general than Selenium specifically, but still important. Its interesting how the issues/features brought up seem to fit into a few broad categories.
Technical Issues w/ Selenium
- Popups and multiple window are hard to manage and keep straight.
- Web-apps that rely heavily on DHTML and visual changes w/o a page load are hard to test b/c the only difference between a valid page and an invalid one might be the application of a specific css style (which the designers could change later w/o breaking the page)
- Some test constructs behave differently in different browsers
- Other technical issues: SSL certificates, downloading of files, proxy chaining (I have struggled with solving this for some time now. I just can’t get Se to either keep my proxy settings or use the -DproxyHost setting I provide)
Test Maintenance / Execution / Management
- How do we abstract out the Selenium environment (browser, test server, etc) from the tests while keeping test execution for the build as well as the developer simple and fast?
- How do we keep the time to execute individual tests down? What happens when the test suite takes FOREVER to run?
Best Practices (Testing/Selenium/Agile)
- How do we keep the logical idea of an element on the page seperate from the specific locators we need to tell Selenium how to find it?
- How do we make sure there are IDs or something similar on all important elements so that we don’t have locators which are 50+ character XPath expressions?
- How do we incorporate test data setup/teardown into our tests when using the IDE to generate and/or when they are written by “non developers”?
Testing Writing / Debugging
- Framework Feature Request – easy utility to spit out all of the IDs on the page. (Firebug and XPather in FF are lifesavers here)
- Complicated AJAX creates some real problems when it comes to making sure you have waited until after the logical operation has completed and knowing what has changed.
- We need to provide better error messages and/or write tests which are smaller and more readable so that the time between a test failing and understanding why it failed is minimal.
- XPath can be confusing. Period.
- DSLs and syntactic sugar over the top of Selenium make it sweeter but can become another whole language a team must learn in order to use the tool.