Non-Java tools are often used for functional (black box) regression testing J2EE web applications. Such tools include Mercury WinRunner, SilkTest, (IBM) Rational Robot, and others. The lure of these tools is the “capture/playback” capabilities (making it easy to create tests without writing code), and scripting capabilities.
This seems like a positive step away from tedious, error-prone, and infrequently done manual regression testing. But you may be able to do better.
I label these tests semi-automated (or semi-manual if you’re a “half-empty” type) because of the way I see them applied in practice: I see them initiated manually, followed by interpretation of the results by the team’s “QA person.”
These tools have some disadvantages:
I’ve yet to see them plugged them into a Continuous Integration server such as CruiseControl or Continuum. It is reckless and wasteful not to run regression tests continuously (think hourly) while developers are changing code, especially across multiple teams. The Continuous Integration server needs a way to create a known initial state (i.e. build, deploy, and reset the database), initiate the tests, roll up all their results (along with results of unit tests) into one binary pass/fail assessment, and identify which test has failed when that occurs. These things are relatively easy to do with JUnit-based tests. It should also be technically possible to do this with WinRunner/SilkTest/etc. but I haven’t seen anyone do it yet. If you’ve gotten this to work, please write to me or post a response here.
License restrictions impede the reproducibility of the test results. In XP teams everyone has access to their own “push to test” button, and pushes it a lot.
License restrictions and proprietary scripting languages re-enforce the unagile idea that one person on the team is responsible for testing. It’s hard enough to break this habit without artificial restrictions from the tools.
Overuse of capture/playback can make the tests break easily when user interfaces change. This can be mitigated by using the proprietary scripting languages.
JUnit tests are easy to plug into Continuous Integration, aren’t bound by license restrictions, and are written in Java, one of the best known programming languages (especially on teams that are already developing Java applications!). Most traditional testers have no background in Java, thus would need to pair with Java programmers to create tests.
The JUnit framework was originally intended (and still primarily used) for “unit testing” — that is, white box testing of individual components outside the context of the whole system. That’s great and all, but creating deterministic (very thoroughly tested) control systems for aircraft and spacecraft has taught me that systems tend to fail more than individual components. We need end-to-end (system) tests even more than we need “unit” tests. This is one area I differ (slightly) with the mainstream XP crowd.
The good news is that a great selection of third-party libraries exist for system testing with JUnit. They’re all free and/or open source.
JUnit alone is sufficient for EJB testing. As with the others, your build script will need to deploy and start the server. And you’ll need some way to reset the database to a known state. Apache Cactus may be useful in some cases.
Selenium will drive a wide array of browsers for MS-Windows, Linux, and Mac OS X. This, combined with Selenium Remote Control is the current favorite at Danube.
Abbot is for testing Java Swing and AWT GUIs, such as desktop applications, Java WebStart applications, or Applets. The ScrumWorks desktop client is extensively tested with Abbot on MS-Windows and Mac OS X.
When someone at Danube checks in a code change, the screens of a Mac and a PC in the back room (each running CruiseControl) wake up and start GUI tests doing many of the same things you do as a user, except faster and much more abusively. And yeah, we still need to do a lot of manual testing.
Those are the third-party tools/libraries we can vouch for from personal experience. If you’ve had good experiences with others (or bad experiences with any of these), please let me know!
Software Process Mentor
Danube Technologies, Inc.
Testing during Sprint execution is discussed in this example Daily Scrum Meeting video.