Non-Java tools are often used for functional (black box) regression testing J2EE web applications. Such tools include Mercury WinRunner, SilkTest, (IBM) Rational Robot, and others. The lure of these tools is the “capture/playback” capabilities (making it easy to create tests without writing code), and scripting capabilities.
This seems like a positive step away from tedious, error-prone, and infrequently done manual regression testing. But you may be able to do better.
I label these tests semi-automated (or semi-manual if you’re a “half-empty” type) because of the way I see them applied in practice: I see them initiated manually, followed by interpretation of the results by the team’s “QA person.”
These tools have some disadvantages:
JUnit tests are easy to plug into Continuous Integration, aren’t bound by license restrictions, and are written in Java, one of the best known programming languages (especially on teams that are already developing Java applications!). Most traditional testers have no background in Java, thus would need to pair with Java programmers to create tests.
The JUnit framework was originally intended (and still primarily used) for “unit testing” — that is, white box testing of individual components outside the context of the whole system. That’s great and all, but creating deterministic (very thoroughly tested) control systems for aircraft and spacecraft has taught me that systems tend to fail more than individual components. We need end-to-end (system) tests even more than we need “unit” tests. This is one area I differ (slightly) with the mainstream XP crowd.
The good news is that a great selection of third-party libraries exist for system testing with JUnit. They’re all free and/or open source.
When someone at Danube checks in a code change, the screens of a Mac and a PC in the back room (each running CruiseControl) wake up and start GUI tests doing many of the same things you do as a user, except faster and much more abusively. And yeah, we still need to do a lot of manual testing.
Those are the third-party tools/libraries we can vouch for from personal experience. If you’ve had good experiences with others (or bad experiences with any of these), please let me know!
–mj
Michael James
Software Process Mentor
Danube Technologies, Inc.
Testing during Sprint execution is discussed in this example Daily Scrum Meeting video.
Michael James is a software process mentor, team coach, and Scrum Trainer with a focus on the engineering practices (TDD, refactoring, continuous integration, pair programming) that allow Agile project management practices. He is also a software developer (a recovering "software architect" who still loves good design).
i am a qa person and i am totally sold on junit and the benefits of continuous integration. however, you omit a key benefit of tools like silktest, winruuner, etc., which is that they support data-driven tests. this for qa is imperative. we need to test many permutations of an application event, not simply a few positive ones. for example, when working on an ecommerce site and testing a search engine, i want to script the invokation of the search function just once and make the input string a parameter, where that parameter is fed by an external data file that can contain a large number of test strings. we need to test way more than one or a few positive test cases, and data-driven testing allows us to do this. you have mentioned to me, michael, that we should look into FIT for this purpose, and i will, i just want to call out that for qa people this is a critical piece of the puzzle.
Carol’s message illustrates the value of involving QA from the very beginning. An isolated programmer in a hurry to get “code complete” is less likely to think of the negative test cases necessary to get work done/done/done.
While many JUnit teaching examples focus on the positive cases (what we used to call the “happy path” or “main success scenario” back in the days of use cases) it’s just as applicable for testing error handling.
I’ll be running a webinar on how to do this in the near future.
Michael James
Software Process Consultant
http://www.danube.com
Beginners may find more on jUnit and Unit Testing in the following article:
Unit Testing 101
Beginners may find more on jUnit and Unit Testing in the following article:
http://weblogs.java.net/blog/cayhorstmann/archive/2006/10/the_worlds_simp.html
I applaud that professor for teaching automated testing (which should be an integrated part of programming) in his programming course. I’d be a bit wary of continuing the System.out.println() approach beyond one course though. JUnit’s not too hard to learn, especially compared to learning to code.
–mj
Michael James
Software Process Consultant
http://www.danube.com
Just a quick comment about data driven tests with watij or other tools.
Since watij is a java api you are free to construct any type of testing framework. You can model your automation framework to mirror your applications model and build test cases which are also data driven. It would not be difficult to build a test case which pulls data from a file or internal data structure to execute a test multiple times with varying data scenarios.
For a simple example: I could model a Login with a LoginTest class which uses a User object to login and validate on login. You could then create multiple scenarios for LoginTest by just creating new User objects with varying scenarios and validate expectations.
I am hoping to cover some of this at my talk at Agile 2007 this year.
Happy Testing
Jake Dempsey
http://www.watij.com
http://www.explainpmt.com
Your point illustrates there’s a lot to be said for writing the tests in a programming language like Java, which lets you do *anything*, rather than a proprietary scripting language which seems easier to learn at first but is ultimately more limiting.
Congrats on the 3.1.0 release of a great tool, and give a great talk at Agile 2007.
–mj
Michael James
Software Process Consultant
http://www.danube.com
nice topic, testing is always a major hurdle to get ‘leet’ programmers to become business useful.
http://www.junitee.org/ seems missing from your list though. Cactus does the same, but the core JunitEE code is nice if you want to integrate the test code right into the program environment and get away from unit testing. (free too)