Mock Objects Considered Insufficiently Harmful

Someone wrote to me:

Hey Michael – So I was at the Innotech conference this week in Portland and attended a talk about using mock objects to make unit testing easier. It definitely sounded like a way of getting some test coverage of “big ball of mud legacy code,” but it also threw up a bunch of red flags in my non-engineer brain.

Since I’m not an engineer, I don’t have any experiences to look back on to have an opinion on whether or not using mock objects is a good idea. But I felt like something was missing. Let’s say you have some code you want to test, but you want to isolate it from other layers of the architecture so there are no messy dependencies. So you put in some “perfect” mock objects to test the code with. GREAT! It works. Then you put in your real database or other real module that your code is supposed to work with and it seems to me that it might suddenly cease to work because you didn’t code it to your real world environment.

It feels like having a bunch of clothes tailored to a size six model, because you are also a size six but when you go to try on your new clothes, none of them fit because the fit model didn’t ACTUALLY have the same proportions as you. Sure, it takes more time out of your day to tailor everything to the “real world system” but it seems like that’s the only way to get a real fit and know that your investment will work.

Let me preface my response with the disclaimer that any kind of automated testing is better than what most teams are doing.

One goal of eXtreme Programming is to get the test to run quickly (like in a few milliseconds) so people will run it over and over as they’re refactoring code. Running the application in the context of the actual database, actual GUI, actual web environment takes seconds, minutes, or hours. Using mock objects for these parts makes it easier to develop tests, and much easier to run them. This provides near-instant feedback. So mock objects are a good thing, and skill at using them is worth developing. New development environments for Java and C# make it even easier to build these.

But do functioning components guarantee we have a functioning system?

The second test flight of the Grumman F-14 Tomcat (the huge fighter jet in Top Gun) suffered a hydraulic failure and crashed because of a harmonic relationship between the speed of an engine and the speed of a fuel pump. This combined with the lack of any mechanism to smooth out the pulses caused the resonance that burst the hydraulic lines. All parts (which are actually systems themselves) worked perfectly, but the system failed. The pilots ejected (and survived) 1.3 seconds before the plane crashed.

In software it turns out to be even easier to construct flawed systems out of working components. The most frequent source of the flaws? The presentation layer and interactions with other systems sharing the same database tables, the parts it’s most tempting to mock up.

I can’t give a blanket prescription for everyone. When I look at the tests that gave us the most bang for the buck in developing control systems for aircraft and spacecraft, they were always the end-to-end tests. I would generally prioritize those above the unit tests. Sure they take longer to rerun. Fortunately we now have continuous integration tools to do ease some of that pain.

This isn’t a black and white issue; it turns out there’s a continuum between pure unit tests and “system” tests. For example, I describe how to use a unit-testing tool for “system” testing here.

Think about your bathroom floor. Does more crud build up on the tiles, or in the grout between the tiles?

–mj
Michael James
Software Process Mentor
http://www.danube.com

Testing during the Sprint is explored during this example Daily Scrum Meeting video.

Michael James

Michael James is a software process mentor, team coach, and Scrum Trainer with a focus on the engineering practices (TDD, refactoring, continuous integration, pair programming) that allow Agile project management practices. He is also a software developer (a recovering "software architect" who still loves good design).

Posted in Agile
7 comments on “Mock Objects Considered Insufficiently Harmful
  1. roy says:

    I agree that there is a place for mock objects (or Strata tests) in the process, especially if you need to break dependancies, but a recent experience has made me wary of using them extensively.
    A major system integration / deployment failed, and one developer held up his hands and said “it’s not my problem, my tests passed with the mock objects…”. I was unable to respond politely.
    I guess, like most things, you’ve got to use your judgement, but I’ll try to ensure we can speed up whole-system tests rather than use mock objects in the future: load a smaller test dataset into the database, use a servers/services hosted on the same box.

  2. Michael James says:

    Thanks for the response.

    I heard that Ken Schwaber recently challenged the XP community on this very issue — does the overuse of mock objects compromise the robust definition of “done” we value in Scrum?

    While XP and Scrum and generally well aligned, I question the idea that tests have to run in a few milliseconds to get run frequently. Is this so much an issue now that we have continuous integration servers (CruiseControl, Continuum, etc.) to rerun tests every checkin and still get back to us reasonably fast? Running higher fidelity tests at a lower frequency (say, every hour) may provide greater assurance than low-fi tests every minute.

    –mj

    Michael James
    Software Process Consultant
    http://www.danube.com

  3. Jay Conne says:

    By ‘mock objects’, do you mean anything different than traditional stubbing out future or remote functionality?

  4. Michael James says:

    Yes, I mean something quite different from that.

    Michael James
    Software Process Consultant
    http://www.danube.com

  5. Aaron Bridges says:

    The way you’ve framed the question seems a little dangerous to me. Unit and functional tests are both indispensable to the quality of the product and the quality of my life as an engineer.

    But I don’t really think that there is any disconnect here:

    Unit tests guarantee INTERNAL quality of the code: how well class A fulfills its responsibilities, how well module B interacts with Module X, etc. If you are relying on these types of tests to ensure that your plane flies or your website collects money, you’re on thin ice.

    The kinds of tests that guarantee the EXTERNAL quality of your application are functional, system-level tests that should never be confused or conflated with unit tests and are equally important to a well-tested product.

    Unit and functional tests fulfill completely separate responsibilities. I’ve never read any credible source that said that unit tests could guarantee anything but EXACTLY WHAT THEY ARE TESTING, the individual pieces of the app, so I’m not sure what the motivation is for asking the question in the first place. If your team is in a position in which you’re choosing between unit and functional tests, IMO you’re already f-ed, probably not releasing often enough and relying on manual verification of your releases.

    SO why not just test through the UI? We could easily (a day or so) record every path through the system and then play it back right through a web browser, finding any problem we may have introduced, right? I’ve heard (and said) this several times, but that’s dangerous too. First, unit tests, and especially TDD, contribute to the quality of your design. Second, they make refactorings a breeze because they are already ensuring everything that you will need to test after your refactoring is complete.

    The combination of exhaustive, TDD-created unit tests and exhaustive functional acceptance tests gives you the kind of safety net to say that every build you produce can be deployed to production. Without either side, you’re either relying on luck or manual testing.

    Also, a note about running your full test suite on every checkin: we have found that any delay in finding an error or fixing it can have very long-running effects on our ability to produce a clean build. If we detect a defect an hour after the checkin, and my team is making 10 checkins per hour, we get a logjam of failed builds that can take days to sort out because there are failures on top of failures. This can really throw off your productivity!

    Run your longer-running acceptance tests hourly (or even daily) and keep the unit tests running extremely quickly and independently.

  6. Michael James says:

    As I already acknowledged, a commitment to qualtity means a greater emphasis on test than on creating new code. In successful aerospace projects I saw a ratio of about 4 to 1.

    Most of the teams I encounter lack any automated test coverage other than a couple smoke tests and some unit tests that don’t prove anything. Sure, it would be nice to have both the belt and the suspenders you advocate, but if I had to prioritize I’d put the higher-fidelity end-to-end tests above the ones that cut corners using mock objects.

    The idea there must be a dichotomy between “acceptance tests” and “unit tests” is part of the problem, enforced by the QA vs. programmer culture, and the tools that re-enforce this artificial barrier. Imagine a world where the “units” get continuously larger and larger, until they enter the realm of “acceptance tests.” Sure, they take a little longer to run. Machine time is cheaper than shipping bugs.

    –mj
    Michael James
    Software Process Mentor
    http://www.danube.com

  7. Chris Horn says:

    I know when telling others about the advantages of TDD the point I stress most is that TDD is a development practice not a testing practice and that it does not limit the need for any other testing or QA practice, it just should reduce the number of errors found by them.

    IMO, one of the big keys behind TDD is that at a module level every scenario should be covered by a test, if you’re to focus on integration tests instead of unit tests, you won’t be able to reasonably cover every scenario as the possibilities could be endless, so you need to strike the balance between the comprehensive set of unit tests, and a set of integration and functional tests that should cover all the integration points, but not necessarily every single possible path.

    Especially on agile projects I find the need for a dedicated tester creating and running automated functional tests absolutely essential.

    Great food for thought though, especially your last comment about unit tests growing into acceptance tests.

Leave a Reply

Your email address will not be published. Required fields are marked *

*