Today, I chose to write about one of those questions that I’m asked over and over again in my CSM classes and coaching opportunities: “Why is management so different in an agile organization than in any other?” I decided that I was going to find out.
My search for answers took me back to whitepapers and research studies that go back to the initial days of Scrum and the concepts behind Takeuchi and Nonaka’s New, New Product Development Game as published in the Harvard Business Review. What I found, though not new, really had a new meaning for me based on what I’ve learned since diving into the oft-stormy waters of agile development.
To start, let’s clarify something. The problem isn’t that management in an agile environment is different. The problem is that management in SOFTWARE DEVELOPMENT needs to be different and, prior to agile development, we’ve been trying to squeeze a square management peg through a round hole in the organization.
Starting in the 1960s, as software development organizations emerged in businesses, first as Data Processing (DP) or Management Information Systems (MIS) departments and later as Information Technology (IT) departments, there was little to no change in management approach. Frederick Taylor’s view of employee disenfranchisement was coupled with combinations of under-considered or over-blown quality management processes and IT management was born.
It didn’t work, but it was born nonetheless. Oh, well….we had to start somewhere. Don’t get me wrong, there’s nothing wrong with most IT managers; it’s just that they’ve been taught to use the wrong tools in the wrong manner. We’ve been managing software development projects and software developers incorrectly for years!
So, how should we be managing software projects and software developers?
Enter complexity theory and the concept that systems (and we’ll consider groups of developers working together to build a product as a “system”) come in three flavors: stable, unstable, and chaotic. Stable systems are like the orbits of planets – even when impacted by changes in their environment, they generally re-establish equilibrium on their own (and yes, to those of you who want to argue this, I understand the concepts of orbital drift and the fact that our moon is actually slowly escaping Earth’s orbit – let’s not quibble over a quarter inch a year, OK?).
On the other hand, unstable systems last a brief period, failing as soon as their equilibrium is upset by an external effect. Picture your two-year-old learning how to walk (or just about any newbie on ice skates) – it takes only a slight loss of balance and down they go. That’s what we call an unstable system.
But – and this is BIG – unbeknownst to most of us, there is actually a third type of system. Prior to the introduction of complexity (chaos) theory, a third system wasn’t theorized. But now we have the chaotic system, and I don’t mean “chaos” in the classic definition of “out of control.” Rather, a chaotic system embraces properties of both stable and unstable systems, following rules of its own while remaining wholly unpredictable.
Additionally, chaotic systems are highly sensitive to initial conditions. Those initial, very complex conditions, known and unknown, cause the chaotic system to respond with wild behavioral perturbations that seem random and uncontrolled. Best example I can give you – the weather. Even with years upon years of observations, our meteorologists still can’t reliably predict the weather more than 24 or 48 hours in the future (unstable). However, a monsoon has never struck Toronto and the Nile River has never frozen (stable).
Interestingly enough, the activity of software development seems to exhibit the same characteristics of a chaotic system. It obviously follows some clear principles – we know what we mean by “analysis,” “design,” “unit testing,” and “refactoring.” At the same time, there are many, many initial conditions, from known and unknown risks inherent in every feature we build (or attempt to build) to known and unknown defects that will directly affect our developer’s ability to write working code. However, much of how we actually build each application is fraught with uncertainty. In fact, there are documented principles that specifically address this.
Ziv’s Uncertainty Principle says that uncertainty is inherent and inevitable in software development. Watt’s Humphrey has postulated that, for a new software system, the requirements of that system will not be completely known until after the users have used it. Wegner’s Lemma also suggests that an interactive system can never be fully specified, nor can it ever be fully tested.
So, from day to day as we develop our applications, we truly have no way of knowing if tomorrow holds a project–zapping surprise for us, or if it’ll just be another day at the office. Couple the uncertainty of software development with the uncertainty of the market, the time pressures from customers, and the variety of equipment on which our applications can be run and you have a chaotic system that is pretty much like the weather. The further forward we try to predict (plan), the more wrong we are.
But wait, Frederick Taylor says that workers will do as little as possible and are generally not smart enough to know the best way to do their job. Experts are supposed to show everyone how to do their jobs and then they’re just supposed to do it. And doesn’t that feel like how IT management has been run for most of the last forty years?
So, why is management so different in an agile organization? It’s different because software development is a chaotic system. It defies prediction and long range planning, yet we continue to try to do long range planning, estimation, and analysis. Let’s be realistic – no one is surprised when a project overruns it’s budget and/or schedule.
Proper software development requires that everyone be able to make decisions, collaborate, and change direction based on the current realities. Management cannot assume that plans remain unchanged and rather than trying to manage to a plan, must change approach to manage to current opportunities instead.
In an upcoming blog post, I’ll discuss more about management responsibilities in an agile environment.
Download the PDF version: We’ve Been Managing Software Development All Wrong_blog