First and foremost, I’m not a complexity/chaos theory expert, but I find the subject fascinating and have done some light reading on the subject. But I’m finding that reading even pop books on complexity really help me understand the underpinnings of why Scrum seems to work so well for many organizations. As a primer, I recommend the book Surfing the Edge of Chaos which provides a business-centric overview of complexity and chaos theory, with a handful of real life business case studies.
The authors of Surfing summarize the principles of complexity theory concisely as:
- Equilibrium is a precursor to death. When a living system is in a state of equilibrium, it is less responsive to changes occurring around it. This places it at maximum risk.
- In the face of threat, or when galvanized by a compelling opportunity, living things move toward the edge of chaos. This condition evokes higher levels of mutation and experimentation, and fresh new solutions are more likely to be found.
- When this excitation takes place, the components of living systems self-organize and new forms and repertoires emerge from the turmoil.
- Living systems cannot be directed along a linear path. Unforeseen consequences are inevitable. The challenge is to disturb them in a manner that approximates the desired outcome.
The authors then place these principles in the context of business. In nature, any organism that isn’t constantly changing is eclipsed by predators and the ever challenging environment in which it lives. I’ve found this to be generally true in business. As a business man, I have learned the hard way that as soon as I start feeling the comfort of a status quo, some new threat is likely lurking. The authors contend that successful businesses never stay still for very long; they are periodically morphing their strategies and making moves to stay ahead of competitors and on top of the market.
This near constant motion places the company in a state the authors of Surfing call “the edge of chaos”; a periodic injection of chaos keeps the organization fit and on its toes. Too much chaos, though, will overwhelm the organization. So the success pattern looks like periods of upheaval and change followed by brief stabilization periods where the organization catches it breath.
But how does this relate to software and Scrum? Good question! Scrum was actually designed to simulate this oscillation between periods of chaos and stability.
- Chaos: the product backlog is constantly evolving and churning, always adapting to new business needs and inherent changes due to feedback from sprint reviews past. This manifests physically as changes in the priority of individual product backlog items.
- Stability: sprint provide safe-haven from the chaos of the product backlog. The sprint planning meeting lets some chaos in, but Scrum rules are firm on keeping any further changes from entering the sprint. This gives the team a period of stability (the sprint) to make progress toward a short-term goal without incessant distraction and interference.
- Self-organization: the team self-organizes to meet its goals, fueled by the recent injection of chaos and bounded by the relative stability of the sprint.
- Emergence is an oft observed phenomenon of self-organizing systems. This is an elevated level of performance beyond that which is typical for individuals. Scrum teams often exhibit extraordinary productivity gains.
- Chaos again: at the end of each sprint, chaos is again injected into the system as the product owner inserts new requirements and changes priorities to keep the evolving software system fresh and relevant in the face of encroaching change. And so the cycle repeats.
I hope this sheds some light on relationship between Scrum and chaos/complexity theory, even if my knowledge of complexity is cursory. In general, complexity theory is in line with the Scrum idea that “linear” thinking is outmoded in the current, highly complex business era. It has been replaced by empirical models, where self-organization is used to harness emergent behavior and novel solutions to complex problems.