The gift of failure

Scrum will help you fail in 30 days or less. — Ken Schwaber, c2001

I spent some time talking about risk and failure in my last post, noting the pathogenic fear of failure endemic at most waterfall development organizations. To a person conditioned to such an environment, the above quote probably appears nihilistic or dangerously clueless. At traditional organizations, failure is hidden, denied, rationalized and otherwise treated in ways that ensure its continued reappearance in the most destructive ways. In other words, the culture of fear of failure prevents most organizations from using it not only to learn and improve, but also to innovate.

A powerful benefit of Scrum is that it provides an environment to fail safely, in small ways, and offers plenty of feedback loops to make sure you glean maximum learning from each fresh failure. Which is exactly the opposite of waterfall, which in isolating disciplines, creating lossy knowledge handoffs, and lengthening feedback loops, virtually guarantees that failures are big, messy, and happen too late for much good to be made of them, at least for the project in which they occur.

By keeping failures small, we defuse much of their danger, thereby increasing their relative enlightenment value, because arguably it is through failure that most learning and innovation occurs. Angry Birds, for example, was the 52nd game the company created. All its predecessors more or less failed, but gave the company valuable feedback on what didn’t work. For contrast, take the sequel to the popular 1996 video game Duke Nukem (remember Duke?). They absolutely had to get the sequel right, at all costs. After twelve years with no release, it cost dearly indeed. And when it finally came out, nobody cared.

Agile isn’t academic. Still, beware the ‘pragmatists’.

There’s a perception that agile is somehow academic – that it was cooked up as part of a PhD dissertation, or is the rarefied result of years of private funding and fevered intellectual inquiry by ivory-tower eggheads at some software equivalent of the Brookings Institution. Unfortunately for anyone subscribing to this misperception, the exact opposite is true. The folks that came up with agile practices did so out of long experience and frustration with the failures of the dominant method of software development, waterfall. And ironically waterfall itself owes its popularity in large part to the seductiveness of a theoretical approach.

Nevertheless, the perception persists. A disappointingly common result – used in defense of reluctance to shed unproductive habits, roles, and expectations preventing organizations from being more agile – is an appeal for a need to ‘be pragmatic’. This is odd, because agile is the most pragmatic approach to software development anyone has yet articulated. It doesn’t conflate the inherently chaotic, unpredictable, creative process of building software with a stable, predictable process like building a house or a bridge. It doesn’t confuse the subjective and infinitely changeable – even personal – success criteria of most software development with the objective and immovable criteria of say, mathematics or physics.

Unfortunately none of this means it’s easy to implement against the inertia of existing structures and decades of embedded assumption. But at what point is ‘being pragmatic’ simply an excuse for not trying to change?

In the face of repeated failure, wouldn’t the pragmatic approach be to try something new that might offer a chance of improvement? Most large organizations investigating agile are in a crisis state of perpetual project failure already, so they’re not actually risking much except the ability to blame the usual suspects. It’s not the risk of failure they fear – It’s the discomfort of change that’s the scary part. It’s the pain they know versus the one they don’t.

So many react to agile like the Frenchman in the joke about an American and a Frenchman struggling to craft a solution to a complex problem: The American drafts a straightforward but unconventional solution and passes it to the Frenchman, who examines it, frowning and puzzling over it at length, rereading it again before shaking his head and passing it back, saying: “This solution works in practice, but I’m afraid it doesn’t  work in theory.”

5 Pitfalls to Avoid in Agile Transformations

Over the past several years, I have listened to new clients talk about the challenges they face with their Agile adoptions. At some point in the conversation the client inevitably says something like, “You have to understand, our organization is unique.” The irony is that the phrase is usually followed by a description of an environment that sounds pretty much like one found at every other company just getting started with Agile. It is natural that every client would view their organizational challenges as unique because they’re providing unique products and services. However, the fundamental challenges that organizations have at their core generally boil down to people issues. Human beings require similar environments for success – largely independent of the kinds of work being performed. After engaging with several hundred clients along these conversational lines, I have identified some of the most common pitfalls organizations run into during their initial efforts to become Agile.

1)      Reading a Book on Agile and Thinking It’s All You Need to Get on Track

Reading about Agile is useful and absolutely encouraged. However, reading alone does not provide the more esoteric elements of an Agile transformation. I have spoken with countless team members at large organizations who have read a few books and started doing Scrum without any professional guidance. Every single one of them has told me some version of the following: “We read some books and got started with Scrum teams and sprints, and after a year, we don’t feel like all of us are on the same page as to what Scrum should look like.”

People go to therapists because it is impossible to objectively analyze one’s self and provide appropriate guidance. Shouldn’t this be the same for organizations? My colleague, Angela Druckman, uses a more universal metaphor: Professional sports teams are made up of highly skilled athletes, but still require a coach to perform well and win. If highly paid, highly skilled professional athletes still require a coach, why shouldn’t we expect our organizations to require something similar to achieve greatness?

2)      Breaking the Rules Before You Give the Rules Time to Work

Ken Schwaber, Scrum co-founder, once said, “Scrum is like chess. You either play it as its rules state, or you don’t. Scrum and chess do not fail or succeed. They are either played, or not.”

Scrum exposes various kinds of organizational dysfunction, but does not tell you how to resolve them. Often teams will modify the Scrum framework because it does not conform to an established business practice. This defeats the whole purpose of Scrum right from the outset. Scrum is supposed to feel uncomfortable and chaotic at first because the point is to uncover organizational issues that need addressing. Changing the rules in an attempt to fit them to existing processes leads an organization into a long period of doing “sort-of-Scrum” in which no discernible benefit is achieved and which usually causes the organization to quit after a year or two because “Scrum didn’t work.” Scrum works just fine, but only if you follow the rules. One of our clients who led a global Agile transformation effort involving hundreds of people on three continents likes to say, “I’m not smarter than Scrum, and neither are you.”

3)      Not Involving Anyone from the Business Outside of IT

High-performing Scrum teams have been compared to jazz ensembles because communicative self-organization is the hallmark of a good improvisational jazz. However, most organizations today are more appropriately characterized as orchestras in which the musicians play their parts as written on the sheet music in front of them. Having a start-to-finish plan is necessary for orchestras because their performances are replications of a completed product.

The orchestral methodology is impractical for nearly all business organizations, but many try to do it anyway rather than operating as a jazz ensemble. Imagine if an orchestra’s percussion section started playing in a different time signature in the middle of a piece. The whole performance would go off the rails. In a good jazz group, though, if a player abruptly changes the time signature, the other players will adapt and go with the flow.

Transitioning to Agile is a significant change that should affect the whole organization. Not everyone in the organization needs to be on a Scrum team, but everyone needs to be familiar with the values and benefits of Agile in order to play their parts in a communicative, collaborative way.

4)      Lack of Commitment

An Agile transformation requires organization-wide effort. At the outset, there is often a period of chaos and uncertainty which must be endured. This is something many beginning Agilists don’t understand and it can get them pretty freaked out once they realize what they’ve gotten into. If a company is not committed to learning about Agile, how to practice healthy Scrum, and how to resolve the organizational issues that are exposed, then there is no point in even trying to attempt an Agile transition. Yes, there will be people who do not want to learn yet another process. Yes, there will be people who are uninterested and unengaged. Find the people who are engaged, get them together and continue to push for more commitment, especially from upper management.

5)      Unwillingness to Mentor Talent to Achieve Long-Term Quality

An unfortunate practice in many organizations is the constant attempt to do too much with too little. This can be particularly short-sighted when it comes to personnel. Have you ever heard the phrase, “Stepping over a dollar to pick up a nickel”? When training budgets are cut and mentoring activities are shunned because management considers mentoring to be paying two people to do one person’s job, the long-term consequences can be disastrous. People will become stagnant in their skill sets, get frustrated and eventually move on to an organization where their natural desires to learn and produce will be fulfilled. If a company does not foster a sense of worth and appreciation for their employees and does not invest in training and mentoring programs, the company is bound to suffer losses large and small, sooner and later.

*****************************

This is by no means a comprehensive list, but it is something you can use to analyze your organization’s approach to an Agile transformation. Try self-assessing your organization on each of these 5 pitfalls and then prioritize them. If nothing else, it could get the right conversations going within your organization.

Agile Beyond Software – Agile for the Whole Organization

In my last few Agile Basics classes, I have noticed a welcome trend: People with roles outside of IT are attending classes in conjunction with the development teams. People from sales, marketing, accounting, HR and even executive branches have started to recognize that learning about Agile has a positive impact on the whole organization.

Agilists have long held that Agile transformations require buy-in from entire organizations, not just the IT and development teams which have adopted the most popular Agile framework, Scrum. To understand why this is, it is important to go back to Agile’s roots and understand what exactly Agile means.

As the Agile movement has expanded, misuse of the word itself has become common. It is easy to say, “We want our company to be more agile.” This stands to reason. No one wins over shareholders by saying something like, “We prefer to remain beholden to our processes and plan-driven approaches instead of focusing on innovation and high-quality delivery.” Typically then, an executive will decide the business needs to “be Agile” so that the company appears hip and lean to shareholders, and leaves it to the rest of the business to figure out what Agile is and how they’re supposed to do it.

Let’s go ahead and clear up a couple of common misconceptions:

First, Agile is a set a values, not a process. This is a critical point to understand, so I’ll go ahead and say it again: Agile is a set of values, not a process. There are Agile practices that enable Agile values to take hold. Agile does not prescribe any specific practices, but instead presents Agile values through the Agile Manifesto and the 12 Agile Principles (http://agilemanifesto.org/). An Agile organization adheres to these values and probably uses a combination of Agile frameworks such as Scrum and XP to foster an Agile environment. Agile prescribes human interaction, complete pieces of working product, collaboration and response to change over all the ways we’re used to documenting, planning and channeling product development by skill set and development cycle. Agilists understand that product development comes with inherent uncertainty and inevitable change to the original plan. Agilists welcome change and consider uncertainty a platform for creativity.

Second, Agile exposes organizational dysfunction, but it doesn’t tell you how to solve the problem. Agile practices can often prompt more questions than answers. Agile practices do not pretend to be silver bullet solutions. They are simply methods people have used to successfully ferret out organizational waste, technical debt and micro-management, while also focusing on high-quality, high-value delivery.
Organizations that have done the hard work of examining and adapting their internal structures to adhere to Agile values have had great success with higher quality, more frequent delivery. Organizations that have expected ingrained business processes to stay in place while also working toward a more Agile environment have not.

A problem many organizations are running into is that their Agile development teams are delivering high-quality product at such a rapid pace that the organization’s non-Agile teams cannot keep up. As an example, take the following scenario: Marketing is working on web pages based on the original, loose plan and does not know the development team was able to get more features into the release than originally anticipated. Legal has no idea the License Agreement needs an addendum as a result of the development team’s use of some new open source tools. Salespeople have been telling customers about the planned features for the new upgrade, but do not realize they could be selling even more desirable features to clients.

What is the point of delivering high-quality, high-value product based on consistent customer/stakeholder feedback if the parts of the organization responsible for marketing and selling the product rely on outdated and demonstrably false estimations and plans? Having Agile development teams working in conjunction with Agile marketing and sales teams is the next logical step for many companies in the midst of Agile transformations. For a business to reach its full potential as an Agile organization, it must jettison its outmoded business practices and bring all of its teams into the fold of Agile values.

How Did Scrum Become a Straitjacket?

The ideas behind Agile (before it was a buzzword) largely arose from practitioners looking to eliminate painful or silly management practices between them and their craft. When I first did Scrum, it was more fun than any way I’d worked before. As a member of a self organizing team, we negotiated Sprint Goals with the PO and collaborated to meet them our own way. Scrum’s freedom from micromanagement opened the door to learning Agile technical practices that kept the code enjoyable to work on. So the autonomy, the social environment, and the state of the product itself made it fun. If it’s not fun, it’s not Scrum. This is the way I try to teach it in my classes anyway.

It makes me sad to see that Scrum and Agile are often implemented as just another way to constrain people and squeeze work out of them. Innovation, intrinsic motivation, teamwork, and employee retention do not thrive in those environments, so eventually the companies that do this won’t be able to compete. But in the meanwhile, they’re sure causing a lot of pain in the name of Agile. One person with strong feelings about this wrote the following to me via LinkedIn:

Michael,

I like what you said, I appreciate the way you said it, and I am glad you were the one to respond to my comment on xp and scrum. In the agile world (and I demoted the “a” a long time ago out of frustration) all the pigs are the same. Only in the way scrum has devolved, some pigs are better that others.

I was hooked on XP after reading my very first agile book (…Embrace Change). Almost a decade later I am jaded, confused, saddened, and have lost the will to challenge the status-quo in agile. In one company I left, the scrumlord actually enforced the waterfall method for project management. (Rally lists and charts were far more important in the presentations to the client than our application or hearing their feedback.) And the company was proud of their “agile” teams. And pair programming, refactoring, TDD, revision control, version control, continuous build and all the rest was forbidden by the scrumlord. “Waste of time,” he always told the CEO.

Why has this happened?

My first guess is that the professional manager found a way to insert themselves into agile — in a particularly contrary move to the spirit of agile. My second guess is that to be a good scrumlord, you really do need to know the technical side and there are too few middle managers who are capable of that. Lastly, I would say that no practice, no methodology, and no trend can do away with the ego of the middle manager.

But, as I say, I’m jaded.

I suppose it was the end of me at that fateful job but I knew it was time to go when I told the scrumlord that a team with only XP can still produce a superior code product with few errors and little technical debt. But a team with only scrum cannot. It would have just been an opinion except I was proved correct in my hyperbole and they had to punish someone.

You use words in ways with which I was once familiar. It is the same melody that drew me to agile so long ago. Is it still being heard in some fair corners? Not where I’ve been. And I don’t look for it any time soon. If you truly live the words you write, then your team is fortunate. And your clients even more so.

So long as the command and control business model is more valuable to corporate leaders than their product and their people, I will keep spelling agile in the lower case.

[name omitted]

After I reported on this email, my friend Kane Mar sent another example of a someone who’s been burned by the discrepancy between what we say and what we do: http://www.halfarsedagilemanifesto.org/

One way to check whether your Scrum implementation is anything like what we had in mind is to go through the ScrumMaster Checklist. The items you mark with deltas may be organizational impediments worth discussing with the highest level person in your company who will listen. If they still won’t listen, please make the best of your short lifespan by finding people to work with who are more committed to human innovation.

–mj
(Michael)

Building a Better Backlog Q&A

On March 5th, I presented the webinar, “Building a Better Backlog: Strategies for Long Term Success in Agile Development.” In the session, I shared strategies on how to build and maintain a good product backlog by describing the overall concepts and techniques for backlog management and how each of the project contributors can contribute to its overall effectiveness.

Specifically, I covered:

  • What a product backlog is and how to create product backlog items
  • How to write good user stories
  • How to estimate product backlog items
  • How to groom the product backlog, and
  • The importance of treating the product backlog as a living artifact

During the webinar, we encouraged our attendees to interact with us by asking questions for the live Q&A session that we had at the end of the presentation. As a result, we received many great questions that we just couldn’t get to during the live event so we have provided answers to the audience’s questions below.

If you missed the session live, you can watch the recording and download my slides.

Don’t forget to register for our upcoming webinars! CollabNet offers multiple webinars a month that cover a variety of topics related to software development.

As promised, here’s the follow up to some of the live audience questions we received:

Q: What was that 2nd user story you mentioned?  It was a good example – As a user I want to login to a website so that I can save my preferences…
A: We were pointing out how important it is to listen for not only what is said in a user story but also what is inferred.  Compare these two examples:

  • “As a user, I want to log in to the website so I can use it.”
  • “As a user, I want to log in to the website so I can save my preferences and don’t have to reenter them each time I visit.”

In the first story, the implication is if I don’t log in I cannot use the website at all.  The second story, however, implies that I can use the website without logging in but if I want it to remember me (and my preferences) I need to log in.  Agile teams need to get good at identifying not just what is stated in a user story but also what is inferred.

Q: Is everything in the Product Backlog in some “User Story” format?
A: While it is certainly not required to use the user story format to create product backlog item, it is a very popular and useful way to describe requirements.  However you choose to write your product backlog items, do try to keep them succinct for readability.  You can link backlog items to more detailed documentation when necessary.

Q: Does the level of detail contain something about the how?
A: As we said, user stories are about the “what”.  Tasks, created by the team, are about the “how”.  Acceptance criteria is right in the middle, with some characteristics of both.  Likewise, supporting documentation, such as design documents, may have strong elements of “how”.

Q: Are there instances where a backlog item is labeled as “L” or “XL” without it being an epic?
A:  I consider “L” and “XL” to be valid story sizes.  Anything bigger than that, however, would be an “epic” and the team would need to have that story broken down into more granular pieces before they could even consider including them in a sprint.

Q: What do you do when you have a Dev Manager that wants to use real numbers (in days) for User stories rather than story points? Mostly because he has 20 years in the business and feels he knows the workload well.
A: How does the team feel about it?  If a team is keen to try relative estimation (which they often are) as ScrumMaster I would work with the manager to help him understand that what really matters is getting the team to the point where they make sprint commitments they can successfully fulfill.  This is what will win over the Product Owner and, for that matter, the rest of the business.  I feel you are describing an individual who has not yet internalized agile values.  As long as the team is told how long work “should” take, they won’t truly self-manage.

Q: If the Product Owner role is a full time job, and you have a Product Owner that is not available full time – how do we make Scrum work? The Dev mgr has been grooming the backlog and the developers have been writing user stories in the Product Owners absence. This has been challenging for the team to understand the Product vision and to have “non-technical” user stories. Any advice?
A: A couple suggestions. First, stop doing this person’s job for him.  As long as you give the Product Owner “excessive” help, he will never do the job.  If you have not done so, start scheduling regular Story Time meetings to show him how to develop his backlog.  Many Product Owners fail because they have received no training and don’t really understand what is expected of them.  Next, work with the organization to make sure the Product Owner is held accountable for actually doing the job.  If he sees it as something to be squeezed in during his spare time, he will never give it the thought and effort it requires to be successful.

Q: How do you manage dependencies between stories?
A: Discuss them.  As a Product Owner, as soon as I did a preliminary ordering of my backlog the very next thing I wanted was feedback from my team.  Often, they has ideas for moving user stories around to reduce dependencies or gain some efficiencies.  I had final say on the order of the backlog but their input was priceless to me.

Q: Can we Use Case in place of User Story? What is the difference?
A: Use cases, at least most of the use cases I have worked with, a sometimes large, multi-page documents.  A user story is a succinct single sentence.  Use cases can support user stories but they are not the same thing.

Q: How do you prioritize based on “business need” driven by PO vs Dev need to develop high risk feature first.  Looking at risk/value grid!
A: The short answer is: through discussion.  A good Product Owner never works in a vacuum.  He wants input from all stakeholders and that includes the team.  But because the Product Owner is the one ultimately held accountable for the product, he has final say on priority.

Q: Who facilitates the Story Time session?  And who be involved?
A: The ScrumMaster facilitates the discussion, since SMs are often master facilitators!  However, the Product Owner should set the agenda.  The goal of the meeting is to improve the product backlog.  He should be intimately familiar with where it needs improving.  Stakeholders may attend this meeting at the PO’s discretion.  This meeting is a great tool for gathering requirements.  And, of course, Story Time meetings, like all agile meetings, should be time-boxed.  Regular meetings of 60-90 minutes are much more effective than one marathon session.

Q: Are there any guidelines around estimating the size of user stories, should the sprint length be taken into account?
A: Teams will come to consensus remarkably quickly on what, for example, constitutes a “medium” story.  Leave this up to the team.  But all stories must fit within a sprint.  If they do not, they are “epics” and must be broken down.

Q: How do we organize support-related work (assuming it’s performed by same team)?
A: Some teams reserve capacity in their backlog for support work, based on historical data.  But a necessary discussion to have when these “emergencies” pop up is “Can this wait until the next sprint?”  True, sometimes the answer is no.  But if you are doing, for example, 2-week sprints sometimes the “wait” would only be a few days.  Then the work can written in a user story and added to the backlog.

Q: What is the most effective way a ScrumMaster could do in order to help product owner to maintain and clean up product backlog?
A: Show them what to do but don’t do their job for them.  Writing the Product Backlog for your Product Owner only teaches him that, if he ignores his job, you will do it for him!

Q: As presented earlier the acceptance criteria should be negotiable. Does it mean that during demo meeting PO can change one of the acceptance criteria?
A: The “negotiating” of acceptance criteria happens in sprint planning, before any commitment has been made.

Q: Who or what types of people should attend the Story Time Meetings?
A: The ScrumMaster, Product Owner, Scrum team and any stakeholders invited by the Product Owner.  On larger projects, the PO may find it useful to have a business analyst there to help document user stories.

Q: If you use t-shirt sizes to estimate, how do you derive sprint velocity?
A: Teams that use t-shirt sizing often use a numeric “weight” for each size to calculate velocity.  The “powers of 2” scale is quite popular:

XS-1

S-2

M-4

L-8

XL-16

In this scenario, a team that committed to 1 large, 3 medium and 3 small user stories would have a sprint commitment of 26 story points.

Q: Can you please elaborate on what should happen during the planning meeting?
A: During the sprint planning meeting, the Product Owner wants the team to tell him how many user stories they are willing to commit to deliver in the upcoming sprint.  To do this, the team and Product Owner must establish acceptance criteria for each story, or what it will take to call the story “done”.  The team will also probably want to do estimation and also to begin task creation.  They may even start to plan the work of the sprint and who will do what.

Q: What are the criteria / conditions for a “groomed” story ready for Sprint Planning meeting?
A: A “groomed” user story is one that is clearly written and of reasonable size, meaning it can fit into a single sprint.

Q: What are the competences required to be a good product owner?
A: Thick skin!  Organizations almost always have more wants than they do money or time.  Product Owners have to make tough trade-offs to get the most value possible given those limitations.  Good Product Owners can make decisions, but never do so in a vacuum.  They know how to prioritize and maximize value.  And they take the job seriously.

Q: Regarding breaking down epics, is that process typically a PO effort or a team effort?
A: the Product Owner will almost always require help in breaking down epics.  If he could do it by himself he probably would have already.

Q: Do you think that all items of the backlog should be estimated or is not necessary for the items at the bottom?
A: It is nice, after teams get some experience with agile, to estimate out a few sprints.  This helps the Product Owner with planning.  Much more than that is probably so inaccurate as to be useless.

Q: Are there any additional pointers for a product with a backlog that is thousands of items big?
A: Close your eyes and delete!  Seriously, you need to winnow it down.  Assign a Low/ Medium/ High ranking to each item.  If it makes you nervous to actually delete items, move the “lows” to an “on hold” backlog.  I personally have never seen an actionable and well-organized product backlog with more than a couple hundred items at any given time.

Q: If product backlog ordering is done in backlog meetings then is this overlapping with the Sprint

Q: Good afternoon Angela, when are you brining PO training to NYC? :-)
A: We currently do not have PO training in NYC, but I will pass along the request for it. ;-)

Q: Assuming you have several released stories, what if the product owner decides that these stories must be changed?
A: Then he can write new user stories for these changes—no problem whatsoever.

Q: We have some great Product owners and then we have some that won’t participate that we are having to be stand-ins for, is there one key thing we can do to make this more successful?
A: Make sure you have the right people in the job and that they have the tools to do that job.  Look at those “great” Product Owners and try to figure out why they are great.  What do they do or have that the others lack?  As an Agile coach, one of the top issues I am called in to help with is Product Owner problems.  When I arrive I often find POs that have not been trained, don’ t know what is expected of them and are not getting help and support.

Q: If you work at an organization where the executives want the final say on the order of backlog items, how would you handle that?
A: The Product Owner has final say on backlog order so, if an exec wants that privilege, she has to take on the rest of the PO role as well.  That usually doesn’t work out and eventually these individuals see they must delegate some authority.  Delegation should not be a foreign concept to the average executive.  Giving someone authority as Product Owner is just a form of delegation.

Q: Any hints about how to recognize if our backlog is “not in a good shape”? (Without reviewing each PBI one-by-one if possible)
A: A mature backlog has smaller, more detailed user stories at the top, less detail at the bottom.  It is “chunked” into bodies of work like releases.  It is full enough to keep a high-performing agile team busy.  If this is your backlog, you are on the right track!

Q: What role does a business analyst play on the product backlog?
A: Business analysts often help Product Owners understand and write backlog items.  They may also write supporting documentation and even help with testing.