How to measure success on agile projects from a customer point of view

December 18, 2007

This topic is a thread on Scrum-Dev (the Yahoo list) these days. Excellent topic, with many good posts. Go and see. This is an involved topic, so I will give some views in this post and more later.

I start with the word “measure.” I am concerned that we are measuring too many things (not good), and the wrong things (even worse). Many people don’t take to being measured like a thing. (People have lots of characteristics and variability, but being a “thing” is something they are especially not good at, or so I have found.)

So, what to do if one wants a simple measure?

The simplest “measure” would be to ask the Product Owner on the team, “Is this team effective and are you guys producing Business Value?”

This is one measure. It might even be binary (yes or no), and it is low cost to collect, except that the Product Owner might give you (let’s say for now, you are a senior manager) more of the truth than you want, and ask you to assist in making things better (and things can always be better).

(As an aside, if the Product Owner were really good, I might accept a yes/no answer. From a less strong Product Owner, I would always want a longer answer, probably with follow-up questions.) It might be a conversation that is forward-looking and action-oriented.

The next problem becomes, “What is Business Value really?” To me, this is the key direction to take this issue (finding a simple measure). (There are other directions we must also take, but for later.)

Let me emphasize this: As I view business, efforts (projects) are successful to the degree that they deliver Business Value. There are no technical successes (“We did what we were asked to do,” “We’re within scope and budget,” “It’s a beautiful system,” etc., etc.). There is only delivery of Business Value — at least somewhere along the chain of value to the ultimate customers.

Therefore: “What is Business Value really?” There are many views on what Business Value is. We have discussed some in earlier posts. It is an important and difficult subject, worthy of much discussion.

Lean says that value is defined by the customer in the context of a specific product (service) at a specific time. That says, among other things, that customers are redefining value all the time. If you take your own personal experience, you know that your “values” (at least in the material world) are changing all the time. Your own wants and needs change, with great rapidity sometimes.

OK, so what do we do about this in Scrum?

Well, Scrum does not prescribe any specific method. (That’s a good thing, because there are so many situations.)

What I often suggest is this — and it seems to fit many situations well. Let’s assume you can determine a dollar amount representing the Business Value of a release or for the whole project. Then, assign 1,000 (or 10,000) “gummy bears” to all the existing user stories (or the equivalent if your Product Backlog is not in stories). You might put those BV Points on one corner of each Story Card. Make sure the BVPs are relative, meaning: A card with 50 BVP has half the Business Value of a card that has 100 BVP.

Then, as each iteration is completed, count the BVPs “done.” Say 1,000 are done after the first iteration and the total is 10,000. And say that the total Business Value is $26 million. So, 1,000 divided by 10,000 says you have completed 10% of the Business Value, or $2,600,000 in Business Value.

It’s a good enough approximation for most situations.

The next problem is if total Business Value changes during the project — which it always does to some degree — and new Business Value is discovered (if anyone is paying attention) usually.

Is that a simple enough answer?

Of course there is much more to say on this topic. I particularly recommend “Software by Numbers” by Denne and Cleland-Huang.

“Things should be as simple as possible, and not simpler.” I think Einstein said that.

Let me deal with one more issue (in this post) that I have often seen. The issue: The Product Owner is not always “good enough.”

This is always a risk, but, in a way, unavoidable. Someone must be a ‘stand-in’ for our customer set. These days, we never get the real final end user to be the Product Owner. (I suppose there are some edge cases, but you know what I mean.) So, the PO is trying to determine the overall Business Value of the product (or our change to the product), and Scrum prefers, to avoid long delays in making decisions, to name only one Product Owner, who serves as that customer stand-in. Of course, one or more people (perhaps on the team, perhaps not) might still assist that Product Owner in determining the Business Value.

Just as the one Product Owner can fail, so can any group of people fail. Scrum has no silver bullet to eliminate failure by people. What it does is make clear and visible where the (relative) failures are. Often that can allow corrections to be made.

Being a good Product Owner is extremely useful. Get the best people you can find, and help each one learn to become a better and better PO.