Metrics Gone Bad

While providing operational support to the expansion stage software companies in OpenView’s venture capital investment portfolio in the areas of product management and development, we try to help management teams and their staff set up metrics to measure progress, performance, and identify areas to improve.

Typically, I see teams either have no metrics, trying to pilot themselves blindly or on intuition, or have too many metrics or the “bad” metrics. Too many metrics create noise, and can make it more difficult to zero in on the things that matter and act.

So it is crucial that you select metrics that you absolutely need to understand what’s going on, where the issues are, and what to do about it.

How to Select Good Metrics (and Avoid Bad Metrics)

When selecting the metric, ask yourself, “So what?” That is, if this metric shows X, what will I do about it? What about Y? If you can’t answer that question, consider not collecting this metric.

In my view, a “good” metric is one that tells you something useful without causing harm. This includes answering the following questions, among others:

• Is a process that you need to be followed being followed (in general and by individuals)?
• Is work that you need to be done being done (in general and by individuals)?
• Are you getting the outputs you expected?
• Are you and/or your team getting better or worse at what you do?
• Where are the areas for improvement and how can you improve?
• Do you have a problem?
• If you have a problem, where is it? And what is it impacting?
• If you do X, what will happen to Y?
• Should you go with A or B? 

“Bad” metrics are those that:

• Are misused
• Take more time and resources to collect and review than they add in value
• Create perverse incentives, causing teams to act in a way that is actually harmful to the business
• Diminish a team’s passion and morale, hurting their productivity and quality of output
• Are liable to be inaccurate or are easy to manipulate
• Don’t actually tell you anything
• Don’t tell you what you think they’re telling you

Many metrics are “good” when used one way, and “bad” when used in another way.

For example, let’s take velocity in Scrum. It is useful for a team look at its velocity to indicate if they are improving their performance. It is also an extremely useful tool for release planning.

It is a “bad” metric for management to use to measure and evaluate a team’s (or even worse, individual’s) performance and manage them to it. It is especially “bad” to compare teams’ performance by velocity.

This is because velocity is a measure that is completely controlled by the Scrum team, and it is a measure that only captures a team’s speed on its own terms. It doesn’t capture anything else, like quality for example. And since it is different for each team, it cannot be used as a comparison tool.

As a result, if management suddenly starts hammering the team for higher velocity, you can get the following counterproductive results:

• The team starts to frantically work harder, producing lower quality software or burning out
• The team changes the definition of Done to allow for faster velocity, but lower quality product or product that is actually not finished and ready to ship
• The team changes its reference stories, doing the same amount of work but clocking a higher velocity
• The team loses its passion for driving higher velocity and doing better, leading to worse performance, as its intrinsic motivation to improve is replaced by extrinsic motivation and stress from management, but in general, for a lot of work, especially “thinking” work like software development, intrinsic motivation, which is a passion that comes from within the worker, is generally a better motivator and driver of performance than extrinsic motivation, such as punishment or reward from management. Furthermore, extrinsic motivation reduces intrinsic motivation)

And if the team starts doing any of the above, and seeing velocity as a tool for management, they can no longer use it as a tool for accurate planning or for measuring their own progress, on their own terms.

A great talk I attended at Agile 2009, Metrics in an Agile world, was given by Rob Myers and James Shore and focused on the pros and cons of metrics and several times concluded that while some metrics are valuable to teams for informational and self-management reasons, they are horribly dangerous in the hands of management and must stay hidden at all costs, or best not to be measured at all. 

The talk featured a 30 minute therapy session where dozens of attendees told horror stories of how ill-considered metrics demanded by management led to absurd behavior and failure. Balanced Score Cards were deemed as ineffective at best, damaging at worst.

In short, metrics are valuable, but be careful on what metrics you choose and how you use them.