What to look at, what to look for, and what it all means.

Agile metrics, are lagging indicators of how the work has gone for a team in the past. Financial institutions are fond of pointing out that “past achievements are no guarantee of future performance”. This is true in agile teams too.

At the same time “Yesterday’s Weather”, is a handy rule-of-thumb to help teams spot sub-optimal performance and mitigate it.  After all, that is EXACTLY why we use agile principles: to surface problems early so that we can explore and resolve them as cheaply and as quickly as possible.

I just need to say here…

Going faster and / or going cheaper are only by products of agile working in the early adoption phases. An agile team’s metrics can and should level off as new practices getting embedded.  It WILL NOT continue to trend positively for ever!

These are the 3 most helpful metrics I have found when I am looking for the next most important thing to focus on.

With metrics, we are looking for patterns and trends. Like pixels in a picture, individual metrics are far more useful when looked at in groups than in isolation. In this case most definitely the whole is greater than the sum of its parts.

At NewVoiceMedia, these are the metrics we care most about:

Cycle Time

What are we measuring?

This is the amount of time a story takes from starting it to having it accepted by the product owner.
What do we want to see?

Ideally we want to work towards our cycle time being <5 days. This is because we check our code into the main trunk constantly, even before it is finished. This main trunk goes live with everything every week. If we can get our cycle time down to less than 5 days from story start to story acceptance, we have much fewer problems caused by incomplete code, and less code toggling is needed.

What should we look for?

Stories taking too long to go through this cycle

What could this mean?

  • The Stories too big
  • The stories not well understood before work starts
  • The stories being started and then abandoned
  • Are new stories started before ones in progress are finished? (Look for a high WIP to support this theory).


Work In Progress (WIP)

What are we measuring?

This is the number of stories currently ‘in play’, either for a particular part of the process, or maybe for the whole process. It is always a snapshot in time at the moment you are looking.

What do we want to see?

Ideally we want to see the WIP (Work In Progress) level be less than the total number of developers + testers.

We greatly value pair working at NewVoiceMedia, so it is logical we want less stories in play than people to do them.  A ratio of less than 1 story to 1 team member.

It is worth mentioning again, that context is important. Each team’s optimal WIP can be different.  It is also feasible that a team’s optimal WIP might change over time as the type of work changes.

What should we look for?

Stories (multiple) that have been started by the team, but not yet finished.

What could this mean?

  • We want to aim for great flow with our work, a high number of stories in progress at any one time implies we do NOT have great flow.
  • Stories are blocked – there are infinite reasons why a story may be blocked, but you will see that nothing happens with it for a long time.
  • Are stories being started and then abandoned?
  • Are new stories being started before ones in progress have been finished? (this theory can be supported by evidence of a high Cycle Time)



What are we measuring?

The principle use for velocity is to estimate how long a particular set of work is likely to take.

You can only predict something if it is predictable (otherwise it is a guess rather than a prediction).

Predictable velocity relies on consistency.  Consistency of team members, type of work, number and likelihood of interruptions, working hours, working days per week etc.

Given all those being consistent we then want to see consistency in velocity. Even then, with all these things being consistent, velocity is only an indicator, and is the least scientific of all the metrics we use at NewVoiceMedia. That said, it is certainly much better at estimating a finish date for a collection of work than anything else we have.

What do we want to see?

Ideally we want to see consistent velocity over time for a team, although inconsistency of velocity can also be very useful to show underlying problems.

What should we look for?

A steady velocity will tell us when we can reasonably expect to deliver to the business.  Of course, the only time we are certain when we can deliver to the business is when we have just done so!

A steady velocity can also indicate that

  • we are working ‘in the zone’
  • we are developing at a sustainable pace
  • Other things that effect velocity are probably ok too. (I have never yet seen stable velocity for a team that has serious problems).

What could this mean?

I think this is the most interesting question here. The patterns in velocity over time can be hugely insightful to a scrum master or team.

When velocity is not stable, you can learn by observing what is happening: for example, imagine you are running 2 week sprints, but there is a velocity pattern

This mythical team seem to be really working in iterations 3 times as long as they are claiming.

This mythical team seem to be really working in iterations 3 times as long as they are claiming.

This tells you all is not well.  Maybe you are starting some stories are that are too big, taking consistently longer than a full iteration to get complete.

This often happens when you start a story early in the iteration, but can’t quite finish it until the following iteration.  Asking before you pull any story into play ‘can this be done in a single iteration?’ can be useful here.

Another reason may be that your team functions are quite siloed. Developers might do all their development work before passing over to the testers. The testers in turn may then have a large amount of manual testing to do (automated testing will shorten this, but it is hard to make it go away without fixing the problem).

Alternative to Velocity

One last point: velocity it is a fickle measure (as illustrated by the caveats above).

It may be that the nature of the work coming through the team means that velocity will never be consistent. It might be that the team deal with a lot of bugs or chores as often as new features – perhaps in a more DevOps type team.

I have found the following worked well when that is the case:

  1. Shorten the feedback loop – we dropped from 2 weekly to weekly iterations.
  2. We abandoned estimating in the way usually taken by a team, and instead asked ourselves only:
  • Can we finish this piece of work by Friday? (ie within a single iteration)
  • Are we sure this is NOT a task, and part of a bigger story?

After that, we stopped counting story points and started counting work items.   The number of work items (bugs, chores and stories) gave us a far more significant measure of our progress through the work than story points alone could have done.

To sum up…

Ultimately, agile working is about finding what works, looking to do things better, and understanding when you’ve found the root cause of a problem. The tools and methods used to do that need to work for your team, your domain, and most importantly your customers.

I am not the first person to say this is very simple to understand but very difficult to do well.  We all learn more about doing it well every single day.