CHOW #113- Tracking Productivity using Story Points Delivered and Effort

KillerApps Solutions is a software products and services provider in the banking, cargo, and travel verticals. Product teams in India have been working for the international market during the last two years. Teams have delivered two releases of products in each of the verticals during the last year but there were major quality issues as well as scope reductions that had to be made to achieve timeline goals for each of the product releases.

KillerApps management has implemented Agile/Scrum from the start with an elaborate process to guide the teams on product development and release engineering. Given the challenges associated with rapid ramp-up of teams as well as the lack of a good process culture and discipline, the US Management team has decided to implement a common productivity tracking mechanism across all teams to measure how teams are doing and drive improvements.

The measure the management team has decided to track productivity of teams is Story Points per person per day. In order to ensure uniformity across teams, they have also decided to standardize on story point definition – with 1 SP equals one average programmer day (what an average developer and tester can deliver in terms of functionality working together per day). Teams are asked to size stories using this standard and story points delivered in each iteration and the capacity of the team in person days used to measure the required metric for each iteration for every team. In addition to this defects leaked from one iteration to the next are also tracked to ensure quality is not compromised. These two metrics in addition to the post release defects reported by customers serve as the primary dashboard for management to track how well teams are doing with respect to each other.

1. What do you see are some of the challenges that the management team is likely to face with implementing this productivity metric? Where do you see this initiative heading?
2. What would be your recommendations to the management team at KillerApps to deal effectively with productivity goals?

Suggested Solution:

The productivity metric defined here is story points per person per day. If this needs to be implemented, the first challenge is ensuring consistency of story point measurements across teams. With the definition of a story point that they have, it is extremely difficult to get any level of consistent measurements. How do you define an average developer or tester? Is it based on their experience or skill? How do you ensure everyone in the team – especially with so many ramp-ups happening – understands what an average developer or tester means in the same way? Even the denominator in this case – which is person days of effort – is not likely to be uniform across teams given that the experience level in each team is likely to be different from the others.

If this initiative is pursued and teams are compared on this metric, you could end up influencing teams to change their behaviour to give what you are looking for from a management perspective. They could use story points sizing data to report favourable metric numbers given the pressure on them to do so. Even the effort data may not get correctly reported. As a coach, I have seen this with many organizations and the end result is that the numbers and metrics generally look good making you believe you are getting what you want but the outcomes are not what you want, outcomes typically include value and quality of what is delivered and not team productivity.

Given this what would our recommendations be to the KillerApps management team? When you define metrics, look at what you need as outcomes. In this case value of what the team is delivering is a lot more important than story points per person day of effort and quality of what is delivered is higher priority than story points delivered. So if the focus is too much on productivity, you end up forcing teams to behave in a way that you will get what you ask. What is the need for a common definition of a story point? What is the need for comparing two teams on how well they are doing with respect to each other? Isn’t it more important first for a team to feel that they are at their productive best as a team and then improving from iteration to iteration? Two teams are invariably not loaded with similar experience and skills and complexity of work so any productivity comparisons across teams, more often than not, is not meaningful. Your Product Management determines the value and quality of what a team delivers and the focus should be on defining measures for those and for teams to keep improving with respect to these expectations from product management.  In addition to this, you should be looking at factors that contribute to these outcomes – such as how well the team is collaborating and working together, how good is the collaboration with product management and so on.

Watch out this space for a future blog on measuring outcomes in an Agile environment.