Lies, Damned Lies, and Statistics

Posted on February 24, 2013

Reading a post by Patrick Kua on An Appropriate Use of Metrics got me thinking about the type of statistics and metrics I have seen organisations use and the naive way in which they can be interpreted.

Using inappropriate metrics is lazy at best!

The very first example Patrick used was one related to the focus on bugs raised in production. Focusing purely on metrics led to the decision to work harder to close bugs that are raised but generated little insight into the underlying causes of those bugs. Were they caused by poorly defined requirements? Were they caused by poor development practices? Are they really bugs that are being raised or just poor understanding of the product features? A bit more insight might have highlighted that they were as a result of poor user understanding of the product so it would have been better to focus on user awareness of designing intuitive user interfaces.

Patrick goes on to say:

Organizations love metrics because it makes setting targets easier, and discourages people from questioning the goal behind the target. This leads managers into a false sense of organizational efficiency. Strong incentives tied to strong metrics force people to concentrate on just one part of the work, neglecting other contributing factors that might make a goal more successful.

This sort of lazy, dumbed down, manage by numbers approach is why many organisations are run by those pesky accountants!!

An example that I have seen a number of times is that of maximising office densities. An organisation has been growing (or shrinking) and the executive committee decide that they must maximise the return on their office space. A manager decides that the best way to achieve this is to increase the densities of workers in the office space by fitting more people in. Or perhaps using hot desks or if he is a really innovative manager he might even decide to allow people to work from home. “With the money that I’ll save the company I’ll be in line for that promotion” (he thinks…and sometimes, sadly, this even happens).

The thing is, he squeezes more people into that office and it starts to irritate our highly mobile knowledge workers. They start to notice more and more the working environments that other companies (cough, Google etc.) are providing and decide they want more.

This causes churn which effects the bottom line, subtly at first but more and more pronounced as time passes. It becomes harder to back fill those roles because people start to hear that the offices are cramped and uncomfortable. These who stick around are less engaged because forcing employees together from unrelated areas of the company has meant they have less affinity with those around them. They no longer stick around after work to get stuff over the line because the increased noise in the office leaves them drained after a long day.

The guys using the hot desks find it an irritant to start with as it makes for a very sterile environment. They lose 30 minutes in the morning finding a desk and setting up and 15 minutes in the evening packing their things into lockers (which ironically take up more space than the original desks!). They find that the loss of cohesion means that there is no longer sense of team and this is effecting their productivity. This is particularly difficult for the junior members of staff and new hires.

The guys who work from home start to feel isolated and so raise requests for better communications equipment and help in paying for increased lighting and heating costs. Sure, instant messaging and video chat sort of feel the same but they don’t hear much of what is going on unless they are directly involved. Think about the water cooler effect!

But this is all ok because the accountant that made the decision has moved on and its no longer his problem.

It’s not all bad news for metrics! But the right metrics used appropriately proved to be very useful. Patrick identifies some guidelines for appropriate use of metrics:

Explicitly link metrics to goals
Favour tracking trends over absolute numbers
Use shorter tracking periods
Change metrics when they stop driving change

These guidelines are described in further detail and it’s well worth reading Patricks post.