The Art of Measurement
I want to talk a bit about management measurement. I recently spent a number of years in a good sized multinational, and I watched management trying to gain control through measurement. And mostly I watched as they gained the wrong sort of control; as they crystallized behaviour in ways that lose more from employees than they gained.
When you’re dealing with small numbers of people, simple measurements are all you need, and indeed the time spent measuring can be a simple waste of time. For larger groups, and as management becomes disassociated from the actual work of the organization, measuring is necessary so that management knows what is happening and can modify it. The old saying (which I’m sick of) is that “you can’t manage what you can’t measure.” It’s a statement with a lot of truth to it, but so is this – “you measure what you manage, so you’d better be sure you’re measuring what you want to manage.”
Here’s an example. A friend of mine used to do customer support for laptops. He was measured on how long he was on the phone and how quickly he picked up. If he spent too long on the phone on average, then he was taken aside and reprimanded. These measurements encouraged tech support employees to get people off the phone as quickly as possible, whether their problem was solved or not. Assuming management actually wanted happy customers (ie, that they saw tech support as a way to sell the next laptop, rather than something they had to do as cheaply as possible) then the way to measure this would be to have an automatic survey at the end of the phone call, asking how satisfied the client is. Since there will always be jerks who are never happy with phone support, you set the threshold at a certain percentage of “unhappy” customers and then if someone goes over that you investigate. To keep productivity up you measure phone time and compare to satisfaction ratios and (horrors) investigate individual reps who spend more time than normal on the phone, then coach them individually on how to solve problems with less chit-chat while still keeping the customer happy.
I’m going to discuss five issues related to measurement. The first is the problem of measuring what you can easily measure. Simply put, it may be more difficult to measure some things than others. Management tends to measure those things that are easy to measure. In a call center there are plenty of systems which will allow you to track a wild variety of phone stats, but you can’t measure one CSR helping another with a call. In sales you can measure how many sales a salesman makes and how much they’re worth, but it’s more difficult to measure whether he’s made verbal promises your company will have trouble living up to. You can measure the number of code lines a programmer put out, but it’s harder to measure how easy they will be to maintain down the line.
This is often a systems issue. Whatever the system assists your employees to do, is easy to measure. So if you have a system that presents work items, and which employees close those work items, it’s easy to measure how fast they’re doing them. But what if some work items are harder than others? And what happens to those employees who are taking calls or e-mails you can’t track and are helping customers or other employees with those problems – is that behaviour you don’t want to encourage? Because if you’re measuring only processing times then those who do other things will be measured as less productive. So they stop helping customers, and soon you have a reputation as having unresponsive employees who never want to take time to help people.
And this leads to the second issue, which is what I call Putting your Fingers Down. Another way of putting it, is “you get the behaviour you measure.” If a job involves 10 activities, and you publicly measure only 5 of them, your employees will gravitate towards those activities. It often seems obvious what an employee does. Let’s say you have repair techs in a retail store and you decide to measure their productivity by measuring how many appliances they repair. Sounds good eh? Productivity increases and you’re happy.
Until you start getting complaints that the repair techs don’t want to talk to customers, and that when they do all they seem to want to do is get away from them. You also hear that some techs are taking easy repairs and leaving the hard repairs for others, who put them off, because that boosts their stats. So easy repairs are getting done fast, the hard ones are getting done slower, and customers aren’t getting individual personal attention any more, so they aren’t happy. That worked well!
Which leads to what I call the The Limits of Coercion. Public measurement is a form of coercion. The idea is to measure people and then push them to do better and get rid of the ones who don’t measure up. You put your fingers down and say, “do this!” And you can absolutely do it. Whatever behaviour you are able and willing to take the time to measure, you can and will get. But what you can’t get is positive cooperation. You can’t make people do the extra things. And people resent the wrong type of measurement. The problem is that you as management think you understand the job. Problem is, unless you still do it yourself, you probably don’t. Outside of the sort of jobs that are truly subject to Taylorization, most jobs require a myriad of little tasks and if people don’t do them, the overall job suffers. If you start measuring the wrong specific things then people’s attitude when you pull them in for a talk is “I’m doing fine on the stats you said you want, I don’t have time for the other stuff.”
The other problem is that people subvert the measurements. There are almost always ways to make the numbers come out better than they should, and people will take the time to find them and do them. Which leads to the fourth issue, the question of “Public metrics and private metrics.” Simply put, when you’re setting up metrics you should first find out which metrics track each other; figure out why they track each other; and measure both sets. But one set you keep private and the other is the public set. If the private set starts diverging from the public set then you should investigate if people are fiddling with the public set. Odds are they are.
But the real, final point is that you should be looking for your “Bottom Line Metrics”. In a call center it might be the percentage of happy callers divided by the average time per call. In a processing center I once worked in the VP (a very wise man) used to publicly (I’m sure privately he had a number of measurements which had to remain satisficed) measure only one thing – the average time from a piece of work entering the center to the time it left. He didn’t measure any specific processing times – only how well the center was working overall. If that number went up he’d want to know why, and when he wanted it to go down he let people tell him how they were going to get it down, not the other way around. The center ran very well. When he left his successor started putting his fingers down and both customer satisfaction and employee happiness declined.
In the end you should ask yourself “what are we trying to accomplish?” Then you publicly measure that, and only that. It may seem that you want to do multiple things, but in most cases you can boil it down to one thing – as with the customer service center where happiness was divided by call times. You want people to go away happy after their call with the least time necessary to make them happy. If you can’t break it down then you either don’t understand what the job actually entails (or what your division or company does) or you may need to break the work into different functional groups.
Finally, don’t fall into the MBA trap. As a manager you probably don’t really know what your employees are doing. You probably don’t really understand what is required to do the job well. However unless you’ve beaten them down too hard, or you’ve got a crew of reprobates, most people want to do a good job. Most people want to be able to say “damn, we’re good!” Don’t treat them like untrustworthy children, and you may find that they’re on your side and that measuring only the bottom line, on the minimum, is sufficient. When you go to war with your employees and try and measure every specific behaviour, generally both sides lose.
(Originally published in 2004 at BopNews.)
Comments are closed.
This is quite thought provoking analysis. It rings true when it describes the effects of attempts at control, and it does not prescribe outlandsih remedies. I am not sure I can subscribe to the notion that most people want to do a good job. In my more pessimistic moods, it appears that most people want to say, “Damn, I am so good, that I should not put in any effort.” After all, my youthful motto was “Minimum face, eumque bene.” (Do the minimum and do it well.)
Ian, I so want to side with ya.
But.
The worker bee’s v. the management . . .
A great divide.
I’ve been both.
It’s hard to measure . . . most management sucks.
And many workers suck, too.,
I’ve been a co-worker. Often, they sucked.
I’ve been a manager, most workers sucked.
On this topic, it’s hard to deliberate, as it’s honestly case by case driven.
You can’t generalize.
But thanks for a heck of a read.
The problem is what management represents anymore, and what ownership demands.
Workers are fucked.
And you know it.
😉
Ian, you’ve hit it head on (and I say that with a degree in organizational behavior).
In school, classmates would ask what was going to be on the final. They were adapting to the metrics that were going to be applied to them, as opposed to learning as much as possible. (This tells us something important about the US obsession with standardized testing)
At the supermarket, one of the metrics was items/minute checked out. So when a mother came through the line with 50 jars of baby food, instead of entering the price X 50, we’d ring one jar through and then hit repeat 49 times. Someone had spread the word that this made a higher count/minute…I’d bet that was an incorrect bit of lore.
As a manager, I found the Hawthorne Effect was real. Employees don’t like being micro-managed and treated with distrust; they also don’t like being ignored. Ignoring them teaches them that what they do doesn’t matter. If you educate them as to what the organization’s objectives are and how their work matters in the overall scheme, they can engage their minds figuring out ways to adapt their work to the objectives instead of just the metrics. I think this also makes a lot of work more interesting.
Another thing I discovered when I was a manager was that many of my peers would talk about how lazy their staffs were and how they needed to be watched like hawks. In the same conversations, they would talk about how our boss was an oppressive micro-manager. EVERYONE in a hierarchy happens to be at that exact level where they work best when given a free rein to exercise their creativity on behalf of the company; and where everyone below them needs to be regimented, measured, and punished to prevent shirking. Funny how that works out.
Ian, have you read any Mintzberg? Or did I already ask you that at FDL?
I’m hoping and praying that that this poorly written piece and its comments will gather dust. I suspect none of you could manage a woman in full heat much less a company.
John are you angry?
I owned small business with employees and now work for large regional company in Facilities. I don’t talk to management for days or until they want something out of the normal day. They don’t trust any of us and only a few people in the company even know what we do. Ian almost everything you wrote is true and we are soon to be measured in how much we can do and I hope to make another year before they fire me because I’ll be to slow. My goal with a large project that might take several months or a few hours is to not come back when I’m done. I was trained as a Millwright and in that trade you only got one chance to do it right and if you didn’t a lot of money/time went out the door.
On another note my group was just transferred to the under 30 who are reinventing the wheel. I didn’t make any friends when I reminded them that it always ends up round, only maybe with a white wall or red lettering.
Thanks Ian
Wow hit the double jackpot of sexism and libertarian nuttery.
My first troll. Yay!
Barry – if I have read Mintzberg, I don’t recall it.
Tallifer and Larue: I’ve managed teams but not above that level, and I have found that if I treat most people the right way, I get good performance. Now this isn’t universal, some people don’t respond. But I’ve gotten good performance from folks other people couldn’t, and that’s mostly because I both have high expectations and high regard for people. I think they can do a good job, I find things to like about them, and they want to keep/get my good regard because most people don’t get nearly enough good regard.
In the same way I am a pretty tough teacher (though obviously I have far less experience teaching than Tallifer). But, again with some exceptions, I find that I can get most people to learn because I think they can do it, and I admire them when they do.
I don’t want to oversimplify, but in my experience in workplaces, you more often get what you expect from people than not. If you expect them to be lousy, they will be.
I just lifted this from a comment by Paul Rosenberg on OpenLeft. He’s quoting the transcript of a recent Bill Moyers program:
BILL MOYERS: Yes, one of my favorite scenes, in Season Four, we get to see the struggling public school system in Baltimore, through the eyes of a former cop who’s become a schoolteacher. In this telling scene, he realizes that state testing in the schools is little more than a trick he learned on the police force. It’s called “juking the stats.” Take a look.
ASSISTANT PRINCIPAL: So for the time being, all teachers will devote class time to teaching language arts sample questions. Now if you turn to page eleven, please, I have some things I want to go over with you.
ROLAND “PREZ” PRYZBYLEWSKI: I don’t get it, all this so we score higher on the state tests? If we’re teaching the kids the test questions, what is it assessing in them?
TEACHER: Nothing, it assesses us. The test scores go up, they can say the schools are improving. The scores stay down, they can’t.
PREZ: Juking the stats.
TEACHER: Excuse me?
PREZ: Making robberies into larcenies, making rapes disappear. You juke the stats, and major become colonels. I’ve been here before.
TEACHER: Wherever you go, there you are.
DAVID SIMON: You show me anything that depicts institutional progress in America, school test scores, crime stats, arrest reports, arrest stats, anything that a politician can run on, anything that somebody can get a promotion on. And as soon as you invent that statistical category, 50 people in that institution will be at work trying to figure out a way to make it look as if progress is actually occurring when actually no progress is. And this comes down to Wall Street. I mean, our entire economic structure fell behind the idea that these mortgage-based securities were actually valuable. And they had absolutely no value. They were toxic. And yet, they were being traded and being hurled about, because somebody could make some short-term profit. In the same way that a police commissioner or a deputy commissioner can get promoted, and a major can become a colonel, and an assistant school superintendent can become a school superintendent, if they make it look like the kids are learning, and that they’re solving crime. And that was a front row seat for me as a reporter. Getting to figure out how the crime stats actually didn’t represent anything, once they got done with them.
BILL MOYERS: And you say that’s driving the war on drugs, though, right? The stats, not the-
DAVID SIMON: Dope on the table. Stats, you know, “We’ve made so many arrests.” I mean, they used to ride around Baltimore under one administration, and say, “If we can make 54 arrests a day, we’ll break the– we’ll have an all-time record for drug arrests.”
DAVID SIMON: Some of the arrests, well, it was people sitting on their stoops and, you know, loitering in a drug free zone, meaning you were sitting on your own steps on a summer day. Anything that is a stat can be cheated, right down to journalism. And I was sort of party to that.
So, I would be– I would be watching what the police department was doing, what the school system was, you know, you would look outward. But if you looked inward you’d see that the same game is played everywhere. That nobody’s actually in the business of doing what the institution’s supposed to do.