David Maister - Professional Business, Professional Life
David’s ResourcesAbout David
NEW! Browse my materials by topic of interest:StrategyManagingClient RelationsCareersGeneral

Passion, People and Principles

Don’t Measure, Judge!

post # 58 — April 25, 2006 — a Managing, Strategy post

The mantra of modern management is (still) ‘If you can’t measure it, you can’t manage it’ and much analytical creativity goes into trying to invent new metrics (eg ‘the balanced scorecard’) in order to justify and monitor new strategies. Corporate officers are always asking for (impossible) ROI calculations on such things as knowledge management, customer satisfaction, innovation and the like.

What this measurement mentality misses is the fact that, as human beings, we are remarkably well-equipped to judge things, and we can form an accurate picture of what is going on by looking, talking and interacting, even for only a short period of time.

Take, for example, the issue of whether or not the employees in a business are excited, enthused, energetic and engaged.

In spite of corporate efforts to design questionnaires to monitor performance (including the infamously ineffective and bureaucratic 360-degree feedback programs), anyone walking into a business unit would be able to detect in a minute the difference between a ‘turned-on’ group of employees and one where everyone was operating like a good, compliant soldier. It would have to be a very sophisticated ‘measurement’ system to capture what any one of us could detect in an instance.

Embedded in the ‘management by measurement’ mentality is a false hypothesis. That effective management can be achieved through detachment – that measurements will be reviewed by a top-level (corporate-style) manager who is not involved in a hands-on way in the business, but monitors what is going on through the scorecards.

There is much in business (perhaps most of the crucial decisions) that cannot be measured in such a way that an accurate meaning can be conveyed to the person examining ONLY the measurements. Measurements can be invaluable in giving guidance to someone who must decide and judge, but are a profoundly pathetic substitute for effective judgments.

The challenge of modern business is not to improve our measurement metrics, but to redesign managerial activities so that the organization is better able to make good judgments.

How would you think about designing an organization that outperformed others by making effective judgments? I don’t think we yet have any science on this, but it would probably include:

a) Delegation of decision making to local managers who are more likely to understand what is going on and are able to make better, informed judgments;

b) A clear set of enforced guiding principles and objectives, so that trade-offs are made consistently across the firm

c) Careful selection and apprenticeship of those who are going to be placed in judgment (managerial) roles

d) Finding ways to ensure that the judges are credible and held accountable for their effectiveness

A good starting point for reading on this topic is an article entitled ‘The New Organization’ in January 21st issue of The Economist quoting Yves Morieux of the Boston Consulting Group.

M. Morieux says that, in the organization of the future, the main tasks will be to help improve the judgment-based decision-making processes that knowledge workers use, not find better ways to measure their performance.

Anyone else have some views on how you improve an organization’s ability to judge, rather than measure?

11 Comments

James Bullock said:

I think you have half of it. Talking with some colleagues, some years ago a few of us astonished the other half of the conversation by claiming that “assessments” of large IT projects or SW development organizations can take as little as half a day, or even half an hour to start. Much like the “engaged employees” example above, it is obvious what is going on. What to do about it takes longer. Doing something takes longer still. The idea is the easy part.

With an assessment like this, “they” know what is going on. They are living it, after all. They’ll tell you, if you listen. So the skill is a kind of open awarness and receptivity, to take in what you are being shown and told. The antitheseis of imposing a preconcieved solution, actually, which can unfortunately look like the same rapid response.

That said, one way to describe how we work is in terms of models of how the world works. Your own work, (Mr. Maister) particularly “Managing the Professional Services Firm” describes some models that are uncommon in people’s minds, yet accurate, evocative, and predictive. Thus the power of describing the models.

I think there’s a time for reflection, and “measures” are tools of reflection, actually. “Hey, I’m pouring more hours into making software, and the useful stuff rate is going down. What’s up with that?” Not “this is wrong” but “what’s up with that?” Measures can help make it obvious when the world operates differently than we think. The measures we pursue reflect, in the end, how we think the world works. So, deciding what to collect is a way of exposing tacit assumptions. Measures are a learning tool, really, which as internalzied, leads to better judgments in the moment.

I’d like to say that this is original. Yet, it’s just what Deming really said, what Toyota has institutionalized, and what Austin wrote about in his book about measurement, among many others. In a serendipity event, just yesterday I got a note from a colleague who talks all the time about “embodiment” so traded him one of my favorite quotes:

“Software development is a performance art. Managing that doubly so.”

Still, I think the addiction to superficial measurement is as debilitating for it’s erosion of reflection as it is for usurping judgment in the moment. But that’s just my opinion, right now.

Thanks.

posted on April 25, 2006

David Koopmans said:

Measuring makes sense for stuff that allows itself to be measured with a reasonable level of accuracy; productivity is a good one. I recently participated as a communications consultant in a change program for a large financial institution and got a good idea why these measurement systems don’t work; the people interviewed are still largely driven by their own agendas and prejudices, hence the objectivity of the measurement tool is comprimised. I am sure that there will be people who argue that there are good and bad systems in this. To me it is more a matter of using the right tools for the right problem.

posted on April 25, 2006

David (Maister) said:

David and James – thanks for the comments. What both of your comments have in common is the recognition that a true understanding of a measurement system can only be obtained by looking at how measurements are actually used.

One of the points I find myself making repeatedly is that no measurement system can ever be “objective” or reflect ‘truth’. Pursuit of more accurate or ‘better’ measurements may be a vain pursuit.

Measurements (which are critical) are best seen as ‘signals to investigate’ – signs that something should be explored further.

A great system would be one that had good signals, but also the managerial capacity and wisdom to react to those measures with conversaation, inestigation and judgment, not knee-jerk reactions.

posted on April 25, 2006

annette said:

I’d offer a view from a human systems perspective. Measurement is also predicated on the assumption that the future can be controlled (all strategic plans are tools for the measurement and control of future uncertainty, as are most of the current psychometric testing tools)…I’m with you on the issue of judgement which is a subjective exercise. The trap is to turn the use of subjectivity into an objective exercise! (I’m a sceptic when it comes to objectivity!). I believe an organisation of this type would build in systemic reflexive spaces where learning from the subjective experience of workers was leveraged in the service of the group and organisational task.

Technically speaking organisations don’t exist — they are social constructs in the service of a task and as such are a collection of subjective experiences…

My own area of interest is in how emotion is generated and managed in organisational contexts, which is often accused of being “irrational” and “subjective” – yet systemic exploration of this area reveals extraordinary information about systemic relationships.

posted on April 26, 2006

James Cherkoff said:

I work to help brands and marketeers get to grips with new web technologies – such as blogs and the user generated world.

One of the first questions is always how do we measure what is happening in the blogopshere. (Perfectly reasonable of course.) Now while there are tools that let you look around and start to understand what is being said about specific issues, it’s not yet possible to actually suck everything in and carve it up into neat bundles – not least because machines can’t read. This can be an alarming response for some clients and some don’t feel much better when I tell them the only way to understand what is being said about them, their products or their market is to participate in the online chatter. By that I mean collecting a few hundred blogs and reading them. that way, you quickly get a good feel for the main issues that people and customers are dealing with. And while it’s certainly possible to gather specific, real information – understanding the all important sentiment can still only be done with the human eye. However, this is sometimes greeted with a rather disappointed response – because it is “subjective”.

posted on April 26, 2006

David (Maister) said:

If you are intersted in this topic, go to http://www.accmanpro.com/?p=726 where an exchange between Dennis Howlett and I is taking place. We both want your comments – here OR there!

posted on April 26, 2006

Shawn Callahan said:

Hi David, we must be on a similar bandwidth this week because I’ve also written a piece on appreciating the unmeasurable. (http://www.anecdote.com.au/archives/2006/04/evaluating_the.html)

We are using a technique (narrative based of course) called Most Significant Change. It was written up in the American Journal of Evaluation (full reference in my post) and involves asking people to retell what is the most significant change they’ve witnessed over, say, the last six month and then describe why they think that is significant. These stories are collected at the front line then passed up the hierarchy where each level decides what they think is significant and why. There selections and feed back to the troops. No measurements are made but is a systematic way of getting people to talk about the things that matter.

The technique was orginally developed to assess overseas aid programs. We are now using it in large corporations.

posted on April 26, 2006

Ian Welsh said:

Strangely enough, this is a topic dear to my heart, and one I wrote on at some length in the past at: http://www.bopnews.com/archives/004474.html#4474

(The Art of Measurement).

In general the problem with measurement is that people react to it. If I decide that I want my employees to answer X number of calls a day, then I can be sure they will do so – even if it means that they don’t deal properly with each caller; ditch hard calls to other people; and let non-call work go. If I want my partners to give me X billable hours, publish it publicly and tie comp to it, I can be sure they’ll do everything they can to get it, and I can be sure they’ll not mentor other people if it means missing it. I can also be sure that they’ll pack hours into the months where they need them, inflate hours if necessary, etc… because I’ve told them nothing else matters as much as their billable hours.

posted on April 26, 2006

Bill Peper said:

Here is a link to the article from the American Journal of Evaluation that Shawn referenced:

http://ntru.aiatsis.gov.au/ifamp/research/pdfs/Dart_and_Davies_2003.pdf

posted on April 27, 2006

John Zapolski said:

The tension between measurement and judgement reminds me of the validity vs. reliability opposition being espoused by Roger Martin, dean of the University of Toronto’s Rotman School of Managment. Martin argues that most businesses have become dominated by the quest for reliability—production of consistent, replicable outcomes, substantiated using past data. But as innovation becomes increasingly necessary to achieve organic growth, validity-oriented operating methods becomes important. Validity-seeking managers look for production of outcomes that meet objectives (regardless of whether it can esily be repeated), and substantiate based on future events. Because validity-oriented approaches seek to increase judgement where reliability-oriented managers seek to avoid it (ie, let the data tell you what to do), judgement will increase in importance in management decision-making over the next several years, Martin argues.

http://www.businessweek.com/innovate/content/sep2005/id20050929_872877.htm

posted on May 22, 2006

Ariel said:

One of the best;s site of research companies is http://WWW.FULLSURVEYSGUIDE.COM

It contain free information about researh companies,free surveys and focus groups,paying for read mail,paid for drive and for test,work from home opportunities,scam list of sites and some free games!

posted on April 11, 2007