Why measurable outcomes aren’t always a good thing

“Enough of this feel-good stuff! How do we know whether people have learned anything unless we measure it?”
—A little voice, heard once in a while in learning designers’ heads

Ah, the lure of measurement! Yes, it’s important. From a scientific perspective, better understanding of the world we live in requires doing experiments that involve quantifying properties in a statistically meaningful and repeatable way. Science has no opinion about ghosts, life after death, and astrology, for example, because we can’t reliably measure associated attributes.

Once the power of scientific thinking became widely evident at the start of the twentieth century, it was inevitable that it would be applied to management. The result was the concept of scientific management, developed by Frederick Winslow Taylor. Even though Taylorism is no longer a dominant management paradigm, its Victorian influence on how we view working with others still persists to this day.

I’m a proponent of the scientific method, but it has limitations because we can’t measure much of what’s important to us. (Actually it’s worse than that—often we aren’t even aware of what’s important.) Here’s Peter Block on how preoccupation with measurement prevents meaningful change:

The essence of these classic problem-solving steps is the belief that the way to make a difference in the world is to define problems and needs and then recommend actions to solve those needs. We are all problem solvers, action oriented and results minded. It is illegal in this culture to leave a meeting without a to-do list. We want measurable outcomes and we want them now…

…In fact it is this very mindset, one based on clear definition, prediction, and measurement which prevents anything fundamental from changing.
—Peter Block, Community: The Structure of Belonging

One of my important learning experiences occurred unexpectedly in a workshop where a participant in a small group I was leading got furious after something I had said. He stood up and stepped towards me, shouting and balling his fists. At that moment, to my surprise, I knew that his intense anger was all about him and not about me. Instead of my habitual response—taking anger personally—I was able to effectively help him look at why he had become so enraged.

There was nothing measurable about this interchange, yet it was an amazing learning and empowering moment for me.

So, one of the dangers of requiring measurable outcomes is that it restricts us to concentrate on what can be measured, not what’s important. Educator Alfie Kohn supplies this example:

…it is much easier to quantify the number of times a semicolon has been used correctly in an essay than it is to quantify how well the student has explored ideas in that essay.
—Alfie Kohn, Beware of the Standards, Not Just the Tests

Another reason why we are fixated on assigning a number to a “measured” outcome is that doing so can make people feel they can show they’ve accomplished something, masking the common painful reality that they have no idea how to honestly measure their effectiveness.

Measured learning outcomes can be relevant if we have a clear, performance-based, target. For example, we can test whether someone has learned and can apply cardiopulmonary resuscitation (CPR) by testing them in a realistic environment. (Even then, less than half of course participants can pass a skills test one year after training.)

This leads to my final danger of requiring measurable outcomes: it turns out that measurements of learning outcomes aren’t reliable anyway!

For nearly 50 years measurement scholars have warned against pursuing the blind alley of value added assessment.  Our research has demonstrated yet again that the reliability of gain scores and residual scores…is negligible.
—Professor Trudy W. Banta, A Warning on Measuring Learning Outcomes, Inside Higher Ed

Given that requiring measurable outcomes often inhibits fundamental change and is of dubious reliability, I believe we should be considerably more reluctant to insist on including them in today’s learning and organizational environments.

[This post is part of the occasional series: How do you facilitate change? where we explore various aspects of facilitating individual and group change.]

Image attribution: xkcd

3 thoughts on “Why measurable outcomes aren’t always a good thing

  1. Interesting article Adrian. As a company we put measurement as a top priority, but perhaps from a slightly different angle. We use psychometric measurement to rate attendee experience, performance and productivity through out flagship product – our Meetology Audit. Our emphasis is on understanding how the meetings environment affects attendees with the goal of creating a benchmark rating that we can then aim to improve on. I agree there is a fine line to tread though ad measurement CAN be time intensive and therefore relative high investment. However, I believe that if we give up and say ‘we can’t measure meetings’ then we have given up and are resigned to decades more of pointless, badly focused meetings with little structure and no measurable outcomes. If you’re interested find out more here http://www.meetology.com
    Jon Bradshaw
    CEO
    The Meetology Group (UK)

    1. Jon—I’d never say “we can’t measure meetings”. And I think there are things it makes sense to measure about meetings. One of the points I’m making above, echoed by my comment quote of the Seth Godin piece that was published today is that just because we can measure something doesn’t mean it’s important. We need to spend more time thinking carefully about what, if anything, it might be important to measure before we spend more time and resources making measurements.

  2. Sometimes I feel like I’m channeling Seth Godin; sometimes I feel like he’s channeling me. Here’s his January 2014 post, written a couple of weeks after the one above:

    Measuring nothing (with great accuracy)
    “The weight of a television set has nothing at all to do with the clarity of its picture. Even if you measure to a tenth of a gram, this precise data is useless.

    Some people measure stereo equipment using fancy charts and graphs, even though the charts and graphs say little or nothing about how it actually sounds.

    A person’s Klout score or the number of Twitter followers she has probably doesn’t have a lot to do with how much influence she actually has, even if you measure it quite carefully.

    You can’t tell if a book is any good by the number of words it contains, even though it’s quite easy and direct to measure this.

    We keep coming up with new things to measure (like processor speed, heat output, column inches) but it’s pretty rare that those measurements are actually a proxy for the impact or quality we care about. It takes a lot of guts to stop measuring things that are measurable, and even more guts to create things that don’t measure well by conventional means.”
    —Seth Godin, http://sethgodin.typepad.com/seths_blog/2014/01/measuring-nothing-with-great-accuracy.html

Leave a Reply

Your email address will not be published. Required fields are marked *