Why measurable outcomes aren’t always a good thing

“Enough of this feel-good stuff! How do we know whether people have learned anything unless we measure it?”
—A little voice, heard once in a while in learning designers’ heads

Ah, the lure of measurement! Yes, it’s important. From a scientific perspective, better understanding of the world we live in requires doing experiments that involve quantifying properties in a statistically meaningful and repeatable way. Science has no opinion about ghosts, life after death, and astrology, for example, because we can’t reliably measure associated attributes.

Once the power of scientific thinking became widely evident at the start of the twentieth century, it was inevitable that it would be applied to management. The result was the concept of scientific management, developed by Frederick Winslow Taylor. Even though Taylorism is no longer a dominant management paradigm, its Victorian influence on how we view working with others still persists to this day.

I’m a proponent of the scientific method, but it has limitations because we can’t measure much of what’s important to us. (Actually it’s worse than that—often we aren’t even aware of what’s important.) Here’s Peter Block on how preoccupation with measurement prevents meaningful change:

The essence of these classic problem-solving steps is the belief that the way to make a difference in the world is to define problems and needs and then recommend actions to solve those needs. We are all problem solvers, action oriented and results minded. It is illegal in this culture to leave a meeting without a to-do list. We want measurable outcomes and we want them now…

…In fact it is this very mindset, one based on clear definition, prediction, and measurement which prevents anything fundamental from changing.
—Peter Block, Community: The Structure of Belonging

One of my important learning experiences occurred unexpectedly in a workshop where a participant in a small group I was leading got furious after something I had said. He stood up and stepped towards me, shouting and balling his fists. At that moment, to my surprise, I knew that his intense anger was all about him and not about me. Instead of my habitual response—taking anger personally—I was able to effectively help him look at why he had become so enraged.

There was nothing measurable about this interchange, yet it was an amazing learning and empowering moment for me.

So, one of the dangers of requiring measurable outcomes is that it restricts us to concentrate on what can be measured, not what’s important. Educator Alfie Kohn supplies this example:

…it is much easier to quantify the number of times a semicolon has been used correctly in an essay than it is to quantify how well the student has explored ideas in that essay.
—Alfie Kohn, Beware of the Standards, Not Just the Tests

Another reason why we are fixated on assigning a number to a “measured” outcome is that doing so can make people feel they can show they’ve accomplished something, masking the common painful reality that they have no idea how to honestly measure their effectiveness.

Measured learning outcomes can be relevant if we have a clear, performance-based, target. For example, we can test whether someone has learned and can apply cardiopulmonary resuscitation (CPR) by testing them in a realistic environment. (Even then, less than half of course participants can pass a skills test one year after training.)

This leads to my final danger of requiring measurable outcomes: it turns out that measurements of learning outcomes aren’t reliable anyway!

For nearly 50 years measurement scholars have warned against pursuing the blind alley of value added assessment.  Our research has demonstrated yet again that the reliability of gain scores and residual scores…is negligible.
—Professor Trudy W. Banta, A Warning on Measuring Learning Outcomes, Inside Higher Ed

Given that requiring measurable outcomes often inhibits fundamental change and is of dubious reliability, I believe we should be considerably more reluctant to insist on including them in today’s learning and organizational environments.

[This post is part of the occasional series: How do you facilitate change? where we explore various aspects of facilitating individual and group change.]

Image attribution: xkcd

How to gauge the energy for creating a new conference

 

Wouldn’t it be wonderful if we could know whether a great idea would lead to a successful new event?

Creating a new conference is a chancy business that requires a certain amount of guts. I have worked on several conference projects that haven’t panned out, and there’s no question that failing to engage a critical mass of registrants for an event that you thought would be popular is a disheartening experience.

Despite success in creating many successful new conferences over the last thirty years, I still don’t know how to predict whether an event will be viable. But I have learned one thing.

One requirement for success is being able to easily find others—my rule of thumb is at least four people—who share your vision for the conference and are willing to help make the event a reality. In the past, I’ve tried to gauge commitment intuitively. Recently, while rereading Peter Block’s wonderful book Community: The Structure of Belonging with my Consultants and Trainers monthly network group (now in its 16th year!) I came across his Four Early Questions, which he uses to negotiate the social contract between group members to “shift the ownership of the room”. Peter suggests asking people to rate on a seven-point scale, from low to high, their responses to these four questions:

  • How valuable an experience (or project, or community) do you plan for this to be?
  • How much risk are you willing to take?
  • How participative do you plan to be?
  • To what extent are you invested in the well-being of the whole?

When building any kind of new endeavor, I think these questions provide a useful way to learn more about the people you may be working with, and so I’m sharing them with you here.

Changing the way we come together at events

Slide 7

“…if we do not change the way citizens come together, if we do not shift the context under which we gather and do not change the methodology of our gatherings, then we will have to keep waiting for great leaders, and we will never step up to the power and accountability that is within our grasp.”
—Peter Block, Community: The Structure of Belonging

Another stirring quote from Peter Block, from a chapter titled The Stuck Community.

Change citizens to conference attendees and you have a good description of what continues to happen at traditional conferences, where attendees listen to session leaders, rather than collectively reaping the benefits of co-creating an event and associated community.

Leadership, conferences, and freedom

Leadership at conferences

“[There are] almost as many definitions of leadership as there are persons who have attempted to define the concept.”
—Ralph Stogdill, Handbook of Leadership: A survey of theory and research (1974, p.259)

I’m not going to add to the thousands of existing definitions of leadership. I believe that defining what leadership is—essentially, process that influences others to accomplish something—misses the point. What we need to understand first, is the purpose of leadership. Once we’ve decided that, we can think about what leadership qualities we need to carry out that purpose.

The task of leadership

Here’s the wonderful Peter Block musing in a recent book:

“…perhaps the real task of leadership is to confront people with their freedom.”
—Peter Block, Community: The Structure of Belonging

I love this expression of leadership mission.

Too often our vision of leadership is clouded and restricted by 19th century ideas of leadership, emphasizing autocratic, bureaucratic, and charismatic leadership styles that are still commonly held up for us as models of what leadership is about. Even though more enlightened leadership models (e.g. servant and transformational styles) are becoming more widely used, there’s still a tendency to revert to the old models in some situations.

Leadership at conferences

For example, how do we treat conference attendees?

At most conferences, attendees have very little say in what happens. The event revolves around a set of limited preselected session choices made by the conference leadership. Such an event culture implies a default passivity. Organizers, not attendees, make decisions—organizers who are, perhaps unknowingly, using leadership styles more appropriate for young children.

It’s perfectly possible, however, to offer freedom to conference participants. Unconference designs provide structure and support for participants to determine what they want to learn, share, and discuss. Participants are then free to make the event their own.

Most of us who are asked to try something new feel a natural reluctance or wariness. First time attendees at an unconference often feel apprehensive about the prospect of taking a more active role. That’s why Peter’s phrasing confront people with their freedom is appropriate. Unconferences offer an environment that gently confronts attendees with their power to influence what happens. In my experience, once attendees experience what it is like to have a real voice in shaping their event, the vast majority of them embrace this new freedom.

What do you think of Peter Block’s musing on the real task of leadership?

Photo by Flickr user Dunechaser.