“Enough of this feel-good stuff! How do we know whether people have learned anything unless we measure it?”
—A little voice, heard once in a while in learning designers’ heads
Ah, the lure of measurement! Yes, it’s important. From a scientific perspective, better understanding of the world we live in requires doing experiments that involve quantifying properties in a statistically meaningful and repeatable way. Science has no opinion about ghosts, life after death, and astrology, for example, because we can’t reliably measure associated attributes.
Once the power of scientific thinking became widely evident at the start of the twentieth century, it was inevitable that it would be applied to management. The result was the concept of scientific management, developed by Frederick Winslow Taylor. Even though Taylorism is no longer a dominant management paradigm, its Victorian influence on how we view working with others still persists to this day.
I’m a proponent of the scientific method, but it has limitations because we can’t measure much of what’s important to us. (Actually it’s worse than that—often we aren’t even aware of what’s important.) Here’s Peter Block on how preoccupation with measurement prevents meaningful change:
The essence of these classic problem-solving steps is the belief that the way to make a difference in the world is to define problems and needs and then recommend actions to solve those needs. We are all problem solvers, action oriented and results minded. It is illegal in this culture to leave a meeting without a to-do list. We want measurable outcomes and we want them now…
…In fact it is this very mindset, one based on clear definition, prediction, and measurement which prevents anything fundamental from changing.
—Peter Block, Community: The Structure of Belonging
One of my important learning experiences occurred unexpectedly in a workshop where a participant in a small group I was leading got furious after something I had said. He stood up and stepped towards me, shouting and balling his fists. At that moment, to my surprise, I knew that his intense anger was all about him and not about me. Instead of my habitual response—taking anger personally—I was able to effectively help him look at why he had become so enraged.
There was nothing measurable about this interchange, yet it was an amazing learning and empowering moment for me.
So, one of the dangers of requiring measurable outcomes is that it restricts us to concentrate on what can be measured, not what’s important. Educator Alfie Kohn supplies this example:
…it is much easier to quantify the number of times a semicolon has been used correctly in an essay than it is to quantify how well the student has explored ideas in that essay.
—Alfie Kohn, Beware of the Standards, Not Just the Tests
Another reason why we are fixated on assigning a number to a “measured” outcome is that doing so can make people feel they can show they’ve accomplished something, masking the common painful reality that they have no idea how to honestly measure their effectiveness.
Measured learning outcomes can be relevant if we have a clear, performance-based, target. For example, we can test whether someone has learned and can apply cardiopulmonary resuscitation (CPR) by testing them in a realistic environment. (Even then, less than half of course participants can pass a skills test one year after training.)
This leads my final danger of requiring measurable outcomes: it turns out that measurements of learning outcomes aren’t reliable anyway!
For nearly 50 years measurement scholars have warned against pursuing the blind alley of value added assessment. Our research has demonstrated yet again that the reliability of gain scores and residual scores…is negligible.
—Professor Trudy W. Banta, A Warning on Measuring Learning Outcomes, Inside Higher Ed
Given that requiring measurable outcomes often inhibits fundamental change and is of dubious reliability, I believe we should be considerably more reluctant to insist on including them in today’s learning and organizational environments.
[This post is part of the occasional series: How do you facilitate change? where we explore various aspects of facilitating individual and group change.]
Image attribution: xkcd
We are informed about conferences by email, we arrive by airplane, and we gaze at fancy PowerPoint presentations, but, year after year, over a hundred million people experience a conference process that has changed very little since the 17th century.
—Conferences That Work: Creating Events That People Love
We need to make our conferences flexible. We need an exercise program for our events. Why? Here are three reasons.
First, watch the video above.
I learned to code at school when I was 15. No big deal? It was 1966. Learning to program a computer changed my life. Far more important than nearly everything else—facts I have long since forgotten—that I was “taught”.
Learning to code didn’t change my life because I could then make big bucks writing software—though my fourth career, as an IT consultant, was very kind to me. And the important truth of the video’s opening quote by Steve Jobs “Everybody in this country should learn how to program a computer…because it teaches you how to think” isn’t the main reason my life was changed.
Our organizations are built on 19th century learning styles coupled by 20th century leadership models fused with 21st century technologies
—Dan Pontefract, Future of Work
Replace “organizations” with “conferences” in Dan’s great quote, and you encapsulate much of what is wrong with conferences today.
What are you going to do about it?
(Ironic) Powerpoint photo courtesy of Flickr user wmcap
I rarely use the word learning these days. Business managers hear learning and think schooling and don’t want to invest a dime in it. I’m tired of having doors slammed in my face, so I now talk about Working Smarter. I’ve yet to meet a manager who didn’t want her organization to work smarter (even though learning is a major component of doing so).
—Bringing Informal Learning Up To Date, by Jay Cross
Given Jay’s experience it surprises me how often I’m asked how to justify attending participant-driven conferences that don’t have a nice neat program of sessions to show to the “I-decide-whether-you-go” boss. I’d like to think that managers are able to:
Sadly, it’s clear that many managers see learning as a dirty word however it’s defined in their minds: whether as schooling/training or as just-in-time, focused, relevant peer learning. Having to recast “learning” into “Working Smarter” to get management on board reveals management with a fundamental misunderstanding of modern business realities.
The industrial age, when employees trained in a static skill set generated long term returns, is over. Management needs to embrace this simple truth: continuous, self-directed learning in all its forms—experiential, social, and formal—is key to sustained business success today. To paraphrase Derek Bok: if you think learning is dirty, try ignorance.
Have you experienced push-back from management when you’re making a case for learning? Do tell!
Photo attribution: Flickr user happeningfish
Johnnie Moore wrote about this sketch: “I think it captures very succinctly the perils of retrospective coherence – the myriad ways we tidy up history to make things seem more linear.” And: “I think learning needs to be messier; amid all those twists and turns are the discoveries and surprises that satisfy the participant and help new things stick.”
Great points, Johnnie, and I’d like to add one more. Models of success and learning like the one on the left lead to tidy, simplistic conference models (with those deadening learning objectives). When we embrace the reality of messy and/or risky learning, embodied by the sketch on the right, we become open to event designs that mirror this reality and provide the flexibility and openness to address it.
Sketch attribution: Babs Rangaiah of Unilever (“& learning” added by me)
Did the flight attendants travel up and down the aisles checking that I’d fastened my seat belt and my personal item was fully under the seat in front of me? No, they didn’t.
Did they retire to their little jump seats, while the locations of the exits were described in a comforting baritone narrative they’d heard a thousand times before. Nope.
Instead, the crew stood, unmoving in the aisles, facing the passengers for the whole three minutes. The conscientious ones stared at the nearest monitor, even though they couldn’t see what’s on it because they were looking at the back!
Why did they do this?
My flight attendants were modeling listening.
Why? Well, if they appeared to be ignoring the safety video (which they could probably repeat backwards perfectly in their sleep), here’s the message that I would receive:
You don’t need to listen to this.
And my interpretation would be:
Not only is this stuff they’re telling me not important, the flight attendants also think it’s a waste of time too.
Let’s face it; listening well is something that’s extremely hard to do for any length of time. During the facilitation of a large peer conference roundtable that lasts a couple of hours, I find it impossible to do perfectly. But even though, at times, I revert from listening to hearing, I always try to model listening. As the facilitator, if I appear disengaged from what participants are saying I send a message, not only to the person who is speaking but also to everyone present, that what is being said is unimportant. Such behavior, dis-empowering in so many ways, can seriously weaken the building of connections and intimacy amongst conference participants.
I hope I never need to urgently know the positions of the six emergency exits on an Airbus A320, but if that day comes and I do, it will be due to the consistent and persistent modeled listening of the flight attendants on all the airplanes I’ve traveled on over the years. Thanks guys!
During a conference session I was facilitating recently, I met a man—I’ll call him Paul—who had no problems. Since the session was described as an opportunity to get answers from a small group of your peers to problems you were having in your professional life, I found Paul’s attendance surprising. “If you have no problems, why are you here?” was my first question. “I just came to help.” was Paul’s reply.
The group of peers at his table questioned Paul further. Paul apparently had no problems at work at all. His boss loved his performance. Paul felt happy and fulfilled at his job. Even one of my favorite questions in circumstances like these—So Paul, if you had a problem, what would it be? (It’s surprising how often this works!)—elicited a short silence followed by a further protestation of problemnessless. Just to see how far we could go, I asked Paul if he had a problem with any aspect of his life. “Well,” Paul admitted, “I’m no longer married.” I allowed that this problem was outside the scope of our session, and we moved on to the next participant.
Of course, as my mentor Jerry Weinberg wrote long ago: There’s always a problem. I don’t know for sure, but perhaps Paul’s biggest problem was that he was in denial about his problems.
Whatever the reason, Paul missed a great chance to work on some important aspect of his professional life. It’s rare to be offered such an opportunity, but, as we can see from Paul’s example, it’s still possible to turn it down.
Photo credit: Flick user themaxsons
With the rise of social learning and the decline in importance of formal learning, perhaps we should use experiential learning. On the other hand, in the same time needed to experience a limited set of participation techniques we can comprehensively describe many more. There again, perhaps experiencing a participation technique directly is a more effective way to cement both learning it and truly understanding its relevance. So, if we are teaching participation techniques, which of these two approaches is a better path for learning?
J’s light-bulb moment
Earlier this week I led a workshop at Meeting Professionals International’s World Education Congress (WEC). The 150-minute session covered a variety of techniques that foster and support meaningful participation during meetings. Participants spent most of their time using these techniques to learn about and connect with each other and explore questions about their experience at WEC and in the session itself.
As the workshop progressed, and I heard from the forty-six participants, it became clear that one of them, whom I’ll call J, had considerable prior experience with the techniques I was facilitating.
Near the end of the workshop I ran Plus/Delta (described in Conferences That Work: Creating Events That People Love): a method that provides a fast, public evaluation of a session or entire meeting. As an advocate for transparency and feedback, I chose the subject of our Plus/Delta to be a group evaluation of the workshop itself. During the evaluation, J commented that he had hoped that I would cover more techniques by talking about them rather than having attendees experience them directly. He then contributed a simple and ingenious way to extend Plus/Delta that was new to me.
My heart sank, just a little. Here was J, an experienced facilitator of participation techniques, proposing that I should spend the workshop talking about techniques rather than facilitating experiences of them. Could I be going about this wrong?
I moved immediately into the last technique of the workshop, running fishbowl: a simple way to facilitate focused discussion with a large group. All participants sit in a large circle of chairs, but only people in the “fishbowl”, a small circle of chairs at the center, can speak. After a few minutes of comments, J entered the fishbowl.
J said that he had read about fishbowls many times before and he understood how they worked, but he had never tried one.
And then, to my surprise and delight, he told us that experiencing the fishbowl had been a revelation to him, because he had directly experienced the power of the technique in a way that significantly enhanced his understanding of it, which he had previously believed to be sufficient. It was poignant for me to hear J express a new point of view that contradicted what he had said only a few minutes earlier, and I admired his courage in sharing his learning with us all.
I too have struggled over the years to define the best balance between understanding techniques through description and understanding them through direct experience. J’s light-bulb moment fits for me; these days I am content to let attendees learn participation techniques, first through direct experience and then, if necessary, via reflection and discussion.
At the end of the workshop, J hung around and we talked while I was packing up my equipment for a flight home.
He told me that his fishbowl sharing had unexpectedly reminded him of a session he had once attended, entitled “One hundred icebreakers in one hundred minutes”, consisting of rapid descriptions of a hundred ways to introduce attendees to each other.
His rueful comment?
“I don’t remember any of them.”
Download five free chapters here!
I have been reading your book, and if I were Oprah, it would be my featured book of the month!