Why meeting evaluations are unreliable and how we can improve them

evaluation 5201223017_52a7453f27_o

A fatal flaw
Just about all meeting evaluations are elicited within a few days of the session experience. All such short-term evaluations of a meeting or conference session possess a fatal flaw. They tell you nothing about the long-term effects of the session.

What is the purpose of a meeting? Unless we’re talking about special events, which are about transitory celebrations and entertainment (nothing wrong with these, but not what I’m focusing on here), isn’t the core purpose of a meeting to create useful long-term change? Learning that can be applied productively in the future, connections that last and reward, communities that grow and develop new activities and purpose—these are the key valuable outcomes that meetings and conferences can and should produce.

Unfortunately, humans are poor objective evaluators of the enduring benefits of a session they have just experienced.

Probably the most significant reason for this is that we are far more likely to be influenced by our immediate emotional experience during a session than by the successful delivery of what eventually turn out to be long-term benefits. We like to think of ourselves as driven by rationality, but as Daniel Kahneman eloquently explains in Thinking, Fast and Slow we largely discount the effects that our emotions have on our beliefs. Although information provided by lectures and speeches is mostly forgotten within a week, the short-term emotional glow fanned by a skillful motivational speaker can last long enough for great marks on smile sheets. And paradoxically, the long-term learning that can result from well-designed experiential meeting sessions may not be consciously recognized for some time.

Other reasons why evaluations of conference sessions can be unreliable include quantifiable reason bias (the distortions that occur when attendees are asked to justify their evaluations) and evaluation environment bias (evaluations are influenced by the circumstances in which they’re made). These biases are minimized if evaluations are made in the environment in which hoped-for learning can actually be applied: i.e. back in the world of work. But instead—worried that no one will provide feedback if we wait too long—we supply evaluation sheets to fill out at the session, or push evaluation reminders right away via a conference app.

How can we improve meeting evaluations?
If we want meeting evaluations to reflect real-world long-term change, we need to use evaluation methods that allow participants to report on their meeting experiences’ long-term effects.

This is hard—much harder than asking for immediate impressions. Once away from the event, memories fade, our professional lives center around our day-to-day work, and we are less amenable to being refocused on the past.

While I haven’t formulated a comprehensive approach to evaluating long-term change related to meetings, I think an effective long-term meeting evaluation should include the following activities:

  • Individual participants document perceived learning and change resolutions before the meeting ends.
  • Follow-up with participants after an appropriate time to determine whether their chosen changes have actually occurred.

In my next post I’ll share a concrete example of one way to implement a long-term evaluation that incorporates these components.

Photo attribution: Flickr user jurgenappelo