The Reminder—a new way to obtain long-term evaluations of events

letter 8909849224_832820ea27_kCan conference organizers get evaluative feedback on the long-term outcomes of their events? Last week, I pointed out that short-term evaluations routinely solicited at events are unreliable. If we want to honestly learn whether our conferences create long-lasting change, we need evaluation methods that can be applied after an appropriate length of time (three months? six months? a year?—you choose!) rather than within a few hours or days of the meeting taking place.

Here’s one way I’ve devised to obtain long-term feedback. It’s based on an old technique “A Letter To Myself” (ALTM, aka “A Letter To My Future Self”) which you may have experienced at meetings over the years.

I call it The Reminder.

In the standard ALTM version, described in Conferences That Work: Creating Events That People Love, the organizers set aside around 30 minutes just before the end of the event, supply each participant with notepaper and an envelope, and ask them to reflect on the changes they would like to make in their lives as a result of the event over the next [3 months/6 months/year/appropriate time period]. People then write letters to themselves about their changes and insert the letters into the supplied envelopes, which they then seal and address to themselves. The envelopes are collected and mailed out, unread, by the conference organizers once the announced time period has passed.

ALTM works because the recipients find value in being reminded of their resolutions after time has passed. They can note what they have accomplished, what is yet to be done, and what they may have forgotten but still have energy to pursue.

When I run a Personal Introspective at the end of a peer conference, I often add the ALTM exercise to provide a personal “tickler” reminder of the changes participants decide to make.

The Reminder
To modify ALTM to incorporate long-term feedback, add the following to the envelope supplied to each participant:

Sample feedback form to be included in the A Letter To Myself envelopeBefore the end of the ALTM session, briefly go through the feedback form with the group. Explain that completing the form on receipt and promptly mailing it back will provide the conference organizers valuable information about the long-term effectiveness of the conference, and this will help make the event even better next time.

It’s harder to implement long-term evaluations of our events because participants have less motivation to provide the information we need. The Reminder combines the effect of receiving the participant-created letter with a quick request for feedback. Motivation can be increased by adding an incentive for returning the feedback form, like a small prize or chance to win a raffle from those who return the form. In this case, name/contact information should be added to the form.

What do you think? Can The Reminder be a useful tool for evaluating your events? If you use it, share how it worked in the comments below.

Photo attribution: Flickr user gufoblu

Why meeting evaluations are unreliable and how we can improve them

evaluation 5201223017_52a7453f27_o

A fatal flaw
Just about all meeting evaluations are elicited within a few days of the session experience. All such short-term evaluations of a meeting or conference session possess a fatal flaw. They tell you nothing about the long-term effects of the session.

What is the purpose of a meeting? Unless we’re talking about special events, which are about transitory celebrations and entertainment (nothing wrong with these, but not what I’m focusing on here), isn’t the core purpose of a meeting to create useful long-term change? Learning that can be applied productively in the future, connections that last and reward, communities that grow and develop new activities and purpose—these are the key valuable outcomes that meetings and conferences can and should produce.

Unfortunately, humans are poor objective evaluators of the enduring benefits of a session they have just experienced.

Probably the most significant reason for this is that we are far more likely to be influenced by our immediate emotional experience during a session than by the successful delivery of what eventually turn out to be long-term benefits. We like to think of ourselves as driven by rationality, but as Daniel Kahneman eloquently explains in Thinking, Fast and Slow we largely discount the effects that our emotions have on our beliefs. Although information provided by lectures and speeches is mostly forgotten within a week, the short-term emotional glow fanned by a skillful motivational speaker can last long enough for great marks on smile sheets. And paradoxically, the long-term learning that can result from well-designed experiential meeting sessions may not be consciously recognized for some time.

Other reasons why evaluations of conference sessions can be unreliable include quantifiable reason bias (the distortions that occur when attendees are asked to justify their evaluations) and evaluation environment bias (evaluations are influenced by the circumstances in which they’re made). These biases are minimized if evaluations are made in the environment in which hoped-for learning can actually be applied: i.e. back in the world of work. But instead—worried that no one will provide feedback if we wait too long—we supply evaluation sheets to fill out at the session, or push evaluation reminders right away via a conference app.

How can we improve meeting evaluations?
If we want meeting evaluations to reflect real-world long-term change, we need to use evaluation methods that allow participants to report on their meeting experiences’ long-term effects.

This is hard—much harder than asking for immediate impressions. Once away from the event, memories fade, our professional lives center around our day-to-day work, and we are less amenable to being refocused on the past.

While I haven’t formulated a comprehensive approach to evaluating long-term change related to meetings, I think an effective long-term meeting evaluation should include the following activities:

  • Individual participants document perceived learning and change resolutions before the meeting ends.
  • Follow-up with participants after an appropriate time to determine whether their chosen changes have actually occurred.

In my next post I’ll share a concrete example of one way to implement a long-term evaluation that incorporates these components.

Photo attribution: Flickr user jurgenappelo

Mission Impossible

Mission Impossible 8477081649_e951df1931_oIf you know your mission — you do have a mission, right? — then your long-term strategy becomes much clearer. You know where you want to go; now, all that remains is how to get there.

Of course, life is rarely that simple.

There’s always that must-do-now stuff that gets in the way. As Seth Godin puts it:

“This interim strategy, the notion that ideals and principles are for later, but right now, all the focus and resources have to be put into the emergency of getting successful—it doesn’t work.
It doesn’t work because it’s always the interim. It never seems like the right time to stop doing what worked and start doing what we said was important.”
—Seth Godin, The interim strategy

How can we stay focused on our mission when there’s always something demanding our attention right now? There are four core steps:

  1. Notice what’s going on. (“A week has gone by, and I’ve spent fifteen minutes, tops, working on my mission.”) Sometimes this is the hardest step. We can’t change when we are unaware or avoiding the changes we really need/want to make.
  2. Make a plan. End/delegate/deprioritize the short term stuff that’s getting in the way. Set goals for your mission-related work.
  3. Carry out your plan. Sometimes this is the hardest step.
  4. Steps 1-3 aren’t a one-time process. Loop ’em. Keep noticing, making new plans, and acting on them. That’s how you’ll grow and, potentially, succeed in your mission.

As Seth concludes his aforementioned post: “The interim is forever, so perhaps it makes sense to make act in the interim as we expect to act in the long haul.”

And remember this.

If you remain continually immersed in interim work, executing your Mission becomes Impossible.

Photo attribution: Flickr user taylor-mcbride