short-term

The Reminder—a new way to obtain long-term evaluations of events

Monday, November 9th, 2015 by Adrian Segar

the reminder: a black-and-white photograph of a man sitting on a bench and reading a letterCan conference organizers get evaluative feedback on the long-term outcomes of their events? Check out The Reminder and find out!

Last week, I pointed out that short-term evaluations routinely solicited at events are unreliable. If we want to honestly learn whether our conferences create long-lasting change, we need evaluation methods to apply after an appropriate length of time (three months? six months? a year?—you choose!) rather than within a few hours or days of the meeting taking place.

Here’s one way I’ve devised to obtain long-term feedback. It’s based on an old technique, “A Letter To Myself” (ALTM, aka “A Letter To My Future Self”) which you may have experienced at meetings over the years.

I call it The Reminder.

In the standard ALTM version, described in Conferences That Work: Creating Events That People Love, the organizers set aside around 30 minutes just before the end of the event, supply each participant with notepaper and an envelope, and ask them to reflect on the changes they would like to make in their lives as a result of the event over the next [3 months/6 months/year/appropriate period]. People then write letters to themselves about their changes. They put the letters into the supplied envelopes, seal and self-address them and hand them in. After the announced period, the meeting organizers mail the letters.

ALTM works because the recipients find value in being reminded of their resolutions after time has passed. They can note what they have accomplished, what is yet to be done, and what they may have forgotten but still have the energy to pursue.

When I run a Personal Introspective at the end of a peer conference, I often add the ALTM exercise to provide a personal “tickler” reminder of the changes participants decide to make.

The Reminder

To modify ALTM to incorporate long-term feedback, add the following to the envelope supplied to each participant:

the reminderBefore the end of the ALTM session, briefly go through the feedback form with the group. Explain that completing the form on receipt and promptly mailing it back will provide the conference organizers valuable information about the long-term effectiveness of the conference, and this will help make the event even better next time.

It’s harder to implement long-term evaluations of our events because participants have less motivation to provide the information we need. The Reminder combines the effect of receiving the participant-created letter with a quick request for feedback. You can increase motivation by adding an incentive for returning the feedback form, like a small prize or a chance to win a raffle from those who return the form. In this case, name/contact information should be added to the form.

What do you think? Can The Reminder be a useful tool for evaluating your events? If you use it, share how it worked in the comments below.

Photo attribution: Flickr user gufoblu

Tags: , , , , , , , | Posted in Event design | No Comments »

Why meeting evaluations are unreliable and how we can improve them

Monday, November 2nd, 2015 by Adrian Segar

meeting evaluations are unreliable: an illustration of a skeptical person receiving forms from the hands of many peopleTraditional meeting evaluations are unreliable. We obtain them within a few days of the session experience. All such short-term evaluations of a meeting or conference session possess a fatal flaw. They tell you nothing about the long-term effects of the session.

What is the purpose of a meeting? Excluding special events, which are about transitory celebrations and entertainment (nothing wrong with these, but not what I’m focusing on here), isn’t the core purpose of a meeting to create useful long-term change? Learning to apply productively in the future? Connections that last and reward? Communities that grow and develop new activities and purpose? These are the key valuable outcomes that meetings and conferences can and should produce.

Unfortunately, humans are poor objective evaluators of the enduring benefits of a session they have just experienced.

Why are we poor evaluators in the short term?

Probably the most significant reason for being poor evaluators in the short term is that we are far more likely to be influenced by our immediate emotional experience during a session than by the successful delivery of what eventually turn out to be long-term benefits. We like to think of ourselves as driven by rationality. But as Daniel Kahneman eloquently explains in Thinking, Fast and Slow we largely discount the effects that our emotions have on our beliefs. Information provided by lectures and speeches is mostly forgotten within a week. But the short-term emotional glow fanned by a skillful motivational speaker can last long enough for great marks on smile sheets. Paradoxically, the long-term learning that can result from well-designed experiential meeting sessions may not be consciously recognized for some time.

Other reasons why evaluations of conference sessions can be unreliable include quantifiable reason bias (the distortions that occur when attendees are asked to justify their evaluations) and evaluation environment bias (evaluations are influenced by the circumstances in which they’re made). These biases are minimal if we receive evaluations from the environment in which participants can implement hoped-for learning: i.e. back in the world of work. But instead—worried that no one will provide feedback if we wait too long—we supply evaluation sheets to fill out at the session or push evaluation reminders right away via a conference app.

How can we improve meeting evaluations?

Want meeting evaluations to reflect real-world long-term change? Then we need to use evaluation methods that allow participants to report on their meeting experiences’ long-term effects.

This is hard—much harder than asking for immediate impressions. Once away from the event, memories fade. Our professional lives center around our day-to-day work, and we are less open to refocusing on the past.

While I haven’t formulated a comprehensive approach to evaluating long-term change related to meetings, I think an effective long-term meeting evaluation should include the following activities:

  • Individual participants document perceived learning and change resolutions before the meeting ends.
  • Follow-up with participants after an appropriate time to determine whether their chosen changes have occurred.

In my next post, I’ll share a concrete example of one way to implement a long-term evaluation that incorporates these components.

Photo attribution: Flickr user jurgenappelo

Tags: , , , , , , , | Posted in Event design | No Comments »