Did anyone learn anything?

Animated graphic of icons of a puzzled person surrounded by other puzzled people, with the blinking legend: "Did anyone learn anything?"The meeting is over. Did anyone learn anything? And how would you know?

An EventTech Chat discussion

I greatly enjoy participating in EventTech Chat, “a weekly conversation about meeting and event technology, including software, hardware, and audiovisual for in-person and online events” hosted by pals Brandt Krueger and Glenn Thayer.

During last week’s chat, one of the topics we discussed was whether there are differences in how people learn online, as opposed to face-to-face. This led to conversations about learning styles (be careful, they’re mostly mythical and barely useful), the importance of taking responsibility for your own learning at meetings, and how meeting formats affect what people learn.

Are you a regular reader of this blog? If so, you might have guessed—correctly—that I had plenty to say about these important issues. There is plenty of solid research on the best ways to support effective learning. We know that:

Of course, even if we know the best ways to maximize useful learning and connection at meetings, that doesn’t mean we implement them. Unfortunately, our meetings are still full of lectures.

This brings us to an important question we hardly ever ask about meetings…

Did anyone learn anything?

In my book Conferences That Work, I shared a story about when I—and everyone else in my graduate class—never admitted we didn’t understand what our teacher was teaching us for weeks.

…toward the end of my second year I was understanding less and less of a mathematics course I was taking. The professor seemed to be going through the motions—he asked few questions, and there was no homework. Elementary particle physicists are either mathematicians or experimentalists. I was the latter, so my lack of mathematical understanding was not affecting my research work. But the experience was disconcerting.

And, as the semester went on, the percentage of class material I understood gradually declined.

One day, our teacher announced that we would be studying Green’s Functions, a technique used to solve certain kinds of equations. After the first 20 minutes of the class I realized that I understood nothing of what was being said, and that I was at a crucial turning point. If I kept quiet, it would be too late to claim ignorance later, and it was likely I would not understand anything taught for the remainder of the semester. If I spoke up, however, I was likely to display my weak comprehension of everything the teacher had covered so far.

Looking around, I noticed that the other students seemed to be having a similar experience. Everyone looked worried. No one said a word.

The class ended and the professor left. I plucked up my courage and asked my classmates if they were having trouble. We quickly discovered, to our general relief, that none of us understood the class. What should we do? Somehow, without much discussion, we decided to say nothing to the teacher.

The class only ran a few more weeks, and the remaining time became a pro forma ritual. Did our teacher know he had lost us? I think he probably did. I think he remained quiet for his own reasons, perhaps uncaring about his success at educating us, perhaps ashamed that he had lost us.

When I didn’t speak up, I chose to enter a world where I hid my lack of understanding from others, a world where I was faking it…

…Probably you’ve had a similar experience; a sinking feeling as you realize that you don’t understand something that you’re apparently expected to understand, in a context, perhaps a traditional conference, where nonresponsiveness is the norm. It’s a brave soul indeed who will speak out, who is prepared to admit to a conference presenter that they don’t get what’s going on. Have you stayed silent? Do you?

Silence isn’t golden

Silence during a presentation and a lack of questions at the end does not mean that anyone learned anything. As Jonah Berger reminds us in Contagious, “Behavior is public and thoughts are private.”

If my teacher had bothered to periodically ask his class whether they understood what he was attempting to teach, or, better, asked questions to check, we’d likely have told or shown him we were lost.

So, how can we discover if anyone learned anything?

Well, I’m sorry, but smile sheets dropped in a box at the end of a session, app-based evaluations, and online surveys that must be completed within a few days do not provide an accurate picture of the long-term benefits of a meeting.

Why? Because we are far more likely to be influenced by our immediate emotional experience during a session than by the successful delivery of what eventually turns out to be long-term benefits.

Three better ways to obtain long-term evaluations of events are Net Promoter Scores, A Letter to Myself, and The Reminder. Check out this post for more details.

Ultimately, we can’t ensure or guarantee that anyone learned anything at a meeting. As Glenn pointed out during our EventTech Chat, the ultimate responsibility for learning is the learner’s. Attend a meeting expecting that the leaders will magically transfer learning to you without doing any work yourself? You probably won’t learn much, if anything.

Nevertheless, we can actively help people learn at meetings by implementing the principles listed above. (Check out my books for complete details.) But there’s one additional thing we can do to maximize and extend learning during our meetings.

How to help people consolidate what they learn at meetings

During our EventTech Chat, several participants shared how they consolidate learning during or immediately after an event. Folks who have learned the value of this practice and figured out the ways that work best for them may not need what I’m about to share (though even they can often benefit).

What I’ve found over decades of designing meetings is that the majority of meeting attendees do not know how to consolidate what they learn there. So I designed a closing plenary that gives each participant a carefully structured opportunity to review, consolidate, and reinforce what they have learned at the conference. They also get to develop the next steps for changes they will work on in their professional lives. It’s called a Personal Introspective and takes 60 – 90 minutes to run. (You can find full details in Chapter 57 of The Power of Participation.)

Did anyone learn anything? There are no guarantees. But, following the above advice will make it significantly more likely that your attendees will learn what they want and need to learn. Do you have other thoughts on how to improve how you or others learn at meetings? If so, please share them in the comments below.

What your conference evaluations are missing

conference evaluations: a photograph of a milk carton with a message on the side "MISSING: HAVE YOU SEEN ME?" surrounding an evaluation sheetOne of the easiest, yet often neglected, ways for meeting professionals to improve their craft is to obtain (and act on!) client feedback after designing/producing/facilitating an event. So, I like to schedule a thirty-minute call at a mutually convenient date one or two weeks after the event, giving the client time to decompress and process attendee conference evaluations.

During a recent call, a client shared their conference evaluation summaries that rated individual sessions and the overall conference experience.

This particular annual conference uses a peer conference format every few years. The client finds the Conferences That Work design introduces attendees to a wider set of peer resources and conversations at the event. This year, The Solution Room, was a highly rated session for building connections and getting useful, confidential peer consulting on individual challenges.

As the client and I talked, we realized that the evaluations had missed an important component. We were trying to decide how frequently the organization should alternate a peer conference format with more traditional approaches. However, we had no attendee feedback on how participants viewed the effectiveness of the annual event for some key performance indicators (KPIs):

  • making useful new connections;
  • building relationships;
  • getting current professional wants and needs met; and
  • building community.

Rating your KPIs

Adding ratings of these KPIs to conference evaluations provides useful information about how well each event performs in these areas. Over time, conveners will see if/how peer conference formats improve these metrics. I also suggested that we include Net Promoter Scores in future evaluations.

The client quickly decided to include these ratings in future conference evaluations. As a result, our retrospective call helped us to improve how participants evaluate his events. This will provide data that will allow more informed decisions about future conference design decisions.

Do your evaluations allow attendees to rate the connection and just-in-time learning effectiveness of your meeting? Do they rate how well your meeting met current professional wants and needs? If not, consider adding these kinds of questions to all your evaluations. Over time you’ll obtain data on the meeting designs and formats that serve your participants best.

The meeting industry’s biggest dirty secret

Illustration of a group of people crowded around a table. Two people sit at the table, which holds a giant padlock.There are some things that the meeting industry doesn’t like to talk about in public. For example:

But the meeting industry’s biggest dirty secret is so embarrassing, we don’t even talk about it in private.

We make no attempt to discover (let alone evaluate) the long-term value of our meetings to attendees!

Before the event, we turn up our marketing to eleven, create FOMO, and do everything we can to get potential attendees to register. It’s going to be great!

During the event, we do our best to put on a heck of a show.

After the event? Crickets.

Once the event is over, it’s done. We barely look back; we’re on to the next one. Don’t look back! (You might not like what you see.)

This is fine for special events, which are primarily about creating an emotional experience and celebrating cultural rituals.

It’s not OK for any other kind of meeting.

Long-term meeting value for participants

Did the event have any significant long-term impact on the lives of the participants? Did anything significant change as a result? Or was the event effectively a short-term-feel-good, short-term-high-impact experiential blip?

If you think about it, this is shocking. We spend vast sums of money and consign umpteen person-hours to a meeting. And we have no idea whether it made any significant long-term difference to the people who attended it!

Notice I’m not concentrating on other meeting stakeholders here. Those who run meetings as a business can determine whether a meeting met their profit goals or not. Sponsors have metrics to help them decide if the event business booked, new connections made, and existing connections strengthened were worth the time and money expended. Even presenters, to some extent, can discover via feedback their audience impact and often find out whether their exposure led to future opportunities to speak or consult.

But long-term value to participants? We have no idea, because we don’t do anything to discover if there even was any, and, if there was, what it might be.

The invisible elephant in the room

Why do we ignore what I believe is the most important outcome of meetings from the attendee perspective: long-term change? After all, if you go to a meeting and nothing significant changes in your life as a result, what was the point of going?

I think we avoid talking about attendee long-term benefits from our meetings because we secretly know that most meetings do not lead to long-term change for most attendees. And we are embarrassed by this reality.

The meeting industry’s invisible elephant is that most meetings are incapable by design of producing significant long-term attendee benefits.

But, but … we do evaluate our meetings!

If you think that smile sheets dropped in a box at the end of a session, app-based evals, and online surveys that must be completed within a few days provide an accurate picture of the long-term benefits of a meeting, there’s a bridge I’d like to sell you.

Even if we include them (and we often don’t), short-term evaluations measure emotional impact, not long-term change and are unreliable[1, 2] measurements of event value. They tell us nothing about the impact of meeting attendance months later, when the stirring closing session story told by the motivational speaker that you thought was great at the time has been forgotten and, most likely, hasn’t changed your life or perspectives on it one bit.

OK, I hear you. So, if were brave enough to explore my event’s long-term impact, how would I do it?

How do we expose and face the meeting industry’s biggest dirty secret?

<soapbox>First of all, if you really want to create events that are maximally likely to provide long-term attendee value, you need to think about event designs that are likely to provide it! That’s why I design and facilitate meetings that meet participants’ actual wants and needs, rather than the traditional pre-determined, best-guess programs that, from my extensive experience, don’t.</soapbox>

Here are three tools that can be used to explore the long-term impact of an event.

Net Promoter Scores (NPS)

A time-delayed evaluation of conference and sessions’ Net Promoter Scores provides summary feedback about long-term value. (The link explains how NPS is calculated from respondents’ answers.) Use questions of the form:
“Considering the value you gained from Conference C/Session X, Y & Z, how likely is it that you would recommend Conference C/Session X, Y & Z to a friend or colleague?”

Adding an open-ended request for respondents to share their reasons for each rating will provide useful feedback on the value of a conference and its sessions.

A Letter To Myself

Set aside around 30 minutes just before the end of the event, supply each participant with notepaper and an envelope, and ask them to reflect on the changes they would like to make in their lives as a result of the event over the next [3 months/6 months/year/appropriate time period]. Attendees write letters to themselves about their changes and insert the letters into the supplied envelopes, which they then seal and address to themselves. The envelopes are collected and mailed out, unread, by the conference organizers once the announced time period has passed.

A Letter To Myself provides participants with a personal method to evaluate the long-term effect of an event, but meeting organizers remain in the dark unless participant responses to receiving their letters are shared back. That’s why I developed the following tool.

The Reminder

The Reminder is a more sophisticated version of A Letter To Myself that adds valuable long-term feedback to the conference organizers. See the link for full details.

Do it!

If you want to really know about the long-term value of your meetings to attendees, I’ve given you concrete things you can do to find out. I know it can be scary to ask these questions, because there’s usually that little voice in your mind that says, “Suppose I find out that what I’m doing really sucks? Then what?”

I’m here to tell you that the likelihood is very high that you will hear really good stuff about what you’re doing — and yes, there will be some unsettling responses too, because you can’t please everyone no matter what you do. Buried amongst the “not-so-great” responses will be nuggets of gold feedback you can use to improve your event for next time. Working on continuous improvement is increasingly important in the competitive world of meetings, and here are some implementation tools.

It’s your choice whether you use them or not.

Do you agree that not evaluating the long-term value of our meetings to attendees is the meeting industry’s biggest dirty secret? Do you have other suggestions for taking an honest look at what we’re providing to our meeting attendees? Please share your thoughts in the comments below!