September 2021 — Are in-person events COVID safe? The worldwide meeting industry is desperately, and understandably, waiting for the answer to this question to be “Yes”. Well, Freeman, the largest global event management firm, has just announced “the truth is, in-person events are actually safer than many daily activities, like trips to the grocery store — and we have the data to prove it”.

Sadly, while I acknowledge and appreciate Freeman’s significant work on the case for recommencing meeting in person, I believe this claim is misleading, and the underlying modeling and research include flawed assumptions.

Make no mistake; I love to design and facilitate in-person meetings. I strongly desire to be able to safely return to facilitating and attending in-person events. But, as meeting professionals, we have a professional duty of care during the COVID-19 pandemic, and I think it’s important to provide a realistic assessment of risk for meeting stakeholders—especially potential attendees. Articles are already appearing in meeting publications (1, 2) that highlight the one-line summary of the Freeman announcement above. Such opinions, buttressed by what seems to be solid research and modeling, can easily give our industry the impression that in-person meetings can safely recommence.

My concerns about Freeman’s statements

I have two broad concerns about Freeman’s summary of research “Inside LIVE: The data you need to navigate the Delta variant for events” on the safety of in-person events. You can watch Freeman’s 55-minute webinar, posted on August 25, 2021, below.

1—Freeman’s overall conclusion is misleading

My first concern is that Freeman’s big picture conclusion that “in-person events are actually safer than many daily activities, like trips to the grocery store” is a misleading characterization of the statistics they present.

Here are the statistics (from the next webinar slide).

This slide compares the mid-August, 2021 rate of COVID cases amongst the entire population in the United States with the reported rates from four recent large in-person events. The second column shows the infection rate as a percentage.

The entire U.S. infection rate is indeed higher than the reported rates from the listed recent in-person events. (I’ll add that we know that reported rates are typically significantly lower than actual rates, but let’s assume that both sets of statistics are undercounted to the same degree.)

Unfortunately, Freeman’s statement that “in-person events are actually safer than many daily activities, like trips to the grocery store” does not follow from the data on this slide!

Why?

Because the statement conflates the risk of a masked visit to a grocery store with the overall risk in the United States of getting infected! The latter regrettably includes a significant fraction of the U.S. population who won’t or can’t get vaccinated, don’t wear masks to protect against airborne transmission of COVID-19, and don’t social distance. The risk of contracting COVID-19 during a grocery store masked visit is far less than the overall risk for everyone in the U.S.

The headline statement is, therefore, comparing apples to oranges. You’d expect any event that implements precautions against COVID-19 transmission to have a lower infection rate than the entire United States. That doesn’t mean that attending an event is a safe enough choice for attendees and staff.

This brings us to what’s actually important to people trying to make a decision whether to attend an event. The event modeling, performed for Freeman by Epistemix, and discussed later, suggests that those who are currently likely to attend a large in-person business event that implements mitigation strategies such as vaccination requirements, masking, and social distancing, are significantly more likely to be vaccinated (~80%). That statistic seems credible to me.

Such potential attendees, who are already more careful than the average American about how they live their lives in a pandemic, aren’t interested in whether an event environment provides a risk of getting COVID-19 comparable to the average risk of the entire population of the U.S. Rather, they want to know if attending the event will significantly (defined by them) increase their likelihood of contracting COVID-19. And that brings me to the second concern about the assumptions made by Epistemix’s event risk model.

2—The event risk model used for risk calculations is flawed and incomplete

When I heard about the Freeman webinar (thanks Julius Solaris!), I posted some initial responses. Freeman’s Jessica Fritsche was kind enough to reach out to me and arrange a Zoom call with John Cordier, the CEO of Epistemix, to walk through the data modeling used in the research. And John generously offered an hour of his time for us to talk. We were also joined by Sarah Shewey, Founder/CEO of Happily who was also interested in learning more about how infection rates at meetings could be modeled.

During our hour together, John shared an overview of the Epistemix model. This gave me a better understanding of Epistemix’s approach. The model essentially attempts to simulate the entire population of the United States at an impressive level of detail. It includes numerous geographic and social factors that affect infection risk. However, during our conversation I asked about a number of important factors that I believe Epistemix has not incorporated into their model of calculating meeting risks.

Probably the most important of these is adequately modeling the air quality at the event, given the paucity of information available about the safety of specific venues and properties from an air quality perspective. In addition, the model does not include the additive risks for travel to and from an event, and staying in a hotel during an event. Though it’s likely possible to model the increased risk during (unmasked) eating and drinking social activities during the event, it doesn’t appear that the Epistemix model does this. Finally, though the Epistemix model incorporates information about COVID-19 variants as they become known, I’m skeptical that it can accurately predict in a timely manner the impact of brand new COVID-19 variants.

In the following sections, I’ll expand on these issues in more detail.

Flaws and omissions in Epistemix’s meeting model

First, a tiny introduction to modeling human systems. All models are an approximation of reality. Consequently,

“All models are wrong, but some are useful“
—aphorism generally attributed to the statistician George Box

Two fundamental considerations when building and trusting computer models are:

1. The assumptions one makes in building a model are key to the model being actually useful rather than wrong. Computer models are very seductive. They seem precise and authoritative, and it’s hard to discover and accept their limitations and/or even their completely wrong predictions. Choosing the right assumptions is an art, not a science. One poor assumption can doom a model’s reliability.
2. Even if you choose good assumptions, implementing them correctly in computer code is difficult. It’s hard to be sure that an implementation faithfully reflects core assumptions. An incorrect implementation of a potentially useful model typically leads to incorrect predictions. If you’re lucky, it’s obvious that a model’s outputs are wrong. But sometimes, predictions are subtly wrong in ways that are easy to overlook.

I’m going to assume that Epistemix models faithfully implement the assumptions made to create them (#2 above). However, I’ve identified four factors that I feel Epistemix has not incorporated into their model of calculating meeting risks. Some of these factors are interlinked.

1—Adequately modeling airborne COVID-19 transmission at a specific event

While talking to John, it became clear to me that the current Epistemix approach does not adequately model the air quality—and the consequent risk of COVID-19 transmission—at a specific event. The model has some capacity to estimate risks (which are generally minimal) in very large, high-ceiling spaces like convention halls. But, of course, the typical meeting venue contains multiple meeting spaces, some of them small, and, critically, the venues do not in general have a good handle (if any) on the air quality in those spaces. (Or, if they do, they’re not talking publicly about it.)

When I wrote about this issue six months ago, I put out an industry wide request to learn of venues and properties that had upgraded their HVAC systems to current ASHRAE recommendations (typically ~5 air changes/hour plus MERV 13 or better air filtering). I promised to publicize the venues that had made these upgrades.

I know such upgrades can be costly, but you’d think that venues and properties that have implemented them would love to promote themselves as having air quality that meets current pandemic-based standards.

To date, I have not been told of a single venue that is now compliant with ASHRAE pandemic recommendations. (I hope that by now there are some, and that they will let this be known.) During the webinar, Freeman said that such work has and is being done. Please share this information, folks! Meeting planners want to know!

Frankly, without this information a) being made available and b) being incorporated into the Epistemix model it’s hard to have much confidence in the infection risks Epistemix’s model predicts.

2—Additive risks for travel to and from an event, and staying in a hotel during an event

Epistemix’s model does not include the additive risks for attendees (and staff) traveling to and from an event. The main concern is air travel. The air industry has stressed that air change rates in aircraft are high (over 10 air changes/hour) and, now that masks are mandatory, infection risks should therefore be low. An excellent investigation by the New York Times “How Safe Are You From Covid When You Fly?” has tempered this assessment somewhat. Of particular interest are comments from a couple of readers who monitored the carbon-dioxide level—an excellent proxy for air quality—during their entire travel. They found that boarding and deplaning air quality was drastically reduced, as well as during the last thirty minutes of one person’s flight. Exposure at terminal restaurants, where masks must be removed, is also potentially risky.

Quite apart from the “event” itself, staying in a venue may greatly increase one’s risk of infection. I wrote about venue and property ventilation concerns in detail in April, 2021, and later articles by PCMA (1, 2) and the New York Times (1) have echoed this concern.

Again, travel risks are not included in Epistemix’s model. They can be significant, and have to be included to determine the relative risk for an event attendee who is choosing whether to participate or staff an event, or not.

3—Modeling the increased risk when masks are off for socials and group meals

Most in-person meetings include meals and socials, when masking is not possible. Unless you hold such unmasked get-togethers outdoors or in safely ventilated venues, airborne transmission of COVID-19 amongst everyone present (attendees and staff) is a potentially significant and unknown risk. Outdoor locations are only possible for limited periods in much of the U.S. As mentioned above, venues and properties remain silent on whether they’ve upgraded and certified their facilities to current ASHRAE recommendations on air quality.

We have also seen reports of numerous cases of reduced, unmasked social distancing at socials and meals. This is understandable in a world where we’ve been masked and apart for so long. But it is still a risky activity, especially in spaces where ventilation is inadequate.

My understanding (which may be incorrect) of Epistemix’s model is that masking is a global parameter for an event. The model does not handle unmasking in specific event spaces for periods of time. Even if the model does have this flexibility, the lack of knowledge of whether such spaces are safely ventilated prevents an accurate risk assessment.

4—Can Epistemix model the appearance of brand new COVID-19 variants?

I am also still skeptical that Epistemix can build new variants into the model predictions in a timely fashion, given how the world took about six months after the delta variant was first identified to realize that it was radically changing COVID-19 transmission rates. While Epistemix’s model includes the infection characteristics of multiple variants, and new variants can be added once they are identified, I wonder if an event organizer who made a go/no-go decision about a fall meeting early this summer based on the Epistemix model would be happy about the increased predicted risks once the delta variant was added.

But John and I didn’t have time to fully explore this issue, so this concern may be overblown.

Are in-person events COVID safe?

I really appreciate John Cordier’s willingness to share an overview of Epistemix’s infection risk model for events. Obviously, my brief introduction means there’s no way I can authoritatively review the extensive assumptions that are built into the model. Epistemix’s model is impressively detailed and, if correctly implemented (which I have no reason to doubt), seems to comprehensively cover core demographics, the data needed to model infection spread in regional populations, and most major components for predicting infection at a specific event.

When I brought up the concerns I’ve listed in this post, I felt that John largely talked past me, continuing with an explanation of the model without responding directly to what I was asking. This was somewhat frustrating. The two exceptions to this were:

1. My question about whether the model could accurately predict in a timely manner the impact of brand new variants. This arose at the end of our meeting. John indicated that he believed the model was able to do this, but we didn’t have enough time to explore this issue fully. I’m still skeptical, though he might well have been able to convince me otherwise if we’d had more time.
2. My primary concern about modeling air quality in detail. John admitted during the meeting that the model does not currently handle specific venue air quality architecture at the detail that’s necessary to simulate, say, what happens when you have a session in a smaller classroom with an HVAC system that is not up to current ASHRAE recommendations. It also omits risks due to event participants (and staff) spending time in properties that may have inadequate air quality. He wrote to me afterwards that “he’d be glad to follow up on the air-quality parameters that you think are most important”.

I’ve seen so many pretty models of systems over the decades. To a casual viewer, they look impressive. It’s only because I spent years building and validating such models that I know how misleading they can be, and the difficulty and importance of identifying the key factors and approximations that form the basis of the model and limit its scope and/or accuracy.

Leaving out detailed venue specific air quality modeling, plus the incoming and return travel risks and accommodation risks during an event, plus inadequate modeling of the risk of transmission during socials and food & beverage sessions make the outputs of the Epistemix model suspect. And I’m skeptical that Epistemix can build new variants into the model predictions in a timely fashion.

Finally, I haven’t covered in this article the feasibility of implementing the various mitigation strategies that are available to reduce the risk of COVID-19 infection at meetings. Personally, I’d insist on proof of vaccination (no exceptions) and maximal masking at any event I’m likely to attend in the near future. But I’ll just add here this observation from the Healthcare Information and Management Systems Society‘s HIMSS21 Las Vegas conference for 19,000 attendees. Vaccination was mandatory for all attendees. There were six positive test results (0.03% infarction rate). However, this PCMA article on the event includes the statement:  “…you will not be able to service your show if you require every single vendor employee, every single supplier employee, every single temp employee to be vaccinated — there’s just not enough labor out there.” Something to bear in mind.

Are in-person events COVID safe? Ultimately, each of us needs to decide the answer to this question. But, in my opinion, until the COVID-19 case count drops drastically and/or venues can show that their facility ventilation is safe, it’s a violation of our professional duty of care to mislead attendees and those who work in our industry by telling them “in-person events are actually safer than many daily activities, like trips to the grocery store”.