Building conference programs: lessons from the $1 million Netflix Prize

How can we get better at building conference programs?

building conference programs

“In 2006 we announced the Netflix Prize, a machine learning and data mining competition for movie rating prediction. We offered $1 million to whoever improved the accuracy of our existing system called Cinematch by 10%.”
Netflix Recommendations: Beyond the 5 stars (Part 1) by Xavier Amatriain and Justin Basilico (Personalization Science and Engineering)

The above quote comes from an interesting blog post by two Netflix engineers who explain why the company has never fully implemented the algorithm that won the Netflix $1 Million Challenge to improve Netflix’s customer movie recommendations.

Why didn’t Netflix use the improved movie recommendation algorithm?

Although an earlier part of an earlier version of the algorithm was incorporated into the way Netflix recommends movies, by the time the prize was awarded in 2009, Netflix’s world had drastically changed:

“Netflix launched an instant streaming service in 2007, one year after the Netflix Prize began. Streaming has not only changed the way our members interact with the service, but also the type of data available to use in our algorithms. For DVDs…selection is distant in time from viewing, people select carefully because exchanging a DVD for another takes more than a day, and we get no feedback during viewing. For streaming members are looking for something great to watch right now; they can sample a few videos before settling on one, they can consume several in one session, and we can observe viewing statistics such as whether a video was watched fully or only partially.”

Conference program building: DVDs or streaming?

The traditional way we build conference programs is like how we order DVDs on Netflix. We decide in advance the content we want and order it from presenters. Once the conference starts we have no choice except to view the presentations we’ve ordered.

At participant-driven events, building the conference program is like how we pick a streaming movie to watch on Netflix. When we settle down on the couch, we pick the movie we want at that moment and start to watch. Not to our liking? We can stop the stream and try something else.

Notice there are two things different about the streaming movie choice. First, we get to choose what we want to watch when we’re ready to watch it. We don’t have to make a choice several days in advance. And second, we can switch to any other movie right away if what we choose isn’t to our satisfaction. That’s an option not available with DVDs.

Netflix’s move towards streaming delivery of movies is the reason why the carefully tuned, Prize-developed-at-great-expense algorithm has been largely abandoned. It turns out that people choose different movies when they can decide what they want to watch when they’re ready to watch, and when they can change their mind if their choice is unsatisfactory.

Lessons for your event, courtesy of Netflix

Netflix’s observations parallel my experience of building conference programs. Programs that participants create at the event are noticeably different (and, in my experience, better) than those chosen in advance by a program committee. As I’ve observed before, half or more of the session topics that participants choose via the Conferences That Work design are not predicted in advance by a program committee. If program committees do such a poor job of creating conference sessions that registrants actually want to attend, perhaps you should consider replacing at least some of your conference sessions with those chosen via an effective tool like Open Space, World Café, or Conferences That Work‘s peer session sign-up.

I subscribe to both DVDs & streaming from Netflix, and notice that my DVD queue is getting shorter and shorter these days; we watch far more streaming movies than we did just a year ago. We’re not alone; Netflix streaming users now outnumber DVD subscribers 2:1. With the rise of online, providing topical, up-to-the-minute conference sessions is becoming more important than ever. Let’s plan for a streaming rather than a DVD future for building conference programs.

Photo attribution: Flickr user jovino

Hat tip to the always interesting Techdirt where I first came across the Netflix article.

Leave a Reply

Your email address will not be published. Required fields are marked *