Articles, Blog

Data and Predictive Analytics: Moneyball in Hollywood w Legendary Entertainment (CXOTalk #276)

Okay. You go to a movie. You’re totally enthralled and engrossed in
the movie, and you think the acting is great. But, did you ever stop to think about the
data, the analytics, the machine learning, the predictive analytics that go behind all
of it to make it happen? Today on Episode #276, that’s what we’re talking
about. I’m Michael Krigsman. I’m an industry analyst and the host of CxOTalk. I want to say a quick thank you to Livestream,
which provides our video streaming infrastructure. Those guys are great. They support CxOTalk. If you go to, in fact,
they’ll give you a discount on their plans. Now, before I say anything else, I need to
ask you in the most heartfelt way to please tell a friend and like us on Facebook. Not only that, get them to like us on Facebook. For extra points, subscribe on YouTube. Without further ado, I am so thrilled to introduce
our guest today. Matt Marolda is the chief analytics officer
at Legendary Entertainment, which is the blockbuster movie house behind some of the movies that
you know of. Matt, welcome to CxOTalk and thank you for
being here. Great. No, thank you. I appreciate it. Matt, tell us about Legendary Entertainment. Sure. Sure. Legendary, as you mentioned, is a producer
of both movies and television shows. The types of movies we produce are large-scale,
things like Godzilla, Kong: Skull Island, The Dark Knight series, movies of that scale,
which are intended to be large, what people often refer to as tentpole movies that are
big, global events all around the world. Okay, so you’re producing these major films,
and your role is chief analytics officer. I think, for many of us, the notion of what
a chief analytics officer in this world, in the Hollywood filmmaking world, is probably
kind of alien. Yeah. Certainly, yeah, it was for me. [Laughter] Yes. No, I can give you the genesis of it, where
we started from, and what the objectives were. Initially, thinking back, this must have been
back in 2012 or ’13, in that timeframe. Our founder and former CEO, a guy named Thomas
Tull, he came to the realization that there was an opportunity in our space in movie marketing
and movie production, in general. That opportunity was to apply data analytics
in a way that was different and unique than what other people were doing at that time. It fell into two major categories. One category was very early in the sense that,
as we were considering what films to produce, how to cast them, the release dates, things
of that nature, there’s an opportunity for data analytics to step in and provide input. To not take over the process, but to be an
important kind of consideration along the way. Similarly, and on the other side, we also
thought there was a real opportunity for us to be able to add efficiency to the marketing
of the films. The movies I described, I think folks might
have a sense as to what they might cost. These are $100 million to $200 million production
budgets, which is usually a pretty publicly known thing. Also, somewhat publicly known, but maybe less
well-known, is how much we spend on media, which can be anywhere from $80 million or
$90 million to $120 million, $150 million worldwide, so these are large bets. These are big bets, and we wanted to make
sure that, before we entered in those bets, we were as informed as possible. Then as we took those movies to the world,
literally, we wanted to be as efficient in the marketing as we could be. Your background was in sports. Yeah. Essentially, Moneyball. The idea was to apply those techniques to
Hollywood filmmaking. Yes. In hindsight, it’s funny because we didn’t
really know what we were going to do. [Laughter] We knew, at a high level, what
it was. We knew we needed to use data and analytics
to inform the process. The Bridges are similar in a certain sense. The first thing I said to our chief creative
officer when I joined Legendary, and again these are two roles that could effectively
be oil and water, creative and analytics. Those could be things that are opposing forces,
basically. What I said to him is the attitude that we’ve
had from the beginning from the creative side, which was that analytics, especially in sports,
but the same with content, never produced a player, but all it tries to do is put the
player in the best position to succeed. That was the attitude from the beginning. On the marketing side is a little bit different. The marking side was how could we use data
and analytics to gain a competitive advantage? On that front, what we realized very quickly
was that there was a real opportunity in how we addressed our audiences, meaning the traditional
approach, and this is still often the dominant approach for these kinds of movies is what
we always call the spray and pray. Meaning, quite literally, spray the population
with TV ads and pray they go to the box office. That works in a certain world, but maybe not
even the world of today, but at some point it worked. What we realized back four, five years ago
was that we needed to be much more precise. It’s a game of impressions, meaning how do
we deliver the trailer, the TV spot, or the poster to the right people? Which of those things do we deliver, and in
what format? Doing that in a very precise and individual
way. What we’ve built are tools that enable us
to, at individual levels, predict people’s propensity, as we call it, their likelihood
to take the action we want, which may be a trailer view. It may be buying a ticket. That meant we had to use some very sophisticated
tools and techniques that we had to build ourselves. We built up a suite of assets and capabilities
that are all rooted back in AI. This was all when AI was not cool. [Laughter] This is a time at which AI was
Skynet or something. It wasn’t embedded really as broad as it may
be becoming now. The reason why we knew we needed to go down
that path and to use machine learning, neural networks, computer vision, all of that was
the scale which we needed to operate was so massive that, without those kinds of tools,
you’re almost back to that spray and pray mode where you are quite literally taking
broad guesses at large groups of people. Mm-hmm. This is really interesting. Let’s start to decompose this. Sure. You’ve got your data sources. Your goal then is to have these very micro-micro-micro-micro-targeted–
Yes. –can we say ad campaigns, marketing campaigns? Sure. Yep. Okay, so you’re targeted these micro-campaigns. You need the data, so you have sources of
data. You have these tools and techniques that you’ve
developed. Then somehow, there’s some link where you’re
now applying that to your ad platforms: Facebook, YouTube, wherever it is that you’re going. Yeah. Yeah, so you’ve got the right ballpark. I’ll go ahead and get more specific. The first step in that process for us is to
really try to understand people. The best way for us to understand people is
with data. Now, what is unique [laughter], and I tell
a story often. It’s true, and it’s not meant to be anything
other than just factual. More or less, the first day I walked into
Legendary, my first question was, “Where’s all the data?” Again, coming from a world that wasn’t connected
to Hollywood, I didn’t really understand how the dynamics worked, which was, we produce
a movie, we deliver it to a distributor, who then hands it to exhibitors. Then the exhibitors or, ultimately, maybe
an Apple, Amazon, or whomever, all the transactions, all the customer interactions happen at that
level, which are too removed from us. When the answer came back to me, “Oh, what
data do you mean?” I said, “Anything on people,” they came back
with an Excel spreadsheet of about 50,000 email addresses. I realized at that point that there was a
different challenge we had to face, which was, how do we actually get data on people? I’ll put that to the side for a second, but
the principle, though, that we were taking wasn’t data, necessarily. It was analytics. Our bet was not necessarily on getting the
best and most precise data on people. It was, how do we build the analytic tools
to take whatever data is available to us and use that to do our targeting. That is a recognition of a lot of factors
that I think were true then, but even more true now: privacy issues, social platforms
and how they share data and what levels of granularity they’ll provide you, regulatory
issues, all sorts of things. The data will shift. What’s available to you Monday might not be
available to you the next, or new things will pop up that weren’t there before. We knew we had to have data. That was table stakes, and so we invested
a lot of money, millions of dollars, into data to acquire data on people, on content,
unstructured data from social networks, everywhere we could find it. That wasn’t so much the bet. That was the table stakes. The real bet for us came at the next level,
which was, what can we build on top of that data? With that, what we drove towards was these
AI solutions. Meaning, could we take a billion or more email
addresses and attach hundreds, if not a thousand or more, attributes to those email addresses
that we created from, sourced them from, some kind of partnership to constructing them from
unstructured data, meaning of text, image, and things of that nature? We produced a very robust picture of people. Then once we had that robust picture, we needed
to do something with it. It’s inert if we don’t act upon it. The next option to take is to use, effectively,
that big table of data on people, which is not what it literally is, but that’s a good
visual of it, and create audiences from it and to make individual predictions. The first step in our process is to use our
models. There are many different inputs into them,
but to use them to hone in on who we think the most likely audience is. It’s not binary. In fact, we have three major categories that
we kind of drill in on specifics. The three major categories for us are people
we consider to be given, meaning they’re going to watch the movie no matter what. They’re wearing the Godzilla T-shirt. They’ve watched the Kong movie, from 30 years
ago or even 10 years ago, dozens of times, that kind of person. There’s a small number of them, but they’re
there. There’s a much larger number of people who
will never watch, who are never going to consume this content. That’s fine. We don’t want to spend impressions on them. Who we really care about are the people in
the middle of those two groups. We call them the persuadables. The people who we can persuade by giving them
the right piece of content or the right creative at the right moment through the right channel
is key. Those are trite things now. People talk about that a lot, but we try to
be very precise about it. The first step we’ll do is take that persuadable
audience and define them sort of exclusive of the givens and the nevers. Then, within the persuadable audience, we
will effectively score every single person. In the U.S., for a movie of a scale we typically
would work on, it could be 40 million or 50 million people. They’ll get a score from zero to 100, literally,
100 being very likely, zero being very unlikely. Once we have that, where we can, we deploy
media to them specifically and individually. A lot of people use the term onboarding. We might onboard them into, say, programmatic
buy on websites, publishers that you would see on the sidebar or across the top. That includes social media. That includes search. That includes video like YouTube. Wherever we can find these people, we’ll reach
them, and so we’ll launch these at the lowest granularity that that platform will accept. Sometimes it’s small audiences. Sometimes it’s individuals, but wherever we
can. Then, once that’s launched, the next thing
we’ll do is take very small pieces of those audiences, so not only cutting into small
microsegments, but now I’m taking even small subsegments of them to test. We call it calibration. We’ll launch many, many combinations, hundreds
or maybe thousands of combinations of subsegments and creative. That will give us an indication as to which
of those segments will respond better to which pieces of creative. Once we’ve done that, then we start scaling. Then we start actually applying more spend,
and that will lead us to a more global kind of scale. At that level, once we’ve done that, where
we can, and China happens to be the territory we can do this the best, we will actually
try to measure conversion. Meaning, we will actually try to see who is
buying tickets. Those ticket purchases will then feed into
our models and enable us to be more honed. What’s interesting about our approach is we
tend to do things that a lot of them are the opposite of what others do. A lot of folks will start to become narrow
and them maybe even get panicked and go broad. We do the opposite. The closer we get to release, the more honed
we’re trying to get and the more precise we’re trying to get. Right. We’re trying to get to as close to the actual
number of people that are going to buy tickets as possible. The platforms, this is really fascinating. The platforms that you’re using then are what? Can you be specific about that? Oh, sure. Yeah, it would be, in no particular order:
social media, so Facebook, Twitter, Instagram, Snap, all those platforms. It would include the Google platforms, which
would be search or YouTube. We’ll also do things programmatically, so
we’ll be able to target people across many different websites. Those are the major categories. We do try to do analytics to help us guide
what we would consider being nonaddressable media, like television buys and outdoor ads. But, it’s using the same concept of audience. We’re just now deploying it in a more coarse
way. You mentioned earlier that you’re going down
to the lowest level of granularity that these platforms will accept, essentially. Yes, that’s correct. In certain cases, we can provide individuals
and track them. That’s rare, but we can do that. In other cases, it’s these sort of subsegments
that may be hundreds of people or a thousand or two thousand, something like that. In other cases, for example, a television
buy, you’re buying against the people who watch that show. We’re making a prediction as to who we think
are going to actually watch the show, but we can’t actually precisely say, “Oh, these
are the 700,000 people we want to get through this show.” We are taking a bet that they’ll be watching,
but we don’t know precisely what they are. We have a question from Twitter, an interesting
one, from Gus Bekdash. He’s asking, “Do you apply this on a real-time
basis or just to get trends? How does that aspect work?” It’s a great question. The pace of these kinds of campaigns is very
fast. What will happen is, for any given movie,
the vast majority — when I say vast, 80%, 90% will be spent over the course of about
4 or 5 weeks. This is where I think people who are just
kind of awake and alive will see these massive sorts of media dumps out into the world. We knew that was the phenomena, and we knew
that we had to be able to react very quickly within those timescales. If this were an always-on campaign that ran
over the course of years, it’d be much different. We do try to operate very precisely within
that sort of very short window of four to six weeks. I would say our cadence for changes and adjustments
are typically within a day or so. It’s not real time in a sense of every minute
or every hour, but once a day we’re recalibrating and adjusting. You’ve had a set of tools and techniques that
let you take the data that’s coming back. Correct. You’re selecting the data to work on, to analyze,
and build models. Then you’re running your campaigns. Then you’re taking the data back from those
campaigns and doing what with it to optimize? Sure. I’ll just clarify the loop, but the loop you
described is very accurate. We have a source set of data we begin with
that creates audiences. Those audiences we then launch. We had to build our own technology platform. It was really crazy to scale. The data we were storing wasn’t sort of serviceable
by most data storage solutions, so we actually built our own data storage solution. Once we had our data storage solution–
I don’t mean to interrupt, but that’s incredible. Oh, sure. Yeah. I mean I’m just thinking about all the tools
you had to build. That’s wild. Yes. Yeah, it’s crazy. It was funny, going back to the beginning. I was a guy in a room. [Laughter] I had a checkbook, effectively,
and we could have done a number of things. We could have built a mosaic of solutions. What we found was that that didn’t exist,
and so we built out a team. Our team is about 70 people. Of the 70, we’ve got about half are some form
of an engineer, whether they’re data scientists or computer scientists, and we have people
who have all kinds of disciplines. We accumulated these people, and we built
these tools because they didn’t exist, and we couldn’t find that solution. It’s really that singular solution that goes
from front to back. There were a lot of good point solutions along
the way, but they didn’t have the full integration. The loop you described is a very logical loop,
and that’s exactly what we were trying to build toward, but we had a hard time finding
the solution that would meet both the speed and the pace at which we were spending, along
with the sophistication with which we wanted to spend. To go back to your loop, this data platform
that we’ve created will suck in data from whatever sources we’ve started with, the initializing
sort of data. Then it will launch the media out into the
different platforms. To your point, as the campaigns run, new data
is being created constantly. That comes back into the system, enables us
to calibrate and change dynamically, and then re-spend. It’s sort of a virtuous cycle that continues. I think I know the answer to this question,
but here you are. You’re basically giving away your process,
and you seem willing to go to whatever level of detail I ask, or the audience asks. And so, why aren’t you uncomfortable sharing
your secrets and your process with the world? Sure. I hope this doesn’t come across as arrogant,
but it’s a hard thing to do. [Laughter]
[Laughter] I was going to say, probably because how would you replicate it? Right. Yeah, there’s a certain alchemy we had to
create even to be able to pursue these goals. That alchemy included people. It includes data. It includes technology. It’s a lot of things. No, the process and the approach, I think
a lot of things hopefully that I’m saying are sort of borderline initiative. Certainly, we’re transparent about a lot of
this. The real secret sauce is what’s happening
the next level or two down, the actual tools we have, the actual techniques we’re using,
the specific data types we have, and the ways in which we connect to these different places,
some of which are ones that people could reverse engineer and kind of anticipate. Others may be more subtle. That’s the level where it gets a little more,
I think, proprietary. But, at this level, again, it’s just a hard
problem to solve. We also, frankly, are these sort of data nerds,
right? We like to share, to learn what other people
are doing, and there may be ways in which we can learn from others by being open about
what we do. My mind is kind of reeling with what you’re
doing. The targeting then, you must have developed
real areas of expertise in each one of these platforms. You must know as much about these platforms
as the developers do, essentially, I would imagine at this stage. I think a good way to think about it, and
I appreciate the compliment, but a good way to think about it is we have a very specific
set of goals and objectives, and they’re very practically oriented. We need to have people buy tickets to movies
or watch our TV shows, whatever it is. We are willing to invest the time and resources
to best understand these platforms for that purpose. I think, for those purposes, for the kinds
of things that we want to do, and they’re broader than just movies and TV shows. It’s when you want to build brand awareness;
when you want to build consumer intent. Those objectives are ones that I think are
shared by a lot of industries. The platforms are trying to solve many, many
more problems than just those specific things. Yeah, I do think we’ve spent as much time
as anybody on trying to figure out how to take full advantage of those platforms for
that particular set of activities. Understood. I want to remind everybody we’re speaking
with Matt Marolda, who is the chief analytics officer at Legendary Entertainment. Right now, there’s a tweet chat going on using
the hashtag #CxOTalk. You can ask your questions directly at Matt. Matt, as you said, the general approach is
one that’s being applied by many different companies in many different industries. Do you ever have companies like, say, retailers
coming to you wanting to license your technology because these techniques would work for anything? Yeah, it’s interesting. This is not false modesty by any means at
all. When we started back in 2012 and ’13, I wasn’t
sure any of this was going to work. We had no idea. It was a gamble. It started to work and pay off quickly. We were fortunate that it got us our own kind
of ROI very quickly. For about a year or two into that process,
we were relatively quiet about it. We didn’t want to pat ourselves on our back
or take undue credit because there are a lot of factors into any of these successes or
even failures, frankly, that might happen with a movie or TV show. That being said, folks did begin to notice. People, especially in our own industry, initially,
saw things that they were surprised by. They saw patterns that you could observe but
weren’t quite sure what was causing the pattern. For example, people use what they call tracking
in Hollywood, which is kind of like polling. It’s sort of a national poll that assesses
people’s interest in a movie. Tracking has been used for decades to predict
box office outcomes. Our movies consistently outdid tracking, meaning
the old ways of predicting the box office outcome were being broken very consistently
by what we were doing. And so, that drew attention. For a while, we said, “No, we won’t share
this with anybody. This is a competitive advantage.” But then, what we realized was maybe not so
much for the creative side, which is something we’ve held closer, but when it comes to the
marketing, this is an asset and an engine that we’ve created that could be deployed
for others. And so, over the last 18 months or so, we’ve
been sort of slowly, but with some aggression, taken on what we call our partners, people
who do use our tools and license them for their own campaigns, whether it’s for other
movies or TV shows. We’ve done things in CPG [and] hospitality
as well. You mentioned the tracking. I know that one of the things that you’ve
done is used biometrics to see how people respond to movies. Oh, yeah. Maybe you can tell us about that. Sure. The phases, I think of at least, for a movie,
very broadly, there’s, for lack of a better name, a greenlighting phase, meaning the point
at which you decide whether to make a movie or not. Then there’s a production phase. Then a postproduction phase, which means the
editing and any other refinements you do to the movie after it’s been filmed. Of course, this is from our point of view
on how we look at it. Then the marketing of it. In that postproduction phase, which I think
is what you’re describing or asking about, is the tools we’ve used to try to hone in
on a movie. It’s the same thing, I would say, at the greenlighting
stage. By the way, we were never involved in the
production stage at all. All we’re trying to do is provide inputs back
to the filmmakers and the producers that they can then make judgments and make decisions
based on them. The difference is, our goal is to provide
it at a level of granularity that is really robust and unique so that they can be as best
informed as possible. Biometrics is one path towards that. We have done things where we’ve taken movie
theaters, and we’ve put several hundred people, maybe 300, 400 people in a movie theater,
and put the wristbands on them. They’re like a Fitbit that would track things
like heart rate and others. We have an iPad on the back of a receipt to
then capture a face to then grab facial emotion and other recognition sort of indicators. We’ll tie that back to a second-by-second
view of the movie and allow us to at least give these broad, but also very specific sometimes,
comments about what’s happening. I would say we wouldn’t discard other things,
too. We will also use what people call sort of [surveys]. It’s the cards, meaning the little card you
fill out afterward. We still use those. We try to blend traditional approaches with
the slightly more avant-garde ones because we think that there’s value in all of them,
especially when they’re integrated. You mentioned the relationship to the creative
side, and so I’m really wondering how. We don’t think of the artistic process as
being informed by predictive analytics. Is there a tension there? How do you work with the creative side? Well, it’s certainly a collaboration, right? On the creative side, especially, we’re trying
to make sure we recognize kind of where we fit into the process. There are many, many things that the creative
team is working on at any given moment on a movie – many. There’s only a handful of them we can actually
impact. But, just like you’d want to understand what
an actor might be like to work with or you’d want to watch their previous work, those kinds
of things are readily available to a producer. They can call and find that out. Why not also understand what kind of audience
the property might generate on its own; what kind of box office potential it has in advance;
what kinds of talent might align well with that? Again, it’s not to say that I’m prescriptively
[saying] here’s what you have to do and you’re dumb if you don’t. It’s more like helping to either hone options
and get a little more specific or uncover ones that they might not have thought of. Again, they have to use judgment and perspective
to be sort of the final arbiter on how it all pulls together. But again, the tools we’ve tried to build
are inputs in that process that can be very enlightening. I think one of the things that at least I’ve
learned even from sports into here, but certainly here as well, is there’s a real premium on
humility, so coming in with an open mind. An open mind helps in a lot of ways. It helps look at problems in a different way
and to understand that there are maybe solutions that exist, but also helps in how you present
the solutions or the findings you might have uncovered. Humility is important because we want to be
able to be confident and clear about what the data analytics are telling us, but also
humble in its presentation so we understand the broader context that it’s been used in. How do you get folks who are on the creative
side that, in their entire life, would never have thought of applying data and analytics
and probably resist it? How do you get them to have that open mind
and be receptive? Sure. Sure, so the attitude overarching, of course,
is that humility, but that confidence, that sort of combination of the two. The real practical steps have been a learning
curve, quite literally. Starting with things like here’s a small sample
of what’s possible. Also, like a lot of things, it’s about listening
and asking questions. I’m trying to understand what is it they’re
trying to solve, or what’s the riddle that they haven’t been able to crack on their own? Going back and then building a tool that can
help them actually get there in a way they couldn’t have done through intuition or other
more traditional means. It’s both the attitudinal thing, but also
a sort of learning curve where it’s dolled out. We’ve been fortunate to see that people become
kind of voracious about it. They ask more questions and you get more in
depth. It becomes a real conversation, which is exhilarating,
I think, for everybody. It’s led to stars being cast in TV shows. Even recently, there’s a well-known actor
right now who has a guest spot, basically, on a network television show because we used
these analytics. Again, it’s a great example where the producer
came in with a list of, I think it was, 30 or 40 actors. They wanted to understand who had the best
fit with the show, who could bring new audience but still have a consistency with the core
of the show and be able to provide a ratings boost. The analytics helped us hone the 30 or 40
that they were considering down to 3 or 4. Of the three or four, then they were able
to go and focus on the one they thought was the best, and they were able to bring him
in. It’s a process that looks like that where
it’s collaborative and communicative. It seems to work and, over time, then the
creative side has become receptive because they’ve seen the benefit. I’m assuming that must be the cast. Yes. It was two things, I think, and they are both
human nature. One is it’s a great way to be able to have
a justification. It’s a great way to be able to say, “Hey,
we did all these things and here is hard evidence to support the decision.” That’s really helpful. I think, also, they have seen the efficacy. Being the type of data nerds that we are,
we would love to be able to do the test. You’d love to use a really basic sort of typical
nomenclature. You want to do the A/B test. It would be great if you took actor A and
actor B, created the same show, and saw which one was better. That’s obviously impossible. That level of efficacy is hard to measure. The confidence that the decision-maker has
to make that final casting call is much greater with these tools behind them. What about the ethical issues? You alluded to that earlier of the targeting. You’ve got so much data. How do you balance the privacy? You want to target very narrowly based on
intense interests or subtle interests, but you don’t want to be creepy and overstep your
reach. How do you balance that? Ah, so there are lots of different ways in
which we need to think about that problem. There’s one you hit on, which is a regulatory
one. To be very clear and without ambiguity, we
have bright yellow lines around all that. We don’t take anything that’s anywhere near
a risk. That being said, that’s an easy boundary to
create. That’s a simple thing. You would just do that by nature. That includes things like we don’t scrape
data. All the data we acquire is acquired legitimately. We have an agreement, an open relationship,
and often a monetary relationship with the source of the data. Not only are we staying well within the bounds
of the regulatory constraints, but we’re also well within the bounds of the data itself. Then there’s another level I think you kind
of touched on too, which is, how do you interact with a consumer in a way that doesn’t feel
creepy? We certainly are conscious of that. That’s where the tools and the platforms we
tend to use for these purposes, these addressable platforms we talked about earlier–social,
search, video, display–those are ones where as far as the consumer of whatever they’re
watching on that platform are doing and they’re getting these ads served to them, it’s hard
to discern whether that ad is anything special for them. It’s just one of others. That feels like a very natural way. It doesn’t feel creepy. If we were to suddenly start popping up in
places and addressing you directly, and you didn’t realize that we were there, it’s the
kind of thing that I think would throw people off. We’re well short of that. We’re operating within channels that the advertisements
are coming in a way that you would expect. They just happen to be very precisely directed. That’s really interesting. If your data science is really working well,
and you’re operating on the right set of data, from the user perspective they’re receiving
advertisements that are natural, that seem organic, and that just flow. Right. Therefore, there’s no creepiness factor because,
actually, this makes sense. Yeah. In a lot of ways, and I don’t think people
always think of it this way, but that’s the implicit barter we make with all those platforms. That’s the implicit barter we make for having
the ability to search the Internet for free is that there will be ads served to you. We’re just kind of working within that ecosystem. Again, it goes back to what I think I said
a little earlier. The bet we made was always on the analytics
themselves. We recognized, because of these shifting,
even sometimes broader political — like Europe was a great example where the regulatory issues
there are very different than they are here, and people’s view of data and even that implicit
barter is different. We knew that those things would change. We couldn’t predict. [Laughter] Even for people who spend all their
time thinking about these things, it’s impossible to predict how those tides will ebb and flow. But, what we could predict was that they would
ebb and flow. Therefore, we wanted to play at this other
level above it where the analytics are, and then we will just adapt to whatever we can
do. As things become more available, that’s great. If they become constrained, we’ll look for
other alternatives. We’ll kind of balance all those activities
together. At this point, I’m assuming. I don’t want to put words in your mouth, but
I’m assuming that, inside Legendary, your team, your work must be kind of diffuse in
a way throughout the whole organization, which is to say at every stage of the process your
team is involved in some type of integral way. Yeah, often. It’ll vary a little bit by project just depending
on how things are going and where the need is. Some movies develop very quickly and nicely
right through postproduction and don’t need a lot of intervention. Others are a little more challenged. We’re always very involved in the marketing
end. That’s the part that’s consistent. On the other side, there are movies that come
down the pike that we don’t need to do a lot of analysis on. We have a movie called Skyscraper. Of course, you want Dwayne Johnson to star
in a Die Hard type movie. [Laughter]
[Laughter] It doesn’t require a lot of analytics. But, there are others that are much more carefully
considered. It varies just a bit based on the specific
project. Yeah, we touch all those parts, again with
the real exception, of course, truly being like scriptwriting and production. That’s an area we don’t ever get involved
in at all. But, the other parts in terms of greenlighting,
some of the postproduction, and certainly the marketing, we’re very involved in. Why not scriptwriting? I’m sure you could give insights into scriptwriters
that would be really valuable to them. Why not? Yes, sort of. [Laughter]
[Laughter] Again, we still believe that there are very
creative aspects to all of this. This is a human endeavor, still. There are probably ways in which you could
create artificial intelligence. I’m sure people are doing this now–we haven’t
spent a lot of time on it–to construct things more specifically. I think what we do, I know, is that we do
try to at least provide the same kinds of inputs I was describing earlier: thematic
kind of observations or other signals we might pick up from prior movies that are similar
and the audiences around those movies, and again very broad brushstroke type things. I can think of a showrunner, in particular,
a very well-known showrunner that we have a show with, who is very receptive to those
kinds of things. He’s very opened minded about the data signals
we would pick up that might help him shape the show, but we don’t get into anything specific. We’ll bat around a few big ideas, but he then
takes those and puts his creative brain toward it and then figures out a way to solve it
on his end. Yes, it’s an interesting dynamic, for sure. In a number of different areas, it seems like
you have constructed very careful boundaries like you were describing how you collect data,
regulatory issues, the relationship to screenwriting, things like that. It seems like you’re very careful where you
go deep and, at the same time, you’re equally careful circumscribing where you won’t go. Yeah, that’s true. We tried to establish a set of rules. I think it comes back a little bit to some
of the questions you’re asking earlier around how we get buy-in and adoption. By self-declaring those boundaries and making
it very clear, the perspective we have and what we’re trying to do, it makes the conversation
much easier. It’s human nature; as soon as there is any
feeling of encroachment or someone trying to usurp someone else’s role and responsibility,
that really gets the dynamics misaligned. The objective is to almost self-declare and
say, “Hey, look. This is what we think we can do. Here’s where we believe the impacts can be,”
and they’re well short of anything that’s near encroachment, right? It’s something that’s additive, supportive,
and with again that humility kind of wrapped around it. Then that is a true thing, it’s sincere, but
it also has that byproduct of making people feel comfortable that we’re not trying to
go too far or push things in the wrong direction. There are a few questions. We only have about ten minutes or less left,
and there are a number of different big areas that I want to ask you about. Very quickly, can you describe the composition
of your team? Sure. Who do you need if you want to create this
kind of magic? Yeah. We need the right people, for sure. I’ve said humility, I feel like, four times. It must be the word of the day for me. That’s an initial starting point for us. We look for people like that. Of course, we have other things we’re really
interested in, and so the specific skill sets we have accumulated. On a data science side, it’s multidisciplinary. The person who runs our data science team
has a Ph.D. in astrophysics. That’s a discipline you wouldn’t expect at
a Hollywood studio. Just like that discipline, there are people
who have different backgrounds in social sciences like human decision sciences, or they are
statisticians or econometricians. That’s a whole category of people we have
as data scientist folks. On the software development side, we knew–and
we talked about it briefly earlier–that we were going to have these very large data sets. We needed people who had the skills to be
able to build these repositories to query and analyze data at remarkable speeds, to
be able to even build the infrastructure and the thousands of servers we have running at
any given time to support all that, to build the user interfaces that make it all work. Those were skillsets we were very specific
and targeted on. We also needed the other half of our team,
basically, of people who are experts at applying these kinds of outputs into a campaign. That last group I just mentioned was by no
means the last. In fact, we considered all three simultaneously
because we knew that if the data science team and the development team built all the amazing
tools they build but they were just shiny toys on a shelf, it was all for not. We needed to make sure we had a group of people
who knew how to translate those tools into action. That creates the whole iterative loop we use
to further develop. Here you are trying to build this team and
you realize, for example, well, the storage systems that are out there are not going to
work. You now have to go out and hire a bunch of
storage system designers. Right. Exactly. That’s insane! [Laughter]
[Laughter] It is. Yeah. You’re not the first to say that. It’s actually probably a little insane, yes. Okay. I say it’s insane with the greatest of respect
and admiration. Oh, it’s very practical. I think we were able to solve a problem we
couldn’t have solved otherwise, but it’s crazy we went to that extreme, I think. Yeah. But, you had to do it. You really did. Yeah, we had to. We were forced to out of a practical need. Yes, that’s right. Okay. I guess when you’re keeping that practical
need and really monitoring efficacy, “Does it work at every step?” then you can explore
these new territories, actually make it work, and execute it. Yeah. I think there’s another thing, too, which
we were empowered from early on. A lot of people are punished, penalized, or
even fired for taking risks and not making the safe choice. We were the opposite. I think we would have been punished or penalized
for taking the safe choice and not taking a risk. We had the attitude from the beginning that
failure was great. Failure is fine. It’s a learning opportunity, but you want
to do it fast and cheaply. [Laughter] That’s the idea. Yeah, we were willing to take a lot of [risk],
and everything I’ve described was built progressively. It wasn’t like we necessarily set out one
way or the other. It’s that we built from a need and grew it
from there. You were layering. Correct. You build something, you layer, you experiment,
you layer, and so forth. Exactly. Okay. In the spirit of trying to accomplish large
topics within the space of just a couple minutes because we’re really running out of time,
where is the future of this? You’re on the cutting edge, and so you’re
viewing. What’s next, not 20 years out, not 10 years
out; what’s coming down the pike? For us, I think the thing that we think a
lot about is two things. One is the increasing addressability of media
channels, so the ability to get more and more precise. That feels around the corner. Whether that means addressable TV in that
you can send an ad over whatever form of viewing you were doing, that’s one thing for sure. The other thing for us, which is always that
holy grail–in a lot of industries this is not the case, but for things like ours–are
conversion measurement is hard. Meaning, can we tell if someone took the action
we wanted? As data becomes stronger and better there,
that just makes everything better. Those are the two things that feel, in these
sort of 18 to 24 months, or maybe even a little longer than that, but the 1 to 3- or 1 to
5-year range. The more narrow targeting then, you have the
data. Correct. You have a lot of the data. The more narrow targeting then would depend
on the enhancements of the platforms. Correct. Is there something about what Facebook is
planning or YouTube, or Google that you should be sharing with us? No, no. [Laughter]
[Laughter] No, I have no special knowledge. We just know the day is coming. Our goal is to be prepared for that day. The day that we’re ready for is the day where
we could find every single person we want to find, wherever they are, and however we
want. That day isn’t here yet, but it’s coming,
and that’s the day we’re preparing for. You’re just anticipating that. Right. Given the evolution of these things, that’s
going to happen pretty soon. Correct. Yeah, it’s a bet. I think it’s a pretty safe bet, but it’s definitely
a bet. Okay. Finally, what advice do you have for businesspeople,
and not necessarily in the movies, but businesspeople in retailing, in marketing, and whatever it
might be who want to use data and analytics approximating for, say, CMOs in their marketing? What would you recommend to get in below the
surface, but obviously not to the extent that you’ve taken it? I think, first and foremost, it’s adopting
this attitude. I think what I’ve seen, at least in a lot
of organizations, is that data or analytics, or both, spring up as departments. They sort of become their own unit or something. It’s almost like a reverse factitial effect. There never was a place for them before, so
now they have to have a place, and they get created. I think it’s almost the opposite. Even though my title even has analytics in
it, really it’s a discipline applied to subject areas. I think the first sort of step is to be able
to break that thinking that, “Oh, data is over here in this corner and people need to
think about it as its own sort of discrete thing,” and in fact, it’s actually almost
the opposite. It’s a tool and a toolset that can be applied
to something specific. That’s just a mindset thing. I think, more practically, data collection
itself is hard. That’s why we had to build what we built. If people can capture data and keep it at
a level — like, we don’t throw any data away. We have innumerous amounts of data that may
not even be any good, but we never know if we could use it or if it’s somehow useful
for us in the future. Storing data and keeping it is really another
thing. Then having said that, this sort of higher
level thing that, like I’ve mentioned a couple times that we try to do, which is, realize
that it’s about the analytics of the data, not the data itself. That’s where the value comes. Okay. We have another question from Twitter. This is from Zachary Jeans. He actually is asking the question that I
meant to ask you about what’s coming down the pike. He’s asking it in a better way. He asks, “What is a technical challenge that
you will be facing during this next year that is new?” Hmm. There are so many. [Laughter]
[Laughter] It’s hard to almost pick. The ever-growing challenge for us is working
through the shifts in the technology and data that’s available. Part of it is just the nimbleness of me to
have to be able to adapt. Those are typically the ones that are at the
forefront for us. It’s not so much a technical challenge. It’s the functional challenge, which is, we’re
always trying to solve the problem around measurement and conversion. Meaning, after all the work we’ve done, sometimes
it’s very hard to tell, did we actually get the person we wanted to buy the ticket? That’s a challenge that we’re always confronted
with and we’re always working toward. Okay. Wow! This has been a very, very fast 45 minutes. Matt, thank you for taking time to come talk
with us today. Oh, no, I appreciate the opportunity. It’s great. I thank you very much. You have been watching Episode #276 of CxOTalk. We’ve been speaking with Matt Marolda, who
is the chief analytics officer at Legendary Entertainment. What an action-packed and very fast and interesting
show this has been. Everybody, come back next week; we have more
shows. Don’t forget to like us on Facebook. Thanks so much, everybody. Have a great day. Bye-bye.


Leave a Reply

Your email address will not be published. Required fields are marked *