You know how accurate you need to be — great. But how do you actually create your estimate?
You can follow these steps to make your estimate as robust as possible while minimizing the amount of time you spend on it:
Let’s say you work at Netflix and want to figure out how much money you could make from adding games to the platform (if you monetized them through ads).
How do you structure your estimate?
The first step is to decompose the metric into a driver tree, and the second step is to segment.
Developing a driver tree
At the top of your driver tree you have “Games revenue per day”. But how do you break out the driver tree further?
There are two key considerations:
1. Pick metrics you can find data for.
For example, the games industry uses standardized metrics to report on monetization, and if you deviate from them, you might have trouble finding benchmarks (more on benchmarks below).
2. Pick metrics that minimize confounding factors.
For example, you could break revenue into “# of users” and “Average revenue per user”. The problem is that this doesn’t consider how much time users spend in the game.
To address this issue, we could split revenue out into “Hours played” and “$ per hour played” instead; this ensures that any difference in engagement between your games and “traditional” games does not affect the results.
You can then break out each metric further, e.g.:
- “$ per hour played” could be calculated as “# ad impressions per hour” times “$ per ad impression”
- “Hours played” could be broken out into “Daily Active Users (DAU)” and “Hours per DAU”
However, adding more detail is not always beneficial (more on that below).
Segmentation
In order to get a useful estimate, you need to consider the key dimensions that affect how much revenue you’ll be able to generate.
For example, Netflix is active in dozens of countries with vastly different monetization potential and to account for this, you can split the analysis by region.
Which dimensions are helpful in getting a more accurate estimate depends on the exact use case, but here are a few common ones to consider:
- Geography
- User demographics (age, device, etc.)
- Revenue stream (e.g. ads vs. subscriptions vs. transactions)
“Okay, great, but how do I know when segmentation makes sense?”
There are two conditions that need to be true for a segmentation to be useful:
- The segments are very different (e.g. revenue per user in APAC is multiple times less than in the US)
- You have enough information to make informed assumptions for each segment
You also need to make sure the segmentation is worth the effort. In practice, you’ll often find that only one or two metrics are materially different between segments.
Here’s what you can do in that case to get a quick-and-dirty answer:
Instead of creating multiple separate estimates, you can calculate a blended average for the metric that has the biggest variance across segments.
So if you expect “$ per hour played” to vary substantially across regions, you 1) make an assumption for this metric for each region (e.g. by getting benchmarks, see below) and 2) estimate what the country mix will be:
You then use that number for your estimate, eliminating the need to segment.
How detailed should you get?
If you have solid data to base your assumptions on, adding more detail to your analysis can improve the accuracy of your estimate; but only up to a point.
Besides increasing the effort required for the analysis, adding more detail can result in false precision.
So what falls into the “too much detail” bucket? For the sake of a quick and dirty estimation, this would include things like:
- Segmenting by device type (Smart TV vs. Android vs. iOS)
- Considering different engagement levels by day of week
- Splitting out CPMs by industry
- Modeling the impact of individual games
- etc.
Adding this level of detail would increase the number of assumptions exponentially without necessarily making the estimate more accurate.
Now that you have the inputs to your estimate laid out, it’s time to start putting numbers against them.
Internal data
If you ran an experiment (e.g. you rolled out a prototype for “Netflix games” to some users) and you have results you can use for your estimate, great. But a lot of the time, that’s not the case.
In this case, you have to get creative. For example, let’s say that to estimate our DAU for games, we want to understand how many Netflix users might see and click on the games module in their feed.
To do this, you can compare it against other launches with similar entry points:
- What other new additions to the home screen did you launch recently?
- How did their performance differ depending on their location (e.g. the first “row” at the top of the screen vs. “below the fold” where you have to scroll to find it)?
Based on the last few launches, you can then triangulate the expected click-through-rate for games:
These kind of relationships are often close enough to linear (within a reasonable range) so that this type of approximation yields useful results.
Once you get some actual data from an experiment or the launch, you can refine your assumptions.
External benchmarks
External benchmarks (e.g. industry reports, data vendors) can be helpful to get the right ballpark for a number if internal data is unavailable.
There are a few key considerations:
- Pick the closest comparison. For example, casual games on Netflix are closer to mobile games than PC or console games, so pick benchmarks accordingly
- Make sure your metric definitions are aligned. Just because a metric in an external report sounds similar doesn’t mean it’s identical to your metric. For example, many companies define “Daily Active Users” differently.
- Choose reputable, transparent sources. If you search for benchmarks, you will come across a lot of different sources. Always try to find an original source that uses (and discloses!) a solid methodology (e.g. actual data from a platform rather than surveys). Bonus points if the report is updated regularly so that you can refresh your estimate in the future if necessary.
Deciding on a number
After looking at internal and external data from different sources, you will likely have a range of numbers to choose from for each metric.
Take a look at how wide the range is; this will show you which inputs move the needle on the answer the most.
For example, you might find that the CPM benchmarks from different reports are very similar, but there is a very wide range for how much time users might spend playing your games on a daily basis.
In this case, your focus should be on fine-tuning the “hours played” assumption:
- If there is a minimum amount of revenue the business wants to see to invest in games, see if you can reach that level with the most conservative assumption
- If there is no minimum threshold, try to use sanity checks to determine a realistic level.
For example, you could compare the play time you’re projecting for games against the total time users currently spend on Netflix.
Even if some of the time is incremental, it’s unrealistic that more than, say, 5% — 10% of the total time is spent on games (most of the users came to Netflix for video content, and there are better gaming offerings out there, after all).