is a god-send for market research. If you want to understand interest in a particular term you can just look it up and see how it’s changing over time. This is the kind of data we could do some serious data science with. Or rather, it would be if the data was actually usable.
In reality, Google Trends exists solely to do what it says: show trends. The data is normalised and regionalised to the point where it’s impossible to get a hold of comparable data to do any meaningful modelling with. Unless we have a few tricks up our sleeve.
In my last post on this topic we introduced the concept of chaining data across overlapping windows to get around the granularity limitations of google trends data. Today we’re going to learn how to compare that data across countries and regions so you can use it for real insights.
Motivation: Comparing Motivation
Google trends allows the downloading and reuse of Trends data with citation, so I’ve gone and downloaded the data on motivation for five years and scaled it so we have one dataset of motivation searches for each country that gives us a rough idea of how each country’s interest in motivation changes over time. My goal was to compare how motivated different countries are, but I have a problem. I don’t know whether a google trends score of 100 searches in the US is bigger or smaller than a score of 100 in the UK, and my first suggestion for how to work that out fell flat. Let me explain.
So when I started this project I wasn’t a connoisseur of Google Trends and I quite naively tried typing in UK motivation, then adding a comparison and typing it motivation again and changing the location to the US. Admittedly, I was confused as to why it was the same graph. So then I thought it was just that UK and US were too similar so I added Japan and it wasn’t until I got to China that I realised that the graph was changing all of the lines to be that country’s motivation.
So if I can’t get the countries on the same graph then I can’t compare them. Unless I find a more creative way…
My next brainwave came from looking at the US, because if you scroll down on google trends you’ll see that there’s this subregion section showing the states in the US in relative terms. So the state with the highest search volume is set to 100 and the other states are scaled accordingly.
So I thought I was a genius, I’ll just set the region to be worldwide, see the different numbers that come out for my countries of interest and just multiply the results for that country by that number.
But it turns out, I had misunderstood something fundamental again. And I’m sorry but we’re going to need to do some maths to explain it.
The Maths Behind Google Trends Normalisation
So I grabbed ninety days of data from the US and the UK from the 24th of April on two separate google trends graphs as you can see here. They’re both scaled so the maximum is at 100 which occurs on a different day for each country.
The problem is that because we’re looking at two different countries, the google trends scores are in fundamentally different units for each country. Just like inches and centimetres are different units of measurement, so are US Google Trends units and UK Google trends Units. And unlike inches to centimetres, we don’t know the conversion factor here.
Let’s assume that on the worldwide graph the US is given a score of 100 and the UK is given a score of 50. The UK score of 50 means that the peak of UK is 50% of the peak of the US. On a first look this might suggest that the conversion factor between these two units is a half, ie UK units are half the US units or equivalently one US unit is 2 UK units. I’m now going to convince you why this isn’t true.
Let’s take this to a day that’s not a peak day. Let’s look at the 30th April and say hypothetically that its score was 70 in the US and 80 in the UK. This means that the score in the US that day was 70% of its peak and the score in the UK that day was 80% of its peak. Let’s look at it with some maths:
70% of US peak = 70% * 100 US units = 70% * 2 * 100 UK units (based on the scaling factor of one US unit = 2 UK units) = 140 UK units
Now looking at it from a UK perspective:
80% of UK peak = 80% * 100 UK units = 80 UK units
And last time I checked, 140 was not double 80.
Just because the peak of US is twice the peak of UK doesn’t mean that for the whole time period the US data is twice the UK data!
So okay, we can’t just take the worldwide ratios to compare the data of different countries. So what can we do?
The thing I love the most about data science is that the underlying science and methodologies we use can translate across multiple different domains so for this problem I’m going to take a similar approach.
Because I learned my data scientist skills before I even knew what a data scientist was, forged in the chaos that is the trading floor of an investment bank. If you’ve ever heard of the term “Exchange Traded Fund” then that might give you a little bit of an idea of what you’re in for, but if not do not fear.
Taking Inspiration from the Stock Market
So the stock market, as you’re probably aware, is a place for buying and selling equity, or shares in a company. These shares are a partial ownership and usually come with things like voting rights or the ability to receive dividends, like a small bonus for being an owner of the company. Stocks can be held by individuals like you and I or big investors like banks and hedge funds or other private companies.
The stock market can be used as a measure of the economic health of a country. When stocks are going up, we’re in a bull market and the country is, in theory, financially prosperous. When the market starts to fall we enter a bear market and things are going less well. This is a huge simplification, the markets move according to human behaviour which is a notoriously difficult thing to understand, but for our purposes this generalisation holds : we can gain an understanding of a country’s economic health based on its stock market.
Tracking the Market Through Indices
So how do we track the stock market as a whole? Well the obvious thing to do is to take all the shares on the stock exchange and add up all their prices to get an overall number for the value of the stock market. But this isn’t how it works in reality. In reality, we use indices.
You’ve probably heard of the S&P 500, an index built up of the 500 biggest companies in the US. It’s used to track the US market because, being the biggest companies, it covers about 80% of the total market capitalisation, that’s value effectively, and are also very liquid, that means they’re easily traded and their prices move a lot.
Because they cover the majority of the market, it’s a good representation of the whole market in a smaller collection of 500 stocks. Why 500? Well, for starters the S&P 500 was introduced in 1957 and I was going to say that the computational power available to calculate the market capitalisation of thousands of stocks wasn’t there like it is today but it’s even more interesting than that because the S&P 500 was only created with 500 stocks because of a new electronic calculation method that enabled 500 stocks to be included in the calculation. Before that, indices were even smaller because they were calculated by hand!
Why you’d estimate in this big data world
Now we do have the computation power to calculate the entire market if we want, a few thousand stocks is small fry in today’s big data world, but it’s not really necessary. Adding in smaller companies means an increase in overhead in tracking them all and also some of them might not get traded very often, meaning the information about them goes stale. The pros of adding them are outweighed by the cons.
And this conversation pops up all over finance. The UK has the FTSE-100, a basket of 100 stocks. Commodity baskets can be used to track the health of specific industries such as oil or agriculture. And inflation, measured by CPI, is made up of a basket of goods to track price changes over time.
So if a basket of representative items can be used to measure the entire stock market, or inflation, why not use it to track search volumes?
Applying ETFs to Google Trends Data
So if I want to use this concept, what I really need is some idea of the most commonly searched terms that I can use to build a S&P-500-esque index for each country. One of the things we can use is Google Trend’s Year In Search functionality to get basket candidates from popular search terms.
So let’s say for now that I did have the average search volumes for at least one country, let’s say the US. The way we get around this is to average the scaling factors for a subset of my basket (or the whole basket) and have this as an average US google trends units to real world search volumes. And I can then use this number to get an idea of the absolute search volumes for motivation.
Making Search Data Truly Comparable Across Countries
Now there are a couple of caveats here. I don’t know how representative my basket is. In reality, I’m constrained by how much google trends data I can manually download so my basket was small, just nine items. In addition, some countries will have very large search volumes for particular terms that are completely absent from my basket. For example, I have Facebook and Instagram in my basket which are very popular in places like the UK, US et cetera. But in China, the equivalent would be WeChat which isn’t used very much outside of the country.
I wouldn’t put WeChat in my basket, because it’s not representative of the vast majority of countries around the world. But it is highly representative of China.
The other problem I have to solve is that even if I can benchmark for one country, how do I scale the other countries which I don’t have a benchmark for?
In order to tackle this problem I had a think about things that might influence the search volumes of a country. An obvious one is the population of the country. The US has five times as many people as the UK so it wouldn’t be surprising if the US had five times the search volume of the UK. But actually I think we can do better.
Because internet access is not uniform across the population. There are still many places in the world where people find themselves without internet access. There are older people who grew up without technology and have no interest in learning, toddlers who haven’t yet been given a tablet or people who just for whatever reason decide to opt out. The demographics of these non-internet users will be very country dependent, and so a more accurate figure could be the percentage of internet users in each country.
I actually managed to find this data and combining that with population we can get a figure for the absolute number of internet users in each country. By taking the ratio of internet users in the country and the US, we can calculate an adjustment factor for the US scaling factor for each country to leave us with a method to calculate the absolute search volume of any term for any country.
When the maths simplifies itself
Now with that in mind, I do have one more caveat. Because in order to compare countries and model motivation trends, what we’re modelling isn’t absolute search volumes for motivation. If we were then we’d conclude the US is less motivated than the UK because it searches for motivation more, but in reality we know that they’re not necessarily less motivated, there’s just more of them.
So to solve this problem I’d need to look at search volumes of motivation as a proportion of total search volume and we’ve already built something to model this: our basket of terms. So I can calculate absolute search volume for all of these terms, add them up for the basket and divide absolute motivation by absolute basket.
You might have noticed something here. If I do that, won’t all my scaling factors cancel out? And actually the answer is yes. All of these scaling factors cancel out rendering the work we’ve done before unnecessary, from a certain point of view.
But actually, it’s not unnecessary. Because if I’d started this post saying “let’s just add up the google trends score of the basket and divide motivation by it” you probably would have thought “why? Is that something we can actually do?”. Until we did this analysis, we didn’t know we could.
There’s also an extra benefit of this. I was aware that by the time we’ve chained all the data and scaled all the numbers we’ve actually accumulated a lot of estimations and as a result a lot of noise that would pollute our numbers. By cancelling out our scale factors, we’re actually removing a lot of that noise.
So yes, we did work that is unnecessary to the final calculation. But we did it because it enabled us to understand the problem and have confidence that what we’ve actually come up with is robust. And that makes it worthwhile.
At Evil Works we’re all about improving the life of the data scientist, through showcasing real world projects and building the tools to just do data science better. Click the links to find out more.