Bootstrapping is a useful technique to infer statistical features (think mean, decile, confidence intervals) of a population based on a sample collected. Implementing it at scale can be hard and in use cases with streaming data, very complicated. When trying to learn how to bootstrap at scale, I came across a Google blog (almost a decade old) that introduces Poisson bootstrapping. Since then I’ve discovered an even older paper, Hanley and MacGibbon (2006), that outlines a version of this technique. This post is an attempt to make sure I’ve understood the logic well enough to explain it to someone else. We’ll start with a description of classical bootstrapping first to motivate why Poisson bootstrapping is neat.
Classical bootstrapping
Suppose we wanted to calculate the mean age of students in a school. We could repeatedly draw samples of 100 students, calculate the means and store them. Then we could take a final mean of those sample means. This final mean is an estimate of the population mean.
In practice, it’s often not possible to draw repeated samples from a population. This is where bootstrapping comes in. Bootstrapping sort of mimics this process. Instead of drawing samples from the population we draw samples (with replacement) from the one sample we collected. The psuedo-samples are referred to as resamples.
Turns out this is extremely effective. It’s more computationally expensive than a closed form solution but does not make strong assumptions about the distribution of the population. Additionally, it’s cheaper than repeating sample collections. In practice, it’s used a lot in industry because in many cases either closed form solutions don’t exist or are hard to get right—for instance, when inferring deciles of a population.
Why does bootstrapping work?
Bootstrapping feels wrong. Or at least the first time I learned about it, it didn’t feel right. Why should one sample contain so much information?
Sampling with replacement from the original sample you drew is just a way for you to mimic drawing from the population. The original sample you drew, on average, looks like your population. So when you resample from it, you’re essentially drawing samples from the same probability distribution.
What if you just happen to draw a weird sample? That’s possible and that’s why we resample. Resampling helps us learn about the distribution of the sample itself. What if you original sample is too small? As the number of observations in the sample grow, bootstrapping estimates converge to population values. However, there are no guarantees over finite samples.
There are problems but given the constraints we operate in, it is the best possible information we have on the population. We don’t have to assume the population has a specific shape. Given that computation is fairly cheap, bootstrapping becomes a very powerful tool to use.
Demonstrating classical bootstrapping
We’ll explain bootstrapping with two examples. The first is a tiny dataset where the idea is that you can do the math in your head. The second is a larger dataset for which I’ll write down code.
Example 1: Determining the mean age of students in a school
We’re tasked to determine the mean age of students in a school. We sample 5 students randomly. The idea is to use those 5 students to infer the average age of all the students in the school. This is silly (and statistically improper) but bear with me.
ages = [12, 8, 10, 15, 9]
We now sample from this list with replacement.
sample_1 = [ 8, 10, 8, 15, 12]
sample_2 = [10, 10, 12, 8, 15]
sample_3 = [10, 12, 9, 9, 9]
....
do this a 1000 times
....
sample_1000 = [ 9, 12, 12, 15, 8]
For each resample, calculate the mean.
mean_sample_1 = 10.6
mean_sample_2 = 11
mean_sample_3 = 9.8
...
mean_sample_1000 = 11.2
Take a mean of those means.
mean_over_samples=mean(mean_sample_1, mean_sample_2, .. , mean_sample_1000)
This mean then becomes your estimate of the population mean. You can do the same thing over any statistical property: Confidence intervals, deviations etc.
Example 2: Determining the 95th percentile for ‘time to process payment’
Customers on a food delivery app make payments on the app. After a payment is successful, an order is placed with the restaurant. The time to process payment calculated as the time between when a customer presses the ‘Make Payment’ button and when feedback is delivered (payment successful, payment failed) is a critical metric that reflects platform reliability. Millions of customers make payments on the app every day.
We’re tasked to estimate the 95% percentile of the distribution to be able to rapidly detect issues.
We illustrate classical bootstrapping in the following way:
- We assume that there’s some population that has a million observations. In the real world, we never observe this data.
- We randomly sample 1/10th of this population. So we take 10,000 observations. In reality this is the only data we observe.
- We then apply the same procedure we discussed above. We resample observations from our observed data with replacement. We do this many, many times.
- Each time when we resample, we calculate the 95th percentile of that distribution.
- Finally we take the mean of the 95th percentile values and find confidence intervals around it.
We get the graph below. Magically, we find that the confidence intervals we just generated contain the true 95th percentile (from our population).
We can see the same data at the level of the bootstrapped statistic.
The code to generate the above is below, give it a try yourself!