Funnel experiments use Bayesian statistics with a beta model to evaluate the win probabilities and credible intervals for an experiment. Read the statistics primer for an overview if you haven't already.
What the heck is a Beta model?
Imagine you run a pizza shop and want to know if customers say "yes" to adding pineapple. Some customers will say yes, others will say no.
The beta distribution is a statistical model that's great for analyzing proportions or probabilities. It helps us understand:
- The true probability of customers saying yes to adding pineapple.
- How certain we are about that probability.
For example, if:
- Only 2 out of 4 customers (50%) say yes, the Beta distribution will be wide, indicating high uncertainty.
- 150 out of 300 customers (50%) say yes, the Beta distribution will be narrow, showing we're more confident about that 50% rate.
Within PostHog, our Beta model uses a minimally informative prior of ALPHA_PRIOR = 1 and BETA_PRIOR = 1.
Win probabilities
The win probability tells you how likely it is that a given variant has the highest conversion rate compared to all other variants in the experiment. It helps you determine whether the experiment shows a statistically significant real effect vs. simply random chance.
Let's say you're testing a new signup flow and have these results:
- Control: 100 signups from 1000 visitors (10% conversion)
- Test: 150 signups from 1000 visitors (15% conversion)
To calculate the win probabilities for the experiment, our methodology will:
Model each variant's conversion rate using a Beta distribution:
- Control: Beta(100 + ALPHA_PRIOR, 900 + BETA_PRIOR)
- Test: Beta(150 + ALPHA_PRIOR, 850 + BETA_PRIOR)
Take 10,000 random samples from each distribution.
Check which variant had the higher conversion rate for each sample.
Calculate the final win probabilities:
- Control wins in 40 out of 10,000 samples = 0.4% probability
- Test wins in 9,960 out of 10,000 samples = 99.6% probability
These results tell us we can be 99.6% confident that the test variant performs better than the control.
Credible intervals
A credible interval tells you the range where the true conversion rate lies with 95% probability. Unlike traditional confidence intervals, credible intervals give you a direct probability statement about the conversion rate.
For example, if you have these results:
- Control: 100 signups from 1000 visitors (10% conversion)
- Test: 150 signups from 1000 visitors (15% conversion)
To calculate the credible intervals for the experiment, our methodology will:
Create a Beta distribution for each variant:
- Control: Beta(100 + ALPHA_PRIOR, 900 + BETA_PRIOR)
- Test: Beta(150 + ALPHA_PRIOR, 850 + BETA_PRIOR)
Find the 2.5th and 97.5% percentiles of each distribution:
- Control: [8.3%, 12%] = "You can be 95% confident the true conversion rate is between 8.3% and 12.0%"
- Test: [12.9%, 17.3%] = "You can be 95% confident the true conversion rate is between 12.9% and 17.3%"
Since these intervals don't overlap, you can be quite confident that the test variant performs better than the control. The intervals will become narrower as you collect more data, reflecting your increasing certainty about the true conversion rates.