Definition
Regression to the mean describes the phenomenon that when the first measurement of a variable is extreme (i.e. significantly different from the average) it is usually followed by a measurement much closer to the mean. Alternatively if the first observation is close to the mean. The second measurement it likely to be further away from the mean. This occurs whenever there is a less than perfect correlation between two variables (e.g. high value game players and size of deposit).
Extreme groups, such as an exceptionally high or low performing sales region or country, will regress to the mean over time. To measure performance of extreme groups it is necessary to compare against a control group. This would improve regression alone. Only by having a control can you tell whether a new experience (e.g. a new landing page) is influencing performance. Or whether it is due to regression to the mean.
A/B test showing signs of regression to the mean
It is not uncommon for A/B tests to start with an extreme measurement and then gradually disappear as the variant regresses to the mean. A common mistake with low performing sales regions or countries is to introduce a new incentive or landing page with the hope of improving conversion. However, this is often implemented without a control (i.e. an A/B test). How can we be confident that any rise in conversion is due to the intervention or just regression to the mean?
Conclusion:
This is an important phenomenon that can easily be confused with correlation in experiments and conversion reporting. It should make us wary of stopping experiments too early. Many tests will start with an abnormally high or low score but then regress to the mean.
It also shows the value running A/B tests to compare the performance of extreme groups. Don’t fall into the trap of assuming that an improvement in conversion in a poor performing region or country is down to an intervention. It may just be due to regression to the mean.
If we find that a winning variant from an A/B test does not deliver the uplift when it is implemented. This could suggest that your A/B testing software is not selecting visitors at random. If this is the case then your test result could actually be down to regression to the mean rather than a sustainable uplift.
Resources:
Conversion marketing – Glossary of Conversion Marketing.
Over 300 tools reviewed – Digital Marketing Toolbox.
A/B testing software – Which A/B testing tools should you choose?
Types of A/B tests – How to optimise your website’s performance using A/B testing.
