Margin of Error
Margin of Error
Definition
The margin of error indicates the level of random sampling error generated by drawing a sample as opposed to conducting a census. It is + or – of the statistic and falls as the sample size rises (it’s highest at 50%)..
It can be calculated by multiplying the critical value by either the standard deviation or the standard error of the statistic. P is the percentage that the sample is seeking to measure (e.g. conversion rate of 6%).
It is important to remember that the margin of error applies to the whole sample and not sub-samples that will be definition be smaller than the total sample size. This means that for example a sample of 1,000 users might comprise roughly 500 new visitors and 500 returning visitors. Rather than having a margin of error of say +/- 3% for a sample of 1,000 users, each sub-sample of new and returning users will have a margin of error of around +/- 4%.
Often A/B tests will only sample new visitors because they are unlikely to have previously experienced the new design. This means any analysis by demographic or persona segments will be based on even smaller sample sizes. Such segments will suffer from a much larger margin of error than the overall sample. This means that unless the difference is very large it could just be random error and not statistically significant. Indeed, it is rare to find anything that can be relied on with samples of less than 100.
The margin of error only makes allowance for random sampling error. It does not account for other sources of error, such as if the measurement system (e.g. A/B testing tool) has not been configured correctly or if decisions to exclude or include user segments influence the outcome of the experiment.
Resources:
Conversion marketing – Glossary of Conversion Marketing.
Over 300 tools reviewed – Digital Marketing Toolbox.
A/B testing solutions – Which A/B testing tool should you choose?
Types of A/B tests – How to optimise your website’s performance using A/B testing.
