Confidence Level

Definition

A confidence level or confidence interval, is a measure of uncertainty about whether a data set is within the range that we would expect from the default population. It refers to the percentage of all possible samples that the parameter can be expected to fall within the bounds of the interval.

When we get a test score it is important to understand what proportion of the standard normal distribution the score represents (i.e. how likely that it is part of the same population). In most instances we use a 95% or 99% standard error of the mean which we derive from a z score table. The z score tells us what proportion of the population falls between the mean and the standard deviation .

1. Sampling Distribution and Standard Errors:

When we have a sufficiently large sample of observations from a defined population (e.g. at least over 100) the resulting averages or percentages, if plotted on a graph, should form the shape of a bell-shaped curve known as the standard normal distribution.

The standard normal distribution is useful when conducting experiments because it tells us that a known proportion of values lie beyond certain multiples of the standard error. For example, 95 per cent of values are within the range of population average + or – 1.96 standard errors. Only 5 percent of values fall outside of this range. Further, 99 percent of values are within the range of + or – 2.58 standard errors. This characteristic of the standard normal distribution is valuable when we want to make inferences about the population average.

2. Calculating the Standard Error:

If we take a sample n (greater than 100) and obtain a numerical measure from each visitor to a website (e.g. basket value) we can calculate the standard deviation of the sample values. The standard error of the mean of the sample is calculated using this formula:

Standard error (x) = s / √ n

So, the standard error is calculated by dividing the sample standard deviation by the square root of the sample size (n).

Example for Mean Score:

A sample of n = 200 visitors to a website are tracked to provide the following data on average basket value.

Mean = x = £5.60

Standard deviation = £2.1

Standard error (x) = 2.1 / √ 200 = £0.148

For binomial data, such as conversion rate, the mean of the sample is the proportion (percentage) of the sample who convert. The standard error of this percentage is derived from this formula:

Standard error (p%) = √ p% (100 – p%) / n

Example for Percentage:

A sample of n = 400 sessions is taken and generates a conversion rate of p = 40%.

n = 400, p% = 40%.

100 – p% = 60%.

Standard error (p%) = √¯ 40 × 60 / 400 = √‾ 6 = 2.45%.

3. Calculating the Confidence Interval:

We can use the the standard errors and the normal distribution theory to calculate confidence intervals or limits. We know that:

• 95 percent of the samples will have a mean falling in the range of x ± 1.96 standard errors of the mean.
• 99% of the samples will have means falling in the range of x ± 2.58 standard errors of the mean.

Given this is the case it is logical to say that the true mean (x) of the population will have a 95 percent chance of falling within this range.

Confidence Interval Example for Mean Score:

To calculate the confidence level around the mean (x) we use the following formula:

x ± 1.96 (s / √ n)

Where 1.96 is the standard errors of the mean (for 95% or 2.58 for 99%), s is the standard deviation and n is the sample size taken (e.g. the number of sessions in the experiment).

For example, the average order value is £11.2, with a standard deviation of 4.2 and a sample size of 200 sessions.

£11.2 ± 1.96 (4.2/ √ 200)

± 1.96 (0.297)

£11.2 ± 0.58

Therefore, the 95% confidence level or interval indicates that the average order value will range from £10.62 to £11.78. This means there is a 5% (one in twenty) chance that this is wrong.

Confidence Interval Example for Percentage:

n = 400 sessions

Conversion rate – p% = 40%

Standard error (p%) = 2.45%

95% confidence interval for p% = 40% is given by:

40% ± 1.96 (2.45%)

40% ± 4.8%

i.e. 35.2% to 44.8%

See how wide the confidence level or interval is for such a small sample.

This demonstrates the vital role of the sample size in the precision of an experiment or survey. It is an integral part of the standard error formula and so determines the width of the confidence interval. It is of course the square root of the sample size that sets the width of the confidence interval. This means that if we wish to double the precision of the experiment, we need to increase the sample size by four times. For an experiment this means allowing for a longer duration to build up a much larger sample size.

Resources:

Z score table – The Z Score Definition and how to use it. 

Conversion marketing – Glossary of Conversion Marketing.

Over 300 tools reviewed – Digital Marketing Toolbox.

A/B testing tools – Which A/B testing tool should you choose?

Credit:

Confidence icons created by Smashicons – Flaticon