Bandit Testing

Definition

Bandit testing or Multi-Armed Bandits (MAB) is a testing methodology which uses algorithms that seek to optimise for your conversion goal during rather than after an experiment is completed. It does this by aggressively switching traffic to the highest converting variant as the test progresses with the aim to maximising the conversion rate.

This is especially suited to testing headlines and other content that has a very short lifespan (e.g. limited offer promotional content) and continuous, long-term testing. The chart below illustrates graphically how the bandit algorithm allows for a smooth transition of traffic from the exploration phase of testing to exploitation of the winning variant.

Image source:

In a bandit test for 10% of the time the traffic is split equally between the two variants (the exploration phase). But for the remaining 90% of the test the algorithm sends traffic to the best performing variant (the exploitation phase). MABs allow you to weight traffic according the estimated value of different variants from the beginning of the test. This is usually based upon experience but could still be completely wrong.

Multi-armed bandit algorithms begin to switch traffic to the best performing variant before full statistical confidence is achieved. This increases the risk that a variant will be chosen that has an above average conversion rate purely by chance and the uplift will therefore not be sustainable.

So when should you use MABs?

Multi-armed bandits are best suited to the following campaigns:

• To simultaneously explore and exploit optimisation opportunities. This is potentially a great growth hacking strategy.
• Testing radically different variants where there is a high chance of large differences in conversion rates.
• Headlines and short-term campaigns.
• Automation for scale.
• Targeting.
• Combining optimisation with attribution.

Conclusion:

Multi-armed bandit algorithms are a different methodology from A/B testing and are not a replacement to traditional experimentation. A/B testing is still most suited for identifying the best performing variant with a high level of statistical confidence. However, bandit testing can be useful for short-term and continuous optimisation to obtain a high average conversion rate.

Resources:

Conversion marketing – Glossary of Conversion Marketing.

Over 300 tools reviewed – Digital Marketing Toolbox.

A/B testing software – Which A/B testing tools should you choose?

Types of A/B tests – How to optimise your website’s performance using A/B testing.

Credit:

Gambling icons created by justicon – Flaticon