Definition
Bayesian A/B testing uses Bayesian statistical theory to determine the probability that a hypothesis (e.g. B is different from A) is true or false. Bayesian theory assigns a probability to a hypothesis. Whilst in frequentist statistics a hypothesis is tested without being given a probability.
Furthermore, with Bayesian A/B testing past knowledge of the phenomena is fed into a prior probability which is then updated to a posterior probability using the data from the experiment at hand. The prior and the posterior allows us to come a conclusion about the result of the A/B test.
Bayesian statistics often generate results for parametric models that are close to the results produced by frequentist methods.
Advantages of the Bayesian Approach:
- It provides a logical way of combining prior knowledge of a similar situation within a robust statistical framework. This allows you to include past information about a parameter and calculate a prior distribution for future analysis. When a new A/B test is conducted for the same parameter, the previous posterior distribution can be used as a prior.
- It generates inferences that are conditional of the data and are precise, without reliance on asymptotic approximation. Small sample inference works in the same way as if we had a large sample.
- The approach obeys the likelihood principle. If two separate samples generate proportional likelihood functions for ϴ, then all inferences about ϴ should be identical for both designs.
- It generates interpretable statistics on the probability of a hypothesis.
The Main Disadvantages:
- There is no agreed method for choosing a prior and it requires skill to translate subjective prior beliefs into a mathematically calculated prior. If not done correctly it could lead to misleading results.
- The posterior distribution can be heavily influenced by the selection of the prior. The selection of the prior is a subjective process this can easily be challenged by other stakeholders.
- Bayesian analysis can require a high level of computational resource, particularly in models with a large number of parameters. Chris Stucchio provides a useful example of using Bayesian statistics for conversion in his post Analysing conversion rates with Bayes rules.
Conclusion:
Finally, both VWO and Google Experiments use the Bayesian approach for A/B testing. Bayesian analysis can often generate similar results to the frequentist approach for A/B testing. We could consider the statistical method to be a secondary issue not worth a great deal of attention.
The main advantage of Bayesian statistics is the ability to include historical data and to select a prior distribution. The main danger though is the subjective nature of the selection process for the prior could be distraction from the test conclusion. However, I firmly believe that the key issue is whether the A/B testing tool you choose is appropriate and meets the needs of your organisation. The nature of the statistical engine is less of a concern.
Resources:
Conversion marketing – Glossary of Conversion Marketing.
Over 300 tools reviewed – Digital Marketing Toolbox.
A/B testing software – Which A/B testing tools should you choose?
Types of A/B tests – How to optimise your website’s performance using A/B testing.
