Base Rate Fallacy
Base Rate Fallacy
Definition
Base rate fallacy (or base rate neglect) is the tendency to mistakenly estimate the likelihood of an event without taking account of all the relevant data (e.g. generic information on probabilities). People often focus on specific information that only relates to a certain case. As a result sometimes jump to inappropriate conclusions.
In his book Thinking, fast and slow, Daniel Kahneman gives the following example to show how people often ignore the base rate. A taxi cab was involved in a hit-and-run accident at night. You are given the following data.
- 85% of the cabs in the city are Green and 15% are Blue.
- A witness identified the cab as Blue. The court tested the reliability of the witness under the circumstances that existed on the night of the accident and concluded that the witness correctly identified each one of the two colours 80% of the time and failed 20% of the time.
What is the probability of the cab involved in the accident being Blue rather than Green?
To answer the question people have the base rate (i.e. 15% of cabs are blue) and the reliability of the witness (80% accurate). If the two cab operators were identical in size the base rate would be neutral. You would only have to consider the reliability of the witness and would conclude the probability is 80%.
However, despite the obvious difference in the size of the cab companies most people give 80% as their answer to this question. Even those who attempted to make allowance for the base rate estimated a probability of over 50%. The correct answer using Bayesian inference is 41%.
If you would like to work out the answer use this Bayesian calculator. Only enter data in the first column (0.15 in the first row, 0.8 in the second row and 0.2 in the third row). Then hit “Compute” and you will see the answer adjacent to the compute button.
Why does it matter for optimisation?
It is easy to get excited about a customer segment that has a high average order value or a cool new feature to improve the customer experience. However, always ensure you check what proportion the segment makes up of your target audience. Or how much features are used by customers before using up valuable resources. If you commit effort to a small segment you are likely to end up reducing the time you can spend on a larger, more important audience.
Base rate neglect is also easy to fall foul of if you do not segment you’re A/B tests. By segmenting your tests you can identify the conversion rate of your target population and understand what proportion of your visitors are in your target group. Optimising for an aggregate conversion rate can lead to poor testing decisions as your conversion rate will vary according to the customer segment. This can result in an increase in unqualified leads, a higher conversion rate for non-target visitors and a lower conversion rate for your target visitors.
Implications for marketing:
People are poor at understanding statistics and percentages so avoid using such data to explain features or performance of your product or service. If you want to show savings or cost per item display it in monetary terms so that people don’t have to estimate it themselves. Allow people to understand how your proposition solves their problem by giving specific examples, show case studies and testimonials.
Resources:
Conversion marketing – Glossary of Conversion Marketing.
Over 300 tools reviewed – Digital Marketing Toolbox.
A/B testing software – Which A/B testing tools should you choose?
Types of A/B tests – How to optimise your website’s performance using A/B testing.
