Dunning-Kruger Effect
Dunning-Kruger Effect
Definition
The Dunning-Kruger Effect is a cognitive bias in which unskilled people form wrong conclusions or make poor judgements. They rate their level of competence as being much higher than it is in reality. People’s own incompetence prevents them from recognising their own ineptitude.
The phenomenon was first observed by the psychologists Dr. David Dunning and Dr. Justin Kruger (published in 1999). They considered the bias to be an internal illusion amongst people of low ability or expertise and an external underestimation of the competence of people with high ability.
1. What prompted the discovery of the Dunning-Kruger Effect?
Dunning and Kruger were prompted to investigate a possible cognitive bias after hearing the story a bank robber called McArthur Wheeler. In 1995, Wheeler decided to rob a bank. He thought that by smearing his face with lemon juice it would become invisible to security cameras. He had come across a chemical property of lemon juice that means it can be used to write invisible letters which only become visible when the paper is close to a heat source. This led him to think that this would also make his face invisible to a camera.
Wheeler tested his theory by taking a selfie with a polaroid camera. Unfortunately for him, the camera produced a blank picture. Instead of checking if the camera was faulty in some way, this made him even more confident about the properties of lemon juice. As a consequence of this confidence he decided to rob two savings banks in Pittsburgh on the same day.
The police then checked the security cameras and managed to get film from it shown on the local TV news. Wheeler was arrested before the end of the same day. During his police interview he was totally shocked by how his own ignorance had let him down so badly.
The Dunning-Kruger effect came from studying the complete confidence Wheeler had that made him think he could make his face invisible to security cameras. They observed that the less competent a person is at a particular topic or task, the more likely they are to exaggerate their own self-perception of their ability. This may result poor judgement and sometimes catastrophic decisions. We don’t know what we don’t know is one saying to consider when we become confident about our competence.
2. Dunning-Kruger Effect Studies:
The psychologists conducted a number of studies, including “Why people fail to recognise their own incompetence” (2003). This research indicated that a person’s inflated self-perception of competence is due to an individual’s ignorance of the level of expertise needed to perform the task competently. This results in a double whammy. Their lack of skill prevents them from both performing the task to a high standard and the self-awareness to recognise they are performing sub-optimally.
“People base their perceptions of performance, in part, on their preconceived notions about their skills. Because these notions often do not correlate with objective performance, they can lead people to make judgments about their performance that have little to do with actual accomplishment.” – Dunning, Johnson, Ehrlinger and Kruger, 2003.
In paper published in 2005; Self-insight: Roadblocks and Detours on the Path to Knowing Thyself, Dunning points out that “if you’re incompetent, you can’t know you are incompetent”. The skills needed to judge your level of performance are identical to those needed to perform the task competently.
They also tested the cognitive bias of illusory superiority by evaluating psychology student’s self-assessments of their intellectual abilities. Students were then given their self-assessment scores and were asked to rank where they thought they would be in relation to the whole class. This demonstrated that incompetent students overestimated their class rank and competent students underestimated their class rank.
3. Website Optimisation:
For conversion rate optimisation, the Dunning-Kruger effect may explain why initial success with online experiments is often followed by a large dip in performance. The valley of despair perfectly describes how optimisers feel when they realise their initial success is often not sustainable. It is based upon quick-wins and gut instinct.
Most optimisers naturally begin with low hanging fruit, fixing the obvious deficiencies first. Once these are resolved it becomes progressively more difficult to achieve significant uplifts without first obtaining insights from users or other sources of qualitative and quantitative data. Fixing things that are broken gives optimisers false confidence in their ability. People like to take the credit for improvements even if they are simple changes.
A/B Testing
The fact that most A/B tests fail confirms that our gut instinct is often a poor indicator of success. We also tend to value our own ideas more highly than those of other s and so we often suffer from not invented here bias. This is why the most successful optimisers rely on data and insights to inform decision making. They don’t rely on a single piece of data as this can lead to confirmation bias. Instead, they build hypothesis based upon multiple data sources that give real insights and take particular notice of customer behaviour and needs.
Competent optimisers also don’t accept data at face-value. It’s important to validate your tools. Consider running an A/A test to check the implementation of your A/B testing solution. It’s also wise to use Google Analytics to analyse your test and not rely solely on your A/B testing platform. This allows you to compare the data and break results down into more relevant custom segments which can help avoid Simpson’s Paradox.
Further, there is increasing evidence that process is more important than analysis. Ensure you have a systematic framework for evaluating your customer experience. Our 8 step process for conversion rate optimisation reduces the risk that you will forget to check the obvious and it increases the chance you will identify real insights from multiple data sources. Always remember, we don’t know what we don’t know.
Conclusion:
The Dunning-Kruger effect should remind us not to get too confident about our abilities. Optimisation is a complex and never-ending process that requires a structured approach and collaboration to share knowledge and to learn from our mistakes. The fact that the majority of experiments often fail to result in any significant uplift in the conversion rate demonstrates how difficult it is to predict how changes to a website will influence visitor behaviour.
To avoid falling into the trap of the Dunning-Kruger effect encourage a culture of experimentation and evidence based optimisation. Focus on learning from experiments, whether they generate an uplift or not and avoid placing too much value on your own ideas (not invented here bias). Evaluate all tests equally. Explore whether changes (positive or negative) are responsible for the difference in the success metric.
Resources:
Dunning-Kruger Effect – CRO and the growth strategy that’s being ignored.
Simpson’s Paradox – A definition and examples of Simpson’s Paradox.
Framework for conversion rate optimisation – Conversion rate optimisation is 8 simple steps.
Over 300 tools reviewed – Digital Marketing Toolbox.
A/B testing software – Which A/B testing tools should you choose?
Types of A/B tests – How to optimise your website’s performance using A/B testing.
