Why Do People Prefer Gut Instinct to Research?

Behavioural economics tells us that people become overconfident and ignore their ignorance as they develop expertise

Behavioural Economics and Gut Instinct:

Business people pride themselves in their decision making and many businesses embed market and competitor research into this process. However, business people often prefer their gut instinct as they are prone to the same human frailties as everyone else. This can discourage the use of research and experimentation as a way of gaining feedback and learning from our mistakes.

Behavioural economics suggests that as people gain experience and knowledge in their area of expertise they have a tendency to become overconfident and complacent about their ability to understand the past and predict the future. Our brains assume that we are living in a simpler, more predictable world than is really the case.

Image of Dilbert cartoon where boss has gut instinct - behavioural economics indicates people rely on their gut instinct for many decisions

Image Source: Dilbert.com

This is one of the most useful insights of behavioural economics and yet professionally it is a difficult truth to acknowledge when we like to be seen as an expert our field. Our decisions are sometimes from the advice of an ‘expert’ as if this guarantees the quality of the decision. Behavioural economics though tells us that experts are often no better than non-experts at predicting the future.

It’s All About Us

As humans we are certainly prone to the illusion of understanding. Our minds create narrative fallacies from our continuous attempt to make sense of the world.

We notice the small number of unusual events that happen rather than the multitude of events that failed to occur. We focus on the attributes of survivors and ignore the influence of the process that shaped them.

Our memory is selective and biased by the workings of our mind. We construct vivid accounts of the past based on memories that change every time we recall them. Also, we believe they are a true reflection of past events.

We suffer from a tendency to like (or dislike) everything about a person. This helps generate a simpler and more coherent representation of the world than is really the case. We fill gaps in our knowledge about a person using guesses that fit our emotional response.

Short-term emotions are probably the most powerful force in our decision making arsenal. Many of our judgements and decisions are from feelings of liking and disliking rather than rational deliberation.

We hate uncertainty and suppress ambiguity because inconsistencies slow our thought processes and interfere with the clarity of our feelings. People like confidence and prefer decision makers that demonstrate such qualities above someone who may be equally competent but wants to think through a decision before giving an answer.

Image of Dilbert cartoon where boss wants to use his gut rather than read document

Image Source: Dilbert.com

The WYSIATI Rule

People are heavily influenced by the What You See Is All There Is (WYSIATI) rule. We naturally work with the fragmentary information that we have access to as if it were all there is to know. The paradox is that it is easier to construct a consistent story when you have little knowledge. People make fallible guesses from incomplete information by making a leap of faith about how things should work. Steven Pinker points out our only defence is that it worked sufficiently well in the world of our ancestors.

“Our comforting conviction that the world makes sense rests on a secure foundation: our almost unlimited ability to ignore our ignorance.” Daniel Kahneman, Thinking, fast and slow.

This can lead us to frame our choices too narrowly and consequentially reduces our options. Research and working collaboratively can help by widening our horizons and introducing new insights to challenge our perception of the topic.

Changing Our Beliefs

We are also very good at changing our beliefs after an un-predicted event without being aware of it. We often unconsciously adjust our view of the world and find it difficult to recall what we believed before the event. Hindsight bias leads us to evaluate the quality of decisions by the nature of the outcome rather than the process by which the decision was made.

“Asked to reconstruct their former beliefs, people retrieve their current ones instead.” Daniel Khaneman, Thinking, fast and slow.

The danger here is that people get blamed for a decision that had a negative outcome despite the unpredictability of the event. In corporate decision making this can result in people relying on bureaucratic solutions to avoid blame which leads to extreme risk aversion.

Image of Dilbert cartoon where boss explains decision making process

Image Source: Dilbert.com

It can also result in business people receiving unjustified rewards (e.g. bonuses) for being irresponsible with risk taking and just being lucky. Before the 2008 financial crisis, banks and other financial institutions paid risk takers massive remuneration packages for activities that put the whole financial system at risk. Due to the complexity of some of the assets their AAA ratings proved to be illusory.

Kahneman asserts that any comparison of how successful or not companies have been is to a large extent a comparison between how lucky or not they have been. In every story of a successful company there will have been moments when the destiny of a firm could easily have turned in an instant.

So, is the analysis of the situation more important or is it the process that is they key? Research conducted by Dan Lovallo and Oliver Sibony studied 1,048 major business decisions over 5 years. They found that “process mattered more than analysis by a factor of 6”.

An Illusion

I have no use whatsoever for projections or forecasts. They create an illusion of apparent precision. The more meticulous they are, the more concerned you should be. We never look at projections … —Warren Buffett

This does not mean that analysis is unimportant. Treat it with caution as most economic forecasts are wrong. When making decisions it is essential that we explore uncertainties and encourage discussion of opinions that may contradict the views of senior stakeholders. This helps to prevent confirmation bias taking hold and gets people to acknowledge gaps in expertise. Behavioural economics challenges many assumptions about human decision making that have been unchallenged by conventional economic thinking.

Image of Dilbert cartoon where data with human biases is given to boss

Image Source: Dilbert.com

Intelligence and a high IQ are not normally associated with stupidity. But behavioural economics suggests that our propensity to make rash, foolish or irrational decisions is often not related to our IQ. No one is immune from making daft decisions and our reliance on IQ and educational qualifications as an indicator of competence can be a recipe for disaster. When the business culture gives too much reverence to people with certain qualifications and skills this can lead to rewarding decisions based mainly upon intuition rather than evidence.

CAVE Men!

Colleagues Against Virtually Everything. People invest a lot of time and effort into their existing strategy or ideas. Dan Ariely calls this the not-invented here bias. People also have a tendency to value their own ideas significantly more than others’ ideas. This can result in obsessive focus on poor ideas and probably explains some of the less successful decisions that we come across in business.

Confirmation bias

Means that we tend to ignore information that does not align with our existing beliefs. We subconsciously seek and are drawn to evidence that confirms our view of the world. People are very good at overlooking facts that undermine their opinions and will follow the crowd that most closely supports those beliefs.

Image Source: Dilbert.com

So where does this leave the customer insight professional?

It demonstrates the need for a comprehensive strategy for promoting the use of insight and collaboration to facilitate innovation and evidence based decision making.

Stakeholder management is essential not just to obtain the buy-in and support of senior management, but also to counteract many of the myths about how research and insight is undertaken.
Use storytelling to engage people at an emotional level. Our brains become more active when we are told a story, not only the language processing part of our brain, but also other areas we would normally use to experience the events of the story in real life. Some evidence suggests that our brains can synchronize with the brains of the person telling a story.
Spend time getting to understand your audience and their preconceptions. “What You See Is All There Is” tends to be strongly influenced by survey research when it comes to insight. Voice Of the Customer surveys are often given as an example of what research involves and yet these can be fundamentally flawed by relying on asking people direct questions.

The Importance of Knowing

Never underestimate the importance of how choices are presented and ensure you are fully prepared so that you avoid uncertainty about your recommended approach. Behavioural economics demonstrates that there is no such thing as a neutral choice architecture.
Immerse yourself in the customer facing side of your business by meeting and observing how your organisation interacts with your customers. Don’t rely on third parties or management to identify the real challenges customer facing staff have to deal with.
Identify information gaps to highlight the need for research and insight. Our illusion of understanding sometimes needs reminding how little we really know about the world.
Challenge default methods of conducting research. Examine the potential for alternative approaches to insight, including experiments, observation and collaborative methods.
Image Source: Dilbert.com

Encourage, Test and Understand

Encourage a culture of experimentation. For instance use A/B and multivariate testing on your website to understand what content most engages and motivates your existing and potential customers.
To counter hindsight bias always ask key stakeholders before you commission a project what they expect the outcome/findings to be. You can then use these as hypothesis to prove or disprove their views.
Encourage all areas of the business to share insights and engage with the research process. It shouldn’t just be marketing and customer services that buy-in to customer insight. This helps avoid group think by bringing diversity into the decision making process.

Behavioural economics provides a valuable framework for developing strategies for behavioural change. It can also helps us to better understand some of the barriers in organisations to developing a culture of research and experimentation. This does have to come from the top of the organisation, but if you can get senior management on board then behavioural economics can be used throughout your organisation to assist in decision making and behavioural change.

Why are Voice Of the Customer surveys fundamentally flawed? A behavioural economics view of VOC surveys.

5 ways to get more valuable insights from your Voice Of the Customer programme. Strategies to ensure your VOC programme has impact and delivers genuine insights.

What does mountaineering tell us about human motivations? A look at how what people say is not consistent with what they do and what really motivates us humans.

Are most of our decisions the result of other’s behaviour and opinions? How social learning and copying drives much of our behaviour.

What do business people really think about market research? The misconceptions about market research from my personal experience on the client-side.

Is financial decision making more rational?A review of how rational people are when making purchasing decisions and how we use rules-of-thumb to reduce cognitive load assist decision making.