Stop Using focus Groups To Evaluate Your Home Page

Why do people use focus groups to evaluate new designs?

“Focus groups are not usability tests” says the usability expert Steve Krug in his book Don’t Make Me Think. But focus groups are also a very flawed method of research for evaluating any concept or design.

That’s why I was surprised when I attended a conference on conversion optimisation that a speaker explained how they had used focus groups to evaluate alternative designs for their new home page. One of the designs received much more favourable feedback than any of the other variants. This gave the client confidence that this design would win over their existing home page. However, when they used A/B testing to measure the performance of each variant the preferred design completely bombed. They were relieved that they had decided to test the design before changing their home page.

The outcome of the test did not surprise me. How we articulate what we think about a user interface design is very different from how we are likely to behave in reality. When we browse the internet we are normally seeking to complete a task that takes us closer to achieving a personal goal. This is not a group experience.

Focus Groups

Further, neuroscience indicates that most of our decisions are taken intuitively, often with little conscious awareness. We use behavioural short-cuts called heuristics to minimise cognitive load and our emotions often over-ride our rational thought processes.

How often do you browse the internet with a group of people you have never met whilst being observed through a one way mirror? Website navigation and task completion is more often than not undertaken by an individual without any group interaction. This may be changing to an extent with mobile devices, but in most cases these are still people we know.

Context is critical in how we respond to any product or service. And yet focus groups are often conducted in a completely alien and artificial environment. This does not match the reality of website browsing. Viewing facilities exasperate this problem as they create a laboratory atmosphere.

Problematic

Some times it is suggested that focus groups are used before designs are drawn up to understand user’s wants, needs and likes. I would also caution against this as focus groups are problematic on a number of counts:

A single person can dominate the discussion. Respondents will sometimes change their opinion to conform to the majority or because they mistake confidence for knowledge.
People over-analyse and rationalise the topic when we are looking for their emotional and intuitive response to achieving a goal. People will say one thing but behave in a totally different way in reality. This is because people are extremely poor at predicting future behaviour as we are prone to the influence of short-term emotions and cognitive bias.
Presentation is critical. Despite explaining to people that a design is a mock-up or not fully complete. You can guarantee that the stage of development of a design will influence how people respond to it.
People dislike uncertainty and when it does exist people look to the actions of others to guide them. ‘Group think’ may also set in if a group is too homogeneous. We seek a consensus of opinion rather than having a show of hands. This is particularly worrying as participants to focus groups are normally recruited on the basis of being aligned to agreed demographic characteristics.

Individual Usability Interviews

As a result I would certainly avoid using focus groups for evaluating new website pages or journeys. All research methods have their limitations. Focus groups are especially problematic and inappropriate in the context of website design.

Individual usability interviews can provide much more meaningful insights about how a website or web page works or not. Learning to listen and observe rather than asking direct questions is likely to provide more useful feedback on how to improve your site. There are also many online voice of customer tools that can provide relevant sources of insight.

Conclusion:

There is no valid reason to use focus groups to evaluate online designs. They offer a flawed approach to research and don’t provide a reliable method of predicting user behaviour. So, let’s stop using focus groups. Start looking at more appropriate ways of assessing new designs.

For wireframes and new designs you can use online design feedback tools that have quick turnaround times to avoid any delay in your design sprint. For online testing there are also some great remote usability testing solutions to choose from. They can again turn the research around very quickly.

For understanding what in particular visitors are focused on when they look at your home page there are eye tracking solutions available which measure gaze and attention metrics for you new design. To understand psychological responses to your design you can consider various implicit methods of research that don’t rely on self-reporting.

Finally, if you have a somewhat larger project, such as developing a new personalised home page, you could consider a Google 5 day design sprint. This is a 5 day workshop with key stakeholders and experts to rapidly develop and test a new concept with your target audience.