How Should You Prioritise Your A/B Test Ideas?

Getting Maximum Return on Investment:

Before you begin A/B testing it is critical to ensure you prioritise A/B tests so that you achieve maximum benefit for your organisation. This means you must be careful to prioritise the pages and journeys most likely to have a large impact on your organisation’s goals and related key metrics. However, at the same time you also needs to take into account the difficulty and time taken to develop your test ideas.

Before you even think about A/B test ideas you should review and fix your user experience by conducting a thorough evaluation of your digital experience. Otherwise it is likely that your A/B tests will fail because new designs will be overwhelmed by existing usability and user experience problems.

Below I’ve outlined a simple process that is designed to help you prioritise A/B test ideas to get maximum impact from your efforts. If you haven’t selected an A/B testing tool yet then you can read this guide on how to choose an A/B testing tool which summarises all the main solutions.

It’s important to have evidence behind your test ideas and develop a strong hypothesis. Otherwise we are prone to confirmation bias and only change the things we perceive are worth testing.

Indeed, some experts believe this leads to a majority of winning A/B test results being an illusion. Preparation is very important with A/B testing programmes as it heavily influences whether you are successful or not.

How to prioritise:

Begin by looking at your web analytics to identify where there is most opportunity to test.

Use data to identify top entry pages and view data at a template level.
Combine all pages that have the same template to identify traffic levels for testing.

Prioritisation framework (PIE):

Potential – How much improvement can be made – how poor is the current page?
Importance – How valuable is the traffic to the pages.
Ease – How complicated will the test be to implement on the page or template.

1.Potential – Identify really bad pages:

Use your experience and best practice to identify pages that could most benefit from testing. Use the following metrics and tools to help guide you prioritise tests with most potential:

High bounce rate
Top exit pages
Funnel drop-off rates
Usability testing
On-site surveys
Customer email surveys
Eye tracking tools
On-page click tracking heatmaps
Use above as inputs into a heuristic evaluation

2.Important pages – What makes a page important?

Use your web analytics and your marketing spend to assess:

High traffic pages
Top entry pages
Pages with expensive visits are more important
Identify source of traffic and cost

3.Easy test pages – Consider technical implementation:

Tests that include the following are generally more complicated:

Site-wide elements like buttons, banners and navigation bars
Alternative site templates
Dynamic content
Pages controlled by CMS or platform
Alternative flows – multiple pages
Pages with server-side validation or interaction
Phone call tracking·
Multi-goal tracking
Experiments with multiple languages
Where multiple stakeholder opinions need to be satisfied
Consider organisational barriers – such as internal politics

But also remember that challenging tests can be most rewarding – particularly site-wide templates.

4. Prioritise pages –

Score each page from 1 (low) to 5 (high) on each of the 3 criteria and position accordingly in a matrix.

Prioritise according to their total ranking for all three criteria in a matrix like the example above. This should be circulated around teams involved in proposing test ideas so that they understand how tests are prioritised. To get buy-in from other teams it may be worthwhile asking some of the more important stakeholders to have an input into the prioritisation process.

This process is based upon the Widerfunnel approach which you can find in the excellent book: You should test that by Chris Goward. The book is a must read for anyone wanting to create an optimisation programme.

Review your prioritised list of A/B test ideas on a regular basis to check that it still aligns with business goals and what you have learned from your A/B test results.

For more on A/B testing, please read our blog; how to build a strong A/B testing plan.

Conclusion:

Prioritisation is crucial to any conversion optimisation programme because resources are finite and often scarce. Getting time with designers or developer resource can be challenging and so you don’t want to waste any resource you get with tests that won’t potentially have a good return on investment. The PIE framework is a simple and useful way of prioritising your ideas and yes there are more complicated methods available, but I like the fact that it is not difficult to implement.