Iterative Testing
Iterative Testing
Definition
Iterative testing is the process of basing tests on insights gleaned from previous experiments to make changes gradually and that are evidence based. Iterative testing is an important component of conversion rate optimisation. It allows a site or app to evolve gradually over time with small steps rather than a large redesign. This has the advantage that users are often able to continue to navigate and browse without evening noticing when changes are being evaluated.
Iterative testing has a number of advantages:
- It minimises risk to your conversion rate because you only make relatively small changes in each iteration.
- It allows for agile working because the changes are normally easy and quick to implement.
- It doesn’t disrupt returning visitors because they are often excluded from such experiments. The changes are often too small for a noticeable difference in the site’s navigation.
- It is easier to debug/QA a iterative test because the changes are usually small and limited.
- It is less costly that some other forms of testing because the changes often require little, if any, developer resource.
- It is easier to identify which changes have a positive or negative impact on your North Star metric. Iterative tests normally only involves one change to an experience.
- It can use split tests which can be created in a matter of minutes.
For example, after testing a new landing page against the home page, we decided to begin an iterative testing plan to understand the impact of individual changes on the conversion rate. The test altered the social proof below the call-to-action button. Just by changing the number from all players a month to the number of unique active players we measured a 5% drop in registration conversion. This confirmed the power of social proof and that we should test other forms of social validation.
1. Follow a systematic approach:
Iterative testing can provide more clarity about what changes improve or reduce the conversion rate. Making lots of random changes to the user experience can make it more difficult to understand what is driving the conversion rate. It also shows a lack of strategic direction. Online experiments are most successful when there is a systematic process in place to create and develop a roadmap of tests based upon both qualitative and quantitative data.
2. Create a plan:
Creating a specific plan based upon previous tests enhances the learning process that A/B testing and multivariate testing provides to digital marketers. Each experiment follows a systematic evaluation of the previous test. Marketers are less likely to fall into one of the many logical errors and cognitive biases that we are all prone to.
The Dunning-Kruger effect is where people misjudge the amount of expertise required to be competent at a task. This can lead to overconfidence and poor decision making. Working collaboratively in cross-functional teams can help avoid this problem and seeking peer reviews of your hypothesis.
3. Types of experiments:
A test and learn process is a much more scientific approach to website improvement than completely redesigning your website from scratch. Iterative testing is one of six kinds of tests that you should be considering using as part of a conversion rate optimisation programme. These include real estate tests where we move elements around to find their optimal position and inclusion and exclusion tests. Apart from innovation tests, these other kinds of tests may be included in iterative testing. Provided you have evidence to suggest they are appropriate to improve your key metric.
4. Should you redesign your site or app?
Redesigning your site is sometimes required when you have strong evidence to suggest that your site architecture and content is a barrier to further improving your conversion rate. However, iterative testing and other types of experiments should be attempted first. For example, Facebook did at one time regularly go down the route of a site redesign. What they found was that it resulted in a huge increase in complaints from users as they struggled to navigate and complete their common tasks.
People generally dislike change and redesigns are problematic. Users have to quickly re-learn how to navigate and browse the new site. Site redesigns will normally lead to a temporary drop in the conversion rate. Some visitors may never return again if they find the new site too difficult to use.
For this reason be careful not to go down the route of a site redesign until you have exhausted all other options. Iterative testing and the other types of tests can achieve a great deal on most websites and apps. The most important aspect of all online experimentation is to ensure you follow a systematic process to evaluate the existing user experience and use data to develop strong hypothesis for testing.
Conclusion:
Iterative testing is a powerful means of improving your website or app by gradually evolving over time. This avoids disrupting and confusing users who dislike change and may abandon your site if you make lots of changes all at once. A site wide redesign should only be employed if all other forms of testing have been exhausted and you have evidence to suggest the current site has little potential for improvement.
Resources:
Optimize: How to set up and run tests in Google Optimize.
Split test: How to create a split test in Google Optimize.
A/B test plan: How to build a strong A/B testing plan.
Framework for optimisation. Conversion Rate Optimisation in 8 simple steps.
A/B testing software – Which A/B testing tools should you choose?
Types of A/B tests – How to optimise your website’s performance using A/B testing.
Mistakes in A/B tests – Are most A/B test results a lie?
