Which Are The Best A/B Testing Tools?
A Guide To Selecting A/B Testing Tools:
A/B testing tools are an important component of conversion rate optimisation. It allows you to serve and measure the relative performance of different experiences by managing online controlled experiments. Increasingly A/B testing tools also enable you to personalise your customer experience and discover new customer segments based upon behaviour rather than just demographics.
What is A/B Testing?
A/B testing tools allow you to run online controlled experiments to measure the difference in performance between an existing webpage design (e.g. your home page) and a change to the page or a totally new design or user journey (see split path test).
A/B testing tools randomly select visitors for each design and use robust statistical analysis to measure the performance between the control and the variant. If there is a statistical difference between the two experiences we can say with a high degree of confidence (normally 95% or 99%) that it is down to the design and not other factors (e.g. traffic source) that often influence the success metric. Your conversion rate will fluctuate on a daily basis and so it is essential to have a control to compare against.
Most A/B testing tools offer both A/B and multivariate testing. Multivariate tests allow you to test changing content in two or more locations within the same experiment. With traditional MVT software this does require more traffic and as a result the test will take longer to conclude. However, AI algorithms can conduct massively complex multivariate tests using evolutionary processes to maximise conversion without needing as more traffic than traditional solutions.
Questions to consider:
Selecting the A/B testing tool is key to the success of your optimisation strategy. I’ve summarised below over 10 A/B testing tools that offer at least basic testing capabilities. When deciding on an A/B testing tool it’s worth considering these questions:
- What web analytics support do you have to provide insights for test ideas?
- Do you have access to usability testing and other user feedback tools to identify problems with your site?
- How easy it is to get internal IT development resource to code more complex tests?
- Do you have designers who can assist in creating new web pages and user journeys?
- What level of technical expertise does your team have to set up tests?
- How does your conversion rate compare to some of your competitors and what would be the value to your business of a 1% uplift in conversion?
- How much resource can you afford to dedicate to testing?
- What level of traffic do have on your site and therefore how long will it take to complete tests? Take an average weekly count of unique visitors over the last 12 months to estimate traffic levels.
- Who will manage the implementation of successful A/B tests?
Client-side or server-side?
The vast majority of A/B testing software is either client-side (where JavaScript manipulates the page design in the user’s browser) or server-side (where the test is built and delivered from your own server).
Client-side A/B testing tools also create potential IT Security issues as new experience are delivered by calls to the software vendor’s server. This can be a big concern when a test involves payment pages, such as checkout. This is why normally a QA of a test and the launch of an experiment are the responsibility of the client.
Server-side tools offer you greater control on what is being displayed and for this reason any security concerns from internal stakeholders are much easier to resolve. You will require support from IT developers and can even build your own testing engine if you have the in-house expertise.
WYSIWYG Visual Editor:
One of the benefits of client-side A/B testing tools is that you can build simple tests using an easy to use visual editor (What you see is what you get – WYSIWYG). This means you can build and serve simple test experiences with little, if any, support from IT developers. Even marketers with no technical training can quickly become competent with using most visual editors.
However, be careful not to rely on WYSIWYG editors because they can cause problems with more complex tests or with tests where content will need to be changed during the duration of the experiment. This can result in a build up of code which may slow down the page load speed and potentially influence the test result.
Before you buy a A/B testing tool ensure you have allocated some developer resource to the CRO programme. For apart from very simple tests you will need to get developers to build your tests. This will require some training from the supplier unless you plan to use a partner and outsource developer resource.
Managed or Self-Service?
When choosing conversion optimisation software it is necessary to consider whether you will need a managed service, self-service or hybrid solution. A managed service contract is where the software solution’s team provides consultancy, build tests and analyses the results for you. A self-service contract means you build and analyse your own tests.
A managed service tends to be expensive, but because the software provider is building and analysing the tests for you it can give your testing programme a good kick-start. A self-service contract is great if you have the technical know-how and resource or if you have little budget. However, it can be difficult to build momentum when there is no testing culture already established in the organisation. A hybrid contract may be right for those organisations that need some level of consultancy and support to get started, but plan to develop the resource and expertise in-house over time.
Dedicated Resource:
Ideally you should have dedicated resource to get maximum ROI from your A/B testing tool as it is a time consuming and expensive process. However, even if you can get a full-time testing manager you still need to consider what support they can be given with developing, building and implementing the successful ones.
Testing is a collaborative process, one person cannot do it on their own. For this reason you may want to hire a conversion rate optimisation consultant to kick-start the process of establishing a CRO programme. A good CRO consultant can advise you on your overall strategy and help evaluate the best A/B testing tool for individual needs.
Getting An Optimization Process In Place:
You will need to have a clearly defined process for website optimisation. Without this you will struggle to maintain momentum or achieve the kind of consistent and sustainable uplifts that we would expect from a systematic approach to optimisation. You should also consider the types of A/B tests that can be used to improve your conversion rate.
User Ratings:
You are now ready to consider A/B testing tools. TrustRadius, an independent site for business software users, recently undertook a review of 9 of the most popular A/B testing tools in terms of user ratings and functionality.
Perhaps surprisingly, Google Analytics, which is a free tool, was rated bottom of the pile by users. However, this is likely to have changed with the launch of Google Optimize, the free version of Google 360, its enterprise testing and personalisation product.
What You See Is What You Get (WYSIWYG) visual editors are popular for client-side A/B testing tools because they allow even the least technical of digital marketers to design and publish A/B tests on existing web pages within a matter of minutes. Most WYSIWYG editors now allow you to set up page re-directs (split-tests) so that you can test whole new pages rather than simply changing the colour of a button or text. A re-direct enables you to test an experience on a different URL against your existing page design.
The review was segmented by size of business. In the small business sector Unbounce and Visual Website Optimizer are the market leaders and received the highest user ratings.
Source: TrustRadius.com
Mobile App Testing Capability:
A/B testing tools that allow you to test native apps is one of the most important criteria for many digital organisations. Apps have become one of the most common ways users access digital experiences and so it would be dangerous and fool-hardy to assume that the user journeys they deliver are anywhere near optimal.
If your organisation has an app with a reasonable amount of traffic then you should certainly be considering integrating with a solution that allows you to test and personalise the in-app customer experience. Many providers now offer SDK solutions to test within native apps and so it is worth exploring this as an option when you decide to begin A/B testing.
As you will see from the providers summarised below there are many different kinds of A/B testing tools on the market, and depending upon your situation and budget, there should be at least one that meets your needs. If you are going through the process of deciding on an A/B testing tool then I would strongly recommend that you download the TrustRadius Buyer’s guide to A/B testing.
19 A/B Testing Tools:
1. Adobe Target & Adobe Premium:
One of the most popular enterprise A/B testing tools and personalisation platforms on the market. Now part of the Adobe Cloud which offers full integration with Adobe Analytics and content management products. Includes visual editor for micro-tests.
Target is popular with e-commerce sites for its real-time automated self-learning personalisation engine. Target lets you automate the targeting process with its machine learning algorithm that continually and automatically makes associations with various events and differences between predicted and observed responses.
It allows you to use pre-set or customised rules, including those defined by visitor location, to target content to a specific audience based on real-time data. Target has an extensive range of out-of-the box targeting rules for you to choose from.
You can also use APIs to integrate Target with data from web analytics, CRM, partner and other off-site sources to enable further customised segmentation and targeting.
2. Change Again:
A Visual editor A/B testing tool for making changes to existing pages. Allows for full integration with Google Analytics and other popular analytic tools.
3. Convert:
Uses a visual editor for A/B testing, multivariate and split URL testing. Includes a WYSIWYG editor, Google Analytics integration, revenue and conversion tracking, behaviour and segmented targeting.
4. Google Optimize
This is the free version of Optimize 360, Google’s new A/B testing and personalisation tool. This is vastly superior to its old testing tool and includes a visual editor and a reporting dashboard. See my post on how to setup and run experiments with Google Optimize as this is an excellent tool for organisations with limited budget.
Optimize 360 (currently in beta) is Google’s paid for enterprise A/B testing solution which is part of the Google Analytics 360 suite of integrated solutions. Optimize 360 includes the ability to target audiences identified in Google Analytics, testing based on data layer variables and a visual editor (WYSIWG) interface to bring it in line with other similar A/B testing tools. This fully integrated solution now puts Google in direct competition with other major solutions such as Adobe and Oracle Maxymiser.
5. Leanplum:
One of the best known A/B testing tools designed specifically for optimising mobile user experiences. Leanplumb provides mobile personalised messaging, user experience optimisation, A/B testing and analytics platform.
6. Omniconvert (Previously Marketizator):
A simple, but flexible to use visual editor tool for A/B and multivariate testing. A tool Free for up to 5,000 views per month.
7. Monetate:
An enterprise A/B, multivariate and personalisation platform with functionality for delivering totally new experiences to test. Integrates with common analytics packages and offers a visual editor for self-service basic testing implementation.
8. Oban International:
Targeting international websites Oban International have wealth of experience with optimising sites at a cultural level. They focus on cultural A/B and multivariate testing to identify factors that significantly influence conversion rates for cross-cultural sites.
9. Optimizely:
One of the most popular A/B testing tools on the market. This provides the ability to quickly A/B test changes to existing pages or redirect to a new page using their user friendly visual editor interface. The WYSIWYG editing tool allows for A/B and multivariate testing and offers real-time reporting. For more complex tests or technical support Optimizely have many authorised partners to provide advice and developer assistance.
They also offer an SDK solution for personalising and optimising mobile apps. Their predictive analytics and machine learning technology helps identify which user experiences are worth personalising by revealing high value audiences and identifying under- and over-performing segments.
10. Oracle Maxymiser:
A comprehensive enterprise A/B, multivariate testing and personalisation platform with the ability to build and test totally new pages without having to involve your developers. Maxymiser can split traffic randomly and display new experiences from its own server. Integrates with most analytics (e.g. Google Analytics & Omniture) and visual behaviour tools (e.g. Click Tale).
The self-service visual editor tool (Visual Campaign Builder) allows you to test changing elements on existing pages (e.g. headline, image or CTA changes) within a matter of minutes and now offers the ability to set up A/B redirect campaigns and multivariate tests. Also has an SDK solution for mobile app optimisation.
This was the market leader in A/B testing tools until it was taken over by Oracle. I understand a lot of experienced personnel have left and the business has moved away from the self-service model.
11. Personyze:
A SaaS platform for real-time visitor segmentation and personalisation. Offers a suite of services designed to improve visitor engagement, retention and conversion. Offers audience-orientated A/B and multivariate testing.
12. Qubit:
An enterprise analytics, segmentation, personalisation, A/B and multivariate testing platform. Includes a visual editor for changing existing page designs. Now has full mobile App testing capability.
13. Convertize:
This is is a new client-side A/B testing tool with a simple visual editor to quickly create and execute online experiments. To assist users in developing hypothesis to test Convertize have identified and developed over 200 tactics based upon neuroscience and consumer behaviour.
Simply select whether your site is an e-commerce, Saas or lead generation website and the software generates a menu of tactics to choose from. This means you don’t have to be a psychologist to develop persuasive ideas to test. Multivariate testing is not yet available, but is in the product road map for later in 2017.
Convertize offers new users a 14 day free trial period and a free plan after that expires for up to 5,000 visitors a month. However, a starter plan (for up to 50,000 visitors a month) costs just £29 per month. The Team plan for committed marketing teams (up to 300,000 visitors per month) costs £179 per month and the Agency plan (from 1,000,000 visitors per month) costs £499 per month.
14. Evolv (Previously Sentient Ascend):
The first pure artificial intelligence (AI) conversion rate optimisation software on the market to offer complex and fast multivariate testing. Evolv uses evolutionary algorithms to learn, adapt and respond to user interaction to identify the best performing combination of changes for your website. The AI technology needs less time and less traffic than traditional multivariate testing methods and by automating the testing schedule it speeds up the whole process from end-to-end.
Evolv is a client-side product which uses a visual editor to allow you to make changes. It allows you to stop or adjust tests mid-way through to add new ideas based upon initial results. However, unlike traditional A/B testing algorithms, Evolv is always testing in the background and so if your traffic mix changes and a different experience would improve conversion, Ascend will automatically respond by serving the best performing design for that audience.
Evolv claims it can increase the speed of testing from around 10 to 100 times that of traditional testing software. Indeed, for underwear brand Cosabella Ascend tested 15 different changes to the homepage header, category page, product page and shopping cart design. Using standard multivariate testing would have required 160 tests, instead of the automated process that Evolv manages for you. This improved conversions by 35% compared to the control experience.
15. SiteSpect:
An enterprise A/B, multivariate and personalisation platform for medium to large businesses. Includes a visual editor for micro-changes to existing pages.
16. Taplytics:
This is an enterprise native app A/B testing tool that also offers push notifications with campaign automation to optimise both the timing and content of your communications. A visual editor allows for easy development and implementation of simple A/B and multivariate tests. More complex tests can be built by developers. Clients such as Tinder demonstrate that this tool has the capacity for scale and they offer a 30 day free trial.
17. Unbounce:
Offers A/B testing tools for building, publishing and measuring the performance of landing pages using best practice templates provided by Unbounce. The solution also offers fully mobile responsive pages to A/B test.
18. Visual Website Optimizer:
Positions itself as the world’s easiest A/B testing tool. Provides a simple to use visual editor tool that offers A/B and multivariate testing capabilities, together with personalisation, heatmaps and user insights. Now offers an SDK solution for full mobile app optimisation.
VWO uses bayesian statistical analysis which allows you to incorporate past experience when developing your tests.
19. Webtrends Optimize:
helped found the industry in 2000, originally built for Microsoft. They have an all-inclusive model offering Full Stack (Web, Server, Mobile SDK, API), Testing (AB, ABn, Split, Full Factorial MVT, Fractional Factorial MVT), Personalisation, Social Proofing, Product Recommendations, 3rd party integrations etc. all at no extra cost.
They now also have a completely free tier, offering all features of the platform for free to small businesses and are a top-rated tool for satisfaction on G2.com.
Conclusion:
With the advent of artificial intelligence A/B testing tools are becoming increasingly powerful and responsive to the needs of digital marketers. However, it is important to take your time and consider the questions I raised above before choosing a solution.
A/B testing tools have great potential to improve your conversion rate and allow your optimisation team to become a revenue generator rather than a cost centre. But this will only happen if you follow a structured approach and create a culture of experimentation in your organisation. CRO is a collaborative and creative process that requires patience and resourcing appropriately.
is an inspiring book. It’s a highly readable and engaging book full of brilliant anecdotes and short stories. It made me reflect on life and work. For me it has some great insights that are very salient to both digital marketing and conversion optimisation. Dave recently shared this quote on Twitter which sums up advertising and marketing in a sentence.
