AB Testing Tools Customer Reviews – Movers and Shakers

Analysis of A/B Testing Tools Customer Reviews & Market Presence

As a greater proportion of sales move online. The use of A/B testing tools is likely to increase as more organisations see the value in conversion rate optimisation (CRO). I’m a self-taught developer, and worked my way to running the Product team at Webtrends Optimize and sitting on the board of directors. I’ve worked at agencies and run my own too, in my ~8 years in CRO. As part of my role, I need to understand the landscape of AB Testing tools better than the average user.

There are a lot of A/B testing tools. Let’s be honest – keeping track of what each of them have been up to can feel like a job in itself. Thankfully for you, it’s my job – here’s what I’ve seen. This post looks at their trajectory on G2.com, and possible reasons for it.

For The Uninitiated

G2.com is a review site that holds heaps of reviews for countless tech vendors. They have an AB Testing Tools section, with an easy-to-interpret grid. It measures market presence and satisfaction scores.

There’s a lot to be interpreted from G2 scores. While one review in isolation means very little, dozens are certainly meaningful. Often, user feedback is the fastest way to take a pulse of the situation with a company. Whether positive or negative, you’ll see where they’re heading faster than feelings that reach the wider public.

1. All A/B Testing Tools:

The All segment is the general, aggregated view of how A/B testing tool vendors are doing. Looking at the video above, there are a few key things to point out, when monitoring the key vendors.

Optimizely:

Once considered the leader in the AB Testing space. Optimizely has dropped both in Market Presence and Satisfaction and left the Leaders section of G2 – now being considered a Contender.

There are numerous reasons for this. Most notably the changes in their practise since their Series D fundraising round. Part of this deal was to bring the global services consultancy Accenture on as a partner. Something which has had a knock-on effect for even long-term agency partners.

Alongside was a move in billing units from Unique Visitors to Impressions. This has seen several customers have drastically increased pricing.

Finally, the abundance of alternative tools increasing in popularity (see below) has likely seen Optimizely plateau for the time being. Perhaps until they make a sizeable shift in the practices mentioned above.

Adobe Target:

This has seen a margin decrease in market presence and satisfaction. The same direction as Optimizely, but a lesser magnitude.

Satisfaction has always been a challenge for Target. It’s one of the worst-rated tools on the grid at (now) under 10%. However, the declining market presence may well be an interesting thing for the landscape as a whole.

Target was billed as a key component of the Adobe Marketing Cloud. A decline in presence may well mean users are moving to specialist/dedicated toolsets. For the industry, we view this as hugely beneficial – no one company (even Google) gets to do everything perfectly, and dedicated tooling is preferred when integrations are seamless.

VWO:

VWO seems to have had a good few months – seeing an increase in both satisfaction and market presence. It should be noted that this is a difficult thing to achieve – we more commonly see growing tools suffer in satisfaction as resources are focused on selling more than service.

Their ability to have a very linear pricing model overseas is likely one of the reasons for growth and satisfaction – whilst it’s not considered to be the most competent tool, “cheap and cheerful” is certainly also a great thing.

Webtrends Optimize:

WTO has seen one the more positive trajectories of growth on G2, with a sizeable increase in market presence and satisfaction, and from a tool which is still run as privately-owned. Rated as the top tool for satisfaction (after GA, but no that doesn’t count!) is an important feat regardless of market presence.

Being half a square from being a Leader will likely see the recognition in 1-2 months time at this rate of progression.

There are a few important reasons for this movement during the Covid crisis:

• Free support provided to several organisations large and small.
• Free platform provided to anyone who needed it (for 60 days).
• A fully-free tier released for smaller organisations, with no restrictions on feature-set.

With a combination of their toolset (Full Stack, Social Proofing, Recommendations) etc. and the current price-point, continued growth seems likely. As mentioned above, maintaining satisfaction during growth is a challenge, and one to keep an eye on.

AB Tasty:

Marginal increase in satisfaction.

Not huge movement for AB Tasty so far, but it seems change in the near future is likely:

By running their Series C round, they seem to be focusing on the same market Optimizely was stepping away from – partnerships. Often viewed as the biggest reason Optimizely saw the success they did, this focus on partnerships is likely to see their business continue to grow. That is, of course, if managed well – we’ve seen other companies in CRO see large cash injections and start to suffer under the needs to provide a return.

Honourable (or not!) mentions:

• Dynamic Yield: A sizeable down-turn in both satisfaction and presence. 

• Convert: A decline in satisfaction since October 2019 (start of comparison).

• Sitespect: A decline in satisfaction since October 2019 (start of comparison). Currently fighting with Adobe Target for the worst-satisfaction on the grid of major tools.

2. Medium businesses:

This segment targets companies of 50-1000 people (yeah, it’s a big range, but I didn’t make the rules!).

Only three platforms made notable changes in direction – most have been marginal:

Webtrends Optimize:

• The best-rated tool (including GA!) for Satisfaction, with a sizeable increase in both Satisfaction and Presence over the last few months.

Dynamic Yield:

• Mirroring what we see in ALL (and probably causing it), a downturn for Satisfaction and Presence.

Google Optimize:

• An increase in market presence, but still the worst-rated tool on the grid for Satisfaction. This is a positive thing for CRO though – being largely used as Freeware, Google Optimize will encourage more companies to get involved with CRO and thus become acquainted with A/B testing tools. For the industry, more people taking an interest is a brilliant direction – experience can then shine through as people struggle with inferior tools and less experienced users.

3. Small and Enterprise videos:

These consider Small and Enterprise – demographics which do not apply to most of the companies reading this post! If you’re interested, I’ll let you watch and analyse them yourself.

Changes in reviews among small organisation are shown here:

Changes in customer review of enterprise organisations:

4. A time for change in AB Testing Tools:

We appear to have entered what I’d call a 3rd wave in AB Testing Tools. The first was about 10-15 years ago, with the tools that established the market (Webtrends Optimize, Maxymiser, SiteSpect etc.). A revolution then occurred about 6-7 years ago where tools such as VWO and Optimizely became well-known for WYSIWYG capabilities, and opened the door to DIY (self-serve) testing.

Along with that came an agency boom in usage, with that knowledge no longer being siloed in organisations which often offered services too. With so many capable users now, the current landscape seems to penalise price and/or poor service, seeing the decline of certain tools but the growth of tools like Google Optimize and Webtrends Optimize (both have fully free-tiers) and VWO (quite affordable).

As for a fourth-wave

We seem to be at the start of an AI revolution, with more tools considering it’s adoption into their processes. There may well be a time where an outstanding AI-driven tool once again begins to sweep up the market who are looking towards being cutting-edge with their technology choices.