The 5 Step Process for A/B Testing Your Chatbots

According to Zendesk and Dimensional Research¹, a good first impression inspires 24% of users to become loyal customers for two years or more; whereas a bad first impression causes 39% of users to avoid companies for two years or more. The way that your chatbot interacts with people can make the difference between a loyal consumer and a one-time visitor.

This week we’re talking everything experimentation and conversion rate optimisation following on from the newest episode of our podcast Everything eCommerce. After Tuesday’s article, which detailed the importance of A/B testing your live chat strategy, we also wanted to talk about how you can apply this to chatbots and ensure they are working the best they can. The key metrics here will remain the same, but chatbots need to be measured in a different way.

1. Choose the right platform to conduct the A/B testing

There is a range of software out there which you can use to test your chatbots and it’s important to make sure you choose the one which will work best for your needs.

Some top picks are as follow: Botanalytics allows you to analyse conversations and build better conversational flows, Botlytics lets you keep count of all the messages your bot sends and receives, Chatbase in an analytics platform made by Google and Dashbot an analytical tool for conversational interfaces, like Facebook, Alexa, Google Home, Slack, and Kik.

Choosing your software wisely is super important, as ultimately the results of the A/B test lie in the results, the way you gather the data and how you utilise it.

2.Analyze the chatbot funnel and decide which factors you want to analyse – check out out Tuesday’s article for more ideas on this.

The easiest way to do this is by looking at both visual and conversational factors. Visual factors can be things such as the branding and design of the chat window, colours and fonts, the use of emojis and where the chatbox is placed on the site.

different algorithms, different structures

Conversational factors can be the formality and tone of the chat, proactives and greetings, utilising user profiles to tailor the conversations to your customers, and customisation of messages.

3. Decide on the test method to use, control for interactions between the factors to be tested, then gather as much data as possible.

Paying close attention to the metrics you are measuring is key for understanding the results of your test. Key factors to measure are:

• Activation rate – how long it takes for a site visitor to respond to your chatbot with a relevant question or answer
• Confusion triggers – the number of times your chatbot misinterprets a message or fails to understand a message
• Fall back rate (FBR) – how often your chatbot fails a task
• Goal completion rate (GCR) – how well your chatbot engages with users
• Retention rate – the number of users who return to a chatbot within a specific time period (varies, depending on the analytics tool you’re using)
• Self-service rate – how well a chatbot can resolve a request without human intervention
• User satisfaction – surveys for a good understanding of how pleased a user is with your chatbot’s performance

4. Compare and analyse the data, then test for additional factors

Your next A/B test should build on the insight gained from the first experiment. For example, if customers reacted better to the bot displaying multiple products instead of just one, start testing how many you display and the categories you display them under. Do you want to show the lowest price, the best-reviewed, the most popular?

Keep the process going until you are happy you have found the best result. A/B testing should be an ongoing process and should be something you do every so often to ensure your chat is still optimised.

5. Improve your chatbot and watch your conversions rise!

Using the data you gathered, implement changes in your chatbot strategy and see the payoff. This could be an increase in leads or sales or simply happier customers who have had their support query resolved by a chatbot.

Final Thoughts

If all of this sounds like too much for your business, consider having the hard work done for you. At The Chat Shop, we not only offer managed human chat, but we can build chatbots and offer hybrid solutions if it suits your needs best.

Find out more about what we can do for you here.

Watch the full episode of Everything eCommerce here.

¹https://www.zendesk.com/resources/the-impact-of-customer-service/