Digging Into A/B and Multivariate Testing Results

Ebusinesses can clearly see the value of testing before making design decisions or implementing online marketing strategies. That’s why A/B and multivariate testing are so popular today. But with either an A/B or multivariate method, you are making critical decisions based on one metric alone: "Did the variable perform well or not?"

In review of everything involved in the experiment, do you think that ONE metric is really enough?

One of the best examples of the danger of relying on one metric comes from Marianina Manning’s Web Analytics Princess blog. Marianina is responsible for web analytics and customer experience at Rightmove. The blog entry describes how she investigated the results of a multivariate test with Tealeaf. Marianina wasn’t sure why property renters were converting at a higher rate than property buyers—on the same form in her application. After reviewing some sessions in Tealeaf, Marianina discovered that a content feature about energy efficiency, which appeared only occasionally, was distracting attention from the call to action.

Rightmove is not alone—many ebusinesses have told me that they found unexpected behavior or unplanned results when they conducted deeper customer behavior analysis during multivariate testing. Often, they find unexpected downstream consequences to a change or test in their application, such as a promotional price not showing up at checkout or navigation that doesn’t make sense to some customers based on the new application flow.

As a best practice, you should look not just at which variable performed best but also ask why a particular variable worked well or poorly.

1. Review a select group of sessions—for example, users who got the new functionality (that is being tested before being rolled out to all users on the site), but who still abandoned the purchase process.
2. Use a customer experience management solution to replay a sampling of those sessions to see what actually happened. The number of sessions to replay is dependent on how large a difference was seen in conversion rates (or task completion rates) in the variable that performed well or poorly. Use the same techniques described in my “Where Do I Start Customer Behavior Analysis? I Have So Many Sessions!” blog entry.
3. After investigating why the variable did well or poorly, you might come to a decision to revise the test and run it again before fully deploying the best-performing variable.

Customer behavior analysis is a very valuable complement to your A/B and multivariate testing processes. Whether your test is as small as trying different home page promotions or as big as selecting a new design for a checkout process, you need to be sure that you really understand what’s happening. Customer behavior analysis does just that.

-- John Dawes, Vice President, Product Management