The Many Shades of CEM: Rapidly Building a Business Case

By Brian Strauss, Director of Sales Engineering —

Here's the latest installment in my ongoing series on the importance of Business Impact Analysis—the holy grail of Customer Experience Management. So far in this series we've taken a look why being able to conduct a business impact analysis is such an important part of your Customer Experience Management (CEM) practice, and we've also taken a look at how the various web site optimization tools stack up in their ability to provide actionable, business-case-driven data.



Now we're going to take a look at how business impact analysis is performed in Tealeaf. We'll show how to move from discovering an interesting anecdote to building a business case for prioritizing a fix. The key to creating such a business case is understanding that you need to go beyond merely counting how many times online customers encounter an obstacle. The much more important information is how many transactions aren't completed as a result of that obstacle. With this knowledge, you can easily quantify, in monetary terms, how much the issue is really costing your online business. Once you've got that, you can prioritize issues more effectively by their actual business impact. So let's get to it!

Step One: Identify the Root Cause

Let's say that we see that a good number of people are struggling with a specific form field validation problem on our billing page. Drilling down into one of the web sessions, we see that a customer was trying to enter her billing information for the first time. She made it through the first several steps of the process, but when it came time to validate her email address, she hit a roadblock because apparently she didn't remember that she already had an account with us. So the application sent her a validation message specifying that her email address has already been used, as shown below.



Rather than merely clicking on the login button at the top of the screen (which she may not have noticed) and moving on, she curiously edited her password instead and re-submitted the form, only to get the message a second time. On the third pass, she edited the password, again submitted the form and received the validation message. Completely confused, she abandoned the site after the fourth attempt.

After we replay a few other sessions from users that were struggling with this same message, we notice the same behavior several times. What could be wrong? There doesn't appear to be a system error, or a slow performing page. In fact, it seems as though the application is performing correctly. Why would all of these people attempt to edit the password when it's the email address that we’re calling to their attention?

And then a hypothesis dawns on us. For every other field in the checkout process we place our validation message directly above the field that needs correcting. In this case we've placed the message several lines below the target field—right above the password field! We suspect that even though the text is clear, we have a design flaw that's confusing users. We are not following our typical validation convention for this page, which causes the user to struggle.

Okay, so we've now seen this anecdotally a few times, but how can we be sure that this message is blocking a large number of users? How can we build the justification to quickly fix this problem? A mini business case will help.

Step Two: Compare Conversion Rates

Now we'll need to figure out if the users who get this message convert at a lower level than those who do not. Visitors get this message only after they fill out the billing form and click the continue button. So we are going to set up a control population as the group of people who click “Continue” on this form. We first need to isolate the conversion rate for this group as a whole.

To calculate the conversion rate, we'll search Tealeaf data over the past week for the number of people who clicked the continue button on this billing page at least once. It turns out that there were 27,135 of them. We then ask Tealeaf how many of the people who clicked continue on the billing page also made it all the way to a purchase. The answer is 26,550. Dividing the latter number by the former, we can see that our control groups converts at a very high 97.84%. After all, this is the bottom of our funnel and we should expect as much.

Next, we need to get the conversion of our study group—those who received the error after clicking continue. So we go back to Tealeaf to ask who clicked continue AND got the error: 1,449 sessions over the same time period of seven days. To finish the study, we now ask how many of the sessions with errors successfully completed their transaction: 1,347 or 92.96%

Right off of the bat we can see that, yes, this issue is confusing enough for users that their conversion rate is nearly 5 percentage points lower than the entire population!

To learn what's next, read next week's post, where we'll discuss how to complete the analysis by determining the opportunity cost of not addressing the issue.