Measuring up to the Competition Part 2 - What Are Your Competitors Measuring on Their Websites and Why?

By John Kim, Product Manager —

If there is one sport that's all about statistics, it's baseball. While I'm not a huge baseball fan (in what other sport do people get excited when nothing happens, i.e., the no-hitter?), I appreciate the fans' obsession with numbers. For example, you'll hear baseball announcers say things like, "he's batting only .127 against left-handed pitchers with runners in scoring position during night-time games played outside when it's less than 65 degrees on the East Coast." At first I thought it was just to fill time, since nothing much was happening on the field, but then I came to realize that these statistics represent a fundamental departure from traditional statistics. That difference is context.



It's not enough to have raw numbers. You need to know what happens when. Old school baseball scouts and general managers used to make their draft picks based on height, weight, speed, etc. But these measurements were poor indicators of success. Only 31 of the 53 first round draft picks made in 1997 made it to the Major Leagues. That's because those traditional statistics don't actually measure how good a player is. They measure only how many times that player got on base or hit a home run, regardless of the competition or situation.

This started to change in the late 90's and early 2000's with the Oakland A's. Known as one of the most frugal organizations in the league, they developed a way of measuring the value of a player not just by the raw numbers, but also contextualized player statistics to create a better team for less cost. They considered the batting average a poor measure of actual performance. Instead, they measured things like Value Over Replacement Player which means "how does this player perform vs. a player that replaces him?" The idea is that a mediocre player can look good on a powerhouse team and a good player can look mediocre on a bad team. Using statistics in context, the A's went to the playoffs from 2000-2003 and 2006 even though they consistently had one of the lowest payrolls in all of baseball. This success was quickly copied by other teams, who now employ many of the same "sabermetrics" to build their rosters. (By the way, to get the whole story, read the Michael Lewis book "Moneyball" or wait for the soon-to-be-released Brad Pitt movie to come to a theater near you). Oh, and for those who think,"but the 2011 A's are terrible," that’s because everybody is doing it now and, all things being equal, money can still buy the better players.

Measure Repeated Attempts to Enter Information on Key Form Pages

So what does all this have to do with web analytics and usability? Glad you asked. When trying to analyze a website's usability, companies often rely on just raw numbers with very little context as to how they actually affect user experience or their bottom line. At Tealeaf, we worked with one company who was worried about the sheer number of errors that their other web analytics applications were showing. Tealeaf confirmed that the website was throwing a lot of errors. However, our solutions went further and showed that most of those errors occurred before the checkout process. Users would often just refresh the page and continue. The overall conversion rate if the errors occurred before the checkout process was the same as for users with no errors. So were the errors before checkout a problem? Yes? Did it affect usability? Yes, because it forced the end users to perform an action (refresh) that they didn't need or want to do. Was it a problem that had to be fixed right away? No. It had very little impact on revenues.

However, if users experience the same errors during the checkout process, conversion dropped. It seems like users found the exact same issue more upsetting during checkout than during regular browsing of the site. Perhaps this was due to lingering concerns about getting double billed or general wariness about revealing personal information over the web. Regardless, this was an error that needed to be fixed quickly because it was affecting revenue.

Measure Repeated Access to Key Conversion Process Steps

Another area where context is important is in measuring conversion rates. Some companies measure conversion as starting the checkout process. If their conversion rate is low, they may feel one solution is to simply increase the number of users who start checkout. An increased number of checkout starts multiplied by the same conversion rate means increased revenue, right? That assumes the conversion rate stays the same. But what about users who start checkout more than once? That typically means they are having a problem checking out, so while your checkout start increases your conversion rate decreases. So three start checkouts spread over three users are great! Three start checkouts for the same user in the same session are not, assuming there aren't three complete checkouts for that user.

What should you be tracking? Many of the same metrics that you're tracking now, such as start and complete checkout, error rate, etc. For an example, click here. However, also make sure that you track when those issues occur. This may require creating events that track issues that are not on the happy path, or that you create more events to determine when they occur. While this requires greater effort, creating those events will help save time overall by reducing false positives and helping you focus on the truly critical issues. And just like in baseball, this can save or create significant revenue. And who knows? Someday they might make a movie about you starring Brad Pitt!

Stay tuned for part 3 of the series of what your competitors are measuring on their websites. In the meantime, please chime in on this topic. How do you measure the usability of your website?