Gorillas in the Mist

By John Kim, Product Management —

Before reading further, watch the video. The goal is to count how many times people wearing white pass a ball around. The video is only 90 seconds. Go ahead, I'll wait.



If you watched at least 45 seconds of it, you should've noticed the gorilla. If you didn't watch past 45 seconds, you may have ADHD. If you didn't see the gorilla, don't worry, most people don't see the gorilla. All they see is amazing lack of athleticism by the people passing the ball (the Harlem Globetrotters they are not). This phenomenon is called "inattentional blindness." Basically, you only see what you expect to see and completely ignore the unexpected. Everybody does it.

Is the World of Analytics Blind?
This phenomenon is especially true in the world of analytics. With most analytics suites, you're not actually looking at what the users are doing. You're seeing only a small subset of those metrics that you have specifically instrumented your system to track (i.e. actions that you expected enough to instrument). Not only that, you're not even seeing the actual behavior, just abstract, numerical representations of the behavior. It gives new meaning to the term "double blind." The issue is that it is the unexpected issues that cause a large percentage of usability problems. Because if you knew something was a problem, you would’ve done something about it already, right?

So how do you find something you weren't even looking for? Admittedly, it's not easy but it's possible. The first step is to acquire as much data as possible. If you’re worried about data overload, you can do a couple things. One is use tools that automatically look for anomalies. That way you can manage by exception. The other route is to simply not look at all of available data all the time. Instead, look at only the key metrics. What typically happens is you'll look at one of those keys and notice something a little odd. Perhaps conversion is down a couple points from what is expected. It is then that you'll want to look at more data to see what happened. Then come up with a hypothesis as to what could be causing the drop. You're probably thinking, "This still requires that I pre-instrument my system to get the extra data, which requires some expectation of what could happen. It can't be totally unexpected." That's not entirely true. If your system just grabs all of the user sessions and not just the instrumented metrics from those sessions, you may be able to run a search against the sessions that already occurred that match the conditions of your hypothesis. A search query is like a rules engine after all (e.g. return sessions that match conditions X and Y).

Analyzing Unknown Issues
As an example, on my first day at a new customer, the future Tealeaf admin asked if I could track an issue that the developers didn't think was possible. She claimed that sometimes if you click on a certain page of the signup process, you're dumped back to the home page. She said she saw it once herself but couldn't reproduce it and to a developer, non-reproducible basically means impossible. Our system had been running for a couple days before I arrived so I ran a search with those general conditions. After looking through some of those sessions, we were able to find one session that exhibited the issue. One session isn't enough to change code for, but it is enough to create more specific search criteria. Once we did that, we found that the issue occurred several thousand times a day. So now the client could go to the developers and not only show a reproducible case, but also quantify the severity of the issue and the affect on the bottom line. We went on to create an event to track how many times it would occur in the future, if there is a correlation between occurrence and hour of day, and discover the exact scenarios.

Did I know that the issue existed and pre-configured the system to track it upon installation? Of course not, it was my first day on site. But we were able to find it by having all the session data available so that we could quickly test any hypothesis we came up with, then narrow down the results, and repeat until we found the issue. The process would have taken much longer if we could only test against future sessions, which would've cost the client not just in billable services hours, but also in lost revenue from abandoned checkouts while the issue remained unresolved. For the customer, she will probably run into more issues like that in the future. No website is perfect and they're also constantly changing. By gathering as much data as possible, she'll be able to track down and therefore resolve issues much more quickly. And we all know that time is money.

Regardless of what tools you have, the biggest key is to keep looking. Understand what you're measuring and their limitations. There are gorillas out there.