Testing Web Applications in Production

Web applications are organic dynamically changing delivery vehicles. Even if the application or the environment isn't changing -- the user community is in a constant state of flux -- as new users come to the site or repeat visitors return.

I recently read a white paper 'Practical Guide to Controlled Experiments on the Web: Listen to Your Customers not to the HiPPO'. In this case HiPPO stands for Highest Paid Person's Opinion.  What do a British ship captain from the 1700's, Amazon's shopping cart and your customer's experience all have in common?  The three authors from Microsoft tie it all together.

"One accurate measurement is worth more than a thousand expert opinions"

— Admiral Grace Hopper               

Many of us are familiar with things such as controlled experiments such as A/B tests and use them regularly.  After speaking with a number of Tealeaf customers, I was fascinated about how controlled experimentation is being applied -- and the value that Tealeaf brings when running these tests.

Prior to making any change -- you must test. Only way to test is to have a controlled release and assess impact.

• Did I make the changes correctly? (Code)
• Did I get the changes correctly published? (Software Logistics)
• Did users appreciate the change? (Usability, etc)
• Did other factors change? (Need to do this in an A/B Controlled scenario)  As a teenager, I sold lottery tickets in the inner city of Baltimore. Sales always picked up on the first day of the month because government aid checks came out. In order to run a fair test -- you have to evaluate the control group versus the experimental group under the same conditions.

Anecdotes

Here are just some of the stories that I've heard recently.

• A programmer unknowingly changed the font size of the text. Conversion rate went up by 2X. Nobody knew that this change went in – only by replaying sessions did they understand why.
• One of our customers had a 4 page form to enroll for services. They were convinced that if they changed the form to be a 1 page form -- they would get a better conversion rate. They didn't -- conversion rate tanked -- because people were afraid of doing everything on 1 page. Surprised by the results, they ran a second test with a 2 page form and got improved results.
• A/B testing on every new code push -- controlled release. Every time you fix a bug or improve an algorithm to improve performance -- you are also at risk for introducing a new bug or causing other bad consequences. It just flat out makes sense to treat 'new code' just like any other marketing treatment -- i.e. you accept the code only if the conversion rate remains the same or improves.

Visibility is Essential. One of the key benefits of Tealeaf is that we capture the complete user experience. This allows us to observe exactly what the customer did and review not just the conversion rates -- but the whole conversion experience.

In an earlier blog 'Tealeaf and Web Analytics: Quantitative + Qualitative', I outlined how we integrating with leading Web Analytics Packages -- I see a similar need with A/B testing tools. I believe that we can help our joint customers through automating the process of understanding A/B tests through automated Tealeaf event creation and more.

-- Robert Wenig, Founder, CTO, Member of Board

Comments

[Eric Hansen]

Hi Robert,

nice posting about the need to validate site changes (whether front-end or back-end driven). There are lots of pieces to the puzzle of understanding user behavior; serving a controlled experience and tracking behavior by site treatment and customer segment are essential. Only when marketers know what experience was presented, to which visitors, and the response behavior can they truly optimize a site for effectiveness.

Eric