Tealeaf: Visibility. Insight. Answers.

A common Tealeaf question and a new way to reply

As the CTO of Tealeaf, I’m fortunate to talk to many organizations interested in improving their customer’s experience.  Whether I’m speaking to the CIO, the VP of eBusiness or the team managing the online channel the question always arises when discussing online customer experience:  “Why does Tealeaf capture everything, what is the data and just what is it used for?"

I’d guess I’ve had this conversation thousands of times since Tealeaf was founded in 1999 and I enjoy seeing how companies can immediately grasp how visibility into their customer can really change the way they conduct their business.  Because I haven’t had a chance to talk to everyone who might be interested in how Tealeaf operates I decided to visually walk someone through how Tealeaf captures the complete user experience.

How does Tealeaf Capture the Complete User Experience?
Here is my answer in video format...

I'm interested to know your thoughts, comments and what questions you come up with next...

-- Robert Wenig, Founder, CTO, Member of Board

Testing Web Applications in Production

Web applications are organic dynamically changing delivery vehicles. Even if the application or the environment isn't changing -- the user community is in a constant state of flux -- as new users come to the site or repeat visitors return.

I recently read a white paper 'Practical Guide to Controlled Experiments on the Web: Listen to Your Customers not to the HiPPO'. In this case HiPPO stands for Highest Paid Person's Opinion.  What do a British ship captain from the 1700's, Amazon's shopping cart and your customer's experience all have in common?  The three authors from Microsoft tie it all together.

"One accurate measurement is worth more than a thousand expert opinions"

— Admiral Grace Hopper               

Many of us are familiar with things such as controlled experiments such as A/B tests and use them regularly.  After speaking with a number of Tealeaf customers, I was fascinated about how controlled experimentation is being applied -- and the value that Tealeaf brings when running these tests.

Prior to making any change -- you must test. Only way to test is to have a controlled release and assess impact.

• Did I make the changes correctly? (Code)
• Did I get the changes correctly published? (Software Logistics)
• Did users appreciate the change? (Usability, etc)
• Did other factors change? (Need to do this in an A/B Controlled scenario)  As a teenager, I sold lottery tickets in the inner city of Baltimore. Sales always picked up on the first day of the month because government aid checks came out. In order to run a fair test -- you have to evaluate the control group versus the experimental group under the same conditions.

Anecdotes

Here are just some of the stories that I've heard recently.

• A programmer unknowingly changed the font size of the text. Conversion rate went up by 2X. Nobody knew that this change went in – only by replaying sessions did they understand why.
• One of our customers had a 4 page form to enroll for services. They were convinced that if they changed the form to be a 1 page form -- they would get a better conversion rate. They didn't -- conversion rate tanked -- because people were afraid of doing everything on 1 page. Surprised by the results, they ran a second test with a 2 page form and got improved results.
• A/B testing on every new code push -- controlled release. Every time you fix a bug or improve an algorithm to improve performance -- you are also at risk for introducing a new bug or causing other bad consequences. It just flat out makes sense to treat 'new code' just like any other marketing treatment -- i.e. you accept the code only if the conversion rate remains the same or improves.

Visibility is Essential. One of the key benefits of Tealeaf is that we capture the complete user experience. This allows us to observe exactly what the customer did and review not just the conversion rates -- but the whole conversion experience.

In an earlier blog 'Tealeaf and Web Analytics: Quantitative + Qualitative', I outlined how we integrating with leading Web Analytics Packages -- I see a similar need with A/B testing tools. I believe that we can help our joint customers through automating the process of understanding A/B tests through automated Tealeaf event creation and more.

-- Robert Wenig, Founder, CTO, Member of Board

Free the Data

In today's all digital world, there is a never ending onslaught of data. For example, in 2007, it's estimated that there are 20 million users getting their email on wireless devices(i.e. Phones, PDA's, etc). This number is estimated to grow to 350 million users by 2010.

Whether it's email, digital pictures, mobile transactions or just 'more data', the critical business problem is:

'The Information Value Chain' - allowing companies to share and understand data to support better decision making. In other words, more data makes your SAN salesperson very happy, transforming that data into actionable information allows your business to prosper.

Data is a lot like milk. Being the nerd that I am, I don't really care for milk, but I do like pizza which requires the transformation of milk into cheese. Left untreated or cared for, milk spoils and creates a stench. 

So, what does this have to do with Customer Experience Management, and Tealeaf?
Tealeaf creates a lot of data -- we need it to power a visual replay of a user's web session. However, just as there are milk consumers, soda-pop consumers and pizza eaters our customers have different needs for Tealeaf data -- and different processes for turning the data into valuable information. Some folks need a snippet of the data in real-time, others need a full forensic interface to the data. How do we keep everybody happy?

Here's an interesting example of how to leverage the value of Tealeaf data: Does the tail wag the dog, or does the dog wag the tail?

The convergence of Web 2.0, SOA (Service Oriented Architecture) and EDA (Event Driven Architecture) are causing some interesting changes in the application development, deployment and monitoring spaces.
In classical engineering education, we are taught the concepts of feedback loops. We study how an airplane adjusts its course by determining its location and how far off course it may be. The following sentence is always the word of caution to 'not adjust course too often', otherwise you induce the plane into instability and crash.

Today's applications are being built with the notion of an information highway. The applications, consume, generate and transform information put onto the 'bus', i.e. the highway. Not only do the application components inter-operate via the highway and its associated on/off ramps, the application monitoring and diagnostic tools also make use of this common infrastructure. Remember Field of Dreams, 'If you build it, they will come.'

In a Web 2.0 world, subroutine calls have been replaced by HTTP/HTTPS invocations. The abstraction of HTTP/HTTPS allows for components to operate anywhere -- and not just within a single computer or single network. This integration/disintegration can occur at any spot on the highway -- i.e. it can take place during the page creation or it can take place within the user's browser. In plainer words, content aggregation can take place either on the server or on the client.  Welcome to the world of Mashups (Wikipedia).

During the Internet Bubble, few things were as wildly hyped as Personalization and the touted ability to customize each and every user's online experience. The reality didn't quite match the hype and/or the cost outweighed the benefit -- however I feel this class of 'feedback' coming back -- albeit with a new infrastructure.

Just what does it take to influence somebody's behavior?

• Is it a billboard for a gas station when your car is approaching empty?
• Is it the offer for help when your car has a dead battery in the airport parking lot?

It's the right data at the right time
Have you ever noticed that many of the internet acronyms (SOA, EDA) are 3 letter abbreviations for 'now', as in I want it 'now'. All of this data flowing, oozing and cruising down the highway needs to be analyzed and acted on in real-time as well as new applications and application flows constructed dynamically.
With the technology stack that we now have, we can even change the behavior of an application without changing the structure of the underlying application.

On the Tealeaf website, we say:
See your online customers...Capturing the details of every single interaction, Tealeaf delivers insights that explain why your customers succeed or fail in their transactions.

In the past, we would have talked about our ability to discover sessions, to replay them and/or to score them/evaluate them. With the latest Tealeaf release -- we can do more -- we can influence the app itself and build a foundation to leverage applications dynamically.

We now have an API to send the data out of Tealeaf in real-time. We can send our view of the application that we are monitoring back onto the information highway. For lack of better terms, we call this the 'Event Bus.' As web 2.0 applications started using RIA(Rich Internet Application) Frameworks such as Ajax, Flex and Flash -- we've also evolved our data collection capability. While the bulk of our data is collected passively via a server side packet sniffer, we now complement that view with methods to see what's happening inside of the browser. Initially we did this work in order to power replay of Ajax applications.

Our hooks for capturing activity from inside the browser come from including some simple JavaScript on the page. Ordinarily, the JavaScript collects client-side activity and asynchronously posts this information using an Ajax call. What happens when there is an intelligent listener to this Ajax posts? This opens up a channel to talk back to the user's browser. Utilizing this mechanism, we could dynamically offer:

• Chat or customer assistance to people who appear to be lost or who are having unproductive sessions.
• Surveys or other forms of customer follow-up.
• Targeted offers based on activity.
• Detection of fraudulent behavior and

I am not suggesting that Tealeaf will enter the 'spaces' described above -- but that our data, analysis and infrastructure can be used to power and partner these application solutions to deliver additional value.

Clearly, Tealeaf is not the only vendor working to exploit opportunities that are arising from the 'Now paradigm' -- load balancers from F5 and Cisco now have full external API's that can be used by the applications themselves. There's even the potential of browser bloat as every vendor of chat, click-to-call, survey offer, etc want to get their own JavaScript stack embedded into your application. It's not just the code -- these hooks also need to talk to a server taking up more resources, etc. The analytics companies are also feeling the tension to perform real-time analysis.

I am suggesting that the Tealeaf infrastructure makes building these class of applications a lot easier and will enable more robust interactions.  It's been said that 'There's a thin line between love and hate', there's also a thin line between application monitoring and feedback -- Welcome to the new world order.

What's worth more, an ounce of insight or a pound of data?

Free the data, let it be used to produce the insight that you need.

-- Robert Wenig, Founder, CTO, Member of Board

Tealeaf and Web Analytics: Quantitative + Qualitative

In the new whitepaper, Customer Experience Management and Web Analytics: From KPIs to Customer Transactions, authored by Eric Peterson of Web Analytics Demystified, Eric talks about the different ways of understanding the online experience. He covers the quantitative measures (traditional web analytics tools like Omniture, Webtrends, Visual Sciences) and the qualitative measures -- Customer Experience Management (What is CEM? - i.e. Tealeaf). In short, he makes the conclusion that both are foundational and both are needed.  The “What” to compliment the "Why".

So, let's build on the foundation of his conclusion -- what are the benefits of bringing together Q+Q (Quantitative and Qualitative).

Does Q+Q = 2Q or do we get Q squared?
As long as Q is a positive integer greater than 1, you want Q squared, right? After all, this is CTO chatter…

Some background on the situation
In earlier blog entries I talked about where Tealeaf data comes from, Tealeaf captures every byte of the HTTP/HTTPS requests and the HTTP/HTTPS response. We need all of this data to power replay of a session. We're able to capture the complete user experience passively -- i.e. with no risk, latency, additional CPU or code changes.

In general, a web analytics tool operates on a subset of the data. This data comes from page tags, weblogs or server side sensors.

Page tags can be an interesting analytical tool -- properly done, pagetags may contain information that is not normally on the page -- and the information can be better suited for analysis by encoding more structure and context. Good pagetags require code changes and proper foresight -- i.e. you have to know what information that you want/need to capture in advance. For many of us, changing a pagetag across a large site could take days to weeks to months to implement and that doesn’t factor time to then collect enough data to properly analyze.

Q Squared -- What are the benefits?

• Drill down

• A web analytics tool will do a good job of understanding customer fall-off. They will show you a funnel analysis of where people stop in a buying process. You will be able to see how many people entered the funnel and at what point people stopped/diverged, looped, etc. This answers the question of "What Happened" -- but it leaves you thirsting for more. With our combined solution, we can now pivot from "What to Why" -- i.e. we can drill down and launch visual replay.
• How can you trust aggregate data if you can't drill down to the actual sessions and replay them?

• Drill up

• What happens when some of the users on your site experience an obstacle in completing their transaction? A small percentage of people get 'Price Not Available' when they attempt to book a flight.
• Did you even know that your pricing engine could respond with such a message? Since this message appeared on the page, the page had a valid, good HTTP status code. It's doubtful that your pagetag could have picked up this message because it would have required scanning the HTTP response for a message that you didn't even know that could happen.
• Obviously, in such a situation, it would be impossible to understand the behavior of visitors who experienced 'Price Not Available' because this dimension was not known/observed by your web analytics solution.

With Tealeaf, because we capture the complete user experience, we could search for all people who received this message. We can then export a result set of sessions back into your web analytic product. In effect, we can dynamically create a new dimension of 'Price Not Available' by creating a synthetic filter or dimension on-demand through the integration.

Synthetic Page Tags      
Tealeaf Events are beneficial. In a Tealeaf environment, you create events by establishing rules to evaluate each and every page. Tealeaf events aid in creating structured, analytical information out of unstructured content. Since tealeaf is passive and invisible to the application infrastructure, events can be easily created/changed on demand. Tealeaf uses these events to create KPI's (Key Performance Indicators) and to aid in search and session scoring.

Taking the example above of 'Price Not Available', we can easily create an event for this. Out of this event, we can send a fake page tag containing this payload up into your web analytics solution -- without having to modify the original application. Pretty darn cool, if I say so myself.

Historical data is valuable.   
Since a web analytics tool is operating on a subset of the data -- you have to define ahead of time what data to capture. If you don't capture the data at execution time -- it's lost forever -- and obviously you can't analyze what you don't see. One of the most commonly requested Tealeaf features has been the ability to event on historical data. People want to be able to make observations on past data and create analytical insight. We've come up with utilities to replay/respray old data and persist these new observations into an analytical shell.

Q Squared -- let the revolution begin.

-- Robert Wenig, Founder, CTO, Member of Board

New Twists in Capturing the Online User Experience

When people ask me, what Tealeaf does, I usually respond that Tealeaf provides needed visibility into what's actually happening on a web site. I typically give the analogy that Tealeaf allows you to see/understand the experience of each and every user on a website as if you had a camera over the shoulder of each and every user.

• What happens when the user experience goes across multiple web sites and multiple companies?
• How can this happen?
• Is this a mashup, SOA, a train wreck or opportunity?

Let's start with a simple example:

An online retailer chooses to outsource their catalog search function. When the user hits 'Search', the search functionality is mapped via a DNS Lookup(Cname) to an outsourced service provider. The ASP (Application Service Provider) returns complete pages of HTML content. Only when the user clicks on an item is the user returned back to the online retailer's actual web site. The user of the web site is completely unaware that they have traveled between web sites and company borders. We like to say that on the Web, your competitor is only 1 click away -- however the power of making 'subroutine calls' via HTTP means that any service or service provider is even less than a click away.

In the extreme sense, this is like driving down the highway and you enter a long tunnel. No lights are in the tunnel and your car's headlights are not operating. This is clearly not a good situation. Search is an integral part of the online shopping experience -- how can I not have Tealeaf level of visibility to the search navigation and content? What happens if the external search content is misleading, broken or confusing? How would the site owner become aware of these issues and fix them?

How do you overcome this problem?

Do you go to using 'multiple cameras' -- one per company and join the data streams -- or is there a better way?

At first glance, the multiple camera story sounds good -- it's just an extension of normal tealeaf. However, this will result in data capture in multiple data centers, requires installation of hardware across multiple companies and data centers and all sorts of other management headaches.

Should people shy away from utilizing 3rd party services as part of their application flow -- I don't think that we can easily close Pandora's box here -- the potential value is too high -- this 'issue' will not easily go away.

So, is there a better way to capture the complete user experience using a single camera?

We could capture from inside the user's browser. While this would be 'one camera' -- it would also be 1 camera per user -- so this would not be too good a solution.

Why not capture using the same techniques that enabled this class of application in the first place? Many people feel that all computer science problems can be solved with an additional level of indirection -- so here goes. Instead of just using the 'DNS remap' i.e. CNAME (DNS Alias) to allow services to be located anywhere, can we setup a common access point through which all content flows? In plainer terms, can we 'proxy' the request to the search application so that all content flows past a single inspection point, i.e. 1 camera. Such a proxy would introduce minimal latency and give the needed visibility at a minimal TCO (Total Cost of Ownership).

If you have complete visibility to the Mashup/SOA application stack -- then you can deploy with confidence -- and deliver better applications faster.

-- Robert Wenig, Founder, CTO, Member of Board.

AJAX Angst, a real world experience (part 2 of 2)

In AJAX Angst - Part 1, I covered how integration of AJAX on the web is becoming ubiquitous in the interest of building a better user experience, delivering richer content quicker and lowering development overhead.  In Part II I’ll detail what happens with adding complexity and how good plans can go awry.   

Let’s cover how AJAX can help solve problems, and deliver a better customer experience.

A travel site creates an AJAX application using JavaScript to validate passwords.  This was designed to enable a smooth user experience and low-overhead (or “server expense”) by removing the page reloads.   The feature would allow users to enter non-alphanumeric characters (e.g., @#$%, etc..) when creating a password, a good idea for passwords, making them harder to guess and bolstering the company’s security.

However, when coded, the JavaScript that validates the password only allowed valid alpha-numeric characters.  Any non-alphanumeric characters were rejected.  Whoops! 

Users could create complex passwords when setting up the account and trying to complete the purchase, however when they tried to login after creating their account. they would get an error message and abandon… In rare cases, and only if the company was “lucky”, the users would contact customer service to try to complete their transaction.

In the interest of delivering a smoother customer experience with AJAX, the company has now inadvertently put a road block in the way of their conversion rates, overloaded the support department and sent online revenues plummeting... What makes it really sting… this could be an “unknown site error” until elevated, by an infamous QA group labeled “your customers”.

However errors happen, let’s focus on fixing them and recovering both lost customers and revenue.  With AJAX capture and replay, Tealeaf can immediately locate all affected sessions, allowing developers to pinpoint the password error and work on a solution, while customer service can reach out to all affected users who didn’t complete their transaction and extend an offer or discount to reengage the customers. Marketing can create a message or campaign to encourage new account sign-up and creation.

This all becomes possible, because with Tealeaf you have the data…

AJAX provides great power for improving the customer experience.  But with great power comes great responsibility (Thanks Spiderman!). 

Have a AJAX challenge...let me know.

-Mark Neuman, Vice President of Engineering

AJAX Angst (part 1 of 2)

So someone at your company decided that you need to jazz up your web site with “Web 2.0” technology.  You need a Rich Internet Application (RIA), which might involve using this new AJAX technology.    What is AJAX? (And no, it is not a household cleanser).How does it change the customer experience? How can you measure it? 

AJAX stands for Asynchronous JavaScript and XML, but it’s really a combination of multiple technologies including Dynamic HTML, JavaScript, and dynamically created requests for data from a web server.  The result is a web page that behaves more like an application installed on your PC (like the word processor that I am writing this with).  This means that instead of the page, page, page interaction typical of most web sites, you just operate one web page, which changes as you interact.  This can greatly improve the user experience by avoiding the sometimes jarring experience of each page unload/reload where the entire window changes appearance.

If you use the web regularly, you have probably used AJAX applications, even without knowing.  Address books are a current popular use of the technology,  the page might include a search field where you can type in search terms to find in your address book, which is located on a web server far, far away.  Under the search field is a list of names and addresses, maybe the first 10 contained in the address book.

When you type “John” into the Search field and hit the enter key, the JavaScript program running in the web page formats a search request and tells your web browser to send the request to the web server, just like you tell the web browser to go to google.com.  The web server receives the request and responds with a list of names and addresses containing the word “John”, formatted using the XML standard (eXtensible Markup Language, aren’t acronyms fun?).  The browser provides this data to the calling JavaScript which creates Dynamic HTML that the browser formats and displays to you.  If you change the search term, the process repeats.

Compared to a “regular” web page, the screen did not refresh with a new version of the page showing the list of “John”s, only that part of the screen listing the first 10 addresses changed.

At this point, it is interesting to consider how the back button would work, or more importantly, how it would affect the user’s experience?

• If they hit the back button now, would they expect to see the original list of addresses?
• Will they be surprised when they return to the page preceding the address book?
• Will they hit the back button, lose focus and leave the site?

This simple example is just one of the considerations developers and designers face when integrating AJAX to make web pages more application-like.  Within the customer experience, the user interface metaphors start to get mixed up.  The end user can become confused as paradigms change and the effort to enhance the experience, could actually lower conversion and cause users to abandon..

In addition to changing the user interface paradigm, AJAX has also added complexity to the development team. Before, if there was an error on the page, say the server returned a blank list instead of the first 10 “Johns”, the programmer would look at the code on the server to troubleshoot.  With AJAX  the programmer needs to look at both the code on the server and the JavaScript running in the web page, then consider what user variables could be in play and attempt to recreate the error.  This takes us back to a core QA issue, with yet another set of variables.

• How can I accurately monitor and measure the user’s experience?
• How can I troubleshoot off averaged analytical website data?
• Is this new feature helping or hurting the overall customer experience?

To address the challenges within development and testing (as well as measuring the impact of the AJAX initiative against marketing goals), you really need to have the entire user’s experience and that starts with data capture.

I’m often asked, “How does it work, can you really track AJAX?”

With a regular web page, Tealeaf CX captures the data stream between the web server and the browser.  But for AJAX, the user interactions are within the web page, so no data is being transmitted, therefore how can Tealeaf see what is happening?

To capture this interaction, the web page programmer adds a line of code to the web page (usually near the top) that tells the browser to include Tealeaf-supplied JavaScript code from the server.  This code “instruments” the regular JavaScript application.  It can collect information about the user environment - what is the current screen size set to?  Is it big enough to show the entire page?, etc. 

The code also makes sure that all the objects on the page (e.g., that “nice” tabbed look across the top of the address book that lets you show different fields) have a unique ID.  And most importantly, the JavaScript captures every character typed, every button pushed,and (if desired) every movement of the mouse. 

Within AJAX applications,  we have extended the insight available, and our customers now have the data they need to create actions.

However, sometimes even the best intentions can go awry.  In part II, I’ll address an interesting  “real-world” scenario.

If you have an AJAX challenge or want to  determine if your RIA is adding value, I encourage comments...

-Mark Neumann, VP of Engineering

Keeping Tealeaf data forever

Recently, more and more customers are talking about the desire to keep Tealeaf data forever. “Forever” is loosely defined as a period of greater than 1 year.

The desire to keep the data for this period comes from the need to handle interesting business challenges, including:

• Dispute Resolution
• Security/Fraud Forensics
• Compliance

The use cases described above require the retention of all Tealeaf sessions. There is no 'pre-ordained' trigger to decide which sessions to keep - they require keeping all the session data, permanently.

To effectively utilize the data, our customers also want the simplicity and flexibility of Tealeaf’s search interface - so we also need to keep the indexes we create of the data. In addition to keeping the source data and the indexes, our customers are also asking for the capability to create retroactive events and construct unusual queries on the data. Customers have seen great utility value in our ability to 'respray' historical data onto Tealeaf CX’s event bus into Complex Event Processing (CEP) engines.

The desire to create retroactive events and 'respray' the data comes from the fact that while our search techniques are robust -- they don't allow for all data to be indexed (HTML constructs, comments) and CEP engines allow for queries that measure things differently.

Today, our largest customers are generating more than a terabyte of data each day. By year's end, the cost of 1 terabyte of disk will be less than $300. (Of course, serious enterprises likely want to have redundant copies of the data, etc). With these kind of storage requirements (ease of access, cheap, relatively fast),  keeping the data on disk is really the only way to go. Tapes would be too slow, too costly and too labor intensive.

In Tealeaf CX, the data is logically organized by day. The current day's disk activity is intense - the disk subsystem is under extreme stress to keep up with persistence and indexing of the data. Logically, once the current day is done, the data is seldom accessed -- and then only for read activity.

In order to effectively scale to current day real-time activity, many of our high-volume customers are setting up and provisioning Storage Attached Networks (SAN). Their Tealeaf data resides on as many as 40 drives (Fiber Channel or high end SCSI) with multiple fiber channel connectors. They use this class of high speed storage (Raid 0+1) because we are writing data at an incredible data rate. We also need redundancy and speed - thus the Raid 0+1. We have customers who are capturing ~500+ Megabits per second of data into Tealeaf.

For days 2-n, Tealeaf access to the data is purely read-only. We also don't read the data that often - typical customers do between 40 and 300 searches a day.

So, we need to have a different storage architecture for the truly long term, read-only data.

We have the Tealeaf 'Archiver' which can move data from one storage subsystem to another. With the right 'SAN Tools' -- it's possible that this data movement can be accomplished with 'snapshots' or 'mirroring' -- i.e. an optimized block or volume level copy. Even with this level of transformation (i.e. moving the storage from fiber channel to lower cost SATA) -- there is still a problem. A SAN doesn't have infinite storage. It is limited by the number of physical connections for drive attachment. Even if this weren't an issue, keeping all of the media 'spinning' would be an energy hog and would eventually lead to drive failure.

As a teenager in Baltimore in the 1970’s (before that fateful day when I checked “Born to Run” out from the public library) I (like everyone else) listened to the great orator Earl Weaver (the legendary manager of the Baltimore Orioles). One of Weaver’s famous sayings was: “It’s what you learn after you know it all that counts.”

So, I thought I had a pretty good handle (no pun intended) on the problem above. Having been in California for 20+ years, I’ve put solar panels on my house, bought a hybrid car, ride my bike to work (although, I would have to say that the fact that you can’t get good unhealthy food in California is not a plus – where’s the hot dog wrapped in fried bologna?, steamed crabs?, tastykakes?.... even good fried chicken?) I even started looking at new-age storage ideas like spin-up storage and hierarchical storage systems.

As I talked with our customers about these ideas – the customers quickly brought up another real problem: while disk drives got denser and cheaper, the underlying operating systems didn’t keep pace.

Microsoft Technet Article on File Systems

In some ways, the limits made sense. Who wants to “chkdsk” a 100tb drive on start-up? But, on the other hand – people are selling 1tb drives and the basic file system limit is 2tb?

So (after considering the advice of the great Mr. Weaver) we will soon begin the work of allowing each day of a Tealeaf archive to reside on a separate volume.

After all, it’s what you learn after you know it all that counts.

--Robert Wenig, Founder & Chief Technology Officer

Where does Tealeaf Data Come From -- and Why

 As web applications have grown in usage, feature richness and complexity the need for understanding the end user experience has skyrocketed. This has led to a lot of different ways to try and gather insight into end user experience.

In the beginning, the web was a place to consume static content. Web servers contained access logs to show which items were requested. Since the content was static, a successful status code (i.e. HTTP 200) was adequate for knowing that information was delivered. Since the content was static, if you knew the resource that was being requested, you could also infer the content.

The Web didn't stay static for very long.

However, the methods of logging the web requests haven't changed much in well over a decade.

Web logs

• The original standard. Contains the request for the resource along with the associated status code.
• Requires extra work and/or extensions to get 'Post Data'. Understanding 'Post Data' is vital for transactional web sites.
• Requires one data collector per Web server. In a large farm of web servers, this gets to be problematic.
• Yields a server side view of the world. The page may have been requested by the user and not received or canceled mid-flight.

The main blind spot of Web logs is that a Web log has no knowledge of what content has actually been delivered to the user! If the dynamic web page contains pricing and product availability -- the web log has no record of this. If you request the page later -- pricing and availability may change  -- since pricing and availability are conditions, not lookups. (I.E. availability is a temporal attribute -- 2 days later, the concert ticket for Springsteen is no longer available) You would not know this from looking at the Web log.

PageTags

As the web grew more sophisticated, there was no longer a guarantee of having a direct path between the user's browser and the web server(s). The advent of caches and application acceleration services (Akamai et all) meant that some or all content could be served by an intermediary. In practice, this was mostly for static content.

Since 'caches' could intercept the request, the web server would not get accessed for the content, thus there was no log entry.

In addition, collecting web logs from a whole farm of servers was a tedious and sometimes error prone process.

For these and other reasons, 'PageTagging' was born. While pagetagging can take on many forms, the most common form of pagetagging takes the form of a snippet of Javascript which is automatically run every time the page is loaded.

The Javascript makes a dummy request back to the origin site (i.e. the webserver) or a third party AASP(Analytics Application Service Provider). In this request, the JavaScript can supply information to be analyzed. The Javascript can pickup environmental data from the browser (true page load times, screen resolution, locale) along with other information that the programmer was instructed to include as part of the page (shopping aisle, contents of the cart, true name of the page, etc).

Since the Javascript runs on page load, most of the issues related to caching will disappear.

With the movement towards Rich Internet Applications (Ajax/Flex) the notion of having a 'Page' for every state transition will go away. The application will then need to trigger/execute Javascript for each major state change.

The next step in the page tagging evolution was to create Javascript which actually monitors the user interaction within a page. This class of Javascript monitors DOM (Document Object Model) events -- i.e. it has the ability to understand/see every keystroke or every User Interface event (mouse move, etc).

This allows a company to see what's happening within a page -- field to field navigation, dwell time per form field, last form field entered prior to abandoning the application.

The problems of  pagetagging include:

• The cost of putting in the tags. It takes programming, qa, staging and more.
• The more complex tags can break the application -- since they are part of the application.
• The job is never done, the website changes constantly -- tags need to be added to new pages, changed, etc. At some companies, the lag between requesting a new or changed page tap and actual implementation can be lengthy. This requires coordination between IT and marketing.
• There is additional overhead on the client. Pages are now bigger to download because you have to download the Javascript. While this can be ameliorated through caching, it is still an issue.
• The Javascript needs to talk back to the server -- creating additional bandwidth from the client and a multitude of new requests back to their server(s). If the client doesn't send the data back -- it is lost forever.
• There are potential security and privacy issues about collecting data from the client. Data must be encrypted in transit, sensitive data such as passwords must never be collected.
• The JavaScript can take additional CPU resources on the client.
• Most important -- The value of a tag represents the 'pre-ordained idea' of the developer who implemented the tag. The tag is one slice of the overall picture. If the analysis of the data requires actual prices presented to the user and the tag doesn't contain it -- the opportunity for analysis is forever lost.

Application Instrumentation

In the page-tagging model, the Web page is the carrier or transport of the analytic payload. Traditionally, a good software development practice is to build additional logging capability into the application. This logging capability can yield analytic insight or extra diagnostic information.

Why should server side components need to put extra information into the Web payload? The ability to join extra information 'Out of Band' from the server side along with the user experience can be very helpful. This 'join' can take place in real time in the analytics framework -- or it can occur post-mortem through effective pivoting or slicing/dicing of the data.

For example, if the application uses log4j as a common logging framework, it can be very beneficial to join this data with the end user experience to diagnose root cause or to expose additional analytic information for subsequent analysis. You don't want to include this information on the page body.

What does Tealeaf do?

Tealeaf, by default, passively captures the complete HTTP/HTTPS stream of each and every user on a website. Tealeaf receives its data by plugging into the span port of a switch. As such, Tealeaf adds no latency, takes no CPU, adds no risk.

You can think of Tealeaf as a reverse application server, reassembling TCP packets back into the HTTP request and response. The process of gathering the hit (i.e. the HTTP request and its associated response) generates interesting metadata that can be turned into value added metrics. These metrics include timestamps for:

• Timestamp for first byte of request
• Timestamp for last byte of request
• Timesetamp for first byte of response
• Timestamp for last byte of response
• Timestamp for last acknowledgement from the browser. This can be used to measure network latency with some degree of success.

In addition, data elements are parsed and decoded. The URL is broken down into name value pairs. Post data is also decoded into name value pairs as well.

After the hit is assembled, it is run through a pipeline of functions. Operations such as privacy (removing or encrypting sensitive data), sessionization (grouping related hits per user) and routing until the data goes into an in-memory database. As the data enters the in-memory database, rules are run against the hit and the session. These rules can scrape values out of the request or the response for subsequent scoring and thresholding. The rules can also look across the 'session object' and make determinations about the online experience using more complex logic.

This scoring of user activity yields interesting metrics which are published in dashboards. In addition, these observations can be pumped back to SaaS (ASP) web analytics products as synthetic page tags.

Creating these rules requires work, however it doesn't require changing the application. These rules run in the context of the Tealeaf CX data store.

If the application is already using page-tags -- these tags can also be used to yield additional information. For example, if the tags contained semantic information such as 'aisle' or 'category' or 'class of user', this can be easily harvested and reused within Tealeaf.

When a session is over (usually determined by a timeout), the session is persisted and the essence of the session is searchable by dynamically converting the session into an XML document and streaming the session through an embedded XML search engine.

In many of the recent articles about Business Intelligence (BI), the future is all about using search as one of the most powerful tools for discovery. Tealeaf has worked this way since its inception. Hindsight may be 20/20 -- however in this case, our foresight was pretty good too.

The secret to Tealeaf's success is that we have optimized the TCO of acquiring and managing this amount of data. In addition, we're hard at work at further exploiting the value of this data.

Tealeaf Multi-Point Capture

Tealeaf always depends upon network capture of the complete HTTP/HTTPS stream.

We can marry the HTTP/S stream with application instrumentation. In addition, we can also take the insight from within the user's browser. This allows us to capture and replay Ajax, Flash, Flex in addition to getting more insight from traditional HTML applications. Along with this, we can read other 'tags', create awareness with our rules and can even do operations like shadow browsing of  active sessions.

The ability to have all of these different types of information in one flexible container is a unique value proposition of Tealeaf.

How can you trust aggregate data?

Many times, the aggregate data leads to the observation of 'What Happened.' How many visitors to my site, how many went down this path, how many orders did I get?

This leaves two fundamental gaps.

• Why? Why did they abandon at step 'X' and what did the site do to force this to happen? Was it price, page content, performance, confusing navigation, errors on the site or more?
• How can you believe aggregate data if you can't drill down to the underlying data? Summarizing data has its merits -- but not at the expense of fundamental understanding.

What is the right analytic approach?

While each approach has its own relative strengths and weaknesses, I believe that only Tealeaf has the complete solution. The richest data set, the richest set of functions on the data and the biggest opportunity for exploitation and true ROI.

-Robert Wenig