Avinash Kaushik
The following is the transcript of an interview of Avinash Kaushik, Director of Web Analytics for a Fortune 500 company, conducted on January 19, 2007. Avinash Kaushik is a Web Analytics expert and a blogger (Occam’s Razor blog). He is also the author of a book on Web Analytics called Web Analytics: An Hour A Day being published by Wiley in May 2007. You might also find him speaking at an industry event such as the Emetrics summit.
Interview Transcript
Eric Enge: Can you start by providing a brief background of yourself and how you got into analytics?
Avinash Kaushik: I am a mechanical engineer, that’s what I have a Bachelor’s in, and I have an MBA in Finance and MIS with a minor in marketing. My first job offer out of MBA School was from Silicon Graphics Inc., a pretty big company here in California at the time. It was to help the marketing team at SGI make better decisions about their marketing campaigns by finding a better way to query and access the website data. My very first project was to work on a web-based querying tool that would go back query a site based database and bring back results on sales, on different countries, and products, and campaigns, and things like that. Nine years ago there was no such thing available in the market. So, we built the very first one for SGI.
Interestingly, at that time there was already a standard business intelligent client-server based tool in the company, used by three hundred people. In the first week after we launched our web-based tool, roughly eighty percent of the people abandoned that for the web-based version. Even though the web-based version had fifty percent less data. It was just so much easier to use and better. And, that was a very important lesson about what worked.
Since then, I have always been in decision making. One way or the other, I have worked on really large ERP systems, and CRM systems, and data warehouses. For the last four years, I have been one hundred percent focused on the web and in making decisions on the web. I don’t think I am ever going to go back, I love the web.
Currently, I am the director for research and analytics for the web for a large Fortune 500 company. The platform we have built out supports both qualitative and quantitative analysis which makes it a little bit unique. Few other companies are doing this. It involves analyzing clickstream data, analyzing the back-end Oracle data, order data, as well as a group of folks who do surveys and lab usability studies, and various other qualitative types of analysis.
We merge this information with our quantitative analysis to make much, much richer decisions about what’s happening on our websites.
Eric Enge: Right, and in a Fortune 500 company small percentage improvements have large bottom line impacts.
Avinash Kaushik: Absolutely. I think Shop.org’s, Q3 benchmark for conversion rate was around 2% for most eCommerce websites. So, the interesting thing is that is if 2% of five million people coming to the website convert, you can imagine what 10% improvement would look like.
Eric Enge: Yes. What caused you to start the Occam’s Razor Blog?
Avinash Kaushik: I have been invited to speak at a bunch of conferences, I have done that for a while, and people said, you know, you should talk more about what you do. I was at a Frost & Sullivan event last year, where I spoke in Phoenix around this time last year in January. One of the participants there was Andy Beal.
Well, he is a very famous blogger of marketingpilgrim.com, and he just convinced me that what I needed to do to change my life is to start a blog. And, I did listen to Andy, so he gets the credit for getting me going. And, it indeed has turned out to be a very, cathartic process for me personally; I love the teaching aspect of it, I love the sharing aspect of it, and I have greatly benefited from people commenting on my blog. I don’t blog every day, I only blog a couple of times a week at most. So, there tend to be much longer posts. But, the interesting thing is that, so far I have written roughly a hundred and twenty thousand words in my blog posts in roughly eight months.
And, the readers of my blog have contributed one hundred thousand words, which means that the readers of the blog write almost as much as I do, which makes it an amazing process for me, because I get to hear from so many different people, all of their opinions on what I write, and it’s a very good learning experience for me. It turns out to be much harder and way more work than I had ever imagined, but it’s very fun.
Eric Enge: So the interactivity of the blog ends up giving back as much or more than you put out it seems.
Avinash Kaushik: My average post will get eight to twelve comments. Usually, they are excellent people, who are in the industry, practitioners, and marketers, and some CEO’s, and search analysts, and great people like Eric Peterson, Jim Sterne, or Jim Novo, or others. I think it makes it such a valuable forum for me as well as everybody else.
Eric Enge: I saw a post by you recently where you had a screenshot of the IndexTools Analytics application. And lo and behold you had a comment from Dennis Mortensen (IndexTools COO, who we interviewed here).
Avinash Kaushik: Yes, the vendors have been kind enough to allow me to implement their tools on my blog and try them, and so I have just had the greatest of luck working with some really great people.
As of today, I have about eight hundred odd subscribers to the feed of the blog, RSS subscribers; and many people who subscribe by email. There are people from every vendor on it, there are people from NASA, and the Social Security Administration, and the British Government. I am just amazed at all these people, who sign up to get these updates, it’s just gratifying.
Eric Enge: Next, I would like to explore a few areas of really high-value analytics, along the lines of the types of things you seem to enjoy putting in your blog.
For example, you recently posted about search funnels. I didn’t realize MSN offered such a service, and it’s actually an awesome tool.
Avinash Kaushik: One of the wonderful things about the web is that there are so many free things out there sitting and waiting for you to use. If you had suggested a few years ago that there would be so many great free tools on the web people would have laughed. I am just stunned by the amount of data that’s available for you for free.
You can use these tools for your own strategic advantage. It often happens to be more complex than you realize, but if you are able to put some small amount of effort into it, it truly can be a strategic differentiator for you.
Eric Enge: Can you expand on this a bit?
Avinash Kaushik: Sure. One of the threads that you will see throughout my blog is that most of us define web analytics in a very, very narrow way. For most of us, web analytics tends to refer to simply to the analysis of clickstream data.
When in reality, web analytics should be expanded to include analysis of any kind related to your website, yours or your competitor’s. It should include things like competitive analysis, and there are several specific examples of tools out there that you can go to analyze your competitor’s behavior and not just yours. That should be considered part of web analytics. Qualitative research should be web analytics. I have posted for example on my blog; I think this was probably six months ago, that one of the web analytics tools that I am personally excited about, a tool that was thought of more as a pure play web analytics tools was Instadia, by a company in Europe.
Eric Enge: The company that Omniture just acquired.
Avinash Kaushik: Today, as a matter of fact, they just did that today.
What is great about their tool is that Instadia has the ability to perform surveying on your website from the exact same tool that does basic web analytics, using the exact same JavaScript tag. You need to do no additional incremental implementation, and you don’t have to go buy anything additional. You can get extra contacts information from your customers with the same tool that you use to analyze clickstream data. This can help you understand your clicks so much better than you would otherwise have been able to.
We need to radically change how we think of web analytics, we should expand the definition to include; I call this the Trinity Strategy.
I have a post on the blog specifically explaining what the Trinity is, but essentially it’s the analysis of customer experience, customer behavior, and customer outcome, and that is web analytics. It’s not the analysis of only clickstream data. I am a big fan of doing competitive analysis. More than any other medium out there, people don’t realize how much they are at the mercy of their competitors. You could be doing every single thing right, and yet your competitor could define the destiny for your website. For example, let’s say you are H&R Block, and you sell tax software.
The interesting thing is that what customers expect on your website in terms of navigation and search are actually being dictated by Amazon.com, a company you don’t even compete with.
But, so many people use Amazon, and so many people expect certain things in certain places in Amazon, they expect them to be in the same places on different websites. That’s fascinating. You also need to be concerned about what your competitor is doing in terms of search keyword bidding, or what your competitor might be doing in terms of affiliate marketing. They could totally destroy you, even if you are trying to do a lot of right things strategically.
So on the web, it’s extremely important to do, depending on your size, depending on how much money you have to spend, there are many options to do competitive analysis, there are some that are free, there are some that are expensive.
But, whatever your place in the pecking order is, you should definitely consider doing competitive analysis, so that you can react much better. You need to look beyond web analytics tools, and there are many other tools, not just web analytics, that give you all kinds of great information.
Web analytics don’t provide insights into your competitors, and I believe that is a key thing. Everybody should be using tools like Hitwise, or Alexa, or MSN, or Google Trends, or all of these tools that I have blogged about (Choosing Competitive Intelligence tools and Best Practices in Competitive Intelligence). You are not too small to use any of these tools, you are really not too small, everybody should use them and they are easily available.
Eric Enge: Can you talk a little bit about the MSN search funnel tool in detail?
Avinash Kaushik: Search funnels are wonderful because they help us understand customer intent. Let’s say you type in your branded key term, such as the one that I mentioned in my blog post, Peachtree. Peachtree is very well known software, available to small businesses, in order to manage their accounting and finances. What a search funnel will help Peachtree understand is, what do people look for before they look for you?
Avinash Kaushik: And what do people look for after they have found you? And, it was really fascinating to see all of the competitors that people had searched for before they went to Peachtree. Or, the kinds of unbranded key terms that people search for when they were still in the prospecting phase, that they had not decided to buy. This can actually help Peachtree make much better decisions about what keywords to buy, and also understand the mindset of the shopping experience of their customers.
For Peachtree, it was fascinating that after they had finished searching for the word Peachtree, they searched for even more terms with the word Peachtree. This is a directional indicator that the web page that they landed on after searching for the word Peachtree wasn’t doing its job in convincing them to buy or giving them all of the information they were looking for. Many people were going to their direct competitor after looking for them, which should be just amazing because if I am Peachtree and I am running my website, and if my website is doing a really great job, there is no need for you to go to my competitor. I mean I know some people who will, but not a big chunk of them.
These are very important insights that a simple search funnel tells you: What do people search for before they search for me? What do people look for, after they have searched for my keywords? It’s a fascinating insight into customer intent and customer behavior.
Eric Enge: Can you talk a bit about AB testing and multivariate testing?
Avinash Kaushik: I am extremely passionate about testing and experimentation. This could be multivariate testing, this could be AB testing, of this could be experience testing. I have written about an experimentation and testing primer in my blog, it is about two pages and it will provide people a flying start on the topic of testing., Too often when we analyze the basic analytics data, especially after we analyze clickstream data, we are left with lots of questions about how to improve our customer experience on the website. For example, how can we help people purchase better, how can we help people find support answers better, how can we help people find the retail store closest to them so that they can buy our products and services.
In the past, it was so expensive to test all of the genius ideas that the designers might come up with that we imposed on our customers the experiences that we thought they wanted. What testing and experimentation allow any company to do is create experiences on its website that the customers want.
It is amazing how many times I am proven flat out wrong about what I think will absolutely work on our website. And, I am happy about this and want to learn these things.
In a speech I gave at the Emetrics Summit creating a decision making culture, I used this term called HiPPO, the Highest Paid Person in the Organization and typically when people make decisions about customer experiences on their website, the HiPPO wins.
Avinash Kaushik: Because you can’t go against the HiPPO. And now, what testing and experimentation allows us to do is say, okay, Mr. or Ms. HiPPO, we will take your idea, and as well as the idea of the lowly designer of the website or the web analytics analyst, put them all on the website, and let the customers vote about what works.
It is the best way for designing customer experiences, creating promotions and offers, and of radically changing your website, and it is severely underutilized. This is more critical on the web because there is a lot more ambiguity in the data than in other cases, such as CRM systems or ERP systems. In web analytics, clickstream data is often ambiguous. It is mostly anonymous unless we force people to log in, but it’s really, really hard to understand what would really work with a customer.
Testing and experimentation is a great way for you to know what works for your customers, and for them to vote on what they want. People often think that all you can do with testing and experimentation is to improve the navigation structure, or improve the landing page, not true. We have used testing and experimentation to do price sensitivity analysis. If you are a big company, and you have to figure out what’s the best price to sell, you may already do price sensitivity analysis in retail stores, maybe try different prices in different states, or you do price sensitivity analysis by sending email campaigns, you know isolated email campaigns by geography, or customer segments and things like that. These are very, very expensive to do.
On the web, you could run a very easy test to figure out what’s the best price that would work for maximum business goals very, very quickly, very, very cheaply, and very, very effectively.
It’s the best way to for a company to really kick it up a notch when it comes to creating an awesome website.
Eric Enge: One of the things you mentioned in a recent post was the idea of using Alexa as a traffic tool. So, it does seem to me that there are some accuracy problems with this tool, aren’t there?
Avinash Kaushik: It is important to point out that the way Alexa collects its data is from people who have installed their toolbar. If you do not install the toolbar Alexa gets no data from you.
As I pointed out, in my blog post that Alexa data is skewed data. You have to know that. For example, any websites that are rated greater than ninety thousand in rank probably has garbage data. You should not even look at it. And there are many, many other flaws. But let’s look at where it’s useful. Let’s say that you are with CNN.com.
CNN is a broadcast website, and its direct competitors potentially are NBC.com, FOXNews.com, CBSNews.com, and ABCNews.com, or whatever the URLs are. If you want to compare a peer group of websites, Alexa is a good way to compare traffic trends over a long period of time. Because you are comparing peer websites, you are controlling the bias. For example, let’s say only people who are white and have blond hair, and only know Unix, use Alexa.
You are still comparing the performance of your website CNN.com against all of these other peer websites controlling for that bias. It is, of course, much more difficult to compare the trend of Procter & Gamble and CNN on the same chart. Now, that is vague, because the kind of people who use those two websites would be drastically different. In this case, the bias would be a big concern.
But, comparing your sites against its peers can provide you with a directional read for your performance in that group of people who use Alexa, whoever they may be.
That’s a quick free tool. If you want to get a much better read, then my recommendation that I have made in my blog is that you should go and use Hitwise. Hitwise is a tool that is significantly more robust. It has very wide coverage. I really like the way it collects data, the ISP driven collection of data, rather than a panel-based collection of data, and it is not that expensive. If you are a good-sized business, it costs money, but it is well worth the price that you would pay.
Eric Enge: With Hitwise, you can get specific competitor keyword data for example.
Avinash Kaushik: Yes, I have a long post on competitive analysis. You can get search keyword data, you can get referrer data, and you can get affiliate marketing data. One unique thing about Hitwise is that it is integrated with the PRIZM database.
So, you can actually get demographic information, you can get zip code location of everybody who buys from your website. You can create targeted promotions and campaigns based on the PRIZM data. So, let’s say you want to target everybody, who makes more than ten million dollars a day, I am kidding obviously, that is over the age of sixty. Hitwise can give you back the websites that these people use, or where they are located, things like that and you could send campaigns to them, or you could do marketing deals with those websites. You can do a lot with Hitwise. You have to pay for it, but you can do a lot with it if your organization has the capacity to analyze and leverage the data.
Eric Enge: How does Web 2.0 and rich media affect the landscape?
Avinash Kaushik: The core essential problem is that the web is increasingly becoming a much more complex experience, using Flash and AJAX, and these types of things. As a result, the concept of a single pageview or a single action on a page is disappearing. Increasingly web experiences are becoming more like desktop software application experiences. In that sense, they are much more complex. For example, when you use Microsoft Word, it is really hard to capture data on what you are doing.
The current set of web analytics tools largely rely on the existence of a page, or a unique definition of a page in order to capture data accurately. For example, I was watching a video this morning of Stephen Colbert interviewing Bill O’Reilly.
I saw him on YouTube, and all YouTube knows is that I saw the page that had the video. They don’t know exactly how much of the video I watched, where did I do pause, rewind, forward, etc., because it is very hard to measure. So, the challenge of measuring rich Media is that they are significantly more complex experiences.
The interesting thing about Web2.0 is that you have to think upfront about what is it that you are trying to do with your experience? Figure out what your key business events are that are occurring in that experience, and plan to capture those specific business events, so, later on, you can analyze them. Because there isn’t such a thing as taking a standard tag, you have to do much more preplanning, and integrate analysis and data capture into your development process. This is what is new, different, and challenging. The fact, that a business analyst or a marketer has to think upfront about why the particular experience exists, and what are the core interaction points of a customer in that experience is new.
Because most of us are not used to thinking upfront. We are used to throwing the tag out there, and then analyzing whatever it sends out, but this is not going to work in the new world. The other facet, of Web2.0, is that it is using all of these new and emerging methodologies for accessing content, such as mashups, like those on Zillow.com where they are using data from Google, and MSN Live Maps, and many, many, many different sources in one web page. When you have all of this content available in so many different places, what is a page?
The other thing is new ways of consuming content, such as RSS. At the moment, the way a website knows that it had a visitor, is that you visit the page, view the page, execute the JavaScript tag, and you have data. Well, that’s out of the door, because I have as RSS reader with forty blogs and websites in it. I never visit the websites, and yet I am consuming the content, and the website is blind to me. How do you measure that?
At the moment there is only one tool, where you can put a customized tag that would measure the impact of RSS in an integrated way in your web analytics tools, but there is not a single tool that does that except that one, which is from Visual Sciences. They have a methodology, where you can tag your RSS feed, and get integrated data into your web analytics application. (Reference Avinash’s posts about Measuring Rich Media.
But, none of the other guys have it, because this is not even on their radar yet. And so, that poses a big challenge in terms of Web2.0. Richer experiences, new content consumption methodologies, both pose challenges of different kinds. Some of the vendors are making progress, but much more needs to be done. For example, on my blog, I have data from FeedBurner, and I have data from my web analytics tools, but I don’t have an integrated view of both sets of data. I should not have to go to two different places. Even worse, FeedBurner and web analytics tools define a visitor in a completely different way.
Eric Enge: Right. I suppose that’s true a lot of these web tool companies were started by someone in a garage, and they just mushroomed into something bigger, and this wasn’t an environment for making a large upfront investment in tracking.
Avinash Kaushik: Right. In the past companies were in control of customer experiences. They could create any website, any look and feel, anything that they wanted, but increasingly companies have lost control of creating customer experiences. And, it is actually customers who are creating their own customer experiences, and this is a fundamental problem for any web analytics tool, and for many companies about how do they measure data, because the customers are creating their own experiences they want to see, the data the way they want to see it, and in that scenario where I can create my own custom experience, how do you measure success? It is really hard to do. And it is going to be a challenge that more and more analytics tools will work hard towards solving.
Eric Enge: Another thing I read about recently was Eric Peterson’s notion of an engagement metric. Can you comment on that?
Avinash Kaushik: Sure. You know that Eric is obviously a leader in the industry. We are all following the trail that Eric has blazed. He is just an awesome guy and a really great thinker. And, in terms of the specific post that you are referring for engagement, I think Eric’s initial proposal for the methodology is a very good one, and it does extend the conversation in terms of what it is possible for us to measure, because Eric obviously has access to some pretty good tools that allow for deeper analysis. But my preference is to ask a random sampling of people or every single person who comes to a website, are you engaged, here is my definition of engagement, do you like this site or product, are you going to recommend it, or whatever is the case.
My preference is to capture the actual vote of the customer to tell us if we are creating customer experiences that are engaging, and that is resulting in our customers finding the information that they are looking for, not the information we are shoving in their face. So, I think Eric is exactly on the right path, and I think we will all evolve towards that. But, on our website, for example, we measure customer satisfaction as a metric, and we measure your likelihood to recommend our website/products to others. And to me, those are great proxies of “engagement” in the sense that we are creating experiences on our website that are engaging and solve customer problems and allow our customers to complete the task that they are looking for to accomplish. I am a bit more in the camp of measuring it that way, does this make sense?
Eric Enge: Yes, this gets back to the survey capabilities of Instadia, right?
Avinash Kaushik: Yes.
Eric Enge: The ability to have integrated survey capability is helpful here. Of course, there can be a bias here too, but if the percentages of responses that are positive or negative go up or down over time, that’s meaningful.
Avinash Kaushik: Yes. The methodology we use is the University of Michigan ACSI index. The American Customer Satisfaction Index. The way we show the survey to the customer uses things called loyalty factor and a standard sampling percent. That it is a way for us to show the survey to certain types of people who can answer the question in a pertinent way. It is not a survey that shows up randomly to everybody who comes to the website, it doesn’t work that way. There are some ways in which we can control sample bias. But even if there is a bias, if you watch the trends over time for that sample and control for the bias, you will be fine.
For example, our success metrics for our website are revenue, completion rate and let’s say customer satisfaction. I think all three of them together measure engagement. But, are we making money, are we helping our customers complete their task? No matter why they come, we measure task completion rate for every task that people come to our website for. So for example, on an eCommerce website, if people come to look for tax-report, we measure if the site is doing a good job of helping those customers find it. So, are you making money, are we helping our customers complete their task on the website, and will they recommend us to others?
This differs greatly from site to site. For example, on my personal blog, I know that I am creating an engaging experience, because of the number of people who are submitting comments on the blog. On average most blogs posts will receive one comment or a few comments. So I benchmarked that and I said, okay so on my blog I have roughly eight to twelve comments a post and that is customers telling me that it’s an engaging experience because they are taking the time and trouble to write something. That their voice is represented. So, there are different ways of representing the voice of the customer. But, I always prefer to have the voice of the customer to come out as clearly and directly as possible from them.
Eric Enge: So one common recommendation for a business that is starting to get engage with analytics is that they should start by defining what their business objectives are with the website. In addition, define the few key things that you think that will really help you understand that in easy fashion, then go figure out what analytics packages might meet those needs, and get the implementation put together and then expect to do a process of continuous improvement and tweaking item from the item.
Avinash Kaushik: That’s a fairly accurate overview of what most people do.
Eric Enge: Can you expand upon that discussion and let’s talk a little bit about the process for taking the best advantage of analytics?
Avinash Kaushik: I could talk about this for a long time. So, let me touch on a couple of things very quickly. You are getting into the heart and the soul of analytics here. I am very passionate about it too. The first part is really very fundamental and also I think a very critical flaw of how people go about either identifying and then picking a tool or starting an analytics program. This is a very important thing to state. The process that you described is a very, very sub-optimal one.
I have a post on my blog about a radical alternative to picking a web analytics tool. The core recommendation of it is that, do not go out and ask people for their business requirements, do not set your objectives, do not do any of these things. Because if you do these things, for the most part, it is highly likely that you will end up picking the most expensive web analytics tools out there, and you will not be able to have a lot of success with it. The reason is that you have no idea what you are getting into. It makes no sense to define the requirements up front when you really don’t what you are getting into. Because human beings are risk averse, they will ask you for the earth and the moon, and hence you will go out and buy the vendor that will cost you $500,000 a year.
Because the guy will give you everything. But when you get everything, it turns out that everything you asked for was wrong.
My recommendation is, the first thing you should do if you don’t have a tool, is going implement Google Analytics or if you have a log file based environment, go get ClickTracks Appetizer; both of these tools are one hundred percent free. The paradigm should be let me go, sign up for a free tool, five minutes to get the tool, ten minutes to put it on the site, and half an hour later data shows up. Then start playing with the data. The best way for you to figure out what you want is to actually start getting the data.
You will figure out all of the things that are right about the data, you will figure out all of the things that are wrong about the data; and more importantly, you will figure out that your website is not coughing up all of the data it needs. You will learn that your web analytics tool is implemented wrong, and your marketers will figure out what kind of rich and wonderful data that a real web analytics tool will provide. This is not fake stuff they have read on somebody’s blog. This is not crazy things told to you be some web analytics vendor who comes to your office and makes a presentation. This is data on your website, and about your website that you are looking at. My recommendation is to pick a free tool and implement it.
If you want a JavaScript-based tool, go with Google Analytics; it is a very good free tool, it has eighty-four reports I believe, when I last looked at it, that’s a lot.
Avinash Kaushik: Or you could use ClickTracks Appetizer if you prefer log file-based data. Just deploy ClickTracks Appetizer. You can do lots of great analysis with it. But the first thing you should do is, implement one of these free tools and start experiencing the joys and frustrations of web analytics.
In my blog post about this I lay out a series of steps that you need to go through; and at the end of step seven or step eight, I forget exactly which one, you could very well go out and collect your requirements from everyone in the company, and you can go out and pick a tool. But now you would have done it from a place of education, rather than a place of ignorance. And, that will lead to a radically different choice by your company. It could very well turn out that, your company is only ready to leverage Google Analytics in which case; you have just saved $100,000.
Eric Enge: The whole notion is, learn on the cheap.
Avinash Kaushik: Exactly, and become smarter. It may turn out that you will learn that Google Analytics does everything for me, but it doesn’t do these one thing or two things, or ClickTracks does all these things, but it doesn’t do these five things. Now you know what to ask the vendor, right?
Eric Enge: Second thing you can decide is, if it is only these two things you are missing, what’s it worth?
Avinash Kaushik: Oh, brilliant, exactly true.
Eric Enge: If it’s going to cost you $250,000 to get it, maybe you will live without it.
Avinash Kaushik: Yes, and again it’s a choice you can make. With this approach, you will make an educated choice. The problem with the model that you described earlier, which by the way is rather prevalent in the world, is that you are making an uneducated choice, and that is suboptimal. And this is especially true for web analytics, because, you are collecting anonymous clicks, and you have all these limitations, people don’t accept cookies in JavaScript, and your URL strings are too long, and you are not setting the right cookie, etc. You have all this complexity. People don’t realize how complex this is. So, you can learn on the cheap, exactly as you said.
Eric Enge: Thank you so much. It’s been a great conversation.
Avinash Kaushik: Thank you, my pleasure!
