Jim Sterne
Recently, I had the pleasure of interviewing Jim Sterne, President of Target Marketing. Jim is recognized as one of the leaders in the world of web analytics and is also President of the Web Analytics Association.
As for the rest of his background, Jim has spent more than 20 years selling and marketing technical products. He began his career as a professional explainer, helping people understand Visicalc at a time when “personal computer” was an oxymoron. He successfully described sales order processing systems to people using hand-cranked tabulators. He was at the forefront of the Computer Aided Software Engineering (CASE) tool revolution and has clarified the salient points of object-oriented programming to software engineers across the country.
The following is a transcript of our discussion:
Interview Transcript
Eric Enge: Can we start with a brief background of yourself and Target Marketing?
Jim Sterne: I was in sales and marketing of technology products, business computers, and software development tools. I was a marketing consultant when I became aware of the Internet in 1993. I have been fascinated with it ever since I have been writing books about it; five books in total. The latest is Web Metrics. And, from that launched the Emetrics Summit and the Web Analytics Association. This year, the Emetrics Summit is in San Francisco, London, Washington DC, and Dusseldorf. The Web Analytics Association is about three years old now with a thousand members, all doing their best to increase the knowledge of web analytics around the world; what it takes, what it is, and how to do it well. I have remained a public speaker on the subject and a consultant on the subject.
Eric Enge: What advice would you give to a company that has just decided to implement web analytics, or is rethinking of their analytics strategy from the ground up?
Jim Sterne: The very most important thing that any company needs to do is, answer the question “why do you have a website?” And by that, I mean how do you define success? The biggest mistake that people make is, “oh yeah we want to know what’s going on a website, everybody is getting web analytics tools, let’s get one”.
They put it in place and they start cranking out reports, and they fool with the reports trying to figure out what they mean. And that’s backward. What they need to do instead, is figure out what constitutes success; determine which metrics will reveal whether or not they are getting closer or further away from success, and then start getting involved in continuous improvement. Continuous improvement means making a little change, see if it has an impact, positive or negative, make another little change and so forth.
There was a great presentation at Emetrics Summit in London a couple of years ago about all the horrible mistakes one company made, and the first one was just mind-bogglingly dumb. They instrumented their entire website with web analytics and turned it on and the whole system, the whole website crashed for about a day and a half.
So, they back backed out and then did it again. And, they crashed again. So, they learned to do it on a test server before you put it into a production build server. But then, they said okay we have everything working finally and the reports started coming out and we said, “what a horrible problem we have on our website. We got to fix this, this, this, and this”. And, they started making a bunch of changes, and the conversion went down, and the traffic went down, and customer satisfaction went down, and they couldn’t figure out what was wrong until they realized that they hadn’t quite implemented the web analytics tool right. There were some mistakes. They had missed some pages, and they were double counting some pages. So the lesson there was, whatever you do, don’t believe the first reports that come out of it. You are not going to have implemented it correctly.
Eric Enge: Can you talk about the notion of actionable measurement?
Jim Sterne: People look at the reports that come out of web analytics tools, expecting them to have all the answers. And then, that doesn’t happen. But, the reports that do come out are fascinating. They tell you things you didn’t know, they express things in ways you haven’t thought about before, and they are really, really interesting, which turns out to be incredibly useless.
Unless you can take action on the reports that come out of the system data that you get is a total waste of time.
One of the common things that I hear from people who are responsible for web analytics is that they are being overwhelmed by people with ad-hoc questions. Now, I am a big fan of ad-hoc questions. But, instead of reading the reports that come out, you should be turning to the web analytics tool and asking questions. How do I improve this? What happens if I do that? Those are great. But, when someone asks you an ad-hoc question the correct response is to ask them why they want to know, and what actions are you going to take based on it. If I tell you the answer is seven, will you take action that is different than if I tell you the answer is twelve? And, do you know what the range is against which you will or will not take action? And what action will you take? If they can answer all those questions, give them the report. But if not, then they are just curious. Unless it is connected to some sort of action it is a waste of time.
Eric Enge: And I suppose that they may have asked the wrong initial question, one that won’t give them the real answer they are looking for.
Jim Sterne: Absolutely, and chances are excellent that they won’t understand the fine nuances of why one number is better than another for whatever purpose. There is a great quote from Gary Beberman, Director of Technical Research, at www.macys.com from another Emetrics Summit. He said, “sometimes you just have to tell them what the numbers are, tell them what the numbers mean, and tell them what to do about it”, because marketing people are not raised in an IT, and they don’t know what the different numbers mean.
Somebody has to be the connection between IT and marketing; somebody has to be able to understand the importance of the value and the differences in the data and the business application of those nuances. That’s the big challenge.
Eric Enge: You mean somebody like an analyst, who can really get to the bottom of what the numbers mean.
Jim Sterne: We have four different kinds of people that we are going to bring together to try to make all of this web analytics stuff work. First, the technician, the guy who can implement and tag the pages, and make sure the reports get generated, and data flows where it should flow, and that’s the systems work. Then, the IT guy. Then, we need the analyst, who is the number cruncher, the guy who loves statistics, the woman who is just fascinated by data diving. Then we need the designer, the website contents/design manager, who understands the value of experimentation, if I change this will it do that, if I change it this way, if I change the button from blue to green will it have an impact to find how much it wasn’t required.
Then we need the business people. What are we trying to accomplish here? Well, we are trying to sell widgets. We are trying to sell ballpoint pens. Do you want to sell more ballpoint pens at a higher profit, do you want to lower cost, do you want to open up new territories, what’s the goal du jour that we are trying to meet or beat here, and that will direct what all of those other people should be doing? We need to get those four people in the same room, speaking the same language and working together. It’s a big challenge.
Eric Enge: I would imagine that a significant number of companies don’t see the complete picture.
Jim Sterne: Or don’t have that kind of talent on hand.
Eric Enge: Isn’t there also the issue that different people on the team have different needs with regard to analytics data?
Jim Sterne: I call that web analytics on a need-to-know basis. The higher I go up the chain, the less data I want to see, because I don’t need to know all the little, tiny fine details of what’s going on at the bottom. What I need is, in general, are we getting enough traffic, are they moving through each of the customer processes appropriately, and is our conversion working? And, if I can get a dashboard that says, all those things are sort of in the green, here is the one this in the orange, oh, oh here is one that’s in the red, now I will make a phone call, or I will click on that and drill down, and see why it has that turned red. But the higher up in the organization I am, the less data I want to see.
Eric Enge: Right. Too much data just doesn’t have the right impact.
Jim Sterne: Yeah, I need to get high-level support, I have to have a sponsor, who understands what the goals are, and why the budget should be spent, and determine whether or not the money should be spent. And then, I need to keep an eye on whether or not we obtained a return on value. So, if I am going to take on a web analytics project, it is not going to work solely because of a bottom-up groundswell, there has to be somebody up high in the organization, who can say, “we are going to spend money on this”.
Eric Enge: Right, I think it was one of your books actually, where I first encountered the phrase “ROI of measuring ROI”.
Jim Sterne: Web analytics tools are expensive. One of my favorite quotes is Avinash Kaushik from Intuit, someone who is really respected for his work in analytics.
He likes to say that for every $10 that you spend on web analytics tools, you should spend $90 on the people to make it work. That’s a huge commitment, that takes serious understanding from senior management, and it takes somebody to say, “if we are going to spend a million dollars, we need to understand what the value is going to be to us”.
It gets tricky because there is nobody in the organization that can understand the whole picture. The person doesn’t need to know the deep dark secrets down below and doesn’t need to get involved in what percentage of my visitors are deleting their cookies, and how that throws off the numbers. All he needs to know is, if we buy this tool, and we optimize our website by so much, we will get some return.
Eric Enge: What about these free tools, and the very inexpensive tools? For example, Google Analytics has a lot of people using it. And, now Microsoft has announced that they are going to be releasing something similar in the May timeframe.
Jim Sterne: Google Analytics and Microsoft can’t touch the high-end tools, such as Omniture, WebSideStory, WebTrends, and Coremetrics. These companies have been focused on this area for a dozen years, they have implemented them in giant enterprises over and over again, they know what the problems are, and they know how to help people out. What makes Google fabulous is its scalability, and that means that they have figured out ways to deliver services to people without having to have thousands of technical support people behind it.
They have a great frequently asked questions document. You have a tool that does some very basic things very well, but if you want to get deep into the data, it can’t do it. For instance, if I am Wal-Mart, it may be that increasing conversion on one of my products in sales online by a quarter of a percent, is going to result in a million dollars to the bottom line, because my volume is so huge. That’s really worth doing, and it is certainly worth spending a lot of money on web analytics tool, and web analytics team to get that quarter of percent improvement because a million bucks mean something.
The tools that Google has, and that Microsoft is going to make available are not intended to go down to that depth of knowledge, they are not intended to hook up to dynamic content management systems to personalize an experience based on behavior. They are intended to give you very specific information on which of your keywords are resulting in the most income to your business. It’s a fabulous tool to help Google extract more dollars out of advertisements. Google proves to you, that the keywords you are buying have value and that encourages you to buy more. Brilliant, everybody wins.
The only people, who are not happy about Google Analytics are the low-end web analytics tool vendors, who are offering the same quality of the tool, but at a price instead of for free.
Eric Enge: Who is it that you see as the low-end vendors in this picture?
Jim Sterne: There are a whole bunch that come and go. The one that has escaped this fate is ClickTracks. ClickTracks started its life as a product that told you which one of these keywords is actually selling something, and when John Marshall created that product, that was the question he was trying to answer. Well, as soon as Google said, “we are buying Urchin, and we are going to make it available for free”, he, like the very intelligent marketing person he is said to himself, “okay, what are we going to do to differentiate ourselves from Google”? He chose to address click fraud, which is something that Google doesn’t want to go after the whole wholeheartedly, because it diminishes the amount of money that they make. So, that was a brilliant move. ClickTracks started out as a very inexpensive, very fundamental tool, but has been around long enough, and has had enough technical investment, that it is now quite robust. It’s not up in the top tier with Omniture and WebSideStory, but it is definitely a real serious consideration if you are a medium-sized business. For a small business, free is the right choice.
Eric Enge: It sounds like you see three major segments of the market. These are free products, such as Google and soon Microsoft, and then some amount of middle ground which includes companies like ClickTracks that have found a niche, and the enterprise products.
Jim Sterne: Right.
Eric Enge: Where would you put IndexTools?
Jim Sterne: IndexTools is another one that has had enough time to make their systems more robust. IndexTools has also taken the tack of tying in with keyword bid management. So, it’s not just what the people do, but how does their behavior differ based on which keywords they clicked on, and that’s huge.
Eric Enge: Okay, great. Another aspect of this is integrating with offline systems, call center data, and those sorts of things. With regard to these external systems, how do you see the current state of the analytics market, and how do you think it is going to evolve in the near future?
Jim Sterne: Right now it is one of the biggest problems. Data integration, in general, is a difficult problem to solve. If you have multiple web sites with different brands for instance, they are probably using different tools. Maybe they are even using the same tools, but the tools have been implemented differently. All of that’s going to create headaches and make it difficult to bring the data together. Now, we talk about the multiple touch points. I have got people coming into my store. I have got them calling the call center. They are coming to the website. They are e-mailing me. They are watching my ads on TV, and somehow I have got to bring all that together, and it’s a big problem to solve, but it is the long term problem that all of the vendors recognize is where the money is going to be. I want a 3600 view of touch points, and it’s a challenge.
Eric Enge: Right, isn’t one of the challenges that the data comes out of your web analytics packages in enormous quantity, and most of it is not directly pertinent to the problems that you are trying to solve, versus the data from the other systems which is much more compact in nature?
Jim Sterne: Well, that is one of the problems of data integration. That we have more data than we know what to do with has pretty much always been the case.
But, data storage is becoming less expensive, and data analysis tools are becoming more sophisticated. We will always be able to capture more data than we can manage, that’s a given. Can we take samples of data from our website? Sure. Can we identify specific individuals whose behavior we want to track? Sure. Can we then integrate that with call center data that we have? Sure, but there are also companies out there, that are listening to the calls, recording the calls, and analyzing the voice stress, analyzing the language used. That’s a huge amount of data as well, but tremendously valuable when brought together with online behavior and in-store behavior. There really is no end to it. So you need to decide which bits of information are really the most revealing and the most valuable, so that you can make solid business decisions, and can take action.
Eric Enge: I have heard that one of the largest sources of error in analytics is the accuracy and problems with implementing the Javascript. Does that make sense?
Jim Sterne: Well Eric, there are a number of things to discuss here. First of all, web analytics numbers are not precise. They are not exact. How many people came to your website? Well, there are cache files, there are proxy servers, people are deleting cookies, and more. Then why bother measuring it? Because, I am going to use this large yardstick consistently, and as a result, I am going to get good comparative numbers day over day over day. So, the precision might not be what I would like. But, the difference between how many came yesterday and how many came today because I am using the same imprecise tool, is still very meaningful. So, if I make a change to my website and it improves visits or it improves conversion that is a true result. So, the question about precision is a long process. The day you implement a tool you may well get bad data. So, verify everything that you possibly can. But eventually, you are going to reach the point of diminishing returns, and say, I can tweak this until the end of time and you will only 0.3% more accurate. When you reach a point like that it’s time to say that’s as good as it’s going to get.
Now, in the marketing world, this is so much more accurate than any other media we can measure. How many people saw your ad on TV? What is the impact of your ad in a newspaper? Well unless they call the 800 number, you are never going to know.
Eric Enge: Can you talk a little bit about how the WAA was formed?
Jim Sterne: There were three of us. Bryan Eisenberg who is the current chairman and Andrew Edwards from Technology Leaders called me and said, “okay we have discussed it at the Emetrics Summit. Now it is time to actually create this association. And you are going to be the first president, right?”
Eric Enge: Some vendors take an approach built around data warehousing versus a custom data store. Do you have any thoughts on what that does in terms of capability and complexity?
Jim Sterne: The web analytics data by itself is useful. But, it is more useful combined with other information from eCommerce systems,
CRM systems, general sales contact management systems, sales force automation systems, and so on. The more you integrate your marketing data, the more valuable it becomes. So, what’s the best way to do that? Some organizations are saying, “the data repository in this web analytics tool is robust enough that we are going to create pipelines from our other systems and dump that data into the web analytics tool, and it will be able to slice and dice all of these together”. Others say, “our data warehouse tool is so powerful, let’s take, let’s cherry-pick the data we want out of the web analytics site and put it into our joined data warehouse and use it there”.
The majority of companies have not made a big decision like that. And, the result is that for each type of question that comes up, they are handcrafting a solution, i.e. we are going to take this specific data out of the web analytics tool and put it into this specific data mart, in order to answer this specific question. There end up being many options. In an ideal world, there is a really powerful heavy lifting data analysis tool into which all data flows; and from which all questions get answered. But, this is actually the heavy lifting; this is the single most difficult problem that I continue to hear from companies all over, that the data integration is a nightmare.
So then the classic problem is how much money do I spend in order to optimize what I am doing in order to measure everything? What is the cost of measurement and what is the point of diminishing return?
Eric Enge: Can you comment on the relative merits of a log file based analytics approach and a Javascript based analytics approach?
Jim Sterne: The best practice is that you use both because you get different data out of the different types of capture. Log files are still the repository for error messages; 404s, time outs, unauthorized access; the log files are where that is scored and Javascript won’t give you that.
And then, the Javascript is there to do the cache busting and to help get around the proxy servers. But most bots don’t execute Javascript, so if you want to know what they’re up to on your site, you need to go back to the log files. No one technology provides all of the answers. We all started with log file analysis because that was what was available.
Eric Enge: Can you talk about the major sources of error in analytics?
Jim Sterne: Web analytic data is not precise, but as long as the inaccuracies are consistent, then the delta is true. So, let me explain all that. So, if I have 10% more people showing up to my website this month than last month, that’s a good measurement. If I changed the button from blue to green and I got a 3% increase in conversion, that’s also correct. The exact accuracy of how many people showed up, I don’t know. How many page views were there really? Well, there are proxy servers and cache file issues, and Javascript doesn’t always resolve those, because if somebody clicks on a link, as soon as the page loads, but before the Javascript executes, then I could miss a click.
As long as I am using the same method consistently, then the difference between yesterday and today, or the difference between version A and version B, is correct and extremely useful.
Eric Enge: What about Latent Conversion Tracking?
Jim Sterne: Yeah, that’s tricky. How do you deal with latent conversion tracking if people are deleting cookies, and you don’t know that they came and visited you six times and then purchased? You only know that this visitor showed up this one time. My favorite example is, shopping for a baby carriage. You look online and you see what’s available, and then you go to the store and you see what they look like in person and kick the tires. Then you go back to the Internet to see what the different features are, and what the different price ranges are. Then you go to the store to find exactly the model and if it really will work and that it really is good. Then you go to the Internet and you find the best price, but then you go back to the store because you want it right now. So, how does the web analytics tool explain that behavior, especially if we’ve got cookie deletion? You know in one case they show up, they look around for a while, they come back later, and they look at different things into different depth. They come back another time and it looks like they are about to buy, but they don’t, they disappear. It’s a big challenge.
Eric Enge: So it seems like one of the things that might be smart for a business to do then is just try to size their problem, is to survey people who purchased products, but according to their analytics didn’t come through their paid marketing campaign. Get a feeling for what percentage of the people, in fact, did at some point see an ad and click on it?
Jim Sterne: That’s one approach, yeah.
Eric Enge: It’s sort of a sizing the side of the barn kind of approach. Even without the offline visit examples that you used, you can have someone who comes through a paid click, comes back on a free click and buys. And, it’s a simple two-step and most packages will credit that sale to the second free click.
Jim Sterne: That brings up the whole issue of sale crediting which is also hard. Actually impossible. Somebody drives down the street and sees a billboard; then they see an ad in the newspaper. Then they hear an ad on the radio and they talk to their neighbor about it. Then, they go online and they see a banner ad and then click on the banner ad and they buy the product. Which should get the credit for making the sale? It’s an unanswerable question.
Eric Enge: Correct, but it’s of great interest just to even get a gut feeling for it. You have these businesses that take this wonderful data they have, because of all its precision “I am losing money on this advertising campaign I am doing over here. I’ll turn it off”. And meanwhile, the total margin of the business drops, because, they didn’t take into account that that campaign was generating latent conversion.
Jim Sterne: Exactly. Then we go back to the idea of surveys. How many times did you visit our website before you purchased?
Eric Enge: What are the biggest challenges you see for the analytics industry going into 2007?
Jim Sterne: Finding people that can do the job. There is a huge shortage of people who have any experience because it’s this weird combination of being technology savvy, being mathematically capable, and being overwhelmingly curious, as a character trait. And, understanding what it is that’s important that makes businesses tick.
So companies are putting together teams; one person from IT, one guy who is an analyst, one guy who is a designer, and another who is a business person. And, it’s hard to find enough people, and if you can find somebody who knows all of those things, make them CEO.
Eric Enge: I very much appreciate your taking the time to speak with me; it was a great conversation.
Jim Sterne: I enjoyed it too. Thanks for the questions.
