Part 2 – 2007 Web Analytics Shootout – Final Report

This is the second part of the Final Report of the 2007 Web Analytics Shoot Out. You can see Part 1 of the Report here.

Section 6: JavaScript Placement Matters

One of the things I heard from the vendors was a concern about where their JavaScript was placed on the page, with respect to that of the other vendors. The notion is that people may arrive at a web page and leave the page by clicking on something or leaving the site before the JavaScript finished executing.
Obviously, the longer it takes a page to load, the worse this problem would be. The further down the page the JavaScript is placed will affect that code’s chance of executing. In addition, if there is a problem with one vendor’s server, vendors further down in the stack of JavaScript would never get to execute.
So after hearing this concern a number of times, we decided to see just how much it really mattered. To test this, we took City Town Info and shuffled the order of the JavaScript. The original order was as follows:

1. IndexTools
2. Clicktracks
3. Google Analytics
4. HBX Analytics
5. Affinium NetInsight

In the second stage of the test, the order became:

1. HBX Analytics
2. Affinium NetInsight
3. IndexTools
4. Clicktracks
5. Google Analytics

Here is the data we received using the original JavaScript ordering:

Here is the data we received with the revised JavaScript ordering:

Clearly, the JavaScript execution order is a factor in the test. We decided to summarize the data by comparing the placement of the JavaScript with the ranking of the 3 traffic metrics, as follows:

Here is the same table, but showing the results after the JavaScript order changed:

A few observations emerge from this:

1. In the first set, the order of the results almost exactly mirrored the order of the JavaScript, with the notable exception of Unica’s Affinium Net Insight, which finished in the middle of the 3 packages, even though it was in the 5th position.

2. In the second set, Affinium Net Insight scored first from the second position, and Visual Sciences HBX Analytics scored third, even though it was in the first position.

3. The clear indication is that JavaScript placement matters. Bear in mind that in Section 4 we showed two different sets of results for City Town Info that were run over different and non-overlapping time intervals, and the differences in the relative data comparisons were small. So the differences in numbers due to JavaScript placement, as documented in this section, are significant.

It’s important to focus on this in the right way. The critical issue affecting how a package counts visits is the delay between when the page begins to load until the JavaScript finishes executing. It’s the time delay that matters because it provides users with increasing opportunities to see what they want and to click on the next link or to simply leave the site.
We have no reason to suspect any other interaction issues between the various packages.

1. From this data, we can hypothesize, but cannot prove, that HBX tends to count a bit lower than other packages on sites like CTI, and that Unica Affinium NetInsight tends to count a bit higher than other packages on sites like CTI. Clearly, the JavaScript execution order is not the only factor at work here in the differences between our packages.

2. Lastly, we should point out that data lost due to JavaScript placement should be considered an error, not a variance. By this we mean, it represents truly lost data.

For example, if a user arrives on your site’s home page, and then clicks on a link to go to another page on the site before the JavaScript for your analytics package can execute, your analytics will not count that visitor as having landed on the home page.
Most likely, it will count the next page as the landing page, and consider the home page of your site as the initial referrer. If the user had, in fact, come from a search engine you will likely have lost information on the keyword used as well.
In addition, if a user arrives on your home page, and then leaves (bounces) before the JavaScript runs, the analytics software will not count that user in the bounce rate calculations for your site.

Tracking pixel test

Next, we did an additional test. Working together with IndexTools, we implemented a tracking pixel at the top of the page code on City Town Info. This tracking pixel was placed just after the tag, and the remainder of the JavaScript was left in place at the other end of the file, immediately before the tag.

When you look at this data the Visitors numbers jump out at you. It’s a startling 18% higher for the tracking pixel at the top of the source code than the JavaScript at the bottom of the source code. However, it turns out that visitor counting using a tracking pixel has some inherent inaccuracies that result in a bit of double counting.
The short explanation of what is transpiring is that the analytics server is undertaking multiple communications with the user’s machine to determine if it can set a cookie. The server has a timeout (in IndexTool’s case the timeout used is 5 seconds), after which it assumes a cookie can’t be set, and then moves on to performing IP and User Agent based tracking.
Some of the time, the user’s computer actually does allow a cookie, and it comes back after 5 seconds, and says OK. When this happens, the software in fact still sets the cookie, and the user gets double counted.
In any event, this error affects only the visitor count in our data above, and it means we can’t use that data to make our determination of the difference between being at the top of the page and the bottom. However, we can still use the unique visitor numbers to get a flavor as to how much the placement affected our counting.
This turns out to be about 1.6% for page views and 4.3% for unique visitors. What is driving this difference? Ultimately, it is the time delay between when the tracking pixel finishes executing, and when the Javascript finishes executing. Some visitors will leave the site, or click on a link and move onto the next page before the JavaScript finishes executing.
In a few sample data points we determined that the average delay between the completed execution of the tracking pixel and the JavaScript was about 1.4 seconds. The site on which we ran the test has a pretty fast execution time, even with all the JavaScript that was present on the pages.
Slower loading sites are likely to have a more significant impact due to the positioning of the JavaScript at the top or the bottom of the page. Some industry studies suggest that users make their decision about your web page in roughly 3 seconds. On our test site, the JavaScript generally finished executing at around 2.5 seconds.
This was still enough to be a factor in counting, but a relatively small factor as noted above. I would conjecture that conducting this test on a site where the JavaScript did not finish executing until 4 or 5 seconds would show even more dramatic results.

Google Analytics and Tool Parts Direct

We saw earlier in Section 4 that Google Analytics reports significantly higher numbers than the other packages that were run at the same time on the site. We also stated our belief that this was due to the Google Analytics JavaScript being placed in the HTML header of that site, while the JavaScript for the other packages was installed just before the tag in the HTML.
We measured the time from when Google Analytics JavaScript finished executing to when the next package’s (IndexTools) JavaScript finished executing, and we saw that the average time between the two was 3.3 seconds.
It’s worth looking at this in a bit more detail, to see if we can estimate the impact of this additional delay of 3.3 seconds, to see how it compares to our 1.4-second delay that we measured and discussed above. To look at how Google Analytics fares in general compared to other sites in the test, let’s look at the following table:
Google Analytics % of Other Package Averages (Visits)

What this table shows is the number of Google Analytics visits divided by the average of the other analytics packages we ran on the same site. For example, the AMD number of 124.99% represents the Google Analytics number of visits, divided by the average of the other five packages installed on the AMD site.
Google Analytics consistently tended to be in the middle of the pack on HP, CTI, and CTI JS Changed. These sites derive most of their traffic from organic search. On AMD and TPD the traffic is largely PPC based.
To be conservative in our speculation, let’s use AMD as an indicator of what to expect from Google Analytics on a given site. We can then further speculate that the delay of 3.3 seconds results in a loss of 12.2% of the data (we divided the TPD number by the AMD number from the above chart to come up with that number).
This is all pure speculation. However, it does stand to reason that a greater delay would provide more users with the opportunity to leave the site prior to the JavaScript executing.

JavaScript Execution Time

Since JavaScript placement does seem to have an impact, we decided to measure the exact execution time of the JavaScript of the respective vendors. In theory, page placement is a factor because the longer it takes for your analytics JavaScript to begin and finish executing, the more likely it is that a user will click on some link they see and leave the page they are on before the JavaScript can finish.
As a result, analytics software with a long execution time would be more prone to these types of errors. The following measurements were taken using a tool known as HTTP Watch (http://www.httpwatch.com/).
For each analytics package, we measured the JavaScript execution time on 3 of the sites in the study (Advanced MD, City Town Info, and Tool Parts Direct). On each site, we took 4 measurements, two on one day, and two more 14 days later. All measurements were taken during the early afternoon Eastern Standard Time.

For an aggregate look at the data, here are the average results for each tool.

Omniture, with the sample size of the 1 site we ran it on, seemed to take the longest. Second in length was IndexTools, although it should be noted that the time it took for the IndexTools JavaScript to run on Tool Parts Direct was quite a bit longer than for the other sites.
The IndexTools code running on Tool Parts Direct was the same code that was run on Advanced MD and City Town Info, so this suggests that some external factor was in play here. We have not identified what that factor may be, if any, at this point in time.
Have comments or want to discuss? You can comment on the Final Analytics Report here

Section 7: Qualitative Comparisons

This section will discuss some of the strengths and weaknesses of each of the 5 packages that worked closely with us during the 2007 Web Analytics Shootout. We don’t suggest that this is a comprehensive analysis of all the aspects of the products reviewed, but it does cover a number of factors with regard to each package.
In particular, we try to focus our efforts here on information that may impact your purchase decisions or use of your analytics software.

7.1 Clicktracks

In general terms, Clicktracks focuses on ease of installation and setup, ease of use, and also offers excellent pay per click campaign management tools. The product does not offer the same level of configurability and options available to customers of Omniture, Visual Sciences, or Unica.
However, it offers a powerful package for those companies that are ready to go deeper analysis than that you can do with Google Analytics, at a lot lower price tag than some of the other companies. In addition, the ease of installation and set up will be a big positive for those who want to go deeper with analytics but are not yet ready to invest heavily in web analytics development tasks.
Clicktracks also has a lot to offer customers who want to manage their pay per click (PPC) campaigns. This includes basic bid management capability build directly into the analytics application and a Click Fraud Report that helps customers track down potential click fraud. This is a report that carries some authority in the eyes of the search engines.
Lastly, it should be mentioned that Clicktracks also offers a free application known as Clicktracks Appetizer (http://www.clicktracks.com/products/appetizer/). The most significant difference between Clicktracks Appetizer and Google Analytics is that Clicktracks Appetizer reads log files, and does not rely on JavaScript tags.
As a result, it makes a great choice for a free application for website owners that want to include measuring search engine robot data with their analytics software.
Key Technical Points:

Clicktracks – Key Strengths:

The first 10 items are taken from the article we developed with the help of Clicktracks titled: 10 Cool Things you can do with Clicktracks.
The article provides a more complete description of each of the first 10 items listed below, along with a rich array of screenshots. Additional strengths (beyond these 10) are listed starting with item 11.

1. Optimize your PPC Campaigns: Clicktracks allows you to perform basic bid management functions without having to purchase an additional software component, as is required by some of the other vendors. All that is required is to enter your Google Adwords Account and Yahoo Search Marketing information into the Campaign Manager Screen (or import any other campaign reports using the ClickTracks Campaign Template).

2. Slice and Dice your Visitors with Segmentation: Clicktracks offers great segmentation capability in the form of “labels”. Labels allow you to select a wide range of criteria to look at specific groups of visitors, and analyze their behavior separately. For example, you can look at your organic traffic and PPC traffic separately. You can also analyze visitors based on the where they are located geographically. Or you can segment based on the page on which they visited or entered or exited your site.

The Clicktracks label setup is simple, elegant, and powerful. Best of all, the labels automatically show up in all of the reports throughout Clicktracks. You can make the labels invisible in some or all reports by clicking a box in a configuration screen.
One of the niftiest aspects of this feature is that you can create labels which are combinations of other labels. This allows you to do some pretty sophisticated analysis.
For example, you could create a label that tracks people who came from Canada and stayed on the site for longer than 3 minutes. Or, you could create a label for someone who came to the site from Google, visited a certain page, and generated at least $10 in revenue.

1. Apply New Analysis to Old Data: Ad hoc analysis is a key capability in analytics. An example of an ad hoc analysis occurs when you decide that you want to see how conversion rates were last Christmas sorted by product, yet you were not specifically capturing that data last Christmas. With ad hoc analysis on Clicktracks, you can define a new label, do a reanalysis of the data, and presto, you have that label applied to the historical data.

2. Improve Your User Experience: Clicktracks offers some nice tools for gaining insight into your site’s visitors. Clicktracks provides Entrance Path and Exit Path analysis as a way to see the flow of traffic through your site. This type of information can help you better understand how users are experiencing your site.

You can also configure an unlimited number of funnels to track progress through your site. Combined with labels, you can track specific groups of users.

1. SEO Optimization Tools: Clicktracks offers a search engine report that provides you with information on the keywords that are bringing traffic to your site, broken out by search engine. This helps you see what terms are ranking highly on a search engine by search engine basis. You can also easily see plenty of other data, such as the conversion rate, average time on site, cost per visitor, revenue per visitor, and total revenue for each keyword:

2. Contextual Analysis: By displaying the statistics for one of the reports in a browser, Clicktracks’ Navigation Report allows you to see information in context, while navigating your site the same way a user would travel through your site. Seeing your actual web page the way a user sees it, along with analytics data embedded on the screen, and other key data on the right, allows you to see the analytics data in context.

3. Powerful Testing Capabilities: Testing is the easiest way to get a positive ROI on your analytics investment. Clicktracks’ What’s Changed Report provides you with quick access to information about significant site changes, even with just a few days worth of data. This can help you in A/B testing, where you can create two different ad groups with different landing page URLs, and then swap the landing page URLs after a while, and see at a glance which combination of ads and landing pages brought the best results.

4. Keyword Analysis and Research: Clicktracks’ Search Report helps you rapidly determine which of your keywords are performing the best. Chances are that these are vertically oriented terms, rather than major brand names, and rapidly identifying the most productive keywords (using metrics such as high average time on site, highest conversion rate, total revenue, and ROI) quickly helps you increase the profitability of your PPC campaign.

5. Click Fraud Detection: This is one of the gems of Clicktracks. Clicktracks offers a Click Fraud Report that provides you with a way to identify click fraud when it happens. This is useful when it helps you identify click fraud, and it can also help you rapidly identify when suspicious looking activity is not likely to be click fraud, but something else, such as a poor performing ad. 10. Track KPIs Over Time: Clicktracks makes it easy to track your Key Performance Indicators (KPIs) over time. Once you have your KPIs setup, all you need to do is select it in one of the reports you are looking at, and right there on the fly, you will see the KPI data in that report.

6. Log file support: While the 2007 Web Analytics Shootout focused on analyzing all the vendors using JavaScript data collection, one of the limitations of JavaScript is that it can’t track the behavior of search engine robots (because search engines don’t execute JavaScript).

Clicktracks offers you the ability to purchase their software, install it on your own server, and then perform its data analysis on your log files. This provides you with the ability to capture search engine robot behavior analysis and provides you with the security of having all of your website data kept under your own roof.
Better still, Clicktracks allows you to operate in a mixed mode, where you can use log file analysis for robot tracking, and JavaScript data collection for visitor tracking.

1. Labels support Perl Compatible Regular Expressions: This may sound pretty specific and detailed to some, but if you are looking to create content groups (i.e., a collection of related URLs that you want to analyze as a single entity), there will be times when the string processing power of Perl is the only way you can accurately identify the group of pages you want to isolate.

For example, on City Town Info, we wanted to look at all the city pages as a single content group. But the URL naming structure of the site did not make it possible to do something like look for a specific string in the URL itself to identify a page as a city page. But we were able to describe city pages accurately using a Perl Compatible Regular Expression.

1. What’s Changed Report: The What’s Changed Report makes it easy to makes it easy and fast to see what happens when you implement a new campaign, or when you are doing A/B testing.

2. Campaign Analysis: Setting up PPC campaigns and tracking their performance is simple in Clicktracks. To do this, you enter your Google & Yahoo account information or import a report from any other engine (such as MSN) you can easily see what’s losing money, and where the big winners are, and take action to improve your overall results.

3. Education: Clicktracks offers regular free education sessions and demos that are available to all customers to help them get more out of the product.

Clicktracks Weaknesses:

1. Clicktracks does not yet offer a true ASP mode. All versions of Clicktracks, even its browser-based version, currently require you to download large compilations of log file data and perform detailed processing on your own machine. Clicktracks has let me know that they will be offering a new product with a true ASP mode in the near future.

2. The interface is hard to follow at times. While this is obviously a bit subjective, we found that remembering where to configure one item versus another was at times confusing. Many times I found if difficult to remember where to configure one thing or another, even though I had done it before.

3. No customizable dashboards. Clicktracks is a one size fits all interface. Individual users can customize some aspects of the data they see, such as creating their own labels, setting up revenue, importing campaigns, configuring funnels and Internal Search/Data Dissection reports, but the level to which the experience can be individualized is fairly limited.

However, Clicktracks does offer the ability to selectively deliver reports via email, so that your CEO and VP of sales can find the numbers they are looking for in their email inbox, without having to log in at all.

1. Labels are great, but the ability to apply them selectively is limited. You can configure what reports each label shows up in through a straightforward configuration screen, but you can’t do it directly in any one report. This makes it a multi-step process if you just want to look at one label at a time, without the clutter of the other labels. One way to work around this is through the use of multiple datasets.

Note that this is only an issue if you have a large number of label scenarios you are interested in looking at, and as a result, having all labels active on the screen at one time is an obstacle to your goals.

1. No MSN support. Campaign tracking does not currently support MSN’s adCenter API. You can, however, run the Dynamic Ad Text report from MSN and import it into ClickTracks.

7.2 Google Analytics

Google made a big splash when they acquired Urchin Software Corporation. They overhauled the hosted version, renamed it Google Analytics, and made it free to all comers. The philosophy behind Google Analytics is extreme ease of use and installation, and quick easy access to analytics information to promote faster decision-making throughout the organization.
In May of 2007, Google Analytics had its first major update in some time, and lots of weaknesses in the product were addressed, including the addition of a configurable dashboard, and the ability to email out reports. Google has also made at least two smaller incremental releases since May 2007.
Google Analytics does not contain the deeper analytics capabilities of the other packages surveyed here but offers a lot of information for website owners to get started with. In fact, many website owners will find that this is all the capability that they will ever want.
Google Analytics does have some enterprise level customers that are evidently satisfied with what they get in the tool, such as RE/MAX. So whether there is enough for you in Google Analytics is not solely determined by your company’s size. How you are using the analytics is the determining factor, but, of course, larger companies tend to have more complex requirements.
Key Technical Points:

Google Analytics – Key Strengths:

Items listed 1-6 below are taken from the article we wrote titled: 10 Cool Things you can do with Google Analytics.
The article provides a more complete description of the 10 things you can do with Google Analytics; only the items which are differentiating are listed below. The article provides additional information and screenshots.

1. Email out your reports: While this feature exists in all the other packages, it did not exist in Google Analytics until the May 2007 release of the service. This was one of the major weaknesses of Google Analytics prior to that release.

2. A/B Testing: Google Analytics offers support for A/B testing, and tracking the results to see how the two tests compare. It also integrates with Google’s complementary multivariate testing platform, Website Optimizer (also free).

3. Drilling down on referrers. Google Analytics makes it easy to track what types of content gets the biggest response from a particular referrer. For example, you can use this to see how a social media site responds to different content on your site.

4. Navigation summaries for individual pages: This feature allows you to quickly and easily track the traffic history for a specific page on your site. This is very useful in understanding and analyzing the life of a particular article or piece of content.

5. Entrance Sources Report: This report provides a really simple way to see your inbound traffic on a page by page basis. Want to see who the biggest referrers are for a newly launched piece of content? This report makes it easy to do.

6. Entrance Keyword Analysis: Google Analytics provides a standard report that allows you to see the entrance keywords on a page by page basis. This is helpful in combination with the Entrance Sources Report.

7. Entrance Pages Report: This report is similar to the entrance sources report, but is structured around internal traffic on your site. It makes it easy to examine and monitor the path of traffic through your site.

8. Price: When evaluating Google Analytics you need to include its price — free — as one of its strengths. Not ready to spend a lot of money on a higher end package just yet? Use Google Analytics to get started, learn a lot about your site, and better understand what your requirements are before deciding on a higher end, and more expensive, package.

9. Ease of sign-up and setup: No other package is easier to get started with than Google Analytics. Simply sign up with your Google account, place the JavaScript on your site, and you are collecting data.

10. Compare Traffic Between Sites: As the data in this report demonstrate, analytics packages measure differently from one another. So if you are trying to compare traffic between Site A and Site B, and they are running different analytics packages, you simply can’t rely on those numbers. The solution? Install Google Analytics on both sites and now you have data that you can compare.

Google Analytics Weaknesses:

1. Limited Ability to Customize: You won’t find a wide range of capabilities to customize Google Analytics. That’s not to say that there is no such capability, it’s just limited in scoped compared to that of the other packages.

2. No Log File Analysis Capability: As a result search engine crawling data is not available within Google Analytics. Google does offer a paid Urchin Web Analytics solution that reads log files and can report on all search engine robot activity.

3. Ad Hoc Analysis is not Possible: All of your historical data is available within Google Analytics, but once you set up a new filter, you cannot reprocess the historical data. The result is that you can only look forward in time with Google

4. Email Only Support from Google: Google provides free email support in the 19 languages they currently support with Google Analytics. They also have created the Conversion University, a detailed help center and a GA Google Group for peer-to-peer support. For professional services, they have partnered with a third party network of companies who specialize in a variety of services, but Google does not offer these services directly unless you are a large AdWords customer with a direct account relationship.

7.3 IndexTools

In general terms, IndexTools positions itself as offering enterprise-level capability, but at a much lower price than competition such as Omniture, Visual Sciences, and WebTrends. IndexTools is designed to be very easy to set up and install, but still offers a powerful set of customization capabilities.
In fact, IndexTools philosophy is to provide minimal out of the box reports and to provide users with access to powerful customization capabilities. This approach emerges quickly within the interface where all reports can be easily customized.
The IndexTools interface was one of the best we reviewed. Information was easy to access and find. The product allows a large array of filters, content groups, and segments to be set up directly within the UI, without a need to resort to editing the JavaScript. However, customizing the JavaScript is an option in those situations where the complexity of the filter exceeds what can be done in the UI.
IndexTools has a wide array of customers of all sizes, including a substantial number of enterprise customers. The customization capabilities exceed that of Google Analytics or Clicktracks by a substantial margin but is not quite as extensive as the level of customization that can be done with Omniture, Web Trends, and Visual Sciences.
However, installation and setup are in general easier than with those other applications, and in our experience the pricing IndexTools offers is substantially more aggressive.

Key Technical Points:

IndexTools – Key Strengths:

The first 10 items are taken from the article we developed with the help of IndexTools titled: 10 Cool Things you can do with IndexTools.
The article provides a more complete description of each of the first 10 items listed below, along with a rich array of screenshots. Additional strengths (beyond these 10) are listed below, starting with item 11.

1. Customize Reports: IndexTools makes it easy to customize any of its reports. Just go to the report you want to customize and click on the “Customize Report” button, and you are presented with a drag and drop interface to do your customizations. You can pick any of a wide range of groupings or metrics, so there is a lot of flexibility here. In addition, you can bookmark the report so you can look at your customized report any time you want.

2. Customize Dashboards: IndexTools provides you with the ability to offer different dashboards for each user, or even a custom set of dashboards for each user. This allows your CEO to have one experience, your VP of sales to have a different experience, and your business analyst another experience altogether.

3. Ad Hoc Scenarios: Business analysts constantly want to know what has happened in the past. IndexTools supports this ability quite simply. All you need to do is to set up the scenario you want to analyze (by setting up filters or a custom report) and then use the calendar settings to go back to any date or date range setting you choose.

4. Filters: In any of the standard reports in IndexTools you can apply custom filtering right there on the screen. All you need to do is pick “Show Filters”, select your filter type, and you can immediately see the results. You can take this further by implementing more than one filter, customizing the report as described above, or you can bookmark the report for later retrieval.

5. Merchandising: IndexTools allows you to upload a spreadsheet with all the custom category information for your eCommerce site. Once this is done, these categories are readily accessible within IndexTools as filters or in merchandising reports. The merchandising reports also provide you with a broad capability to mix, match, and sort your various categories on the fly, so you can see the data you want to see.

6. Path Explorer: The visual overlay feature within IndexTools is particularly powerful. For example, it deals with some of those hard to handle scenarios, such as DHTML menus, Ajax, Flash, and other interactive elements. In addition, you can define specific content areas within each screen, and analyze just that portion of the screen.

For example, in a newspaper business, knowing the historical click-through rate for position 1 of a page can help you see if the current article in that position is doing better or worse than historical averages. This is easily handled in IndexTools.

1. Alerts, Events, and Color Coding: IndexTools allows you to define events that you want to know about, such as a jump in traffic, or a drop in sales. Once an event occurs, an alert can be triggered, and you can see a color-coded report that provides you with the details of what has taken place.

2. Segmentation: IndexTools segmentation capabilities are straightforward and easy to use. One of the powerful features that IndexTools offers is that you can apply more than one segment at a time, to get a more detailed look at the data based on the segments you have already defined, rather than needing to implement another segment combining those attributes. Segments in IndexTools also get applied in real time, so there is no need to wait for a few hours after creating them before you can see the results.

3. Campaign Management: All the analytics tools we looked at offer campaign management capabilities. IndexTools does this as well, but IndexTools also treats your organic traffic as a campaign. This allows you to see all of your results, including the organic results, in one place.

4. Custom Fields: IndexTools provides you the ability to define custom fields. You can use these custom fields to track attributes of your business that are specific just to your business. Sell shirts? You might want to track men’s vs. women’s, or shirt size, or color, or manufacturer. These can be set up using IndexTools professional services and are then available within the UI.

5. Real-Time Tracking. IndexTools offers real-time tracking (with a 3-second delay). While this may not matter to many businesses, there are businesses where real-time data can have a large impact, such as high volume media businesses where the performance of each piece of content in generating page views is a key driver of financial results. Have content that’s underperforming? Get it out of there quickly and replace it with something that provides better results.

6. Filters support Perl Compatible Regular Expressions: This newly added feature provides the power of Perl Compatible Regular Expressions in defining segments, scenarios, filters, and content groups within IndexTools. The big benefit of this is that it should reduce the amount of custom JavaScript tagging you need to do on your site.

7. Scenario Analysis: Using scenario analysis in IndexTools allows you to model loose funnels that don’t require users to take steps in an exact order, but still show you whether or not your users are progressing towards the close.

IndexTools Weaknesses

1. Measures daily unique visitors on a rolling 24-hour basis. The industry standard is to measure daily unique visitors within a calendar day. IndexTools tells me that this is being changed as of their next release, which is coming soon.

2. No log file analysis. As a result, you can’t get search engine robot crawling data from IndexTools.

3. Can’t place scenarios in custom reports. The tool provides a powerful ability to do your own scenario analysis. However, those scenarios cannot be customized or placed in a custom report and then bookmarked. This prevents you from e-mailing the data to someone.

As a result, sometimes you end up implementing something in a scenario and realize that you want to email the data to someone on a recurring basis. To do this, you find that you need to recreate the definition of the scenario in a custom report.

7.4 Unica Affinium NetInsight

Unica did not start out as a web analytics company. They originally focused on the management of offline customer data, and in providing related marketing tools (campaign management, optimization, lead management, e-mail, marketing management, etc.). As a result, the company has an enterprise customer focus and provides a rich array of cross-marketing capabilities.
The interface is simple, clean, and elegant. It was one of our favorite interfaces to work with. Customization of reports within Affinium NetInsight is usually simple and slick. The highly graphical presentation format is also nice.
The special differentiator for Unica is the degree to which they have integrated offline customer data into Affinium NetInsight. This enables some neat features to cross reference the offline data, and perform integrated marketing. Some of the things you can do with this are detailed below.

Key Technical Points:

Unica Affinium NetInsight – Key Strengths:

The first 12 items are taken from the article we developed with the help of Unica titled: 12 Cool Things you can do with Unica’s Affinium NetInsight.
The article provides a more complete description of each of the first 12 items listed below, along with a rich array of screenshots. Additional strengths (beyond these 12) are listed starting with item 13 below.

1. Creating custom dashboards: It’s easy to set up custom dashboards in Affinium NetInsight. The benefit of this feature is that it allows you to provide a custom experience, based on the needs of the person using it. For example, the VP of sales can see what she wants to see, without the clutter of a bunch of other numbers that are of no interest to her.

2. Ad-Hoc Analysis: Affinium NetInsight provides you with the ability to apply segments, filters, and content groups to historical data. This allows you to conduct historical research on user behavior that can be incredibly valuable to your business.

3. Drag, Drop, and Drill Down: Affinium NetInsight offers extensive filtering capabilities. You can apply as many filters as you want simultaneously, and as mentioned above, you can look back in time. The range of filtering options provided is extensive.

4. Correlate Data: Cross-referencing data is another strength of Affinium NetInsight. Not only can you set up multi-level tables of data, but you can also easily re-arrange the table structure, dragging parameters and dropping them where you want them, and the table dynamically rebuilds itself on the fly.

5. A/B Analysis mode: Affinium NetInsight provides a simple and elegant way to set up and see the results of two different scenarios. These can be seen on a single split screen so you can look at the results side by side.

6. Integrate Offline Customer Data: Due to its origins, Unica has provided extensive capabilities for integrating offline customer data together with online customer data. Among other things, this allows offline data attributes to be used as filters and segments within Affinium NetInsight. For example, you may want to filter visitors based on whether or not they are customers through your brick and mortar stores, or belong to your “Gold Membership Program”.

7. Examine Individual Click streams: Affinium NetInsight offers you the ability to review the detailed behavior of any single individual. B2C sites can use this to sample customer behavior data, but the real power is for B2B sites. For example, a B2B site’s salesperson can be looking at the clickstream data of a potential B2B customer while handling a sales call with them.

8. Robot/Spider analysis: Affinium NetInsight offers the ability to read log file data to extract information on the behavior of search engine robots. As an interesting add-on to this, you can also follow the precise path of the robot step by step through your site, to see how the site is being crawled.

You can either run Affinium NetInsight as a pure log file based solution, or in a hybrid mode that uses the log file to get robots data, and JavaScript to get user traffic data.

1. Remarketing: With Affinium NetInsight you can automatically implement fast direct marketing responses to events on your site. For example, if a user abandons their shopping cart part way through it, and you have their email address, you can see what product they were thinking of buying and automatically email them an offer for a 10% discount on that product.

2. Ask NetInsight Wizard: This is a unique feature. Not sure how to get the data you want? Use the Ask NetInsight Wizard to get it. Pick from a list of standard questions, and the report is generated on the fly, or, ask a completely unique question, and the wizard will try to get you the data you want.

3. Heat Map Overlay: The Affinium NetInsight site overlay capability has the added feature of allowing you to look at segments within the overlay. The segment data will show up in a heat map format, making it visually easy to figure out how that particular segment behaved on that page.

4. Date Comparison Reporting: Affinium NetInsight was unique in the packages we reviewed in its ability to allow you to compare the results of 2 different dates side by side.

5. Perl Compatible Regular Expression Support: The ability to use Perl Compatible Regular Expressions in defining content groupings, segments, and filters make it possible to do less custom JavaScript work for some sites. This comes in handy especially if you have a site with dynamic URLs or complex content groupings.

6. On Demand or On Premises: You can run Affinium NetInsight in either an ASP mode with the data hosted at Unica, or buy the software and run it in-house, and keep the data on your own servers.

Unica Affinium NetInsight Weaknesses:

1. No Default Cookie Handling. You can’t get started with Affinium NetInsight On Demand unless you have set up a custom DNS record for your cookies, or until you have set it up to piggyback on another existing persistent cookie. If the site does not have a persistent cookie, NetInsight offers a web server plug-in that can be installed on the web servers to automatically generate a server-side cookie.

2. No Web Services API for Retrieving Reports – Affinium NetInsight offers an enterprise level solution but does not have a web services API for retrieving reports such as the ones offered by Omniture and Visual Sciences. A workaround does exist, in that a database API exists, but using it is a bit more involved on the JavaScript-based product from Unica.

3. Real-time data needs to be requested – You can get data with 15-minute freshness from Affinium NetInsight, but you need to request it from Unica. It also may cost extra, depending on the rest of the relationship, to get it.

4. Education Curriculum is not as Extensive. Unica does offer an educational curriculum, but it’s not as extensive as that of some of the other vendors.

7.5 Visual Sciences HBX Analytics

Visual Sciences, Inc. (formerly known as WebSideStory), has been one of the major players in web analytics for many years. The company focuses on high-end solutions that offer a wide range of flexibility and power. Correspondingly, there may be some configuration and setup work to do, in order to gain access to that additional capability.
The company was originally named WebSideStory but re-branded itself to Visual Sciences after its acquisition of the company by that name in February of 2006. The move marks the company’s increasing focus on multi-channel applications.
Visual Sciences Report Builder is a powerful tool that provides direct access to the data within the HBX Analytics database as an Excel plugin, and with which that data can be analyzed and manipulated like any other data in Excel.
This is part of a family of capabilities that emerge from the built-in APIs within the tool that allows third-party applications that access the API to extract data and be used in a wide variety of ways. This programmability is one of the hallmarks of HBX and permits flexible data access and manipulation that can be built directly into your web application.
You may be able to use a cheaper package to meet your needs, but HBX Analytics and its companion products offer the type of capability required by the most demanding applications.

Key Technical Points:

Visual Sciences HBX Analytics – Key Strengths:

The first 11 items are taken from the article we developed with the help of Visual Sciences titled: 11 Cool Things you can do with HBX Analytics.
The article provides a more complete description of each of the first 10 items listed below, along with a rich array of screenshots. Additional strengths (beyond these 10) are listed starting with item 11 below.

1. Active Viewing: Visual Sciences refers to site overlay functionality as Active Viewing. In HBX Analytics the expanded capability includes the following features:

1. Look inside DHTML menus to see click-through data on individual menu items
2. Place an overlay on top of a form
3. Place an overlay on top of Flash
4. Apply segmentation within your overlay

The segmentation feature allows you to perform path analysis on different types of user groups on your site. In addition, when multiple links on a given page point to the same page, HBX will track the click-through rates of each link as separate items (most vendors do not do this).

1. Customizable Dashboards: HBX offers customizable dashboards so you can customize the experience of each individual user with HBX to meet their specific needs. This enables each user to get what they need, and only what they need.

2. Navigate the Funnel: All the analytics packages provide funnel capability. However, not every package enables you to define a “loose funnel”, which HBX defines as one which can be interrupted, yet the funnel will keep tracking the user.

A common scenario is where users get part way through the funnel, and then decide to check the About Us page or the Privacy Policy page on your site before continuing. In a strictly defined funnel, this would be seen as an exit. With a loose funnel, you can still get a clear indication of how these users progressed through the process.

1. Filtering Conversions: HBX Analytics lets you set up filters based on conversion type (e.g. type of product purchase, newsletter signup, contact us request). This provides you a way to track the path of visitors specific to each conversion type.

2. Active Segmentation: HBX provides a simple and elegant way to take advantage of the segments you have defined. A list of all available segments shows up as a pulldown list on the top of all the report screens, so just select the segment and you are off to the races.

3. Custom Metrics: Even with all the slicing and dicing power of HBX, sometimes you need to do more. HBX does allow you to define Custom Metrics and build them into your analyses. For example, Visual Sciences’ Site Search tool can be integrated into HBX Analytics to offer a rich array of specific attribute data. For example, if you are an eCommerce vendor you may want to capture data on price, size, color, etc., and this is something you can do with HBX and Custom Metrics.

4. Campaign Attributes: You can also isolate and filter on different types of campaign attributes within HBX. For example, if you are running a banner ad campaign, you can look at the size, color, and message as different attributes, or in an email campaign, you can look at the effect of link position in the email. This capability helps in rapid optimization of campaign results.

5. Report Builder: Of the packages reviewed in this section, only HBX offers anything like Report Builder, and it’s special. You install it as a separate module that is a Microsoft Excel add-on. With it installed, you can extract data directly from HBX into Excel, and then use that information the same way you use any other data in Excel.

With this capability, you have a powerful combination of Excel and HBX that allows you to do a lot with your data. Better still, the information in the spreadsheet can be set up to automatically update on a regular basis (e.g., daily) so that you can see the updated information in Excel without having to do anything at all.
In addition, I believe that the ability to copy and paste that data, to expand upon the amount of data extracted from HBX on the fly, is absolutely unique to Report Builder.

1. Active Dashboard: HBX allows for simple modeling of “what if” scenarios with its Active Dashboard functionality. Basically, this allows the integration of Flash into powerpoint, with the result that you can take certain KPIs and implement them as sliders. Moving the sliders around will then dynamically update the rest of the data (according to the business rules you have defined).

While this takes some time to set up, it makes for a very powerful modeling and presentation tool.

1. APIs: Everything within HBX Analytics and Report Builder can be accessed through APIs provided by Visual Sciences. This allows for the design of rich applications that leverage that data on behalf of your site. For example, you can integrate the API into your content management system, and have it use live analytics data to drive content decisions on your site.

2. Ad Hoc Analysis: HBX allows you to perform analysis on historical data. This is useful in looking back in time to check past results and apply filtering and segmentation where it’s of interest.

3. Regular Classes: Visual Sciences runs free online training classes on a regular basis that any customer can join. These are invaluable in helping customers get more out of the Visual Sciences tools.

Visual Sciences HBX Analytics Weaknesses:

1. More Setup Required. HBX requires more setup than some of the other packages. For example, the default type of cookie used is a third party cookie, and you need to modify your DNS record to be able to use a first party cookie.

There are also options that are not available by default, like the Navigation Report and Site Overlay. To get these capabilities, you need to set the hbx.lt parameter to “auto” on all the pages of the site for which you want them.

1. JavaScript tuning is a potential source of error. While JavasSript tuning offers substantial power and flexibility to do whatever you might want to do, it is also a potential source of error. Simple coding errors can result in misleading data. Also, when changes are made to the web site, updating the HBX JavaScript needs to be part of the development plan.

2. Can’t measure ROI on a keyword without an additional tool: The basic HBX Analytics tool does not come with the ability to measure ROI on a keyword by keyword basis. However, Visual Sciences offers a separate bid management tool with this capability.

3. Can’t trap on URLs to look back in time: HBX does not store URL data by default, and this means if you want to do an Ad Hoc analysis that you need to rely on other parameters (such as Page Name and Content Group as defined in the JavaScript) to do this type of analysis.

Have comments or want to discuss? You can comment on the Final Analytics Report here

Appendix A – Placement of JavaScript on the participating sites

The following table shows the order of the JavaScript for each of the sites.

It’s been suggested that JavaScript order will have a significant impact on the results. We specifically analyzed this during the course of the study. See above (supply section reference) for more information on the results (hint: there are bigger issues than JavaScript order!).
In the table above, CTI Initial refers to the initial order of the JavaScript placed on City Town Info. CTI Second refers to the order that was used in the second phase test for City Town Info, in which we tested the effect of the order in which the JavaScript executed. This is reported on in detail in Section 6 of this report.
It should also be noted that JavaScript is disabled on some users’ computers. When that happens, none of the web analytics packages will return data in our study, with the exception of WebTrends, whose analysis was log file based.
Checking across our 4 participating web sites, we found that about 3.5% of users had JavaScript disabled. This means that the true user counts are potentially 3.5% higher than what was reported in the study.
Note, however, there are plenty of other sources of error in analytics, and this particular factor would be in common in all analytics packages that rely on JavaScript.

Appendix B – Sessionization

One of the challenges of web analytics that uses a JavaScript implementation (as we did for all packages in the 2007 Web Analytics Shoot Out) is that the software can only tell when a user loads a page that causes the JavaScript to execute.
This is naturally very accurate when counting page views because every time the JavaScript executes it equates to a page view. There are still a source of error, such as a user clicking on a link and leaving the page before the JavaScript executes, but these sources of error are relatively small.
Total Visitors and Unique Visitors, however, rely on cookies and session tracking. Every time the JavaScript executes, it must:

1. See if a cookie already exists. If it doesn’t, try to set a cookie, and record the time of the user’s visit in it. This equates to the start of a new session. In the event that the cookie does exist, then proceed to the following step.

2. Open the cookie and see if this is a continuation of an existing session (in principle, a continuous visit to the site), or a new one. Update the internal tracking data appropriately.

This still sounds relatively simple in principle, but it gets complicated when you start factoring in the processing that then has to take place. The analytics software has to parse through all the clickstream data in the log file created by the Javascript, and match up the various clicks it sees with sessions.
It does this by keeping all sessions open for a fixed period of time. Once a session is deemed to be closed, as usually determined by there being no new clicks related to that session for a fixed period of time, such as 30 minutes, the session gets written out to disk.
As you can imagine, this task can place a heavy demand on processing power and memory. For practical purposes, analytics software can’t keep sessions open forever. In general, the industry has decided that a session ends after 30 minutes of inactivity. Note that Clicktracks actually uses 15 minutes as a default, and both Clicktracks and Unica make this a user-configurable setting.
Let’s see how that affects two common scenarios:

1. A user comes to your site and then goes to lunch for an hour or so. They then come back and continue browsing your site. In theory, this should be considered one session, but all the packages in our study would treat this as two sessions.

2. A user comes to your site, then goes to another site, and 10 minutes later comes back to your site by typing in the URL directly, or via a bookmark. In theory, this should be considered two sessions, but all the packages in our study would treat it as one session.

Another factor to be considered is what happens when cookies are blocked on the user’s machine. Some packages simply ignore the visitor for purposes of counting visits and unique visitors. Others fall back on pixel tracking and/or IP and user agent based tracking.
Across the sites in our study, only about 3.5% of users did not have JavaScript enabled, but 3.5% is a pretty large potential source of error. So even though pixel tracking and IP and user agent based tracking is less accurate in tracking sessions, it nonetheless is more accurate than ignoring 3.5% of the visitors.
The following table outlines some of the session handling defaults of the active participants in our study:

Have comments or want to discuss? You can comment on the Final Analytics Report here
Have comments or want to discuss? You can comment on the Final Web Analytics 2007 Shootout Report here
This is the end of the second part of the Final Report of the 2007 Web Analytics Shoot Out. You can see Part 1 of the Report here.