Core Improvements in Live Search

This post will provide some more info on September 26, 2007, Searchification event that Microsoft put on. In particular, this post will focus on the core search engine improvements part of the presentation. I will provide some brief comments on each announcement, as well as pictures of the speakers.
1. Introduction by Brad Goldberg. Brad Goldberg, general manager of the Windows Client Product Management Group at Microsoft, who manages the search team from a business perspective, started things off for the event:

Brad spoke about data that indicates some of the basic problems with search. For example, 40% of search queries fail to provide an answer, and 50% of these queries require refinement before an answer is found. People find that getting what they want requires a high level of cost and commitment.
One of the more interesting things he spoke about was the search market data share data from comScore:

Based on this data, he stated that Microsoft’s focus is on getting more repeat queries from their user share, or doing a better job of delighting their current customers.
2. Overview by Satya Nadella. Satya Nadella, Group VP, Search and Advertising Platform Group, was up next and dug into a bit more detail about Microsoft’s areas of focus for this update:

Note that one of the comments Satya made in a pre-show discussion I had with him was that the Microsoft infrastructure is finally getting caught up, and this is enabling them to do much more with their search product.
The last major update that Microsoft did was in September of 2006. Microsoft has been doing rolling updates through the year, including a number of performance and relevance changes. In this release, several of Microsoft’s search products were affected:

1. A major update was done to the web search index
2. Maps
3. Mobile Search
4. Shopping Search
5. Health Search
6. Image Search
7. A Microsoft Video Search Product was Announced

Here is a screenshot of one of Satya’s slides:

Next up was a summary of customer feedback:

Choosing a Global Software Development Partner to Accelerate Your Digital Strategy

To be successful and outpace the competition, you need a software development partner that excels in exactly the type of digital projects you are now faced with accelerating, and in the most cost effective and optimized way possible.

Get the Guide

The data in the above pie chart was based on an analysis of user click behavior. The sidebar point about relevance was based on an analysis of over 10,000 feedback submissions.
Given the preponderance of concerns about relevance, Microsoft did some further research to get a better understanding of the nature of the relevance concerns. This broke out as follows:

Based on what Microsoft learned from these analyses, they invested in 6 major areas:

1. Coverage – They increased their index size from about 5B pages to about 20B pages
2. Query intent – Making a better determination of what the user is really looking for
3. Query refinement – Determining how to refine a query to provide the user with better results
4. RankNet improvements – A variety of tweaks to Microsoft’s Neural Net algorithm to improve results
5. Structured information extraction – Doing a better job of using structured databases to improve relevance
6. Rich answers – Incorporation of blended data from verticals, such as image and video search

For those of you who want a brief definition of what RankNet is, you can see it here:

Demos of Improvements. At this point in the presentation, Ramez Naam joined Satya to run some live demos:

Ramez demoed a variety of search queries and their results. Here are some of the queries that were demonstrated:

1. 1. EPRML – Microsoft did not show acronym based answers before, and now does. They also only showed about 1,700 results for this query previously, and now show more than 10,000.

the office – search engines normally strip off “the” from this query, but many users are actually looking for the TV show. This now comes up in Live Search results.

1. 1. c.n.n. – Live search used to look for “c n n” after seeing this query, but now it knows to look for CNN.

IL soccer – The new Live Search understands that in this query that “IL” means Illinois.

1. Groig Freiderich Nicolai – Live Search now auto-corrects this query to “Groig Friederich Nicolai”.
2. China – This search shows off some of the rich media integration, as well as the “Related Searches” functionality at the top right of the results screen.
3. Volkswagen Kaefer – Shows a German page, which probably is the best result for most users. The page can be translated on the fly, and in fact, Live Search offers a mode in which you can see the original German version and the English version side by side. You can also modify how this works using on-screen controls.
4. San Jose weather – now shows the weather right on the screen. This feature will be live to the public soon.
5. San Jose traffic – You now can near real-time traffic info right there on the screen.
6. MSFT – Provides an intraday stock chart, along with pricing and volume information on the search results page.
7. Barack Obama – News results are integrated in for those queries where that would be relevant.
8. space shuttle videos – Video results are incorporated directly in the results, and you can play the videos inline on that page.

This is a sampling of some of the more interesting queries demonstrated during this part of the presentation.
Ultimately, the objective of this effort was to increase their search results relevance. Microsoft then did some testing with live human subjects to assess relevance. Each participant was trained on how to assess relevance. They had this group do a large number of searches and presented them with results in a format where they did not know which search engine the results were from.
Net-net, the results of this testing showed a dramatic improvement in Microsoft’s relevance scores. Here is a chart showing the details:

In summary, it is clear that Microsoft made a lot of improvements to their core search results. The real tale of the tape will emerge from tens of millions of searches done by real users. That said, a number of different issues were presented, and Microsoft has addressed them, so this presents good progress.
The other concern I would have would be is whether the underlying strategy of trying to capture more market share from their existing users will work. After all, is the reason they capture the initial search, but not the follow-on searches, simply because users know which search engine they already trust?
This could provide some resistance to getting comfortable with Live Search results. However, I can tell you that I am more intrigued by Microsoft’s improvements and Live Search then I have ever been. I know I will be doing some testing of it on an ongoing basis.