Rajat Mukherjee
The following is the transcript of an interview of Rajat Mukherjee, product lead for Google’s Custom Search Engines project.
Dr. Rajat Mukherjee is a group product manager on the search team at Google, Inc., working on several products, including Google Custom Search.
Prior to joining Google, Rajat was senior director of product management, Yahoo! Search Technology, and was responsible for managing the product platform for Yahoo! Web Search, ultimately providing innovative search products for consumers to find, use, share and expand content across the Web.
Prior to joining Yahoo!, Rajat was the director of software engineering at Verity, Inc., where he was responsible for managing and creating new applications including Verity Ultraseek (formerly Inktomi Enterprise Search), Verity Enterprise Web Search, Verity Response and Verity Federator. Before Verity, Rajat was a research staff member at IBM’s Almaden Research Center, conducting research on Web technology and content management.
Rajat began his career as a research staff member at IBM’s Thomas J. Watson Research Center in Hawthorne, New York. There he conducted research on parallel and scalable Internet servers, high availability in clustered computing and scalable transaction processing. His experience included contributing to the design of infrastructure used during the 1996 Atlanta Olympics and the Deep Blue-Kasparov chess match.
Dr. Rajat Mukherjee holds a Ph.D. and a Master of Science in Electrical and Computer Engineering from Rice University, as well as a Bachelor of Technology from the Indian Institute of Technology in Madras, India.
Interview Transcript
Eric Enge: Tell me what’s been going on with custom search engines?
Rajat Mukherjee: Over the last couple of years many things have happened. It’s been very exciting to be part of a team that’s spearheading a new hosted search paradigm. And it’s been a lot of fun developing the product features with the help of an outstanding development team and watching the product grow.
Eric Enge: Can you give us a top-to-bottom overview of the major initiatives you have and the goals you are trying to accomplish?
Rajat Mukherjee: When we spoke two years ago, we had over 100,000 registered search engines. Today, we have more than one million registered custom search engines. Since I joined the project in January 2007, we have had about one hundred times of growth in traffic.
So, the program has been doing really well, and we have pretty significant traffic. I think this effectively validates the need for high-quality contextual search, and it proves that there is a need for this kind of product in the business world. Our goal is to satisfy this need with the best product that we can build.
Eric Enge: Absolutely. You have also focused a lot of on-site search, correct?
Rajat Mukherjee: We have a range in terms of product capabilities and deployment of technology. Site search is one requirement, but we also have a range of different types of search experiences.
For example, we launched a Blogger widget on Blogger’s experimental site. With this, people can actually create a search experience that’s not just searching their own blog, but also pages they’ve linked and their blogrolls. It’s like a blog neighborhood if you will.
Many personal search experiences have been created, e.g., around bookmarks, and many developers are doing similar things around their communities. So, that’s one category of search engines.
Google Site Search targets businesses. There, you have a whole bunch of organizations that are developing search across a single site, or collections of sites. So, if you have a publisher with many publications, for example, you can search across all those different magazines or brands.
Then, we have interesting community-based implementations. Adobe has implemented Community Help, which allows people to search both Adobe.com as well as content from their developer communities. Thousands of sites are encompassed within Community Help, and Adobe has taken the further step of integrating this search experience into Creative Suite 4. That was very exciting. Basically, users and developers have created this range of different search experiences based on contextual needs.
Eric Enge: Right. This notion of searching comments from the community is an interesting one.
Rajat Mukherjee: Yes, I think the community aspect is something with the potential to develop and we hope to add interesting new features as we go forward. I think that’s going to be an exciting area down the road.
Eric Enge: Right. Have you seen anybody just simply investing the time and building high-quality search experiences by editorially selecting a set of sites to use?
Rajat Mukherjee: Yes, absolutely. If you look at About.com, they have a very interesting implementation. They have multiple sections, and each of their sections is handled by an editor or guide. They use Custom Search both within and across sections. So, they’ve used advanced capabilities like Linked Custom Search to create dynamic search engines that morph over time, and that’s a very interesting implementation.
Eric Enge: Yes, that is interesting. How long have they had that out there?
Rajat Mukherjee: It has been around for over a year now if I am not mistaken.
Eric Enge: That must be doing quite well for them.
Rajat Mukherjee: Correct. They’ve also adopted interesting advertising changes. With Custom Search, you can tune your ads to be more topical by using keywords. We use these keywords to further refine the ads that we display. Partners have seen revenue improvements with this feature.
Eric Enge: You could really do some very interesting tuning there. It’s one of those things where you might pick the few obvious phrases, and find that they don’t do as well as when you pick some less obvious phrases.
Rajat Mukherjee: Right. We are interested in looking into this area in greater detail and developing better technology for it because monetization is one of the key aspects of Custom Search. This has also led to its success because we’ve had monetization in place since our initial launch. Publishers are obviously benefiting from that, both in terms of getting the right ads in front of their users and monetizing search. Any improvements in this space can help our webmasters and publishers right away.
Eric Enge: Another thing I have noted is the programmability of custom search engines has increased dramatically.
Rajat Mukherjee: Yes. As I mentioned, Linked Custom Search provides users the ability to create dynamic search engines that change over time. And we update these engines automatically by reading the appropriate definitions that developers have made available to us.
In addition, we’ve also added a programmatic interface for management and creation of Custom Search engines. That is very interesting because we’ve integrated Custom Search with several web hosting providers who’ve provided their users the ability to add search to their websites from the hoster’s control panel. We announced this a while back with a number of hosters like BizLand, IPOWER, StartLogic, and FatCow. They have implemented this provisioning API, allowing webmasters to create search engines and place search boxes on their websites without even coming to Google.
Eric Enge: So, why don’t we talk a little bit about some of the other new things that have happened, such as On-Demand indexing?
Rajat Mukherjee: We’ve made a lot of improvements in our indexing pipeline over the last year in order to go deeper into the sites that we are searching across. We introduced On-Demand Indexing, which allows you to index specific content within 24 hours. That’s very, very important. For example, when you are launching a new product on your own site, you want it to be indexed and searchable, so you can use On-Demand indexing to achieve that.
As part of these indexing improvements, we’ve added support for Sitemaps and integrated more tightly with Webmaster Tools. It helps to have many of these services integrated.
We just launched Google Services for Websites, a program for partners that offer integrated services from Google that are relevant to webmasters. This enhances our offerings to hosters so that their users can quickly sign up for Google services, such as Webmaster Tools, Custom Search and AdSense.
Eric Enge: Right. It’s a way of helping get it out there, and an expansion of what you did with the hosting companies already.
Rajat Mukherjee: Yes, exactly. We are looking to make it easier for hosting companies to use our APIs and services and offer them to their users at the hosting control panel
Eric Enge: What aspects of the product are you focusing on?
Rajat Mukherjee: Over the last couple of years we’ve done a lot of different things. I’ll put them into three categories.
The first one is product improvement; features that we’ve added to improve the product. I think we are starting to see some really interesting implementations of dynamic custom search. We’ve had great feedback on our experimental Blogger gadget and we hope to improve it and make it available to all Blogger users. We’re also excited about our ads improvements.
The second category is the offerings we provide targeting different classes of users. Site Search offers businesses optional ads, branding controls and greater presentation control via XML results. Business customers also have greater needs for On-Demand Indexing. AdSense for Search is for publishers who wish to monetize, and developers can take advantage of Custom Search and AJAX Search APIs. These offerings address the needs of different market segments.
The third area is how Google itself is using the technology. We use Custom Search in many of our products and services. We eat our own dog food.
In addition, Custom Search is available globally. More than half of our traffic comes from outside the US and in languages other than English, so that’s exciting. Custom Search was one of the first Google products that was internationalized into forty languages in one shot. It’s really pushing this global approach to the product from day-one.
Eric Enge: Right. What country outside of the US do you receive the most volume from?
Rajat Mukherjee: There are quite a few. We, of course, get a lot of results from the UK. We also have countries like Germany, France, Brazil, Turkey, and China. It changes on a weekly basis. There are many languages that are showing tremendous growth in traffic, e.g., Spanish.
Eric Enge: What improvements are you making on ads?
Rajat Mukherjee: We have the ability to tune your ads through keywords that you can provide for your search engine. We are looking to expand this as we move forward in terms of being able to automatically figure out what ads we should show for a given topic. We hope to improve user experience and publisher revenues with more topical ads. And we also talked about integrating promotions. I think we talked about subscribed links last time — we’ve tightly integrated that so that people can provide promotions right on top of the results for specific queries.
Eric Enge: For the benefit of people reading this interview, what exactly are subscribed links?
Rajat Mukherjee: With subscribed links, a webmaster who is providing search on his website can decide that he wants specific information to show up right on top for specific keywords. Say you are a travel website like Orbitz, you can show a special promotion that you are offering for a trip to Hawaii integrated right into the top of the search results for users who are searching for queries like “hawaii,” or “maui.”
We’ve made it easy for people to configure that. And we’ll be looking to make further improvements for promotions. The idea here is that you can promote a given set of results that you know your users want to see right to the top.
Eric Enge: Tell us a little more about how Google uses Custom Search internally.
Rajat Mukherjee: We learn a lot as we use the product ourselves. Many Google sites are powered by Custom Search, for example, Google Analytics and our Help Centers. We also customized the search experiences for Google’s landing pages for the 2008 Olympic Games and the 2008 elections. That’s where the contextual nature of search shows right through. So, for example, if you do a search for “nuclear power” on Google, you get very good, but generic, results about nuclear power. But, if you performed the same search on the elections landing page, you would get information around the policies of both candidates during the elections. So, that was a very tightly focused search experience, and that’s where custom search really does a great job.
Eric Enge: How does someone go about creating something like this politically-oriented version? What is the process?
Rajat Mukherjee: This was actually quite simple. When the election team was putting together this landing page, they decided that they wanted to have a very tightly-coupled search experience. Effectively, they decided on a number of high-quality sites, including the Convention sites, candidate sites, and top political blogs. There were obviously many active blogs at that point in time.
That was done by a set of Googlers, and they used the collaborative features of custom search over a couple of days. Custom Search allows users to invite others to contribute to the search engine.
Eric Enge: Right. So, there is some upfront work to decide the family of sites that you want to have included, but if you are knowledgeable about your space, you can probably do that fairly quickly, rather than trying to invent something you don’t know so well.
Rajat Mukherjee: Exactly. I mean, this really confirms that people who are experts in the field can collaborate for a better user experience.
When Google celebrated its 10th anniversary, we created a timeline and we used Site Search to search the timeline. So, we’ve been getting a lot of feedback from internal teams in terms of features or enhancements they’d like to see in the product. So, with internal requirements and feedback from webmasters, our users have helped enhance the product significantly over the last couple of years.
Eric Enge: Can you talk a little bit about how the timeline was put together?
Rajat Mukherjee: We were trying to highlight product launches and events that have occurred over the last ten years in the development of Google as a company. An interactive timeline application was built- you should check it out. Site Search was used to search the timeline. The search user experience was very tightly coupled with the timeline. So, when you click on a search result it brings up the interactive timeline, which is a very nice AJAX-driven dynamic browse experience.
Eric Enge: It really sounds like a lot of cool things are going on as programmability is allowing different people do more aggressive things, and the outreach through distribution partners like the hosting companies.
Rajat Mukherjee: We have worked pretty hard on outreach to developers, and we have a strong developer community as well. On that front, I’ve actually had the opportunity to present to webmasters, because Google has organized a set of SearchMasters conferences — three over the last year, Mexico City, Sao Paulo, Brazil and, most recently, Bangalore, India. At SearchMasters we have been able to reach out to more than a thousand webmasters, talking not just about products and APIs, but also about integration with Webmaster Tools and other Google services. So, that’s been really exciting.
Next month, Nick Weininger, our Tech Lead, will be participating at the World Wide Web Conference in Spain.
We also have a Custom Search session at the upcoming Google I/O developer conference in May in San Francisco.
So, we are constantly looking to reach out to our developer community. We get a lot of feedback from them and on the Custom Search user group as to what users want to see in the product.
Eric Enge: Yes, I imagine you do. Can you talk more generally about the need for such technology and where the industry is currently?
Rajat Mukherjee: I think the good news is that our effort to create a contextual search offering that’s relevant and useful for business has been validated. I think over the last couple of years the growth of the product and the feedback we’ve gotten have been tremendously positive. So, I think we are doing really well in terms of addressing the market need.
One point I’d like to make is that we customize not just the product, but have also customized our offerings for different customer classes- webmasters, publishers, businesses, partners and developers. I think that puts us in a very good position in the industry. We cover everything, from addressing small developers’ needs for personal search engines to helping large publishers who have significant configurability and scalability needs. Our traffic is growing and we’re getting positive feedback from our customers.
Eric Enge: Right. So, our good friend from the election, Joe the Plumber, could create a very simple custom search engine for his website, while a massive company like Boeing could create an entirely different kind of custom search engine.
Rajat Mukherjee: Correct. And they have different options. Those who want to monetize have the ability to do so, and they can get the product for free. Those who want to have more control can use Site Search, which is a licensed offering. So, we’ve made sure to address the needs of these different markets, not just focus on technology.
In terms of product, we have a lot of unique features, e.g., Linked Custom Search for dynamic behavior, the support for On-Demand Indexing and deeper coverage of sites, and topical ads. The ability to do promotions easily and drill-down into results using labels and refinements are all unique features.
The other important point is that monetization has been built into the product right from day one. It’s not something that we had to figure out as we went along — it was part of the core value proposition of the product. Now, with the APIs that we have for distribution and the features we have in the pipeline, I am really looking forward to entering the next phase of product growth. So we’ve created a very compelling platform offering.
Eric Enge: Right. Well, ten to one growth in the past two years is a hard act to follow.
Rajat Mukherjee: Yes. 100 times in terms of traffic! It’s true and it’s really exciting. And I would love to see where it is in a couple of years. In terms of future features, there are lots of different things that we are looking at adding to the product. Improving monetization will always be useful for everyone involved; it’s a practical need in the market. And, if people can monetize better, especially in these times, I think that’s great. A lot of people are looking to those solutions. We have been working on features around improved presentation and customization. Obviously, when you say custom search, people want to customize their search results in various ways. Today we have the ultimate end customization via XML results- you can do whatever you want – but, we want to make it easier for people to do this even without major development resources. So, simple product configurability is something we are always trying to enhance. We are looking to improve support around more structured content. Because our product is global, we are looking at better linguistic support, and there are lots of tools that Google has already developed that we can use in custom search, e.g., transliteration.
There are many other features that we’d like to add. For example, we want to add query suggestions contextual to the topic of the search engine. That’s not trivial to implement.
SearchWiki on Google.com means one thing, but what does it mean in Custom Search? Who your community is, right? So, we need to look into how we deploy some of these advanced features in the context of Custom Search.
Customers always want improved analytics, and that’s something that we would like to improve. I am personally a big believer in the mobile experience. So, as Smartphones become more common, what would it mean for us to provide a mobile search experience? So, these are some of the ideas that we talk about. And, of course, some of these get baked into our roadmap as we move forward.
Eric Enge: Mobile search is a very interesting animal just because of its very nature, a very challenging environment with small keyboards and more awkward navigation
Rajat Mukherjee: I agree. And I think that’s where features like suggestions make a lot of sense, in terms of being able to click on something as opposed to typing it in. If you are searching in a language that’s difficult to type, automatic suggestions and transliteration really help.
Eric Enge: Yes, of course. And that will play in some international environments. Well great, there’s a lot going on; and it’s obviously a very successful product line. Where can readers follow the latest developments?
Rajat Mukherjee: All our developments are announced on the Google Custom Search Blog.
Eric Enge: Thanks Rajat!
Rajat Mukherjee: Thank you Eric!
