Rajat Mukherjee
The following is the transcript of an interview of Rajat Mukherjee, product lead for Google’s Custom Search Engines project.
Dr. Rajat Mukherjee is a group product manager on the search team at Google, Inc., working on several products, including Google Custom Search.
Prior to joining Google, Rajat was senior director of product management, Yahoo! Search Technology, and was responsible for managing the product platform for Yahoo! Web Search, ultimately providing innovative search products for consumers to find, use, share and expand content across the Web.
Prior to joining Yahoo!, Rajat was the director of software engineering at Verity, Inc., where he was responsible for managing and creating new applications including Verity Ultraseek (formerly Inktomi Enterprise Search), Verity Enterprise Web Search, Verity Response and Verity Federator. Before Verity, Rajat was a research staff member at IBM’s Almaden Research Center, conducting research on Web technology and content management.
Rajat began his career as a research staff member at IBM’s Thomas J. Watson Research Center in Hawthorne, New York. There he conducted research on parallel and scalable Internet servers, high availability in clustered computing and scalable transaction processing. His experience included contributing to the design of infrastructure used during the 1996 Atlanta Olympics and the Deep Blue-Kasparov chess match.
Dr. Rajat Mukherjee holds a Ph.D. and a Master of Science in Electrical and Computer Engineering from Rice University, as well as a Bachelor of Technology from the Indian Institute of Technology in Madras, India.
Interview Transcript
Eric Enge: Can you provide a brief introduction of yourself and how you got to be involved in the Custom Search Engine program at Google?
Rajat Mukherjee: Sure. I started at Google at the beginning of this year, and Custom Search was something that I thought was a very interesting product. I have been involved with several different projects as of my start date. But, this is one that I find very compelling and very powerful as a paradigm, to help people find domain-specific information on the web. I have been very excited to be a part of this project as one of my first projects at Google and the team I work with is outstanding.
Eric Enge: Where were you before you came to Google?
Rajat Mukherjee: I have been all over the map. I started my career at IBM Research, in technology, on the east coast and then I transferred to the Almaden Research Labs here on the west coast. I went to a startup for a year and then went to Verity, which is in enterprise search. I was there for about four and a half years. I switched to product management when I joined Yahoo. I was on the web search team there, and then I came to Google.
Eric Enge: Can you talk about how you see the vertical search market and the role that Custom Search Engines play in that market?
Rajat Mukherjee: A lot of organizations want to provide high-quality search to their users and there are solutions out there that they have been using, but Custom Search Engines enable a very high quality of search to be made available in a very simple-to-use manner. I think that there is a huge market out there and it is up to products like ours to tap into that market and satisfy the needs of those customers for both high-quality search and ease of use. I think we are doing great in terms of the excitement around the product, and the number of organizations that have approached us with feature requests and requirements, and are ready to take the product to the next level.
Eric Enge: Do you have any information you could share about how the program is going for Google?
Rajat Mukherjee: Sure. There is a lot of excitement around the product, internally at Google, as well as with our users. We have a very active community. We have more than 100K registered Custom Search Engines, and that’s growing pretty rapidly. We are also seeing significant traffic from specific custom search engines as well. So while I can’t go into a lot of details, I think that should give you an indication of the program, the excitement around it, and the success.
Eric Enge: Can you talk about some of the enhancements you have made recently?
Rajat Mukherjee: We have added improvements on internationalization, where you can actually restrict your search by domain, or by a specific language, and that’s been very popular with users, and there are several customs search engines that are now enabling search in different languages. Although the end user experiences are available in other languages, at this point, the administration interfaces are available only in English, so that’s something that we are looking at adding as well.
We have added statistics, which is again very popular, and there is a lot of demand for more statistics around Custom Search Engines. We have added the ability for people to create gadgets, for both custom search, as well as custom search administration, and these gadgets can be included in Google personalized homepages. And, that is something that we believe will make it much easier for people to quickly access the Custom Search experiences of their choice.
There is improved branding that we are rolling out to make it easier for people to format their pages, and have the Google brand associated with their search. That’s important because people really care about the quality of search, and I think that’s one of the compelling features of this product that makes it so successful. They get Google quality search, and then have the ability to further refine, or customize that. And, we are adding more control in terms of the user experience, and look and feel.
We are simplifying the co-branding, so now you are seeing a simple Google-branded logo on the custom search engine results pages.
Eric Enge: Yeah. So it’s a little trimmer, and more easily adapted on to various people’s web pages, because of course, the space that people have available for wherever they are going to put search box varies greatly from site to site.
Rajat Mukherjee: Right, not only does it satisfy that requirement, but it’s also a clearer branding of the product.
As for the other community features that I alluded to, we have this feature called refinements, where you can create different axes or different attributes around your search experience.
There are several organizations that are now using refinements to provide more control to their users to drill down into the search results. We are seeing a healthy mix of users who are using these advanced features. We see a variety of sites using the product, which is very exciting, and it is being used in very innovative and imaginative ways. I think the fact that users across various domains are using this product shows that we hit a sweet spot in terms of this need.
Eric Enge: Can you expand upon the internationalization plans?
Rajat Mukherjee: We are looking specifically at localizing the administration and console interfaces. This will allow more users and webmasters in these countries to easily use and manage their Custom Search Engines. This will make it much more attractive on an international basis. This is important to us because we are seeing interest in, and adoption of the product, all over the world.
Eric Enge: I see some really good opportunities for the collaboration capabilities of the Custom Search Engine product, and for groups of people to really bring a search engine to a much higher level of expertise. Do you have any examples that you can give of people who have done a good job with the collaborative search engine?
Rajat Mukherjee: We have increasingly larger organizations who ask us about some of these new features. For example, our current refinements feature is being used on several sites in very interesting ways; in the beginning many of them allow their own editors, to add new resources to the custom search index, or to add refinements for drill down, but increasingly they are looking at very innovative ways of allowing a broader audience to actually impact the search experience on their sites.
They are looking to expose some of these features to their communities. We are examining exactly how we should roll out some of these enhancements.
Eric Enge: Do you plan to provide more moderation features for the community environments and Custom Search Engines?
Rajat Mukherjee: Community environments typically have a very nice way of moderating themselves, so I am not sure if we are going to provide many moderation tools, at least in the first phase. In the first phase, we would like to expose more capabilities to users, and as we see innovative uses of these capabilities, we can evolve the product. We want to see what kind of usage there is, and then we can figure out what kind of moderation tools we might want to have in place.
Eric Enge: Right. Recently on the custom search blog, you had a post about a most popular list. In the list, I see sites like soundclick.com and e-words.jp for example. Are there any of those that you think are particularly interesting examples from an implementation point of view?
Rajat Mukherjee: I haven’t looked at those specific sites, because they change on a regular basis, but I think that the interesting thing is that all of the sites on our most popular list are early adopters, and they have been getting a lot of traffic. So, we do keep our eye on them to see the interesting things they are doing with the features that we have provided them. We’re expecting interesting requirements and innovative uses as larger organizations that have substantial resources further deploy these technologies.
I think that many of the needs and requirements will be driven by these larger organizations that are pushing for these features. They will make use of features like refinements in different ways, based on the sort of communities that they support.
Eric Enge: Have you added something along the way to allow subscribed links results to be shown in your Custom Search Engines?
Rajat Mukherjee: Yes, that feature of custom search is already available. You can define your own subscribed links for your Custom Search Engine. In fact, that’s one of the features that’s attractive to many users. For example, there are travel sites that want to provide their own promotions, in addition to custom search results for specific queries, and they want to then be able to promote that further to a larger audience as well. That feature is seeing a lot of attention and people are asking more about how they can make subscribed links available to a larger audience.
Eric Enge: I think subscribed links are a real hidden gem of the Co-op program. I think subscribed links are really under appreciated. If you are a site owner, and you have a visitor come to your site, and your objective is to get repeat traffic from people who visit your site; a subscribed link is an awesome way to do that.
Rajat Mukherjee: You are absolutely right. Subscribed links enable a very powerful paradigm, making it possible for publishers to specifically point their communities to specific pieces of information, and to also promote that information broadly. That’s one of the reasons why we want to use the vehicle of Custom Search, which is a very well understood and easy to use a product, to promote this as we move forward.
Eric Enge: How do Topics feature into Google’s plans at this point?
Rajat Mukherjee: The Topics features have a lot of power, and we are using these features as we go forward. Refinements are really an incarnation of Topics in custom search. With Topics, you are able to provide custom drill downs within the context of your own domain, or within the context of your own Custom Search Engine. You can see this on Google for health-related queries, for example.
Eric Enge: Two of the challenges with custom search are getting end users to recognize: (1) when they have actually encountered a Custom Search Engine, and: (2) that it’s better than the horizontal Google search platform that’s out there. Does that make sense?
Rajat Mukherjee: Yeah, that makes sense, but I don’t necessarily agree with you, because I don’t believe it’s important to explicitly let people know that they have a certain experience or a different one. I think the real value and power of custom search is to provide high-quality search at the point of inspiration, where the content resides. It is about users belonging to communities, and webmasters being able to engage those communities. It’s about providing search where it is required in the form that it is required in. It’s not necessarily that one is better or worse.
It is just that it is seamless in the sense that whenever you need to search, you have the ability to do a high-quality search that is then further refined, further instructed by your community, and further useful at that point. People do understand the information need, and the point is to satisfy that need where you know that need exists. People don’t really care where they are getting that information as long as the information is being retrieved when they need it.
Eric Enge: Right. But, what I am trying to get at here is that if I am a site owner, and I can put regular Google search on my site, it may create as much satisfaction for my users in sticking the Custom Search Engine on my site. I am not saying this as a criticism of CSEs, because I think they are great. The concern I am expressing is that the users may not recognize that the search engine they just used on a website was a Custom Search Engine, so it may not cause the users to come back looking for that search engine rather than simply using Google’s horizontal search. Is this an issue?
Rajat Mukherjee: Well, the premise you are making is that the horizontal search experience provides as much stickiness as the Custom Search Engine experience. And, I think the premise we (Google) are making is that if there is value to providing a customized search experience on a given site, then that could potentially provide more stickiness to those specific communities. Now of course, it is the choice of the content provider or webmaster to determine what is the best experience for their user base.
Eric Enge: Sure. And, to maximize stickiness, it seems to me that the search experience needs to be more than incrementally better than horizontal search to stick deeply in the users’ mind.
Rajat Mukherjee: I completely agree, and I think that these experiences, such as domain-specific refinements and domain-specific subscribed links, all lead to highly customized experiences for users. For example, if I am into kite flying, then I might be able to get a better experience if I go to a kite flying search engine that shares my specific interests or perspective on kite flying. Now, this is not to say that you can’t get information on kite flying in a normal Google search. It is just that Custom Search Engines are tuned differently based on editorial input, and that editorial input could be as narrow as one webmaster or as broad as an entire community of kite flyers.
For example, I went to the Oscars website this morning (the interview happened soon after Oscars weekend), and I could not find a search box there. So, you can immediately see that for people who are going to the Oscars site it could be tremendously useful, and potentially far more sticky, to provide a search experience that is tuned to that domain of interest. So, I think that if a Custom Search Engine is constructed correctly and configured correctly, it can provide a sticky experience to those users.
Eric Enge: I agree, and I think the people who are prepared to really invest in building that experience using their editorial expertise are the ones that are going to get the best results from the custom search engine program.
Rajat Mukherjee: That is correct, and many of them are looking to the power of their communities to further scale this.
Eric Enge: Right. One other thing that you recently announced was the integration of supplemental results. Can you comment on that?
Rajat Mukherjee: Most of the major search engines have some degree of tiering in their indexes. Supplemental results constitute one of these tiers, and typically if you had certain queries that don’t hit in the main part of the index, they retrieve results from the supplemental parts of the index. For Custom Search Engines on smaller sites or more obscure domains of interest, it is necessary to go deeper to retrieve relevant information. So, it’s all about relevance, and accessing the supplemental index is important for improving the relevance on some of these queries.
Eric Enge: Right. So, you have heard a lot of demand from the industry at large for the supplemental data to be included?
Rajat Mukherjee: Yeah, it was a specific request from various webmasters, who wanted this higher degree of relevance in the custom search program. And, it was just a matter of time before we added that.
Eric Enge: Right. Do you have any thoughts on how the vertical search market will mature and grow over time?
Rajat Mukherjee: I have lots of personal opinions on this topic. So, I can express some of my personal thoughts on this. Now, the first question is, when you say vertical search what exactly do you mean? Because that term means a lot of different things to different people. Some people think local search, news search, blog search, video search, image search are all vertical search. Sometimes when we talk about vertical search, we are really talking about different kinds of verticals; such as topic-based verticals, e.g., health. So, vertical search itself means different things to different people. That said, I think all major search engines are trying to provide a better-tuned search experience to their users at large. There are all sorts of features on Google, such as personalized search, that allow you to further refine search, and better understand the intent of the users.
When you talk about verticals, I think there are two aspects. One question is can you provide high-quality search at the point of the content? When you have specific domains defined in terms of content, providing search at that specific point of interest is very, very important. It’s important for the webmasters and content publishers because that increases engagement of their users on their sites. But, it is also important for the users because they find contextually rich information based on their interests. I think that this sort of distribution of search is very important in the long term.
Now, whether or not this means that you will have hundreds of search engines that people go to for different types of needs, for example, news, blogs, video, audio, etc., that is a different question. I don’t know if that’s going to happen, that is, a plethora of large search engines, each focusing on specific verticals. My belief is that you are going to see fewer destinations for this type of search.
Custom Search Engines provide a different function, which is providing search at the point of inspiration.
Eric Enge: This really speaks to the notion of distributed search, right?
Rajat Mukherjee: Yes.
Eric Enge: There is also a whole family of people or companies out there that are launching new kinds of search engines with their own crawls and their own tailored algorithms. This is more the notion of creating a new kind of destination search.
Rajat Mukherjee: Right. And, I think that search as a destination is going to be harder and harder to succeed at over the long term. I think that the technology and infrastructure needs; e.g., the capital expenditure needs of providing comprehensive high-quality search, are non-trivial. I do know that there will be startups that will innovate, but, I think the leading search engines will provide these capabilities as well. So, I think that vertical search as we know it, as we sometimes talk about it, which is local search or shopping search, etc., will be getting encapsulated in generic search.
For example, a lot of people use merchant sites to shop for products, but more and more of them are actually just going to search engines and searching for products right from there. So, I hope that makes my point.
Eric Enge: Sure. So, the goal is to have a distributed model so they can find a high-quality search experience right there at the point of inspiration.
Rajat Mukherjee: Right. They can get Google quality results right there, without having to go to Google.com. So, whether it’s a search on a Custom Search Engine, which is a slice of the Google index, or Google.com, it doesn’t really matter as long as the user gets what they are looking for in as short a time as possible.
Eric Enge: Can you make any comments about future plans from the Google Custom Search Engine group?
Rajat Mukherjee: Yeah, sure. First, as we come out with new features, we do talk about them on our blog. So, users can see our blog at: googlecustomsearch.blogsspot.com is where we announce new things that come up.
We are definitely looking at ease of use, configuration, and management of Custom Search; that is something that’s near and dear to our hearts. Not only do we want a lot of users using it, we also want to make it broader in terms of appeal to a larger audience – not just the techies, but also to consumers in general.
We are also often asked about more customization of the look and feel and layout of the search results. That is something that we are constantly working on, and you will see some of those enhancements over time.
We are looking at things like more statistics and more language support.
We will be trying to provide a generic way to integrate results from multiple sources. At a high level, you can consider that federated search, and we are trying to figure out what the right model is going forward. We do already have some of those capabilities built as I mentioned. Subscribed links are one way of getting that information in place. But, we want to look at different models where we can provide this capability in a more generic fashion.
We are also looking to enhance the community features as we move forward. So, I hope that kind of gives you a flavor for some of the things that we are looking at.
Eric Enge: Can you comment on the business problems that Custom Search Engines are solving?
Eric Enge: Okay.
Rajat Mukherjee: One of the things that I would like to draw attention to is the fact that the very premise of this product is high-quality search. You get the benefits of all the data that Google has about all the different sites out there. So various attributes like PageRank, anchor text, etc. that Google has access to are now being made available to the slice of our index that you care about. That gives you high quality out of the box. In fact, many organizations are replacing their existing search solutions with Custom Search because they see better results quality in fifteen minutes. I think that’s a very important business value proposition, where the communities are basically getting a better search experience. The second thing is that it’s so easy to use, and has such a low cost of ownership that organizations that didn’t previously conceive providing search on their websites, because it was too expensive or too complex to do right, are now able to do that.
Eric Enge: Right.
Rajat Mukherjee: I think that’s one of the reasons for the success of the product as well. And, the other one that’s important to understand is that this provides a customized experience for communities. They can actually get much more engagement and involvement from their own users, and I think that’s good for business in general. And, of course, monetization is very important; businesses can actually monetize their traffic, and benefit from that. Using a high-quality advertising network is something that people find attractive.
Eric Enge: Can you help us wrap up by telling our readers where they can get updates on Custom Search Engines?
Rajat Mukherjee: Sure. Users can see our blog at: googlecustomsearch.blogsspot.com. This is where we announce new things that come up.
Eric Enge: That’s all for today. Thank you very much.
Rajat Mukherjee: Thank you.
