One of the questions you see swirling about the forums and blogs these days is whether or not you should noindex your RSS feeds to avoid duplicate content problems. The source of the problem is that RSS feeds are being crawled by the search engines. In addition, many people are now recommending that you include the entire content of your articles directly in your feed. You can read more about this in my recent interview of Rick Klau of FeedBurner.
So if the search engine sees the content on your site, and also sees it in your feed, will that be seen as duplicate content? There certainly have been instances of finding RSS feeds in the search results of the search engines. For those of you worried about this possibility, you can see Yahoo’s spec for Noindexing RSS feeds here. My understanding is that both Google and Yahoo! will honor this Noindex request of your feed.
But let’s step back, and think about how search engines try to deal with duplicate content. They are always trying to figure out who the authoritative source of the content is. It should be pretty obvious to the search engine when it’s crawling an RSS feed, and it should also be relatively obvious that the RSS feed for a site’s content is not the authoritative search.
In my interview with Rick Klau, and my earlier conversation with Google’s Adam Lasnik, we talked about this issue.
Rick points out, as I did above, that it’s really obvious to a search engine when it is crawling a feed, and that feeds are not the authoritative source of content. In addition, providing a feed often helps a search engine more rapidly find new content on your site.
Adam Lasnik commented that during his time at Google that he had not heard of any instances of a site being negatively affected by a duplicate content issue with an RSS feed.
Based on this input, I would conclude that there is extremely little risk in letting your feed get indexed. After I first learned about the issue, I did move forward and Noindex our feeds here at Perficient Digital. But then, after the conversations with Rick and Adam, we became convinced that it’s just not a problem, and this is the recommendation we make to our clients as well.
