If you’re running an e-commerce website, you probably have your customers searching for your products on your site. In most instances, you’ve likely closed those search pages to search engine indexation, since they may be duplicative of some category pages. However, you may not realize that search pages often fill a gap in your category page structure with a wealth of query-relevant content. If done strategically, opening the right search pages to indexation can significantly benefit your organic traffic.
In this article, you’ll learn how to run the necessary analysis for opening query-aligned search pages to improve organic traffic performance. Incorrectly opening search pages to indexation can spell disaster, so read and follow the best practices laid out when attempting this type of advanced SEO optimization.
Identify Your Gaps
To begin, identify your content gaps. Where do you have product inventory but no category page that displays it? For example, you may carry plenty of widgets XYZ but no category page with that inventory available.
Keyword research can help you understand what people are looking for. Take that information and compare the query demand with your available category pages that are a good match. Once you know your inventory, you can identify missing pages, which products aren’t covered by existing category pages, and where your customer demand lies.
Tools like SEMRush help identify these gaps. To make analysis tools easier to use, keep your category pages in a single folder. Enter your category pages and the category pages of your competitors. The gap analysis tools show you which keywords your competitors are ranking for that you’re not, so you can determine if you have enough product inventory to fulfill that need.
Open Search Pages to Indexation
Once you’ve identified your keyword gaps, determine the best way to fill them:
- Pull a list of keywords that your category pages rank for.
- Open a specific set of search pages that fill any keyword gaps not currently filled by category pages
- Let Google index those search pages.
Remember, opening search pages to indexation when an existing category page already meets the same query demand can damage your SEO. Presenting a category page and search page that target the same query and contain the same content dilutes signals for Google and can harm your overall ranking potential.
Excel is a great tool for the type of comparative analysis you’ll need to perform. Using Excel lets you see which of your category pages aren’t ranking in the top 25 or so positions for the specific keywords identified in the gap analysis. Once you understand where you can support unmet demand, you’ll know which search pages to open for indexation.
How to Open Search Pages for Indexation?
A nonindex tag or disallow rule in robots.txt may cut search pages off from indexation. However, a noindex tag requires Google to crawl your pages to discover the tag and may be a waste of crawl budget.
In either case, opening search pages to crawling requires you to either remove the disallow rule from your robots.txt file or remove the noindex tag. It’s also a good idea to add a self-referential canonical tag to the pages. The canonical tag will send an added signal to Google that the pages are intended for indexation.
Getting New Search Pages Indexed
By now, you’ve opened a very targeted set of search pages to indexation that fill obvious demand where a category page isn’t created. After a few days, you find Google hasn’t indexed any of them yet. Why?
Google is cautious about indexing new pages if it thinks they were accidentally opened for indexation. In this case, the changes were very intentional. So, how do you convince Google to start indexing your new pages?
You have several options:
- Request indexation manually in Google Search Console (GSC)
- This isn’t a great option if you’ve opened hundreds or thousands of pages
- This tool is temporarily unavailable as of Oct. 14, 2020 (Source)
- Create an XML sitemap and submit it in GSC, or add them to your existing XML sitemap
- This is an SEO best practice for any new indexable pages
- However, this alone may not be enough to convince Google to index your pages
- Create an HTML sitemap to get the pages into the crawl path
- This is a reasonable option, as well
- However, Google has indicated that they do not pass page rank through HTML sitemap links, so this may be considered a weak signal
- Link your new pages to already indexed and ranking pages
- This is your best bet but requires extra effort
- Look for pages that are relevant to the new search pages, and add a link from that page to the new one
- On a larger scale, look for category pages with an existing template where you can add a block of related links
- Tagging or another automated system for associating links is required, but it can work well once it’s done
Finally, be patient. Google can take weeks or months to decide that your new search pages are worth indexing. However, the effort is worth it once they’re indexed and have the potential to drive organic search traffic.
Once your newly targeted search pages are indexed, it’s time to track performance. It’s best to track keyword ranking and traffic. Add the keywords you identified in your gap analysis into your favorite tracking tool to monitor over time. Page clicks and impressions in Google Search Console are also good to watch.
Hello John, you picked a vital topic for all bloggers and content marketers. Being a digital marketer and agency owner, I, too, sometimes find it difficult to index pages quickly, especially if the content is authored on another blog. Apart from what you have suggested here, like request indexing on the GSC tool, sitemap updating both XML and HTML, and internal linking, we also create a few short snippets of the content and publish them on Quora, growth hackers, etc., with a link to the main page. We always found this helps but still would like to know your opinion on this. Can you please shed light on this?
Hi Soumya, thanks for your comment. Apologies for the delay in my response. When posting on sites like Quora, keep in mind the importance of making them valuable in their own right.