Opening Search Pages for Indexation & SEO – A Data-Driven Approach

If you’re running an e-commerce website, you probably have your customers searching for your products on your site. In most instances, you’ve likely closed those search pages to search engine indexation, since they may be duplicative of some category pages. However, you may not realize that search pages often fill a gap in your category page structure with a wealth of query-relevant content. If done strategically, opening the right search pages to indexation can significantly benefit your organic traffic.

In this article, you’ll learn how to run the necessary analysis for opening query-aligned search pages to improve organic traffic performance. Incorrectly opening search pages to indexation can spell disaster, so read and follow the best practices laid out when attempting this type of advanced SEO optimization.

Identify Your Gaps

To begin, identify your content gaps. Where do you have product inventory but no category page that displays it? For example, you may carry plenty of widgets XYZ but no category page with that inventory available.

Keyword research can help you understand what people are looking for. Take that information and compare the query demand with your available category pages that are a good match. Once you know your inventory, you can identify missing pages, which products aren’t covered by existing category pages, and where your customer demand lies.

Tools like SEMRush help identify these gaps. To make analysis tools easier to use, keep your category pages in a single folder. Enter your category pages and the category pages of your competitors. The gap analysis tools show you which keywords your competitors are ranking for that you’re not, so you can determine if you have enough product inventory to fulfill that need.

Open Search Pages to Indexation

Once you’ve identified your keyword gaps, determine the best way to fill them:

• Pull a list of keywords that your category pages rank for.
• Open a specific set of search pages that fill any keyword gaps not currently filled by category pages
• Let Google index those search pages.

Remember, opening search pages to indexation when an existing category page already meets the same query demand can damage your SEO. Presenting a category page and search page that target the same query and contain the same content dilutes signals for Google and can harm your overall ranking potential.

Excel is a great tool for the type of comparative analysis you’ll need to perform. Using Excel lets you see which of your category pages aren’t ranking in the top 25 or so positions for the specific keywords identified in the gap analysis. Once you understand where you can support unmet demand, you’ll know which search pages to open for indexation.

How to Open Search Pages for Indexation?

A nonindex tag or disallow rule in robots.txt may cut search pages off from indexation. However, a noindex tag requires Google to crawl your pages to discover the tag and may be a waste of crawl budget.

In either case, opening search pages to crawling requires you to either remove the disallow rule from your robots.txt file or remove the noindex tag. It’s also a good idea to add a self-referential canonical tag to the pages. The canonical tag will send an added signal to Google that the pages are intended for indexation.

Getting New Search Pages Indexed

By now, you’ve opened a very targeted set of search pages to indexation that fill obvious demand where a category page isn’t created. After a few days, you find Google hasn’t indexed any of them yet. Why?

Google is cautious about indexing new pages if it thinks they were accidentally opened for indexation. In this case, the changes were very intentional. So, how do you convince Google to start indexing your new pages?

You have several options:

• Request indexation manually in Google Search Console (GSC)
  • This isn’t a great option if you’ve opened hundreds or thousands of pages
  • This tool is temporarily unavailable as of Oct. 14, 2020 (Source)
• Create an XML sitemap and submit it in GSC, or add them to your existing XML sitemap
  • This is an SEO best practice for any new indexable pages
  • However, this alone may not be enough to convince Google to index your pages
• Create an HTML sitemap to get the pages into the crawl path
  • This is a reasonable option, as well
  • However, Google has indicated that they do not pass page rank through HTML sitemap links, so this may be considered a weak signal
• Link your new pages to already indexed and ranking pages
  • This is your best bet but requires extra effort
  • Look for pages that are relevant to the new search pages, and add a link from that page to the new one
  • On a larger scale, look for category pages with an existing template where you can add a block of related links
    • Tagging or another automated system for associating links is required, but it can work well once it’s done

Finally, be patient. Google can take weeks or months to decide that your new search pages are worth indexing. However, the effort is worth it once they’re indexed and have the potential to drive organic search traffic.

Once your newly targeted search pages are indexed, it’s time to track performance. It’s best to track keyword ranking and traffic. Add the keywords you identified in your gap analysis into your favorite tracking tool to monitor over time. Page clicks and impressions in Google Search Console are also good to watch.