Skip to main content

Posts Tagged ‘Robot.txt’

Robots.txt and Robots Metatags

Want to control those pesky bots? There are techniques available to help you control what they crawl, and what they index. This article will discuss the basics and point you at some resources for more information. Robots.txt Overview The most basic technique is the “robots.txt” file. This file allows you to tell search engine robots […]