Want to control those pesky bots? There are techniques available to help you control what they crawl, and what they index. This article will discuss the basics and point you at some resources for more information. Robots.txt Overview The most basic technique is the “robots.txt” file. This file allows you to tell search engine robots […]
Posts Tagged ‘Robot.txt’