are welcome and which need to move on to that less-exclusive joint down the street.
can admit or reject robots on a sitewide, directory-by-directory, or page-by-
SEO folks often feel a special affection for the
file because it provides
a rare opportunity to communicate with a search engine robot. However, its capabili-
ties are really very limited.
files exist only to
indexing. Just as a
bouncer can keep people out but can’t force anyone to come in, the
do anything to entice a robot to spend
time or visit
pages on your site.
Also, compliance with your
file is voluntary, not mandatory. The major
search engines will generally try to follow your instructions, but other, less-reputable
types might not. This is why you should not rely on your
file to prevent spi-
dering of sensitive, private, or inappropriate materials.
Do You Need a
You may not need a
file. Without one, all robots will have free access to non-
password-protected pages on your site. To decide if you need a
file for your
website, ask yourself these questions:
• Are there any pages or directories on my site that I do not want listed on the
search engines, such as an intranet or internal phone list?
• Are there any specific search engines that I do not want to display my site?
• Do I know of any dynamic pages or programming features that might cause
problems for spiders, like getting caught in a loop (infinitely bouncing between
• Does my website contain pages with duplicate content? (These should not be
indexed or you may be penalized.)
• Are there directories on the site that contain programming scripts only, not
If the answers to these questions are no, then you do not need a
You’ve got the rest of the day off! If you have any yes answers, you’ll prepare your
files are very simple text files. To find a sample, go to
and view ours, or go to just about any other site and look for the
file in the
Determine whether you need a
file in your website.
c08. 8:08 202