ou’re nearing the end of the book, so you should have a pretty good
handle on what exactly robots, spiders, and crawlers are, right? No
doubt you do, but did you know that there is much more to these
Internet creatures than just the fact that they crawl from one web site to
Spiders, robots, crawlers, or whatever else you choose to call them can deter-
mine how well you rank in search engines, so it’s best to make friends with
them as quickly as possible. Certain strategies will help you find favor with
these crawlers (which is the name we’ll use to lump them all together), and
others, unfortunately, will help you find your way right out of search engine
A sad fact of crawlers is that sometimes you’ll be treated unfairly when you’ve
done absolutely nothing wrong. Some of the strategies that you may want to
use to improve the optimization of your site can look suspicious to crawlers.
They’ll see those strategies and automatically penalize you for trying to spam
the search engine, even when you’re not.
That’s what makes it so important for you to understand not only the intimate
details about how crawlers work, but also about what makes them happy and
what makes them move on without looking back.
IN THIS CHAPTER
What are robots, crawlers, and
What’s the Robot Exclusion
Robots meta tag
Inclusion with XML site mapping
16 1 9:55 227