Steering and Supporting Search Engine Crawling

Since Google launched the sitemap protocol in June 2005, webmasters and search engine optimizers have to rethink their dealing with search engine crawlers. Assuming Google will remove the sitemap protocol's 'beta' label faster than from it's search engine, this article tries to give web site owners an idea where to place Google sitemaps in their toolset. To define the playground, this article starts as a tutorial on supporting and steering search engine crawlers.

Index

[Expand]

Basic Search Engine Crawler Support

Supporting crawlers in indexing a web site

Identifying and Tracking SE Crawling

Detecting search engine spiders, tracking and analyzing their behavior

The Gatekeeper: robots.txt

Preventing search engine crawlers from fetching particular files and directories

URL Specific Control: the Robots META Tag

Telling search engine spiders how to index and cache a particular page

Link Specific Regulation: REL=NOFOLLOW

Hindring search engines to interpret a link as a vote for the link target

Tagging irrelevant page areas: class=robots-nocontent

How to make cluttered page areas like blocks with ads unsearchable. The class name robots-nocontent can be applied to everything not related to the page's main content.

User and Crawler Friendly Navigation

Leading search engine bots to the content they shall index

Search Engine Friendly Query Strings

If you can't avoid query strings in URLs, keep them short

What Google's Sitemap Protocol May Change

Educating Googlebot and (hopefully, in the future) other crawlers too

Recap: Methods to Support Search Engines in Crawling and Ranking

Webmaster's toolset to support and control search engine spiders

Author: Sebastian
Last Update: Monday, June 20, 2005 Web Feed

· Home

· Internet

· Steering SE Crawlers