Educating Googlebot and (hopefully, in the future) other crawlers too

Steering SE Crawlers · Index · Expand · Web Feed

Previous PageSearch Engine Friendly Query Strings

Recap: Methods to Support Search Engines in Crawling and RankingNext Page

If you're not familiar with Google's Sitemap Service, please read the tutorial How to make use of Google SiteMaps first. Google's sitemap protocol, offered under the terms of the Attribution-ShareAlike Creative Commons License, is open for other search engines. A big player like Google has the power to talk its competitors into using this protocol. Thus, ignoring the 'beta' label, which is there to stay for a while, most probably we're discussing a future standard here.

For large dynamic web sites, Google SiteMaps is the instrument of choice to improve the completeness of search engine crawls. However, each and every web site out there should make use of Google SiteMaps. SEO firms should develop dynamic Google SiteMaps, which are fully automated and reflect a site's current state on every request by a search engine crawler, for all their clients. Content management systems should come with this feature build-in.

In all the euphoria caused thru Google's launch of sitemaps, do not forget to read the fine prints. Webmasters providing a Google SiteMap containing all crawlable URLs shall not become lazy. Google SiteMaps do not replace established methods of web site crawling, they do not affect rankings, and they do not guarantee spidering and inclusion in a search engine's index. Google SiteMaps do give webmasters an opportunity to inform search engine crawlers about fresh content, relative priorities and change frequencies per URL, but they are hints, not commands.

Google SiteMaps should be used as an additional tool for steering and supporting search engine crawling. Provided all page attributes in the sitemap XML file(s) were populated honestly, search engine crawlers should learn to fetch even deeply buried content in-time, while seldom touched URLs get scheduled in less frequent crawls. Webmasters should not expect too much in the beginning. Taking the learning curve effect into account, most probably search engine crawlers will have to perform sitemap based crawls in many iterations, before they come close to 'perfection'. Also, webmasters should not forget, that every now and then search engines might have a very special understanding of importance, which is pretty different from a site owner's point of view.

Recap: Methods to Support Search Engines in Crawling and RankingNext Page

Previous PageSearch Engine Friendly Query Strings

Steering and Supporting Search Engine Crawling · Index · Part 1 · 2 · 3 · 4 · 5 · 6 · 7 · 8 · 9 · 10 · Expand · Web Feed

Author: Sebastian
Last Update: Monday, June 20, 2005   Web Feed

· Home

· Internet

· Steering SE Crawlers

· Googlebot-Spoofer

· Google Sitemaps Info

· Web Links

· Link to us

· Contact

· What's new

· Site map

· Get Help

Most popular:

· Site Feeds

· Database Design Guide

· Google Sitemaps

· smartDataPump

· Spider Support

· How To Link Properly

Free Tools:

· Sitemap Validator

· Simple Sitemaps

· Spider Spoofer

· Ad & Click Tracking

Search Google
Web Site

Add to My Yahoo!
Syndicate our Content via RSS FeedSyndicate our Content via RSS Feed

To eliminate unwanted email from ALL sources use SpamArrest!


neat CMS:
Smart Web Publishing

Text Link Ads

Banners don't work anymore. Buy and sell targeted traffic via text links:
Monetize Your Website
Buy Relevant Traffic

[Editor's notes on
buying and selling links

Digg this · Add to · Add to Furl · We Can Help You!

Home · Categories · Articles & Tutorials · Syndicated News, Blogs & Knowledge Bases · Web Log Archives

Top of page

No Ads

Copyright © 2004, 2005 by Smart IT Consulting · Reprinting except quotes along with a link to this site is prohibited · Contact · Privacy