Supporting crawlers in indexing a web site

Steering SE Crawlers · Index · Expand · Web Feed

Identifying and Tracking SE CrawlingNext Page

If you want SE spiders to fetch your content, the most important hint to a crawler is a link, known by the search engine, pointing to the page. Other hints are URL submissions, unlinked URLs found on the web, and perhaps even now still directory indexing. SEs consider pages with no incoming links pretty useless and usually don't bother indexing them (by the way, a page without outgoing links may be considered useless too). That means, forget submitting your stuff to the major search engines and concentrate your efforts on linkage.

To attract SE spiders, acquire valuable inbound links from related web sites. To keep SE crawlers interested in your site, provide a natural link schema, avoiding too many hops to the last page in the hierarchy. Search engine web robots are designed to find valuable content for search engine users. Ranking algorithms analyze a site's internal linking and honor an easy and user friendly navigation. There is nothing to say against a few shortcuts implemented for robots, but you really should try to design a navigation scheme that leads both users as well as crawlers on the shortest way to the content deeply buried in the site's hierarchy.

Think of the search engine crawler as a user. Build your site comfortable for your visitors, then implement special crawler support where it is needed. Steering and supporting search engine crawling basically is done by steering and supporting visitors on their way to your content they are interested in.

Stay away from cloaking if you're keen on free and highly targeted search engine traffic. Do not deliver 'search engine optimized versions' of your pages to crawlers. Feed spiders with the page as seen by users. There are very few tolerated exceptions from this rule, for example geo targeting and hiding user tracking from robots.

Identifying and Tracking SE CrawlingNext Page

Steering and Supporting Search Engine Crawling · Index · Part 1 · 2 · 3 · 4 · 5 · 6 · 7 · 8 · 9 · 10 · Expand · Web Feed

Author: Sebastian
Last Update: Monday, June 20, 2005   Web Feed

· Home

· Internet

· Steering SE Crawlers

· Googlebot-Spoofer

· Google Sitemaps Info

· Web Links

· Link to us

· Contact

· What's new

· Site map

· Get Help

Most popular:

· Site Feeds

· Database Design Guide

· Google Sitemaps

· smartDataPump

· Spider Support

· How To Link Properly

Free Tools:

· Sitemap Validator

· Simple Sitemaps

· Spider Spoofer

· Ad & Click Tracking

Search Google
Web Site

Add to My Yahoo!
Syndicate our Content via RSS FeedSyndicate our Content via RSS Feed

To eliminate unwanted email from ALL sources use SpamArrest!


neat CMS:
Smart Web Publishing

Text Link Ads

Banners don't work anymore. Buy and sell targeted traffic via text links:
Monetize Your Website
Buy Relevant Traffic

[Editor's notes on
buying and selling links

Digg this · Add to · Add to Furl · We Can Help You!

Home · Categories · Articles & Tutorials · Syndicated News, Blogs & Knowledge Bases · Web Log Archives

Top of page

No Ads

Copyright © 2004, 2005 by Smart IT Consulting · Reprinting except quotes along with a link to this site is prohibited · Contact · Privacy