Webmaster's toolset to support and control search engine spiders

Steering SE Crawlers · Index · Expand · Web Feed

Previous PageWhat Google's Sitemap Protocol May Change

Let's recap the basic methods of steering and supporting search engine crawling and ranking:

  • Provide unique content. A lot of unique content. Add fresh content frequently.
  • Acquire valuable inbound links from related pages on foreign servers, regardless of their search engine ranking. Actively acquire deep inbound links to content pages, but accept home page links. Do not run massive link campaigns if your site is rather new. Let the amount of relevant inbound links grow smoothly and steadily to avoid red-flagging.
  • Put in carefully selected outbound links to on-topic authority pages on each content page. Ask for reciprocal links, but do not dump your links if the other site does not link back.
  • Implement a surfer friendly, themed navigation. Go for text links to support deep crawling. Provide each page at least one internal link from a static page, for example from a site map page.
  • Encourage other sites to make use of your RSS feeds and alike. To protect the uniqueness of your site's content, do not put text snippets from your site into feeds or submitted articles. Write short summaries instead and use a different wording.
  • Use search engine friendly, short but keyword rich URLs. Hide user tracking from search engine crawlers.
  • Log each crawler visit and keep these data forever. Develop smart reports querying your logs and study them frequently. Use these logs to improve your internal linking.
  • Make use of the robots exclusion protocol to keep spiders away from internal areas. Do not try to hide your CSS files from robots.
  • Make use of the robots META tag to ensure that only one version of each page on your server gets indexed. When it comes to pages carrying partial content of other pages, make your decision based on common sense, not on any SEO bible.
  • Use rel="nofollow" in your links, when you cannot vote for the linked page (user submitted content in guestbooks, blogs ...). Do not hoard PageRankô.
  • Make use of Google SiteMaps as a 'robots inclusion protocol'.
  • Do not cheat the search engines.

  • Previous PageWhat Google's Sitemap Protocol May Change

    Steering and Supporting Search Engine Crawling · Index · Part 1 · 2 · 3 · 4 · 5 · 6 · 7 · 8 · 9 · 10 · Expand · Web Feed

    Author: Sebastian
    Last Update: Monday, June 20, 2005   Web Feed

    · Home

    · Internet

    · Steering SE Crawlers

    · Googlebot-Spoofer

    · Google Sitemaps Info

    · Web Links

    · Link to us

    · Contact

    · What's new

    · Site map

    · Get Help

    Most popular:

    · Site Feeds

    · Database Design Guide

    · Google Sitemaps

    · smartDataPump

    · Spider Support

    · How To Link Properly

    Free Tools:

    · Sitemap Validator

    · Simple Sitemaps

    · Spider Spoofer

    · Ad & Click Tracking

    Search Google
    Web Site

    Add to My Yahoo!
    Syndicate our Content via RSS FeedSyndicate our Content via RSS Feed

    To eliminate unwanted email from ALL sources use SpamArrest!


    neat CMS:
    Smart Web Publishing

    Text Link Ads

    Banners don't work anymore. Buy and sell targeted traffic via text links:
    Monetize Your Website
    Buy Relevant Traffic

    [Editor's notes on
    buying and selling links

    Digg this · Add to del.icio.us · Add to Furl · We Can Help You!

    Home · Categories · Articles & Tutorials · Syndicated News, Blogs & Knowledge Bases · Web Log Archives

    Top of page

    No Ads

    Copyright © 2004, 2005 by Smart IT Consulting · Reprinting except quotes along with a link to this site is prohibited · Contact · Privacy