Googlebot downloads my XML sitemap frequently, but didn't crawl and index all pages of my Web site yet. What can I do to improve my search engine visibility?

Google Sitemaps KB · Index · Expand · Web Feed

Previous PageGoogle's time to index

Google Python Sitemap Generator - IntroductionNext Page

To understand why a Web site isn't fully indexed by Google, despite a complete Google Sitemap, it helps to classify the pages by link popularity. Note that link popularity is only one minor factor in Google's rankings, and that your analysis will not lead to the same results as Google's high sophisticated algorithms, but it is a good point to start with, even without proper weighting of your inbound links.

Fire up a backlink checker or site explorer to track Web links pointing to your pages. Ignore useless links from scraper sites like DMOZ clones, low ranked forums, blogs and alike. Count only links from high quality sites and popular blogs, and make a note of how many of these links are topically related to the linked page on your site. Internal links from within your site do count too.

Sort your spreadsheet by the number of related inbound links. You'll spot a correlation between inbound links and search engine visibility.

  • Fairly ranking pages will have attracted the most valuable inbound links, and prominent internal links.
  • Pages without external inbound links, but one or more prominent internal links, will be indexed (findable with a search, appearing with title and a text snippet on the SERP), but don't rank that well for their desired keywords.
  • Pages where the sole inbound links come from navigational page elements on secondary (or deeper) levels (more than one click away from the home page or another high ranking point of entry), most probably were crawled (fetched by Googlebot), but in many cases not (fully) indexed, or they are shown as URL-only listing on the SERPs of a site-search.
  • Pages without inbound links get (sometimes) crawled from the Google Sitemap, they even may appear for a week or two on the SERPs, but get dumped into the supplemental index or even removed, if they don't attract links.

(Results may vary depending on type, size, age, theme and overall popularity of your site.)

The first option to enhance your unlinked pages' crawlability and popularity is to tweak your site's navigation and other internal linkage. For example create static HTML site maps with no more than 100 links per page. The results will be limited, but it has to be done. Make sure your site is perfectly crawlable, and study Google's SEO guidelines.

The next thing to do is a content review of poorly performing pages. If a page is worth the efforts, then add more interesting and original content, enhance non-textual content with well written descriptions, and so on. Then search the Web for other great pages on the page's topic and link to the best ones from within your textual content (more info on link placement). Don't consider PageRank, nor link to high ranking pages only. Add value to your pages by recommending great resources which are most likely interesting for your visitors. Webmasters finding your page in their referrer stats will naturally place a link back, if they like your content. Be creative.

With a commercial site you'll need to adapt the above said, or hire a competent SEO.

Monday, December 19, 2005 by Sebastian

Google Python Sitemap Generator - IntroductionNext Page

Previous PageGoogle's time to index

Google Sitemaps Knowledge Base · Index · Part 1 · 2 · 3 · 4 · 5 · 6 · 7 · 8 · 9 · 10 · 11 · 12 · Expand · Web Feed

Author: The Google Sitemaps Group
Last Update: December 10, 2005   Web Feed

· Home

· Google Sitemaps Guide

· Google Sitemaps KB

· Google Sitemaps Info

· Google Sitemaps FAQ

· Web Links

· Link to us

· Contact

· What's new

· Site map

· Get Help

Most popular:

· Site Feeds

· Database Design Guide

· Google Sitemaps

· smartDataPump

· Spider Support

· How To Link Properly

Free Tools:

· Sitemap Validator

· Simple Sitemaps

· Spider Spoofer

· Ad & Click Tracking

Search Google
Web Site

Add to My Yahoo!
Syndicate our Content via RSS FeedSyndicate our Content via RSS Feed

To eliminate unwanted email from ALL sources use SpamArrest!


neat CMS:
Smart Web Publishing

Text Link Ads

Banners don't work anymore. Buy and sell targeted traffic via text links:
Monetize Your Website
Buy Relevant Traffic

[Editor's notes on
buying and selling links

Digg this · Add to · Add to Furl · We Can Help You!

Home · Categories · Articles & Tutorials · Syndicated News, Blogs & Knowledge Bases · Web Log Archives

Top of page

No Ads

Copyright © 2004, 2005 by Smart IT Consulting · Reprinting except quotes along with a link to this site is prohibited · Contact · Privacy