Under Google’s hood

I recently tried Google Sitemaps, and found it to be a great way to see your website through Google’s eyes. The core purpose of Google Sitemaps is for website owners to feed Googlebot an XML sitemap listing URLs for all pages on their site (including those that may be otherwise inaccessible to robots), but there are many other features that web developers will appreciate.

Google Sitemaps features

Some of the standout features of Sitemaps are:

  • Check the date that Googlebot last crawled your site
  • View any errors (404, restricted by robots.txt, timeouts) Googlebot encountered while crawling your site
  • See which search queries are driving traffic to your site
  • Check page rank distribution across your site
  • See the words other sites use to link to your content

Google Sitemaps screenshot - click to enlarge

These tools offer an ‘under the hood’ look at the way Google sees your website, and can pinpoint problem areas. I recently redesigned a website that historically had barely registered on Google’s radar. I couldn’t figure out why until Google Sitemaps informed me that the site’s robots.txt file was disallowing all user agents from spidering the site. I guess the previous developer didn’t realise that search engine spiders are robots too!

Google Sitemap Protocol

For websites with difficult to index content, such as pages that are only accessible after completing a form, Google Sitemaps offers a great tool for feeding Googlebot URLs for all pages of the site. This functionality comes in the form of an XML sitemap formatted according to the Google Sitemap Protocol. The Sitemap protocol is a supplement to the way Googlebot crawls your site, and does not replace the conventional crawl-based approach.

Google offers it’s own tool for generating sitemap files, a python script that can run as a cron job, and will inform Google when pages on your site have changed. There are also plenty of third party tools to spider your site and generate sitemap files. One that I found easy to use is GSitemap, a free app for Windows.

Creating an account

Signing up for Sitemaps is easy, you just need a Google account. After registering you can add as many sites as you please to your Sitemaps account. A simple verification procedure allows Google to check that you really do own the site in question – it’s just a case of adding a Google generated meta tag to your site’s root level document.

For new websites Google Sitemaps is a great way to keep an eye on search engine performance, and I highly recommend giving it a spin.