Monday, August 16, 2010

Notifying Major Search Engines with sitemap_gen.py

If you're using sitemaps on your website, you can notify the major search engines whenever your sitemap is updated. They usually offer a "ping" URL that you can hit. The ping URLs for the major search engines are below. You would substitute the full URL to your sitemap.xml (or .gz) file, encoded, wherever you see sitemap_url below. And read the note at the bottom of this post for more info on the YahooDemo appid used below:

  • http://www.google.com/webmasters/tools/ping?sitemap=sitemap_url

  • http://search.yahooapis.com/SiteExplorerService/V1/updateNotification?appid=YahooDemo&url=sitemap_url

  • http://www.bing.com/webmaster/ping.aspx?siteMap=sitemap_url

  • http://submissions.ask.com/ping?sitemap=sitemap_url


In addition, if you're using Google's sitemap_gen.py python script for generating your sitemap, they have a hash of tuples for each site you want to notify. Only Google's is included by default. Here is a version of the NOTIFICATION_SITES hash that has Google, Bing, Yahoo, and Ask.com:

NOTIFICATION_SITES = [
('http', 'www.google.com', 'webmasters/tools/ping', {}, '', 'sitemap'),
('http', 'search.yahooapis.com', 'SiteExplorerService/V1/updateNotification', {'appid' : 'YahooDemo'}, '', 'url'),
('http', 'submissions.ask.com', 'ping', {}, '', 'sitemap'),
('http', 'www.bing.com', 'webmaster/ping.aspx', {}, '', 'siteMap')
]


This should work as-is, but if the Yahoo one fails for you, or if you want to follow Yahoo's rules more closely, you should sign up for an official Yahoo Developer appid, and replace "YahooDemo" in the code above with your appid.