If you're using sitemaps on your website, you can notify the major search engines whenever your sitemap is updated. They usually offer a "ping" URL that you can hit. The ping URLs for the major search engines are below. You would substitute the full URL to your sitemap.xml (or .gz) file, encoded, wherever you see
- http://www.google.com/webmasters/tools/ping?sitemap=sitemap_url
- http://search.yahooapis.com/SiteExplorerService/V1/updateNotification?appid=YahooDemo&url=sitemap_url
- http://www.bing.com/webmaster/ping.aspx?siteMap=sitemap_url
- http://submissions.ask.com/ping?sitemap=sitemap_url
In addition, if you're using Google's sitemap_gen.py python script for generating your sitemap, they have a hash of tuples for each site you want to notify. Only Google's is included by default. Here is a version of the NOTIFICATION_SITES hash that has Google, Bing, Yahoo, and Ask.com:
NOTIFICATION_SITES = [
('http', 'www.google.com', 'webmasters/tools/ping', {}, '', 'sitemap'),
('http', 'search.yahooapis.com', 'SiteExplorerService/V1/updateNotification', {'appid' : 'YahooDemo'}, '', 'url'),
('http', 'submissions.ask.com', 'ping', {}, '', 'sitemap'),
('http', 'www.bing.com', 'webmaster/ping.aspx', {}, '', 'siteMap')
]
This should work as-is, but if the Yahoo one fails for you, or if you want to follow Yahoo's rules more closely, you should sign up for an official Yahoo Developer appid, and replace "YahooDemo" in the code above with your appid.