Real-Time search indexing
Googlebot has gotten faster at indexing real-time sites. I posted a link to friendfeed.com's API room at 3:57pm. Googlebot indexed it at 4:14pm, 17 minutes later.
Another bot running on the Amazon Web Services indexed the link even faster, at 4:01pm - just 4 minutes after posting it. That bot doesn't identify itself in the User-Agent, PyCURL is the name of the opensource software package it is using to fetch URLs. It also does not appear to honor robots.txt, which is a questionable decision.
2 comments
Aug 11, 2009
Ahsan Ali liked this post.
Aug 11, 2009
Svartling liked this post.
