Wednesday, 27 June 2007

Re-visit policy

The Web has a very dynamic nature, and crawling a fraction of the Web can take a really long time, usually measured in weeks or months. By the time a Web crawler has finished its crawl, many events could have happened. These events can include creations, updates and deletions.

Source: http://en.wikipedia.org/wiki/Web_crawler

No comments: