The Web has a very dynamic nature, and crawling a fraction of the Web can take a really long time, usually measured in weeks or months. By the time a Web crawler has finished its crawl, many events could have happened. These events can include creations, updates and deletions.
Source: http://en.wikipedia.org/wiki/Web_crawler
Subscribe to:
Post Comments (Atom)
No comments:
Post a Comment