Thursday, 28 June 2007

Examples of Web crawlers

The following is a list of published crawler architectures for general-purpose crawlers (excluding focused Web crawlers), with a brief description that includes the names given to the different components and outstanding features:

Source: http://en.wikipedia.org/wiki/Web_crawler

No comments: