Saturday, 30 June 2007

Mission Statement

"The Inktomi mission is to build scalable software applications that are core to the Internet infrastructure."

Source: http://en.wikipedia.org/wiki/Inktomi

Friday, 29 June 2007

Freelane by Excite

In a bid to compete against Internet Service Providers like NetZero and Juno Online, which offered free or low-cost dial-up access in the United States, Excite started offering their own "no-pay" service for private customers by partnering with 1stUp.com to create FreeLane by Excite: 1stUp would allow Excite customers to download software in order for them to able to log-on to the Internet. The software would then rotate a series of sponsored banner advertisements on the user's computer while they surfed the Internet. 1stUp.com soon went out of business, and Excite switched to another partner named WorldShare, rebranding Freelane as FreeLane version 2.0. As of March 1, 2001 Freelane was discontinued.[1]

Source: http://en.wikipedia.org/wiki/Excite

Thursday, 28 June 2007

Examples of Web crawlers

The following is a list of published crawler architectures for general-purpose crawlers (excluding focused Web crawlers), with a brief description that includes the names given to the different components and outstanding features:

Source: http://en.wikipedia.org/wiki/Web_crawler

Wednesday, 27 June 2007

Re-visit policy

The Web has a very dynamic nature, and crawling a fraction of the Web can take a really long time, usually measured in weeks or months. By the time a Web crawler has finished its crawl, many events could have happened. These events can include creations, updates and deletions.

Source: http://en.wikipedia.org/wiki/Web_crawler

Saturday, 23 June 2007

Human-edited directories

A human-edited directory is created and maintained by editors who add links based on the policies particular to that directory.

Source: http://en.wikipedia.org/wiki/Web_directories

Geospatially enabled search engines

(See e.g. Wikipedia:Summary style.)

Source: http://en.wikipedia.org/wiki/Searchengine

Friday, 22 June 2007

Storage costs and crawling time

Storage costs are not the limiting resource in search engine implementation. Simply storing 10 billion pages of 10 kbytes each (compressed) requires 100TB and another 100 TB or so for indexes, giving a total hardware cost of under $200k: 100 cheap PCs each with four 500GB disk drives.[citation needed]

Source: http://en.wikipedia.org/wiki/Searchengine