Online published on 30 September, 2013.
As the web is expanding abruptly, so need to extract knowledge from the web is becoming essential and is accepted. This is possible because of fortune and ease of information. In order to get web pages search engines are used that are based on Web crawling framework. This paper gives the outline of how the Web crawlers are connected with search engine and also explains the fundamental task performed by the search engine.
Web Crawlers, Focused Crawling, Distributed Crawling