An Analytical Study of Intelligent Parallel Web Crawler

The World-Wide Web has undergone explosive, exponential growth. As a consequence, users find themselves unable to browse the ever-changing, distributed hyperlink structure of the web. Furthermore, they are subjected to information overload information is too abundant. With the increasing number of information resources on the Web, it is often more difficult to locate the resources that are relevant to a given need. Henceforth, Search Engines are becoming equally important tool in locating relevant information. It is very difficult to search information from such a huge collection of World Wide Web as the web pages/documents are not organized as books on shelves in a library, nor are web pages completely catalogued at one central location. Search engine is basic information retrieval tool, used to access information from WWW. Users provide search queries in the Search engine's interface. In response to the search query provided, Search engines use their database to search the relevant documents and produce the result after ranking on the basis of relevance. In fact, the Search engine builds its database, with the help of Web Crawlers. This article describes standard crawler architecture and the modules of a proposed system.