A Cloud-based Web Crawler Architecture

The cloud computing paradigm provides support for elastic resources and unstructured data, and provides pay-per-use features that allow individual businesses to run their own web crawlers for crawling the Internet or a limited web hosts. In this paper, we propose a cloud-based web crawler architecture that uses cloud computing features and the Map Reduce programming technique. The proposed web crawler allows us to crawl the web by using distributed agents and each agent stores its own finding on a PostgreSQL using amazon web services. The proposed web crawler also could store unstructured and massive amount of data on PostgreSQL.