Web Crawlers, also known as Web Spiders, are programs that search for information on the World Wide Web. Web Crawlers are most widely employed by search engine companies and are used to create ever-growing indexes of web pages and their content. At its most elementary, a Web Crawler visits a known web page and then explores all the links on that web page (discovering new web pages in the process and repeating the sequence on them). Content that is not accessible to Web Crawlers is known as the Deep Web.
- By Jason Fitzpatrick on 05/11/13