A web crawler is like a web browser with a robot at its controls. Crawlers are employed by all search engine companies to retrieve copies of websites into the search engine’s index.