Web and technology
Crawler
In short and in plain language
A crawler is a program. It visits web pages by itself and reads their content.
Search engines use crawlers to collect pages.
Other crawlers gather content for other uses.
Many visits from crawlers can put a load on a server.
Definition
A crawler is a program that automatically visits web pages and reads their content. Search engines use crawlers to capture pages; others collect content for other purposes. Many requests from crawlers can put a load on a server.