Crawler

In short: A crawler (bot, spider) is a program that automatically fetches web pages and follows them.

In more detail: Search engines like Google use crawlers (Googlebot) to index the web.

In Depth

The file robots.txt says which areas crawlers may visit. Reputable bots follow it; malicious bots ignore it.

Price comparisons, archive services, AI training crawlers, but also scrapers and attack bots (web scraping).

Rate limiting, bot detection, firewall (WAF).

See also: Web Scraping, Tracking, Firewall