curl and wget: Fetching web pages and APIs in the terminal

In short: Two command-line programs that download data over HTTP and other protocols or send it to servers.

In more detail: curl and wget fetch content from web addresses without a browser. They are standard tools for scripts, troubleshooting and trying out interfaces. wget is designed for downloading whole files and pages, curl for flexible requests with all HTTP methods.

In Depth

curl: common examples

curl https://example.com                 # print content
curl -I https://example.com              # headers only (HEAD)
curl -o page.html https://example.com    # save to file
curl -L https://short.link/x             # follow redirects
curl -X POST https://api.example.com/items \
  -H 'Content-Type: application/json' \
  -d '{"name":"Test"}'                   # send JSON
curl -u user:password https://...         # basic auth
curl -H 'Authorization: Bearer TOKEN' ... # token

-v shows the complete communication, -s suppresses progress, -w '%{http_code}' prints the status code.

wget: common examples

wget https://example.com/file.zip         # download
wget -c https://example.com/big.iso       # resume an interrupted download
wget -r -np https://example.com/docs/     # fetch a folder recursively
wget -O name.zip https://...              # set file name

It resumes downloads after connection drops and can mirror whole websites (--mirror).

Typical uses

• Test an interface before writing code.
• Check whether a page is reachable and which status code it returns.
• Installation scripts: curl -fsSL URL | sh (caution, see below).
• Load files on servers without a graphical interface.
• Narrow down problems with certificates, redirects or headers.

Security notes

Piping scripts from the net straight into a shell (curl ... | sh) runs foreign code without checking; better to download and read it first. Passwords and tokens on the command line appear in the shell history and the process list; prefer environment variables or configuration files. -k (turn off certificate checking) only for testing.

See also: REST API design, Linux commands, TLS, HTTP status codes