grep, sed and awk: editing text in the terminal

In short: Three command-line classics: grep searches, sed replaces, awk works with columns. Together with pipes (|) they process text files and logs in a flash.

grep: search

grep "error" log.txt           # lines with "error"
grep -i "error" log.txt        # ignore upper/lower case
grep -rn "TODO" src/           # recursive, with line number
grep -v "debug" log.txt        # lines WITHOUT "debug"
grep -E "^(GET|POST) " log.txt # extended regular expression

For regular expressions see Regular expressions.

sed: replace

sed 's/old/new/g' file.txt         # replace all, output on screen
sed -i 's/old/new/g' file.txt      # directly in the file (-i)
sed -n '5,10p' file.txt            # only lines 5 to 10
sed '/^#/d' file.txt               # delete comment lines

awk: columns

awk '{print $1}' log.txt                 # first column
awk -F: '{print $1, $3}' /etc/passwd     # separator ":"
awk '{s+=$2} END {print s}' numbers.txt  # sum of the second column

Together

grep "404" access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head

Shows the addresses with the most 404 errors. Remember: before sed -i always keep a backup or check first without -i.

See also: Writing shell scripts