dmesg and journalctl: Reading system messages

In short: dmesg shows the kernel’s messages (hardware, drivers, errors), journalctl the collected logs of all services under systemd.

In more detail: When something does not work, the cause is often already in a log message. On Linux there are two main tools for this: dmesg for the kernel ring buffer and journalctl for the systemd journal, which also records services, sign-ins and applications.

In Depth

dmesg

dmesg                    # all kernel messages since boot
dmesg -T                 # with readable time
dmesg --level=err,warn   # only errors and warnings
dmesg -w                 # follow live (e.g. when plugging in a USB stick)

Typical finds: detected devices, driver errors, disk errors (I/O error), memory problems (Out of memory), hardware problems. The buffer is limited, older messages are overwritten, and on some distributions ordinary users may only read it with sudo.

journalctl

journalctl -b                       # since the last boot
journalctl -b -1                    # previous boot
journalctl -u nginx                 # messages of one service
journalctl -u nginx -f              # follow live
journalctl -p err                   # only priority error and higher
journalctl --since "1 hour ago"     # time range
journalctl -k                       # kernel only (like dmesg)

With -o json the output can be processed by machine. The journal is not persistent everywhere by default: if /var/log/journal/ exists, it survives a reboot.

Approach to problems

1. Narrow down the time of the error.
2. Look at journalctl -u <service> --since ... and -p err.
3. For hardware problems dmesg -T | tail.
4. The first error message is usually more important than the follow-up errors.
5. Look up the error text in search engines or in the wiki.

Space and retention

The journal limits itself (SystemMaxUse= in /etc/systemd/journald.conf). sudo journalctl --vacuum-time=7d removes old entries. You hand an error report to others with journalctl -b -p err > errors.txt; watch out for personal data such as user names or addresses.

See also: Logs and scheduling, Kernel panic and oops, systemd, logrotate