Linux system logs: reading journalctl and dmesg

When a Linux computer behaves oddly, the answer is almost always written down somewhere. The trouble is that there are thousands of messages and, without filters, finding the right one is like reading a phone book. On a modern system there are two main tools: journalctl, which queries the systemd journal, and dmesg, which shows kernel messages. A handful of options is enough to get to the point in seconds.

Where messages end up

On distributions that use systemd, every message, from the kernel to the services, is collected by journald into a binary, indexed journal. Many distributions still run rsyslog alongside it, which writes text files in /var/log/, such as syslog on Debian and Ubuntu or messages on Fedora and its derivatives. Some programs also keep their own logs: the web server in /var/log/nginx/, the package manager in /var/log/apt/ or /var/log/dnf.log.

The journald journal can be volatile, kept only in memory and lost at every reboot, or persistent on disk in /var/log/journal/. If journalctl --list-boots shows only one boot, the journal is not being kept: just create the /var/log/journal folder and restart the systemd-journald service, or set Storage=persistent in /etc/systemd/journald.conf.

journalctl: the options you actually need

Run on its own, journalctl shows everything from oldest to newest, opened in a pager where you scroll with the arrow keys and search with /. The useful options are few:

The options can be combined. For example, errors from the SSH service in the last hour:

journalctl -u ssh -p err --since "1 hour ago"

Or everything that happened yesterday afternoon between 3 and 4 pm:

journalctl --since "yesterday 15:00" --until "yesterday 16:00"

Priorities

Every message has a severity level inherited from the syslog tradition. From most to least severe: emerg, alert, crit, err, warning, notice, info, debug. With -p warning you see warnings and everything more severe. In the pager, errors appear in red.

dmesg: kernel messages

dmesg reads the kernel's ring buffer: device detection, drivers loading, disk errors, memory running out. On many distributions it needs sudo. The handiest options:

• dmesg -T shows readable timestamps instead of seconds since boot;
• dmesg -w follows new messages live;
• dmesg --level=err,warn filters by severity.

A typical use: you plug in a USB stick that doesn't show up. With sudo dmesg -w open in a terminal, you plug it in and immediately see whether the kernel recognised it, under which device name, and possibly which error it returned.

What to look for in common cases

• A failing disk: lines with I/O error, blk_update_request or SATA link resets. Take them seriously and follow up with a check of the SMART values.
• Out of memory: lines with Out of memory: Killed process tell you which process the kernel sacrificed. Find them with journalctl -k | grep -i "out of memory".
• A service that won't start: journalctl -u name -b almost always shows the exact message, for example a syntax error in the configuration file or a port already in use.
• Sudden shutdowns: look at the end of the previous boot with journalctl -b -1 -e. If the journal stops abruptly without the shutdown lines, the computer went off without an orderly command, as happens with a power cut or a power supply problem.
• Login attempts: on an exposed server, journalctl -u ssh | grep "Failed password" shows failed password login attempts.

How much space they take

The persistent journal grows but has automatic limits, by default a percentage of the filesystem's space. To see how much it uses:

journalctl --disk-usage

To shrink it right away you can keep only a certain period or size:

sudo journalctl --vacuum-time=30d
sudo journalctl --vacuum-size=500M

For a permanent limit, set SystemMaxUse= in journald.conf. The text files in /var/log are handled by logrotate, which rotates, compresses and deletes them according to the rules in /etc/logrotate.d/. Deleting logs by hand with rm while the service is writing them is a bad idea: the space may not be freed until the process reopens the file.

A method to avoid getting lost

In my work I always follow the same order: first I narrow down the time when the problem happened, then the service or device involved, and finally the severity. Three filters, combined, turn tens of thousands of lines into a few dozen. Only then do I read the messages one by one, starting from the first error rather than the last: later errors are often consequences of the first.

In short

• journald collects almost everything; rsyslog and individual programs may also write to /var/log.
• journalctl -b, -u, -p and --since narrow the search straight away.
• dmesg -T -w is the right tool for devices, drivers and hardware problems.
• Log space is checked with --disk-usage and limited with --vacuum or SystemMaxUse.
• Read starting from the first error in the relevant period, not the last.