>_ Operating systems & Linux

Disk full on Linux: finding what takes up space with df and du

A method for freeing space on Linux: reading df, finding heavy folders with du and ncdu, cleaning logs, caches and old kernels, inodes and open files.

«No space left on device» is one of the messages server administrators dread most: the database stops writing, updates fail, sometimes you can't even log in. On a desktop the problem is less dramatic but just as annoying. The instinctive reaction is to delete something at random; the right one is a three-step method: work out which filesystem is full, find where the space is concentrated, decide what can be safely removed.

Step 1: which filesystem is full

The command is df:

df -h

It shows every mounted filesystem with size, used space, available space, percentage and mount point. On Linux there can be several partitions: the root /, sometimes separate /home or /var, the /boot partition. Knowing which one is full changes the search entirely. Lines of type tmpfs are in-memory filesystems and can usually be ignored; with df -h -x tmpfs -x devtmpfs they disappear.

A special case: df -h shows free space but the system still says the disk is full. Then you check the inodes:

df -i

Every file uses an inode, and on ext4 their number is fixed when the filesystem is created. Millions of tiny files, such as a web application's sessions or temporary files, can exhaust them before the space runs out.

Step 2: where the space is concentrated

du measures the space taken by folders and files. The most useful way to use it is to look one level at a time, sorting the results:

sudo du -h --max-depth=1 / 2>/dev/null | sort -h

The last line is the heaviest folder. You go into it and repeat:

sudo du -h --max-depth=1 /var 2>/dev/null | sort -h

Two options help: -x stops du from crossing into other mounted filesystems, so you measure only the partition you care about; 2>/dev/null hides errors about virtual folders such as /proc.

For more comfortable exploration there is ncdu, available in the repositories of almost every distribution. It scans the given folder and shows a list sorted by size that you navigate with the arrow keys; d deletes the selected item after confirmation.

sudo ncdu -x /

On desktops there are graphical equivalents, such as GNOME's Disk Usage Analyzer or KDE's Filelight.

Step 3: the usual suspects

In my experience, space almost always disappears into the same places:

Location What it holds How to act
/var/log system and application logs journalctl --vacuum-size=, check logrotate
/var/cache/apt or dnf's cache downloaded packages sudo apt clean or sudo dnf clean all
/var/lib/docker images, containers and volumes docker system df, then docker system prune
/var/lib/snapd or Flatpak previous package versions keep fewer revisions, flatpak uninstall --unused
/boot old kernels sudo apt autoremove
~/.cache browser and program caches can be emptied with the programs closed
trash in ~/.local/share/Trash files deleted from the desktop empty the trash

With Docker, be careful: docker system prune removes stopped containers, unused networks and untagged images; adding --volumes also deletes unattached volumes, which may contain database data. Better to read first what would be removed.

On a small /boot partition, a build-up of old kernels is a frequent cause of failed updates. autoremove removes them while keeping the one in use and the previous one.

Files deleted but still open

Sometimes you delete a huge log file, but df still shows the disk as full. The reason: a process still has the file open. On Linux the file vanishes from the folder, but the space is only freed when the last process closes it. To find them:

sudo lsof +L1

The list shows deleted files still in use, along with the process holding them. The fix is to restart that service. To empty a log without this problem, truncate it instead of deleting it:

sudo truncate -s 0 /var/log/application/large.log

ext4's reserved space

On ext4 a small percentage of the disk is reserved for root, 5% by default, so the system can keep working even when users have filled everything. That is why df may show a partition at 100% while root can still write. On a disk used only for data, the reserve can be reduced with sudo tune2fs -m 1 /dev/sdb1; on the system partition it is better left alone.

Prevention

  • Set limits on logs, with SystemMaxUse for journald and logrotate rules for text files.
  • Periodically clear package caches and unused container images.
  • Monitor free space with an alert below a threshold, for example 15%, instead of noticing when it has run out.

In short

  • df -h tells you which filesystem is full; df -i checks the inodes.
  • du --max-depth=1 | sort -h or ncdu -x find the culprit folder one level at a time.
  • Logs, package caches, Docker and old kernels are the most frequent causes.
  • A deleted but open file takes up space until the process closes it: lsof +L1.
  • Logs are truncated with truncate, not deleted while in use.
MM
Marco Marzaduri

Marco Marzaduri works in cybersecurity, digital forensics and web development. With degrees in computer science and psychology, he combines technical analysis with an understanding of the human factor.

Read also