7 / 25 File Systems

Deleted But Still Growing

A long-running daemon has /var/log/app.log open on fd 3 and is still appending to it. An operator, out of disk space, runs rm /var/log/app.log.

$ du -sh /var/log
412M    /var/log
$ df -h /var
Filesystem  Size  Used Avail Use%
/dev/sda2    20G   19G  240M  93%

du no longer counts the 8 GB log, but df still reports the space as used, and it keeps climbing. What is happening?

A

rm only ever removes names. Blocks belonging to deleted files are reclaimed by fsck at the next boot, which is why a file system that is never checked slowly fills up.

B

unlink() removed the directory entry and dropped the link count to zero, but the daemon's open descriptor is a second kind of reference. The inode and its blocks are freed only when the last descriptor closes, and until then writes keep allocating. Restart the daemon, or truncate through /proc/<pid>/fd/3.

C

The freed blocks are still dirty in the page cache. Running sync writes them back and the data bitmap update makes the space visible as free.

D

df reads a cached statfs summary that the file system refreshes lazily. The space is already free; the number catches up at the next mount or after a mount -o remount.