Deleted But Still Growing

A long-running daemon has /var/log/app.log open on fd 3 and is still appending to it. An operator, out of disk space, runs rm /var/log/app.log.

$ du -sh /var/log
412M    /var/log
$ df -h /var
Filesystem  Size  Used Avail Use%
/dev/sda2    20G   19G  240M  93%

du no longer counts the 8 GB log, but df still reports the space as used, and it keeps climbing. What is happening?

rm only ever removes names. Blocks belonging to deleted files are reclaimed by fsck at the next boot, which is why a file system that is never checked slowly fills up.

unlink() removed the directory entry and dropped the link count to zero, but the daemon's open descriptor is a second kind of reference. The inode and its blocks are freed only when the last descriptor closes, and until then writes keep allocating. Restart the daemon, or truncate through /proc/<pid>/fd/3.

The freed blocks are still dirty in the page cache. Running sync writes them back and the data bitmap update makes the space visible as free.

df reads a cached statfs summary that the file system refreshes lazily. The space is already free; the number catches up at the next mount or after a mount -o remount.