May 6, 2019

Who Is Eating My Disk

procedure article

I got a message from the monitoring this morning:

the usage of /var is 100% !

Well maybe I forgot to setup the rsyslog/logrotate or something. Then I ssh into my server, Try to figure out where is the huge file.

First I use df to see if it's really 100%:

$ df -h
Filesystem                  Size  Used Avail Use% Mounted on
/dev/mapper/var              18G   18G   20K 100% /var

K, it's indeed full, something is eating my disk. Then I use du to check the /var:

$ cd /var

# -s: summary
# -h: human-friendly
$ du -sh .
1.6G	.

Interesting, It's only 1.6G, But why I got 100% with df ?

If you also got this just like me, most likely there are some files have been deleted but the process didn't release them. use lsof to check:

$ lsof | grep delete
filebeat  ... /var/log/node.log-2019050618 (deleted)
filebeat  ... /var/log/node.log-2019050613 (deleted)
filebeat  ... /var/log/node.log-2019050615 (deleted)
filebeat  ... /var/log/node.log-2019050614 (deleted)

In my case, filebeat is the bad process. The /var is back after I restart the filebeat.