I got a message from the monitoring this morning:
the usage of /var is 100% !
Well maybe I forgot to setup the rsyslog/logrotate or something. Then I ssh into my server, Try to figure out where is the huge file.
First I use
df to see if it's really 100%:
$ df -h Filesystem Size Used Avail Use% Mounted on ... /dev/mapper/var 18G 18G 20K 100% /var
K, it's indeed full, something is eating my disk. Then I use
du to check the /var:
$ cd /var # -s: summary # -h: human-friendly $ du -sh . 1.6G .
Interesting, It's only 1.6G, But why I got 100% with
If you also got this just like me, most likely there are some files have been deleted but the process didn't release them. use
lsof to check:
$ lsof | grep delete filebeat ... /var/log/node.log-2019050618 (deleted) filebeat ... /var/log/node.log-2019050613 (deleted) filebeat ... /var/log/node.log-2019050615 (deleted) filebeat ... /var/log/node.log-2019050614 (deleted) ...
In my case,
filebeat is the bad process. The /var is back after I restart the filebeat.