I noticed something very bizarre today that has me beat.
Observation 1 is that I run “df” and can see that /tmp is full, with Use = 100%.
Observation 2 is that I run “du /tmp” I can see where it’s gone (I have pretty fine grained lightpd logging on and the log files filled it up).
Intervention 1 is that stop lightpd (/etc/init.d/lightpd stop) then I remove the log files.
Observation 3 is that I run “du /tmp” again. Yep it’s gone. Job done.
Observation 4 is that I run “df” again. And /tmp is still full. Use=100%.
What gives? That’s bizarre.
Immediately conjures images of a trash can or recycling bin that rm moves files to that needs emptying. Which would be nice except that rm doesn’t work like that, not last time I checked. So what gives?
I have no idea why du would report something different than df when it’s tmpfs. But what I am sure is if you have configured lighttpd to spill out more logs or even configured access log than you should configure logrotate to roll those away. Google up tutorials on logrotate…
1 Like
I’ll check out logrotate, it should really be on by default, naughty lightpd! ;-). Turned off logs for now.
du and df reporting different things is weird. I should have added:
Observation 5 is that df is probably right because if run “/etc/init.d/resolver start” then “ps | grep kresd” I see nothing!
Observation 6 is that if I reboot, df shows space on /tmp, and “ps | grep kresd” shows me the kresd process and I can stop it and start it with init.d/resolver and still see it, so all is order on fresh restart.
There is no supported way to enable verbose logging in lighttpd unless you edit configuration files directly and in that case we can’t ensure anything. You should know what you are doing in such case. This isn’t TO specific. It’s common administrative task to ensure that you have correct logrotate configuration when you change or create log file.
Are you, by any chance using df on a btrfs filesystem? In that case it is known that df is not reporting the correct information try “sudo btrfs filesystem df /tmp”. Or even better (if supported) “sudo btrfs fi usage /tmp”, but best to read:
https://btrfs.wiki.kernel.org/index.php/FAQ#How_much_free_space_do_I_have.3F
for details…
Best Regards
Yep it’s tmpfs not btrfs.
# df -T
Filesystem Type 1K-blocks Used Available Use% Mounted on
/dev/mmcblk0p1 btrfs 7633920 640896 7004292 8% /
tmpfs tmpfs 1035384 15352 1020032 1% /tmp
tmpfs tmpfs 512 4 508 1% /dev
If happened again I could nab screenshots to share, but I’ve turned off lighttpd logging for now to prevent it happening ;-). I guess understanding the weird df/du disagreement is a secondary concern to being on-line reliably. But I might find time and interest to turn it on again and replicate some time. Because it did drive me a bit batty, as the only way I could in fact empty /tmp was rebooting, no amount rm was working as evidence (not conclusively mind you) by df reports and the fact that kresd would not start.
Though I forgot another weirdism I noted. (I’m collecting them):
Observation 7 is that while df reported 100% usage on /tmp, the this would yield nothing:
#/etc/init.d/resolver/start
Called /etc/init.d/kresd start
#ps | grep kresd
yet this:
#kresd
started kresd just fine and I could type verbose(true) and it would spew forth an endless stream of stuff (probably stuff on my LAN asking for name resolutions, I didn’t dwell on it, it was bundles of activity).
And after a reboot:
# ps | grep kresd
24425 root 30652 S /usr/bin/kresd -c /tmp/kresd.config -f 1 /tmp/kresd -a 0.0.0.0 53 -a :: 53 -k /etc/root.keys
24904 root 1088 S grep kresd
And to test I did this:
# /etc/init.d/resolver stop
Called /etc/init.d/kresd stop
# ps | grep kresd
31431 root 1088 S grep kresd
# /etc/init.d/resolver start
Called /etc/init.d/kresd start
# ps | grep kresd
24425 root 30652 S /usr/bin/kresd -c /tmp/kresd.config -f 1 /tmp/kresd -a 0.0.0.0 53 -a :: 53 -k /etc/root.keys
24904 root 1088 S grep kresd
In short all fine after a reboot. But while tmpfs is at 100% use not so, but kresd will start in interactive mode just not as a service. Weird.
The plain kresd command won’t load that config file; in particular, defaults will setup cache etc. in the current directory instead (which isn’t full presumably).
1 Like