Full /dev & loss of LAN

[jada4p]

Recently I noted that /dev directory on my TO is full as stated by df command; which was something like

Filesystem 1K-blocks Used Available Use% Mounted on
tmpfs 512 512 0 100% /dev

Later, before I was able to investigate more (even though I don’t know what and how) I encountered loss of LAN connectivity Repeated reboots didn’t help

Fortunately, WiFi was still OK, so I was able to return to last schnapps, which saved my ass. From that moment LAN (and SSH) is back again.

Could’nt full /dev directory be the cause of lost LAN connectivity? How could one prevent it? And, BTW, was it not the cause of loss of LAN connectivity, mentioned in some other forum topics?

[vcunat]

Uh. For reference, my working Omnia shows zero usage, just as a Turris 1.x:

root@turris:~# df /dev
Filesystem           1K-blocks      Used Available Use% Mounted on
tmpfs                      512         0       512   0% /dev

[Pepe]

Hello,
isn’t this related to BOINC somehow? But I have BOINC on Odroid U2 and it doesn’t write anything to /dev, so this seems weird. Wold you be so kind and if this happen again can you check which file or folder took all the space for /dev?

Hopefully you have BOINC on external storage and you don’t run LXC containers in internal storage.

// edit @vcunat pointed it right. Edited my post.

[vcunat]

/dev is a mounted tmpfs, so I don’t expect this to be as simple as filling the internal flash.

[jada4p]

One of very first thing I did when starting with LXC containers was that I mounted USB on /srv/lxc for containers.

Today afternoon state of /dev is as follows:

Filesystem 1K-blocks Used Available Use% Mounted on
tmpfs 512 108 404 21% /dev

root@turris:~# ls -l /dev
crw-r–r-- 1 root root 10, 235 Feb 13 01:39 autofs
drwxr-xr-x 2 root root 60 Feb 13 01:39 bsg
crw-r–r-- 1 root root 10, 234 Feb 13 01:39 btrfs-control
drwxr-xr-x 3 root root 60 Feb 13 01:39 bus
crw-r–r-- 1 root root 5, 1 Feb 13 01:39 console
crw-r–r-- 1 root root 10, 63 Feb 13 01:39 cpu_dma_latency
crw-r–r-- 1 root root 10, 58 Feb 13 01:39 crypto
crw-rw-rw- 1 root root 1, 7 Feb 13 01:39 full
crw-r–r-- 1 root root 10, 229 Feb 13 01:39 fuse
crw-r–r-- 1 root root 10, 183 Feb 13 01:39 hwrng
crw-r–r-- 1 root root 89, 0 Feb 13 01:39 i2c-0
…
crw-r–r-- 1 root root 89, 8 Feb 13 01:39 i2c-8
crw-r–r-- 1 root root 1, 11 Feb 13 01:39 kmsg
srw-rw-rw- 1 root root 0 Feb 13 01:39 log
drwxr-xr-x 2 root root 60 Feb 13 01:39 mapper
crw-r–r-- 1 root root 1, 1 Feb 13 01:39 mem
crw-r–r-- 1 root root 10, 60 Feb 13 01:39 memory_bandwidth
brw-r–r-- 1 root root 179, 0 Feb 13 01:39 mmcblk0
brw-r–r-- 1 root root 179, 8 Feb 13 01:39 mmcblk0boot0
brw-r–r-- 1 root root 179, 16 Feb 13 01:39 mmcblk0boot1
brw-r–r-- 1 root root 179, 1 Feb 13 01:39 mmcblk0p1
brw-r–r-- 1 root root 179, 24 Feb 13 01:39 mmcblk0rpmb
crw-r–r-- 1 root root 90, 0 Feb 13 01:39 mtd0
crw-r–r-- 1 root root 90, 1 Feb 13 01:39 mtd0ro
crw-r–r-- 1 root root 90, 2 Feb 13 01:39 mtd1
crw-r–r-- 1 root root 90, 3 Feb 13 01:39 mtd1ro
brw-r–r-- 1 root root 31, 0 Feb 13 01:39 mtdblock0
brw-r–r-- 1 root root 31, 1 Feb 13 01:39 mtdblock1
drwxr-xr-x 2 root root 60 Feb 13 01:39 net
crw-r–r-- 1 root root 10, 62 Feb 13 01:39 network_latency
crw-r–r-- 1 root root 10, 61 Feb 13 01:39 network_throughput
-rw-rw-rw- 1 root root 721397 Feb 14 16:10 null
-rw-rw-rw- 1 root root 1000056 Feb 14 16:10 null.1
-rw-rw-rw- 1 root root 1000077 Feb 14 02:31 null.2
-rw-rw-rw- 1 root root 1000086 Feb 13 20:44 null.3
crw-rw-rw- 1 root root 1, 3 Feb 13 01:39 null.4
crw-r–r-- 1 root root 1, 4 Feb 13 01:39 port
crw-r–r-- 1 root root 108, 0 Feb 13 01:39 ppp
crw-rw-rw- 1 root root 5, 2 Feb 14 16:10 ptmx
drwxr-xr-x 2 root root 0 Jan 1 1970 pts
crw-r–r-- 1 root root 1, 8 Feb 13 01:39 random
lrwxrwxrwx 1 root root 4 Feb 13 01:39 root → b301
crw-r–r-- 1 root root 254, 0 Feb 13 01:39 rtc0
brw-r–r-- 1 root root 8, 0 Feb 13 01:39 sda
brw-r–r-- 1 root root 8, 1 Feb 13 01:39 sda1
lrwxrwxrwx 1 root root 8 Feb 13 01:39 shm → /tmp/shm
drwxr-xr-x 2 root root 60 Feb 13 01:39 snd
crw-r–r-- 1 root root 153, 0 Feb 13 01:39 spidev0.2
crw-r–r-- 1 root root 5, 0 Feb 14 15:08 tty
crw-r–r-- 1 root root 4, 64 Feb 13 01:39 ttyS0
…
crw-r–r-- 1 root root 4, 79 Feb 13 01:39 ttyS15
crw-r–r-- 1 root root 10, 59 Feb 13 01:39 ubi_ctrl
crw-r–r-- 1 root root 1, 9 Feb 13 01:39 urandom
crw-r–r-- 1 root root 10, 130 Feb 13 01:39 watchdog
crw-r–r-- 1 root root 253, 0 Feb 13 01:39 watchdog0
crw-rw-rw- 1 root root 1, 5 Feb 13 01:39 zero

I can’t exclude that Boinc is using /dev; but most of its data files are stored in /var/lib/boinc-client (on container, off course).

But, I have to admit, that often, e.g. after reboot and even sometimes later, the /dev directory is empty, as I see it just now… I’ll don’t pretend I understood it.

This is only to explain my point.

[jada4p]

A moment ago, at about Mon Feb 19 17:15:57 CET 2018 (uptime: 17:15:57 up 4 days, 21:36, load average: 0.26, 0.30, 0.23; openwrt_version: 15.05; turris-version: 3.9.5; Turris Omnia), situation regarding /dev directory was:

Filesystem 1K-blocks Used Available Use% Mounted on
tmpfs 512 388 124 76% /dev

There is a lot of block and character special files, some directories:

root@turris:~# ls -l /dev
drwxr-xr-x 2 root root 80 Feb 14 19:39 bsg
drwxr-xr-x 3 root root 60 Feb 14 19:39 bus
drwxr-xr-x 2 root root 60 Feb 14 19:39 mapper
drwxr-xr-x 2 root root 60 Feb 14 19:39 net
drwxr-xr-x 2 root root 0 Jan 1 1970 pts
drwxr-xr-x 2 root root 60 Feb 14 19:39 snd

Also there are some other files:

srw-rw-rw- 1 root root 0 Feb 17 04:35 log
lrwxrwxrwx 1 root root 4 Feb 14 19:39 root → b301
lrwxrwxrwx 1 root root 8 Feb 14 19:39 shm → /tmp/shm

Most peculiar are null files:

-rw-rw-rw- 1 root root 355992 Feb 19 17:16 null
-rw-rw-rw- 1 root root 1000106 Feb 19 17:10 null.1
…
-rw-rw-rw- 1 root root 1000077 Feb 16 00:30 null.16
crw-rw-rw- 1 root root 1, 3 Feb 14 19:39 null.17

Note that 1st null file is 355992 B, others, null.1 - null.16 each about 1 MB, but null.17 is character special file!

root@turris:~# file /dev/null /dev/null.17
/dev/null: data
/dev/null.17: character special (1/3)

Could any Linux expert please explain this situation?

[Skywalker-11]

I also have multiple null files under /dev :

# ls -lhe /dev/null*
-rw-rw-rw-    1 root     root      584.1K Mon Feb 19 20:50:35 2018 /dev/null
-rw-rw-rw-    1 root     root      976.6K Mon Feb 19 17:18:06 2018 /dev/null.1
-rw-rw-rw-    1 root     root      976.6K Fri Feb 16 23:21:22 2018 /dev/null.10
-rw-rw-rw-    1 root     root      976.6K Fri Feb 16 17:46:00 2018 /dev/null.11
crw-rw-rw-    1 root     root        1,   3 Thu Feb 15 23:35:38 2018 /dev/null.12
-rw-rw-rw-    1 root     root           0 Mon Feb 19 07:40:43 2018 /dev/null.2
-rw-rw-rw-    1 root     root      976.7K Sun Feb 18 23:41:36 2018 /dev/null.3
-rw-rw-rw-    1 root     root      976.6K Sun Feb 18 17:34:00 2018 /dev/null.4
-rw-rw-rw-    1 root     root      976.7K Sun Feb 18 10:19:55 2018 /dev/null.5
-rw-rw-rw-    1 root     root      976.6K Sun Feb 18 03:53:21 2018 /dev/null.6
-rw-rw-rw-    1 root     root      976.7K Sat Feb 17 21:49:55 2018 /dev/null.7
-rw-rw-rw-    1 root     root      976.6K Sat Feb 17 15:20:33 2018 /dev/null.8
-rw-rw-rw-    1 root     root      976.7K Sat Feb 17 03:44:10 2018 /dev/null.9

From the dates of the files it looks like they are files that are logrotated ??
The POSIX standard (http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap10.html) explicitly states:

/dev/null
An empty data source and infinite data sink. Data written to /dev/null shall be discarded. Reads from /dev/null shall always return end-of-file (EOF).

Same goes for the LSB standard (linux) in http://refspecs.linuxbase.org/FHS_3.0/fhs/ch06.html#devDevicesAndSpecialFiles:

/dev/null
All data written to this device is discarded. A read from this device will return an EOF condition.

The files definitely shouldn’t be there. Also notice that if you pipe output to /dev/null you are able to read that output for some time until it is removed.
The other /dev/null.x files contain ssh and haas-proxy logs

Did the turris team messed something up?

EDIT: added linux standard reference

[jada4p]

I don’t think turris team messed anything up… (I hope I’m affraid it was in OpenWrt, earlier

BTW, I think that even Linux begginer knows what /dev/null is This was main reason of this thread.

Could please anybody from the team - @Pepe? @Miska? escalate this problem ?!? Maybe these “null” files dissolve by themselves, given some time, or they can be removed by cron… but such a solution isn’t nice and, moreover, system solution… Anyhow, done as temporary mitigation

[Pepe]

Hi,
no need to mention me in other post, when you mention me in your first post.

We’ll look into this.

//EDIT: Fixed and will be part of Turris OS 3.9.6 release.

[jada4p]

Nice.

Could I ask when, approximately, is rel 3.9.6 planned? Or, where one can find such info?

I didn’t noted your edited reply Wouldn’t it be possible to improve forum system in such a way, that when somebody edit his post, it’ll be mailed as well as original post?

[Pepe]

The latest info what I get today is that we’d like to release 3.9.6 to RC tomorrow, but unexpected things can delay it. It won’t be a later than this week.

//EDIT: Postponed. We’d like to release 3.9.6 to RC tomorrow (28/2).

[jada4p]

Thanks…(bloody 20 chars required

---

Anyway, I’d be interested in cause of this problem…

[Pepe]

It should be fixed in 3.9.6, which we released today.