Omnia / 5.0 keep writing to mmcblk0 even though there are no apps and no reason to do so

I noticed since I moved to HBT - 5.0, there are periodical writes to MMC. I’m quite sure it was not there when I was on 4.x version, the Write red part was almost clean.
My Omnia is almost in default settings, with just some custom Network Interfaces and Firewall rules and collectd statistics. Hence no LXC, no additional apps, no one checkbox at used.

/srv is mapped to /dev/sda, what is internal SSD, but currently it is used only for storing RRDTool output [/srv/rrd].

root@turris:~# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/mmcblk0p1            7.3G    722.3M      6.6G  10% /
devtmpfs                512.0K         0    512.0K   0% /dev
tmpfs                   512.0K         0    512.0K   0% /sys/fs/cgroup
tmpfs                  1009.8M     24.4M    985.4M   2% /tmp
tmpfs                   512.0K         0    512.0K   0% /dev
/dev/sda                 55.9G     17.7M     54.9G   0% /srv

Any idea what could cause that, how to investigate, and more importantly - is it harmful to the MMC wearing off?

manual monitor disk writes by processes with htop (customized columns)

did you see eMMC - live span, health monitoring - Omnia HW problems - Turris forum?

To be honest 4.0.5 behaves the same.

I can confirm there are every 30 minutes writes from two pocesses:

htop shows ‘no perm’ for DISK_WRITE, searching for solution suggest it is something with kernel what goes beyond my knowledge.
I read the eMMC… discussion but do not see any resolution. On this forum I saw many people having dead MMC though not using extensive-write apps on it.
mmc extcsd read /dev/mmcblk0 shows 0x01, what should be fine, but for how long…
I have used the STORAGE plugin in Foris that shows green Device currently in use is sda...
I would like to use the internal SSD for everything write-related.

The graph posted in your initial report shows writes every hour or so, you would have to watch the htop output for one hour (worst case) to catch up. A more automated (but also more complex) approach could perhaps be with sysstat.

Every type of storage medium has a limited lifespan and NAND flash is perhaps less durable than SSD.

as mentioned in the thread

Not sure if the developers could cover via the Foris storage plug-in each and every userland, that may potentially write to the OS’s drive, that is available in the distro - seems they got the mainstream apps covered.

Or else you could run the OS entirely off

and thereby prevent eMMC usage altogether.

It still shows no perm, even when I let it run for hours and also I do extensive writing to the SSD:

As I said, I don’t have anything more than default settings + Collectd, such config should definitely be covered.

For each option, you need to have the serial cable and flash disk with the medkit.
That’s not reachable for me :frowning:

It can be done/prepared through ssh whilst running the OS from the eMMC. Unfortunately, it is not documented but there are few related threads in this forum. Only UART serial cable should be available in case something goes wrong. :warning:

Well, apologies for having sent you on a wild goose chase since the relevant kconfs are unset.

What i would like to know is what i can do to have all writes done to SSD. I used the Storage function in Foris, that is supposed to make it work (in case of all default settings) ?

I think you should better specify what you mean by “all writes”. If you want to run the system from eMMC, then there will always be system updates which need to write to the eMMC. So at least these writes are unavoidable.

One idea I got is the following - could you try to remount the root (/) readonly after boot? That way you could be absolutely sure nobody is writing anything to the eMMC. And looking in the logs, you would probably find the apps that try to write someting. I’d also be quite interested in knowing who the bad guys are. But if I remember correctly, every time my filesystem got into readonly mode (due to some error) on TOS 3.x, resolving (a.k.a. kresd) stopped working immediately.

At least kresd is under the maintenance of CZ.NIC, so they should be able to explain why and what does it write. procd is the generic “init” process, so that one is very unspecific.

Another way of finding out what’s writing and where could be periodical saving of the output of lsof (with a well curated set of parameters). If you roughly know the times at which the writes happen (they seem periodical), you can just run this for a few minutes and if you’re lucky you’ll catch the bad guys in action.

As @anon82920800 said, disk utilization via htop/iotop is unavailable in TOS 4/5. In 3.x it works. If you’d like to have it available in TOS 4/5, please write it in Where are iftop, iotop, nethogs and friends? to show CZ.NIC there is a non-singular user base that is interested in having the required kernel flags set. They said it’s possible to enable everything that’s needed for showing disk utilization, but maybe they just need a little bit motivation to do so :slight_smile:

Do you think it is possible to test this in a LXC container?

Nope. LXC doesn’t IMO have such a level of filesystem virtualization.

I’m not sure about the lxc/lxd capabilities, but the underlying mechanism of mount namespaces does allow such things. BTW, uses this in practice: the directories with installed packages are read-only by default to avoid accidental writes even by root, and the “installer” runs in a mount namespace where it’s read-write.

If you really want to have all writes on SSD, move the root partition to your SSD and stop using the emmc. This is what I done. It is working fine but the migration may give you some difficulties (need to change UUID, need to update uboot settings,…).

I mean ‘unnecessary’ writes, where I understand software updates are necessary to get written, but application things like DNS resolver or DHCP server should not need to write to eMMC, they can store all their running data in RAM, or I would like to redirect those to the SSD.

This goes far beyond my knowledge and I’m afraid to brick my router :roll_eyes:

will do

I don’t feel confident to perform such operation

Do not try then. It may require you to use serial access and such and else may render your router unusable.

Hmm, I did a few tests, but no known method worked on Omnia. The RW-mounted filesystem refuses to remount readonly.

Even echo u > /proc/sysrq-trigger did not work. That’s weird, because it is the lowest level kernel command…

Couldnt that be adressed by a kresd developers here?
I have 3.X branch, and I remember exact same thing - time to time eMMC write (even when kresd and all other apps like adblock have defined storage on SSD card)