Omnia / 5.0 keep writing to mmcblk0 even though there are no apps and no reason to do so

[xsys]

I noticed since I moved to HBT - 5.0, there are periodical writes to MMC. I’m quite sure it was not there when I was on 4.x version, the Write red part was almost clean.
My Omnia is almost in default settings, with just some custom Network Interfaces and Firewall rules and collectd statistics. Hence no LXC, no additional apps, no one checkbox at http://192.168.1.1/foris/config/main/updater/ used.

/srv is mapped to /dev/sda, what is internal SSD, but currently it is used only for storing RRDTool output [/srv/rrd].

root@turris:~# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/mmcblk0p1            7.3G    722.3M      6.6G  10% /
devtmpfs                512.0K         0    512.0K   0% /dev
tmpfs                   512.0K         0    512.0K   0% /sys/fs/cgroup
tmpfs                  1009.8M     24.4M    985.4M   2% /tmp
tmpfs                   512.0K         0    512.0K   0% /dev
/dev/sda                 55.9G     17.7M     54.9G   0% /srv

Any idea what could cause that, how to investigate, and more importantly - is it harmful to the MMC wearing off?

[anon82920800]

@xsys

how to investigate

manual monitor disk writes by processes with htop (customized columns)

---

@xsys

and more importantly - is it harmful to the MMC wearing off?

did you see eMMC - live span, health monitoring?

[k_205]

To be honest 4.0.5 behaves the same.

I can confirm there are every 30 minutes writes from two pocesses:
/sbin/procd
/usr/bin/kresd

[xsys]

htop shows ‘no perm’ for DISK_WRITE, searching for solution suggest it is something with kernel what goes beyond my knowledge.
I read the eMMC… discussion but do not see any resolution. On this forum I saw many people having dead MMC though not using extensive-write apps on it.
mmc extcsd read /dev/mmcblk0 shows 0x01, what should be fine, but for how long…
I have used the STORAGE plugin in Foris that shows green Device currently in use is sda...
I would like to use the internal SSD for everything write-related.

[anon82920800]

@xsys

htop shows ‘no perm’ for DISK_WRITE

The graph posted in your initial report shows writes every hour or so, you would have to watch the htop output for one hour (worst case) to catch up. A more automated (but also more complex) approach could perhaps be with sysstat.

---

@xsys

I read the eMMC… discussion but do not see any resolution.

Every type of storage medium has a limited lifespan and NAND flash is perhaps less durable than SSD.

@anon82920800

OpenWrt utilises F2FS (Flash-Friendly File System) for that purpose as are quite a few of Android flavours.

---

@xsys

mmc extcsd read /dev/mmcblk0 shows 0x01, what should be fine, but for how long…

as mentioned in the thread

@cynerd

You should know that this is not linear. This is secondary statistics and tells you nothing about complete wear. Common wear is an avalanche effect. Nothing is happening for a long time and then all blocks starts failing at once. Meaning percentage of used reserved blocks might start rising very quickly.
[…]
Flash memory in TO is (up to my knowledge) SK hynix eMMC4.5 which manufacturer specifies, for density of 4GB, 2.4TB total bytes written before EOL.

---

@xsys

I would like to use the internal SSD for everything write-related.

Not sure if the developers could cover via the Foris storage plug-in each and every userland, that may potentially write to the OS’s drive, that is available in the distro - seems they got the mainstream apps covered.

Or else you could run the OS entirely off

@anon82920800

• interior SSD
• interior USB drive (requires additional adapter)

Running from an exterior USB drive is currently only supported on the Mox platform but not for the Turris or the Omnia.

and thereby prevent eMMC usage altogether.

[xsys]

@anon82920800

you would have to watch the htop output for one hour

It still shows no perm, even when I let it run for hours and also I do extensive writing to the SSD:

[xsys]

@anon82920800

if the developers could cover via the Foris storage plug-in each and every userland

As I said, I don’t have anything more than default settings + Collectd, such config should definitely be covered.

[xsys]

@anon82920800

interior SSD

For each option, you need to have the serial cable and flash disk with the medkit.
That’s not reachable for me

[anon82920800]

@anon82920800

@anon82920800

interior SSD

For each option, you need to have the serial cable and flash disk with the medkit.
That’s not reachable for me

It can be done/prepared through ssh whilst running the OS from the eMMC. Unfortunately, it is not documented but there are few related threads in this forum. Only UART serial cable should be available in case something goes wrong.

[anon82920800]

@xsys

It still shows no perm, even when I let it run for hours and also I do extensive writing to the SSD:

Well, apologies for having sent you on a wild goose chase since the relevant kconfs are unset.

@Pepe

These kernel options necessary for disk activity in htop are indeed not enabled and shows no perm .

[xsys]

What i would like to know is what i can do to have all writes done to SSD. I used the Storage function in Foris, that is supposed to make it work (in case of all default settings) ?

[peci1]

I think you should better specify what you mean by “all writes”. If you want to run the system from eMMC, then there will always be system updates which need to write to the eMMC. So at least these writes are unavoidable.

One idea I got is the following - could you try to remount the root (/) readonly after boot? That way you could be absolutely sure nobody is writing anything to the eMMC. And looking in the logs, you would probably find the apps that try to write someting. I’d also be quite interested in knowing who the bad guys are. But if I remember correctly, every time my filesystem got into readonly mode (due to some error) on TOS 3.x, resolving (a.k.a. kresd) stopped working immediately.

At least kresd is under the maintenance of CZ.NIC, so they should be able to explain why and what does it write. procd is the generic “init” process, so that one is very unspecific.

Another way of finding out what’s writing and where could be periodical saving of the output of lsof (with a well curated set of parameters). If you roughly know the times at which the writes happen (they seem periodical), you can just run this for a few minutes and if you’re lucky you’ll catch the bad guys in action.

As @anon82920800 said, disk utilization via htop/iotop is unavailable in TOS 4/5. In 3.x it works. If you’d like to have it available in TOS 4/5, please write it in Where are iftop, iotop, nethogs and friends? to show CZ.NIC there is a non-singular user base that is interested in having the required kernel flags set. They said it’s possible to enable everything that’s needed for showing disk utilization, but maybe they just need a little bit motivation to do so

[Leonardo]

@peci1

could you try to remount the root ( / ) readonly after boot

Do you think it is possible to test this in a LXC container?

[peci1]

Nope. LXC doesn’t IMO have such a level of filesystem virtualization.

[vcunat]

I’m not sure about the lxc/lxd capabilities, but the underlying mechanism of mount namespaces does allow such things. BTW, NixOS.org uses this in practice: the directories with installed packages are read-only by default to avoid accidental writes even by root, and the “installer” runs in a mount namespace where it’s read-write.

[X-dark]

If you really want to have all writes on SSD, move the root partition to your SSD and stop using the emmc. This is what I done. It is working fine but the migration may give you some difficulties (need to change UUID, need to update uboot settings,…).

[xsys]

@peci1

what you mean by “all writes”

I mean ‘unnecessary’ writes, where I understand software updates are necessary to get written, but application things like DNS resolver or DHCP server should not need to write to eMMC, they can store all their running data in RAM, or I would like to redirect those to the SSD.

@peci1

remount the root ( / ) readonly

This goes far beyond my knowledge and I’m afraid to brick my router

@peci1

please write it in Where are iftop, iotop, nethogs and friends? to show

will do

@X-dark

move the root partition to your SSD

I don’t feel confident to perform such operation

[X-dark]

@xsys

@X-dark

move the root partition to your SSD

I don’t feel confident to perform such operation

Do not try then. It may require you to use serial access and such and else may render your router unusable.

[peci1]

@peci1

remount the root ( / ) readonly

Hmm, I did a few tests, but no known method worked on Omnia. The RW-mounted filesystem refuses to remount readonly.

Even echo u > /proc/sysrq-trigger did not work. That’s weird, because it is the lowest level kernel command…

[Jirka]

Couldnt that be adressed by a kresd developers here?
I have 3.X branch, and I remember exact same thing - time to time eMMC write (even when kresd and all other apps like adblock have defined storage on SSD card)