Omnia / 5.0 keep writing to mmcblk0 even though there are no apps and no reason to do so

Leonardo · June 10, 2020, 3:58pm

I didn’t notice that the rpz was saved in the MMC memory until now, I changed it to the sda.
At least one less write for me.
Thank you.

loom · June 10, 2020, 4:32pm

on my system the “/etc/root.keys” file was updated every hour so i symlinked it to the ssd card. “schnapps diff” does not show any updated files on the system. but i usually have all http frontends like forris and the lucid stopped.

vcunat · June 10, 2020, 4:49pm

Yes, that’s how it’s been packaged for Turris so far. I personally believe these keys should instead be packaged as static and updated as normal package data, just as we do this for most other distributions. So far I haven’t found time to look into doing that. (So far the keys have only rotated once during whole DNSSEC history!)

Leonardo · June 10, 2020, 5:25pm

The file is continuously be modified, but the content is the save.
I kept checking the modify time using “ls -l”, when it get modified, I can see writes in the MMC.
Since the key doesn’t change, why is the file modified?

vcunat · June 10, 2020, 5:37pm

The implementer simply didn’t optimize for this case A tiny write once an hour is nothing normally, especially compared to the cache.

Leonardo · June 10, 2020, 9:10pm

I think this is the file:

github.com

CZ-NIC/knot-resolver/blob/master/daemon/lua/trust_anchors.lua.in

-- SPDX-License-Identifier: GPL-3.0-or-later

-- Load the module
local ffi = require 'ffi'
local kres = require('kres')
local C = ffi.C

local trust_anchors -- the public pseudo-module, exported as global variable

-- RFC5011 state table
local key_state = {
	Start = 'Start', AddPend = 'AddPend', Valid = 'Valid',
	Missing = 'Missing', Revoked = 'Revoked', Removed = 'Removed'
}

local function upgrade_required(msg)
	if msg then
		msg = msg .. '\n'
	else
		msg = ''

This file has been truncated. show original

Should we open a ticket to remind the implementer to optimize for this case too?

This is the only process writing regularly to mmc.

vcunat · June 10, 2020, 9:26pm

Instead of that optimization:

vcunat · June 10, 2020, 9:31pm

Symlinking the file probably won’t help. To ensure atomicity, the file is moved into the location instead of rewritten in-place.

loom · June 11, 2020, 12:09am

are you sure? my root.keys on the ssd gets updated and the symlink is still there. maybe the update script fails, i haven’t verified this.

Jirka · June 11, 2020, 4:08am

Thats great find. I overlooked this as well. Because resolver config already have rundir directive, I thought that rpz and rundir would be enough.

What about modify path directly in /etc/config/resolver instead symlink?

option keyfile <SSD path>?

I just did this fix, will again turn on disk @collectd to see, how it is.

Edit:
Hmm, something else seems to do still some writes (root.keys change was done 6.10, I leave original keys in /etc just to be sure)

root@Doma:/etc ls -la | grep root.keys
-rw-r--r--    1 root     root           416 Jun 11 05:09 root.keys
root@Doma:/etc ls -la /mnt/SSD/kresd/root.keys
-rw-r--r--    1 root     root           416 Jun 11 06:10 /mnt/SSD/kresd/root.keys

Not sure what it could be.

vcunat · June 11, 2020, 2:11pm

How did you determine that it’s updated every hour? Normal update is on resolver (re)start and then once a day.

vcunat · June 11, 2020, 2:38pm

Anyway, the the /etc/root.keys part should be solved soon. You can follow details on https://gitlab.labs.nic.cz/turris/turris-os-packages/issues/609

loom · June 12, 2020, 12:13pm

The file never had a last write time older than 1 hour. Maybe i found it somewhere in the code, but i investigated this long ago. I assume i also tried to modify the roots.key path in the init.d / lucid config. but that didn’t help either. Symlinking was my easy approach to get rid of the writes to the mmc without touching the code itself.

Leonardo · June 12, 2020, 8:43pm

Looking at “function keyset_write(keyset)”.
The function writes a temp file and then it renamed the filer to the “root.key”
Maybe the temp file is still written somewhere.

Leonardo · June 22, 2020, 2:30am

I updated my Omnia to v5.0.2
Writing is even worse, not sure why kresd restarts constantly and each time is writing the file.

2020-06-21 19:20:38 info kresd[9515]: [ta_update] next refresh for . in 1.08 hours
2020-06-21 19:21:16 info kresd[9961]: [ta_update] refreshing TA for .
2020-06-21 19:21:16 info kresd[9961]: [ta_update] key: 20326 state: Valid
2020-06-21 19:21:16 info kresd[9961]: [ta_update] next refresh for . in 1.08 hours
2020-06-21 19:21:21 info kresd[10329]: [ta_update] refreshing TA for .
2020-06-21 19:21:21 info kresd[10329]: [ta_update] key: 20326 state: Valid
2020-06-21 19:21:21 info kresd[10329]: [ta_update] next refresh for . in 1.08 hours
2020-06-21 19:23:10 info kresd[10996]: [ta_update] refreshing TA for .
2020-06-21 19:23:10 info kresd[10996]: [ta_update] key: 20326 state: Valid
2020-06-21 19:23:10 info kresd[10996]: [ta_update] next refresh for . in 1.08 hours
2020-06-21 19:23:15 info kresd[11352]: [ta_update] refreshing TA for .
2020-06-21 19:23:15 info kresd[11352]: [ta_update] key: 20326 state: Valid
2020-06-21 19:23:15 info kresd[11352]: [ta_update] next refresh for . in 1.08 hours
2020-06-21 19:24:50 info kresd[11967]: [ta_update] refreshing TA for .
2020-06-21 19:24:50 info kresd[11967]: [ta_update] key: 20326 state: Valid
2020-06-21 19:24:50 info kresd[11967]: [ta_update] next refresh for . in 1.08 hours
2020-06-21 19:24:55 info kresd[12335]: [ta_update] refreshing TA for .
2020-06-21 19:24:55 info kresd[12335]: [ta_update] key: 20326 state: Valid
2020-06-21 19:24:55 info kresd[12335]: [ta_update] next refresh for . in 1.08 hours
2020-06-21 19:25:31 info kresd[13342]: [ta_update] refreshing TA for .
2020-06-21 19:25:31 info kresd[13342]: [ta_update] key: 20326 state: Valid
2020-06-21 19:25:31 info kresd[13342]: [ta_update] next refresh for . in 1.08 hours

vcunat · June 22, 2020, 5:58am

The solution I linked above hasn’t landed yet. Until then, it’s normal to have one write on each (re)start and then every 24h.

Restarts don’t happen normally, and this is especially frequent. To look into that, I’d probably start by getting more logs (especially just before the crashes) by editing /etc/config/resolver section config resolver 'common' line option verbose '0' to '1'.

EDIT: note to self, followed up on ticket #1102025.

anon82920800 · June 22, 2020, 1:03pm

echo 1 > /proc/sys/vm/block_dump

then watch the log, e.g. logread -f | grep mmc.

Turn off the dumping with echo 0 > /proc/sys/vm/block_dump