Turris OS 6.5.0 is released!

viktor · January 23, 2024, 4:56pm

LED, LAN and wi-fi working after reboot?

helios21 · January 23, 2024, 5:07pm

strings /dev/mtd1 | grep U-Boot returns no result. However on mtd0 there is a result:

% strings /dev/mtd0 | grep U-Boot   
U-Boot SPL 2022.10-rc4-OpenWrt-r16653+119-44ce70f0e2 (Sep 15 2022 - 18:21:35 +0000

LED, LAN and WI-FI are working as far as I can tell. One network connection must work anyway, as I am sending this through the router. But I also tested WI-FI briefly and it connects. I am still only using the 2.4 GHz one I think, though. Also internal 1 TB mSATA SSD still works Apparently I did not succeed in bricking my device.

mionica · January 23, 2024, 5:27pm

Final edit: found a thread with pretty much my problem (was in Czech), and some mentions on the RC thread in English (where, unless I understood it wrongly, it was claimed that this exact problem was fixed?!).
Leaving this here for now, for non-Czech speaking users whom might get hit by this on the final 6.5.

–

The rescue update broke the flash for me (Turris Omnia, purchased in March 2017 on alzashop.cz I think), 2GB model.

Had to use kwboot to get to an usable bootenv, and flashed it to uboot-devel (from HBS omnia-uboot_2023.01-5) and omnia-initramfs-zimage* (from the nor_fw location) to get back to an usable system. Found the instructions here.
_ * see PS#2

That allowed me to boot, and I was able to run nor-update (per this page - thanks @jada4p above for the the link); I was kinda surprised to see it actually change stuff,

root@router:/# nor-update 
Verifying /dev/mtd0 against uboot ...
95ee7fbfc670b2655937271b428555c5 - /dev/mtd0
a3a04aa6143fe437e00b96fbcc8cd20f - uboot
Failed
Warning: Bad CRC, using default environment
Unlocking /dev/mtd0 ...
Erasing /dev/mtd0 ...

Writing from uboot to /dev/mtd0 ...     
Verifying /dev/mtd1 against rescue ...
6c453ee022850e0caca0163898ac3ad2 - /dev/mtd1
550b8eb1766f9348fe928ebcbbd3aa49 - rescue
Failed
Unlocking /dev/mtd1 ...
Erasing /dev/mtd1 ...

Writing from rescue to /dev/mtd1 ...     
root@router:/# reboot

but whatever. After a reboot, everything seemed to work and be as expected

root@router:/# nor-update 
Verifying /dev/mtd0 against uboot ...
a3a04aa6143fe437e00b96fbcc8cd20f - /dev/mtd0
a3a04aa6143fe437e00b96fbcc8cd20f - uboot
Success
Verifying /dev/mtd1 against rescue ...
550b8eb1766f9348fe928ebcbbd3aa49 - /dev/mtd1
550b8eb1766f9348fe928ebcbbd3aa49 - rescue
Success
root@router:/# schnapps factory-version
6.5.0

I’ve no idea why I had ended with with different checksums, but fine, seems solved now.

Here’s the relevant portion of my boot log (after the fix), in case it could help devs figure out what the issue was:

BootROM - 1.73
Booting from SPI flash

U-Boot SPL 2022.10-rc4-OpenWrt-r16653+119-44ce70f0e2 (Sep 15 2022 - 18:21:35 +0000)
High speed PHY - Version: 2.0
MiniPCIe/mSATA card detection... MiniPCIe
WWAN slot configuration... PCIe+USB2.0
Detected Device ID 6820
board SerDes lanes topology details:
 | Lane # | Speed |  Type       |
 --------------------------------
 |   0    |   5   | PCIe0	|
 |   1    |   5   | USB3 HOST0	|
 |   2    |   5   | PCIe1	|
 |   3    |   5   | USB3 HOST1	|
 |   4    |   5   | PCIe2	|
 |   5    |   0   | SGMII2	|
 --------------------------------
High speed PHY - Ended Successfully
mv_ddr: 14.0.0 
DDR3 Training Sequence - Switching XBAR Window to FastPath Window
mv_ddr: completed successfully
Trying to boot from SPI


U-Boot 2022.10-rc4-OpenWrt-r16653+119-44ce70f0e2 (Sep 15 2022 - 18:21:35 +0000)

SoC:   MV88F6820-A0 at 1600 MHz
DRAM:  2 GiB (800 MHz, 32-bit, 2T, ECC not enabled)
Core:  71 devices, 24 uclasses, devicetree: separate
WDT:   Started watchdog@20300 with servicing (60s timeout)
MMC:   mv_sdh: 0
Loading Environment from SPIFlash... SF: Detected s25fl164k with page size 256 Bytes, erase size 4 KiB, total 8 MiB
*** Warning - bad CRC, using default environment

Model: Turris Omnia
  MCU type: STM32
  MCU version: b5a8a24e007eb72be16aeb3fff6f03ec647023e4/67eddc9540526d0a9d9660f7a7867af9a28a68d6
  RAM size: 2048 MiB
  Serial Number: 0000000B0001500F
Disabling MCU watchdog... disabled
Regdomain set to **
pcie0.0: Link up
pcie1.0: Link up
pcie2.0: Link down
Net:   eth0: ethernet@70000, eth1: ethernet@30000, eth2: ethernet@34000 [PRIME]
Hit any key to stop autoboot:  0 
switch to partitions #0, OK
mmc0(part 0) is current device
Scanning mmc 0:1...
Found U-Boot script /boot.scr
1199 bytes read in 35 ms (33.2 KiB/s)
## Executing script at 01800000
gpio: pin gpio@71_4 (gpio 68) value is 1
21434 bytes read in 39 ms (536.1 KiB/s)
4328320 bytes read in 285 ms (14.5 MiB/s)
Kernel image @ 0x1000000 [ 0x000000 - 0x420b80 ]
## Flattened Device Tree blob at 02000000
   Booting using the fdt blob at 0x2000000
   Loading Device Tree to 0fff7000, end 0ffff3b9 ... OK

Starting kernel ...

Anyway. While I appreciate the fact that the rescue image updating support was optional and not selected by default, I’m not sure it should’ve been offered at this point in time - I was lucky enough to have the knowledge I needed to fix it (and the bits I wasn’t sure about, I found in the forums and docs), but really guys.

Just because somebody gets an Omnia, you can’t assume they’d be able to do this.

It also helped tremendously that I had the foresight to equip the router with a USB serial port in advance (just in case) - I imagine this would’ve been a fair bit less convenient if I had had to open the thing up and hook up the wires of an FTDI cable (if anybody else wants to do this - I drilled a USB-C-shaped hole in the front of the case, and got through it the USB header off a CH340E which I had sticky-taped to the top the case internally, and had connected RX/TX/GND to the Omnia board header).

Anyway, posting this long rant here so somebody in the same situation would hopefully find everything they needed to recover their router promptly, in one place.

And FWIW, I’m not complaining. I understood the risk, took it, had to recover, and I did.
But still, guys. Seems kinda dangerous to make this kind of options available to potentially less geeky customers.

PS. The initial failure looked like this:

Booting from SPI flash

U-Boot SPL 2022.10-rc4-OpenWrt-r16653+119-44ce70f0e2 (Sep 15 2022 - 18:21:35 +0000)
High speed PHY - Version: 2.0
MiniPCIe/mSATA card detection... MiniPCIe
WWAN slot configuration... PCIe+USB2.0
Detected Device ID 6820
board SerDes lanes topology details:
 | Lane # | Speed |  Type       |
 --------------------------------
 |   0    |   5   | PCIe0	|
 |   1    |   5   | USB3 HOST0	|
 |   2    |   5   | PCIe1	|
 |   3    |   5   | USB3 HOST1	|
 |   4    |   5   | PCIe2	|
 |   5    |   0   | SGMII2	|
 --------------------------------
High speed PHY - Ended Successfully
mv_ddr: 14.0.0 
DDR3 Training Sequence - Switching XBAR Window to FastPath Window
mv_ddr: completed successfully
Trying to boot from SPI
ERROR: Invalid data checksum in kwbimage
Trying to boot from BOOTROM
Returning to BootROM (return address 0xffff05c4)...
BootROM: Image checksum verification FAILED
BootROM: Bad header at offset D4200000
BootROM: Bad header at offset D4400000
BootROM: Bad header at offset D4600000

BootROM - 1.73
Booting from SPI flash

U-Boot SPL 2022.10-rc4-OpenWrt-r16653+119-44ce70f0e2 (Sep 15 2022 - 18:21:35 +0000)
High speed PHY - Version: 2.0
MiniPCIe/mSATA card detection... MiniPCIe
WWAN slot configuration... PCIe+USB2.0
Detected Device ID 6820
board SerDes lanes topology details:
 | Lane # | Speed |  Type       |
 --------------------------------
 |   0    |   5   | PCIe0	|
 |   1    |   5   | USB3 HOST0	|
 |   2    |   5   | PCIe1	|
 |   3    |   5   | USB3 HOST1	|
 |   4    |   5   | PCIe2	|
 |   5    |   0   | SGMII2	|
 --------------------------------
High speed PHY - Ended Successfully
mv_ddr: 14.0.0 
DDR3 Training Sequence - 2nd boot - Skip

(and then nothing, until the watchdog kicked in and rebooted the thing)

However, the SPI flash is clearly still good (or it wouldn’t have worked a second time, I expect), and - this is the bad bit - I received no warning/error of any kind about flashing the NOR having failed. And yes, I did wait after the update until htop showed me that the writing to /dev/mtd1 had finished.

PS#2. Might be time to update the nor_fw folder; also, a note that you’re supposed to use image.fit .lzma from rescue-image_3.6.1-35 would be nice as well - I initially tried image.fit and panicked about having a too old, with too small NOR, router, for the current firmware I had to strings $(which nor-update) | grep rescue to figure I was attempting to flash the wrong file - not that it’d matter, if you just provided the right file as omnia-initramfs-zimage in the nor_fw folder

helios21 · January 23, 2024, 5:34pm

Seems I got really lucky then. Thanks.

Running nor-update again shows the same checksums and success messages as in your post.

Robo_Kupka · January 23, 2024, 5:53pm

Same issue here with postfix too.

michalko58 · January 23, 2024, 6:38pm

I cannot acces my Omnia with NAS perk via SSH after update. reForis and LuCI seems to be accessible, Transmission too, but SSH says Session not connected. LXC running on that Omnia is accessible via SSH.
Reboot didn’t helped…

Also the same with my main Omnia, which was updated, but still not restarted…

Edit:

Syslog message:
Jan 23 19:02:43 Pivnicka_NAS sshd[7602]: Unable to negotiate with 192.168.1.177 port 40952: no matching host key type found. Their offer: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521 [preauth]

Jan 23 19:07:53 turris sshd[12168]: Unable to negotiate with 192.168.1.177 port 57460: no matching host key type found. Their offer: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521 [preauth]

I am using Snowflake SSH client, but I am unable to connect from terminal too. Any advice?

tonyquan · January 23, 2024, 6:56pm

Turris Omnia, never updated the firmware. 6.5.0 installation successful, and nor-update successful:

root@turris:/tmp/log# nor-update
Verifying /dev/mtd0 against uboot ...
02ac2a52595476d095a4a8157bd401ca - /dev/mtd0
a3a04aa6143fe437e00b96fbcc8cd20f - uboot
Failed
Warning: Bad CRC, using default environment
Unlocking /dev/mtd0 ...
Erasing /dev/mtd0 ...

Writing from uboot to /dev/mtd0 ...     
Verifying /dev/mtd1 against rescue ...
d4a7af8e7a2799252d6b555a0a6a2e08 - /dev/mtd1
550b8eb1766f9348fe928ebcbbd3aa49 - rescue
Failed
Unlocking /dev/mtd1 ...
Erasing /dev/mtd1 ...

Writing from rescue to /dev/mtd1 ...     
root@turris:/tmp/log# reboot

root@turris:/tmp/log# nor-update
Verifying /dev/mtd0 against uboot ...
a3a04aa6143fe437e00b96fbcc8cd20f - /dev/mtd0
a3a04aa6143fe437e00b96fbcc8cd20f - uboot
Success
Verifying /dev/mtd1 against rescue ...
550b8eb1766f9348fe928ebcbbd3aa49 - /dev/mtd1
550b8eb1766f9348fe928ebcbbd3aa49 - rescue
Success

mizoch · January 23, 2024, 7:46pm

After upgrade I have problem with OpenVPN connection between two Turris routers. Anybody else?

DIKKEHENK · January 23, 2024, 9:03pm

Nope, running fine here?

dkomrska · January 23, 2024, 10:15pm

Turris Omnia (RTROM01), update from 6.4.4 to 6.5.0. RIPE + LXC installed. No issues observed.
Thank you for the update!

miska · January 23, 2024, 10:17pm

Try newer ssh client/newer configuration, some obsolete ciphers were disabled by OpenSSH update.

miska · January 23, 2024, 10:21pm

Postfix is currently not building due to some dependencies issues, we will look into that.

mionica · January 24, 2024, 12:48am

Completely unrelated topic from my previous post.
fw_printenv complains about a bad CRC; this was previously reported on the RC thread, but it seems 6.5 final hasn’t fixed it either (by providing the correct contents for /etc/fw_env.config.
And yes, I did do a saveenv in U-Boot - so U-Boot itself no longer complains about the CRC error causing it to load the default environment.

My current fw_env.config is

/dev/mtd0 0xc0000 0x10000 0x40000

In my view, the hook script that installs u-boot should update this file as well. Might be something worth doing for 6.5.1 or whenever.

Edit (before I got to hit “Reply” :>): I’ve looked in the 6.5.0 medkit, and it seems there’s no /etc/fw_env.config in it at all?!

Edit #2 (also before I got to hit “Reply”): a look at the device tree (specifically at /sys/devices/platform/soc/f1010600.spi/of_node/spi-nor@0/partitions/partition@f0000/reg) made me find out that the following /etc/fw_env.config contents work (at least with fw_printenv):

/dev/mtd2 0x0 0x10000 0x10000

that said, while it looks correct, I’d like somebody to confirm it actually is, before me - or anyone else - tries to run fw_setenv…

Edit #3: After digging through nor-update, it seems there’s some fw_printenv stuff being done as part of uboot_preupdate()…

This might not be unrelated to my previous post after all

The fact that my old uboot had environment on /dev/mtd0, not /dev/mtd2, at 0xc0000, means that the old uboot image was necessarily smaller than 0xc0000; the new one is 0xe8f00. So attempting to write the new one would necessarily thrash my existing environment (which wouldn’t be so bad), but begs to question whether /dev/mtd0 wasn’t somehow smaller than 0xe8f00… (FWIW, I last installed cleanly 3.11.2 in Feb.2019; it’s been all upgrades since, and I keep a snapshot for every upgrade).

'cause the way I see it, if you attempted to write a 931KB image over a partition smaller than that could only go two ways: either only part of the image would be written (because there’s no room), leaving you with a partial update and missing code, or mtd could simply refuse to flash to a too-small partition.

in the first case, it’s possible nothing was wrong with DDR training - instead, that simply happened to be the last thing printed before uboot jumped to code that simply hadn’t been written to flash
in the second, if the new uboot was required for the updated rescue system image (because the rescue partition, /dev/mtd1, was clearly large enough to be flashed correctly), it’s only natural that the system couldn’t boot.

Now, I have no proof of this, but I could swear I saw a 2015 uboot header when I connected the serial right after it first failed to boot, before I went serial booting and flashing the NOR. (I only remember this because I went, “jaysus, they updated all that code but wouldn’t update the bloody version string” before getting back to the task at hand).

Anyway. If any of those two failure scenarios were the real issue, then I think the problem is that nor-update doesn’t check proprly what it’s attempting to write to.
I’d strongly suggest changing the check logic in uboot_preupdate() to first and foremost check /proc/mtd for the expected partition layout, and refuse to update either uboot or the rescue system unless it looked ok (specifically, unless the mtd partition map looked exactly as expected). Ideally, doing something to mitigate that, like enlarging /dev/mtd0 and ideally relocating the uboot environment for the next boot (not really necessary since we know we can boot just fine with a broken environment, and anyone whom had to modify their bootenv knows pretty well how to fix things for himself in 5mins or less).

My 2 cents. Hopefully the Turris team can do something with the insights above.

peci1 · January 24, 2024, 4:02am

IIRC, /dev/mtd2 is only an “alias” of some space in /dev/mtd0. These two just overlap and are meant to. So it doesn’t necessarily mean that the old image had to end before mtd2 start.

tuomaz · January 24, 2024, 6:13am

Turris Omnia, installed 6.5.0 and updated firmware with nor-update. Result: multiple spontaneous reboots in the 18 hours that has passed since. How can I debug this in the best way?

mionica · January 24, 2024, 7:35am

I’m not sure that’s correct as stated. If you mean that several might be located on the same flash device (in this case), you’re correct, but that doesn’t mean they overlap; indeed, they shouldn’t, and with the current firmware, they don’t (check sysfs).

I honestly have no idea what the old uboot’s device tree looked like, nor do I know whether that gets merged into the kernel DT or the kernel simply provides its own description of the same stuff (I suspect the former), but from what I found in fw_env.config, I strongly suspect there was no /dev/mtd2 with the old uboot.

And here’s the second bit. uboot_preupdate() in nor-update attempts to dump the environment (suceeding), and uboot_postupdate() attempts to restore it. However, unless you reboot inbetween (so you get the updated mtd table from uboot) or update fw_env.config, the DT hasn’t changed, so if you environment was on /dev/mtd0 at 0xc0000, it will be written back in the exact same location (no /dev/mtd2 exists yet FWIW). And if you just wrote wrote 0xe8f000 bytes to /dev/mtd0 at 0, and then another don’t-care-how-many to /dev/mtd0 at 0xc0000… There you go, instant uboot code corruption.

My talk at 2am local about mtd0 being too small doesn’t hold much water now at half 9, true (must’ve been at least 0x100000, since the environment was included at the end of mtd0). But the “check the damn /proc/mtd” bit still makes sense IMO.

The only thing that nags me is that I could swear I saw a 2015-something uboot header on after the presumed update. But it’s not impossible that happened not during the failure, but rather when I was messing around with kwboot, attempting to resurrect my main router (I wonder whether that’s what you get on the turris server at nor_fw…)

AreYouLoco · January 24, 2024, 8:41am

I think that is correct. Look here:

And here:

And here as well:

With new U-Boot there is u-boot-env added. But this was known 1 year ago and there is still no config for fw_printenv in medkit… And the fix is simple as to add one file and its done. Two Issues solved…

mionica · January 24, 2024, 9:44am

That’s a good point; if I understand it correctly, and my reasoning is correct, then,

no system that lacks an /etc/fw_env.config would be broken, since saving/restoring fw_env isn’t going to happen during nor_update
a system that has an /etc/fw_env.config that actually matches the uboot2022 config will have no trouble updating
a system that has an /etc/fw_env.config that has the old layout (mtd0+0xc0000) will be broken like mine was, because the uboot_postupdate fw_setenv -s will overwrite part of the uboot code

Edit: On the bright side, (particularly since I expect /etc/fw_env.config is not present on most omnias), this means that the number of affected people will be fairly low.

Probably the simplest possible solution in order not to break anything would be to not restore the boot environment if fw_env.config is present and has the old layout.
The better but more complicated solution would be to back up the old environment
dd if=/dev/mtd0 bs=0x10000 of=/usr/share/nor-update/ubootenv skip=12 count=1
as part of pre-update) and somehow write it to the new location; no simple solution comes to mind - but what can be done could be something like preparing a mtd0 image that included the old environment, to be flashed before the kernel starts using the new mtd layout:

# in pre-update
dd if=/dev/mtd0 bs=$((0x10000)) skip=$((0xC)) count=1 > /tmp/uboot-env
# prepare update image by assembling the uboot, padding, and the old environment
ubootfile=/usr/share/nor-update/uboot
ubootbinsz=$(ls -l $ubootfile|awk '{print $5}')
if [ $ubootbinsz > $((0xf0000)) ]; then
  echo uboot image too large\; aborting...
  exit
fi
cat $ubootfile > /tmp/uboot-with-env
dd if=/dev/zero count=$((0xf0000-ubootbinsz)) bs=1 >> /tmp/uboot-with-env
cat /tmp/uboot-env >> /tmp/uboot-with-env
# write the created partition image to the current mtd0 (that'll show up as mtd0+mtd after the uboot changes)
dd if=/tmp/uboot-with-env of=/dev/mtd0 bs=$((0x10000))

Edit #2: ok, I’m stupid. If we have the old layout in /etc/fw_env.config, it’s enough to replace the condition of calling fw_env-config-init.sh, which will do exactly what needs to be done.
However. fw_env-config-init.sh sets the config to /dev/mtd0 0xf0000 0x10000 0x10000 only if
U-Boot 2015.10-rc2 is not found in /dev/mtd0 - so calling in before updating uboot is the wrong place.
If however it was called in postupdate (after uboot was flashed) but before fw_setenv -s was called, I expect it’ll do what’s actually intended.
As far as I’m concerned, case solved.

viktor · January 24, 2024, 11:22am

Could you post the new default IPv6 settings please? I would like to consider using them, but I don’t want to reinstall my router.

DIKKEHENK · January 24, 2024, 12:34pm

Hello there, here a ‘enthusiast amateur’ aka nOOb. Whats the verdict for someone with a standard indy omnia ( 2015 U boot version me think) at this stage?