Turris 1.x (Blue) Factory Reset to Turris OS 3.8.5 Catch 22 - no SSH, no internet, no updates, nothing working, bricked?

Software SW bugs discussion
1

After a pretty good uptime on 6.x my Turris 1.x just died today. Had to do a factory reset, which brought it back to Turris OS 3.8.5 recovery image, which is way old, full of bugs and unsupported.

Which kinda 100% broke everything and made it totally unrecoverable even onsite…

  1. No SSH – sshd is not even running for some weird reason, not even visible in LuCI process list.
    ssh: connect to host 192.168.1.1 port 22: Connection refused
    Even tried to run it manually via LuCI’s Command interface, not in the $PATH and not found.

  2. No internet – using PPPoE here, and nowhere to set VLAN ID, thus no connection. Backup internet through LTE or iPhone Hotspot not working either, because 3.8.5 doesn’t even support WPA2 out of the box and only supports WEP or unprotected WiFi which isn’t supported by anything else.

  3. Without internet, no updates to at least latest 3.11.x. Without SSH, not even a chance to update manually via images if even possible on TOS 3.x without BTRFS.

  4. No BTRFS, obviously, even though the SD card is there and was booting just before it borked. No way to start the btrfs_migrate script, since it doesn’t exist in 3.8.5, and cannot even do it manually, since no SSH.

  5. Without BTRFS & SSH, no way to use schnapps to manually flash a 6.x HBS medkit.

  6. Catch 22 – the router is as good as dead.

This is the second time this happened to me – the first time was a fucked up Turris OS 5.x update that bricked old Blue Turris 1.x, quickly corrected via a hot patch a day later. This time, it just died on its own for some reason.

The problem is that once you do a Factory Reset on Turris 1.x, you are stuck with 3.8.5. Which might not be able to connect to modern internet at all (FTTH, PPPoE, WPA3, …), thus not being able to update itself, a Catch 22.

And 3.8.5 does not have a working SSH (yes, I did set a root password for LuCI!), so you can’t even try installing stuff manually from a flash stick.

The only way I was able to recover was to go on‑prem and physically haul the router 10 km away where I could plug it behind a VDSL router where it was able to at least update itself to 3.11.x, where I could finally get in with SSH and run the BTRFS migration script again and upgrade to 6.x via the latest medkit.

Not the greatest way to spend the last day of 2023, though it could have been worse – the other location 10 km away was at least family premises, so it wasn’t all that wasted time…

Hence my questions:

A) Any possibility to update the Factory Reset Recovery Image in Turris 1.x from 3.8.5 to 3.11.6, or whatever is the latest 3.x, with all the packages and scripts? I am even open to flashing the EEPROM, or wherever is it stored. The default Factory Reset 3.8.5 image is just totally useless…

B) Any possibility to update it manually even without working SSH in 3.8.5? I guess I could still get in on‑prem via a serial cable? If so, any way to flash the 3.11.x image to it via a serial connection? Because a router that basically bricks itself on factory reset is kinda useless to me. Especially if some future 6.x possibly borks it again.

I could always buy a modern Microtik and chain the Turris behind it, I guess, but that kinda defeats the whole purpose – it still works great and this was the only second major outage after that single bad 5.x patch, and still serves its purpose great otherwise.

The Turris 1.x documentation is totally outdated and parts of it are even 404, with broken links (had to use Google Cache for fucks sake to find some docs), so asking here in the forums…

1 Like
2

Usually, this kind of problems is solved using a serial cable. But i only have Omnias, so maybe I’m wrong.

This is a guide for Omnia: Omnia - Turris Documentation .

3

Thanks! Unfortunately that doc only addresses the Omnia, not the Turris 1.x :frowning:

And I’d rather not totally bork my 1.x Blue Turris via flashing of the NOR through the serial per incompatible Omnia instructions. Especially as the repo itself is quite different for 1.x firmware and U‑boot files (understandably, as the HW itself is different).

All I am asking if it’s even possible to flash the 1.x with a more recent recovery image than 3.8.5, since that one doesn’t work that well at all.

Perhaps not, the NOR flash might not even have space enough for 3.11.22 or whatever is the latest, for all I know. But it would be really nice to have an official word on that, because the current factory reset of 1.x devices often leaves them near useless and unrecoverable.

I don’t even know why ssh doesn’t work on 3.8.5 there. I tried to run the daemon via the LuCI Commands interface, but that’s just a really poor excuse for a proper terminal access.

Serial might be the only option, though I was really hoping a 3.11.22 (?) could be flashed as a recovery image into the NOR flash. I was able to easily install the latest TOS 6.x via a medkit once I could get into it via ssh once the btrfs boot worked on 3.11.x. As the SD card with the last btrfs install was still inside the 1.x, so just switching the boot to that might have helped, if possible (though the bricked condition could have been due to corruption of the SD card itself, no idea).

4

Oh, I really did not want to suggest you to apply Omnia NOR to 1.x (I only posted it to illustrate how it is done on Omnia).

More interesting for these cases is the possibility to boot over serial. It uploads a temporary OS directly to the RAM, so you can boot it and try to fix whatever remained of your system.

1 Like
5

Thanks. I’d have to really look up the serial connection and possibilities of – any pointers please? As I am not really sure all of the current docs apply to the venerable old 1.x models, as the docs are a bit chaotic :wink:

I might still have to resurrect an old Zyxel or some shit like that just to connect if that happens again, to let the Turris update to 3.11.x, but I’d rather have a better solution if possible at all.

6

3,3 V

https://wiki.turris.cz/en/troubleshooting/serial_link

7

There’s a lot of discussion in the 6.5RC thread about firmware updates, including updating the factory image. It doesn’t quite delve into how to update to a particular named image, but there’s a place to start digging.

(In particular, for a 1.x maybe a recent 6 image as factory might not be the best choice given the kernel issues on that hardware? As an Omnia user I was happy with using 6.x for the recovery image but for 1.x hardware perhaps a late 3.11.x image would be a safer bet. I guess the problem with having ancient recovery image is that the config is so far out of sync as to be unrecognised, at least in part, by the image?)

8

Uh, ohh… What a hell of discussion became here. Several posts are off topic and not helpful. Particularly, I wonder why I am replying since it is not my job anymore and Turris Team does not care about us - no Christmas post, no Happy New Year, etc.

So, let me be helpful here before I will disappear from this forum, because GitLab activity can reveal what is happening and that there is not going to be change anymore. Also, this was my daily job to recover something what was in this state and it was not easy to recover and also it was hard work to do anything systematic. BTW: This is all prepared and Turris OS 6.x can boot and it fits into NAND, this is proven! But what it needs is time. Turris package maintainers should look at it, but I wonder when the last version of Turris OS was released to the stable, well… maybe they should hire more people. :slight_smile:

Tell me about it. I spent Christmas evening helping my sister how to configure PPPoE on Turris Omnia, which was put into the factory version instead of reboot. Old Foris, two version of Foris. Oh man, one hour wasted on this one, which should be there since beginning. LuCI helped and it works there. Not sure about OpenWrt 15.05 or what Turris OS 3.x version is using, but in LuCI, I bet you can do that.

I wonder why it borked. That does not happen to me. Which microSD card are you using? Are you sure that yours microSD card is reliable, trustworthy?

I would not say something like “pull requests are welcome, we appreciate our help” as it was most common response from Turris team. The Turris 1.x documentation can be found here:

This responds to your B question.

Sure, but you need to have working Internet connection without that, you can not do that. Its logic and not useless. :slight_smile: From 3.8.5, you can easily update to its the latest version, but make sure that time is correct, you have valid DNSSEC keys, etc.

Thats only for Turris Omnia and most likely for Turris MOX. It is not for Turris 1.x.

What? Which kernel issues? You mean those from kernel 5.15? These are kinda expected, because several things in Linux kernel for reworked and thats why there is kernel 5.10, perfectly stable and safe to use on Turris 1.x and many users are using Turris OS 6.x on Turris 1.x routers, so please avoid such misleading statements. :raised_hands:

If you are not sure, what are you doing, just write to Turris Support, bring your router to their office and within one hour, it should be up and running with the latest version. I was doing that for free as one-time courtesy for our users and I bet there are still doing that. :slight_smile:

5 Likes
9

Blue Turris 1.x will be migrated to 5.15 kernel in TOS 6.5:

We migrated to 5.15 kernel and fixed the problem that held us back at 5.10. Work well in our tests, but more testing in various configuration would be more than welcome.

10

I doubt it, seriously. Look at the thread for Turris OS 6.5:

  • factory update for Turris Omnia is a completely disaster

    • why? Missing tests, it was not properly tested, even though yeah, manufacturer can state: “You are doing it on your risks”, thats true, but seriously, this should not pass to the testing not even to the stable branch when you clearly know that IGG units will fail and serial cable is required. For me, it looks like as @Nones like to say this term “polotovar” in Czech, in the English, lets translate it to semi-working state.

  • Turris 1.x:
    They can not push the kernel 5.15 to the stable version, why? Because community found out that the system boots more than 10 minutes.

We can say, it is important that it boots after all and it is working, but seriously you want to push it to the normal users?

If they would rather focus on Turris OS 7.0 to have something up-to-date on our routers or rather focusing on upstream support (Kernel, OpenWrt), it will be better. In OpenWrt, it is not happening. I know, since I am still contributing there. :slight_smile:

1 Like
11

In the current state, partially yes (at least for IGG Omnias). But I think that’s what hbt is for, isn’t it? Moreover, the package list upgrading these dangerous things is marked experimental. As I understand it, Turris team wants more testing for this feature and this is one of the easy ways to distribute the changes for testing to users. Non-adventurous users are expected to not enable experimental things.

12

Experimental label isn’t equal to dangerous, thats what we can agree, right? Testing, which leads to bricked routers, hmm. Could be, but no. There should be some internal testing before that. If it was done only one unit instead of all Turris Omnia revisions, thats what should be improved.

I would expect that in the HBD or HBL, but for HBT? Even based on their doc with branches, it should not happen. It happened, because there were no tests, there is no time for that, because the team have just a few people. We can also argue how many people are reading this forum (this was often internal discussion) or if they prefer announcements on social media, etc. There was idea to have Turris News (click here for details), but it did not happen.

2 Likes
13

Hah! :wink: Can’t but love those holidays family tech support sessions, can’t we :smiley:

I did try in LuCI, but frankly, I am not that good at it, and as I wasn’t quite sure 3.8.5 supports my FTTH provider’s PPPoE correctly, or even if I was doing it right, I stopped after a few looks before some automated netmonitoring kicked me off for borking up their network as well – not a prospect I’d like to have or wish on their techs on New Year’s Eve :wink:

It’s a high endurance uSD from a good brand, but quite old by this time. Storage corruption is indeed a possibility I am looking into, though I don’t see anything pointing to that so far in the logs or terminal when formatting it. Anyway, I’ll be getting a new industrial uSD just to be sure…

I meant updating the NOR U‑Boot and rescue image, without a working connection, or at least the NAND via the medkit. I could connect via the USB serial (now that I am aware of it), but the recommended medkit update instructions required migration to btrfs first. Which I couldn’t do on the factory 3.8.5, as the packages or scripts were not there, as far as I could tell (though my terminal access was limited to LuCI commands at that time, as I didn’t have a uUSB cable handy).

Once I got it to autoupdate to 3.11.x via internet access elsewhere, btrfs migration and medkit update to 6.x was a breeze, obviously. But that doesn’t help much when the factory reset to 3.8.5 breaks all connectivity like my PPPoE.

I found this old doc on how to re‑flash the NOR U‑Boot and recovery image from uSD locally for Turris 1.x:
https://wiki.turris.cz/cs/troubleshooting/sdcard_recovery?rev=1503399543
but the boot image sdcard.img is dated 13‑Dec‑2021, though. Not really sure what version might it be? And not really ready to experiment with it without a clue :wink:

I think you mentioned in another thread that there might be a later 3.x U‑boot prepared by Turris devs, though not released (I gather that building for 3.x might be quite complicated by this time)?

Anyway, many thanks for the reply and have a great New Year!

14

I’ll have a look there. I’d agree that having a late 3.11.x rescue image in the NOR might be safer. I don’t care about config being unrecognisable, as won’t be using a backup config from a different version anyway, safer to start anew.

15

IIRC I think boot over serial is only available with Omnia and later HW, at least that’s what the docs say.

16

Hm, I might have missed that part of the 6.5RC announcement that might address the very issue I had:

We added another interesting item to the list of available packages. You can now select that you want to have the Latest firmware and your router will automatically install latest U-Boot, latest rescue system and if you are on Omnia, even latest MCU firmware. Apart from that it will update your factory image, whenever you update to new major release. So now, it will make your factory image contain Turris OS 6.4.4, but once we release Turris OS 7.0, it will get updated to Turris OS 7.0, so you wouldn’t have to migrate again from Turris OS 3.0. As it touches really crucial parts of your router, do not enable it when there is high probability of power failure. It can break your router in a way that is not easy to fix.

I guess I’ll have to read through the whole 6.5RC thread to see how it behaves on the old Turris 1.x (re: the above mentioned long boot time issue), as I am not yet ready for experiments at my end, at least not without another backup router.

17

Nope, Turris OS 3.x was also present on Turris Omnia routers.
See: Versions - Turris Documentation

It will only take care of Turris Omnia and Turris MOX routers as I said. It is not for Turris 1.x running on NAND, because it requires Btrfs. It is done via Schnapps, so for Turris 1.x which is booting from microSD card, it will work, but once your microSD card is dead, then it wont be helpful. You can see it here: