[SOLVED] Filesystem read only

Hi guys,

My filesystem has just turned read only. I can’t edit anything.

I have seen other people have same issues, but they fixed this problem by re-flashing. I have tried to go to a previous snapshot, but no salvation in that.

I have A LOT of options i do not want to lose, including a LXC that is up and running with A LOT of configurations.

Anybody a solution that is already known but i have somehow missed it?

root@K-Router:~# mount
/dev/mmcblk0p1 on / type btrfs (ro,relatime,ssd,space_cache,commit=5,subvolid=271,subvol=/@)
proc on /proc type proc (rw,noatime)
sysfs on /sys type sysfs (rw,noatime)
none on /sys/fs/cgroup type cgroup (rw,relatime,cpuset,cpu,cpuacct,blkio,memory,devices,freezer,net_cls)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noatime)
tmpfs on /dev type tmpfs (rw,relatime,size=512k,mode=755)
devpts on /dev/pts type devpts (rw,relatime,mode=600,ptmxmode=000)
debugfs on /sys/kernel/debug type debugfs (rw,noatime)
mountd(pid4018) on /tmp/run/mountd type autofs (rw,relatime,fd=5,pgrp=4016,timeout=60,minproto=5,maxproto=5,indirect)

Hello,
a few days ago I have similar case on IRC.

Re-flash in this case is solution.

LXC containers you should have on external device (usb flash drive, hard drive and so on).
See it here how you can move it from internal storage to somewhere else

Configuration it depends, what you want and what you would like to backup.
Most configuration files should be located in /etc/config

Eey Pepe long time no see.

Re-flashing in the sense of going back to factory defaults right? As my schnaps list is empty…

root@K-Router:~# schnapps list
mount: wrong fs type, bad option, bad superblock on /dev/mmcblk0p1,
missing codepage or helper program, or other error

   In some cases useful info is found in syslog - try
   dmesg | tail or so.
# | Type      | Date                      | Description

------±----------±--------------------------±--------------------------------
ERROR: cannot find real path for ‘/mnt/.snapshots/@ 5’: No such file or directory
5 | single | |
ERROR: cannot find real path for ‘/mnt/.snapshots/@ 7’: No such file or directory
7 | single | |
ERROR: cannot find real path for ‘/mnt/.snapshots/@ 12’: No such file or directory
12 | single | |
ERROR: cannot find real path for ‘/mnt/.snapshots/@ 138’: No such file or directory
138 | single | |
ERROR: cannot find real path for ‘/mnt/.snapshots/@ 139’: No such file or directory
139 | single | |
ERROR: cannot find real path for ‘/mnt/.snapshots/@ 140’: No such file or directory
140 | single | |
ERROR: cannot find real path for ‘/mnt/.snapshots/@ 141’: No such file or directory
141 | single | |
ERROR: cannot find real path for ‘/mnt/.snapshots/@ 142’: No such file or directory
142 | single | |
ERROR: cannot find real path for ‘/mnt/.snapshots/@ 143’: No such file or directory
143 | single | |
ERROR: cannot find real path for ‘/mnt/.snapshots/@ 144’: No such file or directory
144 | single | |
ERROR: cannot find real path for ‘/mnt/.snapshots/@ 145’: No such file or directory
145 | single | |
ERROR: cannot find real path for ‘/mnt/.snapshots/@ 146’: No such file or directory
146 | single | |
ERROR: cannot find real path for ‘/mnt/.snapshots/@ 147’: No such file or directory
147 | single | |
ERROR: cannot find real path for ‘/mnt/.snapshots/@ 148’: No such file or directory
148 | single | |
ERROR: cannot find real path for ‘/mnt/.snapshots/@ 149’: No such file or directory
149 | single | |
ERROR: cannot find real path for ‘/mnt/.snapshots/@ 150’: No such file or directory
150 | single | |
ERROR: cannot find real path for ‘/mnt/.snapshots/@ 151’: No such file or directory
151 | single | |
ERROR: cannot find real path for ‘/mnt/.snapshots/@152’: No such file or directory
152 | single |

No, I mean you need usb flash drive, download medkit on usb flash drive, plug it to the Turris Omnia.

See it here: https://doc.turris.cz/doc/en/howto/omnia_factory_reset#re-flash_router

WOOW…this is some serious problem then. What is going on? Is this problem known/already been worked on with the Omnia Team?

You have corrupted file system somehow probably because of LXC containers on eMMC or power failure.

We’re working on a few notifications that they will tell you that you shouldn’t install LXC containers on eMMC.

Hmm, well my LXC container is on my SSD. Majordomo even is also on my SSD. What runs on my eMMC is only the OS itself and some scripts. A few days ago we did experience a power outage through the whole city and some neighboring villages that lasted 2-3 hours.

The other thing that i often experience is when i disable/enable some interfaces Turris Omnia crashes/reboots. This however has happened AT LEAST 6 months or longer.

@Pepe

Today i finally found the time to do what it needed to be done. Everything so far succeeded, i am only stuck with LXC. How do i recover everything?

/etc/lxc/lxc.conf

lxc.lxcpath = /mnt/LXC/K-Router-LXC/

command “mount”

root@K-Router:/etc/lxc# mount
/dev/mmcblk0p1 on / type btrfs (rw,noatime,ssd,space_cache,commit=5,subvolid=256,subvol=/@)
proc on /proc type proc (rw,noatime)
sysfs on /sys type sysfs (rw,noatime)
none on /sys/fs/cgroup type cgroup (rw,relatime,cpuset,cpu,cpuacct,blkio,memory,devices,freezer,net_cls)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noatime)
tmpfs on /dev type tmpfs (rw,relatime,size=512k,mode=755)
devpts on /dev/pts type devpts (rw,relatime,mode=600,ptmxmode=000)
/dev/sda1 on /mnt/majordomo type btrfs (rw,relatime,ssd,space_cache,subvolid=5,subvol=/)
/dev/sda2 on /mnt/LXC type btrfs (rw,relatime,ssd,space_cache,subvolid=5,subvol=/)
debugfs on /sys/kernel/debug type debugfs (rw,noatime)
mountd(pid4150) on /tmp/run/mountd type autofs (rw,relatime,fd=5,pgrp=4146,timeout=60,minproto=5,maxproto=5,indirect)

cd & ls

root@K-Router:/etc/lxc# cd /mnt/LXC/K-Router-LXC/
root@K-Router:/mnt/LXC/K-Router-LXC# ls
K-Router-LXC K-Router-LXC.log rootfs

/etc/config/lxc-auto

#config container
#option name container1
# How long to give container to shutdown gracefully before killing it
# Default 300s
#option timeout 30
# Commnads to run before starting container
#list command ‘/bin/command --option’
#config container
#option name container2
#option timeout 60
config container
option name K-Router-LXC
option timeout 60

It just won’t detect. I rebooted multiple times. Maybe i am missing some package or so?

Happened to me too. No containers, no power failure, just removed old snapshots with schnapps. So it was actually your tool causing the issue :stuck_out_tongue_winking_eye:

Flashing from USB to the rescue.

In the future there maybe other people who fall in the same situation as i have and many before me.

Here a quick how to tutorial.

Step 1: Backup your configuration files. In my case the eMMC was READ-only so i could not mount even a USB-Drive. I first tried “SCP”-command through SSH, however somehow because the eMMC was still READ-ONLY i could not make it work. When typing “mount” you can see at the beginning of each line what mount-points are still writeable (RW). In my case my external(internal) drive where i had my LXC running was still writeable. So i made some directories and copied all “/etc/config” content to that backup directory on my internal drive. I also just incase copied the crontab as i had added some content to it. (fstrim lines for example) and some files in my root directory.

Step 2: When everything was backedup to another drive instead of the eMMC, i copied the medkit to the USB drive (used ext4 filesystem for the USB-drive) and put in the FRONT USB of the Turris Omnia.

Step 3: I then pushed and kep pushing the reset butting until LED 4 was burning (number 2 LED) and released it. It directly started to re-flash the entire eMMC. In maybe 1-2 minutes it was apparently already done.

Step 4: The Omnia should now be re-flashed and the router IP is by default 192.168.1.1. Go and type this in the webbrowser and you will reach Floris (default webinterface of the Omnia). Follow the entire wizard.

Step 5: removed

Step 6: When this was done i went to mount points and mounted the partitions and pressed save & apply.

Step 7: I then used ssh and copied all the content of the /etc/config directory i backed up previously in step 1 to /etc/config. Remember the mountpoint will propable be sda1 or sda2 this time. REMEMBER it will rewrite all the files/directories in the /etc/config directory without warning you. When this was done, i restarted the Turris Omnia and many configurations where back to the old. However certain software was still missing. Then again i SSh to the omnia and typed “updater.sh”. It then started to install all the missing software packages (for example LXC and many more).

Step 8: If you have done step 7, do again take a good look what you are missing in packages. In my case some packages were still missing so i had to re-install them manually. In myc case packages such as ath9k-firmware to get a USB-wifi-dongle working again and mwan3 package.

Step 9: If you were running a LXC like me, you have to change some settings now to make it work again. Go to /etc/lxc/lxc.config and change the directory to that of what you previously had on your internal SSD. If nothing is still detected give it a reboot and things should work.

1 Like

Oke, i fixed it. I decided to create another VM from scratch to see if the directory is right. That went well and the new VM had a “config” file, while my original VM did not have. I had made that VM back when i just got the Omnia.

So when seeing the new VM had a “config” file that contained information (network setting, name etc) of the VM, i copied that file to my original VM and changed the settings. Send a reboot command and everything worked fine.

Yes, as described in the manual: https://doc.turris.cz/doc/en/howto/omnia_factory_reset

Just backed up my config before and restored it afterwards. Just needed to start and enable mysqld to access nextcloud (installed to external HD) and set up DDNS and VPN again.

Why not use nextcloud in LXC instead?

Pros:

  • Latest version of PHP
  • Latest version of mysql ( i am using MariaDB)
  • Later version of webserver.

Or you REALLY have attached a EXTERNAL HD to it that is running on 7200 RPM or 5400 RPM which i guess is rather “slow” to have a LXC running.

Wrote it out of convenience, just to be clear it isn’t the flash storage. In fact it’s an SSD attached via the PCI/mSata slot.

OOh well, then having Nextcloud in LXC you will have more benefits as EVERYTHING is written and read from SSD leaving flash storage (eMMC) for the real basic things (routing, internet, VPN…etc).

I am running my Nextcloud in Ubuntu Xenial LXC using repositories to have many packages up to date.

1 Like