[SOLVED] - Extroot - (expanding Turris package space)

Well logically speaking, there is the possibility to do so i think as you managed it to get it to work, as the packages are installed at a certain directory and just have to change the file that points at the right directory right?

Maybe you could post your knowledge on the community wiki. Too much shared knowledge doesn’t exist :P.

The process appeared to be pretty straightforward. But still, I’ve written an article on wiki.

P.S. If you’d find the difference between the instruction I’ve written and the actions you did when failed, it would be pretty useful.

5 Likes

Good article. But … (always a but),

You’ve mounted your external drive as the root filesystem, i.e not as an external overlay. I admit reading this:

https://wiki.openwrt.org/doc/howto/extroot/extroot.theory

I’m still not real clear on the difference and why you’d choose one over the other on LuCI.

You also mention the risk of soft-bricking which is not encouraging at all. To wit I’d take a full backup before trying this, that could be restored, but it leaves the open question of what the best approach to that is (taking a snapshot of the whole Omnia file system for example and reloading it).

I’d also love to see the config issue in your first footnote resolved, and here is one idea:

  1. Fix fstab to permanently mount /rom in read/write mode
  2. then soft-link /etc/config to /rom/etc/config

and voila the whole issue of config syncing is gone, and updates to config are confidently same on / and /rom

I would cherish it if we could update your excellent doc with more overall confidence, for which I think the input of Turris would be very useful. They built Turris OS and can answer some of the uncertainties there.

This has to be a generally desirable thing, to stick an external drive on, and overlay it as / for all the benefits you mention!

The difference between external root and external overlay is simple.

When you do pivot-overlay, then the external drive contains ‘the difference’ between the data in the internal memory and the current data i.e. if you have some file in the internal memory and you delete that file in extroot filesystem then instead of the actual deletion, a special entry would be created in the external filesystem. In other words, every time you access a file with pivot-overlay extroot set up, the kernel asks the file on the external drive first and if that file does not exist - tries to access the internal memory.

I didn’t find any profit of that complication, but the drawback is the complication itself. It much more easier to understand how pivot-root works - during the boot at some moment, root in internal memory got unmounted and root on external drive mounts which means complete replacement. You don’t need to worry about the internal memory anymore, however it becomes mounted again into /rom so you can grab some files from it still.

Regarding stability - it works, but a couple of times after an automatic reboot I was losing any access to the device. The solution for me appeared to be the following: I powered the device off, unplugged hard drive, powered on, waited for boot, powered off again, plugged hard drive back and powered on - pretty simple. Kernel receives updates so I’d say it’s working. Not sure why did I lost an access - it might be related to the wi-fi bug in some of the updates in July or June I guess.

P.S. Regarding linking /etc/config - I don’t like the idea. While the internal filesystem remains completely separate from the external one, you can always remove hard drive and boot. Which means that you have some kind of stable backup for urgent issues. E.g. If I mess up with the configuration of the router at home from work and I cannot reach the device via ssh anymore, I can ask my wife to reboot it with hard drive unplugged.

I am pretty slowpoke but I just realized that kernel was not upgrading all this time. Other packages seems to be upgrading but the kernel is not. I stuck with 4.4.79 which was released Aug 2 for Omnia right before I configured extroot.

Does anyone use extroot except me? If so - did you solve that problem with kernel updates?

P.S. As a quick fix I tried to run:

pkgupdate
mount -o remount,rw /rom
cp -r /boot /rom/boot
cp -r /lib/modules /rom/lib/
reboot

It helped - the kernel upgraded and device booted without errors. I’ll investigate how reliable is that method.

I use direct external boot from SSD with MMC as second boot option. It is a bit like extroot but without the big hassle with updates. Updates work like reboot to MMC and update there. Side effect of my solution: the led counts for recovery etc are off by one (1 LED boots to SSD, 2 LED boot to MMC (normal), 3 LED rollback snapshot on MMC, 4 LED factory reset, …). This will be changed back to original if i find a way to detect if reset was pressed but only for the first LED or not pressed at all.

This may break a bit when you disable extroot but flashing with the medkit (recovery) will work. Another option could be chroot to /rom (mounted r/w) and callling pkgupdate but there may be really nasty side-effects.

Oh, that’s cool! I’d prefer direct boot over extroot if I knew how to achieve that. Did you modify the bootloader? Can you share your knowledge via a wiki article? That would be helpful.

No, i didn’t change the bootloader itself but the u-boot environment variables.

u-boot runs the variable bootcmd on boot. It reads the number of LEDs lid while in reset to the variable rescue and if it is zero it runs $mmcboot else $rescueboot.

bootcmd=i2c dev 1; i2c read 0x2a 0x9 1 0x00FFFFF0; setexpr.b rescue *0x00FFFFF0; if test $rescue -ge 1; then echo BOOT RESCUE; run rescueboot; else echo BOOT eMMC FS; run mmcboot; fi

!WARNING!

Do not do this without alternate means to get online and a mean to use the serial console to change things in the bootloader the hard way! I give no warranty as this is not exactly the environment i use.

I modified bootcmd and added ssdboot

bootcmd=i2c dev 1; i2c read 0x2a 0x9 1 0x00FFFFF0; setexpr.b rescue *0x00FFFFF0 ; if test $rescue -ge 1 ; then setexpr rescue $rescue - 1 ; if test $rescue -ge 1 ; then echo BOOT RESCUE ; run rescueboot ; else echo BOOT eMMC FS ; run mmcboot ; fi ; else echo BOOT SSD FS ; run ssdboot ; fi
ssdboot=scsi scan ; setenv bootargs earlyprintk console=ttyS0,115200 rootfstype=btrfs rootdelay=2 root=/dev/sda rootflags=subvol=@,commit=5 rw ; btrload scsi 0 0x01000000 boot/zImage @ ; btrload scsi 0 0x02000000 boot/dtb @ ; bootz 0x01000000 - 0x02000000

the setexpr rescue $rescue - 1 steals 1 LED count from the recovery mechanism so you have to light one more LED than normal.

2 Likes

In the uBoot command you’re using /dev/sda as root. Do you reckon there would be a way to use UUID instead?

I have one mSATA drive and two drives on a SATA controller which results in my desired root to be /dev/sdc. This is all fine except if I ever change the drive configuration, this may change to some other /dev/sd?.

Also, could you describe how did you made /dev/sda to be bootable?

reformatted ssdboot for reference:

scsi scan
setenv bootargs earlyprintk console=ttyS0,115200 rootfstype=btrfs rootdelay=2 root=/dev/sda rootflags=subvol=@,commit=5 rw
btrload scsi 0 0x01000000 boot/zImage @
btrload scsi 0 0x02000000 boot/dtb @
bootz 0x01000000 - 0x02000000

/dev/sda contains a BTRFS. There is a subvol named @ in the sample. This is just like the original one on eMMC.
scsi scan will detect the available controllers. This works with the SoC-internal one for mSATA. I am not sure if any other ahci controller is detected in u-boot but the config says it will at most have 2 disks.

btrload will the load zImage and dtb from subvol @ of device scsi 0 without any partition. If there were partitions it would probably be btrload scsi 0:0 or 0:1 or something like this.
bootz will run it.

As you see there is nothing special. I only use a filesystem understood by u-boot and load 2 files from it.

I did not use any UUID on purpose.
If i replace the SSD i can still boot as long as it is formatted with BTRFS.
As you can see i use root=/dev/sda which means there is no partition. This is a design decision by me as i do not plan to put anything else than a single BTRFS on this device.

The other way is to configure extroot as you would normally do, but add this start script to the extroot system:

cat <<EOF > /etc/rc.d/S01mount-rom
/usr/bin/mount -o remount,rw /rom
/usr/bin/mount --bind /rom/boot /boot
/usr/bin/mount --bind /rom/lib/modules /lib/modules
/usr/bin/mount --bind /rom/etc/modules.d /etc/modules.d
/usr/bin/mount --bind /rom/etc/modules-boot.d /etc/modules-boot.d
EOF

chmod +x /etc/rc.d/S01mount-rom

This will make to mount bind all kernel related folders very early, so extroot system will always update kernel and modules on mmc.

I have a few questions.
Because I have no guarantee that some of software running on omnia writes to mmc (lxc containers are on mSATA, but there is more software installed, some of them may not use only /tmp and /var), I consider moving entirely to mSATA in your way.

So:

  1. how to change this u-boot variable?
  2. can it be /dev/sda1?
  3. I have no subvolumes, I assume it is necessery to create one.
  4. how to copy entire filesystem to mSATA? rsync?

Thanks in advance.

fw_printenv shows it and fw_saveenv allows changes. Be sure to have a backup of /dev/mtd0. Alternative approach: clearenv to remove automatically generated variables, loadenv to get the ones from flash, setenv to change them and saveenv to save them back and reboot to try it in u-boot will do the same thing.

  1. This should work but with will require some changes to the variables.

  2. There is always a toplevel subvolume with id 5. If you don’t give one then it should work. On the other hand creating a subvolume is simple and gives the option to switch the used subvolume with the change of one variable, holding the reset key or even using a switch on the extension header (to be implemented later). I saved the subvolume name to one variable in u-boot and only have to change it in the boot prompt to switch to a backup snapshot.

  3. btrfs-send ... | btrfs-receive ... may be a better option. This does not only copy data and permissions but also internal attributes like no-cow and compressed and part of its metadata like data-dedup.

2 Likes

Hi,

very inspiring thread. Could you be more specific for us - n00bish - users please? :slight_smile: or you would not recommend it at all ?

For example I have trouble with very first command - fw_printenv - I got response:

Cannot parse config file: No such file or directory

It is specially a bit cryptic to not have some n00bish user do it without thinking about the consequences and risks.

Some for reference:

  • wrong or missing bootloader settings (fixable with the internal serial port)
  • killed bootloader (may require JTAG)
  • even worse things…

It seems /etc/fw_env.config is missing.

WIthout any warranty: Content of /etc/fw_enc.config

I don’t know if the mtd is correct for TurrisOS. The flash sector size may also be different. Be sure to have backups of the mtd device and the environment after reading works. You will need them on another device if something goes wrong.

It should probably have this content:

/dev/mtd0 0xC0000 0x10000 0x10000

Thanks, I’ll probably let it go for now (I wanted to use sata ssd instead of flash safest possible way - with working updates) but I’m not experienced enough for it.

I thought fw_* was “official” or working by default but it doesn’t seem so.

when creating /etc/fw_env.config with values you provided I just got:

Warning: Bad CRC, using default environment
bootcmd=bootp; setenv bootargs root=/dev/nfs nfsroot=${serverip}:${rootpath} ip=${ipaddr}:${serverip}:${gatewayip}:${netmask}:${hostname}::off; bootm
bootdelay=5
baudrate=115200

so I don’t think it would be wise to proceed any further - I would be just “spamming” you or someone else after each step

Most of things i do are far of from “official” or even simple. I try to keep things safe most of the time but bad things happen quite often.

thats ok when you know what to do :slight_smile: but why there’s uboot-envutils package installed but no /etc/fw_env.config - it’s bug or “feature”?

A month back i bought a Khadas VIM2 MAX as i was looking for a SoC that could decode x265 for KODI. They have a AWESOME feature.

As it has the ability to run dual-OS from eMMC, it also gives the possibility to run even a triple OS using a SDCard. Here they have a feature that if a Micro-sdcard is plugged in that contains a OS, it will automatically boot from that micro-sdcard. So why is this a awesome feature? Well the wear leveling can sooner or later be a fact on the eMMC. It could be 10 years…20 years…i do not know. However many of us haven’t bought the Turris Omnia for 2-3 years, rather a investment for the future (replacing wifi cards, Antenna’s, SSD-drive, USB dongles/drives, etc…). When the eMMC of the Omnia has left us, logically thinking it would mean the Omnia is ready for the junkyard right? Or used as a simple hub as the switch functionality wouldn’t even work as there is no software controlling it. So having the ability to put the WHOLE Turris OS on a SSD-drive would be a good alternative right?

Do correct me if i have said something incorrectly or give your additions to what i have come up with.

http://docs.khadas.com/bootcamp/BootingCardVsBurningCard/

2 Likes

So I come back to playing with Omnia and what do I see? This is now officially supported! \o/

If you’ve ended up here because this is top search result when looking how to set up the router from external drive, you need to fetch the medkit and follow the official documentation. Still, there are two things I’d like to mention.

UUID

First of, it is possible to use UUIDs instead of device names (which are not stable and change based on what drives are attached to the router). To do this, run fdisk -l /dev/sda and take note of the ‘Disk identifier’.

# fdisk -l /dev/sda
Disk /dev/sda: 55.9 GiB, 60022480896 bytes, 117231408 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xe17eb372

Device     Boot Start       End   Sectors  Size Id Type
/dev/sda1        2048 117231407 117229360 55.9G 83 Linux

In the listing above, the id we’re after is e17eb372 (note 0x prefix is stripped). Now, when changing bootargs (as documented), use root=PARTUUID=e17eb372-01 instead of root=/dev/sda1 (where e17eb372 is the identifier of your disk).

/var

Second of, one of the reasons I wanted to use external disk is so that I can persist all the data. For example all the log files saved to /var/log. To achieve that, I’ve figured I’ll modify the root file system by dropping the /var symbolic link (which points to /tmp) and make it into a directory (i.e. rm /mnt/ssd/@/var; mkdir /mnt/ssd/@/var). Unfortunately, after that change, the router no longer boots and hands after printing procd: - ubus -.

Anyone has any ideas how this can be addressed? Or do I need to manually configure everything I want to write data someplace outside of /var?

1 Like