[SOLVED] - Extroot - (expanding Turris package space)

Turris Omnia
#1

Guys, could you help me out with some of the things.

I have put a mSATA 500GB in my Omnia (Samsung 850 evo 500GB)

I finally made the decision to give 16 GB as extroot(sda1) and 4 gb as swap(sda2). Now i am kind of confused what filesystem that partition has to be.

https://wiki.openwrt.org/doc/howto/extroot
https://wiki.openwrt.org/doc/techref/filesystems
https://wiki.openwrt.org/doc/techref/flash.layout

  1. SquashFS is a read only compressed filesystem

  2. JFFS2 is a writable compressed filesystem with journaling and wear leveling using LZMA for the compression.

  3. “/overlay” is the writable part of the file system that gets merged with /rom to create a uniform /-tree. It contains anything that was written to the router after installation, e.g. changed configuration files, additional packages installed with OPKG, etc. It is formated with JFFS2.

  4. Whenever the system is asked to look for an existing file in “/”, it first looks in “/overlay”, and if not there, then in “/rom”. In this way “/overlay” overrides “/rom” and creates the effect of a writable “/” while much of the content is safely and efficiently stored in the read-only “/rom”.

  5. it’s recommended to use first partition for the root overlay, second partition for swap

So knowing these things, i understood it like this.

Make 2 partitions. Partition 1 (sda1) will become “/overlay”, because this is the part that is being used for packages and other writeable things. (see point 4). I have to format this partition as “JFFS2-filesystem” (see point 2).

However when i did df -T

root@K-Router:~# df -T
Filesystem           Type       1K-blocks      Used Available Use% Mounted on
/dev/mmcblk0p1       btrfs        7633920    811048   6834140  11% /
tmpfs                tmpfs        1035500      5616   1029884   1% /tmp
tmpfs                tmpfs            512         4       508   1% /dev

That partition is not JFFS2, but btrfs.

My questions:

  1. Do i need to also format partition 1(/overlay)(sda1) as btrfs?
  2. Have i understood it correctly that sda1 will be used as /overlay?
  3. Do i need to setup other partitions beside /overlay and swap?
  4. Is 16 GB overkill for /overlay(extroot)?
  5. Are the log files also stored in /overlay?
  6. If not, where are they stored and can i also make a partition for these?
  7. Have i forgotten something essential or not even essential?
#2

The internal storage is something like a (Micro)SD card or SSD. It does internal wear levelling about the same a SSD does it. Yes, even some MicroSD support trim/discard.
An overlay won’t be useful as the snapshot feature of btrfs is used for recovery and factory reset. As the kernel resides on the filesystem an overlay may even break your boot.

8 GB are more than generous even with 10 snapshots.

You probably don’t want swap on your router either. If you have some software that takes a lot of memory you don’t want to have your system swap out your important things. Let the out-of-memory-killer kill it. It is generally faster and more stable this way.

I would format the SSD with ext4, f2fs or btrfs and mount it as /srv. This way your containers are on the SSD and won’t be back in time on recovery.

The log files are stored in memory. You may change this. Maybe /srv/log ?

SSD durability should be better with one half full partition than one full and one empty because there are less rewrites because you ran out of bigger free space ranges.

#3

So it will not be /overlay?

I am not really planning to make a lot of snapshots. I think it will be like MAX 5 or so. About SWAP, well you indeed have a good argument, however…is a SSD as a swap not even kind of fast? I mean i know RAM is WAY faster, but SSD isn’t one of those old drives that are rather slow.

If i mount it as /srv , will packages that i have installed also be saved on that in the future? just like snapshots etc.?
I am indeed planning to at least have 1 LXC maybe two in the far future.

I am planning to use a very large portion of the ssd for LXC and torrents etc. Will that partition that i have used for LXC only be visible within LXC or will it be just a sort of mount from openWRT? I am curious about this, because of wanting to setup fstrim, so that i know where to setup within LXC or openWRT.

#4

The snapshots are part of the original filesystem so they are on the internal storage. Even fast swap has a high latency.

Packages will still go to the internal storage. The LXC are in /srv/lxc and would be on the SSD. I would put the torrents on /srv/torrent or something like this.

LXC uses directories under /srv/lxc/CONTAINERNAME/root or so. As this directory gets the root of the container it will not have access to something not within this path.

Mounting the SSD to /srv would give you tons of space for LXC and storage but not for OpenWRT itself. I think OpenWRT won’t get this big. My normal desktop runs on 16GB including office and so on. Data and home directory are on a different storage. OpenWRT normally runs on 8MB of flash storage. This is compressed but uncompressed this won’t be more than about 50 MB. Turris OS is a giant installation.

#5

How about this then.

8 GB for OpenWRT, just to be sure and on the safe side(this will be /overlay i guess right?). I am kind of afraid that if i do not use SWAP the memory will be full after some time. If you have more and more torrents (gonna use for the purpose of uploading (legal torrents off course), then this will use more and more RAM. So that is why i wanted to be on the safe side with 4 GB SWAP.

I read somewhere to leave 10-15 GB untouched as unallocated space, because of bad sectors if i could remember it correctly…i could also be for something else, but by doing this you could extend the life-cycle of the SSD.

This will leave roughly 475 GB left. Let’s say i use 400 GB and leave the 75GB for something else in the future.

The 400GB just for /srv/. Right now the torrents that i am uploading are about 50 GB and off course gonna grow in the future.

What do you think?

#6

Your SSD has 512GiB flash. That are 549755813888 Bytes. Available for usage are 500 000 000 000. So you already have 10% more flash than you use. If you create a 480GB ~ 450GiB partition then this will leave about 20GB free space. You may later use it as swap, extend the existing data partition or something else. Free space at the end is always better than in front and it won’t matter for swap or something because the speed is the same.

mSata use recommendation for LXC (and more?)
#7

Yeah i guess you’re right. Just leave the SWAP for now out of the way and leave it as a possible way out if it would be necessary in the future. For now just one for /overlay to be on the safe side and the other…the big one /srv/

Thanks, that helped me a lot to sort things out and clarify them.

BTW: I am still planning to use /overlay…so also /overlay just format as ext4, f2fs or btrfs?

#8

@Big_boss did you have a chance to get extroot working? I’d love to use external HDD instead of internal flash too. I suppose instead of external overlay, external root method must be used but I didn’t get this working - router becomes a brick after reboot

#9

@Grief, no. There is no extroot on Turris OS. It is just a large btrfs filesystem where everything is running on.

What i have done is just make a separate partition and put the LXC on that. So the whole LXC-(ubuntu 16.04) partition is running on my SSD.

#10

@Big_boss sorry, I didn’t make a note here, but I’ve successfully made extroot working in a day after my previous post.

1 Like
#11

Well logically speaking, there is the possibility to do so i think as you managed it to get it to work, as the packages are installed at a certain directory and just have to change the file that points at the right directory right?

Maybe you could post your knowledge on the community wiki. Too much shared knowledge doesn’t exist :P.

#12

The process appeared to be pretty straightforward. But still, I’ve written an article on wiki.

P.S. If you’d find the difference between the instruction I’ve written and the actions you did when failed, it would be pretty useful.

5 Likes
#13

Good article. But … (always a but),

You’ve mounted your external drive as the root filesystem, i.e not as an external overlay. I admit reading this:

https://wiki.openwrt.org/doc/howto/extroot/extroot.theory

I’m still not real clear on the difference and why you’d choose one over the other on LuCI.

You also mention the risk of soft-bricking which is not encouraging at all. To wit I’d take a full backup before trying this, that could be restored, but it leaves the open question of what the best approach to that is (taking a snapshot of the whole Omnia file system for example and reloading it).

I’d also love to see the config issue in your first footnote resolved, and here is one idea:

  1. Fix fstab to permanently mount /rom in read/write mode
  2. then soft-link /etc/config to /rom/etc/config

and voila the whole issue of config syncing is gone, and updates to config are confidently same on / and /rom

I would cherish it if we could update your excellent doc with more overall confidence, for which I think the input of Turris would be very useful. They built Turris OS and can answer some of the uncertainties there.

This has to be a generally desirable thing, to stick an external drive on, and overlay it as / for all the benefits you mention!

#14

The difference between external root and external overlay is simple.

When you do pivot-overlay, then the external drive contains ‘the difference’ between the data in the internal memory and the current data i.e. if you have some file in the internal memory and you delete that file in extroot filesystem then instead of the actual deletion, a special entry would be created in the external filesystem. In other words, every time you access a file with pivot-overlay extroot set up, the kernel asks the file on the external drive first and if that file does not exist - tries to access the internal memory.

I didn’t find any profit of that complication, but the drawback is the complication itself. It much more easier to understand how pivot-root works - during the boot at some moment, root in internal memory got unmounted and root on external drive mounts which means complete replacement. You don’t need to worry about the internal memory anymore, however it becomes mounted again into /rom so you can grab some files from it still.

Regarding stability - it works, but a couple of times after an automatic reboot I was losing any access to the device. The solution for me appeared to be the following: I powered the device off, unplugged hard drive, powered on, waited for boot, powered off again, plugged hard drive back and powered on - pretty simple. Kernel receives updates so I’d say it’s working. Not sure why did I lost an access - it might be related to the wi-fi bug in some of the updates in July or June I guess.

P.S. Regarding linking /etc/config - I don’t like the idea. While the internal filesystem remains completely separate from the external one, you can always remove hard drive and boot. Which means that you have some kind of stable backup for urgent issues. E.g. If I mess up with the configuration of the router at home from work and I cannot reach the device via ssh anymore, I can ask my wife to reboot it with hard drive unplugged.

#15

I am pretty slowpoke but I just realized that kernel was not upgrading all this time. Other packages seems to be upgrading but the kernel is not. I stuck with 4.4.79 which was released Aug 2 for Omnia right before I configured extroot.

Does anyone use extroot except me? If so - did you solve that problem with kernel updates?

P.S. As a quick fix I tried to run:

pkgupdate
mount -o remount,rw /rom
cp -r /boot /rom/boot
cp -r /lib/modules /rom/lib/
reboot

It helped - the kernel upgraded and device booted without errors. I’ll investigate how reliable is that method.

#16

I use direct external boot from SSD with MMC as second boot option. It is a bit like extroot but without the big hassle with updates. Updates work like reboot to MMC and update there. Side effect of my solution: the led counts for recovery etc are off by one (1 LED boots to SSD, 2 LED boot to MMC (normal), 3 LED rollback snapshot on MMC, 4 LED factory reset, …). This will be changed back to original if i find a way to detect if reset was pressed but only for the first LED or not pressed at all.

This may break a bit when you disable extroot but flashing with the medkit (recovery) will work. Another option could be chroot to /rom (mounted r/w) and callling pkgupdate but there may be really nasty side-effects.

Root fs on mSATA drive?
Boot from SSD - Outdated description?
#17

Oh, that’s cool! I’d prefer direct boot over extroot if I knew how to achieve that. Did you modify the bootloader? Can you share your knowledge via a wiki article? That would be helpful.

#18

No, i didn’t change the bootloader itself but the u-boot environment variables.

u-boot runs the variable bootcmd on boot. It reads the number of LEDs lid while in reset to the variable rescue and if it is zero it runs $mmcboot else $rescueboot.

bootcmd=i2c dev 1; i2c read 0x2a 0x9 1 0x00FFFFF0; setexpr.b rescue *0x00FFFFF0; if test $rescue -ge 1; then echo BOOT RESCUE; run rescueboot; else echo BOOT eMMC FS; run mmcboot; fi

!WARNING!

Do not do this without alternate means to get online and a mean to use the serial console to change things in the bootloader the hard way! I give no warranty as this is not exactly the environment i use.

I modified bootcmd and added ssdboot

bootcmd=i2c dev 1; i2c read 0x2a 0x9 1 0x00FFFFF0; setexpr.b rescue *0x00FFFFF0 ; if test $rescue -ge 1 ; then setexpr rescue $rescue - 1 ; if test $rescue -ge 1 ; then echo BOOT RESCUE ; run rescueboot ; else echo BOOT eMMC FS ; run mmcboot ; fi ; else echo BOOT SSD FS ; run ssdboot ; fi
ssdboot=scsi scan ; setenv bootargs earlyprintk console=ttyS0,115200 rootfstype=btrfs rootdelay=2 root=/dev/sda rootflags=subvol=@,commit=5 rw ; btrload scsi 0 0x01000000 boot/zImage @ ; btrload scsi 0 0x02000000 boot/dtb @ ; bootz 0x01000000 - 0x02000000

the setexpr rescue $rescue - 1 steals 1 LED count from the recovery mechanism so you have to light one more LED than normal.

2 Likes
Boot from SSD - Outdated description?
#19

In the uBoot command you’re using /dev/sda as root. Do you reckon there would be a way to use UUID instead?

I have one mSATA drive and two drives on a SATA controller which results in my desired root to be /dev/sdc. This is all fine except if I ever change the drive configuration, this may change to some other /dev/sd?.

Also, could you describe how did you made /dev/sda to be bootable?

#20

reformatted ssdboot for reference:

scsi scan
setenv bootargs earlyprintk console=ttyS0,115200 rootfstype=btrfs rootdelay=2 root=/dev/sda rootflags=subvol=@,commit=5 rw
btrload scsi 0 0x01000000 boot/zImage @
btrload scsi 0 0x02000000 boot/dtb @
bootz 0x01000000 - 0x02000000

/dev/sda contains a BTRFS. There is a subvol named @ in the sample. This is just like the original one on eMMC.
scsi scan will detect the available controllers. This works with the SoC-internal one for mSATA. I am not sure if any other ahci controller is detected in u-boot but the config says it will at most have 2 disks.

btrload will the load zImage and dtb from subvol @ of device scsi 0 without any partition. If there were partitions it would probably be btrload scsi 0:0 or 0:1 or something like this.
bootz will run it.

As you see there is nothing special. I only use a filesystem understood by u-boot and load 2 files from it.

I did not use any UUID on purpose.
If i replace the SSD i can still boot as long as it is formatted with BTRFS.
As you can see i use root=/dev/sda which means there is no partition. This is a design decision by me as i do not plan to put anything else than a single BTRFS on this device.