JFFS2 is a writable compressed filesystem with journaling and wear leveling using LZMA for the compression.
â/overlayâ is the writable part of the file system that gets merged with /rom to create a uniform /-tree. It contains anything that was written to the router after installation, e.g. changed configuration files, additional packages installed with OPKG, etc. It is formated with JFFS2.
Whenever the system is asked to look for an existing file in â/â, it first looks in â/overlayâ, and if not there, then in â/româ. In this way â/overlayâ overrides â/româ and creates the effect of a writable â/â while much of the content is safely and efficiently stored in the read-only â/româ.
itâs recommended to use first partition for the root overlay, second partition for swap
So knowing these things, i understood it like this.
Make 2 partitions. Partition 1 (sda1) will become â/overlayâ, because this is the part that is being used for packages and other writeable things. (see point 4). I have to format this partition as âJFFS2-filesystemâ (see point 2).
However when i did df -T
root@K-Router:~# df -T
Filesystem Type 1K-blocks Used Available Use% Mounted on
/dev/mmcblk0p1 btrfs 7633920 811048 6834140 11% /
tmpfs tmpfs 1035500 5616 1029884 1% /tmp
tmpfs tmpfs 512 4 508 1% /dev
That partition is not JFFS2, but btrfs.
My questions:
Do i need to also format partition 1(/overlay)(sda1) as btrfs?
Have i understood it correctly that sda1 will be used as /overlay?
Do i need to setup other partitions beside /overlay and swap?
Is 16 GB overkill for /overlay(extroot)?
Are the log files also stored in /overlay?
If not, where are they stored and can i also make a partition for these?
Have i forgotten something essential or not even essential?
The internal storage is something like a (Micro)SD card or SSD. It does internal wear levelling about the same a SSD does it. Yes, even some MicroSD support trim/discard.
An overlay wonât be useful as the snapshot feature of btrfs is used for recovery and factory reset. As the kernel resides on the filesystem an overlay may even break your boot.
8 GB are more than generous even with 10 snapshots.
You probably donât want swap on your router either. If you have some software that takes a lot of memory you donât want to have your system swap out your important things. Let the out-of-memory-killer kill it. It is generally faster and more stable this way.
I would format the SSD with ext4, f2fs or btrfs and mount it as /srv. This way your containers are on the SSD and wonât be back in time on recovery.
The log files are stored in memory. You may change this. Maybe /srv/log ?
SSD durability should be better with one half full partition than one full and one empty because there are less rewrites because you ran out of bigger free space ranges.
I am not really planning to make a lot of snapshots. I think it will be like MAX 5 or so. About SWAP, well you indeed have a good argument, howeverâŚis a SSD as a swap not even kind of fast? I mean i know RAM is WAY faster, but SSD isnât one of those old drives that are rather slow.
If i mount it as /srv , will packages that i have installed also be saved on that in the future? just like snapshots etc.?
I am indeed planning to at least have 1 LXC maybe two in the far future.
I am planning to use a very large portion of the ssd for LXC and torrents etc. Will that partition that i have used for LXC only be visible within LXC or will it be just a sort of mount from openWRT? I am curious about this, because of wanting to setup fstrim, so that i know where to setup within LXC or openWRT.
The snapshots are part of the original filesystem so they are on the internal storage. Even fast swap has a high latency.
Packages will still go to the internal storage. The LXC are in /srv/lxc and would be on the SSD. I would put the torrents on /srv/torrent or something like this.
LXC uses directories under /srv/lxc/CONTAINERNAME/root or so. As this directory gets the root of the container it will not have access to something not within this path.
Mounting the SSD to /srv would give you tons of space for LXC and storage but not for OpenWRT itself. I think OpenWRT wonât get this big. My normal desktop runs on 16GB including office and so on. Data and home directory are on a different storage. OpenWRT normally runs on 8MB of flash storage. This is compressed but uncompressed this wonât be more than about 50 MB. Turris OS is a giant installation.
8 GB for OpenWRT, just to be sure and on the safe side(this will be /overlay i guess right?). I am kind of afraid that if i do not use SWAP the memory will be full after some time. If you have more and more torrents (gonna use for the purpose of uploading (legal torrents off course), then this will use more and more RAM. So that is why i wanted to be on the safe side with 4 GB SWAP.
I read somewhere to leave 10-15 GB untouched as unallocated space, because of bad sectors if i could remember it correctlyâŚi could also be for something else, but by doing this you could extend the life-cycle of the SSD.
This will leave roughly 475 GB left. Letâs say i use 400 GB and leave the 75GB for something else in the future.
The 400GB just for /srv/. Right now the torrents that i am uploading are about 50 GB and off course gonna grow in the future.
Your SSD has 512GiB flash. That are 549755813888 Bytes. Available for usage are 500 000 000 000. So you already have 10% more flash than you use. If you create a 480GB ~ 450GiB partition then this will leave about 20GB free space. You may later use it as swap, extend the existing data partition or something else. Free space at the end is always better than in front and it wonât matter for swap or something because the speed is the same.
Yeah i guess youâre right. Just leave the SWAP for now out of the way and leave it as a possible way out if it would be necessary in the future. For now just one for /overlay to be on the safe side and the otherâŚthe big one /srv/
Thanks, that helped me a lot to sort things out and clarify them.
BTW: I am still planning to use /overlayâŚso also /overlay just format as ext4, f2fs or btrfs?
@Big_boss did you have a chance to get extroot working? Iâd love to use external HDD instead of internal flash too. I suppose instead of external overlay, external root method must be used but I didnât get this working - router becomes a brick after reboot
Well logically speaking, there is the possibility to do so i think as you managed it to get it to work, as the packages are installed at a certain directory and just have to change the file that points at the right directory right?
Maybe you could post your knowledge on the community wiki. Too much shared knowledge doesnât exist :P.
Iâm still not real clear on the difference and why youâd choose one over the other on LuCI.
You also mention the risk of soft-bricking which is not encouraging at all. To wit Iâd take a full backup before trying this, that could be restored, but it leaves the open question of what the best approach to that is (taking a snapshot of the whole Omnia file system for example and reloading it).
Iâd also love to see the config issue in your first footnote resolved, and here is one idea:
Fix fstab to permanently mount /rom in read/write mode
then soft-link /etc/config to /rom/etc/config
and voila the whole issue of config syncing is gone, and updates to config are confidently same on / and /rom
I would cherish it if we could update your excellent doc with more overall confidence, for which I think the input of Turris would be very useful. They built Turris OS and can answer some of the uncertainties there.
This has to be a generally desirable thing, to stick an external drive on, and overlay it as / for all the benefits you mention!
The difference between external root and external overlay is simple.
When you do pivot-overlay, then the external drive contains âthe differenceâ between the data in the internal memory and the current data i.e. if you have some file in the internal memory and you delete that file in extroot filesystem then instead of the actual deletion, a special entry would be created in the external filesystem. In other words, every time you access a file with pivot-overlay extroot set up, the kernel asks the file on the external drive first and if that file does not exist - tries to access the internal memory.
I didnât find any profit of that complication, but the drawback is the complication itself. It much more easier to understand how pivot-root works - during the boot at some moment, root in internal memory got unmounted and root on external drive mounts which means complete replacement. You donât need to worry about the internal memory anymore, however it becomes mounted again into /rom so you can grab some files from it still.
Regarding stability - it works, but a couple of times after an automatic reboot I was losing any access to the device. The solution for me appeared to be the following: I powered the device off, unplugged hard drive, powered on, waited for boot, powered off again, plugged hard drive back and powered on - pretty simple. Kernel receives updates so Iâd say itâs working. Not sure why did I lost an access - it might be related to the wi-fi bug in some of the updates in July or June I guess.
P.S. Regarding linking /etc/config - I donât like the idea. While the internal filesystem remains completely separate from the external one, you can always remove hard drive and boot. Which means that you have some kind of stable backup for urgent issues. E.g. If I mess up with the configuration of the router at home from work and I cannot reach the device via ssh anymore, I can ask my wife to reboot it with hard drive unplugged.
I am pretty slowpoke but I just realized that kernel was not upgrading all this time. Other packages seems to be upgrading but the kernel is not. I stuck with 4.4.79 which was released Aug 2 for Omnia right before I configured extroot.
Does anyone use extroot except me? If so - did you solve that problem with kernel updates?
I use direct external boot from SSD with MMC as second boot option. It is a bit like extroot but without the big hassle with updates. Updates work like reboot to MMC and update there. Side effect of my solution: the led counts for recovery etc are off by one (1 LED boots to SSD, 2 LED boot to MMC (normal), 3 LED rollback snapshot on MMC, 4 LED factory reset, âŚ). This will be changed back to original if i find a way to detect if reset was pressed but only for the first LED or not pressed at all.
This may break a bit when you disable extroot but flashing with the medkit (recovery) will work. Another option could be chroot to /rom (mounted r/w) and callling pkgupdate but there may be really nasty side-effects.
Oh, thatâs cool! Iâd prefer direct boot over extroot if I knew how to achieve that. Did you modify the bootloader? Can you share your knowledge via a wiki article? That would be helpful.
No, i didnât change the bootloader itself but the u-boot environment variables.
u-boot runs the variable bootcmd on boot. It reads the number of LEDs lid while in reset to the variable rescue and if it is zero it runs $mmcboot else $rescueboot.
bootcmd=i2c dev 1; i2c read 0x2a 0x9 1 0x00FFFFF0; setexpr.b rescue *0x00FFFFF0; if test $rescue -ge 1; then echo BOOT RESCUE; run rescueboot; else echo BOOT eMMC FS; run mmcboot; fi
!WARNING!
Do not do this without alternate means to get online and a mean to use the serial console to change things in the bootloader the hard way! I give no warranty as this is not exactly the environment i use.
I modified bootcmd and added ssdboot
bootcmd=i2c dev 1; i2c read 0x2a 0x9 1 0x00FFFFF0; setexpr.b rescue *0x00FFFFF0 ; if test $rescue -ge 1 ; then setexpr rescue $rescue - 1 ; if test $rescue -ge 1 ; then echo BOOT RESCUE ; run rescueboot ; else echo BOOT eMMC FS ; run mmcboot ; fi ; else echo BOOT SSD FS ; run ssdboot ; fi
ssdboot=scsi scan ; setenv bootargs earlyprintk console=ttyS0,115200 rootfstype=btrfs rootdelay=2 root=/dev/sda rootflags=subvol=@,commit=5 rw ; btrload scsi 0 0x01000000 boot/zImage @ ; btrload scsi 0 0x02000000 boot/dtb @ ; bootz 0x01000000 - 0x02000000
the setexpr rescue $rescue - 1 steals 1 LED count from the recovery mechanism so you have to light one more LED than normal.
In the uBoot command youâre using /dev/sda as root. Do you reckon there would be a way to use UUID instead?
I have one mSATA drive and two drives on a SATA controller which results in my desired root to be /dev/sdc. This is all fine except if I ever change the drive configuration, this may change to some other /dev/sd?.
Also, could you describe how did you made /dev/sda to be bootable?
/dev/sda contains a BTRFS. There is a subvol named @ in the sample. This is just like the original one on eMMC. scsi scan will detect the available controllers. This works with the SoC-internal one for mSATA. I am not sure if any other ahci controller is detected in u-boot but the config says it will at most have 2 disks.
btrload will the load zImage and dtb from subvol @ of device scsi 0 without any partition. If there were partitions it would probably be btrload scsi 0:0 or 0:1 or something like this. bootz will run it.
As you see there is nothing special. I only use a filesystem understood by u-boot and load 2 files from it.
I did not use any UUID on purpose.
If i replace the SSD i can still boot as long as it is formatted with BTRFS.
As you can see i use root=/dev/sda which means there is no partition. This is a design decision by me as i do not plan to put anything else than a single BTRFS on this device.