User Tools

Site Tools


ExtRoot: How it works

Most routers do not have hard drives. They use flash memory for similar purposes: storing programs and data, even when the system is off (non-volatile memory).

In most systems, flash memory does not appear like RAM and so data and instructions must be copied to RAM to be used. So, for example, the bootloader copies the kernel from flash to RAM and then starts that copy running.

There are two basic types of flash memory: NOR flash and NAND flash.
If the flash chip (no matter what type) is connected directly with the SoC and has to be addressed directly by Linux, we call it "raw flash". If there is an additional controller chip between flash chip and the SoC, we call it "FTL (Flash Translation Layer) flash". Embedded systems almost exclusively use "raw flash", while SSDs and also USB memory sticks, almost exclusively use FTL flash!

Older routers generally have raw NOR flash but many newer routers have raw NAND flash.

Raw NOR flash in typical routers is generally small (4 MiB - 16 MiB) and error-free. This is currently the normal OpenWRT setup. Because it is error-free, the file systems are a little simpler. But they still need to do wear-levelling. Conserving space is important.

Raw NAND flash in typical routers is generally larger (32 MiB - 256 MiB) and may contain bad blocks. OpenWRT isn't configured to handle this yet. One promising approach would be to use UBIFS layered on top of UBI.

→

There are two possible ways to increase Read/Write storage in such an embedded system:

  1. pivot the /overlay to the USB disk (call this external overlay or pivot-overlay)
  2. or pivot the entire / (read: root filesystem) to the USB disk (call this external root or pivot-root)

Note: extroot started as the external overlay thus the external root was called pivot-root only in order to distinguish between the two implementations. To avoid misunderstandings you should always use pivot-overlay and pivot-root respectively. extroot comprises both methods.

Note: The shitty bootloaders the OEMs delight us with, are the only reason why we cannot put everything (including the Kernel) on the USB hard drive! Especially uboot is very well capable of handling diverse file systems, but uboot most often lacks the necessary hardware drivers. While uboot is under GPLv2 and the Linux Kernel is also GPLv2 it is possible to port code, so implementing such functionality is not that big of a deal, but the OEMs are obviously not inclined to give us such luxury and who is going to do that for free? The OEM bootloader of the dockstar is actually NOT capable of booting from USB, only the version provided by Jeff Doozan is! Hello, FOSS ;-)


In order to use any filesystem located on a USB device, the Linux kernel needs driver modules for the USB-circuitry (kmod-usb-core and kmod-usb2), modules for storage devices connected over USB (kmod-usb-storage) and modules for the file system (kmod-fs-ext4). And once you have a kernel with all that functionality built-in, you still need to tell init to mount that particular filesystem on the USB disk as /. Tricky without having a filesystem at all.

  • Older versions of ExtRoot required to include the kernel modules in the SquashFS (or include them in the Kernel) and also a configuration. Then you would change the /etc/config/fstab, which was located on the JFFS2 and this would make ExtRoot work.
  • Newer versions allow you to install everything on the JFFS2-partition, an OpenWrt firmware image file doesn't have to be built specifically with extroot in mind any longer.
    • FIXME "allow you to install everything on the JFFS2-partition" → What do you mean by "everything"? Do you mean all needed kernel modules and configs are now on JFFS2 or Everything is now on a all-JFFS2 partition without any SquashFS partition?

First using SquashFS and then chroot, would make init not PID 1, which is bad.


The external rootfs is done as an overlay of the boot root filesystem. The boot root filesystem will be mounted read-only on /rom, and the external rootfs device will be mounted read-write on /overlay. The root (/) filesystem will be an overlayfs overlay of /overlay over /rom, and mounted on root (/) (obviously).

During the boot process, the configuration used to mount the external rootfs is done as follows: → process.boot

For all block-extroot revisions, determines whether this filesystem is a rootfs for use with block-extroot. If it is a rootfs and block-extroot is installed, then during preinit, this filesystem will be mounted on /overlay and used as the root overlay (like jffs2 on a normal squashfs boot, only with this filesystem). target is ignored for the purposes of a rootfs mount, however when doing the squashfs mount (e.g. a fallback, or on firstboot), it will be used as usual (see extroot).
overlay filesystem

Sequence pivot overlay

  1. IF NOT SquashFS, root is remounted rw on /
  2. IF SquashFS, JFFS2 is mounted rw on /tmp/overlay
  3. IF /tmp/overlay/etc/config/fstab exists we use it as the configuration file, otherwise we use /etc/config/fstab (if it exists)
  4. Fork
    • Backfire: we search for a mount section in the fstab-file with the option is_rootfs set to true, and mount it on /overlay, if we find it
    • Attitude Adjustment: in trunk we prefer to look for a mount section in the fstab-file with the target /overlay
  5. we unmount /tmp/overlay again and/or remount root (/) read only
  6. we pivot the root so that that /overlay on top of /rom is now root, and old root is on /rom

Sequence pivot root

  1. IF NOT SquashFS, root is remounted rw on /
  2. IF SquashFS, JFFS2 is mounted rw on /tmp/overlay
  3. IF /tmp/overlay/etc/config/fstab exists we use it as the configuration file, otherwise we use /etc/config/fstab (if it exists)
  4. we search for a mount section in the fstab-file with target set to / and mount it on /rom
  5. we pivot the root to that what was rom is now root and old root is on /rom
doc/howto/extroot/extroot.theory.txt · Last modified: 2016/04/25 06:29 by RAThomas