To understand the concept of OpenWrt extroot, please read extroot.theory.

1. configure your bootloader to boot from the external USB device directly. Most bootloaders can't…!
2. configure your bootloader to boot over the network →netboot Many bootloaders can't…!
3. make opkg install new packages somewhere else: →Installation destinations
4. use kexec

1. Please read the article extroot.theory for better understanding.
2. This entire extroot functionality is Work-in-Progress, so is this article, so please make sure to check Troubleshooting and the Notes as a preparation and also later on from time to time!

1. you need to mount a file system, on which the overlay will be copied to. You could
   1. follow usb.essentials and additionally usb.storage for USB storage
   2. alternatively follow sata.essentials, ide.essentials or cf.essentials for other storage
   3. alternatively follow Mounting Filesystems to mount a remote network file system

• block-extroot
• block-hotplug
• block-mount

Depending on your medium (USB-Device, IDE-Device, SATA-Device or SD/MMC device) you need:

• kmod-usb-core Kernel support for USB
• kmod-usb2 Kernel support for USB2 (EHCI) controllers
• kmod-usb-ohci Kernel support for USB OHCI controllers
• kmod-usb-storage Kernel support for USB Mass Storage devices

or

• kmod-ata-ahci Support for AHCI Serial ATA controllers
• and any other required SATA drivers

or

• kmod-ide-core Kernel support for IDE, useful for usb mass storage devices
• and any other required IDE drivers

or

• kmod-mmc Kernel support for MMC/SD cards
• and any required required drivers

You also need the kernel module for the filesystem with which you formated the partition you want to mount:

• kmod-fs-ext4

or

• kmod-fs-ext3

or

• kmod-fs-ext2

Optionally:

• e2fsprogs contains essential ext2, ext3 and ext4 file system utilities (e2fsck, mke2fs, debugfs, dumpe2fs, tune2fs, ..) considerably smaller than →e2fsprogs in Debian repos
  • e2fsck contained in e2fsprogs. Cannot be installed alone. Please regard RAM requirements to check big file systems.¹⁾

opkg update
opkg install block-mount block-hotplug block-extroot

1. Copy necessary files from flash to the new root partition:
   1. For pivot overlay you can either use an empty new rootfs OR copy the contents of the current overlay (JFFS2) to the new rootfs (assuming the filesystem for the new external rootfs is mounted on /mnt/sda1):

tar -C /overlay -cvf - . | tar -C /mnt/sda1 -xf -

The configuration of extroot is very simple and is done entirely in /etc/config/fstab.

For backfire the option is_rootfs is required for block-extroot mounts!

This is the only variant available in Backfire.

Configure by setting the option is_rootfs in your /etc/config/fstab. Besides that, there is really nothing to be configured regarding ExtRoot that is different from any other mount.

For Backfire 10.03, and 10.03.1-rc1 to 10.03.1-rc5 use the example given below.

Note: Using the directory /mnt as target will make testing possible, first set is_rootfs to 0, reboot and check if all mounts well and than set it to 1 and reboot again. Make sure the target it is NOT manually set to /overlay. /overlay is automatically used on a successful is_rootfs-mount, but if for some reason this file system could not be made the rootfs it will be mounted on the target listed here. Using /overlay will result in an double /overlay mount and screw thing up.

1. you need to mount a file system, on which the overlay will be copied to. You could
   1. follow usb.essentials and additionally usb.storage for USB storage
   2. alternatively follow sata.essentials, ide.essentials or cf.essentials for other storage
   3. alternatively follow Mounting Filesystems to mount a remote network file system
2. decide whether you want to pivot /overlay (recommended) or to pivot / (entire root)

• block-mount

Depending on your medium (USB-Device, IDE-Device, SATA-Device or SD/MMC device) you need:

• kmod-usb-core Kernel support for USB
• kmod-usb2 Kernel support for USB2 (EHCI) controllers
• kmod-usb-ohci Kernel support for USB OHCI controllers
• kmod-usb-storage Kernel support for USB Mass Storage devices

or

• kmod-ata-ahci Support for AHCI Serial ATA controllers
• and any other required SATA drivers

or

• kmod-ide-core Kernel support for IDE, useful for usb mass storage devices
• and any other required IDE drivers

or

• kmod-mmc Kernel support for MMC/SD cards
• and any required required drivers

You also need the kernel module for the filesystem with which you formated the partition you want to mount:

• kmod-fs-ext4 (recommended, this also supports ext2 and ext3)

Optionally:

• e2fsprogs contains essential ext2, ext3 and ext4 file system utilities (e2fsck, mke2fs, debugfs, dumpe2fs, tune2fs, ..) considerably smaller than →e2fsprogs in Debian repos
  • e2fsck contained in e2fsprogs. Cannot be installed alone. Please regard RAM requirements to check big file systems.

Also required:

• kmod-scsi-generic Linux kernel (2.6.30 and later) uses the SCSI devices to link any storage media.

opkg update
opkg install block-mount

1. Copy necessary files from flash to the new root partition:
   1. For pivot overlay you can either use an empty new rootfs OR copy the contents of the current overlay (JFFS2) to the new rootfs (assuming the filesystem for the new external rootfs is mounted on /mnt/sda1):

      tar -C /overlay -cvf - . | tar -C /mnt/sda1 -xf -

   2. For pivot root (only possible as of r26109!) you must make sure to have a complete root filesystem on the external rootfs device. One possible way to get such a system (assuming the filesystem for the new external rootfs is mounted on /mnt/sda1) is to issue

      mkdir -p /tmp/cproot
      mount --bind / /tmp/cproot
      tar -C /tmp/cproot -cvf - . | tar -C /mnt/sda1 -xf -
      umount /tmp/cproot

The configuration of extroot is very simple and is done entirely in /etc/config/fstab.

But because this is such a WiP, you really need to read on before just switching extroot on!

option is_rootfs is deprecated after r25787! Before Trunk r25787 the option is_rootfs is required for block-extroot mounts!

Before r25787 configure by setting the option is_rootfs in your /etc/config/fstab. Besides that, there is really nothing to be configured regarding ExtRoot that is different from any other mount.

For trunk versions up to, but not including r25787 and the releases Backfire 10.03, and 10.03.1-rc1 to 10.03.1-rc5 use the example given below.

Note: Using the directory /mnt as target will make testing possible, first set is_rootfs to 0, reboot and check if all mounts well and than set it to 1 and reboot again. Make sure the target it is NOT manually set to /overlay. /overlay is automatically used on a successful is_rootfs-mount, but if for some reason this file system could not be made the rootfs it will be mounted on the target listed here. Using /overlay will result in an double /overlay mount and screw thing up.

While option is_rootfs will still work after r25787, the preferred method of configuring the extroot is option target /overlay in the config mount section for the rootfs device in the /etc/config/fstab file.

For trunk versions newer than and including r25787 use this variant. Backfire releases do not support this.

Note: If the overlay mount fails it will appear on /tmp/overlay-disabled instead of becoming the rootfs. No more double mounts even on failure.

After r26109 you can configure a non-overlay rootfs (called a whole_root extroot because the entire filesystem must be present on device, not only the changes from the SquashFS) using option target / in the config mount section for the rootfs device.

Note that this isn't required or preferred, it's just another option for those who want it. Backfire releases do not support this.

In order to set up such a whole root overlay, refer to the example below.

Note: If the non-overlay extroot mount fails before r26311 it will appear mounted to /tmp/rom-disabled, while after r26311 it will appear mounted to /tmp/whole_root-disabled.

The following steps were made on the hg553 using OpenWrt trunk@r28057: openwrt-HW553-jffs2-64k-cfe.bin after correctly setting up usb.essentials to obtain basic support for USB and usb.storage for USB storage.

The HG553 has two USB ports, I've used the one on the top. The USB drive was recognized as /dev/sda.

mkfs.ext4 /dev/sda1
mount /dev/sda1 /mnt
mkdir /tmp/cproot
mount --bind / /tmp/cproot
tar -C /tmp/cproot -cvf - . | tar -C /mnt -xvf -
sync ; umount /mnt
umount /tmp/cproot

After that, fstab should be edited to reflect the desired mount point:

After reboot you should check your mounts and get something like:

/dev/root on /rom type jffs2 (ro,noatime)
proc on /proc type proc (rw,noatime)
sysfs on /sys type sysfs (rw,noatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noatime,size=31020k)
tmpfs on /dev type tmpfs (rw,noatime,size=512k,mode=755)
devpts on /dev/pts type devpts (rw,noatime,mode=600)
/dev/sda1 on / type ext4 (rw,sync,relatime,user_xattr,barrier=1,data=ordered)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
none on /proc/bus/usb type usbfs (rw,relatime)

If all previous don't work, try with a sysupgrade restoring configurations files (especially fstab with the desired configuration). Then be generated correctly a "whole_root-disabled", that is need to fix (and repeat this operation for each new sysupgrade). However, if the files contained in the extroot are already updated, you may not need to perform this operation and extroot starts correctly (check with a df before and after reboot again), although it is best to clean the partition before, especially if we can not it to work when trying to enable:

mkdir -p /tmp/cproot
mount --bind / /tmp/cproot
tar -C /tmp/cproot -cvf - . | tar -C /tmp/whole_root-disabled -xf -
sync
umount /tmp/cproot
/etc/init.d/fstab whole_root_enable
reboot

With any "overlay-disabled" is more easy (note that first we need clean overlay, check that it's ok before do any rm):

rm -r /tmp/overlay-disabled/*
tar -C /overlay -cvf - . | tar -C /tmp/overlay-disabled -xf -
/etc/init.d/fstab overlay_enable
reboot

• Add option force_space in /etc/opkg.conf to allow installation of packets bigger than your /rom partitions free space:

  echo option force_space >> /etc/opkg.conf

• There is still a bug in extroot that prevents it from working with 2.4 kernels: /backfire/10.03.1-rc4/brcm-2.4 and /backfire/10.03/brcm-2.4. Extroot does work with /backfire/10.03.1-rc4/brcm47xx
• If extroot does not work with 10.03.1 try the latest rc (currently 10.03.1-rc6).
• Do not use vfat, it doesn't work! Use e.g. ext4 (install e2fsprogs, then format usb drive mkfs.ext4 /dev/sda1)

	Note: Before OpenWrt 'Backfire' 10.03.1-RC4 the block-extroot package is still existent and cannot be installed later on with opkg. It has to be included in the image. There is a separate howto_build for this extroot.howtobuild
	As of r26314 block-extroot and block-hotplug have been merged with block-mount.

• https://forum.openwrt.org/viewtopic.php?id=32812

I recommend that you DO NOT try to do upgrades using opkg upgrade. You will likely end up with an inconsistent state and bricked router that way:

• The main reason is that the uClibc ABI (Application Binary Interface) is unstable and changes from revision to revision, so binaries for one version of uClibc may be incompatible with versions from another.
• Another problem that can arise is if you try to follow the advice in the Old Notes and upgrade the kernel packages, then flash and reboot, but your operation is interrupted in any way, then you will have a kernel and module mismatch and likely a brick.
• Finally, if you upgrade all packages but the kernel and the kernel modules, some packages like iptables will be broken.

In Trunk r25269 respectably Backfire r25270 the detection of a mismatch in a previously generated key (md5sum) for extroot and JFFS2 was correctly implemented! Since that the following applies: A potentially broken overlay is mounted on /tmp/overlay-disabled instead of as rootfs, to give you a chance to fix it before mounting the extroot as the rootfs. That means you can safely flash the kernel with e.g. sysupgrade imagename.bin and not end up with a brick. Instead the system will have mounted from the SquashFS/JFFS2 instead of the extroot.

In case of a md5sum-mismatch you should:

1. make a copy of any data you want to keep from the extroot
2. recreate the extroot
3. copy back your data back to recreated extroot
4. remove the md5sums (<extroot>/.extroot.md5sum and <extroot>/etc/extroot.md5sum). As of trunk r26312 you can achieve the md5sum removal with

   /etc/init.d/fstab enable_overlay

5. make sure your /etc/config/fstab (located on the JFFS2) is correct
6. reboot.

In future I plan to have a script that removes all non-conffiles (except also preserving files preserved by sysupgrade) pointed to the opkg list on the extroot, and few other cleanups, which will hopefully sanitize extroot so it can be safely used again.

So to upgrade all your installed packages, you could do

opkg upgrade $(opkg list-upgradable)

DO NOT DO THAT! Since the kernel is on read-only flash partition, and all the kernel modules are on your external device, the modules will be updated, but the kernel not. You will end up with not matching kernel <> kernel-modules and your installation will not survive a reboot. To avoid that, you have to exclude all kernel-modules from being updated, for example so:

opkg upgrade $(opkg list-upgradable | awk '($1 !~ "^kmod|Multiple") {print $1}')

To upgrade kernel + kernel modules you need to first upgrade the kernel-modules (with opkg upgrade … –force… ) and then immediately WITHOUT rebooting, reflash Kernel + SquashFS with sysupgrade. Then reboot.