Rootfs on External Storage (extroot)

More often then not, there is a limited amount of storage space available on embedded devices. While the available flash memory will usually accomodate a bare OpenWrt installation, more room for applications and data can tremendously expand a device's potential. Luckily, many of these devices have these expansion capabilities built-in, for example in the form of USB ports, SATA ports, PCIexpress slots, or even storage in a network location. However, many of the applications you want to install are developed with the idea that they should be installed in the root file system (rootfs). By employing OpenWrt's extroot, you can expand the storage capacity of your root file system using the additional space of an added storage device. At a certain point in the boot process the external storage space is mounted as the root file system or in an overlay configuration over the original file system. To understand the technical details of OpenWrt extroot, please read extroot.theory. This article explains how to get it to work.

Alternatives to extroot

To see if extroot is a good option for increasing the amount of storage for new application packages, look at the following alternatives to achieve this goal:

  1. Configure your bootloader to boot from the external USB device directly. However, most bootloaders can't…!
  2. Configure your bootloader to boot over the network →netboot. However, many bootloaders can't…!
  3. Make opkg install new packages somewhere else →Installation destinations. However, installing OPKG packages in root is much more convenient, as its files are installed in locations that the rest of the OS expects them to reside.
  4. Use kexec. However, this may require custom configurations on different devices.

Preparations

  1. Please read the article extroot.theory for better understanding of extroot. In particular, understand the difference between the two extroot variants: pivot overlay and pivot root.
  2. For different OpenWrt versions extroot configuration and features differ. The sections below provide separate instructions for OpenWrt 10 and higher. Notably, in OpenWrt 10 (Backfire) you can only use the pivot overlay variant, not pivot root.
  3. The entire extroot functionality is Work-in-Progress, so is this article, so please make sure to check Troubleshooting and the Notes as a preparation and also later on from time to time!
If for some reason extroot fails at boot, the normal configuration is loaded that is still in the JFFS2 partition on the main flash, which is the old configuration that you copied to the external root device and have since changed from there. This configuration may pose a security risk (for example if its configuration of SSH gives unprotected access from the internet). You should therefore make sure the system is safe and protected before installing extroot. Also, any changes you make to your network environment should be compatible with the security in the original JFFS2 configuration.

Prerequisites

You need to mount a file system, on which the overlay will be copied to.

Make sure that you can mount and have read/write access to your external storage device. For example, check if you can read/write from a manually mounted file system on your partitioned USB drive at /dev/sda:

  mount /dev/sda1 /mnt/sda1

You should check if the mount was succesful by running mount or df. Now check if you can write to /mnt/sda1 by writing a file to it and reading it back.

OpenWrt 12.09 'Attitude Adjustment'

For OpenWrt 12, the second extroot variant pivot root is added. Decide whether you want to pivot /overlay (recommended) or to pivot / (entire root)

Required Packages

  • block-mount

This is the only required extroot package for OpenWrt 12, see block_mount.

Installation

After preparing your storage device and its file system, installation proceeds as follows. First the required packages are installed. Then the existing files from the current JFFS2 overlay partition on flash are copied to the new storage device. Then the system is configured using fstab for activating extroot at boot.

Install packages

opkg update
opkg install block-mount
Also opkg install any filesystem or usb packages you need

Duplicate data

Copy necessary files from flash to the new root partition:

  1. For pivot overlay you can either use an empty new rootfs OR copy the contents of the current overlay (JFFS2) to the new rootfs (assuming the filesystem for the new external rootfs is mounted on /mnt/sda1):
    tar -C /overlay -cvf - . | tar -C /mnt/sda1 -xf -
  2. For pivot root (only possible as of r26109!) you must make sure to have a complete root filesystem on the external rootfs device. One possible way to get such a system (assuming the filesystem for the new external rootfs is mounted on /mnt/sda1) is to issue
    mkdir -p /tmp/cproot
    mount --bind / /tmp/cproot
    tar -C /tmp/cproot -cvf - . | tar -C /mnt/sda1 -x
    umount /tmp/cproot

Configuration

The configuration of extroot is very simple and is done entirely in /etc/config/fstab.

Old external overlay variant (pivot overlay)

Before r25787 configure by setting the option is_rootfs in your /etc/config/fstab. Besides that, there is nothing to be configured regarding extroot that is different from any other mount.

For trunk versions up to, but not including r25787 the following is an example for an extroot mount entry in the file /etc/config/fstab. Note the fstype, options and enable_fchk options are optional, see fstab.

config mount option target /mnt # This is ignored once is_rootfs is set to 1 option device /dev/sda1 option fstype ext3 option options rw,sync option enabled 1 option enabled_fsck 0 option is_rootfs 1

A target option is not mandatory as it will be the overlay file system. However, using the directory /mnt as target will make testing possible: first set is_rootfs to 0, reboot and check if all mounts well and than set it to 1 and reboot again. Make sure the target it is not manually set to /overlay. /overlay is automatically used on a successful is_rootfs-mount, but if for some reason this file system could not be made the rootfs will be mounted on the target listed here. Using /overlay will result in an double /overlay mount and screw things up. In later releases, the target is used to determine if a mount entry is an extroot device and is_rootfs is not used anymore to circumvent this.

option is_rootfs is deprecated after r25787. Use the new overlay variant where the target is used to determine if a mount entry is an extroot device. Before Trunk r25787 the option is_rootfs is required for block-extroot mounts!
New external overlay variant (pivot overlay)

While option is_rootfs will still work after r25787, the preferred method of configuring the extroot is option target /overlay in the config mount section for the rootfs device in the /etc/config/fstab file.

For trunk versions newer than and including r25787 use this variant. This is the entry in the /etc/config/fstab configuration file.

config mount option target /overlay option device /dev/sda1 option fstype ext3 option options rw,sync option enabled 1 option enabled_fsck 0

If the overlay mount fails it will appear on /tmp/overlay-disabled instead of becoming the rootfs. This means that you will have no more double mounts, even on failure .

Whole external root (pivot root)

After r26109 you can configure a non-overlay rootfs (called a whole_root extroot because the entire filesystem must be present on device, not only the changes from the SquashFS) using option target / in the config mount section for the rootfs device.

Note that this isn't required or preferred, it's just another option for those who want it. This is new to OpenWrt 12. Backfire releases do not support this.

In order to set up such a whole root overlay, use the following fstab mount entry in the file /etc/config/fstab

config mount option target / option device /dev/sda1 option fstype ext3 option options rw,sync option enabled 1 option enabled_fsck 0

If the non-overlay extroot mount fails before r26311 it will appear mounted to /tmp/rom-disabled, while after r26311 it will appear mounted to /tmp/whole_root-disabled.

If the system on the extroot is obtained from a prebuilt image the md5sums will not match and you need to copy the md5sum from the active system:

cp /.extroot.md5sum /tmp/whole_root-disabled/etc/extroot.md5sum
or
cp /.extroot.md5sum /tmp/overlay-disabled/etc/extroot.md5sum

Example

The following steps were made on the hg553 using OpenWrt trunk@r28057: openwrt-HW553-jffs2-64k-cfe.bin after correctly setting up usb.essentials to obtain basic support for USB and usb.storage for USB storage.

The HG553 has two USB ports, I've used the one on the top. The USB drive was recognized as /dev/sda.

mkfs.ext4 /dev/sda1
mount /dev/sda1 /mnt
mkdir /tmp/cproot
mount --bind / /tmp/cproot
tar -C /tmp/cproot -cvf - . | tar -C /mnt -xvf -
sync ; umount /mnt
umount /tmp/cproot
Note: mkfs.ext4 is provided by package e2fsprogs.

After that, /etc/config/fstab should be edited to reflect the desired mount point:

config mount option target / option device /dev/sda1 option fstype ext4 option options rw,sync option enabled 1 option enabled_fsck 0

Note the fstype, options and enable_fchk options are optional, see fstab.

After reboot you should check your mounts and get something like:

/dev/root on /rom type jffs2 (ro,noatime)
proc on /proc type proc (rw,noatime)
sysfs on /sys type sysfs (rw,noatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noatime,size=31020k)
tmpfs on /dev type tmpfs (rw,noatime,size=512k,mode=755)
devpts on /dev/pts type devpts (rw,noatime,mode=600)
/dev/sda1 on / type ext4 (rw,sync,relatime,user_xattr,barrier=1,data=ordered)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
none on /proc/bus/usb type usbfs (rw,relatime)

OpenWrt 'Barrier Breaker' (trunk)

For the current trunk of OpenWrt, a new block-mount package is introduced, refer to block_mount and fstab.

Decide whether you want to pivot /overlay (recommended) or to pivot / (entire root)

Required Packages

  • block-mount

This is the only required extroot package for OpenWrt 12, see block_mount. Note that from r36988 the new block utility can be used, which is the new runnable that is responsible for all mounting procedures. See fstab Configuration for more details. The extroot fstab configuration is unchanged from the new external overlay/root variants in OpenWrt 12.

Installation

After preparing your storage device and its file system, installation proceeds as follows. First the required packages are installed. Then the existing files from the current JFFS2 overlay partition on flash are copied to the new storage device. Then the system is configured using fstab for activating extroot at boot.

Install packages

opkg update
opkg install block-mount
Also opkg install any filesystem or usb packages you need

Duplicate data

Copy necessary files from flash to the new root partition:

  1. For pivot overlay you can either use an empty new rootfs OR copy the contents of the current overlay (JFFS2) to the new rootfs (assuming the filesystem for the new external rootfs is mounted on /mnt/sda1):
    tar -C /overlay -cvf - . | tar -C /mnt/sda1 -xf -
  2. For pivot root (only possible as of r26109!) you must make sure to have a complete root filesystem on the external rootfs device. One possible way to get such a system (assuming the filesystem for the new external rootfs is mounted on /mnt/sda1) is to issue
    mkdir -p /tmp/cproot
    mount --bind / /tmp/cproot
    tar -C /tmp/cproot -cvf - . | tar -C /mnt/sda1 -xf -
    umount /tmp/cproot

Configuration

The configuration of extroot is very simple and is done entirely in /etc/config/fstab.

Pivot Overlay

The following is an example for an extroot /etc/config/fstab mount entry for pivot overlay. Note the fstype, options and enable_fchk options are optional, see fstab.

config mount option target /overlay option device /dev/sda1 option fstype ext3 option options rw,sync option enabled 1 option enabled_fsck 0

If the overlay mount fails it will appear on /tmp/overlay-disabled instead of becoming the rootfs. This means that you will have no more double mounts, even on failure.

Pivot Root

After r26109 you can configure a non-overlay rootfs (called a whole_root extroot because the entire filesystem must be present on device, not only the changes from the SquashFS) using option target / in the config mount section for the rootfs device.

Note that this isn't required or preferred, it's just another option for those who want it. This is introduced in OpenWrt 12.

In order to set up such a whole root overlay, use the following /etc/config/fstab mount entry

config mount option target / option device /dev/sda1 option fstype ext3 option options rw,sync option enabled 1 option enabled_fsck 0

If the non-overlay extroot mount fails before r26311 it will appear mounted to /tmp/rom-disabled, while after r26311 it will appear mounted to /tmp/whole_root-disabled.

Example

This example assumes you have a device connected properly, at /dev/sda with a partition at /dev/sda1. The following will format at to an ext4 partition, mount it at /mnt/sda1,

mkfs.ext4 /dev/sda1  
mount /dev/sda1 /mnt/sda1
tar -C /overlay -cvf - . | tar -C /mnt/sda1 -xf -
Note: mkfs.ext4 is provided by the package e2fsprogs.

Next, edit your fstab, for example with vi:

vi /etc/config/fstab

You need to modify the target line and option enabled line so that it looks like this (Note: In this example we have referred to the partition by its UUID instead of its device file. The UUID will be different for you as this is just an example. Alternatively, you can point to the right partition using the device option set to the device file /dev/sda1 as in the previous examples):

config 'global'
        option  anon_swap       '0'
        option  anon_mount      '0'
        option  auto_swap       '1'
        option  auto_mount      '1'
        option  delay_root      '0'
        option  check_fs        '0'

config 'mount'
        option  target  '/overlay'
        option  uuid    '7d3abfaf-493a-46bb-9730-1d793ecb9783'
        option  enabled '1'

You can also use UCI CLI commands to change the fstab file:

uci set fstab.@mount[0].target=/overlay
uci set fstab.@mount[0].enabled=1
uci commit fstab
/etc/init.d/fstab restart

Reboot your system.

To check if extroot is working run df and it should give an output like this:

root@OpenWrt:~# df
Filesystem           1K-blocks      Used Available Use% Mounted on
rootfs                 2758072    118004   2501828   5% /
/dev/root                 2048      2048         0 100% /rom
tmpfs                    63340        76     63264   0% /tmp
/dev/sda1              2758072    118004   2501828   5% /overlay
overlayfs:/overlay     2758072    118004   2501828   5% /
tmpfs                      512         0       512   0% /dev

Notice rootfs and overlayfs:/overlay share the same space and both are mounted on the root (/). Further, /dev/sda1 is mounted on /overlay and is the overlay file system. For pivot root configurations the overlayfs:/overlay entry in df will be absent.

Troubleshooting

  • Add option force_space in /etc/opkg.conf to allow installation of packets bigger than your /rom partitions free space:
    echo option force_space >> /etc/opkg.conf
  • Do not use vfat (FAT/FAT32); it does not work. If you have a FAT preformatted USB drive, you cannot use it for extroot without reformatting. Use e.g. ext4 (install e2fsprogs, then format your FAT formatted USB drive using mkfs.ext4 /dev/sda1 as per the example, also see storage).
  • On Barrier Breaker, block-mount will create a file /etc/.extroot-uuid on extroot filled with uuid of mtd partition rootfs. At boot time when trying to do extroot, block-mount would try to check the actual uuid with the content of .extroot-uuid. If they did not match, extroot would fail. So if you want to continue use extroot after flashing a new firmwre, /etc/.extroot-uuid needs to be deleted first.
  • If the partition containing your extroot isn't mounted during boot, but you can mount it without problems from a shell, you should try to increase config 'global' / option delay_root. On my system I had to set it to 15 to get extroot working. Another hint to this being the culprit is having a working swap or other partitions mounted after booting, but not your extroot.

Notes

Remote File Systems

System Upgrade

Recommended by extroot maintainer

I recommend that you DO NOT try to do upgrades using opkg upgrade. You will likely end up with an inconsistent state and bricked router that way:

  • The main reason is that the uClibc ABI (Application Binary Interface) is unstable and changes from revision to revision, so binaries for one version of uClibc may be incompatible with versions from another.
  • Another problem that can arise is if you try to follow the advice in the Old Notes and upgrade the kernel packages, then flash and reboot, but your operation is interrupted in any way, then you will have a kernel and module mismatch and likely a brick.
  • Finally, if you upgrade all packages but the kernel and the kernel modules, some packages like iptables will be broken.
In Trunk r25269 respectably Backfire r25270 the detection of a mismatch in a previously generated key (md5sum) for extroot and JFFS2 was correctly implemented! Since that the following applies:
A potentially broken overlay is mounted on /tmp/overlay-disabled instead of as rootfs, to give you a chance to fix it before mounting the extroot as the rootfs. That means you can safely flash the kernel with e.g. sysupgrade imagename.bin and not end up with a brick.
Instead the system will have mounted from the SquashFS/JFFS2 instead of the extroot.

In case of a md5sum-mismatch you should:

  1. make a copy of any data you want to keep from the extroot
  2. recreate the extroot
  3. copy back your data back to recreated extroot
  4. remove the md5sums (<extroot>/.extroot.md5sum and <extroot>/etc/extroot.md5sum). As of trunk r26312 you can achieve the md5sum removal with
    /etc/init.d/fstab overlay_enable
  5. make sure your /etc/config/fstab (located on the JFFS2) is correct
  6. reboot.

In future I plan to have a script that removes all non-conffiles (except also preserving files preserved by sysupgrade) pointed to the opkg list on the extroot, and few other cleanups, which will hopefully sanitize extroot so it can be safely used again.

Accessing original root

Sometimes you may need to access the original root overlay, for example to change your extroot settings. A convenient way of doing this is to configure /etc/config/fstab on your extroot partition to mount the original root overlay in another directory, like this:

config mount option target /overlay-boot option device /dev/mtdblock3 option fstype jffs2 option options rw,sync option enabled 1 option enabled_fsck 0

This assumes the original overlay was on /dev/mtdblock3 - check your router's page on this wiki and look at the flash map to confirm what MTD block it is for you. Or run "$ cat /proc/mtd" and search the partition named rootfs_data to know where is your overlay.

If you then create /overlay-boot on the extroot partition, this directory will contain the original root overlay, which is used as the main root overlay until the extroot is up and running. So you can then for example edit /overlay-boot/etc/config/fstab to change your extroot configuration (or temporarily disable it) should you ever need to.

Old Notes

So to upgrade all your installed packages, you could do

opkg upgrade $(opkg list-upgradable)

:!: DO NOT DO THAT! Since the kernel is on read-only flash partition, and all the kernel modules are on your external device, the modules will be updated, but the kernel not. You will end up with not matching kernel <> kernel-modules and your installation will not survive a reboot. To avoid that, you have to exclude all kernel-modules from being updated, for example so:

opkg upgrade $(opkg list-upgradable | awk '($1 !~ "^kmod|Multiple") {print $1}')

To upgrade kernel + kernel modules you need to first upgrade the kernel-modules (with opkg upgrade … –force… ) and then immediately WITHOUT rebooting, reflash Kernel + SquashFS with sysupgrade. Then reboot.

Information on Legacy versions

Backfire see: extroot.old

Back to top

doc/howto/extroot.txt · Last modified: 2014/09/17 12:42 by nicoo