User Tools

Site Tools


Rootfs on External Storage (extroot)

  • More often than not, a limited amount of storage is available on embedded devices. While flash memory will generally accommodate a bare OpenWrt installation, more storage for applications and data can expand a device's potential; with many devices having expansion capabilities built-in (i.e. USB, SATA, and PCIe ports, NAS, etc.).
  • Applications are developed with the idea they should be installed in the root file system (rootfs). By employing extroot, expansion of the storage capacity of your root file system is accomplished by using an added storage device. During the boot process, external storage space is mounted as the root file system, or in an overlay configuration over the original file system. To understand the technical details, please read extroot theory.


  • Configuring the bootloader to boot from an external USB drive, of which most can not.
  • Configuring the bootloader to boot over the network via netboot, of which most can not.
  • Make opkg install new packages somewhere else; however:
    • Installing packages in root is more convenient, as files are installed in locations the OS expects them to reside.
  • Use kexec; however, this requires custom configurations on different devices.

Automatic Setup

  • There's a project called OpenWrt Auto extroot that can build a customized firmware image (using the Image Generator).
    • When this image is flashed, it will automatically set up extroot on any inserted storage device and then reboot (hopefully into the freshly set up extroot).
    • It's a nice foundation for custom applications that require auto provisioning, or just for your convenience at home for an easy extroot setup.
  • This is an ImageBuilder example for building very small images (compatible with 4 MB devices) with only the required packages for manual extroot setup. Please mind that there is no LuCI installed. Once extroot is performed, you can install LuCI and all the packages you require on top of the newly created filesystem.
    make image PROFILE=TLMR3020 PACKAGES="blkid block-mount kmod-fs-ext4 kmod-usb2 kmod-usb-uhci kmod-usb-ohci kmod-usb-storage"


You need to mount a file system, on which the overlay will be copied to.

  • For USB support, follow usb.essentials
  • For USB storage support, follow See storage for general information on partitioning, formatting and mounting storage devices.
  • Alternatively, follow Mounting Filesystems to mount a remote network file system

Make sure that you can mount and have read/write access to your external storage device. For example, check if you can read/write from a manually mounted file system on your partitioned USB drive at /dev/sda (adapt sda to your actual needs):

  mount /dev/sda1 /mnt/sda1

You should check if the mount was succesful by running mount or df. Now check if you can write to /mnt/sda1 by writing a file to it and reading it back.

root@OpenWrt:~# df
Filesystem           1K-blocks      Used Available Use% Mounted on
rootfs                     896       244       652  27% /
/dev/root                 2048      2048         0 100% /rom
tmpfs                    14708        64     14644   0% /tmp
/dev/mtdblock6             896       244       652  27% /overlay
overlayfs:/overlay         896       244       652  27% /
tmpfs                      512         0       512   0% /dev
/dev/sda1              7759872    477328   7221104   6% /mnt/sda1

Chaos Calmer

As described in extroot theory you can use pivot-overlay or pivot-root. This section will only cover pivot-overlay which is the recommended extroot implementation. If you really want pivot-root, check the Barrier Breaker section.


  • block-mount
  • kmod-fs-ext4 or kmod-fs-[filesystem of choice]
  • kmod-usb-storage-extras
    opkg update ; opkg install block-mount kmod-fs-ext4 kmod-usb-storage-extras
  • If opkg errors when installing kmod-usb-storage-extras, you might need to install


  1. Format and partition drive the way you want, either using a Linux Live CD or via uci (it's recommended to use first partition for the root overlay, second partition for swap (if you want swap), and then you can add other partitions as you like)
  2. Prepare your external storage root overlay
    mount /dev/sda1 /mnt ; tar -C /overlay -cvf - . | tar -C /mnt -xf - ; umount /mnt
  3. Create fstab with the following command
    block detect > /etc/config/fstab; \
       sed -i s/option$'\t'enabled$'\t'\'0\'/option$'\t'enabled$'\t'\'1\'/ /etc/config/fstab; \
       sed -i s#/mnt/sda1#/overlay# /etc/config/fstab; \
       cat /etc/config/fstab;

This command will create a fstab template enabling all partitions and setting '/mnt/sda1' partition as '/overlay' partition .

  1. See the following in /etc/config/fstab:
    1. Check if all 'option enabled' is set to '1'
    2. In 'option target' of your root overlay partition must be set to '/overlay'
    3. You can edit 'option target' of your other partition(s) to show whatever mount point(s) you want them to have (make sure to also create the directories those mount points point to)

 vi /etc/config/fstab

  1. You'll end up with an fstab looking something like this:
    config 'global'
            option  anon_swap       '0'
            option  anon_mount      '0'
            option  auto_swap       '1'
            option  auto_mount      '1'
            option  delay_root      '5'
            option  check_fs        '0'
    config 'mount'
            option  target  '/overlay'
            option  uuid    'c91232a0-c50a-4eae-adb9-14b4d3ce3de1'
            option  fstype  'ext4'
            option  enabled '1'
    config 'swap'
            option  uuid    '08b4f0a3-f7ab-4ee1-bde9-55fc2481f355'
            option  enabled '1'
    config 'mount'
            option  target  '/data'
            option  uuid    'c1068d91-863b-42e2-bcb2-b35a241b0fe2'
            option  enabled '1'
  2. Check if it is mountable to overlay:
    root@OpenWrt:~# mount /dev/sda1 /overlay
    root@OpenWrt:~# df
    Filesystem           1K-blocks      Used Available Use% Mounted on
    rootfs                     896       244       652  27% /
    /dev/root                 2048      2048         0 100% /rom
    tmpfs                    14708        64     14644   0% /tmp
    /dev/mtdblock6         7759872    477328   7221104   6% /overlay
    overlayfs:/overlay         896       244       652  27% /
    tmpfs                      512         0       512   0% /dev
    /dev/sda1              7759872    477328   7221104   6% /overlay
    Note that only /overlay has grown but not the /
  3. Reboot the router
  4. Verify that the partitions were mounted properly:
  • via LuCI
    • System - Software should show free space of overlay partition
    • System - Mount Points should show USB partition mounted as overlay
  • via CLI
    • mount should show USB partition mounted as /overlay
      root@OpenWrt:~# mount
      /dev/root           on /rom               type squashfs (ro,relatime)
      proc                on /proc              type proc     (rw,noatime)
      sysfs               on /sys               type sysfs    (rw,noatime)
      tmpfs               on /tmp               type tmpfs    (rw,nosuid,nodev,noatime)
      /dev/mtdblock6      on /overlay           type jffs2    (rw,noatime)
      overlayfs:/overlay  on /                  type overlay  (rw,noatime,lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work)
      tmpfs               on /dev               type tmpfs    (rw,relatime,size=512k,mode=755)
      devpts              on /dev/pts           type devpts   (rw,relatime,mode=600)
      /dev/sda1           on /overlay           type ext4     (rw,relatime,data=ordered)
      /dev/sda3           on /data              type ext4     (rw,relatime,data=ordered)
      debugfs             on /sys/kernel/debug  type debugfs  (rw,noatime)
    • df should show free space available on your /overlay and / partition, all the storages mounted to /overlay and / (the rootfs in the first place) should look the same increased capacity:
      root@OpenWrt:~# df
      Filesystem           1K-blocks      Used Available Use% Mounted on
      rootfs                 7759872    477328   7221104   6% /
      /dev/root                 2048      2048         0 100% /rom
      tmpfs                    14708        64     14644   0% /tmp
      /dev/mtdblock6         7759872    477328   7221104   6% /overlay
      overlayfs:/overlay     7759872    477328   7221104   6% /
      tmpfs                      512         0       512   0% /dev
      /dev/sda1              7759872    477328   7221104   6% /overlay
      /dev/sda3            242846048    163864 230323224   0% /data
  1. You're done!


  • Add option force_space in /etc/opkg.conf to allow installation of packets bigger than your FIXME /rom partitions free space:
    echo option force_space >> /etc/opkg.conf
  • Do not use vfat (FAT/FAT32); it does not work. If you have a FAT preformatted USB drive, you cannot use it for extroot without reformatting. Use e.g. ext4 (install e2fsprogs, then format your FAT formatted USB drive using mkfs.ext4 /dev/sda1 as per the example, also see storage).
  • If the partition containing your extroot isn't mounted during boot, but you can mount it without problems from a shell, you should try to increase config 'global' / option delay_root. On my system I had to set it to 15 seconds to get extroot working. Another hint to this being the culprit is having a working swap or other partitions mounted after booting, but not your extroot.
  • Another possibility to consider and try is to include in /etc/rc.local the commands: export PREINIT = 1;mount_root, as described in 14946 ticket, which in the case of running Chaos Calmer r44266 in the Comtrend AR-5387un, has been the only thing that allowed me to achieve extroot.
  • If after successful extroot mount, swap isn't enabled and other partitions aren't mounted, check your /etc/config/fstab in the overlay partition. If your /etc/config/fstab in the overlay partition only contains a global section, you may need to add the following from the example:
    config 'swap'
            option  uuid    '08b4f0a3-f7ab-4ee1-bde9-55fc2481f355'
            option  enabled '1'
    config 'mount'
            option  target  '/data'
            option  uuid    'c1068d91-863b-42e2-bcb2-b35a241b0fe2'
            option  enabled '1'
  • If you are putting the extroot on a non-USB device such as a mmc card all modules needed acccess the device should be in appropriate file in /etc/modules-boot.d. For example using a sdhci card on a mt7688/mt7628 device /etc/modules-boot.d/mmc needs have two lines added:


Using Image Generator

As the kmod-fs-ext4 package is rather large, it can be a challenge setting up extroot on 4MB flash devices, where extroot is most needed. Therefore, you can build an image to flash to your device with the required packages already included, this method saves considerable space on your device.

Using the image generator makes building the image very easy. See the Image Generator page for more detailed instructions.

  1. Extract archive, change to directory
  2. To build your firmware (Note: Replace TLMR3220 with the profile for your own device, use make info for a list):
    make image PROFILE="TLMR3220" PACKAGES="kmod-fs-ext4 kmod-usb-storage"

Your new firmware file will be located within the bin folder, with the packages to enable extroot already included.

Extroot to a card in a slot of an USB-dongle

The Use 3g/UMTS USB Dongle for WAN connection article advises to include the usb-modeswitch tool in the image.
There is a caveat: if the /overlay points to a memory card sitting in a slot of the dongle - the otherwise working pivot overlay set-up will break in the later stages of OS boot. This is because the usb-modeswitch (while disabling the CDROM and enabling the modem) would also intermittently affect the card-reader in the dongle thus hurting the file system.
To avoid this you need a dongle that can be pre-configured to enable its' modem or network adapter (and the card-reader as well) on the power-up, without the need to do it with the usb-modeswitch on the router.

Insert your dongle in a desktop and use a terminal to send the necessary AT-commands.
Check your dongle's initial configuration:

The meaning of the above report can be understood with the following command:
So, in the example above we have a dongle with CDROM and card-reader available in the first configuration (to the left of the ; character), and with modem, control and diagnostic interfaces, and card-reader available in the other configuration. It is between these configurations the usb-modeswitch switches the dongle on the router.

Your goal is to disable the CDROM and enable the modem (the 1 above) or the network adapter (the 16 above) while leaving the card-reader enabled (the A2 above).
NOTE: Never disable the PCUI (the 2 above) - this will lock you out from your dongle!

Some dongles accept a 'disable all' operand (the FF below).
Place the list of all the functions you need on your dongle by default to the right of the ; character according to their codes from the dongle's answer above:


This sequence has disabled the CDROM and made the modem, control and diagnostic interfaces and the card-reader available by default - without any usb-modeswitch interaction. Thus only one configuration exists now in the dongle - see the ; character, there is nothing to the left of it now.

Dongles known to support the pre-configuration

  • Huawei E3131s-2 f/w v21.

Remote File Systems

System Upgrade

I recommend that you DO NOT try to do upgrades using opkg upgrade. You will likely end up with an inconsistent state and bricked router that way:

  • The main reason is that the uClibc ABI (Application Binary Interface) is unstable and changes from revision to revision, so binaries for one version of uClibc may be incompatible with versions from another.
  • Another problem that can arise is if you try to upgrade the kernel packages, then flash and reboot, but your operation is interrupted in any way, then you will have a kernel and module mismatch and likely a brick.
  • Finally, if you upgrade all packages but the kernel and the kernel modules, some packages like iptables will be broken.

Accessing original root

Sometimes you may need to access the original root overlay, for example to change your extroot settings. A convenient way of doing this is to configure /etc/config/fstab on your extroot partition to mount the original root overlay in another directory, like this:

config mount
	option target	/overlay-boot
	option device	/dev/mtdblock3
	option fstype	jffs2
	option options	rw,sync
	option enabled	1
	option enabled_fsck 0
This assumes the original, internal overlay was on /dev/mtdblock3 - check your router's page on this wiki and look at the flash map to confirm what MTD block it is for you. Or run $ cat /proc/mtd and search the partition named rootfs_data to know where is your overlay.

If you then create /overlay-boot on the extroot partition, this directory will contain the original root overlay, which is used as the main root overlay until the extroot is up and running. So you can then for example edit /overlay-boot/etc/config/fstab to change your extroot configuration (or temporarily disable it) should you ever need to.

Information on Legacy versions

For extroot on OpenWrt versions Barrier Breaker, Attitude Adjustment, Backfire see: extroot.old

doc/howto/extroot.txt · Last modified: 2016/09/07 12:25 by noblepepper