User Tools

Site Tools



This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
doc:howto:extroot [2012/09/12 05:18]
doc:howto:extroot [2016/02/08 14:05] (current)
rhombus [Chaos Calmer] add note about other usb package required
Line 1: Line 1:
 +====== Rootfs on External Storage (extroot) ======
 +  * More often than not, a limited amount of storage is available on embedded devices. ​ While flash memory will generally accommodate a bare OpenWrt installation,​ more storage for applications and data can expand a device'​s potential; with many devices having expansion capabilities built-in (i.e. USB, SATA, and PCIe ports, NAS, etc.).  ​
 +  * Applications are developed with the idea they should be installed in the root file system (rootfs). ​ By employing //​extroot//,​ expansion of the storage capacity of your root file system is accomplished by using an added storage device. During the boot process, external storage space is mounted as the root file system, or in an overlay configuration over the original file system. To understand the technical details, please read [[doc:​howto:​extroot:​extroot.theory|extroot theory]].
 +===== Alternatives =====
 +  * Configuring the [[doc:​techref:​bootloader|bootloader]] to boot from an external USB drive, of which most can not.
 +    * Popular bootloaders (i.e. U-Boot or RedBoot) [[wp>​Comparison_of_boot_loaders|support different file systems]], but they lack the necessary hardware drivers.
 +  * Configuring the bootloader to boot over the network via [[inbox:​howto:​netboot|netboot]],​ of which most can not.
 +  * Make ''​opkg''​ install new packages [[doc:​techref:​opkg#​Installation destinations|somewhere else]]; however: ​
 +    * Installing packages in root is more convenient, as files are installed in locations the OS expects them to reside.
 +  * Use [[doc:​howto:​kexec|kexec]];​ however, this requires custom configurations on different devices. ​
 +===== Automatic Setup =====
 +  * There'​s a project called [[https://​​attila-lendvai/​openwrt-auto-extroot|OpenWrt Auto extroot]] that can build a customized firmware image (using the [[doc:​howto:​imagegenerator|Image Generator]]). ​
 +    * When this image is flashed, it will automatically set up extroot on any inserted storage device and then reboot (hopefully into the freshly set up extroot).
 +    * It's a nice foundation for custom applications that require auto provisioning,​ or just for your convenience at home for an easy extroot setup.
 +  * This is an ImageBuilder example for building very small images (compatible with 4 MB devices) with only the required packages for manual extroot setup. Please mind that there is no LuCI installed. Once extroot is performed, you can install LuCI and all the packages you require on top of the newly created filesystem.\\ <​code>​make image PROFILE=TLMR3020 PACKAGES="​blkid block-mount kmod-fs-ext4 kmod-usb2 kmod-usb-uhci kmod-usb-ohci kmod-usb-storage"</​code>​
 +==== Prerequisites ====
 +You need to **mount a file system**, on which the overlay will be copied to.
 +  * For USB support, follow [[doc:​howto:​usb.essentials]]
 +  * For USB storage support, follow [[doc:​howto:​]]. See [[doc:​howto:​storage]] for general information on partitioning,​ formatting and mounting storage devices.
 +  * Alternatively,​ follow [[doc:​howto:​client.overview#​Mounting Filesystems]] to mount a remote network file system
 +Make sure that you can mount and have read/write access to your external storage device. For example, check if you can read/write from a manually mounted file system on your partitioned USB drive at ''/​dev/​sda''​ (adapt ''​sda''​ to your actual needs):
 +    mount /dev/sda1 /mnt/sda1
 +You should check if the mount was succesful by running ''​mount''​ or ''​df''​. Now check if you can write to ''/​mnt/​sda1''​ by writing a file to it and reading it back. 
 +==== Chaos Calmer ====
 +As described in [[doc:​howto:​extroot:​extroot.theory|extroot theory]] you can use pivot-overlay or pivot-root. This section will only cover pivot-overlay which is the recommended extroot implementation. If you really want pivot-root, check the [[doc:​howto:​extroot.old#​Barrier Breaker|Barrier Breaker section]].
 +=== Prerequisites ===
 +  * //​block-mount//​
 +  * //​kmod-fs-ext4//​ or //​kmod-fs-//​[filesystem of choice] ​
 +  * //​kmod-usb-storage-extras//​ <​code>​opkg update ; opkg install block-mount kmod-fs-ext4 kmod-usb-storage-extras</​code>  ​
 +  * If opkg errors when installing kmod-usb-storage-extras,​ you might need to install <​code>​kmod-usb-core</​code> ​
 +=== Steps ===
 +  - Format and partition drive the way you want, either using a Linux Live CD or via uci //​(it'​s recommended to use first partition for the root overlay, second partition for swap (if you want swap), and then you can add other partitions as you like)//
 +  - Prepare your external storage root overlay<​code>​
 +mount /dev/sda1 /mnt ; tar -C /overlay -cvf - . | tar -C /mnt -xf - ; umount /mnt
 +  - Create an fstab template<​code>​
 +block detect > /​etc/​config/​fstab ; vi /​etc/​config/​fstab
 +  - Edit the following in ''/​etc/​config/​fstab'':​
 +    - Set all ''​enabled '​0'​ ''​ ->  ''​enabled '​1'​ ''​
 +    - Edit ''​option target''​ of your root overlay partition to show **''/​overlay''​** ​ (**__not__** ''​overlay'',​ as it will cause broken, duplicate mounts)
 +    - Edit ''​option target''​ of your other partition(s) to show whatever mount point(s) you want them to have (make sure to also create the directories those mount points point to)\\ You'll end up with an fstab looking something like this:<​code>​
 +config '​global'​
 +        option ​ anon_swap ​      '​0'​
 +        option ​ anon_mount ​     '​0'​
 +        option ​ auto_swap ​      '​1'​
 +        option ​ auto_mount ​     '​1'​
 +        option ​ delay_root ​     '​5'​
 +        option ​ check_fs ​       '​0'​
 +config '​mount'​
 +        option ​ target ​ '/​overlay'​
 +        option ​ uuid    '​c91232a0-c50a-4eae-adb9-14b4d3ce3de1'​
 +        option ​ enabled '​1'​
 +config '​swap'​
 +        option ​ uuid    '​08b4f0a3-f7ab-4ee1-bde9-55fc2481f355'​
 +        option ​ enabled '​1'​
 +config '​mount'​
 +        option ​ target ​ '/​data'​
 +        option ​ uuid    '​c1068d91-863b-42e2-bcb2-b35a241b0fe2'​
 +        option ​ enabled '​1'​
 +  - Reboot the router
 +  - Verify that the partitions were mounted properly:
 +    * via LuCI
 +      * //System - Software// should show free space of overlay partition
 +      * //System - Mount Points// should show USB partition mounted as ''​overlay''​
 +    * via CLI
 +      * ''​mount''​ should show USB partition mounted as ''/​overlay''​\\ <​code>​
 +/​dev/​root ​          on /rom               type squashfs (ro,​relatime)
 +proc                on /proc              type proc     ​(rw,​noatime)
 +sysfs               on /sys               type sysfs    (rw,​noatime)
 +tmpfs               on /tmp               type tmpfs    (rw,​nosuid,​nodev,​noatime)
 +/​dev/​ubi0_1 ​        on /​overlay ​          type ubifs    (rw,​noatime)
 +overlayfs:/​overlay ​ on /                  type overlay ​ (rw,​noatime,​lowerdir=/,​upperdir=/​overlay/​upper,​workdir=/​overlay/​work)
 +tmpfs               on /dev               type tmpfs    (rw,​relatime,​size=512k,​mode=755)
 +devpts ​             on /​dev/​pts ​          type devpts ​  ​(rw,​relatime,​mode=600)
 +/​dev/​sda1 ​          on /​overlay ​          type ext4     ​(rw,​relatime,​data=ordered)
 +/​dev/​sda3 ​          on /data              type ext4     ​(rw,​relatime,​data=ordered)
 +debugfs ​            on /​sys/​kernel/​debug ​ type debugfs ​ (rw,​noatime)
 +      * ''​df''​ should show free space available on your ''/​overlay''​ and ''/​data''​ partition\\ <​code>​
 +Filesystem ​          ​1K-blocks ​     Used Available Use% Mounted on
 +rootfs ​                  ​27180 ​     7796     ​17964 ​ 30% /
 +/​dev/​root ​                ​2304 ​     2304         0 100% /rom
 +tmpfs                   ​127668 ​     1468    126200 ​  1% /tmp
 +/​dev/​ubi0_1 ​           1998672 ​     8056   ​1869376 ​  0% /overlay
 +overlayfs:/​overlay ​      ​27180 ​     7796     ​17964 ​ 30% /
 +tmpfs                      512         ​0 ​      ​512 ​  0% /dev
 +/​dev/​sda1 ​             1998672 ​     8056   ​1869376 ​  0% /overlay
 +/​dev/​sda3 ​           242846048 ​   163864 230323224 ​  0% /data
 +  - You're done!
 +===== Troubleshooting =====
 +  * Add option ''​force_space''​ in ''/​etc/​opkg.conf''​ to allow installation of packets bigger than your ''/​rom''​ partitions free space: <​code>​echo option force_space >> /​etc/​opkg.conf</​code>​
 +  * Do not use vfat (FAT/​FAT32);​ it does not work. If you have a FAT preformatted USB drive, you cannot use it for extroot without reformatting. Use e.g. ext4 (install e2fsprogs, then format your FAT formatted USB drive using ''​mkfs.ext4 /​dev/​sda1''​ as per the example, also see [[storage]]).
 +  * If the partition containing your extroot isn't mounted during boot, but you can mount it without problems from a shell, you should try to increase ''​config '​global'​ / option delay_root''​. ​ On my system I had to set it to 15 seconds to get extroot working. Another hint to this being the culprit is having a working swap or other partitions mounted after booting, but not your extroot.
 +  * Another possibility to consider and try is to include in ''/​etc/​rc.local''​ the commands: ''​export PREINIT = 1'';''​mount_root'',​ as described in [[https://​​ticket/​14946|14946]] ticket, which in the case of running Chaos Calmer r44266 in the Comtrend AR-5387un, has been the only thing that allowed me to achieve extroot.
 +===== Notes =====
 +==== Using Image Generator ====
 +As the ''​kmod-fs-ext4''​ package is rather large, it can be a challenge setting up extroot on 4MB flash devices, where extroot is most needed.
 +Therefore, you can build an image to flash to your device with the required packages already included, this method saves considerable space on your device.
 +Using the image generator makes building the image very easy. See the [[:​doc:​howto:​obtain.firmware.generate|Image Generator]] page for more detailed instructions.
 +  - Download image builder for your architecture\\ (e.g. ar71xx: [[https://​​chaos_calmer/​15.05/​ar71xx/​generic/​OpenWrt-ImageBuilder-15.05-ar71xx-generic.Linux-x86_64.tar.bz2]])
 +  - Extract archive, change to directory
 +  - To build your firmware (Note: Replace TLMR3220 with the profile for your own device, use ''​make info''​ for a list):\\ <​code>​make image PROFILE="​TLMR3220"​ PACKAGES="​kmod-fs-ext4 kmod-usb-storage"</​code>​
 +Your new firmware file will be located within the ''​bin''​ folder, with the packages to enable extroot already included.
 +==== Remote File Systems ====
 +  * Forum: [[https://​​viewtopic.php?​id=32812|Fstab not mounting cifs at boot or through CLI]]
 +==== System Upgrade ====
 +=== Recommended by extroot maintainer ===
 +I recommend that you DO NOT try to do upgrades using ''​[[doc:​techref:​opkg]] upgrade''​. You will likely end up with an inconsistent state and [[doc:​howto:​generic.debrick|bricked]] router that way:
 +  * The main reason is that the uClibc ABI (Application Binary Interface) is unstable and changes from revision to revision, so binaries for one version of uClibc may be incompatible with versions from another.
 +  * Another problem that can arise is if you try to upgrade the kernel packages, then flash and reboot, but your operation is interrupted in any way, then you will have a kernel and module mismatch and likely a brick.
 +  * Finally, if you upgrade all packages but the kernel and the kernel modules, some packages like ''​iptables''​ will be broken.
 +==== Accessing original root ====
 +Sometimes you may need to access the original root overlay, for example to change your extroot settings. ​ A convenient way of doing this is to configure ''/​etc/​config/​fstab''​ on your extroot partition to mount the original root overlay in another directory, like this:<​code>​
 +config mount
 + option target /​overlay-boot
 + option device /​dev/​mtdblock3
 + option fstype jffs2
 + option options rw,​sync
 + option enabled 1
 + option enabled_fsck 0
 +This assumes the original, internal overlay was on ''/​dev/​mtdblock3''​ - check your router'​s page on this wiki and look at the flash map to confirm what MTD block it is for you. Or run ''​$ cat /​proc/​mtd''​ and search the partition named ''​rootfs_data''​ to know where is your overlay.
 +If you then create ''/​overlay-boot''​ on the extroot partition, this directory will contain the original root overlay, which is used as the main root overlay until the extroot is up and running. ​ So you can then for example edit ''/​overlay-boot/​etc/​config/​fstab''​ to change your extroot configuration (or temporarily disable it) should you ever need to.
 +===== Information on Legacy versions =====
 +For extroot on OpenWrt versions Barrier Breaker, Attitude Adjustment, Backfire see: [[doc:​howto:​extroot.old]]