User Tools

Site Tools


doc:howto:extroot

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
doc:howto:extroot [2013/07/06 01:30]
malvineous Add method for mounting original root overlay within extroot
doc:howto:extroot [2016/09/07 12:25] (current)
noblepepper [Troubleshooting]
Line 1: Line 1:
 +====== Rootfs on External Storage (extroot) ======
 +  * More often than not, a limited amount of storage is available on embedded devices. ​ While flash memory will generally accommodate a bare OpenWrt installation,​ more storage for applications and data can expand a device'​s potential; with many devices having expansion capabilities built-in (i.e. USB, SATA, and PCIe ports, NAS, etc.).  ​
  
 +  * Applications are developed with the idea they should be installed in the root file system (rootfs). ​ By employing //​extroot//,​ expansion of the storage capacity of your root file system is accomplished by using an added storage device. During the boot process, external storage space is mounted as the root file system, or in an overlay configuration over the original file system. To understand the technical details, please read [[doc:​howto:​extroot:​extroot.theory|extroot theory]].
 +
 +===== Alternatives =====
 +  * Configuring the [[doc:​techref:​bootloader|bootloader]] to boot from an external USB drive, of which most can not.
 +    * Popular bootloaders (i.e. U-Boot or RedBoot) [[wp>​Comparison_of_boot_loaders|support different file systems]], but they lack the necessary hardware drivers.
 +  * Configuring the bootloader to boot over the network via [[inbox:​howto:​netboot|netboot]],​ of which most can not.
 +  * Make ''​opkg''​ install new packages [[doc:​techref:​opkg#​Installation destinations|somewhere else]]; however: ​
 +    * Installing packages in root is more convenient, as files are installed in locations the OS expects them to reside.
 +  * Use [[doc:​howto:​kexec|kexec]];​ however, this requires custom configurations on different devices. ​
 +
 +===== Automatic Setup =====
 +
 +  * There'​s a project called [[https://​github.com/​attila-lendvai/​openwrt-auto-extroot|OpenWrt Auto extroot]] that can build a customized firmware image (using the [[doc:​howto:​imagegenerator|Image Generator]]). ​
 +    * When this image is flashed, it will automatically set up extroot on any inserted storage device and then reboot (hopefully into the freshly set up extroot).
 +    * It's a nice foundation for custom applications that require auto provisioning,​ or just for your convenience at home for an easy extroot setup.
 +
 +  * This is an ImageBuilder example for building very small images (compatible with 4 MB devices) with only the required packages for manual extroot setup. Please mind that there is no LuCI installed. Once extroot is performed, you can install LuCI and all the packages you require on top of the newly created filesystem.\\ <​code>​make image PROFILE=TLMR3020 PACKAGES="​blkid block-mount kmod-fs-ext4 kmod-usb2 kmod-usb-uhci kmod-usb-ohci kmod-usb-storage"</​code>​
 +
 +==== Prerequisites ====
 +
 +You need to **mount a file system**, on which the overlay will be copied to.
 +  * For USB support, follow [[doc:​howto:​usb.essentials]]
 +  * For USB storage support, follow [[doc:​howto:​usb.storage]]. See [[doc:​howto:​storage]] for general information on partitioning,​ formatting and mounting storage devices.
 +  * Alternatively,​ follow [[doc:​howto:​client.overview#​Mounting Filesystems]] to mount a remote network file system
 +
 +
 +Make sure that you can mount and have read/write access to your external storage device. For example, check if you can read/write from a manually mounted file system on your partitioned USB drive at ''/​dev/​sda''​ (adapt ''​sda''​ to your actual needs):
 +
 +    mount /dev/sda1 /mnt/sda1
 +
 +You should check if the mount was succesful by running ''​mount''​ or ''​df''​. Now check if you can write to ''/​mnt/​sda1''​ by writing a file to it and reading it back. 
 +<​code>​
 +root@OpenWrt:​~#​ df
 +Filesystem ​          ​1K-blocks ​     Used Available Use% Mounted on
 +rootfs ​                    ​896 ​      ​244 ​      ​652 ​ 27% /
 +/​dev/​root ​                ​2048 ​     2048         0 100% /rom
 +tmpfs                    14708        64     ​14644 ​  0% /tmp
 +/​dev/​mtdblock6 ​            ​896 ​      ​244 ​      ​652 ​ 27% /overlay
 +overlayfs:/​overlay ​        ​896 ​      ​244 ​      ​652 ​ 27% /
 +tmpfs                      512         ​0 ​      ​512 ​  0% /dev
 +/​dev/​sda1 ​             7759872 ​   477328 ​  ​7221104 ​  6% /mnt/sda1
 +root@OpenWrt:​~# ​
 +</​code>​
 +==== Chaos Calmer ====
 +
 +As described in [[doc:​howto:​extroot:​extroot.theory|extroot theory]] you can use pivot-overlay or pivot-root. This section will only cover pivot-overlay which is the recommended extroot implementation. If you really want pivot-root, check the [[doc:​howto:​extroot.old#​Barrier Breaker|Barrier Breaker section]].
 +
 +=== Prerequisites ===
 +  * //​block-mount//​
 +  * //​kmod-fs-ext4//​ or //​kmod-fs-//​[filesystem of choice] ​
 +  * //​kmod-usb-storage-extras//​ <​code>​opkg update ; opkg install block-mount kmod-fs-ext4 kmod-usb-storage-extras</​code>  ​
 +  * If opkg errors when installing kmod-usb-storage-extras,​ you might need to install <​code>​kmod-usb-core</​code> ​
 +
 +=== Steps ===
 +  - Format and partition drive the way you want, either using a Linux Live CD or via uci //​(it'​s recommended to use first partition for the root overlay, second partition for swap (if you want swap), and then you can add other partitions as you like)//
 +  - Prepare your external storage root overlay<​code>​
 +mount /dev/sda1 /mnt ; tar -C /overlay -cvf - . | tar -C /mnt -xf - ; umount /mnt
 +</​code>​
 +  - Create fstab with the following command<​code>​
 +block detect > /​etc/​config/​fstab;​ \
 +   sed -i s/​option$'​\t'​enabled$'​\t'​\'​0\'/​option$'​\t'​enabled$'​\t'​\'​1\'/​ /​etc/​config/​fstab;​ \
 +   sed -i s#/​mnt/​sda1#/​overlay#​ /​etc/​config/​fstab;​ \
 +   cat /​etc/​config/​fstab;​
 +</​code>​
 +This command will create a fstab template enabling all partitions and setting '​**''/​mnt/​sda1''​**'​ partition as '​**''/​overlay''​**'​ partition .
 +  - See the following in ''/​etc/​config/​fstab'':​
 +    - Check if all '​**''​option enabled''​**'​ is set to '​**''​1''​**'​
 +    - In '​**''​option target''​**'​ of your root overlay partition must be set to '​**''/​overlay''​**'​
 +    - You can edit '​**''​option target''​**'​ of your other partition(s) to show whatever mount point(s) you want them to have (make sure to also create the directories those mount points point to)\\ 
 +    <​code>​ vi /​etc/​config/​fstab</​code>​
 +    - You'll end up with an fstab looking something like this:<​code>​
 +config '​global'​
 +        option ​ anon_swap ​      '​0'​
 +        option ​ anon_mount ​     '​0'​
 +        option ​ auto_swap ​      '​1'​
 +        option ​ auto_mount ​     '​1'​
 +        option ​ delay_root ​     '​5'​
 +        option ​ check_fs ​       '​0'​
 +
 +config '​mount'​
 +        option ​ target ​ '/​overlay'​
 +        option ​ uuid    '​c91232a0-c50a-4eae-adb9-14b4d3ce3de1'​
 +        option ​ fstype ​ '​ext4'​
 +        option ​ enabled '​1'​
 +
 +config '​swap'​
 +        option ​ uuid    '​08b4f0a3-f7ab-4ee1-bde9-55fc2481f355'​
 +        option ​ enabled '​1'​
 +
 +config '​mount'​
 +        option ​ target ​ '/​data'​
 +        option ​ uuid    '​c1068d91-863b-42e2-bcb2-b35a241b0fe2'​
 +        option ​ enabled '​1'​
 +</​code>​
 +  - Check if it is mountable to overlay:<​code>​
 +root@OpenWrt:​~#​ mount /dev/sda1 /overlay
 +root@OpenWrt:​~#​ df
 +Filesystem ​          ​1K-blocks ​     Used Available Use% Mounted on
 +rootfs ​                    ​896 ​      ​244 ​      ​652 ​ 27% /
 +/​dev/​root ​                ​2048 ​     2048         0 100% /rom
 +tmpfs                    14708        64     ​14644 ​  0% /tmp
 +/​dev/​mtdblock6 ​        ​7759872 ​   477328 ​  ​7221104 ​  6% /overlay
 +overlayfs:/​overlay ​        ​896 ​      ​244 ​      ​652 ​ 27% /
 +tmpfs                      512         ​0 ​      ​512 ​  0% /dev
 +/​dev/​sda1 ​             7759872 ​   477328 ​  ​7221104 ​  6% /overlay
 +root@OpenWrt:​~#</​code>​ Note that only ''/​overlay''​ has grown but not the ''/''​
 +  - Reboot the router
 +  - Verify that the partitions were mounted properly:
 +    * via LuCI
 +      * //System - Software// should show free space of overlay partition
 +      * //System - Mount Points// should show USB partition mounted as ''​overlay''​
 +    * via CLI
 +      * ''​mount''​ should show USB partition mounted as ''/​overlay''​\\ <​code>​
 +root@OpenWrt:​~#​ mount
 +/​dev/​root ​          on /rom               type squashfs (ro,​relatime)
 +proc                on /proc              type proc     ​(rw,​noatime)
 +sysfs               on /sys               type sysfs    (rw,​noatime)
 +tmpfs               on /tmp               type tmpfs    (rw,​nosuid,​nodev,​noatime)
 +/​dev/​mtdblock6 ​     on /​overlay ​          type jffs2    (rw,​noatime)
 +overlayfs:/​overlay ​ on /                  type overlay ​ (rw,​noatime,​lowerdir=/,​upperdir=/​overlay/​upper,​workdir=/​overlay/​work)
 +tmpfs               on /dev               type tmpfs    (rw,​relatime,​size=512k,​mode=755)
 +devpts ​             on /​dev/​pts ​          type devpts ​  ​(rw,​relatime,​mode=600)
 +/​dev/​sda1 ​          on /​overlay ​          type ext4     ​(rw,​relatime,​data=ordered)
 +/​dev/​sda3 ​          on /data              type ext4     ​(rw,​relatime,​data=ordered)
 +debugfs ​            on /​sys/​kernel/​debug ​ type debugfs ​ (rw,​noatime)
 +root@OpenWrt:​~#​
 +</​code>​
 +      * ''​df''​ should show free space available on your ''/​overlay''​ and ''/''​ partition, all the storages mounted to ''/​overlay''​ and ''/''​ (the ''​rootfs''​ in the first place) should look the same increased capacity:\\ <​code>​
 +root@OpenWrt:​~#​ df
 +Filesystem ​          ​1K-blocks ​     Used Available Use% Mounted on
 +rootfs ​                ​7759872 ​   477328 ​  ​7221104 ​  6% /
 +/​dev/​root ​                ​2048 ​     2048         0 100% /rom
 +tmpfs                    14708        64     ​14644 ​  0% /tmp
 +/​dev/​mtdblock6 ​        ​7759872 ​   477328 ​  ​7221104 ​  6% /overlay
 +overlayfs:/​overlay ​    ​7759872 ​   477328 ​  ​7221104 ​  6% /
 +tmpfs                      512         ​0 ​      ​512 ​  0% /dev
 +/​dev/​sda1 ​             7759872 ​   477328 ​  ​7221104 ​  6% /overlay
 +/​dev/​sda3 ​           242846048 ​   163864 230323224 ​  0% /data
 +root@OpenWrt:​~#​
 +</​code>​
 +  - You're done!
 +===== Troubleshooting =====
 +  * Add option ''​force_space''​ in ''/​etc/​opkg.conf''​ to allow installation of packets bigger than your FIXME ''/​rom''​ partitions free space: <​code>​echo option force_space >> /​etc/​opkg.conf</​code>​
 +  * Do not use vfat (FAT/​FAT32);​ it does not work. If you have a FAT preformatted USB drive, you cannot use it for extroot without reformatting. Use e.g. ext4 (install e2fsprogs, then format your FAT formatted USB drive using ''​mkfs.ext4 /​dev/​sda1''​ as per the example, also see [[storage]]).
 +  * If the partition containing your extroot isn't mounted during boot, but you can mount it without problems from a shell, you should try to increase ''​config '​global'​ / option delay_root''​. ​ On my system I had to set it to 15 seconds to get extroot working. Another hint to this being the culprit is having a working swap or other partitions mounted after booting, but not your extroot.
 +  * Another possibility to consider and try is to include in ''/​etc/​rc.local''​ the commands: ''​export PREINIT = 1'';''​mount_root'',​ as described in [[https://​dev.openwrt.org/​ticket/​14946|14946]] ticket, which in the case of running Chaos Calmer r44266 in the Comtrend AR-5387un, has been the only thing that allowed me to achieve extroot.
 +  * If after successful extroot mount, swap isn't enabled and other partitions aren't mounted, check your ''/​etc/​config/​fstab''​ in the overlay partition. If your ''/​etc/​config/​fstab''​ in the overlay partition only contains a ''​global''​ section, you may need to add the following from the example:\\ <​code>​
 +config '​swap'​
 +        option ​ uuid    '​08b4f0a3-f7ab-4ee1-bde9-55fc2481f355'​
 +        option ​ enabled '​1'​
 +
 +config '​mount'​
 +        option ​ target ​ '/​data'​
 +        option ​ uuid    '​c1068d91-863b-42e2-bcb2-b35a241b0fe2'​
 +        option ​ enabled '​1'​
 +</​code>​
 +  * If you are putting the extroot on a non-USB device such as a mmc card all modules needed acccess the device should be in appropriate file in /​etc/​modules-boot.d. For example using a sdhci card on a mt7688/​mt7628 device /​etc/​modules-boot.d/​mmc needs have two lines added:<​code>​
 +mmc_core
 +mmc_block
 +sdhci
 +mtk_sd
 +</​code>​
 +===== Notes =====
 +
 +==== Using Image Generator ====
 +As the ''​kmod-fs-ext4''​ package is rather large, it can be a challenge setting up extroot on 4MB flash devices, where extroot is most needed.
 +Therefore, you can build an image to flash to your device with the required packages already included, this method saves considerable space on your device.
 +
 +Using the image generator makes building the image very easy. See the [[:​doc:​howto:​obtain.firmware.generate|Image Generator]] page for more detailed instructions.
 +
 +  - Download image builder for your architecture\\ (e.g. ar71xx: [[https://​downloads.openwrt.org/​chaos_calmer/​15.05/​ar71xx/​generic/​OpenWrt-ImageBuilder-15.05-ar71xx-generic.Linux-x86_64.tar.bz2]])
 +  - Extract archive, change to directory
 +  - To build your firmware (Note: Replace TLMR3220 with the profile for your own device, use ''​make info''​ for a list):\\ <​code>​make image PROFILE="​TLMR3220"​ PACKAGES="​kmod-fs-ext4 kmod-usb-storage"</​code>​
 +
 +Your new firmware file will be located within the ''​bin''​ folder, with the packages to enable extroot already included.
 +
 +==== Extroot to a card in a slot of an USB-dongle ====
 +The [[doc:​recipes:​3gdongle|Use 3g/UMTS USB Dongle for WAN connection]] article advises to include the ''​usb-modeswitch''​ tool in the image.\\ There is a caveat: if the ''/​overlay''​ points to a memory card sitting in a slot of the dongle - the otherwise working ''​pivot overlay''​ set-up will break in the later stages of OS boot. This is because the ''​usb-modeswitch''​ (while disabling the CDROM and enabling the modem) would also intermittently affect the card-reader in the dongle thus hurting the file system.\\
 +To avoid this you need a dongle that can be pre-configured to enable its' modem or network adapter (and the card-reader as well) on the power-up, without the need to do it with the ''​usb-modeswitch''​ on the router.
 +
 +Insert your dongle in a desktop and use a terminal to send the necessary AT-commands.\\
 +Check your dongle'​s initial configuration:​\\
 +<​code>​
 +at^setport?
 +^SETPORT:​A1,​A2;​1,​3,​2,​A1,​A2
 +OK
 +</​code>​
 +The meaning of the above report can be understood with the following command:\\
 +<​code>​
 +at^setport=?​
 +^SETPORT:​A1:​ CDROM
 +^SETPORT:​A2:​ SD
 +^SETPORT:A: BLUE TOOTH
 +^SETPORT:B: FINGER PRINT
 +^SETPORT:D: MMS
 +^SETPORT:E: PC VOICE
 +^SETPORT:1: MODEM
 +^SETPORT:2: PCUI
 +^SETPORT:3: DIAG
 +^SETPORT:4: PCSC
 +^SETPORT:5: GPS
 +^SETPORT:6: GPS CONTROL
 +^SETPORT:​16:​ NCM
 +OK
 +</​code>​
 +So, in the example above we have a dongle with CDROM and card-reader available in the first configuration (to the left of the '';''​ character), and with modem, control and diagnostic interfaces, and card-reader available in the other configuration. It is between these configurations the ''​usb-modeswitch''​ switches the dongle on the router.\\
 +\\
 +Your goal is to disable the CDROM and enable the modem (the ''​1''​ above) or the network adapter (the ''​16''​ above) while leaving the card-reader enabled (the ''​A2''​ above).\\
 +**NOTE: Never disable the PCUI** (the ''​2''​ above) - this will lock you out from your dongle!\\
 +\\
 +Some dongles accept a '​disable all' operand (the ''​FF''​ below).\\
 +Place the list of all the functions you need on your dongle by default to the right of the '';''​ character according to their codes from the dongle'​s answer above:\\
 +<​code>​
 +at^setport="​ff;​1,​2,​3,​a2"​
 +OK
 +
 +at^reset
 +OK
 +
 +at^setport?
 +^SETPORT:;​1,​2,​3,​A2
 +OK
 +</​code>​
 +This sequence has disabled the CDROM and made the modem, control and diagnostic interfaces and the card-reader available by default - without any ''​usb-modeswitch''​ interaction. Thus only one configuration exists now in the dongle - see the '';''​ character, there is nothing to the left of it now.
 +
 +=== Dongles known to support the pre-configuration ===
 +  * Huawei E3131s-2 f/w v21.158.47.00.1094
 +==== Remote File Systems ====
 +  * Forum: [[https://​forum.openwrt.org/​viewtopic.php?​id=32812|Fstab not mounting cifs at boot or through CLI]]
 +
 +==== System Upgrade ====
 +=== Recommended by extroot maintainer ===
 +I recommend that you DO NOT try to do upgrades using ''​[[doc:​techref:​opkg]] upgrade''​. You will likely end up with an inconsistent state and [[doc:​howto:​generic.debrick|bricked]] router that way:
 +  * The main reason is that the uClibc ABI (Application Binary Interface) is unstable and changes from revision to revision, so binaries for one version of uClibc may be incompatible with versions from another.
 +  * Another problem that can arise is if you try to upgrade the kernel packages, then flash and reboot, but your operation is interrupted in any way, then you will have a kernel and module mismatch and likely a brick.
 +  * Finally, if you upgrade all packages but the kernel and the kernel modules, some packages like ''​iptables''​ will be broken.
 +
 +==== Accessing original root ====
 +
 +Sometimes you may need to access the original root overlay, for example to change your extroot settings. ​ A convenient way of doing this is to configure ''/​etc/​config/​fstab''​ on your extroot partition to mount the original root overlay in another directory, like this:<​code>​
 +config mount
 + option target /​overlay-boot
 + option device /​dev/​mtdblock3
 + option fstype jffs2
 + option options rw,​sync
 + option enabled 1
 + option enabled_fsck 0
 +</​code>​
 +This assumes the original, internal overlay was on ''/​dev/​mtdblock3''​ - check your router'​s page on this wiki and look at the flash map to confirm what MTD block it is for you. Or run ''​$ cat /​proc/​mtd''​ and search the partition named ''​rootfs_data''​ to know where is your overlay.
 +
 +If you then create ''/​overlay-boot''​ on the extroot partition, this directory will contain the original root overlay, which is used as the main root overlay until the extroot is up and running. ​ So you can then for example edit ''/​overlay-boot/​etc/​config/​fstab''​ to change your extroot configuration (or temporarily disable it) should you ever need to.
 +
 +===== Information on Legacy versions =====
 +
 +For extroot on OpenWrt versions Barrier Breaker, Attitude Adjustment, Backfire see: [[doc:​howto:​extroot.old]]
 +
 +{{tag>​USBrelated}}