User Tools

Site Tools


doc:techref:flash.layout

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
doc:techref:flash.layout [2013/02/06 17:20]
buntalo
doc:techref:flash.layout [2015/12/23 14:46] (current)
floyd42 [broadcom with CFE] capitaize section name
Line 1: Line 1:
 +====== The OpenWrt Flash Layout ======
  
 +Most routers do not have hard drives. ​ They use flash memory for similar purposes: storing programs and data, even when the system is off (non-volatile memory).
 +
 +In most systems, flash memory does not appear like RAM and so data and instructions must be copied to RAM to be used.  So, for example, the bootloader copies the kernel from flash to RAM and then starts that copy running.
 +
 +There are two basic types of flash memory: [[wp>​Flash_memory#​NOR_flash|NOR flash]] and [[wp>​Flash_memory#​NAND_flash|NAND flash]].\\
 +If the flash chip (no matter what type) is connected directly with the [[wp>​System-on-a-chip|SoC]] and has to be addressed directly by Linux, we call it **//"​raw flash"//​**.
 +If there is an additional controller chip between flash chip and the SoC, we call it **//"​FTL (Flash Translation Layer) flash"//​**. Embedded systems almost exclusively use "raw flash",​ while SSDs and also USB memory sticks, almost exclusively use FTL flash!
 +
 +Older routers generally have raw NOR flash but many newer routers have raw NAND flash.
 +
 +Raw NOR flash in typical routers is generally small (4 MiB - 16 MiB) and error-free. ​ This is currently the normal OpenWRT setup. ​ Because it is error-free, the file systems are a little simpler. ​ But they still need to do wear-levelling. ​ Conserving space is important.
 +
 +Raw NAND flash in typical routers is generally larger (32 MiB - 256 MiB) and may contain bad blocks. ​ OpenWRT isn't configured to handle this yet.  One promising approach would be to use [[https://​en.wikipedia.org/​wiki/​UBIFS|UBIFS]] layered on top of UBI.
 +
 +===== Partitioning of the Flash =====
 +Almost all embedded systems contain //"raw flash"//​-chips. OpenWrt treats and addresses this storage space as an [[doc:​techref:​MTD| MTD (Memory Technology Device)]] and employs [[doc:​techref:​filesystems]] developed for this purpose. The available storage is not partitioned in the traditional way, where you store the data about the partitions in the [[wp>​Master boot record|MBR]] and [[wp>​Volume boot record|PBR]]s,​ but it is done in the Linux Kernel (and sometimes independently in the [[doc:​techref:​bootloader]] as well!). It's simply defined, that "//​partition **''​kernel''​** starts at offset ''​x''​ and ends at offset ''​y''//"​. Using names allows convenient addressing of partitions by name instead of giving the start offset over and over again.
 +
 +The generic Flash layout is:
 +^ Layer0 |  raw flash  ||||||
 +^ Layer1 |  bootloader \\ partition(s) ​ |  optional \\ SoC \\ specific \\ partition(s) ​ |  OpenWrt firmware partition ​ |||  optional \\ SoC \\ specific \\ partition(s) ​ |
 +^ Layer2 |:::​|:::​| ​ Linux Kernel ​ |  **''​rootfs''​** \\ mounted: "''/''",​ [[doc:​techref:​filesystems#​overlayfs|OverlayFS]] with ''/​overlay'' ​ ||:::|
 +^ Layer3 |:::​|:::​|:::​| ​ **''/​dev/​root''​** \\ mounted: "''/​rom''",​ [[doc:​techref:​filesystems#​SquashFS|SquashFS]] \\ size depends on selected packages ​ |  **''​rootfs_data''​** \\ mounted: "''/​overlay''",​ [[doc:​techref:​filesystems#​SquashFS|JFFS2]] \\ "​free"​ space  |:::|
 +
 +Many newer devices share this scheme, details slightly differ concerning U-Boot and SoC specific firmware images. Please see the wiki pages for each SoC and devices for information about a particular layout. In case the flash layout differs for your device please update the wiki pages.\\
 +Here are some examples how it looks on actual devices:
 +==== Examples ==== 
 +=== Example 1: TP-Link TL-WR1043ND ===
 +[[doc:​hardware:​soc:​soc.qualcomm|Qualcomm Atheros]] based [[toh:​tp-link:​TL-WR1043ND]]. Note the "''​art''​ partition with the SoC specific Wifi data. There is also a [[https://​web.archive.org/​web/​20131021013058/​http://​ubuntuone.com/​2aPBH9pwkxtYzy93S0cS1z|LibreOffice Calc ODS]] that allows "​playing"​ with the numbers. ​
 + \\
 +^ Layer0 |  raw flash, 8192 KiB  ||||||
 +^ Layer1 |  **mtd0** \\ ''​u-boot''​ \\ 128 KiB  ||  **mtd5** \\ ''​firmware''​ \\ 8000  KiB <​sub><​sup>​(= FlashSize-(128+64))</​sup></​sub> ​ |||  **mtd4** \\ ''​art''​ \\ 64 KiB  |
 +^ Layer2 |  code \\ 64 KiB  |  data \\ 64 KiB  |  **mtd1** \\ ''​kernel''​ \\ about 1.2 MiB  |  **mtd2** \\ ''​rootfs'' ​ ||:::|
 +^ Layer3 |:::​|:::​|:::​| ​ **''/​dev/​root''​** \\ around 1.5 MiB  |  **mtd3** \\ ''​rootfs_data''​ \\ around 5 MiB  |:::|
 +
 +=== Example 2: Hoo Too HT-TM02 ===
 +[[doc:​hardware:​soc:​soc.ralink|Ralink RT5350F]] based [[toh:​hwdata:​hootoo:​hootoo_tripmatenano_v15|Hoo Too HT-TM02]].
 +
 +^ Layer0 |  raw flash, 8192 KiB  ||||||
 +^ Layer1 |  **mtd0** \\ ''​u-boot''​ \\ 192 KiB  |  **mtd1** \\ ''​u-boot-env''​ \\ 64 KiB  |  **mtd2** \\ ''​factory''​ \\ 64 KiB  |  **mtd3** \\ ''​firmware''​ \\ 7872 KiB <​sub><​sup>​(= FlashSize-(192+64+64))</​sup></​sub> ​ |||
 +^ Layer2 |:::​|:::​|:::​| ​ **mtd4** \\ ''​kernel''​ \\ about 1 MiB  |  **mtd5** \\ ''​rootfs'' ​ ||
 +^ Layer3 |:::​|:::​|:::​|::: ​                      ​| ​ **''/​dev/​root''​** \\ around 2 MiB  |  **mtd6** \\ ''​rootfs_data''​ \\ around 4.5 MiB  |
 +
 +=== Example 3: TP-Link DIR-300 ===
 +
 +For some devices, the OpenWrt partition ''​firmware''​ may not exist at. The [[toh:​d-link:​DIR-300#​flash.layout|DIR-300 flash layout]] is an example, see the explanation there for details
 +
 +==== Partitions ====
 +Since the partitions are nested we look at this whole thing in layers:
 +  - **Layer0**: We have the Flashchip, e.g. 8MiB in size, which is soldered to the PCB and connected to the [[doc:​hardware:​SoC]] over e.g. the [[wp>​Serial Peripheral Interface Bus|SPI (Serial Peripheral Interface Bus)]].
 +  - **Layer1**: We partition the flash space into:
 +    - one or more partition for the bootloader. A U-Boot partition usually consists of 64 KiB u-boot block and a following 64 KiB data block section which contains things like MAC, WPS-PIN, type description.... If no MAC is configured, Wifi will not work correctly.
 +    - a partiton for the OpenWrt firmware
 +    - one or more partition for SoC specific firmware, e.g. "​art"​ for [[doc:​hardware:​soc:​soc.qualcomm|Qualcomm Atheros]]
 +  - **Layer2**: we subdivide the "​firmware"​ into:
 +    - "​kernel"​ at the start. In the generation process of the firmware (see [[doc:​howto:​obtain.firmware.generate]]) the Kernel binary file is first packed with [[wp>​Lempel–Ziv–Markov chain algorithm|LZMA]],​ then packed with [[wp>​gzip]]. The kernel image is written onto the raw flash, it's not part of any filesystem!
 +    - "​rootfs"​ follow, it contains the file system ​
 +  - Layer3: we subdivide "​rootfs"​ further into:
 +    - fixed ROM data using SquashFS
 +    - "​rootfs_data"​ using JFFS2. This the the free space on the device that can be used
 +
 +==== Mount Points ====
 +  * "''/''"​ is your entire root filesystem, it comprises "''/​rom''"​ and "''/​overlay''"​. Please ignore "''/​rom''"​ and "''/​overlay''"​ and use exclusively "''/''"​ for your daily routines.
 +  * "''/​rom''"​ contains all the basic files, like ''​busybox'',​ ''​dropbear''​ or ''​iptables''​. It also includes default configuration files used when booting into [[doc:​howto:​generic.failsafe|OpenWrt Failsafe mode]]. It does not contain the Linux kernel. All files in this directory are located on the SqashFS partition, and thus cannot be altered or deleted. But, because we use OverlayFS filesystem, so called //​overlay-whiteout//​-symlinks can be created on the JFFS2 partition.
 +  * "''/​overlay''"​ is the writable part of the file system that gets merged with ''/​rom''​ to create a uniform ''/''​-tree. It contains anything that was written to the router after [[doc:​howto:​generic.flashing|installation]],​ e.g. changed configuration files, additional packages installed with [[doc:​techref:​opkg|OPKG]],​ etc. It is formated with JFFS2.
 +
 +Whenever the system is asked to look for an existing file in "''/''",​ it first looks in "''/​overlay''",​ and if not there, then in "''/​rom''"​. ​ In this way "''/​overlay''"​ overrides "''/​rom''"​ and creates the effect of a writable "''/''"​ while much of the content is safely and efficiently stored in the read-only "''/​rom''"​.
 +
 +When the system is asked to delete a file that is in ''/​rom'',​ it instead creates a corresponding entry in ''/​overlay'',​ a whiteout. ​ A whiteout is a symlink to ''​(overlay-whiteout)''​ that mostly behaves like a file that doesn'​t exist.
 +
 +<code bash>
 +#!/bin/sh
 +# shows all overlay-whiteout symlinks
 +
 +find /overlay -type l -exec sh -c \
 +    'for x; do [ "​$(readlink -n -- "​$x"​)"​ = "​(overlay-whiteout)"​ ] && printf %s\\n "​$x";​ done' -- {} +
 +</​code>​
 +
 +==== Directory Structure and Filesystem Hierarchy ====
 +**//''​NOTE1''//:​** If the Kernel was part of the SquashFS, we could NOT control where exactly on the flash the kernel is written (namely, on which blocks the data is written to). Thus we can not tell the bootloader to simply load and execute certain blocks on the flash storage. Now this would not be too bad, but in order for that to happen the bootloader would have to understand the SquashFS filesystem; which it does not. The embedded bootloaders we utilize with OpenWrt generally has no concept of filesystems,​ and thus they cannot address files by path and filename. The bootloaders pretty much assume that the start of the trx data section is executable code.\\
 +**//''​NOTE2''//:​** ​ Generally,​the term ''"​firmware"''​ is used for the all data stored on the flash -comprising the boot loader and any other data necessary to operate the device (such as ART, NVRAM, FIS, etc), but it is also use to name only the parts that are being rewritten. Don't let this confuse you ;-)\\
 +
 +===== Partitioning of JFFS2-Images =====
 +TODO
 +
 +===== Discovery =====
 +
 +==== Kernel ====
 +The kernel knows the flash layout, because it is hard coded somewhere, e.g:
 +  * in the ''​mtd''​ driver. See for example [[https://​dev.openwrt.org/​browser/​trunk/​target/​linux/​ar71xx/​files/​drivers/​mtd/​tplinkpart.c?​rev=46662#​L149|here]]
 +  * in the [[wp>​device_tree|device tree]], see for example: [[https://​dev.openwrt.org/​browser/​trunk/​target/​linux/​ramips/​dts/​HT-TM02.dts?​rev=47935#​L14|here]]
 +
 +==== User ====
 +Look at the kerne'​s boot lor (or run ''​dmesg''​ after a fresh boot) and look for something like:
 +<​code>​
 +Creating 5 MTD partitions on "​spi0.0":​
 +0x000000000000-0x000000020000 : "​u-boot"​
 +0x000000020000-0x000000160000 : "​kernel"​
 +0x000000160000-0x0000007f0000 : "​rootfs"​
 +mtd: partition "​rootfs"​ set to be root filesystem
 +mtd: partition "​rootfs_data"​ created automatically,​ ofs=2C0000, len=530000
 +0x0000002c0000-0x0000007f0000 : "​rootfs_data"​
 +0x0000007f0000-0x000000800000 : "​art"​
 +0x000000020000-0x0000007f0000 : "​firmware"​
 +</​code>​
 +These are the start and end offsets of the partitions as hex values in Bytes. Now you don't have to guess which is nested in which. E.g. 02 0000 = 131.072 Bytes = 128KiB.\\
 +\\
 +Runnig ''​cat /​proc/​mtd''​ will also give a list of partitions:
 +<​code>​
 +dev:    size   ​erasesize ​ name
 +mtd0: 00020000 00010000 "​u-boot"​
 +mtd1: 00140000 00010000 "​kernel"​
 +mtd2: 00690000 00010000 "​rootfs"​
 +mtd3: 00530000 00010000 "​rootfs_data"​
 +mtd4: 00010000 00010000 "​art"​
 +mtd5: 007d0000 00010000 "​firmware"​
 +</​code>​
 +
 +The column ''​erasesize''​ is the [[wp>​Block (data storage)|block size]] of the flash, in this case it's 64KiB (= ''​0x10000''​). The column ''​size''​ gives the size as hex value. Unfortunately,​ this does not show if the partitions are nested. This can be obtained form the kernel boot log only - or from "//​educated guessing//"​.\\
 +Note that on some platforms (FIXME: which :?: Examples :?:) give the column data as [[wp>​Endianess|little endian]] integer, ie not in the natural ''​aabbccdd''​ but reversed ''​ddccbbaa''​.\\
 +
 +===== SoC specific details =====
 +==== Generic ====
 +On power up the SOC's CPU begins executing the code from  boot ROM. It contains code to boot from various sources:
 +  * Flash
 +  * USB
 +  * SD-Card
 +  * Network
 +  * ...
 +Usually, the boot ROM code (1st stage boot loader) is very limited. Sophisticated boot options like network boot are usually not included there. It just contains enough code to load the next boot stage (2nd stage boot loader) from a storage device (Flash, SD-Card, USB). \\
 +\\
 +The 2nd boot loader isn't part of the OpenWrt firmware. This greatly improves the stability of the device, as failing to flash a proper version of OpenWrt will not brick it. The 2nd stage bootloader can still be used to flash a new image - it's just a bit more effort and usually less convenient.\\
 +\\
 +Note that this boot approach is similar to the PC. There is ROM code at the 1st stage, the BIOS at the 2nd stage and then the OS bootloader from hard disk as 3rd state. Also, the BIOS can boot from USB, Floppy, CR-ROM, Network ... in case the OS on the hard disk is broken.\\
 +
 +The bootloader is part of the Flash, e.g.:
 +| ^ Boot Loader Partition ^ Firmware Partition ^ Special Configuration Data ^^^
 +^ Atheros ​ |  [[doc:​techref:​bootloader:​uboot|U-Boot]] ​  ​| ​ OpenWrt ​ |  ''​ART'' ​        |||
 +^ Atheros ​ |  RedBoot ​ |  OpenWrt ​ |  ''​FIS recovery'' ​ |  ''​RedBoot config'' ​ |  ''​boardconfig'' ​ |
 +^ Broadcom |  CFE      |  OpenWrt ​ |  ''​NVRAM'' ​      |||
 +
 +The partition or partitions containing so called //Special Configuration Data// differ very much from each other. Examples: ​
 +* the ''​ART''​-partition exists on  Qualcomm-Atheros based devices and U-Boot is the 2nd stage boot loader.\\
 +* the ''​NVRAM''​-partition of Broadcom based devices is used for much more. There are special utilities (like the "nvram utility"​) to access and modify special configuration partitions. To find out what is written in ''​NVRAM''​ you can run ''​nvram show''​.\\
 +\\
 +Clearing these special configuration data partitions like ''​ART,​ NVRAM''​ and ''​FIS''​ does not clear much of OpenWrt'​s configuration,​ unlike other router software which keep configuration data solely in e.g. ''​NVRAM''​. Instead, as a consequence of using the OverlayFS filesystem configuration with JFFS2 flash partition, the whole file system is writable and allows the flexibility of extending your OpenWrt installation in any way you want. OpenWrt'​s main configuration is therefore just kept in the root file system, using [[doc/​uci|UCI]] configuration files. For convenience,​ many other packages are made UCI compatible. If you want to reset your complete installation you should use OpenWrt'​s built-in functionality such as [[doc/​howto/​generic.sysupgrade|sysupgrade]] to restore settings, by clearing the JFFS2 partition. Or, if you cannot boot normally, you can wipe or change the JFFS2 partition using OpenWrt'​s [[doc/​howto/​generic.failsafe|failsafe mode]] (look in your device'​s dedicated page for information how to boot into failsafe). ​
 +
 +
 +==== Broadcom with CFE ====
 +
 +If you dig into the "​firmware"​ section you'll find a trx. A trx is just an encapsulation,​ which looks something like this:
 +
 +^ trx-header ^^^^^^^
 +| HDR0  |  length ​ |  crc32  |  flags  |  pointers ​ |  data  |
 +
 +"​HDR0"​ is a magic value to indicate a trx header, rest is 4 byte unsigned values followed by the actual contents. In short, it's a block of data with a length and a checksum. So, our flash usage actually looks something like this:
 +
 +|  CFE  |  trx containing firmware ​ |  NVRAM  |
 +
 +Except that the firmware is generally pretty small and doesn'​t use the entire space between CFE and NVRAM:
 +|  CFE  |  trx firmware ​ |  unused ​ |  NVRAM  |
 +
 +(**//''​NOTE''//:​** The <​model>​.bin files are nothing more than the generic *.trx file with an additional header appended to the start to identify the model. The model information gets verified by the vendor'​s upgrade utilities and only the remaining data -- the trx -- gets written to the flash. When upgrading from within OpenWrt remember to use the *.trx file.)
 +
 +
 +So what exactly is the firmware?
 +
 +The boot loader really has no concept of filesystems,​ it pretty much assumes that the start of the trx data section is executable code. So, at the very start of our firmware is the kernel. But just putting a kernel directly onto flash is quite boring and consumes a lot of space, so we compress the kernel with a heavy compression known as [[wp>​Lempel–Ziv–Markov chain algorithm|LZMA]]. Now the start of firmware is code for an LZMA decompress:
 +
 +|  lzma decompress ​ |  lzma compressed kernel ​ |
 +
 +Now, the boot loader boots into an LZMA program which decompresses the kernel into RAM and executes it. It adds one second to the bootup time, but it saves a large chunk of flash space. (And if that wasn't amusing enough, it turns out the boot loader does know gzip compression,​ so we gzip compressed the LZMA decompression program)
 +
 +Immediately following the kernel is the filesystem. We use SquashFS for this because it's a highly compressed readonly filesystem -- remember that altering the contents of the trx in any way would invalidate the crc, so we put our writable data in a JFFS2 partition, which is outside the trx. This means that our firmware looks like this:
 +
 +|  trx  | gzip'd lzma decompress ​ |  lzma'd kernel ​ |  (SquashFS filesystem) ​ |
 +And the entire flash usage looks like this -
 +
 +|  CFE  |  trx  |  gz'd lzma  |  lzma'd kernel ​ |  SquashFS ​ |  JFFS2 filesystem ​ |  NVRAM  |
 +That's about as tight as we can possibly pack things into flash.
 +
 +----
 +
 +===== Explanations =====
 +==== What is an Image File? ====
 +An image file is byte by byte copy of data contained in a file system. If you installed a Debian or a Windows in the usual way onto one or two hard disk partitions and would afterwards copy the whole content byte by byte from the hard disk into one file:
 +
 +<code bash>
 +dd if=/dev/sda of=/​media/​sdb3/​backup.dd
 +</​code>​
 +
 +the obtained backup file ''/​media/​sdb3/​backup.dd'',​ could be used in the exact same manner like an OpenWrt-Image-File.
 +
 +
 +The difference is, that OpenWrt-Image-File are not created that way ;-) They are being generated with the **Image Generator** (former called Image Builder). You can read about the:
 +  * [[doc/​techref/​flash.layout]]
 +  * [[doc/​techref/​header]]
 +  * [[doc/​howto/​obtain.firmware.generate]] If you want to read about the **Image Generator**,​ go back to [[doc:​howto:​obtain.firmware]] and choose the second way.
 +  * About [[http://​skaya.enix.org/​wiki/​FirmwareFormat|Broadcom Firmware Format]]