BackgroundI recently got a CRM Point (being well aware that it it not supported anymore by Ubiquiti) and started playing around with it. For the uninitiated: the Cloud Key is an ARM-based small computer that can be powered over POE. It uses the MediaTek mtk7623 SOC and has 2GiB of RAM memory. The point of the CRM Key was to install it at customers networks which allows a sysadmin to remotely manage compatible (Ubiquiti airMAX) network equipment.
I noticed that the firmware isn't updated that regularly anymore which is sad. Because it has a limited amount of flash memory and any time I run apt-get update && apt-get upgrade -y this fills the read-write partition.
But most packages are so old that installing (for example) an up to date version of Java will take more than half of the available storage. Of course I had purchased this device because I intended to tinker with it. So I did exactly that.
First explorationI installed CRM.mtk7623.v0.6.0.a670e69M.170615.0748.bin on it (to my knowledge this is the latest firmware that was released for this hardware) and was happy to see that a clean install leaves 100% of space on the rw-partition. This got me thinking: if I can install the updated packages in the read-only partition this will leave the read-write partition almost empty! Given that the read-only partition is situated at /dev/mmcblk0p6 I started trying some stuff out:
root@control-point:/data# dd if=/dev/mmcblk0p6 of=/data/dd.img 2097152+0 records in 2097152+0 records out 1073741824 bytes (1.1 GB) copied, 101.841 s, 10.5 MB/s root@control-point:/data# dd if=/data/dd.img of=/dev/mmcblk0p6 2097152+0 records in 2097152+0 records out 1073741824 bytes (1.1 GB) copied, 242.271 s, 4.4 MB/s
So writing to the read-only partition directly does not seem to be blocked in any way. After a reboot everything still works just fine! But interestingly enough the partition is 1.1GB while the size of the read-only filesystem as reported by df is only 207MB. This means that there is some potential room for additional tools and packages!
First attempt at a 'custom rom'
I extracted the squashfs from
/data/, then ran mksquashfs on the squashfs-root folder in
/dataand then I proceeded to dd the resulting squashfs back to /dev/mmvblk0p6. While the filesystem is mostly fine, somehow my root/ubnt user got borked and I can't log in using ssh :(. I can however log in to the AirControl web UI. My theory is that this was an issue caused by the layer _under_ the rw-overlay changing. Therefore the stuff on top became invalid. Trashing the entire rw-layer fixes that.
How to start modifying your read-only squashfs:
- ssh to your crm point
- cd to
apt-get update && apt-get install squashfs-tools
- Follow the step on this page about the policy.d file
- Mount /dev, /dev/pts, /proc sections according to the copy-pasta here
apt-getcommands as if you are actually "running this system". So this includes adding packages, Fixing
/etc/apt/sources.listto also contain jessie-backports, updating existing packages. Just make sure that all of this will still fit in the 1.1GB partition that it will eventually need to squeeze in to. In my case I also removed the aircontrol and postgres packages. Then start your cleanup:
From this point on any command you run will likely fail because the rofs was overwritten while mounted. This confuses the system which is understandable. So just pull the power, plug it in again, and attempt to reset the device with a paperclip or something. It should boot up pretty fast and you are now running your custom squashfs image!
rm -rf /tmp apt-get clean # Next line makes sure that on factory reset a new SSH key is generated rm /etc/ssh/ssh_host_rsa_key.pub /etc/ssh/ssh_host_dsa_key.pub /etc/ssh/ssh_host_ecdsa_key.pub /etc/ssh/ssh_host_ed25519_key.pub rm -rf /usr/sbin/policy-rc.d history -c exit #(this is where you leave your chroot) umount /proc /dev/pts /dev #In my case some of them didn't want to unmount. If this happens, just reboot the CRM point. mksquashfs /data/squashfs-root root.sqfs dd if=/data/root.sqfs of=/dev/mmcblk0p6
Next stepsSo given that I was able to modify my squashfs I started to wonder... What if I could extract the kernel and rootfs from the cloud key firmware update and copy it to the flash of my CRM Point using DD? Would the kernel crash? Would the bootloader do some checksum check on the kernel partition? I didn't know, but I wanted to try. Especially since a lot of people on the forum over the years have said "you can't", "you'll brick it" and things like that. And I am pretty stubborn.
So first I tried to cross-flash the Cloud Key the firmware using ubnt-systool fwupdate but that failed. This is because the tool that checks which product this is probably uses data from one of the partitions that I didn't touch. But that's actually perfect because I wouldn't want an accidental update to brick my hacked cloud key, so I guess this worked out just fine.
1. Determine flash layout
So as most people probably already know, embedded devices running linux usually have multiple partitions. In order to do some analysis on firmwares the tool dd is very useful to make backups of partitions so that if you screw up but you can get the device to boot to some kind of recovery you can restore it. So let's take a look at the partitions:
As we can see there are quite some partitions. Let's back them all up first.
root@control-point:~# parted /dev/mmcblk0 GNU Parted 3.2 Using /dev/mmcblk0 Welcome to GNU Parted! Type 'help' to view a list of commands. (parted) p Model: MMC 004GE0 (sd/mmc) Disk /dev/mmcblk0: 3937MB Sector size (logical/physical): 512B/512B Partition Table: gpt Disk Flags: Number Start End Size File system Name Flags 1 262kB 786kB 524kB uboot 2 786kB 1049kB 262kB config 3 1049kB 1311kB 262kB factory 4 1311kB 34.9MB 33.6MB kernel 5 34.9MB 68.4MB 33.6MB recovery 6 68.4MB 1142MB 1074MB rootfs 7 1142MB 2753MB 1611MB ext4 appdata 8 2753MB 3937MB 1185MB ext4 userdata
2. Back up all partitions
This makes a copy of every individual partition on to your SD card (which should be mounted at /data)
Next I copied these files from the SD card to my NAS, but of course just putting the SD-card in a SD-card reader is also perfectly fine.
root@control-point:~# dd if=/dev/mmcblk0p1 of=/data/mmcblk0p1.img 1024+0 records in 1024+0 records out 524288 bytes (524 kB) copied, 0.0360363 s, 14.5 MB/s root@control-point:~# dd if=/dev/mmcblk0p2 of=/data/mmcblk0p2.img 512+0 records in 512+0 records out # repeat for every partition number (so till mmcblk0p8)
root@control-point:/data# scp mmc* email@example.com:/mnt/user/FliX/Ubiquiti/ The authenticity of host '192.168.1.13 (192.168.1.13)' can't be established. ECDSA key fingerprint is c4:0f:f2:89:57:df:bb:85:ee:ba:fb:41:e6:d3:90:9e. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '192.168.1.13' (ECDSA) to the list of known hosts. firstname.lastname@example.org's password: mmcblk0p1.img 100% 512KB 512.0KB/s 00:00 mmcblk0p2.img 100% 256KB 256.0KB/s 00:00 mmcblk0p3.img 100% 256KB 256.0KB/s 00:00 mmcblk0p4.img 100% 32MB 8.0MB/s 00:04 mmcblk0p5.img 100% 32MB 10.7MB/s 00:03
3. Examining the DD-images and the firmware files
For this I used the tool binwalk. This walks through a firmware image and looks for certain signatures that are known to identify certain 'parts' of a firmware image. So first I did this for the latest firmware updates for the UCK and CRM Point:
So clearly the first 'segments' are ubiquiti specific. Then at offset 324 we see the declaration of a 7079048 byte image. Given that the read-only filesystem is a lot larger than 7 Megabytes this is probably the kernel. The last entry in the table is the actual squashfs filesystem. And this is where it gets interesting!
$ binwalk UCK.mtk7623.v188.8.131.528cc5f.200430.0950.bin DECIMAL HEXADECIMAL DESCRIPTION -------------------------------------------------------------------------------- 0 0x0 Ubiquiti firmware header, header size: 264 bytes, ~CRC32: 0x8ED3EC98, version: "UCK.mtk7623.v184.108.40.2068cc5f.200430.0950" 260 0x104 Ubiquiti partition header, header size: 56 bytes, name: "PARTkernel", base address: 0x00000000, data size: 0 bytes 324 0x144 uImage header, header size: 64 bytes, header CRC: 0x3960E04E, created: 2020-04-30 10:00:43, image size: 7079048 bytes, Data Address: 0x80008000, Entry Point: 0x80008000, data CRC: 0x7D24D70D, OS: Linux, CPU: ARM, image type: OS Kernel Image, compression type: none, image name: "Linux-3.10.20-ubnt-mtk" 14642 0x3932 xz compressed data 14863 0x3A0F xz compressed data 7079436 0x6C060C Ubiquiti partition header, header size: 56 bytes, name: "PARTrootfs", base address: 0x00000002, data size: 0 bytes 7079500 0x6C064C Squashfs filesystem, little endian, version 4.0, compression:gzip, size: 349822683 bytes, 29395 inodes, blocksize: 262144 bytes, created: 2020-04-30 10:00:22 $ binwalk CRM.mtk7623.v0.6.0.a670e69M.170615.0748.bin DECIMAL HEXADECIMAL DESCRIPTION -------------------------------------------------------------------------------- 0 0x0 Ubiquiti firmware header, header size: 264 bytes, ~CRC32: 0x604E5185, version: "CRM.mtk7623.v0.6.0.a670e69M.170615.0748" 260 0x104 Ubiquiti partition header, header size: 56 bytes, name: "PARTkernel", base address: 0x00000000, data size: 0 bytes 324 0x144 uImage header, header size: 64 bytes, header CRC: 0xE24B3D5A, created: 2017-06-15 14:57:10, image size: 6586640 bytes, Data Address: 0x80008000, Entry Point: 0x80008000, data CRC: 0x2526FE9E, OS: Linux, CPU: ARM, image type: OS Kernel Image, compression type: none, image name: "Linux-3.10.20-ubnt-mtk" 14642 0x3932 xz compressed data 14863 0x3A0F xz compressed data 6587028 0x648294 Ubiquiti partition header, header size: 56 bytes, name: "PARTrootfs", base address: 0x00000002, data size: 0 bytes 6587092 0x6482D4 Squashfs filesystem, little endian, version 4.0, compression:gzip, size: 216177339 bytes, 17413 inodes, blocksize: 262144 bytes, created: 2017-06-15 14:56:50
4. Testing a theory
So let's assume that the images in the firmware update files are being copied 1:1 to the partitions (so: kernel part goes to mmcblk0p4, rofs goes to mmcblk0p6) and no other security/encryption happens in that process. That would mean that we can test this theory by comparing the kernel partition image we created on the CRM Point to the kernel partition image in the firmware update file.
dd if=CRM.mtk7623.v0.6.0.a670e69M.170615.0748.bin skip=324 count=6586704 iflag=count_bytes bs=1 of=extracted_crm_kernelSo let's test that theory. So we're copying data starting at offset 324 (where the uImage header starts) and we take image size + header size as the count argument. Comparing the extracted_crm_kernel file to mmcblk0p4.img confirmed my suspicion. The only difference between the two files is that the partition we extracted from the CRM is filled with zero's where the kernel data ends. For obvious reasons the firmware update does not contain all the 0's. We're going to take a wild guess and assume that this is also the case for the data partition:
dd if=UCK.mtk7623.v220.127.116.118cc5f.200430.0950.bin skip=324 count=7079112 iflag=count_bytes bs=1 of=extracted_uck_kernel dd if=UCK.mtk7623.v18.104.22.1688cc5f.200430.0950.bin skip=1769875 bs=4 of=extracted_uckrootfs.squashfs
5. Crafting our update file
Earlier in this post thread I showed that using dd to overwrite a partition works just fine, but everything that reads from the partition (especially the overlay one) is screwed up because the underlying data is completely different. Unfortunately swapping a kernel and not the filesystem may also result in a brick. But the kernel and rootfs partitions have a recovery partition in between. Initially I wanted to make a single file that we can write with dd that includes the kernel, recovery and rootfs partitions. But I took a quick peek in
mmcblk0p5.imgand realised this partition is filled with zeros. This explains why people have no way to recover once they brick this thing :D.
So next up: copy
$ cp mmcblk0p5.img extracted_uck_kernel_padded.img # take 32MiB img filled with zeroes $ dd if=extracted_uck_kernel conv=notrunc of=extracted_uck_kernel_padded.img 13826+1 records in 13826+1 records out 7079112 bytes (7.1 MB, 6.8 MiB) copied, 0.177849 s, 39.8 MB/s $ dd if=mmcblk0p5.img >>extracted_uck_kernel_padded.img # insert 32MiB of zeroes 65536+0 records in 65536+0 records out 33554432 bytes (34 MB, 32 MiB) copied, 1.16823 s, 28.7 MB/s $ dd if=extracted_uckrootfs.squashfs >>extracted_uck_kernel_padded.img # then append the squashfs 683248+1 records in 683248+1 records out 349823248 bytes (350 MB, 334 MiB) copied, 11.6135 s, 30.1 MB/s
extracted_uck_kernel_padded.imgon to the sdcard so we can flash it! 6. Flashing This is the part where everything either succeeds, or fails. This is your last chance to choose for safety. If you enter these commands and you brick your device, you are on your own. Neither Ubiquiti nor I can provide support to you if this fails. First off, let's determine where we have to write this file to.
Ok, cool. So because we have an image file that spans kernel (32MiB), recovery (32MiB) and rootfs (the rest) we can just tell dd to start writing at offset 2560 until we reach the end of the file. This works because the new rootfs is larger than the previous one. Otherwise it would have been wiser to append some more 00's at the end of our image just to make sure we don't have weird garbage at the end of our actual partition data.
root@control-point:/# parted /dev/mmcblk0 'unit s print' Model: MMC 004GE0 (sd/mmc) Disk /dev/mmcblk0: 7690240s Sector size (logical/physical): 512B/512B Partition Table: gpt Disk Flags: Number Start End Size File system Name Flags 1 512s 1535s 1024s uboot 2 1536s 2047s 512s config 3 2048s 2559s 512s factory 4 2560s 68095s 65536s kernel 5 68096s 133631s 65536s recovery 6 133632s 2230783s 2097152s rootfs 7 2230784s 5376511s 3145728s ext4 appdata 8 5376512s 7690206s 2313695s ext4 userdata
So from this point your stick will be a little confused. SSH logins might fail, and that kind of good stuff. For me just a normal paperclip-reset wasn't enough, I had to do a reset from the web interface as well (or SSH... not 100% sure anymore). But after the reset this thing behaves like a UniFi Cloud Key! Enjoy!
root@control-point:/# dd if=/data/extracted_uck_kernel_padded.img of=/dev/mmcblk0 \ seek=2560 814320+1 records in 814320+1 records out 416932112 bytes (417 MB) copied, 60.8669 s, 6.8 MB/s