Modifying the Ubiquiti CRM Key to run Cloud Key firmware

This is an extract from the information that I posted on the Ubiquiti forums in this topic. I'd like to be very clear that if you choose to run any of the commands on this page, you are doing this at your own risk. Don't proceed unless you're okay with potentially bricking your device without any way to recover.

Background

I recently got a CRM Point (being well aware that it it not supported anymore by Ubiquiti) and started playing around with it. For the uninitiated: the Cloud Key is an ARM-based small computer that can be powered over POE. It uses the MediaTek mtk7623 SOC and has 2GiB of RAM memory. The point of the CRM Key was to install it at customers networks which allows a sysadmin to remotely manage compatible (Ubiquiti airMAX) network equipment. 

I noticed that the firmware isn't updated that regularly anymore which is sad. Because it has a limited amount of flash memory and any time I run apt-get update && apt-get upgrade -y this fills the read-write partition. 

But most packages are so old that installing (for example) an up to date version of Java will take more than half of the available storage. Of course I had purchased this device because I intended to tinker with it. So I did exactly that. 

First exploration

I installed CRM.mtk7623.v0.6.0.a670e69M.170615.0748.bin on it (to my knowledge this is the latest firmware that was released for this hardware) and was happy to see that a clean install leaves 100% of space on the rw-partition. This got me thinking: if I can install the updated packages in the read-only partition this will leave the read-write partition almost empty! Given that the read-only partition is situated at /dev/mmcblk0p6 I started trying some stuff out: 

root@control-point:/data# dd if=/dev/mmcblk0p6 of=/data/dd.img
2097152+0 records in
2097152+0 records out

1073741824 bytes (1.1 GB) copied, 101.841 s, 10.5 MB/s

root@control-point:/data# dd if=/data/dd.img of=/dev/mmcblk0p6
2097152+0 records in
2097152+0 records out

1073741824 bytes (1.1 GB) copied, 242.271 s, 4.4 MB/s

So writing to the read-only partition directly does not seem to be blocked in any way. After a reboot everything still works just fine! But interestingly enough the partition is 1.1GB while the size of the read-only filesystem as reported by df is only 207MB. This means that there is some potential room for additional tools and packages! 

First attempt at a 'custom rom'

I extracted the squashfs from /dev/mmvblk0p6 onto /data/, then ran mksquashfs on the squashfs-root folder in /data and then I proceeded to dd the resulting squashfs back to /dev/mmvblk0p6. While the filesystem is mostly fine, somehow my root/ubnt user got borked and I can't log in using ssh :(. I can however log in to the AirControl web UI. My theory is that this was an issue caused by the layer _under_ the rw-overlay changing. Therefore the stuff on top became invalid. Trashing the entire rw-layer fixes that. 

How to start modifying your read-only squashfs:

• ssh to your crm point
• cd to /data directory
• apt-get update && apt-get install squashfs-tools
• unsquashfs /dev/mmcblk0p6
• Follow the step on this page about the policy.d file
• Mount /dev, /dev/pts, /proc sections according to the copy-pasta here
• chroot squashfs-root
• mkdir /tmp

At this point you can run apt-get commands as if you are actually "running this system". So this includes adding packages, Fixing /etc/apt/sources.list to also contain jessie-backports, updating existing packages. Just make sure that all of this will still fit in the 1.1GB partition that it will eventually need to squeeze in to. In my case I also removed the aircontrol and postgres packages. Then start your cleanup:

rm -rf /tmp
apt-get clean
# Next line makes sure that on factory reset a new SSH key is generated
rm /etc/ssh/ssh_host_rsa_key.pub /etc/ssh/ssh_host_dsa_key.pub /etc/ssh/ssh_host_ecdsa_key.pub /etc/ssh/ssh_host_ed25519_key.pub
rm -rf /usr/sbin/policy-rc.d
history -c
exit #(this is where you leave your chroot)
umount /proc /dev/pts /dev #In my case some of them didn't want to unmount. If this happens, just reboot the CRM point.
mksquashfs /data/squashfs-root root.sqfs
dd if=/data/root.sqfs of=/dev/mmcblk0p6

From this point on any command you run will likely fail because the rofs was overwritten while mounted. This confuses the system which is understandable. So just pull the power, plug it in again, and attempt to reset the device with a paperclip or something. It should boot up pretty fast and you are now running your custom squashfs image!

Next steps

So given that I was able to modify my squashfs I started to wonder... What if I could extract the kernel and rootfs from the cloud key firmware update and copy it to the flash of my CRM Point using DD? Would the kernel crash? Would the bootloader do some checksum check on the kernel partition? I didn't know, but I wanted to try. Especially since a lot of people on the forum over the years have said "you can't", "you'll brick it" and things like that. And I am pretty stubborn. 

 So first I tried to cross-flash the Cloud Key the firmware using ubnt-systool fwupdate but that failed. This is because the tool that checks which product this is probably uses data from one of the partitions that I didn't touch. But that's actually perfect because I wouldn't want an accidental update to brick my hacked cloud key, so I guess this worked out just fine. 

 1. Determine flash layout

So as most people probably already know, embedded devices running linux usually have multiple partitions. In order to do some analysis on firmwares the tool dd is very useful to make backups of partitions so that if you screw up but you can get the device to boot to some kind of recovery you can restore it. So let's take a look at the partitions:

root@control-point:~# parted /dev/mmcblk0
GNU Parted 3.2
Using /dev/mmcblk0
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) p
Model: MMC 004GE0 (sd/mmc)
Disk /dev/mmcblk0: 3937MB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:


Number  Start   End     Size    File system  Name      Flags
 1      262kB   786kB   524kB                uboot
 2      786kB   1049kB  262kB                config
 3      1049kB  1311kB  262kB                factory
 4      1311kB  34.9MB  33.6MB               kernel
 5      34.9MB  68.4MB  33.6MB               recovery
 6      68.4MB  1142MB  1074MB               rootfs
 7      1142MB  2753MB  1611MB  ext4         appdata
 8      2753MB  3937MB  1185MB  ext4         userdata

As we can see there are quite some partitions. Let's back them all up first. 

 2. Back up all partitions

This makes a copy of every individual partition on to your SD card (which should be mounted at /data)

root@control-point:~# dd if=/dev/mmcblk0p1 of=/data/mmcblk0p1.img
1024+0 records in
1024+0 records out
524288 bytes (524 kB) copied, 0.0360363 s, 14.5 MB/s
root@control-point:~# dd if=/dev/mmcblk0p2 of=/data/mmcblk0p2.img
512+0 records in
512+0 records out
# repeat for every partition number (so till mmcblk0p8)

Next I copied these files from the SD card to my NAS, but of course just putting the SD-card in a SD-card reader is also perfectly fine.

root@control-point:/data# scp mmc* root@192.168.1.13:/mnt/user/FliX/Ubiquiti/
The authenticity of host '192.168.1.13 (192.168.1.13)' can't be established.
ECDSA key fingerprint is c4:0f:f2:89:57:df:bb:85:ee:ba:fb:41:e6:d3:90:9e.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.1.13' (ECDSA) to the list of known hosts.
root@192.168.1.13's password:
mmcblk0p1.img                               100%  512KB 512.0KB/s   00:00
mmcblk0p2.img                               100%  256KB 256.0KB/s   00:00
mmcblk0p3.img                               100%  256KB 256.0KB/s   00:00
mmcblk0p4.img                               100%   32MB   8.0MB/s   00:04
mmcblk0p5.img                               100%   32MB  10.7MB/s   00:03

 3. Examining the DD-images and the firmware files

For this I used the tool binwalk. This walks through a firmware image and looks for certain signatures that are known to identify certain 'parts' of a firmware image. So first I did this for the latest firmware updates for the UCK and CRM Point:

$ binwalk UCK.mtk7623.v1.1.13.818cc5f.200430.0950.bin

DECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------------------
0             0x0             Ubiquiti firmware header, header size: 264 bytes, ~CRC32: 0x8ED3EC98, version: "UCK.mtk7623.v1.1.13.818cc5f.200430.0950"
260           0x104           Ubiquiti partition header, header size: 56 bytes, name: "PARTkernel", base address: 0x00000000, data size: 0 bytes
324           0x144           uImage header, header size: 64 bytes, header CRC: 0x3960E04E, created: 2020-04-30 10:00:43, image size: 7079048 bytes, Data Address: 0x80008000, Entry Point: 0x80008000, data CRC: 0x7D24D70D, OS: Linux, CPU: ARM, image type: OS Kernel Image, compression type: none, image name: "Linux-3.10.20-ubnt-mtk"
14642         0x3932          xz compressed data
14863         0x3A0F          xz compressed data
7079436       0x6C060C        Ubiquiti partition header, header size: 56 bytes, name: "PARTrootfs", base address: 0x00000002, data size: 0 bytes
7079500       0x6C064C        Squashfs filesystem, little endian, version 4.0, compression:gzip, size: 349822683 bytes, 29395 inodes, blocksize: 262144 bytes, created: 2020-04-30 10:00:22


$ binwalk CRM.mtk7623.v0.6.0.a670e69M.170615.0748.bin

DECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------------------
0             0x0             Ubiquiti firmware header, header size: 264 bytes, ~CRC32: 0x604E5185, version: "CRM.mtk7623.v0.6.0.a670e69M.170615.0748"
260           0x104           Ubiquiti partition header, header size: 56 bytes, name: "PARTkernel", base address: 0x00000000, data size: 0 bytes
324           0x144           uImage header, header size: 64 bytes, header CRC: 0xE24B3D5A, created: 2017-06-15 14:57:10, image size: 6586640 bytes, Data Address: 0x80008000, Entry Point: 0x80008000, data CRC: 0x2526FE9E, OS: Linux, CPU: ARM, image type: OS Kernel Image, compression type: none, image name: "Linux-3.10.20-ubnt-mtk"
14642         0x3932          xz compressed data
14863         0x3A0F          xz compressed data
6587028       0x648294        Ubiquiti partition header, header size: 56 bytes, name: "PARTrootfs", base address: 0x00000002, data size: 0 bytes
6587092       0x6482D4        Squashfs filesystem, little endian, version 4.0, compression:gzip, size: 216177339 bytes, 17413 inodes, blocksize: 262144 bytes, created: 2017-06-15 14:56:50

So clearly the first 'segments' are ubiquiti specific. Then at offset 324 we see the declaration of a 7079048 byte image. Given that the read-only filesystem is a lot larger than 7 Megabytes this is probably the kernel. The last entry in the table is the actual squashfs filesystem. And this is where it gets interesting! 

 4. Testing a theory 

So let's assume that the images in the firmware update files are being copied 1:1 to the partitions (so: kernel part goes to mmcblk0p4, rofs goes to mmcblk0p6) and no other security/encryption happens in that process. That would mean that we can test this theory by comparing the kernel partition image we created on the CRM Point to the kernel partition image in the firmware update file. dd if=CRM.mtk7623.v0.6.0.a670e69M.170615.0748.bin skip=324 count=6586704 iflag=count_bytes bs=1 of=extracted_crm_kernel So let's test that theory. So we're copying data starting at offset 324 (where the uImage header starts) and we take image size + header size as the count argument. Comparing the extracted_crm_kernel file to mmcblk0p4.img confirmed my suspicion. The only difference between the two files is that the partition we extracted from the CRM is filled with zero's where the kernel data ends. For obvious reasons the firmware update does not contain all the 0's. We're going to take a wild guess and assume that this is also the case for the data partition:

dd if=UCK.mtk7623.v1.1.13.818cc5f.200430.0950.bin skip=324 count=7079112 iflag=count_bytes bs=1 of=extracted_uck_kernel
dd if=UCK.mtk7623.v1.1.13.818cc5f.200430.0950.bin skip=1769875 bs=4 of=extracted_uckrootfs.squashfs

5. Crafting our update file

Earlier in this post thread I showed that using dd to overwrite a partition works just fine, but everything that reads from the partition (especially the overlay one) is screwed up because the underlying data is completely different. Unfortunately swapping a kernel and not the filesystem may also result in a brick. But the kernel and rootfs partitions have a recovery partition in between. Initially I wanted to make a single file that we can write with dd that includes the kernel, recovery and rootfs partitions. But I took a quick peek in mmcblk0p5.img and realised this partition is filled with zeros. This explains why people have no way to recover once they brick this thing :D.

$ cp mmcblk0p5.img extracted_uck_kernel_padded.img # take 32MiB img filled with zeroes
$ dd if=extracted_uck_kernel conv=notrunc of=extracted_uck_kernel_padded.img
13826+1 records in
13826+1 records out
7079112 bytes (7.1 MB, 6.8 MiB) copied, 0.177849 s, 39.8 MB/s
$ dd if=mmcblk0p5.img >>extracted_uck_kernel_padded.img # insert 32MiB of zeroes
65536+0 records in
65536+0 records out
33554432 bytes (34 MB, 32 MiB) copied, 1.16823 s, 28.7 MB/s
$ dd if=extracted_uckrootfs.squashfs >>extracted_uck_kernel_padded.img # then append the squashfs
683248+1 records in
683248+1 records out
349823248 bytes (350 MB, 334 MiB) copied, 11.6135 s, 30.1 MB/s

So next up: copy extracted_uck_kernel_padded.img on to the sdcard so we can flash it! 6. Flashing This is the part where everything either succeeds, or fails. This is your last chance to choose for safety. If you enter these commands and you brick your device, you are on your own. Neither Ubiquiti nor I can provide support to you if this fails. First off, let's determine where we have to write this file to.

root@control-point:/# parted /dev/mmcblk0 'unit s print'
Model: MMC 004GE0 (sd/mmc)
Disk /dev/mmcblk0: 7690240s
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start     End       Size      File system  Name      Flags
 1      512s      1535s     1024s                  uboot
 2      1536s     2047s     512s                   config
 3      2048s     2559s     512s                   factory
 4      2560s     68095s    65536s                 kernel
 5      68096s    133631s   65536s                 recovery
 6      133632s   2230783s  2097152s               rootfs
 7      2230784s  5376511s  3145728s  ext4         appdata
 8      5376512s  7690206s  2313695s  ext4         userdata

Ok, cool. So because we have an image file that spans kernel (32MiB), recovery (32MiB) and rootfs (the rest) we can just tell dd to start writing at offset 2560 until we reach the end of the file. This works because the new rootfs is larger than the previous one. Otherwise it would have been wiser to append some more 00's at the end of our image just to make sure we don't have weird garbage at the end of our actual partition data.

root@control-point:/# dd if=/data/extracted_uck_kernel_padded.img of=/dev/mmcblk0 \
  seek=2560
814320+1 records in
814320+1 records out
416932112 bytes (417 MB) copied, 60.8669 s, 6.8 MB/s

So from this point your stick will be a little confused. SSH logins might fail, and that kind of good stuff. For me just a normal paperclip-reset wasn't enough, I had to do a reset from the web interface as well (or SSH... not 100% sure anymore). But after the reset this thing behaves like a UniFi Cloud Key! Enjoy!