LEDE project: prevent having to re-flash the latest nightly build every time you need a new package

I really admire the LEDE project and the reasons why they decided that it was necessary to fork openwrt and continue in a way that they saw fit. Unfortunately as they only recently started (somewhere about May 2016) there are no official releases yet and this means that there are only nightly builds.

This 'guide' will work for both openwrt and lede nightly builds. The reason is that every night the packages and dependencies get rebuilt and packages that rely on a specific kernel/kernel module version can not be installed at a later moment (when the new nightly build replaces the previous one online). To solve this we can make a copy of all packages for a given nightly build. In a nutshell:

1. Download the firmware file from the LEDE/openwrt site
2. Run the commands that download all packages from the LEDE/openwrt site
3. Install/upgrade to recent build
4. Run a local webserver
5. Modify the package repository URL's to match your local webserver

So lets get started!

Download the firmware file

This is easy and probably something you've done before. If not: check out the architecture of your router and try to find whether openwrt/LEDE already supports your router. If they don't you're out of luck for now. For this guide I'll be using the TP-Link Archer C7 which is contained in the ar71xx builds of both LEDE and openwrt. 

Run these commands that download all packages from the LEDE/openwrt site

wget -m --no-parent -e robots=off https://downloads.lede-project.org/snapshots/targets/ar71xx/generic/packages/

wget -m --no-parent -e robots=off http://downloads.lede-project.org/snapshots/packages/mips_24kc/base

wget -m --no-parent -e robots=off http://downloads.lede-project.org/snapshots/packages/mips_24kc/luci

wget -m --no-parent -e robots=off http://downloads.lede-project.org/snapshots/packages/mips_24kc/packages

wget -m --no-parent -e robots=off http://downloads.lede-project.org/snapshots/packages/mips_24kc/routing

wget -m --no-parent -e robots=off http://downloads.lede-project.org/snapshots/packages/mips_24kc/telephony

The commands above should download all the packages that LEDE references within it's "software distribution feeds". For openwrt all references to lede-project.org should be replaced with openwrt.org. If you already have a version of openwrt/LEDE installed you can just make sure that you wget every of the URL's in the list instead of the URL's I provided above. Checkout 'System > Software' => Configuration tab for a list of feeds used by the current firmware.

After downloading the feeds you might want to move the contents of the downloads.lede-project.org/snapshots folder to a new folder. I'm using /Users/flix/Documens/lede-snapshots here as the /Users folder can be shared with Docker containers by default if you're using macOS. Not sure about how permissive Linux is in this regard.

Install/upgrade to the downloaded build

After downloading the packages you can install the firmware. To be sure that your 'offline' package feeds actually will work I suggest not (re)installing any of your packages just yet. Just the firmware.

Run a local webserver

Of course, the goal of downloading the packages is to serve them at a later moment in time. Let's get it started:

docker run --name nginx -v /Users/flix/Documents/lede-snapshots:/usr/share/nginx/html:ro -d -p 8080:80 nginx

The above command will pull an nginx container from docker hub and tell it to serve your local folder (containing the packages) on port 8080 from the machine you're running on. Of course you can copy the files to your home server, NAS, raspberry pi or any other device that can run a web server permanently, but for my use case (occasional desire to try out a package) I find spinning up a docker container the easiest.

Modify the package feed URLs

So now we can just swap the downloads.lede-project.org/snapshots part with the IP address of our webserver. This is it! You can now install packages from your local package mirror! Good luck!

The HP Microserver Gen8 for EUR 200. Is it worth it?

So recently our Drobo at QwikSense (my other company) died. Which wasn't really a problem, except that it was storing our Time Machine backups for my computer and my colleagues' MacBooks. So I was looking for a reliable storage solution with room to spare for an extra disk and preferably with ECC memory.

In the NAS world you're either out of luck if you're looking for something at around €200. Most of the time this will give you an ARM device with HDD-slots. On top of that this locks you into the vendor firmware (and support) and most of the time building something yourself with the provided kernel/firmware sources is not enough to build a robust solution that gives you a 'simple' user interface that just does what it should.

Of course we could build something ourselves. Whack a Mini-ITX board in a dito casing and voilà: hard disks attached to your network. But I don't have the time to do this and the number of Mini-ITX boards that sports an ECC-capable CPU/RAM at a reasonable price is close to zero.

Our saviour

Enter the HP Gen8 Microserver. The model with a Celeron G1610T and 4GB ECC(!) RAM can currently be purchased in the Netherlands for €203 euros at the cheapest reliable internet store at the time of writing! Now the strange (and awesome) thing about this tiny server is that it has a CPU, memory and a motherboard that support ECC! Can you believe it? And as an extra, HP's ILO4 management solution is also present. You can actually turn this thing on remotely if it's switched off. You can forcefully reboot it if the kernel crashes. It has features that "real" servers have, but in a tiny and silent package that looks AWESOME.

Note that the image above was ripped from 3dnews.ru. Also note that this is not the exact same model that you can get for such a low price. My model did not come with an optical drive, but who needs that anyway?

Current setup

We still had some drives from the Drobo, so we put those in and added a 128GB SSD from Plextor to act as a caching drive. This confirmed my suspicion that the Drobo was in fact dead as a dodo. The drives were all okay! Because some of the drives that were used in the Drobo were WD Greens (and everybody is always crying about how WD Greens drop out of RAID arrays) I decided to WDIDLE3 them. The easiest way for me was to get UBCD, put it on a USB drive and boot the Microserver from that. Now I must stress that you should NOT connect ANY other drives than the ONE drive you want to flash (except for USB disks of course, they don't count). When booting from de USB drive, open "Disk Management" or something similar (sorry, I forgot what it actually was) and then select WDIDLE3 at the bottom of the list of tools. You might get a list with a lot of choices and a countdown. Let the countdown pass and wait for a DOS-like prompt. At the prompt enter the following command:

WDIDLE3.exe /s300 (for 5 minutes).

That's it. Power down the system and repeat for any WD Green drives you have lying around. Unraid time!

Unraid

Since I saw LinusTechTips' 7 Gamers 1 CPU build I was very interested in unRaid as a storage solution. While I'm still running my Netgear ReadyNAS 4 Ultra at home (running ReadyNAS OS6 while Netgear only provided updates on OS4) I have wanted to own a machine like this. Docker on the ReadyNAS wouldn't work (because I believe the kernel was too old, not sure if that is still the case since they migrated to Jessie in the latest firmware update) but on unRaid the sky is the limit as VM's, Docker and plugins can all be used out of the box. There is a very active community providing solutions to run Plex Media Server, Resilio Sync, CouchPotato, SabNZBD and many more tools for media management. As we use Resilio Sync at work I decided to install that. 

Note that the basic license for unRaid allows a maximum of 6 internal drives (== SATA) while the MicroServer only has 5 SATA ports (the fifth port is for the optical drive). The USB drive does not count towards the number of drives. Unless you're planning on adding a SAS card and an external storage enclosure you'll be fine with a basic license. And if you're not, you probably weren't planning on keeping the budget low anyway.

Photo from a review on tweakers.net
(dutch)

Power consumption

While I've read different thing about the Gen8 Micro I've not been able to measure the power consumption yet. Hopefully I'll be able to report on that soon.

Upgrades (and potential for win)

The Gen8 Microserver has a regular desktop-formfactor Intel CPU on board in socket 1155. While the CPU has a TDP of just 35 watts, many people have retrofitted their Gen8 with beefier CPU's. This is one of the cheapest ways to create a powerful, compact and mostly silent reliable server for a small company or for media streaming. Especially if you have a few drives lying around, this project (including CPU upgrade, SSD cache disk and unRaid license but excluding HDD's) would set you back around €420. Sounds like a sweet deal to me! But even without the CPU upgrade it's a great deal. Let me know what you think!