Konubinix' opinionated web of thoughts

Setting Up My Raspberry Pi Fleet

Fleeting

setting up my raspberry pi fleet

using hypriot

It works well.

I would have stick with it, but

  1. it is not updated anymore, so at some point it will likely fail (the operating system being too old to even apt update correctly)
  2. it does not support arm64, and more and more project don’t support armv<8 anymore (like distroless)
  3. it does not allow preparing the rootfs (using chroot and qemu) like installing stuff ahead of time. I need to put stuff in cloud-init, them plug the card and wait 30 minutes that the system updates and them run the init script. I would prefer a img that simply works when started.

building hypriot myself

It seemed to me that it was complicated

alpine on raspberrypi

also quite complicated. I have to setup the img layout myself. Seems like a good way to end up yak shaving.

I’d rather find a project that already does the whole work and simply adjust to fit my needs.

debian image-spec

Seemed ideal. It uses vmdb2, so the steps are quite transparent. It may be run in docker. I only need to append the *.yaml file with my custom commands so that I get my custom image.

Thanks to its unpack-rootfs feature, I can create edit my config and avoid running the whole flow every time.

Also, because it can be run in docker, I can take advantage of docker cache to cache the partial rootfs.

It was so great that I started putting it into all my machines. After a few days, I realized that my nomad job did not work as well as they were used to. My docker images took a very long time to be pulled and sometimes failed without obvious reason.

To me, this looked like failing SD cards, so I changed them. But the issue persisted. After realizing that ALL my sdcard were “faulty”, I decided to try getting back to hypriot, and things were back to normal.

I assume that the kernel shipped in debian is “working”, but that the one from raspberry is optimised. So I decided to try raspberry pi based solutions.

pi-gen

It looks nice. It is composed of a bunch of scripts that are run in order. It provides out of the box the configuration of ssh (with the installation of the public key and disabling of password authentication), the keyboard and wifi connection.

By copy pasting the current stages, it is quite easy to get something working.

I created my own stage 2.5, that contained the phase 03-install-packages with a file named 00-packages that contains the additional packages I needed.

Then, I created the phase 04-setup that contains the file 02-run.sh with the final commands to run.

Then, i run it with the environment variable STAGE_LIST="stage0 stage1 stage2 stage2.5", then moved stage2/EXPORT* into stage2.5.

To build for the arm64, I simply need to git co arm64.

It also provides a build-docker.sh script that I can use to avoid installing the dependencies in my machine.

The only drawback is that it does not provide, like vmdb2, a way to store a partial chroot, so I cannot (easily?) cache intermediary steps in earthly.

Also, since earthly doe not support bind mounting, it cannot mount /dev and therefore fails when accessing /dev/loop0p*.