Setting up a Kubernetes cluster on Raspberry Pis is not an original idea, but it's a fun experimentation, at least for me.

Raspberry Pis & read only SD cards

Raspberry Pis are often run using SD cards, even if you can use NVMe SSD nowadays.

The caveat of using SD cards, it's that they're prone to fail fast when doing of lot of write operations on it. Indeed, I've got a stack of non-working SD cards, thanks to the Pis I've used over the years.

As I'm planning to use a few Raspberry Pi 4s I already own, I want to improve the lifetime of the cards I'm going to use, and to do so, I will try to reduce the amount of write operations done on the SD cards by enforcing a read only system.

I'm using the official Pi OS Raspberry PI OS LITE (64-BIT).
After booting from this image, I'm doing the following changes:

• disable systemd services:
  • apt-daily-upgrade
  • apt-daily
  • bluetooth
  • hciuart (bluetooth related)
  • dphys-swapfile (SWAP related)
• uninstall packages:
  • avahi-daemon (*.local discovery)
  • modemmanager (3G/4G related)
  • bluez (bluetooth related)
  • wpa_supplicant (wifi related)
• disable features in /boot/firmware/config.txt:
  • bluetooth
  • fsck
  • swap
• mount files/directories in tmpfs:
  • /etc/resolv.conf
  • /var/lib/dhcp
  • /var/lib/NetworkManager
  • /var/lib/nfs
  • /tmp
  • /var/tmp
  • /var/log
  • /var/lib/logrotate
  • /var/lib/sudo
  • /var/lib/systemd/timesync
• limit log size
• update fstab to mount / & /boot/firmware as readonly

I was not able to configure everything perfectly from the first try, obviously, so I've reduced my feedback loop by working with virtual machines to iterate on it.

Here's a snapshot of my ansible configuration to reproduce this setup.

K3s on read only SD cards ?

Can you can run k3s and containerd on a read only system? yes, with some tinkering.
I made it work, but was it a good idea? no.
The system was forced to write too many files in the RAM, with tmpfs, to work.
I also tried to explore persisting permanent files, like the docker/containerd images before mounting the partitions as readonly, but it completely breaks upgrades of the pods.

The good thing about doing something as crazy as that, I learned a lot about K3s and its components.

(example: some service is trying to write files somewhere when I start a pod, what is this service? What's its role? etc)

When you start to use almost 3GB of RAM to run a single podinfo, you NEED to start reconsidering how you do things.

Finally, the compromise was to create a separate partition, which is not read only, and run kubernetes on it. Yes, I'm sometimes writing on the SD card (new pods, etc) but most the usual writes, like logs are not.

We will see how long my SD cards last 🤞