Lately, I’ve been exploring x86 Linux machines for building and running a Kubernetes development environment. Since I’m using an Apple Silicon MBP as my daily driver, I had to get creative and tried a couple of options ending on a pretty robust workflow I wanted to outline. Throughout the process, I noticed parallels to current trends in cloud development environments, including the benefits of easily reproducible, instant environments for effective experimentation.
x86 on Apple Silicon
The most important requirement I needed to solve was the platform itself. Most deployment targets still run on x86 and some dependencies I wanted to use weren’t even built for Apple Silicon/ARMv8. Running a different architecture meant I couldn’t simply virtualize existing resources, but had to use some emulation layer to translate instructions from the x86 virtual machine ISA to the underlying ARM ISA.
Running with qemu, the go-to solution for binary translation, I tried the latest x86 Ubuntu Desktop release and quickly switched to the much more performant server distribution to reduce any unnecessary processing (and thus translation) overhead. While emulation is obviously not coming close to native speeds, I wouldn’t be able to tell the difference for most day-to-day tasks except for the compilation times for bigger components.
UTM as QEMU UI
UTM is a great visual frontend for QEMU and has all the features needed for managing local VMs. On top of emulated VMs, UTM also offers using the Apple Virtualization framework to achieve native speeds with macOS and Linux VMs. Cloning and sharing VMs also adds a nice UX on top of the standard QEMU processes and is great if you’re working in a team.
macOS VMs
Running macOS VMs in UTM has been an eye-opening experience for my development workflow. Previously, running experiments and installing system-level software like Kernel extensions has always felt like cluttering up my machine or worse, compromising security with untrusted code in the kernel. With virtualized environments, I don’t have to worry about security implications and can install any software packages I can think of. If I happen to break anything, I can just restart the process, no issues whatsoever.
Instant, reproducible (throwaway) development environments
The industry-wide adoption of containers, package managers, and tools like nixOS have put a spotlight on reproducible environments. Especially when building software products in teams and on a larger scale, we have to ensure consistency across environments. We’ve solved this problem on the project layer, but it applies to more than just local files in a repository. System-wide package installations like the Node.js or Go language versions quickly diverge, and tiny differences in configurations make for an error-prone experience.
Snapshotting the entire VM, however, captures the complete system, including kernel version, system libraries, user packages, systemd services, and more that you’re probably not even aware of. Apart from external devices and system hardware that may change, everything that influences your experience within the VM is stored somewhere in the filesystem. Sharing that same filesystem is essentially the same process you run through every time you use a container image.
Traditionally, version control has made experimenting on a codebase easy. With VM snapshots, you can experiment on your entire system. Everything you need is a disk image. From there, you can create snapshots by creating a copy, start up your instant environment by launching the VM, and make any changes you feel like. Feel encouraged to break your VM just to spin up another one, it’s that easy.
Cloud development environments like GitHub Codespaces and Gitpod offer the same ephemeral, reproducible, and collaborative experience, just managed and hooked up to your Git repository of choice. Manually copying around QEMU disk images may not be the most ergonomic setup, but the simplicity of it is almost ridiculous.
Bonus: VS Code Server
By now, you might be wondering how to sync changes between your development environment and local machine. This may be even more relevant if you’re running VMs on a remote instance. Luckily, we don’t have to resort to rsync here, VS Code Server allows you to run the entire development experience within the VM. Simply connect your VS Code instance as a thin client and enjoy all the benefits of your IDE combined with instant VMs.