Using Chroot and QEMU for "cross-compilation"
Before I start, a short disclaimer:
I have not used GitHub Actions before, nor I am a chroot expert, so a better solution to my problem may exist. However, I hope that my personal experience may help anyone trying to start a similar project.
Back to the post. After getting tired of manually building and testing MAMBO — a dynamic binary modification (DBM) system I am the maintainer of — on different architectures, I finally decided to create a simple GitHub Action to do it for me. When creating it, I had two main requirements in mind. First, it must easily run in GitHub Actions, so no interactive inputs, etc. Secondly, it must build and run all three implementations: Arm, Arm64 and RISC-V (64-bit).
On my own machine (M2 MacBook Pro) I use native builds for Arm64, and the complete system emulation with QEMU for RISC-V. My first thought was to run a QEMU VM on the GitHub runner, similarity to my local setup. However, communicating with the VM from inside the GitHub Action seemed like a lot of trouble. Of course, I could cross-compile MAMBO and then run it under user space emulation in QEMU, however MAMBO is not setup for cross-compilation out of the box. After a bit of searching around, it turned chroot and the user space emulation with QEMU can be used together to enable something akin to a regular cross-compilation without any changes to MAMBO and its build process. So, let’s do it!
First things first, we need an chroot environment that we can use. Luckily, Ubuntu (and many other distributions) provide a debootstrap tool that can set up a filesystem for a specific Ubuntu version and architecture. For example, to download and setup Arm64 Ubuntu 20.04 we need to run:
sudo debootstrap --arch=arm64 focal /tmp/chroot/focal-arm64 http://ports.ubuntu.com/ubuntu-ports
However, this is going to fail if we use a non-Arm64 machine, as the part of the configuration process requires running binaries from inside the new filesystem, and we cannot just run Arm64 binaries on any system. This is, however, something debootstrap is able to handle by utilising a two-step bootstrapping process. First, we can download required files without running the configuration step be passing the --foreign flag:
sudo debootstrap --arch=arm64 --foreign focal /tmp/chroot/focal-arm64 http://ports.ubuntu.com/ubuntu-ports
Then we can run the configuration step under QEMU and chroot. However, before doing that, we need to make sure qemu-aarch64-static compiled for the host architecture is inside the new filesystem. This can be easily done with:
sudo cp /usr/bin/qemu-aarch64-static /tmp/chroot/focal-arm64/usr/bin/
Now, executing the chroot command running the configuration step should complete without any issues:
sudo chroot /tmp/chroot/focal-arm64/ /usr/bin/qemu-aarch64-static /bin/bash -c '/debootstrap/debootstrap --second-stage'
And that is almost it. We can finally copy the project we want to build inside the new filesystem, e.g.:
sudo cp -r $GITHUB_WORKSPACE/mambo /tmp/chroot/focal-arm64/root
Then what is left is to build and run it:
sudo chroot /tmp/chroot/focal-arm64/ /usr/bin/qemu-aarch64-static /bin/bash -c 'sudo apt-get update; sudo apt-get upgrade -y; sudo apt-get -y install build-essential libelf-dev ruby; cd /root/mambo/; make; /root/mambo/dbm /bin/ls'
And here we are, a simple method to compile MAMBO for Arm64 (or any other architecture) on a non-native system without using a regular cross-compiler or a complete system emulation. The developed GitHub Action can be found on GitHub here.
Finally, some useful resources I used while developing the described automation:
02/06/2024 Update: Changes to the disclaimer and the workflow link. Remove statement about Arm64 runners.
12/06/2024 Update: Various minor grammar fixes.