Running buildah in forgejo-runner

I've been trying to get buildah working in forgejo-runner for building container images on CI, and it's been a frustrating process. There are some posts around the web about this, but nothing that sums up a working solution that doesn't simply disable all security measures. I eventually pieced it together with a lot of trial and error, so here's a summary to hopefully save somebody else (or me in the future) several days of pain.

The basic situation is that docker build requires a running docker daemon, either by sharing the host docker socket, which is a gaping security hole, or docker-in-docker (dind), which has its own set of problems. Either way, this won't work on public forges like codeberg.

Using buildah avoids this problem, but it still has requirements that are denied by standard security policies. If you try this on a stock configuration, you'll get errors because buildah tries to create a user namespace:

Error during unshare(CLONE_NEWUSER): Permission denied

Then tries to remount the image filesystem:

ApplyLayer stdout:  stderr: remount /, flags: 0x44000: permission denied exit status 1

A lot of advice on the web suggests using these docker options for the runner container, which will work, but is a bad idea because it completely disables security protections:

--security-opt seccomp=unconfined --security-opt apparmor=unconfined

You definitely shouldn't do this for a public runner in particular. A better solution is to use security profiles that only allow the additional operations on top of the default profiles used by docker. The following instructions are for forgejo-runner running in docker on Debian 13 (trixie) on x86-64, but should be straightforward to adjust for other situations.

First, you need a seccomp profile which is defined in a JSON file. This needs to be available to the runner in its container, mine is at /data/seccomp.json in the same directory as forgejo-runner's config.yml. The default profile is available at https://github.com/moby/profiles/blob/main/seccomp/default.json. Copy this, and add these rules to the syscalls dictionary:

{
    "names": ["unshare"],
    "action": "SCMP_ACT_ALLOW",
    "comment": "Allow unshare for buildah"
},
{
    "names": ["mount"],
    "action": "SCMP_ACT_ALLOW",
    "comment": "Allow mount for buildah"
}

Note that these rules allow any unshare and mount syscalls, although it should be possible to further restrict to only the specific arguments used by buildah. The default profile this is based on presumably allows many things which aren't actually required, so that could be further locked down as well.

Apparmor works differently, where profiles are installed on the system and referenced by name. Add this file at /etc/apparmor.d/runner-buildah on the host:

#include <tunables/global>

profile runner-buildah flags=(attach_disconnected,mediate_deleted) {
  #include <abstractions/base>

  network,
  capability,
  file,
  umount,
  mount,
  userns,

  signal (receive) peer=unconfined,
  signal (receive) peer=runc,
  signal (receive) peer=crun,
  signal (receive) peer=unconfined,
  signal (send,receive) peer=runner-buildah,

  deny @{PROC}/* w,
  deny @{PROC}/{[^1-9],[^1-9][^0-9],[^1-9s][^0-9y][^0-9s],[^1-9][^0-9][^0-9][^0-9/]*}/** w,
  deny @{PROC}/sys/[^k]** w,
  deny @{PROC}/sys/kernel/{?,??,[^s][^h][^m]**} w,
  deny @{PROC}/sysrq-trigger rwklx,
  deny @{PROC}/kcore rwklx,

  deny /sys/[^f]*/** wklx,
  deny /sys/f[^s]*/** wklx,
  deny /sys/fs/[^c]*/** wklx,
  deny /sys/fs/c[^g]*/** wklx,
  deny /sys/fs/cg[^r]*/** wklx,
  deny /sys/firmware/** rwklx,
  deny /sys/devices/virtual/powercap/** rwklx,
  deny /sys/kernel/security/** rwklx,

  ptrace (trace,read,tracedby,readby) peer=runner-buildah,
}

You can use a name other than runner-buildah, but remember to change the names in the profile itself or it won't work. After adding this file, either reboot the host or use apparmor_parser to add/replace the new profile:

apparmor_parser -r -W /etc/apparmor.d/runner-buildah

(As an aside, the ptrace permissions are required by gcc/clang sanitizers, so if your sanitizer builds don't work in a container, this is probably why. I'm not sure if/when they're required by buildah.)

Then, add the corresponding options to the runner's docker run command line. For forgejo-runner, this goes in the container section of config.yml:

container:
  options: "--security-opt seccomp=/data/seccomp.json --security-opt apparmor=runner-buildah"

Restart the runner container to complete the configuration. Now, jobs that use buildah to build images like this should work:

name: build-my-image

on:
  push:

jobs:
  build:
    runs-on: docker
    container:
      image: quay.io/buildah/stable
    steps:
      - name: 'Clone'
        run: git clone --depth 1 --branch ${FORGEJO_REF_NAME} ${FORGEJO_SERVER_URL}/${FORGEJO_REPOSITORY}.git .
      - name: 'Build'
        run: buildah --storage-driver=vfs bud --isolation chroot -t myorg/myimage

This is just a simple build job for demonstration, you'll need to come up with a suitable workflow for building and publishing your image. The buildah options used here are conservative for maximum compatibility (the main downside being that vfs is slow), so this should also work on shared forges like codeberg.

Some caveats: I'm not an expert in this sort of thing, and these profiles are definitely not as tight as they could be. In my case, I'm using this in a private runner for branches that only I can push in specific repositories in a private forge that runs in docker in a VM all behind a firewall, so runner security isn't much of a concern anyway. You shouldn't blindly copy this configuration for a public runner, although it's certainly better than using unconfined.

If you're more experienced in Linux security and spot any problems or potential improvements here, let me know via email and I'll update this article accordingly.