ci: make BSD VM build jobs resilient to transient failures

The freebsd/netbsd/dragonfly jobs run in vmactions VMs whose boot and
image provisioning flake transiently; a single hiccup aborted the whole
run and had to be restarted by hand. The existing dragonfly retry was
ineffective: it re-ran the VM action without tearing down attempt 1's
leftover libvirt domain, so attempt 2 hit "Guest name 'dragonflybsd' is
already in use" and timed out again.

Two layers of hardening:

 - In-VM pkg installs (freebsd, dragonfly) now retry with backoff and an
   explicit catalogue refresh, so a transient mirror failure no longer
   fails the build (netbsd already had mirror fallback + curl --retry).

 - Each BSD VM step is retried up to 3x, with a new shared
   .github/actions/libvirt-cleanup composite action destroying the
   leftover domain + orphaned qemu between attempts. Only the final
   attempt can fail the job; a success short-circuits the rest. Rolled
   out to freebsd and netbsd too (previously single-attempt) and added
   timeout-minutes to bound a fully-broken job.

Signed-off-by: Enrico Weigelt, metux IT consult <info@metux.net>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

.github/actions/libvirt-cleanup/action.yml

@@ -0,0 +1,25 @@
+name: 'libvirt VM cleanup'
+description: >
+  Tear down a leftover libvirt domain and any orphaned qemu process from a
+  failed VM attempt, so a subsequent retry can start from a clean state instead
+  of hitting "Guest name '<name>' is already in use" (which otherwise makes the
+  retry collide with attempt 1's leftover VM and time out again).
+inputs:
+  guest:
+    description: 'libvirt domain / guest name to destroy (e.g. dragonflybsd, freebsd, netbsd)'
+    required: true
+runs:
+  using: 'composite'
+  steps:
+    - shell: bash
+      run: |
+        set +e
+        echo "--> tearing down stale VM '${{ inputs.guest }}'"
+        sudo virsh destroy  "${{ inputs.guest }}" 2>/dev/null
+        sudo virsh undefine "${{ inputs.guest }}" --nvram 2>/dev/null
+        # the vmactions runner leaves a single qemu per job; kill any survivor
+        # so it releases the qcow2 image lock and the domain name.
+        sudo pkill -9 -f qemu-system 2>/dev/null
+        # give libvirt a moment to actually release the domain name + image
+        sleep 5
+        exit 0

.github/scripts/DragonFlyBSD/install-pkg.sh

@@ -2,8 +2,31 @@
 
 set -e
 
+# Retry a command a few times with linear backoff. pkg mirrors flake
+# transiently (catalogue refresh / package fetch), and a single network
+# hiccup must not abort the whole CI run.
+retry() {
+    n=0
+    max=3
+    while true; do
+        n=$((n + 1))
+        if "$@"; then
+            return 0
+        fi
+        if [ "$n" -ge "$max" ]; then
+            echo "--> '$*' failed after $max attempts" >&2
+            return 1
+        fi
+        echo "--> '$*' failed (attempt $n/$max), retrying in $((n * 10))s ..." >&2
+        sleep $((n * 10))
+    done
+}
+
+echo "--> refresh package catalogue"
+retry pkg update -f
+
 echo "--> install extra dependencies"
-pkg install -y \
+retry pkg install -y \
     curl \
     libdrm \
     libepoll-shim \

.github/scripts/freebsd/install-pkg.sh

@@ -2,8 +2,31 @@
 
 set -e
 
+# Retry a command a few times with linear backoff. pkg mirrors flake
+# transiently (catalogue refresh / package fetch), and a single network
+# hiccup must not abort the whole CI run.
+retry() {
+    n=0
+    max=3
+    while true; do
+        n=$((n + 1))
+        if "$@"; then
+            return 0
+        fi
+        if [ "$n" -ge "$max" ]; then
+            echo "--> '$*' failed after $max attempts" >&2
+            return 1
+        fi
+        echo "--> '$*' failed (attempt $n/$max), retrying in $((n * 10))s ..." >&2
+        sleep $((n * 10))
+    done
+}
+
+echo "--> refresh package catalogue"
+retry pkg update -f
+
 echo "--> install extra dependencies"
-pkg install -y \
+retry pkg install -y \
     curl \
     git \
     libdrm \

.github/workflows/build-xserver.yml

@@ -363,14 +363,53 @@ jobs:
     xserver-build-freebsd:
         runs-on: ubuntu-latest
         needs: ubuntu-fetch-pkg
+        timeout-minutes: 90
         env:
             MYTOKEN : ${{ secrets.MYTOKEN }}
             MESON_ARGS: -Dprefix=/usr -Dxephyr=true -Dwerror=true -Dxcsecurity=true -Dxorg=true -Dxvfb=true -Dxnest=true -Dxfbdev=false
         steps:
             - uses: actions/checkout@v6
             - uses: ./.github/actions/ubuntu-pkg-cache
-            - name: run in freebsd VM
-              id: xserver-build
+
+            # VM boot / image provisioning flakes transiently; retry the whole
+            # VM step up to 3x, tearing down the leftover domain between tries
+            # (see .github/actions/libvirt-cleanup). Only the last attempt is
+            # allowed to fail the job; a success short-circuits the rest.
+            - name: run in freebsd VM (attempt 1)
+              id: freebsd-1
+              continue-on-error: true
+              uses: vmactions/freebsd-vm@v1
+              with:
+                  envs: 'MYTOKEN MESON_ARGS'
+                  usesh: true
+                  release: "14.3"
+                  run:     ./.github/scripts/freebsd/run-xserver-build.sh
+
+            - name: cleanup stale VM before retry
+              if: steps.freebsd-1.outcome == 'failure'
+              uses: ./.github/actions/libvirt-cleanup
+              with:
+                  guest: freebsd
+
+            - name: run in freebsd VM (attempt 2)
+              id: freebsd-2
+              if: steps.freebsd-1.outcome == 'failure'
+              continue-on-error: true
+              uses: vmactions/freebsd-vm@v1
+              with:
+                  envs: 'MYTOKEN MESON_ARGS'
+                  usesh: true
+                  release: "14.3"
+                  run:     ./.github/scripts/freebsd/run-xserver-build.sh
+
+            - name: cleanup stale VM before final retry
+              if: steps.freebsd-1.outcome == 'failure' && steps.freebsd-2.outcome == 'failure'
+              uses: ./.github/actions/libvirt-cleanup
+              with:
+                  guest: freebsd
+
+            - name: run in freebsd VM (attempt 3)
+              if: steps.freebsd-1.outcome == 'failure' && steps.freebsd-2.outcome == 'failure'
               uses: vmactions/freebsd-vm@v1
               with:
                   envs: 'MYTOKEN MESON_ARGS'
@@ -387,8 +426,14 @@ jobs:
         steps:
             - uses: actions/checkout@v6
             - uses: ./.github/actions/ubuntu-pkg-cache
+
+            # VM boot / image provisioning flakes transiently; retry the whole
+            # VM step up to 3x, tearing down the leftover domain between tries
+            # (see .github/actions/libvirt-cleanup). Without that teardown the
+            # retry collides with attempt 1's leftover VM
+            # ("Guest name 'dragonflybsd' is already in use") and times out.
             - name: run in DragonFlyBSD VM (attempt 1)
-              id: xserver-build-1
+              id: dfly-1
               continue-on-error: true
               uses: vmactions/dragonflybsd-vm@v1.1.4
               with:
@@ -397,8 +442,31 @@ jobs:
                   release: "6.4.2"
                   run:     ./.github/scripts/DragonFlyBSD/run-xserver-build.sh
 
+            - name: cleanup stale VM before retry
+              if: steps.dfly-1.outcome == 'failure'
+              uses: ./.github/actions/libvirt-cleanup
+              with:
+                  guest: dragonflybsd
+
             - name: run in DragonFlyBSD VM (attempt 2)
-              if: steps.xserver-build-1.outcome == 'failure'
+              id: dfly-2
+              if: steps.dfly-1.outcome == 'failure'
+              continue-on-error: true
+              uses: vmactions/dragonflybsd-vm@v1.1.4
+              with:
+                  envs: 'MESON_ARGS'
+                  usesh: true
+                  release: "6.4.2"
+                  run:     ./.github/scripts/DragonFlyBSD/run-xserver-build.sh
+
+            - name: cleanup stale VM before final retry
+              if: steps.dfly-1.outcome == 'failure' && steps.dfly-2.outcome == 'failure'
+              uses: ./.github/actions/libvirt-cleanup
+              with:
+                  guest: dragonflybsd
+
+            - name: run in DragonFlyBSD VM (attempt 3)
+              if: steps.dfly-1.outcome == 'failure' && steps.dfly-2.outcome == 'failure'
               uses: vmactions/dragonflybsd-vm@v1.1.4
               with:
                   envs: 'MESON_ARGS'
@@ -409,13 +477,52 @@ jobs:
     xserver-build-netbsd:
         runs-on: ubuntu-latest
         needs: ubuntu-fetch-pkg
+        timeout-minutes: 90
         env:
             MESON_ARGS: -Dwerror=true -Dxephyr=true -Dxorg=true -Dxvfb=true -Dxnest=true -Dxfbdev=false
         steps:
             - uses: actions/checkout@v6
             - uses: ./.github/actions/ubuntu-pkg-cache
-            - uses: vmactions/netbsd-vm@v1.2.3
-              id: vm
+
+            # VM boot / image provisioning flakes transiently; retry the whole
+            # VM step up to 3x, tearing down the leftover domain between tries
+            # (see .github/actions/libvirt-cleanup).
+            - name: run in netbsd VM (attempt 1)
+              id: netbsd-1
+              continue-on-error: true
+              uses: vmactions/netbsd-vm@v1.2.3
+              with:
+                  envs: 'MESON_ARGS'
+                  usesh: true
+                  release: '10.1'
+                  run:     ./.github/scripts/netbsd/run-xserver-build.sh
+
+            - name: cleanup stale VM before retry
+              if: steps.netbsd-1.outcome == 'failure'
+              uses: ./.github/actions/libvirt-cleanup
+              with:
+                  guest: netbsd
+
+            - name: run in netbsd VM (attempt 2)
+              id: netbsd-2
+              if: steps.netbsd-1.outcome == 'failure'
+              continue-on-error: true
+              uses: vmactions/netbsd-vm@v1.2.3
+              with:
+                  envs: 'MESON_ARGS'
+                  usesh: true
+                  release: '10.1'
+                  run:     ./.github/scripts/netbsd/run-xserver-build.sh
+
+            - name: cleanup stale VM before final retry
+              if: steps.netbsd-1.outcome == 'failure' && steps.netbsd-2.outcome == 'failure'
+              uses: ./.github/actions/libvirt-cleanup
+              with:
+                  guest: netbsd
+
+            - name: run in netbsd VM (attempt 3)
+              if: steps.netbsd-1.outcome == 'failure' && steps.netbsd-2.outcome == 'failure'
+              uses: vmactions/netbsd-vm@v1.2.3
               with:
                   envs: 'MESON_ARGS'
                   usesh: true