Rust CI runners: moving one workload from Kubernetes Spot to Hetzner

Autoscaling CI runners are not automatically the cheapest way to complete a build.

For one client's Rust workload, the team initially ran GitHub Actions Runner Controller (ARC) on Kubernetes workers backed by Spot capacity. On paper, it looked economical: runners scale with the job queue, and Spot capacity is discounted compared with on-demand instances.

In practice, the team found the combination too slow and too expensive for those builds. Runner compute was later moved to Hetzner.

The short answer

Kubernetes runners on Spot capacity are cheap per CPU-hour, but Rust CI pays for queue time, cold caches, long compilations and interrupted jobs. For one client's Rust workload, ARC on Kubernetes Spot proved too slow and expensive, so runner compute moved to Hetzner. Compare runners by cost per completed build, not hourly price.

What we can and cannot claim

We do not have verified before-and-after build minutes or billing totals for this case, so there is no percentage improvement to quote here.

What we have is a first-hand engineering account: the ARC and Spot setup did not suit this particular Rust workload, and a different runner placement did. ARC and Spot are still valid choices for other workloads. This article explains why Rust CI is often a hard fit for that model and how to measure the decision for your own pipelines.

Why Rust CI is a demanding runner workload

Rust builds have a few properties that magnify every inefficiency in the runner platform:

• Compilation is CPU-heavy and long. A clean build of a large workspace with many dependencies can take far longer than a typical interpreted-language test job.
• The build cache is large and valuable. The target directory, the Cargo registry and Git dependency caches can turn a long cold build into a short incremental one. Losing them costs real compile time.
• Cache validity is specific. Compiler version, target triple, enabled features and profile all affect whether cached artifacts are reusable.
• Disk I/O matters. Linking and incremental compilation read and write many files. Slow or network-attached storage can stretch a build even when CPU is idle.

A runner design that is fine for short, stateless jobs can be a poor fit for this profile.

Where the Kubernetes Spot model can lose time and money

These are the mechanisms worth checking in any ARC and Spot setup. They are general engineering points, not a diagnosis of this particular client's pipeline.

Ephemeral runners start cold. ARC typically creates a fresh runner Pod per job. Unless the cache is restored from somewhere, each job starts without the target directory and dependency downloads. Restoring a multi-gigabyte cache over the network on every job is itself time and transfer cost.

Node scale-up adds queue time. If the cluster autoscaler has to provision a new Spot node before the runner Pod can schedule, the job waits for the node, the image pull and the runner registration. Developers experience that as a slow build even though the compile itself has not started.

Spot interruptions waste the longest jobs. Spot capacity can be reclaimed. A short test job that is interrupted is a minor retry. A long Rust compile that is interrupted near the end loses most of its work and then runs again, often from a cold cache.

Resource requests and node shapes may not match. A runner Pod that requests less CPU than the build can use runs slower; one that requests more blocks capacity other jobs could use. Rust builds parallelize well during dependency compilation and poorly during final linking, so the right shape is not obvious.

Paid capacity is not only the Pods. Nodes that stay up between bursts, persistent volumes for caches, load balancers and the Kubernetes control plane all belong in the cost.

Each one is small on its own. Together, on a workload with long, cache-heavy jobs, they can make a cheap hourly price produce an expensive completed build.

What the comparison should measure

The useful comparison is not only the hourly price of a CPU core. It is how long the job waits, how long compilation takes and what the completed build costs across the whole runner setup.

We covered the general method in how to measure CI runner cost per successful build. For Rust specifically, collect per job:

1. Queue time from job queued to runner assigned.
2. Setup time for node provisioning, image pull, runner registration and cache restore.
3. Compile and test time, split into cold-cache and warm-cache runs.
4. Outcome: succeeded, failed, cancelled, retried or interrupted.
5. Attributable cost: compute, storage for caches, network transfer for cache restore and artifacts, and idle capacity.

Then divide total attributable cost by successful completed builds over the same period. Compare like with like: same commits, same workflows, same concurrency.

Checklist before moving runners

Before changing runner platforms, check:

• Which workflows dominate total runner time. Often a few Rust jobs consume most of it.
• How often the cache is actually hit, and how long a cache restore takes.
• How many jobs are retried, and how many retries follow a Spot interruption.
• Peak concurrency and how long jobs queue at peak.
• Whether builds need Docker, BuildKit or other privileged access, and how that is isolated.
• Who will patch, monitor and replace the new runner hosts.

If you can, run a small candidate pool alongside the existing one for a representative period, including a busy period, before switching.

Trade-offs of fixed runner capacity

Moving runners to a provider like Hetzner changes the trade-off rather than removing it.

Fixed hosts can keep a warm local cache on fast local disks and do not get reclaimed mid-build. That suits long, cache-heavy compilation. But fixed capacity is billed whether or not the queue is busy, so utilization becomes your problem. The hosts also need patching, disk cleanup, monitoring and a plan for when one fails.

Shared caches on long-lived hosts need boundaries. Separate cache keys by toolchain and target, keep working directories isolated between concurrent jobs, and decide how credentials are scoped when runners are not destroyed after each job.

A hybrid can also make sense: a small fixed pool for the heavy Rust jobs, with autoscaling capacity kept for short, stateless jobs where cold starts matter less.

Failure modes after the move

Watch for these once runners are on fixed hosts:

• Disk fills up with old target directories and the next build fails.
• Cache poisoning or corruption from concurrent jobs writing to the same paths.
• Queue time grows at peak because fixed capacity no longer scales.
• A host failure removes a large share of capacity at once.
• Toolchain drift between hosts makes builds pass on one runner and fail on another.

The engineering decision

This particular Rust CI workload called for a different trade-off than autoscaling Spot runners. Another workload, with shorter jobs or bursty demand, may still be better served by ARC on Kubernetes.

The decision should rest on queue time, compile time and cost per completed build for your own pipelines. If your CI bill keeps growing without builds getting faster, our cloud cost optimization work starts with exactly that measurement.

---