A virtual machine costs less per hour than your current Kubernetes-based runner pool. That sounds like an easy CI saving.
But developers do not buy VM-hours. They need completed, trustworthy builds, delivered quickly enough that releases and reviews keep moving.
A runner that costs less per CPU-hour may be slower to provision, rebuild the same dependencies on every job, run too few builds at once or require more operational attention. Conversely, a fast expensive runner can remain idle most of the day.
For CI infrastructure, the unit worth measuring is cost per successful completed build, alongside queue time and reliability.
Separate three kinds of time
Runner analysis starts with a timestamped job lifecycle.
Queue time is how long a job waits before a suitable runner becomes available. Setup time covers VM or Pod provisioning, image preparation and dependency retrieval. Execution time includes the build and tests themselves.
Looking only at job execution hides wasted capacity. Looking only at the monthly infrastructure bill hides slow delivery.
For each workload class, collect these durations alongside the job's outcome. Separate Rust, container builds, Android builds and other workloads instead of combining them into one average. Their cache and CPU characteristics differ.
The actual utilization problem
Suppose a company keeps enough capacity for a brief afternoon burst but leaves the same nodes running all night.
That design might offer short queues, but utilization can be poor. The provisioned machine hours remain billable even when no work is running.
Autoscaling improves the situation only if it reacts quickly and if jobs fit the scaling units. A whole eight-vCPU runner reserved for a short single-threaded step can waste capacity even while the job is active. A spot interruption may create additional wasted work, retries and queue delay.
A useful calculation distinguishes:
- Paid runner/VM capacity hours.
- Time spent actively executing builds and tests.
- Successful versus failed, cancelled and retried jobs.
- Fixed capacity needed for predictable queue times.
- Cache and persistent-storage costs.
- Runner management and infrastructure overhead.
Build cost can then be approximated as all attributable infrastructure and operational costs divided by completed successful builds, using an agreed accounting period. The denominator matters.
A real migration signal, without an invented benchmark
In one earlier engineering workload, our team moved Rust CI runner capacity away from Kubernetes-hosted GitHub Actions runners using spot VMs toward bare-metal runner resources. The owner reported that builds on the previous setup took longer and cost more for that particular workload.
We have not retained a controlled before-and-after performance benchmark or fully comparable invoice periods for that change. The observation is useful as a reason to test another runner architecture, not proof that bare metal universally beats Kubernetes or spot instances.
The case also illustrates why you should include network and cache locality in the decision. A job's effective throughput can depend as much on its storage/cache behavior as on nominal core count.
Shared caches are useful, but unsafe without boundaries
A shared build cache can save time by avoiding repeated dependency downloads and compilation work.
It can also introduce subtle failures when jobs use different toolchains or share mutable files unsafely. For example, Android Gradle builds may need distinct plugin versions, JDKs and native toolchains. Rust builds depend on features, target triples and compiler versions. Cached objects should not be treated as interchangeable just because the repository name matches.
Design cache namespaces around the things that affect artifact validity. Keep the storage path stable and the keys specific. Ensure concurrent jobs cannot corrupt one another's state. Reuse dependency downloads where safe; isolate working directories and build outputs where needed.
A runner that is fast only with a warm cache should also be tested from a cold start. Otherwise a benchmark can hide the recovery cost after a cache miss, disk failure or new toolchain rollout.
When Kubernetes is the right runner platform
Kubernetes runners can be a good fit when an organization already operates the cluster, needs short-lived isolated Pods and wants autoscaling around the GitHub job queue.
However, cluster-node capacity, image download time, permissions and storage topology must all be considered. Ephemeral runner Pods are not equivalent to truly zero-cost idle capacity if the underlying nodes remain provisioned.
A scale-set design also has to answer what happens when a build needs Docker, BuildKit or another privileged facility. Giving every build broad host-level privileges is a different risk decision from launching an unprivileged test process.
Do not equate "runner Pod deleted" with "all secrets and caches are isolated." Verify the actual execution environment and credential lifetime. The companion GitHub Actions runner security guide covers those trust boundaries in detail.
When bare metal can be better
Bare metal becomes attractive for predictable, CPU-intensive or cache-heavy workloads with enough jobs to keep machines busy. Large local NVMe caches can reduce repeated downloads and build artifacts can benefit from stable I/O performance.
But a dedicated host has fixed monthly cost whether or not the queue is busy. It also needs disk management, patching, replacement capacity and an isolation strategy for concurrent jobs.
A hybrid pool can combine a small reliable baseline with burst capacity elsewhere. The optimal split depends on observed job arrival patterns and the queue-time objective. It is not determined by the largest advertised CPU count.
The benchmark that would actually decide
Before migrating, choose a week of representative work and record the same metrics on the current pool and on a small candidate pool:
- Successful jobs per billed hour.
- Median and high-percentile time from queued to completed.
- Cold-cache and warm-cache duration for the same job.
- Retry rate, interruption frequency and failed-run root causes.
- Effective cost including storage, network, idle capacity and management.
- Security isolation and recovery behavior under concurrent builds.
Test at least one high-concurrency period; an empty-queue benchmark is not enough.
The outcome should be a decision about workload placement, not a claim that one provider is always cheaper.
What to optimize first
Before buying more runners, investigate whether the existing fleet is actually running work. Then verify whether build time is consumed by CPU, storage, network, dependency resolution or avoidable repeated steps.
The most expensive runner is often the one that remains reserved without doing useful work. The second most expensive is the one that makes every engineer wait.
Big Bad Byte's DevOps engineering connects CI architecture with real workload measurements. A practical review should end with a cost and queue-time model that an engineering team can verify.