gpuprov: a GPU machine with Kubernetes, on any cloud, from one command
Renting a GPU should be the easy part of running your own model. In practice it is a scavenger hunt. The H100 you want is out of capacity in the region you tried, the next cloud has it but names it differently, its API reports "insufficient capacity" in a fourth way, and the VM you gave up on twenty minutes ago is still billing because the delete never went through.
gpuprov is the tool we built to make that one command. It is the provisioning layer under Sky Compute, and it is open source under the Apache 2.0 license: github.com/DumbAI/gpu-provisioner.
$ gpuprov provision --name dev --gpus L4:1 --cloud gcp --region us-central1 --region us-east1
VM 'dev' is UP
Endpoint: [email protected]:22
Connect: gpuprov ssh dev (or plain: ssh dev)
Kubernetes cluster 'gpuprov-dev' is ready.
Kubeconfig: ~/.gpu-provisioner/kube/dev.yaml
$ gpuprov ssh dev -- nvidia-smi
$ gpuprov down dev
What it does
You ask for a GPU (--gpus L40S:1) or an exact instance type, on one cloud,
in an ordered list of regions. gpuprov resolves that against the cloud's
catalog, tries the cheapest matching offers in order, and moves on when one is
out of capacity. When a VM is up it waits for SSH, turns the VM into a
single-node k3s cluster with the NVIDIA runtime and device
plugin, validates the cluster, and writes you a kubeconfig. gpuprov down
terminates everything it created, and checks that it is gone.
It speaks to AWS, GCP, Lambda, DigitalOcean, Nebius, Hyperstack, Massed Compute, Verda, Prime Intellect and RunPod, plus a local Docker "cloud" that runs the same machinery end to end with no account. (RunPod pods are containers, so there you get a plain GPU machine without k3s.)
How it is built
One contract, many clouds. Every cloud adapter implements six idempotent
primitives: plan, prepare, create, observe, delete and cleanup.
Adapters never loop, sleep or retry. Everything stateful lives above them,
once.
Durable state machines. Launch, teardown and cluster setup are explicit state machines, journaled in a local SQLite database. A shared executor owns retries with backoff, polling, deadlines, Ctrl-C, resuming after a crash, and a single-writer lease. Each step observes the cloud before it acts. So when your laptop sleeps halfway through a launch, the next run adopts the instance that was being created instead of creating a second one.
launch Reconciling → Planning → [Preparing → Creating → Booting → Connecting]* → Ready
k3s Vm(launch) → CheckingBootInstall → [Installing] → AwaitingApi → FetchingKubeconfig → Validating → Ready
teardown Requesting → AwaitingGone → CleaningUp → Done | Leaked
Failover without leaks. A capacity error tears down what was started and
moves to the next offer; a quota error skips the whole region. If teardown
cannot be confirmed, the VM is recorded as LEAKED, and the next run must
clean it up before it does anything else. A VM gpuprov cannot account for
is a bug, not a cost you discover on the invoice.
k3s by default. On clouds that boot with cloud-init, k3s installs while
the VM boots, and gpuprov verifies that over SSH, falling back to pushing the
installer itself. The installer is lock-serialized and idempotent. On clouds
with no inbound-port control, gpuprov tunnel reaches the API over SSH.
No server. One binary; all state is under ~/.gpu-provisioner.
A library, too
The binary is a thin layer over a Rust library, and we use the library in two places.
- Live prices. The scanner behind Sky Compute's GPU price pages calls the same vendor API clients to list offers, prices and capacity every day. A field the scanner needs is added to gpuprov, never copied.
- Workspaces (in development). These are apps such as ComfyUI on a GPU VM, with their files on a skyfs volume. They run as Temporal workflows, and the "make me a GPU cluster" step is gpuprov's launch and k3s machines, run inside one activity. The two layers of durability compose: Temporal keeps the workflow, and gpuprov's journal keeps each cloud call exactly-once, so a retried activity resumes the launch instead of buying a second VM.
Embedding it is also how bugs get found. Driving gpuprov from an orchestrator
on a real AWS account (an L40S g6e.xlarge in us-east-2) surfaced a corner
where a launch that ran out of retries could leave a finished state nested
where the cluster machine would step it again. The
fix routes every exit
from the child machine through one function, with a regression test.
Where it stands
Every adapter is tested in CI against a stateful stub of its vendor's API, written from the vendor's API reference rather than from our own parser. The shared scenarios cover lifecycle, capacity failover, interruption and reuse, and a leak check. AWS is also covered through an emulator, and the Docker cloud end to end. A separate acceptance suite runs the full lifecycle against a real account and checks that nothing is left behind. It is the bar for calling a cloud supported, and its real-account runs are still to be recorded, cloud by cloud. Outside that suite, we have taken gpuprov through a full lifecycle on a real AWS account: launch, k3s with the GPU allocatable, and teardown.
Deliberately out of scope: one VM per name, terminate-only teardown (no stop/resume), no spot-preemption recovery, and single-node k3s only.
Try it
cargo install --git https://github.com/DumbAI/gpu-provisioner # binary: gpuprov
# no cloud account needed: the Docker cloud
gpuprov provision --cloud docker --name demo --instance-type container
gpuprov ssh demo -- uname -a
gpuprov down demo
Documentation: dumbai.github.io/gpu-provisioner. Issues and pull requests are welcome. Adding a cloud is a checklist: the trait, a stub of its API, and docs.