← Blog

gpuprov: a GPU machine with Kubernetes, on any cloud, from one command

github.com/DumbAI/gpu-provisioner

Renting a GPU should be the easy part of running your own model. In practice it is a scavenger hunt. The H100 you want is out of capacity in the region you tried, the next cloud has it but names it differently, its API reports "insufficient capacity" in a fourth way, and the VM you gave up on twenty minutes ago is still billing because the delete never went through.

gpuprov is the tool we built to make that one command. It is the provisioning layer under Sky Compute, and it is open source under the Apache 2.0 license: github.com/DumbAI/gpu-provisioner.

$ gpuprov provision --name dev --gpus L4:1 --cloud gcp --region us-central1 --region us-east1
VM 'dev' is UP
  Endpoint:    [email protected]:22
  Connect:     gpuprov ssh dev    (or plain: ssh dev)

Kubernetes cluster 'gpuprov-dev' is ready.
  Kubeconfig:  ~/.gpu-provisioner/kube/dev.yaml

$ gpuprov ssh dev -- nvidia-smi
$ gpuprov down dev

What it does

You ask for a GPU (--gpus L40S:1) or an exact instance type, on one cloud, in an ordered list of regions. gpuprov resolves that against the cloud's catalog, tries the cheapest matching offers in order, and moves on when one is out of capacity. When a VM is up it waits for SSH, turns the VM into a single-node k3s cluster with the NVIDIA runtime and device plugin, validates the cluster, and writes you a kubeconfig. gpuprov down terminates everything it created, and checks that it is gone.

It speaks to AWS, GCP, Lambda, DigitalOcean, Nebius, Hyperstack, Massed Compute, Verda, Prime Intellect and RunPod, plus a local Docker "cloud" that runs the same machinery end to end with no account. (RunPod pods are containers, so there you get a plain GPU machine without k3s.)

How it is built

One contract, many clouds. Every cloud adapter implements six idempotent primitives: plan, prepare, create, observe, delete and cleanup. Adapters never loop, sleep or retry. Everything stateful lives above them, once.

Durable state machines. Launch, teardown and cluster setup are explicit state machines, journaled in a local SQLite database. A shared executor owns retries with backoff, polling, deadlines, Ctrl-C, resuming after a crash, and a single-writer lease. Each step observes the cloud before it acts. So when your laptop sleeps halfway through a launch, the next run adopts the instance that was being created instead of creating a second one.

launch    Reconciling → Planning → [Preparing → Creating → Booting → Connecting]* → Ready
k3s       Vm(launch) → CheckingBootInstall → [Installing] → AwaitingApi → FetchingKubeconfig → Validating → Ready
teardown  Requesting → AwaitingGone → CleaningUp → Done | Leaked

Failover without leaks. A capacity error tears down what was started and moves to the next offer; a quota error skips the whole region. If teardown cannot be confirmed, the VM is recorded as LEAKED, and the next run must clean it up before it does anything else. A VM gpuprov cannot account for is a bug, not a cost you discover on the invoice.

k3s by default. On clouds that boot with cloud-init, k3s installs while the VM boots, and gpuprov verifies that over SSH, falling back to pushing the installer itself. The installer is lock-serialized and idempotent. On clouds with no inbound-port control, gpuprov tunnel reaches the API over SSH.

No server. One binary; all state is under ~/.gpu-provisioner.

A library, too

The binary is a thin layer over a Rust library, and we use the library in two places.

  • Live prices. The scanner behind Sky Compute's GPU price pages calls the same vendor API clients to list offers, prices and capacity every day. A field the scanner needs is added to gpuprov, never copied.
  • Workspaces (in development). These are apps such as ComfyUI on a GPU VM, with their files on a skyfs volume. They run as Temporal workflows, and the "make me a GPU cluster" step is gpuprov's launch and k3s machines, run inside one activity. The two layers of durability compose: Temporal keeps the workflow, and gpuprov's journal keeps each cloud call exactly-once, so a retried activity resumes the launch instead of buying a second VM.

Embedding it is also how bugs get found. Driving gpuprov from an orchestrator on a real AWS account (an L40S g6e.xlarge in us-east-2) surfaced a corner where a launch that ran out of retries could leave a finished state nested where the cluster machine would step it again. The fix routes every exit from the child machine through one function, with a regression test.

Where it stands

Every adapter is tested in CI against a stateful stub of its vendor's API, written from the vendor's API reference rather than from our own parser. The shared scenarios cover lifecycle, capacity failover, interruption and reuse, and a leak check. AWS is also covered through an emulator, and the Docker cloud end to end. A separate acceptance suite runs the full lifecycle against a real account and checks that nothing is left behind. It is the bar for calling a cloud supported, and its real-account runs are still to be recorded, cloud by cloud. Outside that suite, we have taken gpuprov through a full lifecycle on a real AWS account: launch, k3s with the GPU allocatable, and teardown.

Deliberately out of scope: one VM per name, terminate-only teardown (no stop/resume), no spot-preemption recovery, and single-node k3s only.

Try it

cargo install --git https://github.com/DumbAI/gpu-provisioner   # binary: gpuprov

# no cloud account needed: the Docker cloud
gpuprov provision --cloud docker --name demo --instance-type container
gpuprov ssh demo -- uname -a
gpuprov down demo

Documentation: dumbai.github.io/gpu-provisioner. Issues and pull requests are welcome. Adding a cloud is a checklist: the trait, a stub of its API, and docs.