skyfs: a POSIX filesystem that lives in your bucket
A rented GPU is the most disposable computer you will ever use. You stop it the moment you are done, because it bills by the hour, or it disappears because spot capacity was reclaimed. What should not disappear with it:
- the custom nodes you installed;
- the Python environment it took an hour to get right;
- forty gigabytes of models;
- your outputs.
The usual answers each give something up:
- A network volume is tied to one data center of one cloud.
rsyncto object storage at shutdown loses everything since the last sync.- An S3 FUSE mount such as mountpoint-s3 refuses the operations real applications do: random writes, in-place edits, appends, renames, hard links.
skyfs is our answer, and the storage layer of Sky Compute: a POSIX filesystem whose only durable copy lives in blob storage you own (Cloudflare R2, S3). It is open source under the Apache 2.0 license: github.com/DumbAI/skyfs.
skyfs create --store s3://bucket/tenant --region us-east-2 # prints a volume id
skyfs mount --store s3://bucket/tenant --volume <id> --cache-dir /var/cache/skyfs \
--replication async --max-lag 1s /mnt/data
What it does
- Mounts in seconds, whatever the volume size. Only metadata is
downloaded at mount, so every file is listed and
stat-able at once. - Reads on demand. File contents are fetched by range on first read, verified and cached on local disk.
- Everything a filesystem should do works. That covers random and partial writes, in-place edits, appends, truncate, file and directory renames, hard links, symlinks, chmod and chown, and xattrs. Each one is a small log record, never a rewrite of existing objects.
- Synchronous or asynchronous replication, your choice.
syncacknowledges a write once the bucket has it.asyncacknowledges locally and uploads within--max-lag(one second by default). - Standalone. A daemon or sidecar that needs nothing but the bucket: no metadata service, no control plane. On Kubernetes it authenticates with OIDC workload identity: the pod's token is exchanged for an hour-long credential narrowed to its one volume, with no stored secret. For stores without identity federation, such as R2, it uses keys from a mounted secret.
- On a Mac, too, with no kernel extension:
skyfs mountserves the volume over NFSv3 on localhost and mounts it with the system's own client.
How it works
Every change to the filesystem is an operation appended to an immutable
write-ahead log. Segments of that log are uploaded with conditional writes
(If-None-Match: *), so an object is never overwritten. What the bucket holds
is a snapshot plus a short log tail. The snapshot's manifest is the
restore pointer: the log position it reflects.
app ──syscalls──▶ kernel FUSE ──▶ skyfs daemon
├─ metadata: SQLite on local disk = apply(snapshot, log tail)
├─ open segment: local, append-only
├─ data cache: blocks of immutable objects
├─ replicator: seal → PUT (sync or async)
└─ restorer: log → new snapshot (background)
bucket: volumes/<id>/ snapshots/ · wal/<seq>.seg · packs/ · blobs/ · volume.json
While a volume is mounted, the daemon folds the log into a new snapshot in the background, and again at shutdown. The log therefore only ever holds a few minutes of changes, and mounting replays seconds of it, not the volume's whole history.
One writer at a time, enforced by the log. A volume is mounted by one writer. If two mount it anyway, the newer one fences the older through the log: the stale writer's commits fail, and the volume never forks.
What it guarantees
The design is built around one invariant: after any crash, the volume is the result of a prefix of the operations, never a mixture.
| Event | Result |
|---|---|
| Daemon or container crash, same disk | Nothing acknowledged is lost: the local log is recovered and uploaded |
Machine lost, sync replication |
Nothing acknowledged as durable is lost |
Machine lost, async replication |
At most max-lag of operations lost, and reported at the next mount |
| Corrupt or truncated object | Detected by checksums and a hash chain, never served; mount refused |
| A stale writer after a new mount | Its commits fail; the new writer's history wins |
| Bucket unreachable | Cached reads work; writes continue locally until the local log is full |
How it is tested
A filesystem earns trust through its test suite. skyfs has run through the kernel on Linux:
- pjdfstest: the full suite, 8,798 tests.
- xfstests: the generic
quickgroup, 650 tests. The 12 exclusions are each justified in the repo. - Fault injection: at every bucket mutation, every point of recovery, and every torn byte of the local log.
- The NFS transport, tested on macOS and Linux.
Running it under an app
skyfs is designed to run beside the app that uses it, so the app never holds the bucket's keys. On Kubernetes that means a privileged native sidecar that propagates its FUSE mount into the app container. Getting that right took some lessons, now written up in the README:
- Use a native sidecar (an init container with
restartPolicy: Always). The kubelet then stops the app before skyfs, so files are closed when skyfs drains its log. - Mount the volume one level below the shared host directory. A daemon
killed with
kill -9leaves a dead mount behind. If that dead mount sits on the shared directory itself, every later container fails to start. One level down, the next sidecar can clear it. - Forward SIGTERM. As a container's PID 1, a process ignores signals it has no handler for.
That is how Sky Compute's workspaces (in development) use it. We ran ComfyUI on an AWS L40S with its release, its Python environment, models directory and projects on a skyfs volume in S3. We stopped it, and the volume drained cleanly to a final snapshot, ready to mount on the next machine, on any cloud.
Where it stands
v1 is implemented and tested end to end, through FUSE on Linux (the production path) and through NFS on macOS and Linux. The design document covers the format, the alternatives we rejected and the open questions.
It is deliberately not:
- a shared filesystem: one writer per volume;
- a cache-coherent filesystem across machines;
- a Windows mount.
Read the design and the log format, and file issues on the repo.