Registry

There is no database and no consensus protocol. The registry is a set of JSON documents in a bucket, written with conditional PUTs. Every record carries v: 1; readers must reject unknown versions.

All registry documents live under the root prefix _scafo, itself under the configured bucket.prefix when one is set. The one exception is the Celld fleet state, which lives outside _scafo.

#Key layout

The layout is flat and kind-prefixed: one prefix per kind of record, so listing one kind never reads another kind's keys. That is what keeps _scafo/applications/ at exactly one key per application — the listing a node runs whenever the change hint has moved — where a layout filing an application's history beneath its own name would make each of those listings read every deployment and every deploy of the whole installation just to learn the names.

Listing prefixes used by the daemon: _scafo/nodes/, _scafo/operators/, _scafo/identities/, _scafo/placement/, _scafo/observe/, _scafo/domains/, _scafo/applications/, _scafo/deployments/<name>/, _scafo/deploys/, _scafo/deploys/<name>/, _scafo/tls/objects/, _scafo/tls/locks/.

_scafo/deploys/ is listed at its root by the coordinator's reaper: one listing for the whole installation per sweep, rather than one per application.

Every per-application prefix is listed with its trailing separator, so a listing under shop never reaches shop-preview or shop2.

A name under _scafo/applications/ that is not an application name, and a name under a deploys prefix that is not a deploy id, are both skipped rather than read: they name no record of scafo's.

#Id rules

#Record shapes

#RegistryMeta — _scafo/meta.json

Written at bootstrap by the first node to arrive, which also proves the bucket honours conditional writes and ranged reads. A meta document announcing another version is store-nonconforming.

#FreezeRecord — _scafo/freeze.json

The fleet's deploy freeze. Its absence is what "this fleet takes deploys" means; there is no unfrozen record.

Written by one conditional create, so freezing twice is one freeze: a second scafo fleet freeze finds the record already there and neither the stamp nor the reason moves. Lifting it is one unconditional delete.

While it stands, every node refuses deploy intake with deploys-frozen (409) before anything is created, and the refusal names when, by whom, why and that scafo fleet unfreeze lifts it. Nothing else changes: serving, reconciliation, retirement and scafo app delete all carry on.

#ChangeHint — _scafo/changes.json

One object that says the registry moved since you last looked, so an idle installation costs one GET a pass instead of a listing and a read of every record it holds.

Written blind — no precondition, last writer wins — after a successful write under _scafo/applications/, _scafo/placement/ or _scafo/domains/, by a decorator every registry context is built with. So a bucket-direct scafo app retire run from an operator's laptop hints exactly as a deploy through a node does, fifty concurrent deploys never contend on it, and a write that failed hints nothing.

Node records are deliberately not hinted: a heartbeat rewrites one every 30 seconds, and a token minted for each would cost every node a listing several times a minute. A node that arrives, starts draining, or changes what it serves mints a token explicitly, because the fleet's routing is rendered from that.

It is a hint and never authority:

#CoordinatorLease — _scafo/coordinator/lease.json

The holder renews at a third of ttlMs, which is also the coordinator loop's tick.

#NodeRecord — _scafo/nodes/<id>.json

Registration guards: an id belonging to an operator key or to another identity is reference-held; a record with a different public key is reference-held; changing a node's id is immutable.

#IdentityReservation — _scafo/identities/<id>.json

Durable first-claim record. Deleting a node or an operator deliberately leaves this behind: an id is a public-key identity, not a reusable name.

#OperatorRecord — _scafo/operators/<id>.json

#ApplicationRecord — _scafo/applications/<name>.json

The unit scafo deploys, places, serves, blocks, retires and deletes.

v and name are the identity and never change; an update that alters either is immutable. Everything else is mutable state, and every mutation is a functional CAS update that preserves the fields it does not own — so a deploy can never drop blocked or labels.

There is no create command and no create route: the first deploy creates the application, a retried create returns the existing record unchanged, and every other setter refuses a missing one with not-found.

#PlacementRecord — _scafo/placement/<name>.json

Absence means no process may start. A write under a stale fence is lease-lost.

#DeploymentRecord — _scafo/deployments/<name>/<version>.json

One record per thing Celld serves, filed under the version Celld assigned and nothing else.

Immutable: created, read and listed, never updated. Recording the same deployment again is idempotent as long as every field but createdAt matches; otherwise immutable, Deployment already exists. Deletion is the reaper's or scafo deployment prune's.

#DeployRecord — _scafo/deploys/<name>/<id>.json

One deploy, from the moment a node accepted it to the moment it stopped moving. The state lives here rather than in the accepting node's memory, so any node of the installation answers a poll and a node restarting mid-deploy leaves an answer rather than a forgotten id.

deployed and failed are terminal: a deploy that reached either never moves again, and a later write to it is cas-conflict, Deploy has already finished.

GET /v1/deploys/:id reaches the deploy-timeout judgement itself, at read time, on exactly the same three conditions, and answers failed without writing anything — so nobody waits for a sweep to hear that a deploy's node is gone. The sweep is what stores that answer; the two never disagree about the outcome, only about when it is recorded.

A poll for an id whose record is gone is not-found, and the detail says the record is kept for seven days after the deploy finishes.

#TrustPolicy — _scafo/trust/<name>.json

subjects are patterns where * matches any run of characters. One policy belongs to one application, exactly, and it may exist before the application does: an admin pre-grants a name, and the first deploy creates the application under it.

#DomainClaim — _scafo/domains/<host>.json

The key is the host, lowercased and held to the claim rule: fully qualified, at most 253 characters, at least two labels of 1–63 characters from [a-z0-9-] not starting or ending with a hyphen, a last label that is not all digits, and no wildcard, port, scheme, path, trailing dot or underscore. A host equal to or under one of the installation's platform domains is refused as well, because every host there is rendered from an application's name. One application holds at most 64 of them.

A host already claimed by another application is reference-held, and so is a host one label under a base somebody outside the application's own <name>-… family claimed with its subdomains, or a wildcard claim on a base where such an application already holds an exact host one label under it.

The claim is mirrored into that application's domains array — and into wildcardDomains when it covers its subdomains — referent-first: the application is checked, then the claim is created, then the application lists the host. A release does it the other way round, dropping the host from the application before the claim goes, so nothing ever points at a claim that is about to disappear. Both orders make the call safe to re-run.

A claim is created, upgraded from exact to covering, and deleted; it is never downgraded, because dropping the flag silently would take every subdomain out of routing.

#TLS object — _scafo/tls/objects/<key>

The value is stored in a JSON envelope, and the keys beneath the prefix are the caller's, not the registry's — nothing here parses them.

An object under this prefix that is not such an envelope is store-nonconforming.

#TLS lock — _scafo/tls/locks/<name>

A lock currently held is cas-conflict; a write by a holder that lost it is lease-lost. An unparseable refreshedAt counts as stale.

#CAS operations

The store must support conditional PUT with both If-Match and If-None-Match.

Documents are serialized as pretty-printed JSON (2-space indent).

#Retry and backoff

updateJson retries only cas-conflict by default, with full jitter.

The mutate function may run more than once and must stay free of side effects. An AbortSignal stops the loop before another read or conditional write once the caller loses authority.

A store that does not honour conditional writes, or answers a ranged read with the whole object or the wrong bytes, is caught at bootstrap by the probe object and reported as store-nonconforming: Store does not honour conditional writes for the write steps, Store does not honour ranged reads for the read. A store that cuts a listing short, or returns no ETag, is store-nonconforming too.

A listing must carry an ETag for every key it names, in the same form GET and PUT answer with. That is what lets a node list a kind and then read back only the records whose ETag has moved since it last looked, instead of reading all of them; a listing that carries none is store-nonconforming. Records are cached under the ETag the listing named, not the one the read answered with: they differ only when the record moved in between, and then the next listing has to see a difference and read it again.

#Celld fleet state

Each application has its own Celld fleet, whose state lives at fleets/<name> — outside _scafo, because it is handed to celld as a bucket path of its own (--bucket <bucket>/<prefix>/fleets/<name>).

These keyspaces are Celld's. Scafo reads them, to tell what an application is serving and what it has published before, and writes none of them.

scafo deployment prune is the only thing that deletes under deploy/ and deploy-blobs/. It deletes under images/ only by reference — an image no kept deployment's manifest names, once it is older than the age window — and never touches queues/.

scafo app lease drop is the only thing that deletes under nodes/: one lease key the store will not read, on an operator's word, never a prefix. Both commands are unstable, run only against Celld releases this build was verified against, and warn on every run.

#Observations

Observation flushes are Parquet files, one per flush, under a node-scoped and UTC-hour-partitioned prefix:

_scafo/observe/<node>/<yyyy>/<mm>/<dd>/<hh>/<epochMs>-<seq>.parquet

Pages are ZSTD-compressed. A key whose calendar partition disagrees with its timestamp is rejected when parsed, which protects the retention sweep. Read them back with scafo metrics, scafo traces, or any Parquet reader.