Operating a fleet

#Boot sequence

What the node prints, once each, at boot:

#Graceful shutdown

SIGINT or SIGTERM runs, in this order:

#Stop grace

# Compose
stop_grace_period: 50s

# systemd
TimeoutStopSec=50

The grace is a ceiling, not a fixed wait. With no requests in flight, a node normally stops much sooner.

#Draining a node

A graceful stop is the drain: drainingAt is written first, and the coordinator counts the node out on its next pass. A node that dies uncatchably cannot write it and must age past the 90-second liveness window before its applications are reassigned.

An ingress sharing the node's network namespace stays attached to the old namespace if only the node restarts. Restart the pair together:

docker compose restart node caddy

#Restarting or replacing a node

Whatever brings a node back — a restart, a re-created container, a new image tag, a reinstall — two things have to be the same on the other side:

• Its id. Each runtime process runs under it, and that is how the process finds what it left on disk.
• Its state directory. A write is acknowledged to the caller once a peer holds it there, before the bucket has a copy. Coming back on an empty directory is coming back without those writes. It also holds the ports each application is served on here, so a node that keeps it keeps publishing the same addresses; one that loses it hands its applications new ports.

An application's directory is kept on the node through every ordinary stop — retirement, blocking, reassignment, a drain. Only deleting the application from the registry clears it.

#Upgrading nodes

There are two upgrades, and they are not the same procedure. Which one you are doing is decided by one question: may the Celld the nodes run and the Celld they move to serve one fleet together? They may when the Celld version does not change, and when the two releases are a rolling pair upstream declared — 0.5.0 ↔ 0.5.1, 0.6.0 ↔ 0.6.1 and 0.6.1 ↔ 0.6.2, see mixed Celld versions. Then the upgrade rolls node by node. Any other move of Celld stops the whole installation, and that includes moving from 0.5.0 or 0.5.1 onto any 0.6.x, and from 0.6.0 straight onto 0.6.2. The image bundles Celld, so moving the image tag usually moves both.

#A rolling upgrade

Every node keeps the Celld it was already running, or moves to the other release of a rolling pair. This one rolls node by node.

scafo fleet freeze --reason "moving every node to Celld 0.6.1"

scafo health
scafo fleet nodes

scafo fleet unfreeze

#A stop-everything upgrade

Between two Celld releases that are not a rolling pair, a fleet takes one version: a node neither starts an application's process nor accepts a deploy for it while another answering node serving that application, or crash-looping on it, reports the other release (version-skew, 409). So this upgrade stops the whole installation rather than rolling it.

A Celld release can also change what an application's code must look like, apart from the upgrade itself. Scafo hands each application's Wrangler config and code to Celld untouched, so what an application needs on the release you move onto is in the notes of every release between it and the one you leave — onto 0.6.2, the v0.6.0, v0.6.1 and v0.6.2 release notes — and nothing in scafo stands in for reading them first.

scafo fleet freeze --reason "moving every node to a new Celld release"

scafo fleet nodes

scafo fleet unfreeze

Pin and review the Celld release along with the Scafo build.

#Rotating the ingress token

Expand, migrate, contract. The node always moves first: an ingress reload refills its certificate cache from storage, so it must find a node that already accepts the token it is about to present.

SCAFO_INGRESS_TOKEN="new,old"

SCAFO_INGRESS_TOKEN="new"

Serving is unaffected throughout: certificates come from the in-memory cache and routes from the last document that parsed, so a mismatched window stalls renewals and route updates rather than handshakes and requests.

#Rotating an operator key

scafo operator rotate --remote prod

The new key is registered as superseding the old one, proved against the node, and the old one is then dropped. --keep-old leaves both trusted; --purge-old deletes the old key file once it is untrusted. An operator cannot revoke its own key — rotation is the path.

#Taking things out of service

# Stop serving one application, reversibly
scafo app retire   --name shop-preview
scafo app unretire --name shop-preview

# Delete it once the 24h grace has passed, or immediately
scafo app delete --name shop-preview
scafo app delete --name shop-preview --force

# Stop an application entirely: deploys, runtime, routing and certificates (admin only)
scafo app block   --name shop
scafo app unblock --name shop

# Stop every deploy of the installation, leaving serving alone (admin only)
scafo fleet freeze --reason "quarterly maintenance"
scafo fleet unfreeze

Retiring takes the application out of the routing document and stops its processes, reversibly for as long as its grace runs. Blocking does the same and also refuses new deploys with application-blocked; it is an admin's decision about an application the fleet hosts. Both keep every record.

Freezing is narrower than either, and fleet-wide: it stops deploy intake only, everywhere, with deploys-frozen, and the refusal names when, by whom, why and that scafo fleet unfreeze lifts it. Processes keep running, routes stay published, reconciliation and retirement carry on, and scafo app delete still works. The freeze lives at _scafo/freeze.json; its absence is what "this fleet takes deploys" means, and scafo fleet nodes prints a line for it while it stands. Setting it twice keeps the first admin's record, stamp and reason included.

#Bucket growth

Celld keeps every deployment it has ever published for an application and deletes none of them, so a long-lived application's fleet prefix grows with every deploy. Nothing reclaims that on its own.

scafo deployment prune --name shop --dry-run
scafo deployment prune --name shop

It keeps the newest --keep (default 10) and deletes only what is also older than --older-than (default 24h) — both gates always weighed — and never a version a Celld pointer names. It then collects the shared asset blobs and saved container images no kept deployment references, once they are older than the same window. It reads and deletes the bucket directly and calls no node, but it does read the node records there: at least one node must be heartbeating, so the command knows which Celld release the installation runs.

Nothing runs it for you. Until scafo automates retention, schedule it yourself — a cron entry or a CI job per application is enough, with bucket credentials and nothing else. --yes answers the confirmation a terminal would be asked:

# every night at 03:00, keep the newest 10 deployments and anything younger than a day
0 3 * * *  scafo deployment prune --name shop --yes

A run that is refused (a Celld release this build was not verified against, no node heartbeating) exits non-zero and deletes nothing, so a failing schedule is safe to leave failing until you look.

#Durable Object storage

A Durable Object's storage grows in the bucket with every activation of its cell, and the superseded part stays unless something deletes it. celld cell gc --dry-run is Celld's command, not scafo's: a read-only report, per storage cell, of the superseded epochs no restore reads any more, with their sizes. It needs the bucket credentials, not a running node, and reads one application's fleet prefix:

AWS_ACCESS_KEY_ID=<bucket access key> AWS_SECRET_ACCESS_KEY=<bucket secret key> \
  celld cell gc --dry-run --json \
    --bucket s3://<bucket>/<prefix>/fleets/<application> \
    --endpoint <S3 endpoint> \
    --region <bucket region>

<prefix> is the node's SCAFO_S3_PREFIX; leave that segment out when there is none. --region defaults to us-east-1. The image carries celld, so it runs inside a node's container too: docker exec -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY <node> celld cell gc ….

Without --dry-run it refuses, and deletes nothing. The deleting happens inside the running fleet, and only when CELLD_LTX_RETENTION_SECS is set through runtime.celldEnv on every node — once every node runs 0.6.1. The report shows what that would remove. What the setting does, and when a cell's owner gets to deleting, is in Celld's documentation.

#What an installation costs its bucket

Scafo's own idle request rate is small, and flat in the number of applications: a node hosting three applications and a node hosting three hundred make the same requests on a quiet minute. Every number below is derived from the defaults on this page; change a default and the arithmetic moves with it.

Per node, per minute, on an installation where nothing is happening:

And on whichever node currently holds the coordinator lease, on top of that:

So roughly 20 requests per minute for an ordinary node and 28 for the coordinator, with a handful of extra reads whenever something actually changes — a listing of the kinds that could have moved, and a GET of each record whose ETag moved with them. A reap sweep past the first reads no deploy records at all: a terminal record never moves again, so it is read once and judged from what the coordinator already holds.

pnpm scale in the source repository counts these requests by driving the real reconcile, coordinator and deploy code against a metered store, from 5 applications to 5,000, and the test suite holds them to those budgets.

#Sizing a node

Measured with Celld 0.5.0 on two nodes serving ten small applications each, with nobody calling them, under node sizes from 1 CPU / 2 GB to 4 CPUs / 8 GB and a store answering in 0, 100 and 300 ms (pnpm scale:real in the source repository runs the same sweep on your machine):

What follows from it:

• Memory is what a node runs out of, not CPU. Count one runtime process per application the node serves and leave room for what they do under load: very roughly 25 idle applications on a 2 GB node and 50 on a 4 GB one. Your applications' own memory comes on top, and traffic was not part of this measurement.
• The store's latency is the platform's latency. Every deploy, every convergence and every restart stretched with it, and nothing else in the sweep moved them. A bucket close to the nodes matters more than a bigger node.
• Nothing in the sweep was killed for memory, missed a deadline, failed a deploy or changed port, on any size.

Use replicas and node selectors to keep an application off nodes that have no room for it — see placement.

#Routine checks

scafo health                                     # this machine's node
scafo fleet nodes                                # the whole fleet, read from the bucket
curl -s -H "Authorization: Bearer $TOKEN" \
  http://127.0.0.1:7400/v1/state                 # this node's own view
scafo metrics --since 1h                         # what was served
scafo traces  --since 1h                         # what the nodes did

scafo health resolves the listener from the same environment and config file the daemon boots from — not from sail flags — so it follows a listener moved onto the tunnel. It is the container image's healthcheck. A listener chosen only with sail --listen is in neither source, so pass the same flag or set SCAFO_LISTEN.

GET /v1/state reports the node record, the coordinator lease and fence, whether this node coordinates, its processes, its placement views, its last reconcile pass, and any unplaced applications. Operator callers get it filtered to their application scope, marked scoped: true.

#Common responses

#Celld lease note

Each runtime process renews its lease every third of CELLD_TTL_MS (default 10000) and retries a failed renewal inside the remaining lifetime. A process that loses its lease logs a line starting with SELF-FENCE: — node_lease_watchdog_fence for an expired lease, node_lease_record_missing_fence for a removed record, node_lease_record_mismatch_fence for a replaced one — exits with code 3, and is restarted by the node. RUST_LOG=celld=info,store=debug on the daemon makes every process log each read and write of its own lease with its outcome.

Change CELLD_TTL_MS consistently on every node: an unplanned takeover waits up to the lifetime.

#A lease the store will not read

Each process keeps its lease at fleets/<name>/nodes/<node>.json, under the Scafo node id — the same name on every start, so an acknowledged write survives a restart. A store can corrupt exactly that object: it lists the key and answers a HEAD for it, and every GET of it fails. The process can then neither read its lease nor create one, logs refusing to install a lease over an unrecovered predecessor log, exits, and is restarted onto the same key, every few seconds, for as long as the object stands. Scafo reports that — the application leaves what the node serves and what the fleet routes to it — and repairs nothing by itself: the record is Celld's.

scafo app lease show --name shop
scafo app lease drop --name shop --node vps-a

show judges every lease of the application on what a listing, a HEAD and a GET each answer, and names the drop command for any it finds unreadable. drop prints that reading, asks, deletes the one key, and reads it back; the process writes a fresh lease the next time it starts. A lease that is still held is refused, because a running process may own it — stop that process first, or --force on your own word.

Runtime policy is set through runtime.celldEnv, --celld-env or SCAFO_CELLD_ENV. Scafo adds no Celld policy defaults.