Nodes and the fleet

#Node identity

A node is its keypair. On first boot the daemon generates an Ed25519 pair and writes it, with the node id, to node.json in the state directory, owner-readable.

A pinned id that differs from the stored one replaces the identity: a new keypair is generated and written under it. A state file that exists but cannot be read as an identity is never silently replaced — the daemon refuses with invalid-input, because a new keypair under an id the registry still holds the old public key for would have every token rejected.

The id is also the name each runtime process runs under, and the state directory is where that process keeps the writes it has acknowledged before the bucket has them. So a node that comes back has to come back as the same id, on the same state directory. Restarting it, re-creating its container, moving it to a newer image: none of those may change either.

#Registration

On boot the daemon writes _scafo/nodes/<id>.json and claims _scafo/identities/<id>.json. Node and operator ids share one keyspace, and the registration guards enforce it:

The node record carries the node's public key, its addresses (internal address, API port), its labels, the versions of Scafo and Celld its daemon booted with, what it is serving, and its heartbeat and draining stamps. The versions are written at every boot and never by the heartbeat: they say what the running daemon started as. Before any of this, scafo sail runs celld --version on the configured binary and refuses to go on with capability-missing when it is older than 0.5.0, cannot be run, or prints no version.

#Heartbeat and liveness

A node counts as live when its heartbeatAt is inside the liveness window. A node that has never beaten is not live. A node carrying drainingAt is never live, whatever its heartbeat says.

Answering is the other half of the same question, and the one mixed-version checks read: a node heard from inside the liveness window, whatever it has said about leaving. A draining node is still answering until it is gone.

Placement and the routing document use liveness directly: a stale or draining node is dropped from the routing rather than left to an ingress health check.

Because liveness is an age rather than an event, no write announces it. Each node therefore re-reads the fleet's node records on a timer of their own, every 30 seconds by default (registry.nodeRefreshIntervalMs), instead of waiting for the registry change hint to tell it about a death nobody wrote down.

#The coordinator lease

Every node runs the coordinator loop and competes for one lease at _scafo/coordinator/lease.json. One holds it; the rest stay candidates.

A longer lease buys back no failover time. What a dead coordinator held is only reassigned once the fleet has decided the node is gone, and that decision is the liveness window — 90 seconds of missed heartbeats. Both clocks start at the same death, so any lease lifetime up to that window expires while the fleet is still waiting to notice. Shortening it only makes the holder write more often, forever, on an installation where nothing is happening.

The lease carries a fence that increases on every acquisition. Coordinator-only work — assignment, deletion, reaping — is stamped with the fence it was decided under, so a decision made by a previous holder cannot be applied after the lease moved.

One renewal may fail or run late and the lease still holds; a second miss inside the same TTL loses it.

The holder does three things per iteration: take or renew the lease, assign every application to the nodes that should serve it, and reap when a reap is due. Non-holders forward coordinator-gated requests to the holder.

The assignment pass is skipped outright when nothing it decides from has moved since the last completed pass — the applications' replicas, selectors, retired and blocked state, their placements, and the set of live nodes. Rendezvous hashing over the same inputs can only reach the same answer, so a pass over them would write what is already written. The fingerprint is recorded after the pass rather than before it, so a pass that failed half way is finished by the next one; a node that has just taken the lease compares against nothing, because the previous holder may have decided something it never saw.

#Draining

drainingAt is written first, before anything else stops: it is what tells placement the node is leaving rather than dying. A heartbeat landing between that write and the stop cannot undo it — a beat touches its own field only. The write is best effort and never throws, so an unreachable bucket delays shutdown briefly and changes nothing else.

An uncatchable death cannot write drainingAt. That node must age past the 90 s liveness window before its applications are reassigned.

#Transports

A node whose API, advertised address or runtime bind host reaches beyond loopback must state what network that is. Booting without doing so raises invalid-input naming the offending fields.

Defaults follow the transport:

0.0.0.0 is used rather than :: because a host with IPv6 disabled cannot bind ::. A wildcard bind is dialled locally on 127.0.0.1; every specific bind is dialled as it was bound.

With transport: wireguard, the node also refuses to boot until scafo wg init has put a keypair in the state directory: registering an address the tunnel does not carry is exactly what the transport decision was made to prevent.

#Labels

Node labels are key=value pairs an operator sets, and they are what application selectors match against.

scafo sail --node-labels tier=fast --node-labels region=eu-west-1

Set through --node-labels, one pair per flag; through SCAFO_NODE_LABELS as k=v,k2=v2; or through node.labels as a JSON object in the config file. The flag and the environment variable replace the file's set wholesale.

#Listing the fleet

scafo fleet nodes
scafo fleet nodes --json

Reads the bucket directly, so no node has to be up. One row per machine, judged against a single clock:

The header line gives the total and the live count. While a deploy freeze stands, the next line says since when, by whom and why. Then one line per binary counts nodes per version newest first — celld: 0.6.2 on 2 nodes, 0.6.1 on 1, unreported on 1. A record written by a build that did not keep the versions counts as unreported rather than as a version.

A second Celld version in that line is work left to do: an installation is upgraded by stopping every node and starting the new binaries, or one node at a time between a rolling pair of Celld releases upstream declared, and the line says how much of the fleet is still waiting.