Celld runtime

Scafo requires Celld 0.5.0, and checks: scafo sail runs <binPath> --version before it opens the bucket and refuses an older or unreadable binary with capability-missing. The version it read is written into the node's record, and every node serving one application must report the same one, or the other release of a rolling pair upstream declared — see mixed Celld versions. The container image and the Linux installer ship Celld 0.6.2. Each application assigned to a node runs as one Celld process, started by the node's reconcile loop and stopped by it. Scafo owns the process arguments and part of its environment; everything else is upstream Celld policy, which you write through this node's configuration.

#Choosing the binary

The value is resolved against PATH unless it contains a directory separator. The official image ships /usr/local/bin/celld; the installer writes SCAFO_CELLD_BIN=/usr/local/bin/celld into the node's systemd unit. The same binary runs deploy builds, so a node whose binPath is wrong fails both at reconcile and at deploy time.

#Node-wide Celld policy

Variables handed to every runtime process this node starts:

Three spellings, as with node labels:

{
  "runtime": {
    "celldEnv": {
      "CELLD_MAX_RSS_MB": "512",
      "CELLD_MAX_RESIDENT_CELLS": "64",
    },
  },
}

Rules for the set:

• The flag and the environment variable replace the config file's set wholesale; they do not merge into it.
• Each --celld-env states one pair, and its value may hold anything, commas included. SCAFO_CELLD_ENV separates its pairs with commas, so a value written there cannot contain one; write such a value by repeating the flag, or as runtime.celldEnv in the config file.
• Everything after the first = is the value, so NAME=a=b gives a=b. Surrounding spaces are dropped on both halves.
• Names are letters, digits and underscores, never beginning on a digit. Values must be strings: a number or a boolean in the config file is refused rather than stringified.
• A duplicate name is refused rather than resolved. Text that names no variable at all is refused too.

Scafo adds no Celld policy defaults of its own. Whatever upstream Celld defaults to is what a process gets, apart from the telemetry switch below.

#Names Scafo refuses

Setting any of these through runtime.celldEnv is invalid-input ("Reserved environment variable"), refused at load rather than silently overwritten, because the driver writes them itself from this node's configuration, from the listeners its processes bind, and from the private directories it keeps for them:

Every other Celld variable is accepted as-is and handed to every process this node starts. Scafo keeps no list of its own: what a variable does, and which release introduced it, is upstream Celld's documentation. These are the node's Celld policy, not an application's variables: an application's variables travel in its own wrangler.jsonc or wrangler.json, inside the artifact, and the node never touches them.

#What Scafo passes to each process

#Arguments

The two ports come from the node's range, runtime.portRangeStart–runtime.portRangeEnd (24000–24998 by default). They are drawn at random and independently of each other — the internal port is never the public one plus one — and once drawn they belong to that application on this node and are reused on every later start. See port pairs. The advertised host is node.advertise's host when one is set, and node.listen's host otherwise — that is what a peer dials to reach the process that owns a cell, which is why a multi-node fleet needs runtime.bindHost off loopback and a stated transport.

Each application's Celld fleet state lives under fleets/<name> in the same bucket, outside the _scafo registry root.

#Environment

Three steps, each overriding the last — and that is the whole of it:

Only the last three differ between the applications this node serves: CELLD_WATCH and CELLD_ASSET_CACHE_DIR name the application, and CELLD_NODE is the same for all of them. Nothing else of an application's own is added — its variables travel in its own config, inside the artifact.

#Variables per application

An application's variables are not in the process environment at all: they live in the application's own wrangler.jsonc or wrangler.json, under vars, and Celld reads them from the config it was published with. The node hands that file to Celld exactly as the artifact carried it.

#Container engine

For container classes a node hands Celld the engine and nothing else. Two different processes need it, and each gets it out of the daemon's own environment:

DOCKER_HOST is passed on unread: Celld finds the engine's socket itself, at the default paths or at the unix:// URL it names. The publish builds or pulls a container image with the docker client on its PATH — /usr/local/bin/docker in the image, beside the esbuild CELLD_ESBUILD names — and on a bare host, whatever client the unit's PATH finds.

CELLD_DOCKER and CELLD_CONTAINER_PLATFORM reach the publish the same way, unread: the client to run in place of docker, such as podman, and the platform the image is built or pulled for, linux/amd64 unless it names another. Celld reads both in celld deploy alone, so they are set in the daemon's own environment — the env file, the Compose environment:, the systemd unit — and never in runtime.celldEnv, which reaches the runtime processes only, where Celld never reads them.

Celld's runtime settings for containers, such as CELLD_CONTAINER_RUNTIME and CELLD_CONTAINER_DNS, go in runtime.celldEnv like any other Celld variable; none of them is reserved.

What each of those variables does is Celld's, in its containers documentation.

Two more reach the publish that are not the engine's. CELLD_PYTHON_RUNTIME_DIR is where a Python Worker's build keeps the Pyodide runtime it downloads, and only the publish reads it. CELLD_MAX_ASSET_FILE_BYTES, the largest static asset file a deployment may carry (25 MiB unless set), is read on both sides: the publish checks it, and so does every runtime process serving the deployment. Raising it therefore means setting it twice, to the same value — in the daemon's own environment for the publish, and in runtime.celldEnv for every process — and a process left at the default refuses the deployment. Other runtime limits, such as CELLD_LTX_RETENTION_SECS and CELLD_MAX_DYNAMIC_WORKER_CODE_BYTES, are read by the runtime processes alone and go in runtime.celldEnv.

#Durability and the state directory

Each runtime process works out of its own directory under the node's state directory, and comes back under the node's own id:

Scafo derives all three and creates the directories 0700 before the process starts. You cannot set them: runtime.celldEnv refuses them with the rest of the reserved names.

Why it matters. With more than one node, a write is acknowledged to the caller as soon as a peer node holds it on its disk, in this directory — before the bucket has a copy of it. A process picks those writes back up at boot from the directory its predecessor of the same node name left behind, and serves its own copy to the peers doing the same. So two things have to hold:

• The state directory must persist, across a restart and across the node being re-created: a named volume, a retained PVC, a real directory on the host. A directory that is new on every start leaves acknowledged writes nowhere, and the application comes back missing its most recent ones.
• The node id must be stable. It is what a process comes back under; a node that renames itself cannot find what it left.

Left to itself Celld would pick a fresh name each start, a working directory under $TMPDIR new on every start, and one asset cache shared by every process on the machine — which is why Scafo sets these three rather than letting you.

<state dir>/celld/<application>/ is kept through every stop: retirement, blocking, reassignment, a drain, a daemon restart. The disk may still hold writes a peer needs, and the application may be served here again. It is removed only by the reconcile pass that finds the application gone from the registry altogether, never while a process for it is running here, and never on a pass whose registry read failed.

#Shutdown grace

CELLD_SHUTDOWN_TOTAL_MS is Celld's complete process shutdown bound. Scafo reads it out of runtime.celldEnv — not out of the daemon's own environment — and sizes its supervisor grace from it:

supervisor grace = CELLD_SHUTDOWN_TOTAL_MS + 5000 ms

With the upstream default of 40000 that is a 45-second grace between SIGTERM and the supervisor's hard kill, so a process always finishes draining before it is killed. The value must be a positive integer of digits only, no greater than 2147478647; anything else is invalid-input ("Invalid celld shutdown bound").

Whatever supervises the node has to be more patient than the node is with its children:

#Telemetry

CELLD_OTEL is the one Celld variable Scafo defaults for you. With observability.enabled on and no CELLD_OTEL in runtime.celldEnv, every runtime process is started with CELLD_OTEL=1. Set it yourself in runtime.celldEnv — to another value, or to 0 — and your value stands; unlike the reserved names above, this one may be overridden.

#Lease cadence

Each runtime process holds a node lease in its fleet prefix and renews it every third of CELLD_TTL_MS (default 10000). A renewal that fails or times out is retried before the authority expires: Celld keeps the normal cadence while at least half of the remaining lifetime is left, and otherwise retries after a quarter of what remains. A process whose lease expires, disappears, or no longer matches what it published fences itself — it stops its cells and exits with code 3 — and the node restarts it. The log line starts with SELF-FENCE: and carries one of node_lease_watchdog_fence, node_lease_record_missing_fence or node_lease_record_mismatch_fence.

Change the lifetime consistently on every node of the fleet; an unplanned takeover waits up to the lifetime:

{
  "runtime": {
    "celldEnv": { "CELLD_TTL_MS": "30000" },
  },
}

RUST_LOG=celld=info,store=debug in the daemon's environment — one of the variables it hands to every runtime process — makes each process log every read and write of its own lease record with an outcome of found, missing, applied, rejected or error.

#Startup storage test

Before a runtime process serves, Celld tests the bucket: the four conditional writes and one ranged read, on throwaway objects under probe/ in the application's fleet prefix. A store that rejects a required operation, ignores a condition, or returns the wrong range stops the process at once. An operation that fails without a clear cause is retried up to three times, after which the process starts with a warning. CELLD_STORAGE_PROBE=0 in runtime.celldEnv skips the test.

Scafo's bootstrap probe under _scafo/probe/ tests a ranged read as well, so a store that mishandles Range is refused once, at scafo sail, before any process starts. Celld's test is its own and runs per process regardless.

#Mixed Celld versions

One application is one Celld fleet, and a fleet takes one version at a time — except across a rolling pair: two releases upstream declared may serve one fleet together while its nodes are replaced one at a time. Scafo keeps those pairs as a list quoted from Celld's release notes and never infers one from version numbers: a patch release is not assumed to roll, and a release named in no pair shares a fleet with nothing but itself. The list holds three pairs: 0.5.0 ↔ 0.5.1, from the v0.5.1 release notes, 0.6.0 ↔ 0.6.1, from the v0.6.1 release notes, and 0.6.1 ↔ 0.6.2, from the v0.6.2 release notes. Pairs do not chain: 0.6.0 and 0.6.2 are in no pair together, so a fleet on 0.6.0 reaches 0.6.2 through 0.6.1, one full roll at a time. No pair crosses from 0.5 to 0.6: the v0.6.0 release notes let the step from 0.5.1 roll only for a fleet with bucket durability, and scafo sets none, so every application runs on Celld's default, fleet, and moving from 0.5.0 or 0.5.1 onto any 0.6.x stops the whole installation.

A pair can come with conditions scafo does not check. The v0.6.1 notes hold three things back until every node runs 0.6.1: setting CELLD_LTX_RETENTION_SECS, deploying a Python Worker, and raising CELLD_MAX_ASSET_FILE_BYTES above 25 MiB. The rolling upgrade freezes deploys across that step for that reason. The v0.6.2 notes hold nothing back: while 0.6.1 and 0.6.2 nodes share a fleet, a handler that fails on the owner of a forwarded Durable Object fetch reaches the caller as a 500 rather than a rejected stub.fetch(), and a rollback onto 0.6.1 rolls the same way.

Before a node starts an application's process, and again before it accepts a deploy for that application, it reads every other node record: any node still answering — heard from inside the 90-second liveness window, draining included — that serves the same application, or whose process for it is crash-looping, on a Celld release this node may not share a fleet with refuses it with version-skew (409). A crash-looping process counts because every run of it still takes a lease in the fleet. The check is made against every other answering node, one pair at a time, so a node never joins a fleet holding any release it may not share one with. The refusal names every node that has to stop, and the rolling pairs this build allows.

At intake the refusal comes before a byte of the artifact is read. In the reconcile loop it is a pending failure rather than a raised one: the rest of the pass still runs, and the next pass tries again once the other node is gone.

#Adopting a new deployment

Adoption is Celld's, but scafo shortens the wait for it. After a publish, the accepting node asks its local Celld process to adopt the new deployment now, over that process's internal listener and bounded to a few seconds; every other node serving the application does the same on the reconcile pass that sees the application's deployment revision move.

A request that fails is one line in the node's log and nothing else: Celld polls the deployment pointer itself, about every 30 seconds, and adopts the deployment either way. There is no route and no record for any of this, and nothing depends on it having happened.

#Upstream documentation

Celld owns deployment adoption, application health, cells and runtime resource policy. Read its own notes before upgrading a fleet:

• Celld security guidance — its internal operator API is mostly unauthenticated; keep it on a private network or overlay.
• Celld limitations
• Cloudflare compatibility

Celld exposes one current deployment per application fleet. Scafo records a monotonic deployment revision rather than a selected version, and offers no per-deployment routing or historical deployment selector — see Celld issue #181.