# Celld runtime > How a node picks the Celld binary, what it passes to every runtime process, and which variables it refuses to let you set. Scafo requires Celld **0.5.0**, and checks: `scafo sail` runs ` --version` before it opens the bucket and refuses an older or unreadable binary with `capability-missing`. The version it read is written into the node's record, and every node serving one application must report the same one, or the other release of a rolling pair upstream declared — see [mixed Celld versions](#mixed-celld-versions). The container image and the Linux installer ship Celld **0.6.2**. Each application assigned to a node runs as one Celld process, started by the node's reconcile loop and stopped by it. Scafo owns the process arguments and part of its environment; everything else is upstream Celld policy, which you write through this node's configuration. ## Choosing the binary | File key | Env | Flag | Default | |---|---|---|---| | `runtime.binPath` | `SCAFO_CELLD_BIN` | `--celld-bin` | `celld` | The value is resolved against `PATH` unless it contains a directory separator. The official image ships `/usr/local/bin/celld`; the installer writes `SCAFO_CELLD_BIN=/usr/local/bin/celld` into the node's systemd unit. The same binary runs deploy builds, so a node whose `binPath` is wrong fails both at reconcile and at deploy time. ## Node-wide Celld policy Variables handed to every runtime process this node starts: | File key | Env | Flag | |---|---|---| | `runtime.celldEnv` | `SCAFO_CELLD_ENV` | `--celld-env` | Three spellings, as with node labels: ::code-group ```jsonc [scafo.config.jsonc] { "runtime": { "celldEnv": { "CELLD_MAX_RSS_MB": "512", "CELLD_MAX_RESIDENT_CELLS": "64", }, }, } ``` ```sh [environment] SCAFO_CELLD_ENV=CELLD_MAX_RSS_MB=512,CELLD_MAX_RESIDENT_CELLS=64 ``` ```sh [flag] scafo sail \ --celld-env CELLD_MAX_RSS_MB=512 \ --celld-env CELLD_MAX_RESIDENT_CELLS=64 ``` :: Rules for the set: - The flag and the environment variable **replace the config file's set wholesale**; they do not merge into it. - Each `--celld-env` states one pair, and its value may hold anything, commas included. `SCAFO_CELLD_ENV` separates its pairs with commas, so a value written there cannot contain one; write such a value by repeating the flag, or as `runtime.celldEnv` in the config file. - Everything after the first `=` is the value, so `NAME=a=b` gives `a=b`. Surrounding spaces are dropped on both halves. - Names are letters, digits and underscores, never beginning on a digit. Values must be strings: a number or a boolean in the config file is refused rather than stringified. - A duplicate name is refused rather than resolved. Text that names no variable at all is refused too. Scafo adds no Celld policy defaults of its own. Whatever upstream Celld defaults to is what a process gets, apart from the telemetry switch below. ## Names Scafo refuses Setting any of these through `runtime.celldEnv` is `invalid-input` ("Reserved environment variable"), refused at load rather than silently overwritten, because the driver writes them itself from this node's configuration, from the listeners its processes bind, and from the private directories it keeps for them: | Variable | Why | |---|---| | `AWS_ACCESS_KEY_ID` | derived from `bucket.accessKey` | | `AWS_SECRET_ACCESS_KEY` | derived from `bucket.secretKey` | | `AWS_SESSION_TOKEN` | reserved with the rest of the credential set the driver writes | | `AWS_REGION` | derived from `bucket.region` | | `CELLD_BUCKET` | derived from the bucket, prefix and this application's fleet prefix | | `CELLD_ADDR` | derived from `runtime.bindHost` and the allocated public port | | `CELLD_INTERNAL_ADDR` | derived from `runtime.bindHost` and the allocated internal port | | `CELLD_ADVERTISE` | derived from the node's advertised address and the internal port | | `CELLD_TRUST_FORWARDED_HEADERS` | derived from the ingress topology | | `CELLD_NODE` | the node's own id, so a process comes back under the name it left under | | `CELLD_WATCH` | `/celld//work`, the application's working directory on this node | | `CELLD_ASSET_CACHE_DIR` | `/celld//assets`, one asset cache per application rather than one per machine | Every other Celld variable is accepted as-is and handed to every process this node starts. Scafo keeps no list of its own: what a variable does, and which release introduced it, is upstream Celld's documentation. These are the **node's** Celld policy, not an application's variables: an application's variables travel in its own `wrangler.jsonc` or `wrangler.json`, inside the artifact, and the node never touches them. ## What Scafo passes to each process ### Arguments | Argument | Value | |---|---| | `--bucket` | `//fleets/` — the prefix segment is omitted when none is configured | | `--endpoint` | `bucket.endpoint` | | `--listen` | `:` | | `--internal-listen` | `:` | | `--advertise` | `:` | The two ports come from the node's range, `runtime.portRangeStart`–`runtime.portRangeEnd` (`24000`–`24998` by default). They are drawn at random and independently of each other — the internal port is never the public one plus one — and once drawn they belong to that application on this node and are reused on every later start. See [port pairs](/docs/placement#port-pairs). The advertised host is `node.advertise`'s host when one is set, and `node.listen`'s host otherwise — that is what a peer dials to reach the process that owns a cell, which is why a multi-node fleet needs `runtime.bindHost` off loopback and a stated transport. Each application's Celld fleet state lives under `fleets/` in the same bucket, outside the `_scafo` registry root. ### Environment Three steps, each overriding the last — and that is the whole of it: 1. **Base**: `PATH`, `HOME`, `TMPDIR`, `TZ`, `RUST_LOG` and `DOCKER_HOST`, copied from the daemon's own environment when they are set — nothing else of the daemon's environment reaches a runtime process. The publish starts from the same base; see [container engine](#container-engine). 2. **Your policy**: everything in `runtime.celldEnv`. 3. **Derived, non-overridable**: `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` and `AWS_REGION` from the node's bucket configuration; `CELLD_TRUST_FORWARDED_HEADERS=1`, because the ingress in front of the node terminates TLS and supplies the forwarded headers; and `CELLD_NODE`, `CELLD_WATCH` and `CELLD_ASSET_CACHE_DIR`, the name and the directories the process comes back to. Only the last three differ between the applications this node serves: `CELLD_WATCH` and `CELLD_ASSET_CACHE_DIR` name the application, and `CELLD_NODE` is the same for all of them. Nothing else of an application's own is added — its variables travel in its own config, inside the artifact. ::warning Celld child processes therefore hold the fleet's bucket credentials. Scafo provides no OS credential boundary around them. :: ### Variables per application An application's variables are not in the process environment at all: they live in the application's own `wrangler.jsonc` or `wrangler.json`, under `vars`, and Celld reads them from the config it was published with. The node hands that file to Celld exactly as the artifact carried it. ::read-more{to="/docs/deploying#variables"} Setting them at deploy time with `scafo deploy --var`, and what that means for secrets. :: ## Container engine For [container classes](/docs/containers) a node hands Celld the engine and nothing else. Two different processes need it, and each gets it out of the daemon's own environment: | Process | What reaches it | |---|---| | Every runtime process | The base above, `DOCKER_HOST` among it, then `runtime.celldEnv` and the derived variables. | | The publish — `celld deploy`, on the node that accepts a deploy | `PATH`, `HOME`, `TMPDIR`, `TZ`, `RUST_LOG`, `DOCKER_HOST`, `CELLD_ESBUILD`, `CELLD_DOCKER`, `CELLD_CONTAINER_PLATFORM`, `CELLD_PYTHON_RUNTIME_DIR` and `CELLD_MAX_ASSET_FILE_BYTES`, when set, plus the bucket credentials. | `DOCKER_HOST` is passed on unread: Celld finds the engine's socket itself, at the default paths or at the `unix://` URL it names. The publish builds or pulls a container image with the `docker` client on its `PATH` — `/usr/local/bin/docker` in the image, beside the esbuild `CELLD_ESBUILD` names — and on a bare host, whatever client the unit's `PATH` finds. `CELLD_DOCKER` and `CELLD_CONTAINER_PLATFORM` reach the publish the same way, unread: the client to run in place of `docker`, such as `podman`, and the platform the image is built or pulled for, `linux/amd64` unless it names another. Celld reads both in `celld deploy` alone, so they are set in the daemon's own environment — the env file, the Compose `environment:`, the systemd unit — and never in `runtime.celldEnv`, which reaches the runtime processes only, where Celld never reads them. Celld's runtime settings for containers, such as `CELLD_CONTAINER_RUNTIME` and `CELLD_CONTAINER_DNS`, go in `runtime.celldEnv` like any other Celld variable; none of them is reserved. What each of those variables does is Celld's, in its [containers documentation](https://github.com/denoland/celld/blob/v0.6.2/docs/services/containers.md). Two more reach the publish that are not the engine's. `CELLD_PYTHON_RUNTIME_DIR` is where a [Python Worker](/docs/deploying#python-workers)'s build keeps the Pyodide runtime it downloads, and only the publish reads it. `CELLD_MAX_ASSET_FILE_BYTES`, the largest static asset file a deployment may carry (25 MiB unless set), is read on both sides: the publish checks it, and so does every runtime process serving the deployment. Raising it therefore means setting it twice, to the same value — in the daemon's own environment for the publish, and in `runtime.celldEnv` for every process — and a process left at the default refuses the deployment. Other runtime limits, such as `CELLD_LTX_RETENTION_SECS` and `CELLD_MAX_DYNAMIC_WORKER_CODE_BYTES`, are read by the runtime processes alone and go in `runtime.celldEnv`. ## Durability and the state directory Each runtime process works out of its own directory under the node's state directory, and comes back under the node's own id: | Variable | Value | |---|---| | `CELLD_NODE` | this node's id | | `CELLD_WATCH` | `/celld//work` | | `CELLD_ASSET_CACHE_DIR` | `/celld//assets` | Scafo derives all three and creates the directories `0700` before the process starts. You cannot set them: `runtime.celldEnv` refuses them with the rest of the [reserved names](#names-scafo-refuses). **Why it matters.** With more than one node, a write is acknowledged to the caller as soon as a peer node holds it on its disk, in this directory — before the bucket has a copy of it. A process picks those writes back up at boot from the directory its predecessor of the same node name left behind, and serves its own copy to the peers doing the same. So two things have to hold: - **The state directory must persist**, across a restart *and* across the node being re-created: a named volume, a retained PVC, a real directory on the host. A directory that is new on every start leaves acknowledged writes nowhere, and the application comes back missing its most recent ones. - **The node id must be stable.** It is what a process comes back under; a node that renames itself cannot find what it left. ::warning An empty or throwaway state directory is not a clean slate. It is acknowledged writes lost. :: Left to itself Celld would pick a fresh name each start, a working directory under `$TMPDIR` new on every start, and one asset cache shared by every process on the machine — which is why Scafo sets these three rather than letting you. `/celld//` is kept through every stop: retirement, blocking, reassignment, a drain, a daemon restart. The disk may still hold writes a peer needs, and the application may be served here again. It is removed only by the reconcile pass that finds the application gone from the registry altogether, never while a process for it is running here, and never on a pass whose registry read failed. ::read-more{to="/config#files-under-the-state-directory"} Everything else the state directory holds. :: ## Shutdown grace `CELLD_SHUTDOWN_TOTAL_MS` is Celld's complete process shutdown bound. Scafo reads it out of `runtime.celldEnv` — not out of the daemon's own environment — and sizes its supervisor grace from it: ``` supervisor grace = CELLD_SHUTDOWN_TOTAL_MS + 5000 ms ``` With the upstream default of `40000` that is a **45-second** grace between `SIGTERM` and the supervisor's hard kill, so a process always finishes draining before it is killed. The value must be a positive integer of digits only, no greater than `2147478647`; anything else is `invalid-input` ("Invalid celld shutdown bound"). Whatever supervises the node has to be more patient than the node is with its children: | Supervisor | Setting | |---|---| | Docker / Compose | `stop_grace_period: 50s` | | systemd | `TimeoutStopSec=50s`, `KillMode=mixed`, `KillSignal=SIGTERM` | | Anything else | at least 50 seconds, or `CELLD_SHUTDOWN_TOTAL_MS + 10s` when you raised it | ::important Signal the daemon alone, in a private PID namespace, and let it drain: it stops its runtimes itself, in an order that matters. A supervisor that signals every process beside it defeats the drain. :: ## Telemetry `CELLD_OTEL` is the one Celld variable Scafo defaults for you. With `observability.enabled` on and no `CELLD_OTEL` in `runtime.celldEnv`, every runtime process is started with `CELLD_OTEL=1`. Set it yourself in `runtime.celldEnv` — to another value, or to `0` — and your value stands; unlike the reserved names above, this one may be overridden. ::read-more{to="/docs/observability"} What the node records about itself, and how to read it back. :: ## Lease cadence Each runtime process holds a node lease in its fleet prefix and renews it every third of `CELLD_TTL_MS` (default `10000`). A renewal that fails or times out is retried before the authority expires: Celld keeps the normal cadence while at least half of the remaining lifetime is left, and otherwise retries after a quarter of what remains. A process whose lease expires, disappears, or no longer matches what it published fences itself — it stops its cells and exits with code 3 — and the node restarts it. The log line starts with `SELF-FENCE:` and carries one of `node_lease_watchdog_fence`, `node_lease_record_missing_fence` or `node_lease_record_mismatch_fence`. Change the lifetime consistently on **every** node of the fleet; an unplanned takeover waits up to the lifetime: ```jsonc [scafo.config.jsonc] { "runtime": { "celldEnv": { "CELLD_TTL_MS": "30000" }, }, } ``` `RUST_LOG=celld=info,store=debug` in the daemon's environment — one of the variables it hands to every runtime process — makes each process log every read and write of its own lease record with an `outcome` of `found`, `missing`, `applied`, `rejected` or `error`. ## Startup storage test Before a runtime process serves, Celld tests the bucket: the four conditional writes and one ranged read, on throwaway objects under `probe/` in the application's fleet prefix. A store that rejects a required operation, ignores a condition, or returns the wrong range stops the process at once. An operation that fails without a clear cause is retried up to three times, after which the process starts with a warning. `CELLD_STORAGE_PROBE=0` in `runtime.celldEnv` skips the test. Scafo's bootstrap probe under `_scafo/probe/` tests a ranged read as well, so a store that mishandles `Range` is refused once, at `scafo sail`, before any process starts. Celld's test is its own and runs per process regardless. ## Mixed Celld versions One application is one Celld fleet, and a fleet takes one version at a time — except across a **rolling pair**: two releases upstream declared may serve one fleet together while its nodes are replaced one at a time. Scafo keeps those pairs as a list quoted from Celld's release notes and never infers one from version numbers: a patch release is not assumed to roll, and a release named in no pair shares a fleet with nothing but itself. The list holds three pairs: **0.5.0 ↔ 0.5.1**, from the [v0.5.1 release notes](https://github.com/denoland/celld/releases/tag/v0.5.1), **0.6.0 ↔ 0.6.1**, from the [v0.6.1 release notes](https://github.com/denoland/celld/releases/tag/v0.6.1), and **0.6.1 ↔ 0.6.2**, from the [v0.6.2 release notes](https://github.com/denoland/celld/releases/tag/v0.6.2). Pairs do not chain: 0.6.0 and 0.6.2 are in no pair together, so a fleet on 0.6.0 reaches 0.6.2 through 0.6.1, one full roll at a time. No pair crosses from 0.5 to 0.6: the [v0.6.0 release notes](https://github.com/denoland/celld/releases/tag/v0.6.0) let the step from 0.5.1 roll only for a fleet with `bucket` durability, and scafo sets none, so every application runs on Celld's default, `fleet`, and moving from 0.5.0 or 0.5.1 onto any 0.6.x stops the whole installation. A pair can come with conditions scafo does not check. The v0.6.1 notes hold three things back until every node runs 0.6.1: setting `CELLD_LTX_RETENTION_SECS`, deploying a Python Worker, and raising `CELLD_MAX_ASSET_FILE_BYTES` above 25 MiB. The [rolling upgrade](/docs/operations#a-rolling-upgrade) freezes deploys across that step for that reason. The v0.6.2 notes hold nothing back: while 0.6.1 and 0.6.2 nodes share a fleet, a handler that fails on the owner of a forwarded Durable Object fetch reaches the caller as a 500 rather than a rejected `stub.fetch()`, and a rollback onto 0.6.1 rolls the same way. Before a node starts an application's process, and again before it accepts a deploy for that application, it reads every other node record: any node still *answering* — heard from inside the 90-second liveness window, draining included — that serves the same application, or whose process for it is crash-looping, on a Celld release this node may not share a fleet with refuses it with `version-skew` (409). A crash-looping process counts because every run of it still takes a lease in the fleet. The check is made against every other answering node, one pair at a time, so a node never joins a fleet holding any release it may not share one with. The refusal names every node that has to stop, and the rolling pairs this build allows. | Case | Counted as different | |---|---| | Another answering node reports the same `X.Y.Z` | no | | Another answering node reports the other release of a rolling pair | no — `0.5.0` beside `0.5.1`, `0.6.0` beside `0.6.1`, `0.6.1` beside `0.6.2` | | Another answering node reports any other `X.Y.Z` | yes — outside the list the comparison is exact: any `0.6.x` beside `0.5.1` or `0.5.0`, and `0.6.2` beside `0.6.0` | | Another answering node reports no version at all | yes — its record predates the field, so nothing in it says one | | A node past the liveness window | no — nothing it says is current | | The node asking itself | never compared | At intake the refusal comes before a byte of the artifact is read. In the reconcile loop it is a pending failure rather than a raised one: the rest of the pass still runs, and the next pass tries again once the other node is gone. ::read-more{to="/docs/operations#upgrading-nodes"} The upgrade procedures this implies. :: ## Adopting a new deployment Adoption is Celld's, but scafo shortens the wait for it. After a publish, the accepting node asks its **local** Celld process to adopt the new deployment now, over that process's internal listener and bounded to a few seconds; every other node serving the application does the same on the reconcile pass that sees the application's deployment revision move. A request that fails is one line in the node's log and nothing else: Celld polls the deployment pointer itself, about every 30 seconds, and adopts the deployment either way. There is no route and no record for any of this, and nothing depends on it having happened. ## Upstream documentation Celld owns deployment adoption, application health, cells and runtime resource policy. Read its own notes before upgrading a fleet: - [Celld security guidance](https://github.com/denoland/celld/blob/main/docs/security.md) — its internal operator API is mostly unauthenticated; keep it on a private network or overlay. - [Celld limitations](https://github.com/denoland/celld/blob/main/docs/limitations.md) - [Cloudflare compatibility](https://github.com/denoland/celld/blob/main/docs/cloudflare-compat.md) Celld exposes one current deployment per application fleet. Scafo records a monotonic deployment revision rather than a selected version, and offers no per-deployment routing or historical deployment selector — see [Celld issue #181](https://github.com/denoland/celld/issues/181). ::note `/.well-known/celld/health` is the application runtime's health endpoint. The node's own is `/.well-known/scafo/health`. ::