Node configuration

#How a setting is resolved

A node is configured from three sources: the command line, the environment, and a config file. Each setting is resolved on its own, in this order:

scafo sail accepts a flag for every setting on this page. scafo health accepts only --config, the six bucket flags, --listen, --transport and --wg-address — it resolves the listener from the environment and the config file exactly as the daemon does, so a listener chosen with sail --listen alone is invisible to it.

#The config file

Without --config, the daemon looks for these files in order and uses the first one it can read:

Both are parsed as JSONC: comments and trailing commas are allowed. The top level must be a JSON object; an array or a scalar is refused with invalid-input. A syntax error is refused naming the file and the character offset.

With --config <path>, only that path is read, and it must exist — a missing file is invalid-input ("Config file not found"), never a silent fall-back to the defaults.

Keys are addressed by the dotted path in the tables below (bucket.endpoint, runtime.celldEnv), which is how a refusal names them too.

#Blank values

A blank or whitespace-only value counts as unset in every source: an empty SCAFO_S3_PREFIX is the same as no SCAFO_S3_PREFIX at all. The one exception is the ingress token, where an explicitly blank value is refused rather than ignored, because an empty token would admit a caller presenting nothing.

#Booleans

observability.enabled and api.public accept a JSON boolean in the config file, or the text 1, true, 0, false (case-insensitive, surrounding spaces trimmed) from a flag, an environment variable or a JSON string. Anything else is refused with invalid-input. On the command line the switches are --observability / --no-observability and --api-public / --no-api-public; a switch nobody wrote leaves the environment and the config file to answer.

#Lists and maps

Five settings hold more than one value, and each source spells that out its own way:

• On the command line, repeat the flag. Each occurrence is one whole value and nothing is split, so a value may itself contain a comma.
• In the environment, separate the values with commas. A variable cannot repeat, which is why a value written there cannot contain a comma.
• In the config file, write the JSON shape: an array for a list, an object for a map.

For node.labels and runtime.celldEnv the flag and the environment variable replace the file's set wholesale rather than merging into it. Blank entries are skipped, so a trailing comma costs nothing; a duplicate key is refused rather than resolved; text that names nothing at all is refused.

Label keys and values are 1–63 characters, lowercase alphanumerics, dots, dashes and underscores, beginning and ending on an alphanumeric. Environment variable names are letters, digits and underscores, never beginning on a digit, and their values must be JSON strings — a number or a boolean in the config file is refused rather than stringified.

#Addresses

node.listen, node.advertise and wireguard.endpoint are host:port:

• the port is 1–65535;
• an unbracketed host is a lowercase DNS name (labels of 1–63 characters, no trailing dot, at most 253 characters) or an IPv4 address;
• an IPv6 literal must be bracketed the way a URL authority writes one: [::1]:7400.

runtime.bindHost is a bare host rather than a pair, and additionally accepts the wildcards 0.0.0.0 and ::. wireguard.address is an interface address with its prefix length, such as 10.83.0.1/24. node.id matches ^[a-z\d_-]{1,128}$. Every platformDomains entry is validated as a host — no scheme, no wildcard, no port.

Malformed addresses, node ids and domains are rejected before the daemon writes or binds anything.

#A worked example

The same node, written both ways: a two-node fleet on a WireGuard tunnel, serving *.apps.example.com, with observability on and a colocated Traefik.

{
  // The bucket IS the registry: every node pointed at the same bucket and
  // prefix is one fleet.
  "bucket": {
    "endpoint": "https://store.internal.example.com:8333",
    "region": "us-east-1",
    "bucket": "scafo",
    "prefix": "prod",
    "accessKey": "AKIAEXAMPLE",
    "secretKey": "s3cr3t-example-key",
  },

  "node": {
    // Optional: without it the id in the state directory is used.
    "id": "node-a",
    // On the tunnel, because that is what this node's peers can reach.
    "listen": "10.83.0.1:7400",
    "advertise": "10.83.0.1:7400",
    "stateDir": "/var/lib/scafo",
    // What this machine IS, for applications that select on a class of node.
    "labels": { "region": "eu-west-1", "tier": "fast" },
  },

  // Apex domains applications are served wildcard subdomains under.
  "platformDomains": ["apps.example.com"],

  "runtime": {
    "binPath": "/usr/local/bin/celld",
    // Each application holds one pair out of this range, for as long as this
    // node knows it. On --network host these are the machine's own ports.
    "portRangeStart": 24000,
    "portRangeEnd": 24998,
    "portQuarantineMs": 600000,
    // Runtime listeners follow the tunnel, so peers can reach them.
    "bindHost": "10.83.0.1",
    // Upstream Celld policy. Scafo sets no policy defaults of its own.
    "celldEnv": {
      "CELLD_MAX_RSS_MB": "512",
      "CELLD_TTL_MS": "30000",
    },
  },

  // How often this node looks past the change hint it reads once a pass.
  "registry": {
    "resyncIntervalMs": 60000,
    "nodeRefreshIntervalMs": 30000,
  },

  // Only the node currently holding the lease acts on these.
  "coordinator": {
    "leaseTtlMs": 45000,
    "reapIntervalMs": 600000,
  },

  "deploy": {
    "artifactBytes": 1073741824,
    "archiveEntries": 20000,
    "activeTasks": 2,
    "applicationActiveTasks": 1,
    "queuedTasks": 64,
    "applicationQueuedTasks": 4,
    "stagedBytes": 5368709120,
    "applicationStagedBytes": 3221225472,
    "receiveTimeoutMs": 600000,
    "queueTimeoutMs": 600000,
    "taskTimeoutMs": 300000,
    "outputBytes": 65536,
  },

  "observability": {
    "enabled": true,
    // Refused unless "enabled" is true.
    "ingressAccessLog": "/var/log/scafo-ingress/access.log",
  },

  "api": {
    "public": true,
    "jsonBytes": 65536,
  },

  "ingress": {
    // Primary first. Leave the setting out to have the node mint its own.
    "token": ["kZ9x-new-token", "Qp1v-old-token"],
    "entryPoints": ["web", "websecure"],
  },

  "transport": { "driver": "wireguard" },

  "wireguard": {
    "address": "10.83.0.1/24",
    "endpoint": "203.0.113.7:51820",
    "port": 51820,
  },
}

#bucket

The registry store. Every node pointed at the same bucket and prefix belongs to one fleet.

The prefix names the installation. Changing or removing it points the node at a different, empty installation: nothing moves, and the node records, operator keys, trust policies and fleets under the old prefix stay where they were. Every bucket-reading command run from another machine, such as scafo fleet nodes, scafo trust set or scafo remote add --register, must be given the same prefix as the node, or none if the node has none.

The Scafo-specific spellings outrank the AWS/S3 ones: SCAFO_S3_REGION wins over AWS_REGION, SCAFO_S3_ACCESS_KEY over AWS_ACCESS_KEY_ID, SCAFO_S3_ENDPOINT over S3_ENDPOINT.

#node

#platformDomains

#runtime

How this node runs the Celld processes that serve its applications.

A range holding fewer than two ports is refused at startup. The pair an application is given is chosen once on this node and then kept — see port pairs.

#registry

How often this node looks at the registry beyond the change hint it reads once a pass. Both are bounds on how stale its view may get, never on correctness.

#coordinator

What the node does on the ticks it spends holding the coordinator lease. Every node reads these; only the holder acts on them.

A longer lease costs no failover time. What a dead coordinator held is only reassigned once the fleet has decided the node is gone, and that decision is the 90-second liveness window; both clocks start at the same death, so any lifetime up to that window expires while the fleet is still waiting to notice. Shortening it only makes the holder write the lease more often, forever.

Nothing a reap does is urgent — it deletes deploy records a week past their retention and applications a day past their retirement grace, and fails deploys whose node is already gone, which a poll answers at read time anyway. What it costs is one listing of every deploy record in the installation, the largest periodic read there is, so it runs at the cadence of the slowest thing it decides.

#deploy

Bounds applied while receiving, expanding and building a deploy artifact. They are scheduling limits, not an OS admission control.

#observability

#api

#ingress

There is no ingress driver to select. Both supported ingresses pull: Caddy through the scafo dynamic-upstreams module reading /v1/ingress/routes, Traefik through its http provider reading /v1/ingress/routes/traefik. This section is what an ingress needs in order to read this node.

SCAFO_STORAGE_TOKEN is the name this variable had when the token only guarded storage; it is still accepted, after SCAFO_INGRESS_TOKEN.

#transport

#wireguard

Read only when transport.driver is wireguard. Settings written here for a node running another transport are reported as ignored and dropped.

#Required settings

Only the four bucket settings are hard-required: bucket.endpoint, bucket.bucket, bucket.accessKey and bucket.secretKey. A missing one refuses the boot with invalid-input ("Missing required configuration"), naming all three ways to supply it:

scafo: Missing required configuration: "bucket.endpoint" has no value: set it in the
config file, set SCAFO_S3_ENDPOINT or S3_ENDPOINT in the environment, or pass
--endpoint [urn:scafo:invalid-input]

Everything else has a default or is legitimately absent.

#Cross-setting rules

These are checked at load or at boot, before anything is written to the bucket or bound to an address.

The transport check runs before the node registers itself, and the tunnel-keys check runs immediately after it: a node that should not be on this network must not have registered on it first, and a node whose transport is a tunnel it has never been set up on must not register an address its peers cannot reach.

#Other environment variables

These are not settings of the configuration above; the daemon, the CLI or the image reads them directly.

Celld's own variables — CELLD_SHUTDOWN_TOTAL_MS, CELLD_TTL_MS, CELLD_OTEL and the rest of the upstream policy — are set through runtime.celldEnv rather than in the daemon's own environment.

#Files under the state directory

Everything here is owner-readable only (0600), in a directory Scafo creates 0700.

The per-application directories are kept through every stop — a retirement, a block, a reassignment, a drain, a daemon restart — because the disk may still hold writes a peer needs and the application may be served here again. One is removed only by the pass that finds the application gone from the registry altogether, and never while a process for it is running here.

#What the installer writes

scripts/install.sh, run as root on a systemd host, adds these on top of the state directory:

Binaries go to /usr/local/bin as root, or ~/.local/bin otherwise. Uninstalling never touches the bucket, and preserves /var/lib/scafo unless --purge is given.