Observability
Opt-in. A node records its own events and, given an ingress access log, what each application was served — then flushes both to the bucket as Parquet.
#Turning it on
| Setting | Flag | Environment | Config key | Default |
|---|---|---|---|---|
| Observability | --observability / --no-observability | SCAFO_OBSERVABILITY | observability.enabled | false |
| Ingress access log | --ingress-access-log <path> | SCAFO_INGRESS_ACCESS_LOG | observability.ingressAccessLog | none |
scafo sail --observability --ingress-access-log /var/log/scafo-ingress/access.logThe seam is attached only while it is on: every emission site reads it conditionally, so an absent seam means the events are never even built. Naming an access log while observability is off is refused with invalid-input.
#Numbers
| Constant | Value | Meaning |
|---|---|---|
| Ring size | 512 | Events held in memory before the oldest are dropped. |
| Drain interval | 5 s | How often the ring is drained into the pending buffer. |
| Flush interval | 60 s | How often the pending buffer is written to the bucket, when nothing triggers it sooner. |
| Flush threshold | 2000 records | Pending records that trigger a flush ahead of the interval. |
| Pending cap | 20000 records | Held while flushes are failing; past it the oldest are dropped and counted. |
| Retention | 30 days | How long a node keeps its own observations. |
| Sweep cadence | every 60 flushes | At the default flush interval, one retention sweep an hour. |
| Sweep cap | 500 keys | Deletions per sweep; the rest go on the next one. |
A final flush runs at shutdown, after every loop has stopped emitting and the access log has been read to its end.
Only the writing node deletes its own keys.
#Events
| Type | Fields |
|---|---|
reconcile.pass.start | kicked — whether the pass was requested or started by the timer |
reconcile.pass.end | actions, failures, storeOps, listOps, durationMs |
placement.pending | application, reason: "placement-unavailable" |
runtime.started | application |
operator.trusted | actor, subject, role |
operator.revoked | actor, subject, role |
application.blocked | actor, application |
application.unblocked | actor, application |
actor is the key the request was signed with; subject is the key an operator event was about, and application the application an application event was about.
Note
The operator and application events are emitted by the node API routes and by nothing else. The bucket-side ceremonies — scafo operator keygen --register, or a record written straight to the registry — have no daemon in the picture and are covered by the bucket's own access log instead.
Reconcile passes are emitted twice per node every 5 seconds whether or not anything happened, which is why scafo traces leaves them out unless --all asks for them.
#Access-log counters
The node tails the access-log file the ingress writes, starting from its current end at boot — whatever the ingress served before this node came up was already served, and re-counting it on every restart would invent traffic.
Both formats are read, per line, independently:
| Caddy | Traefik | |
|---|---|---|
| Host | request.host | RequestHost |
| Method | request.method | RequestMethod |
| Status | status | DownstreamStatus |
| Bytes | size | DownstreamContentSize |
| Duration | duration, in seconds | Duration, in nanoseconds |
Host and status are required; everything else defaults to zero. A malformed line is skipped and counted, never thrown.
Counters are monotonic, one set per application, since the node booted:
| Field | Contents |
|---|---|
requests | Request count. |
bytes | Response body bytes, as the ingress reported them. |
status | Counts per class: 2xx, 3xx, 4xx, 5xx, other (anything outside 200–599). |
durations | One count per band with upper bounds 10, 50, 100, 300, 1000 and 3000 ms, plus one for everything past the last. |
A host that resolves to no application is counted in the reserved - bucket and reported as unattributed.
#Whose host is whose
A request is attributed by the same three rules the routing is built from, read backwards: the exact host first — an application's name under a platform domain, or a host claimed for it — then the platform grammar, then the single label below a custom domain claimed with its subdomains. Both directions come out of one enumeration, so a counter and a route can never disagree.
The rules are read from the registry, not from what this node is running. A node therefore counts correctly for an application placed on another node, and it keeps counting for one that is between deployments: a log line exists because something already answered, and a claim outlives every process that ever served it.
Counters are flushed alongside the events.
Warning
Both processes need the same path: the ingress writes it, the node reads it. In Compose that is a volume mounted into both services. Without a path, the node logs no ingress access log configured once at boot and counts nothing — the ordinary case for an operator running their own ingress.
#Where it lands
_scafo/observe/<node>/<yyyy>/<mm>/<dd>/<hh>/<epochMs>-<seq>.parquetOne wide-table schema, ZSTD-compressed. Every record carries v, record, node and at; the rest of the columns are per record kind — event fields, flush statistics, and the counter fields with status flattened into five status_* columns and durations into one column per band.
Read them with any Parquet reader:
-- DuckDB
SELECT * FROM read_parquet('s3://<bucket>/_scafo/observe/<node>/**/*.parquet');#Reading them back
Both commands read straight out of the bucket, so no node has to be up: the records were flushed by whichever nodes were running at the time, and those nodes may be gone. Both take the usual bucket flags (--config, --endpoint, --region, --bucket, --prefix, --access-key, --secret-key).
#scafo metrics
scafo metrics --since 1h
scafo metrics --since 7d --name shop --json| Flag | Meaning |
|---|---|
--name | Only this application. |
--env | Narrow to the application <name>-<env> instead. |
--since | A duration (90s, 15m, 1h, 7d) or an ISO-8601 instant. Defaults to 1h. |
--json | Print the summary as JSON instead of a table. |
The snapshots are monotonic totals per node, so a window is a subtraction. Counted names are resolved against the applications that actually exist, so a label nothing answers to is nobody's.
#scafo traces
scafo traces --since 30m --name shop
scafo traces --since 1h --all --limit 500 --json| Flag | Meaning |
|---|---|
--name | Only what belongs to this application. |
--env | Narrow to the application <name>-<env> instead. |
--since | Same forms as above. Defaults to 1h. |
--limit | Events to show at most, newest kept. Defaults to 200. |
--all | Include the reconcile loop's own passes. |
--json | Print the events as JSON instead of lines. |
Reconcile passes are held back by default so operational and audit events are not pushed out of the window. How many were held back is always reported.
#Runtime telemetry
When observability is on and runtime.celldEnv does not set it, the node runs every Celld process with CELLD_OTEL=1. An operator value wins: CELLD_OTEL is the one Celld variable that may override a Scafo-derived default.
Celld's telemetry is separate from Scafo's observations and is configured through Celld.