# Observability

> Opt-in. A node records its own events and, given an ingress access log, what each application was served — then flushes both to the bucket as Parquet.

## Turning it on

| Setting | Flag | Environment | Config key | Default |
|---|---|---|---|---|
| Observability | `--observability` / `--no-observability` | `SCAFO_OBSERVABILITY` | `observability.enabled` | `false` |
| Ingress access log | `--ingress-access-log <path>` | `SCAFO_INGRESS_ACCESS_LOG` | `observability.ingressAccessLog` | none |

```sh
scafo sail --observability --ingress-access-log /var/log/scafo-ingress/access.log
```

The seam is attached only while it is on: every emission site reads it conditionally, so an absent seam means the events are never even built. Naming an access log while observability is off is refused with `invalid-input`.

## Numbers

| Constant | Value | Meaning |
|---|---|---|
| Ring size | 512 | Events held in memory before the oldest are dropped. |
| Drain interval | 5 s | How often the ring is drained into the pending buffer. |
| Flush interval | 60 s | How often the pending buffer is written to the bucket, when nothing triggers it sooner. |
| Flush threshold | 2000 records | Pending records that trigger a flush ahead of the interval. |
| Pending cap | 20000 records | Held while flushes are failing; past it the oldest are dropped and counted. |
| Retention | 30 days | How long a node keeps its own observations. |
| Sweep cadence | every 60 flushes | At the default flush interval, one retention sweep an hour. |
| Sweep cap | 500 keys | Deletions per sweep; the rest go on the next one. |

A final flush runs at shutdown, after every loop has stopped emitting and the access log has been read to its end.

Only the writing node deletes its own keys.

## Events

| Type | Fields |
|---|---|
| `reconcile.pass.start` | `kicked` — whether the pass was requested or started by the timer |
| `reconcile.pass.end` | `actions`, `failures`, `storeOps`, `listOps`, `durationMs` |
| `placement.pending` | `application`, `reason: "placement-unavailable"` |
| `runtime.started` | `application` |
| `operator.trusted` | `actor`, `subject`, `role` |
| `operator.revoked` | `actor`, `subject`, `role` |
| `application.blocked` | `actor`, `application` |
| `application.unblocked` | `actor`, `application` |

`actor` is the key the request was signed with; `subject` is the key an operator event was about, and `application` the application an application event was about.

::note
The operator and application events are emitted by the node API routes and by nothing else. The bucket-side ceremonies — `scafo operator keygen --register`, or a record written straight to the registry — have no daemon in the picture and are covered by the bucket's own access log instead.
::

Reconcile passes are emitted twice per node every 5 seconds whether or not anything happened, which is why `scafo traces` leaves them out unless `--all` asks for them.

## Access-log counters

The node tails the access-log file the ingress writes, starting **from its current end at boot** — whatever the ingress served before this node came up was already served, and re-counting it on every restart would invent traffic.

Both formats are read, per line, independently:

| | Caddy | Traefik |
|---|---|---|
| Host | `request.host` | `RequestHost` |
| Method | `request.method` | `RequestMethod` |
| Status | `status` | `DownstreamStatus` |
| Bytes | `size` | `DownstreamContentSize` |
| Duration | `duration`, in seconds | `Duration`, in nanoseconds |

Host and status are required; everything else defaults to zero. A malformed line is skipped and counted, never thrown.

Counters are monotonic, one set per application, since the node booted:

| Field | Contents |
|---|---|
| `requests` | Request count. |
| `bytes` | Response body bytes, as the ingress reported them. |
| `status` | Counts per class: `2xx`, `3xx`, `4xx`, `5xx`, `other` (anything outside 200–599). |
| `durations` | One count per band with upper bounds 10, 50, 100, 300, 1000 and 3000 ms, plus one for everything past the last. |

A host that resolves to no application is counted in the reserved `-` bucket and reported as `unattributed`.

### Whose host is whose

A request is attributed by the same three rules the routing is built from, read backwards: the exact host first — an application's name under a platform domain, or a host claimed for it — then the platform grammar, then the single label below a custom domain [claimed with its subdomains](/docs/custom-domains#claiming-the-subdomains-too). Both directions come out of one enumeration, so a counter and a route can never disagree.

The rules are read from the **registry**, not from what this node is running. A node therefore counts correctly for an application placed on another node, and it keeps counting for one that is between deployments: a log line exists because something already answered, and a claim outlives every process that ever served it.

Counters are flushed alongside the events.

::warning
Both processes need the same path: the ingress writes it, the node reads it. In Compose that is a volume mounted into both services. Without a path, the node logs `no ingress access log configured` once at boot and counts nothing — the ordinary case for an operator running their own ingress.
::

## Where it lands

```
_scafo/observe/<node>/<yyyy>/<mm>/<dd>/<hh>/<epochMs>-<seq>.parquet
```

One wide-table schema, ZSTD-compressed. Every record carries `v`, `record`, `node` and `at`; the rest of the columns are per record kind — event fields, flush statistics, and the counter fields with `status` flattened into five `status_*` columns and `durations` into one column per band.

Read them with any Parquet reader:

```sql
-- DuckDB
SELECT * FROM read_parquet('s3://<bucket>/_scafo/observe/<node>/**/*.parquet');
```

## Reading them back

Both commands read **straight out of the bucket**, so no node has to be up: the records were flushed by whichever nodes were running at the time, and those nodes may be gone. Both take the usual bucket flags (`--config`, `--endpoint`, `--region`, `--bucket`, `--prefix`, `--access-key`, `--secret-key`).

### `scafo metrics`

```sh
scafo metrics --since 1h
scafo metrics --since 7d --name shop --json
```

| Flag | Meaning |
|---|---|
| `--name` | Only this application. |
| `--env` | Narrow to the application `<name>-<env>` instead. |
| `--since` | A duration (`90s`, `15m`, `1h`, `7d`) or an ISO-8601 instant. Defaults to `1h`. |
| `--json` | Print the summary as JSON instead of a table. |

The snapshots are monotonic totals per node, so a window is a subtraction. Counted names are resolved against the applications that actually exist, so a label nothing answers to is nobody's.

### `scafo traces`

```sh
scafo traces --since 30m --name shop
scafo traces --since 1h --all --limit 500 --json
```

| Flag | Meaning |
|---|---|
| `--name` | Only what belongs to this application. |
| `--env` | Narrow to the application `<name>-<env>` instead. |
| `--since` | Same forms as above. Defaults to `1h`. |
| `--limit` | Events to show at most, newest kept. Defaults to `200`. |
| `--all` | Include the reconcile loop's own passes. |
| `--json` | Print the events as JSON instead of lines. |

Reconcile passes are held back by default so operational and audit events are not pushed out of the window. How many were held back is always reported.

## Runtime telemetry

When observability is on and `runtime.celldEnv` does not set it, the node runs every Celld process with `CELLD_OTEL=1`. An operator value wins: `CELLD_OTEL` is the one Celld variable that may override a Scafo-derived default.

Celld's telemetry is separate from Scafo's observations and is configured through Celld.

::read-more{to="/config/celld"}
Setting Celld runtime policy through `runtime.celldEnv`.
::