# Ingress protocol

> The routing document, its ordering rules, the Traefik rendering, and the TLS storage protocol.

Every ingress **pulls**. Nothing is pushed into an ingress and nothing here calls one: a node renders the fleet's routing and serves it, and whatever sits in front reads it — as the generic document at `/v1/ingress/routes`, or in a native format at `/v1/ingress/routes/<format>`. Both surfaces are behind the same ingress token as `/v1/storage`.

## The routing document

`GET /v1/ingress/routes` answers `FleetRouting`.

```json
{
  "v": 1,
  "routes": [
    {
      "host": "shop.dev.example",
      "application": "shop",
      "upstreams": [
        { "node": "vps-a", "address": "10.83.0.1:24417" },
        { "node": "vps-b", "address": "10.83.0.2:24902" }
      ]
    }
  ],
  "api": {
    "hosts": ["dev.example"],
    "upstreams": [
      { "node": "vps-a", "address": "10.83.0.1:7400" },
      { "node": "vps-b", "address": "10.83.0.2:7400" }
    ],
    "denied": ["/v1/storage", "/v1/ingress/routes"]
  }
}
```

### Fields

| Field | Type | Meaning |
| --- | --- | --- |
| `v` | `1` | Document version. Any other value must be refused whole rather than routed against |
| `routes` | array | Every host the fleet answers on, sorted by `host`, each host named once |
| `routes[].host` | string | Fully qualified: an application's name under a platform domain, or a custom domain that application claimed — **or** one wildcard key, `*.<base>`, standing for everything exactly one label under a custom domain claimed with its subdomains |
| `routes[].application` | string | The application answering on this host |
| `routes[].upstreams` | array | Who answers on this host, **in priority order** |
| `routes[].upstreams[].node` | string | Node id |
| `routes[].upstreams[].address` | string | `host:port` — the node's advertised address and the process's public port |
| `api` | object? | Absent when the rendering node does not publish the fleet API |
| `api.hosts` | string[] | Bare platform apexes routed to the daemon itself, sorted |
| `api.upstreams` | array | The daemons' own addresses, ordered like a route's |
| `api.denied` | string[] | Path prefixes the ingress must answer itself and never proxy |

### Ordering and priority

`upstreams` is an **ordered priority list, never a weighting**. The first entry is where the request goes; the rest exist for when it fails. Nothing in scafo asks for traffic to be split.

The renderer sorts nodes by id. A consumer may ask for one node to be put first, which is how a colocated ingress stays local.

A route never carries an empty `upstreams`: an application nothing is running for right now is left out of the document altogether, so a host that is absent — and that no wildcard key covers — is a host to answer `404` for.

### What is in the document

Routes are rendered from what **live** nodes say they serve. Draining nodes and nodes that have stopped beating are dropped here rather than left to health checks — a node that said it is leaving should stop being chosen now, and a black-holed address times out rather than refusing.

For each application that has upstreams, one route is added for its name under every platform domain, and then every custom domain claimed for it. Then, per custom domain claimed **with its subdomains**, one route for each `<env>.<base>` whose `<holder>-<env>` application is serving, and finally the one key `*.<base>`. **First writer wins** across those classes, in code-unit order, so a claim can never shadow a platform host and a host claimed exactly always wins over the wildcard enclosing it.

A host nobody serves is left out entirely. Under a wildcard base it then falls to the `*.<base>` key, which is the base holder's; anywhere else it is a host to answer `404` for. **There is no wildcard under a platform domain**: every host there is rendered from an application's name.

### The wildcard key

`*.<base>` is the only key that is not a host. A consumer resolves a request host in two steps:

1. Look the host up **exactly**. A hit wins, always.
2. Otherwise replace the host's **first label** with `*` and look that up **once**.

Two labels below the base match nothing, as under a wildcard certificate: `a.b.shop.example` reduces to `*.b.shop.example`, which is a different base.

A consumer that knows nothing of the key never matches it and answers as it would for a host the fleet does not serve — which is what it did before the key existed, and why the document stays `v: 1`.

### `prefer`

| Value | Effect |
| --- | --- |
| absent / empty | The fleet's own order. This is the only way to ask for it |
| `self` | The node answering the request, resolved to its own id |
| any other value | Read as a node id, applied verbatim. One naming no node in the fleet simply has no effect |

`prefer` reorders every route and the `api` upstreams: matching entries first, the rest in their existing order.

### Denied paths

::important
`api.denied` **is the security rule of the fleet API surface**, and the ingress must enforce it itself rather than trusting the daemon behind it. A request for one of these prefixes on one of the `api.hosts` must be answered by the ingress and must never be proxied.
::

The prefixes are `/v1/storage` and `/v1/ingress/routes` — the daemon's own control plane, which name where every process listens. They are meant to be reachable on loopback by whoever holds the ingress token, and never through the public ingress.

Matching is at a **path-segment boundary**: the prefix exactly, or the prefix followed by `/` and anything. With `/v1/storage` denied, `/v1/storage` and `/v1/storage/objects/x` are refused and `/v1/storagex` is not.

An application route always wins over an `api` host of the same name. The renderer already omits apexes an application answers on, so this is a tie-break that should never be needed.

### Boot behaviour

Until a node's first reconcile pass completes it has no document at all, and both routing routes answer:

```
503 application/problem+json
{ "type": "urn:scafo:draining", "title": "The fleet's routing is not known yet", "status": 503,
  "detail": "no reconcile pass has completed on node \"…\" yet, …" }
```

Treat it as any other fetch failure, never as an empty document: `{"v":1,"routes":[]}` would be every route in the fleet withdrawn, where a 503 is a pause of a few seconds in one poller.

Every ingress scafo speaks to keeps the last document that parsed when a poll does not return one. Caddy's module logs the failure once and serves what it holds, or — having never held one — resolves `scafo.route=unavailable` and lets Caddy answer 503. Traefik's http provider logs the error and keeps its previous configuration.

## Traefik rendering

`GET /v1/ingress/routes/traefik[?prefer=…]` answers Traefik dynamic configuration. The entryPoints come from the node's `ingress.entryPoints` setting and default to `["web"]` — a router attached to an entryPoint the operator does not define is a route that matches nothing.

### Names

| Thing | Name |
| --- | --- |
| Router and service for a host | `scafo_<host>` |
| Router and service for a `*.<base>` key | `scafo_wild_<base>` |
| Router and service for one platform apex's API | `scafo_api_<domain>` |
| Router denying paths at one platform apex | `scafo_apideny_<domain>` |
| Failover halves, when `prefer` applies | `<key>_local` and `<key>_peers` |
| The one middleware | `scafo_retry` |

The `scafo_` separator is an underscore, which cannot occur in a host.

### Rules

| Router | Rule |
| --- | --- |
| Host route | ``Host(`<host>`)`` |
| `*.<base>` route | ``HostRegexp(`^[^.]+\.<base with its dots escaped>$`)`` — exactly one label under the base |
| API deny | ``Host(`<domain>`) && (Path(`/v1/storage`) \|\| PathPrefix(`/v1/storage/`) \|\| Path(`/v1/ingress/routes`) \|\| PathPrefix(`/v1/ingress/routes/`))`` |
| API | ``Host(`<domain>`) && !Path(`/v1/storage`) && !PathPrefix(`/v1/storage/`) && !Path(`/v1/ingress/routes`) && !PathPrefix(`/v1/ingress/routes/`)`` |

Each denied prefix expands to exactly two rules — `Path(<prefix>)` and `PathPrefix(<prefix>/)` — which is the segment-boundary match.

### Priorities and services

| Setting | Value |
| --- | --- |
| API deny router priority | `3` |
| API router priority | `2` |
| `*.<base>` router priority | `1` — below every other router this document writes |
| Host router priority | unset (Traefik's default) |
| API deny router service | `noop@internal` — Traefik's internal no-op backend, so the route is active without dialling anything |
| Middleware `scafo_retry` | `{ "retry": { "attempts": 3 } }`, carried by every router except the deny routers |
| Health check for a host service | path `/.well-known/celld/health`, `interval: "10s"`, `timeout: "3s"`, `hostname: <host>` — for a `*.<base>` service the `hostname` is the **base**, never the pattern |
| Health check for an API service | path `/.well-known/scafo/health`, `interval: "10s"`, `timeout: "3s"`, no hostname |
| Server URL | `http://<upstream address>` |

The `*.<base>` priority is stated rather than left to Traefik because Traefik's default priority is the **length** of the rule, which would put a long `HostRegexp` ahead of a short `Host` — precisely backwards for a fallback. The API routers are explicit for the same reason.

With `prefer` naming a node that answers on a route **and** at least one other node answering too, the service becomes a `failover` from a balancer over the preferred node to a balancer over the rest. Otherwise it is a single load balancer over every upstream in document order.

When there are no routers at all the answer is `{}` rather than empty maps — empty maps make Traefik retain its previous provider configuration.

### Example

The document above, rendered with `entryPoints: ["web"]` and `?prefer=vps-a`:

```json
{
  "http": {
    "routers": {
      "scafo_app.dev.example": {
        "rule": "Host(`app.dev.example`)",
        "service": "scafo_app.dev.example",
        "entryPoints": ["web"],
        "middlewares": ["scafo_retry"]
      },
      "scafo_apideny_dev.example": {
        "rule": "Host(`dev.example`) && (Path(`/v1/storage`) || PathPrefix(`/v1/storage/`) || Path(`/v1/ingress/routes`) || PathPrefix(`/v1/ingress/routes/`))",
        "service": "noop@internal",
        "entryPoints": ["web"],
        "priority": 3
      },
      "scafo_api_dev.example": {
        "rule": "Host(`dev.example`) && !Path(`/v1/storage`) && !PathPrefix(`/v1/storage/`) && !Path(`/v1/ingress/routes`) && !PathPrefix(`/v1/ingress/routes/`)",
        "service": "scafo_api_dev.example",
        "entryPoints": ["web"],
        "middlewares": ["scafo_retry"],
        "priority": 2
      }
    },
    "services": {
      "scafo_app.dev.example": {
        "failover": {
          "service": "scafo_app.dev.example_local",
          "fallback": "scafo_app.dev.example_peers"
        }
      },
      "scafo_app.dev.example_local": {
        "loadBalancer": {
          "servers": [{ "url": "http://10.83.0.1:24417" }],
          "healthCheck": {
            "path": "/.well-known/celld/health",
            "interval": "10s",
            "timeout": "3s",
            "hostname": "app.dev.example"
          }
        }
      },
      "scafo_app.dev.example_peers": {
        "loadBalancer": {
          "servers": [{ "url": "http://10.83.0.2:24902" }],
          "healthCheck": {
            "path": "/.well-known/celld/health",
            "interval": "10s",
            "timeout": "3s",
            "hostname": "app.dev.example"
          }
        }
      },
      "scafo_api_dev.example": {
        "failover": {
          "service": "scafo_api_dev.example_local",
          "fallback": "scafo_api_dev.example_peers"
        }
      },
      "scafo_api_dev.example_local": {
        "loadBalancer": {
          "servers": [{ "url": "http://10.83.0.1:7400" }],
          "healthCheck": {
            "path": "/.well-known/scafo/health",
            "interval": "10s",
            "timeout": "3s"
          }
        }
      },
      "scafo_api_dev.example_peers": {
        "loadBalancer": {
          "servers": [{ "url": "http://10.83.0.2:7400" }],
          "healthCheck": {
            "path": "/.well-known/scafo/health",
            "interval": "10s",
            "timeout": "3s"
          }
        }
      }
    },
    "middlewares": {
      "scafo_retry": { "retry": { "attempts": 3 } }
    }
  }
}
```

::read-more{to="/docs/ingress-traefik"}
::

## Storage protocol v1

The other half of the ingress token: what a colocated TLS terminator keeps its certificates, account keys and issuance locks in. The daemon holds the bucket credentials; the ingress holds a loopback URL and a bearer token.

### Headers

Every request carries both:

| Header | Value |
| --- | --- |
| `Authorization` | `Bearer <token>`, the trimmed ingress token |
| `X-Scafo-Storage-Version` | `1` |

The version header exists so a daemon can reject a client it cannot serve. An absent or empty header means version 1; any other value is `invalid-input`.

### Key encoding

Keys use `/` as a component separator and have no leading or trailing slash:

```
certificates/acme-v02.api.letsencrypt.org-directory/example.com/example.com.crt
```

A key maps onto the request path as-is: each component is percent-encoded individually and the `/` separators are preserved, so the daemon sees the same hierarchy the caller used. Keys passed as query parameters (`list`, `stat`) use ordinary form encoding.

A key must be relative, made of printable ASCII, and carry no empty, `.` or `..` segment. Anything else is `invalid-input`, *Invalid storage key*.

### Endpoints

| Method | Path | Request | Success | Not found |
| --- | --- | --- | --- | --- |
| `PUT` | `/v1/storage/objects/<key>` | raw value bytes | `204` | — |
| `GET` | `/v1/storage/objects/<key>` | — | `200` with raw value bytes | `404` |
| `DELETE` | `/v1/storage/objects/<key>` | — | `204`; deletes the key and its subtree | `404` |
| `GET` | `/v1/storage/list?prefix=<p>&recursive=<true\|false>` | — | `200` with JSON `[]string` of keys | `404` |
| `GET` | `/v1/storage/stat?key=<k>` | — | `200` with a key-info object | `404` |
| `POST` | `/v1/storage/locks/<name>` | — | `204`, lock acquired | — |
| `DELETE` | `/v1/storage/locks/<name>` | — | `204`, always, idempotent | — |

### The 404 rule

`GET /v1/storage/list` answers `404` when nothing exists at or under the prefix — a `200` always carries at least one key. CertMagic expects listing a missing prefix to be reported as "not found", not as an empty result, and translates a `404` into an error wrapping `io/fs.ErrNotExist`.

::important
A node daemon must answer `404`, not `500`, for keys that do not exist. Getting this wrong makes CertMagic treat absent certificates as infrastructure failures.
::

### Stat body

```json
{
  "key": "certificates/acme/example.com/example.com.crt",
  "modified": "2026-04-01T12:30:00Z",
  "size": 4096,
  "isTerminal": true
}
```

| Field | Type | Meaning |
| --- | --- | --- |
| `key` | string | The key as it was asked about |
| `modified` | RFC 3339 string or `null` | When the value was last written; `null` for a key that only has keys beneath it, and becomes CertMagic's zero time |
| `size` | number | Length of the decoded value; `0` for a key that only has keys beneath it |
| `isTerminal` | boolean | `true` for a key holding a value, `false` for one that merely acts as a prefix |

### Lock semantics

`POST /v1/storage/locks/<name>` is **try-acquire and never waits**. It answers `409 Conflict` (`urn:scafo:cas-conflict`) when the lock is currently held elsewhere; the client then waits one second and retries indefinitely until the lock is granted or its context is cancelled. Any other error status ends the attempt immediately rather than being retried.

Exactly one holder may hold a given name at a time. **The daemon expires locks left behind by a crashed holder**: a holder restamps every 5 s and a lock unrefreshed for 10 s may be taken over. That expiry is what makes several ingress instances able to share one bucket.

A node that loses a refresh drops the lock locally, whatever went wrong: a refused conditional write and an unreachable bucket are indistinguishable, and going on holding a lock the process may no longer have is the one outcome a lock exists to prevent.

`DELETE /v1/storage/locks/<name>` is idempotent and answers `204` even when the lock is not held.

### Client behaviour

The Caddy storage module uses a dedicated HTTP client per fleet, with a per-request timeout of that fleet's `storage_timeout` (**30 seconds** by default), which does not bound `Lock` — that blocks until the lock is acquired or its context is done. It performs no retries other than the lock poll: the daemon is colocated, so a failure is a fault to report rather than a transient to paper over.

The routing poller uses a **10-second** per-request timeout and waits `poll_interval` *between* attempts rather than on a schedule.

::read-more{to="/reference/caddy-modules"}
::

## Errors

Non-2xx responses on both protocols use `application/problem+json`. The `title` and `detail` members, when present, are surfaced in the error a client reports; otherwise the error reports the status code alone.

::read-more{to="/reference/errors"}
::