# Routing and ingress

> Every node renders the whole fleet's routing from the registry. Every ingress pulls it. Scafo never writes ingress configuration.

## The shape of it

```mermaid
sequenceDiagram
  participant C as Client
  participant I as Ingress (Caddy/Traefik)
  participant N as Node API :7400
  participant P as Celld process

  loop every poll_interval / pollInterval (5s)
    I->>N: GET /v1/ingress/routes[/traefik]?prefer=self<br/>Authorization: Bearer <ingress token>
    N-->>I: FleetRouting (or 503 before the first reconcile pass)
  end
  C->>I: GET https://shop.apps.example.com/
  I->>I: look up the normalized host in the cached document
  I->>P: proxy to the first upstream
  P-->>I: response
  I-->>C: response
```

The node serves the document; the ingress decides what to do with it. Nothing is pushed into an ingress, and operator-authored routes in the ingress configuration are never rewritten.

## Endpoints

| Path | Consumer | Auth |
|---|---|---|
| `GET /v1/ingress/routes` | Caddy, through the `scafo` dynamic-upstreams module | ingress bearer token |
| `GET /v1/ingress/routes/traefik` | Traefik, through its http provider | ingress bearer token |

`traefik` is the only registered native format; any other name is `404`. Both paths accept `?prefer=<node>` or `?prefer=self`.

The token comes from `--ingress-token` / `SCAFO_INGRESS_TOKEN` / `ingress.token`. When none is supplied the node mints one and writes it to `<state-dir>/ingress-token`, owner-readable.

## The routing document

```json
{
  "v": 1,
  "routes": [
    {
      "host": "shop.apps.example.com",
      "application": "shop",
      "upstreams": [
        { "node": "vps-a", "address": "10.83.0.1:24417" },
        { "node": "vps-b", "address": "10.83.0.2:24902" }
      ]
    }
  ],
  "api": {
    "hosts": ["apps.example.com"],
    "upstreams": [{ "node": "vps-a", "address": "10.83.0.1:7400" }],
    "denied": ["/v1/storage", "/v1/ingress/routes"]
  }
}
```

| Field | Meaning |
|---|---|
| `routes[].host` | A fully qualified host — an application's name under a platform domain, or a custom domain claimed for it — or one wildcard key, `*.<base>`, for a custom domain claimed with its subdomains. Sorted. |
| `routes[].application` | The application answering on that host. |
| `routes[].upstreams` | Every live node serving that application, at its advertised address and the process's public port. |
| `api` | Present only when the node publishes the platform API apex. |

### How it is built

1. Only **live, non-draining** nodes contribute upstreams. Stale and draining nodes are dropped here rather than left to a health check.
2. For each application with at least one upstream: `<name>.<domain>` under every platform domain.
3. Then every custom domain claimed for that application.
4. Then, for each custom domain claimed **with its subdomains**: one entry per `<env>.<base>` whose `<holder>-<env>` application is serving, and then the single wildcard key `*.<base>`.
5. First writer wins across those classes, so a custom-domain claim can never shadow a platform host, and a host claimed exactly always wins over the wildcard that encloses it.
6. A host nobody serves is left out entirely — and under a wildcard base it then falls to the `*.<base>` entry, which is the base holder's.

**There is no wildcard under a platform domain.** Every host there is rendered from an application's name, and a request for one nothing serves resolves to no upstream and is answered by the ingress itself. A custom domain may carry a wildcard, because it is your own name rather than the platform's.

A consumer looks a request's host up **exactly first** and, failing that, replaces the host's first label with `*` and looks that up **once**; two labels below the base match nothing. An ingress that knows nothing of the wildcard key simply never matches it.

::read-more{to="/docs/custom-domains"}
Claiming a host, and what `--wildcard` covers.
::

An ingress re-reads this document on a timer of its own, so for a while after something changes it still dials the addresses it last read. That is why an application's port does not move: a runtime process answers everything sent to its port, whatever `Host` the request names, so a port that had changed hands would answer the old application's traffic with the new application. Each pair is chosen once per node and then kept — see [port pairs](/docs/placement#port-pairs).

### Upstream order and `prefer`

`upstreams` is an ordered **priority list**, never a weighting. The first entry is where the request goes; the rest exist for when it fails.

| Query | Effect |
|---|---|
| *(none)* | The fleet's own order, by node id. |
| `?prefer=self` | The node answering the request goes first on every route. |
| `?prefer=<node-id>` | That node goes first. A node id nobody matches has no effect. |

Order among the remaining upstreams is preserved. Traefik's renderer uses the name rather than just the order: it emits a `failover` service, using the preferred node's upstreams while healthy and the others only when they are not.

## The platform API apex

When `api.public` is on (the default), the bare apex of each platform domain routes to the node API, so any node of the installation is reachable at one well-known name.

| Setting | Flag | Environment | Config key | Default |
|---|---|---|---|---|
| Publish the API apex | `--api-public` / `--no-api-public` | `SCAFO_API_PUBLIC` | `api.public` | `true` |

An apex an application already claims is left out of `api.hosts`: application routes win.

### Denied prefixes

```
/v1/storage
/v1/ingress/routes
```

::important
`denied` is the security rule of that surface, and **the ingress must enforce it itself** rather than trusting the daemon behind it. A request for one of these prefixes on a platform API host must be answered by the ingress and never proxied. Both are matched at a path **segment** boundary: the prefix exactly, or the prefix followed by `/` and anything.
::

The routing surface reveals every node's address for every host; the storage surface carries private keys in both directions. Both stay on the node's own listener behind the ingress token.

## Before the first reconcile pass

A node that has not completed a reconcile pass has no document at all. Both routing routes then refuse with `draining` (503) rather than serving `{"v":1,"routes":[]}` — an empty document is a statement that the fleet answers on nothing, and every route would be withdrawn.

How each ingress behaves on a failed poll:

| Ingress | Behaviour |
|---|---|
| Caddy (`scafo` module) | Keeps the last document that parsed and logs the failure once. With no document ever received, it resolves `scafo.route=unavailable` and Caddy answers `503`. |
| Traefik (http provider) | Logs the error and keeps its previous configuration. |

So a 503 for a few seconds after a node restart is a pause in one poller, not a withdrawal.

## Certificates

### Which names get a certificate

The routing document names exactly the hosts that may get a certificate, and the ingress decides from the document it already polls — there is no separate question to ask a node. Caddy's `tls.permission.scafo` approves an on-demand certificate only for a name some fleet's current document names exactly, as a route host or an API host, and answers from the documents it already holds. Traefik, told a certificate resolver, obtains what the rendering asks for on each router: its exact host, or the `tls.domains` wildcard certificate that covers it.

Which hosts the routing document names, and so which may get a certificate:

| Host kind | Named when |
|---|---|
| Bare platform apex | `api.public` is on, as an API host. |
| `<name>.<platform domain>` | That application is serving: a live node runs a ready process for it, which a retired or blocked application never has. |
| Custom domain | The application that claimed it is serving. |
| `<env>.<base>` under a custom domain claimed with its subdomains | `<holder>-<env>` is serving. |

A wildcard certificate for a platform domain does not cover its bare apex, which mints its own.

::important
The `*.<base>` entry of a claim that covers its subdomains exists for **routing, not for issuance**: it answers an arbitrary subdomain from the base holder's process, and no name is ever approved on its strength. Whoever sends the first request chooses the name, so a certificate per stranger's request would exhaust the certificate authority's weekly issuance limit for that domain. Those names are served over HTTPS only where the ingress already holds a certificate covering `*.<base>`.
::

### Certificate storage

Caddy's `caddy.storage.scafo` module keeps certificates, private keys and issuance locks in the bucket **through the node**, over `/v1/storage` with the ingress token. The ingress holds no bucket credential. Every node in the fleet therefore serves from one certificate store. The ACME account names no domain and belongs to no fleet, so it stays on the ingress's own disk.

| Path | Purpose |
|---|---|
| `PUT`/`GET`/`DELETE /v1/storage/objects/**` | Raw bytes. A delete removes the key and everything beneath it. |
| `GET /v1/storage/list?prefix=&recursive=` | Key listing; an empty result is `404`. |
| `GET /v1/storage/stat?key=` | Size and modification time. |
| `POST`/`DELETE /v1/storage/locks/**` | Try-acquire (`409` when held, never waits) and idempotent release. |

::read-more{to="/reference/ingress-protocol"}
The wire contract both surfaces implement.
::

## Platform domains versus custom domains

| | Platform domains | Custom domains |
|---|---|---|
| Configured on | The node (`SCAFO_PLATFORM_DOMAINS`) | `scafo app domain add`, as a claim in the registry |
| Host shape | `<name>.<domain>` and the bare apex | The claimed host, verbatim, and with `--wildcard` the single label below it |
| Certificates | One DNS-01 wildcard covers every subdomain; the apex is separate. HTTP-01 per name works without a DNS plugin. | Issued per name through on-demand TLS, for the claimed host and for a subdomain naming an application; the other subdomains of a base claimed with `--wildcard` need a DNS-01 wildcard. |
| API apex | Yes, when `api.public` is on | No |

::read-more{to="/docs/custom-domains"}
Claiming a host for an application.
::

::read-more{to="/docs/ingress-caddy"}
Configuring Caddy with the scafo modules.
::

::read-more{to="/docs/ingress-traefik"}
Configuring Traefik v3 with the http provider.
::