Routing and ingress

#The shape of it

The node serves the document; the ingress decides what to do with it. Nothing is pushed into an ingress, and operator-authored routes in the ingress configuration are never rewritten.

#Endpoints

traefik is the only registered native format; any other name is 404. Both paths accept ?prefer=<node> or ?prefer=self.

The token comes from --ingress-token / SCAFO_INGRESS_TOKEN / ingress.token. When none is supplied the node mints one and writes it to <state-dir>/ingress-token, owner-readable.

#The routing document

{
  "v": 1,
  "routes": [
    {
      "host": "shop.apps.example.com",
      "application": "shop",
      "upstreams": [
        { "node": "vps-a", "address": "10.83.0.1:24417" },
        { "node": "vps-b", "address": "10.83.0.2:24902" }
      ]
    }
  ],
  "api": {
    "hosts": ["apps.example.com"],
    "upstreams": [{ "node": "vps-a", "address": "10.83.0.1:7400" }],
    "denied": ["/v1/storage", "/v1/ingress/routes"]
  }
}

#How it is built

There is no wildcard under a platform domain. Every host there is rendered from an application's name, and a request for one nothing serves resolves to no upstream and is answered by the ingress itself. A custom domain may carry a wildcard, because it is your own name rather than the platform's.

A consumer looks a request's host up exactly first and, failing that, replaces the host's first label with * and looks that up once; two labels below the base match nothing. An ingress that knows nothing of the wildcard key simply never matches it.

An ingress re-reads this document on a timer of its own, so for a while after something changes it still dials the addresses it last read. That is why an application's port does not move: a runtime process answers everything sent to its port, whatever Host the request names, so a port that had changed hands would answer the old application's traffic with the new application. Each pair is chosen once per node and then kept — see port pairs.

#Upstream order and prefer

upstreams is an ordered priority list, never a weighting. The first entry is where the request goes; the rest exist for when it fails.

Order among the remaining upstreams is preserved. Traefik's renderer uses the name rather than just the order: it emits a failover service, using the preferred node's upstreams while healthy and the others only when they are not.

#The platform API apex

When api.public is on (the default), the bare apex of each platform domain routes to the node API, so any node of the installation is reachable at one well-known name.

An apex an application already claims is left out of api.hosts: application routes win.

#Denied prefixes

/v1/storage
/v1/ingress/routes

The routing surface reveals every node's address for every host; the storage surface carries private keys in both directions. Both stay on the node's own listener behind the ingress token.

#Before the first reconcile pass

A node that has not completed a reconcile pass has no document at all. Both routing routes then refuse with draining (503) rather than serving {"v":1,"routes":[]} — an empty document is a statement that the fleet answers on nothing, and every route would be withdrawn.

How each ingress behaves on a failed poll:

So a 503 for a few seconds after a node restart is a pause in one poller, not a withdrawal.

#Certificates

#Which names get a certificate

The routing document names exactly the hosts that may get a certificate, and the ingress decides from the document it already polls — there is no separate question to ask a node. Caddy's tls.permission.scafo approves an on-demand certificate only for a name some fleet's current document names exactly, as a route host or an API host, and answers from the documents it already holds. Traefik, told a certificate resolver, obtains what the rendering asks for on each router: its exact host, or the tls.domains wildcard certificate that covers it.

Which hosts the routing document names, and so which may get a certificate:

A wildcard certificate for a platform domain does not cover its bare apex, which mints its own.

#Certificate storage

Caddy's caddy.storage.scafo module keeps certificates, private keys and issuance locks in the bucket through the node, over /v1/storage with the ingress token. The ingress holds no bucket credential. Every node in the fleet therefore serves from one certificate store. The ACME account names no domain and belongs to no fleet, so it stays on the ingress's own disk.

#Platform domains versus custom domains