Ingress protocol

Every ingress pulls. Nothing is pushed into an ingress and nothing here calls one: a node renders the fleet's routing and serves it, and whatever sits in front reads it — as the generic document at /v1/ingress/routes, or in a native format at /v1/ingress/routes/<format>. Both surfaces are behind the same ingress token as /v1/storage.

#The routing document

GET /v1/ingress/routes answers FleetRouting.

{
  "v": 1,
  "routes": [
    {
      "host": "shop.dev.example",
      "application": "shop",
      "upstreams": [
        { "node": "vps-a", "address": "10.83.0.1:24417" },
        { "node": "vps-b", "address": "10.83.0.2:24902" }
      ]
    }
  ],
  "api": {
    "hosts": ["dev.example"],
    "upstreams": [
      { "node": "vps-a", "address": "10.83.0.1:7400" },
      { "node": "vps-b", "address": "10.83.0.2:7400" }
    ],
    "denied": ["/v1/storage", "/v1/ingress/routes"]
  }
}

#Fields

#Ordering and priority

upstreams is an ordered priority list, never a weighting. The first entry is where the request goes; the rest exist for when it fails. Nothing in scafo asks for traffic to be split.

The renderer sorts nodes by id. A consumer may ask for one node to be put first, which is how a colocated ingress stays local.

A route never carries an empty upstreams: an application nothing is running for right now is left out of the document altogether, so a host that is absent — and that no wildcard key covers — is a host to answer 404 for.

#What is in the document

Routes are rendered from what live nodes say they serve. Draining nodes and nodes that have stopped beating are dropped here rather than left to health checks — a node that said it is leaving should stop being chosen now, and a black-holed address times out rather than refusing.

For each application that has upstreams, one route is added for its name under every platform domain, and then every custom domain claimed for it. Then, per custom domain claimed with its subdomains, one route for each <env>.<base> whose <holder>-<env> application is serving, and finally the one key *.<base>. First writer wins across those classes, in code-unit order, so a claim can never shadow a platform host and a host claimed exactly always wins over the wildcard enclosing it.

A host nobody serves is left out entirely. Under a wildcard base it then falls to the *.<base> key, which is the base holder's; anywhere else it is a host to answer 404 for. There is no wildcard under a platform domain: every host there is rendered from an application's name.

#The wildcard key

*.<base> is the only key that is not a host. A consumer resolves a request host in two steps:

Two labels below the base match nothing, as under a wildcard certificate: a.b.shop.example reduces to *.b.shop.example, which is a different base.

A consumer that knows nothing of the key never matches it and answers as it would for a host the fleet does not serve — which is what it did before the key existed, and why the document stays v: 1.

#prefer

prefer reorders every route and the api upstreams: matching entries first, the rest in their existing order.

#Denied paths

The prefixes are /v1/storage and /v1/ingress/routes — the daemon's own control plane, which name where every process listens. They are meant to be reachable on loopback by whoever holds the ingress token, and never through the public ingress.

Matching is at a path-segment boundary: the prefix exactly, or the prefix followed by / and anything. With /v1/storage denied, /v1/storage and /v1/storage/objects/x are refused and /v1/storagex is not.

An application route always wins over an api host of the same name. The renderer already omits apexes an application answers on, so this is a tie-break that should never be needed.

#Boot behaviour

Until a node's first reconcile pass completes it has no document at all, and both routing routes answer:

503 application/problem+json
{ "type": "urn:scafo:draining", "title": "The fleet's routing is not known yet", "status": 503,
  "detail": "no reconcile pass has completed on node \"…\" yet, …" }

Treat it as any other fetch failure, never as an empty document: {"v":1,"routes":[]} would be every route in the fleet withdrawn, where a 503 is a pause of a few seconds in one poller.

Every ingress scafo speaks to keeps the last document that parsed when a poll does not return one. Caddy's module logs the failure once and serves what it holds, or — having never held one — resolves scafo.route=unavailable and lets Caddy answer 503. Traefik's http provider logs the error and keeps its previous configuration.

#Traefik rendering

GET /v1/ingress/routes/traefik[?prefer=…] answers Traefik dynamic configuration. The entryPoints come from the node's ingress.entryPoints setting and default to ["web"] — a router attached to an entryPoint the operator does not define is a route that matches nothing.

#Names

The scafo_ separator is an underscore, which cannot occur in a host.

#Rules

Each denied prefix expands to exactly two rules — Path(<prefix>) and PathPrefix(<prefix>/) — which is the segment-boundary match.

#Priorities and services

The *.<base> priority is stated rather than left to Traefik because Traefik's default priority is the length of the rule, which would put a long HostRegexp ahead of a short Host — precisely backwards for a fallback. The API routers are explicit for the same reason.

With prefer naming a node that answers on a route and at least one other node answering too, the service becomes a failover from a balancer over the preferred node to a balancer over the rest. Otherwise it is a single load balancer over every upstream in document order.

When there are no routers at all the answer is {} rather than empty maps — empty maps make Traefik retain its previous provider configuration.

#Example

The document above, rendered with entryPoints: ["web"] and ?prefer=vps-a:

{
  "http": {
    "routers": {
      "scafo_app.dev.example": {
        "rule": "Host(`app.dev.example`)",
        "service": "scafo_app.dev.example",
        "entryPoints": ["web"],
        "middlewares": ["scafo_retry"]
      },
      "scafo_apideny_dev.example": {
        "rule": "Host(`dev.example`) && (Path(`/v1/storage`) || PathPrefix(`/v1/storage/`) || Path(`/v1/ingress/routes`) || PathPrefix(`/v1/ingress/routes/`))",
        "service": "noop@internal",
        "entryPoints": ["web"],
        "priority": 3
      },
      "scafo_api_dev.example": {
        "rule": "Host(`dev.example`) && !Path(`/v1/storage`) && !PathPrefix(`/v1/storage/`) && !Path(`/v1/ingress/routes`) && !PathPrefix(`/v1/ingress/routes/`)",
        "service": "scafo_api_dev.example",
        "entryPoints": ["web"],
        "middlewares": ["scafo_retry"],
        "priority": 2
      }
    },
    "services": {
      "scafo_app.dev.example": {
        "failover": {
          "service": "scafo_app.dev.example_local",
          "fallback": "scafo_app.dev.example_peers"
        }
      },
      "scafo_app.dev.example_local": {
        "loadBalancer": {
          "servers": [{ "url": "http://10.83.0.1:24417" }],
          "healthCheck": {
            "path": "/.well-known/celld/health",
            "interval": "10s",
            "timeout": "3s",
            "hostname": "app.dev.example"
          }
        }
      },
      "scafo_app.dev.example_peers": {
        "loadBalancer": {
          "servers": [{ "url": "http://10.83.0.2:24902" }],
          "healthCheck": {
            "path": "/.well-known/celld/health",
            "interval": "10s",
            "timeout": "3s",
            "hostname": "app.dev.example"
          }
        }
      },
      "scafo_api_dev.example": {
        "failover": {
          "service": "scafo_api_dev.example_local",
          "fallback": "scafo_api_dev.example_peers"
        }
      },
      "scafo_api_dev.example_local": {
        "loadBalancer": {
          "servers": [{ "url": "http://10.83.0.1:7400" }],
          "healthCheck": {
            "path": "/.well-known/scafo/health",
            "interval": "10s",
            "timeout": "3s"
          }
        }
      },
      "scafo_api_dev.example_peers": {
        "loadBalancer": {
          "servers": [{ "url": "http://10.83.0.2:7400" }],
          "healthCheck": {
            "path": "/.well-known/scafo/health",
            "interval": "10s",
            "timeout": "3s"
          }
        }
      }
    },
    "middlewares": {
      "scafo_retry": { "retry": { "attempts": 3 } }
    }
  }
}

#Storage protocol v1

The other half of the ingress token: what a colocated TLS terminator keeps its certificates, account keys and issuance locks in. The daemon holds the bucket credentials; the ingress holds a loopback URL and a bearer token.

#Headers

Every request carries both:

The version header exists so a daemon can reject a client it cannot serve. An absent or empty header means version 1; any other value is invalid-input.

#Key encoding

Keys use / as a component separator and have no leading or trailing slash:

certificates/acme-v02.api.letsencrypt.org-directory/example.com/example.com.crt

A key maps onto the request path as-is: each component is percent-encoded individually and the / separators are preserved, so the daemon sees the same hierarchy the caller used. Keys passed as query parameters (list, stat) use ordinary form encoding.

A key must be relative, made of printable ASCII, and carry no empty, . or .. segment. Anything else is invalid-input, Invalid storage key.

#Endpoints

#The 404 rule

GET /v1/storage/list answers 404 when nothing exists at or under the prefix — a 200 always carries at least one key. CertMagic expects listing a missing prefix to be reported as "not found", not as an empty result, and translates a 404 into an error wrapping io/fs.ErrNotExist.

#Stat body

{
  "key": "certificates/acme/example.com/example.com.crt",
  "modified": "2026-04-01T12:30:00Z",
  "size": 4096,
  "isTerminal": true
}

#Lock semantics

POST /v1/storage/locks/<name> is try-acquire and never waits. It answers 409 Conflict (urn:scafo:cas-conflict) when the lock is currently held elsewhere; the client then waits one second and retries indefinitely until the lock is granted or its context is cancelled. Any other error status ends the attempt immediately rather than being retried.

Exactly one holder may hold a given name at a time. The daemon expires locks left behind by a crashed holder: a holder restamps every 5 s and a lock unrefreshed for 10 s may be taken over. That expiry is what makes several ingress instances able to share one bucket.

A node that loses a refresh drops the lock locally, whatever went wrong: a refused conditional write and an unreachable bucket are indistinguishable, and going on holding a lock the process may no longer have is the one outcome a lock exists to prevent.

DELETE /v1/storage/locks/<name> is idempotent and answers 204 even when the lock is not held.

#Client behaviour

The Caddy storage module uses a dedicated HTTP client per fleet, with a per-request timeout of that fleet's storage_timeout (30 seconds by default), which does not bound Lock — that blocks until the lock is acquired or its context is done. It performs no retries other than the lock poll: the daemon is colocated, so a failure is a fault to report rather than a transient to paper over.

The routing poller uses a 10-second per-request timeout and waits poll_interval between attempts rather than on a schedule.

#Errors

Non-2xx responses on both protocols use application/problem+json. The title and detail members, when present, are surfaced in the error a client reports; otherwise the error reports the status code alone.