# Caddy with the scafo modules

> `scafo-caddy` is Caddy with the scafo modules. You tell it where your fleets are, and it routes each request, keeps certificates in each fleet's bucket and decides which names may get one.

## What it is

Upstream Caddy cannot route for Scafo. `scafo-caddy` is Caddy with five modules compiled in, and all of them read one place: the `scafo` **app**, which says where each fleet's node is and which token opens it.

| Module | Caddy module ID | What it does |
|---|---|---|
| The app | `scafo` | Where the fleets are, which token opens each one, which DNS providers exist. |
| TLS storage | `caddy.storage.scafo` | Keeps certificates in each fleet's own bucket, through that fleet's node. |
| Dynamic upstreams | `http.reverse_proxy.upstreams.scafo` | Resolves a request's backends from the routing every fleet's node serves at `/v1/ingress/routes`. |
| On-demand permission | `tls.permission.scafo` | Decides, at handshake time, which names may get a certificate. |
| Plain-HTTP matcher | `http.matchers.scafo_plain_http` | Matches a host whose fleet asked not to be redirected to HTTPS. |

A **fleet** here is one Scafo installation: one node endpoint, one ingress token, one routing document, one bucket. Caddy holds the endpoint and the token and nothing else. The node holds the bucket credentials and does the reading and writing, so there is no path from a compromised Caddy to the rest of the bucket, and rotating the token never involves the storage provider.

The token is the node's **ingress token**: `SCAFO_INGRESS_TOKEN` on the node, or the one the node mints into its state directory (`/var/lib/scafo/ingress-token`) when that is unset. It opens `/v1/ingress/routes` and `/v1/storage` on that node and nothing else.

## One fleet, from the environment

The published image needs **no Caddyfile and no configuration file**. It ships a default Caddyfile that declares an empty `scafo` block, and the app fills itself in from the environment. This is the intended way to run it.

```sh
docker run -p 80:80 -p 443:443 -p 443:443/udp \
  -e SCAFO_FLEET_ENDPOINT=http://127.0.0.1:7400 \
  -e SCAFO_FLEET_TOKEN=... \
  ghcr.io/sandros94/scafo-caddy:0.2.1
```

That is the whole configuration. The fleet is named `default`. Its routing document is polled every five seconds, certificates go into its bucket, and every name it routes exactly gets a certificate on demand.

With `SCAFO_FLEET_TOKEN_FILE` instead of `SCAFO_FLEET_TOKEN`, the token is read from a file once at startup. Give exactly one of the two.

### Beside a node in Compose

The usual shape is Caddy sharing the node's network namespace, which is what makes `127.0.0.1:7400` a real loopback address:

```yaml [compose.yaml]
services:
  scafo-node:
    image: ghcr.io/sandros94/scafo:latest
    environment:
      SCAFO_INGRESS_TOKEN: ${INGRESS_TOKEN:?set INGRESS_TOKEN in .env}
      # ...the node's own bucket credentials and settings
    ports:
      - "80:80"
      - "443:443"
      - "443:443/udp"
    volumes:
      - state:/var/lib/scafo

  caddy:
    image: ghcr.io/sandros94/scafo-caddy:0.2.1
    network_mode: "service:scafo-node"
    depends_on:
      - scafo-node
    environment:
      SCAFO_FLEET_ENDPOINT: http://127.0.0.1:7400
      SCAFO_FLEET_TOKEN: ${INGRESS_TOKEN:?set INGRESS_TOKEN in .env}
    volumes:
      - caddy-data:/data

volumes:
  state:
  caddy-data:
```

Keep `/data`. Certificates live in the bucket, but the ACME account and Caddy's own housekeeping live there, and losing it means registering with the certificate authority again.

::read-more{to="/deploy/compose"}
The complete single-host recipe.
::

### What the default configuration does

| Behaviour | Detail |
|---|---|
| Routing | Every request over HTTPS is resolved from the fleets' routing documents. No backend address is written anywhere, so a deployment that moves costs no reload. |
| Certificates | Issued on demand for a name a fleet's routing document names exactly, and stored in that fleet's bucket. |
| HTTP | Redirected to HTTPS with a `308`, unless the host's fleet has `SCAFO_FLEET_HTTP_REDIRECT=off`. |
| Unknown hosts | A host no fleet serves is answered `404` with a short page. |
| Access log | JSON, written to `SCAFO_ACCESS_LOG`, which defaults to `/dev/stdout`. |

### Settings you may want

| Variable | Default | What it changes |
|---|---|---|
| `SCAFO_FLEET_PREFER` | `self` | Whose upstreams head every route. `self` is the node answering, so a request stays on the machine it arrived at; a node id prefers that node; `none` takes the fleet's own order. |
| `SCAFO_FLEET_POLL_INTERVAL` | `5s` | How long between fetches of the routing document. It bounds how long a deployment that just moved keeps being proxied to where it was. |
| `SCAFO_FLEET_HTTP_REDIRECT` | `on` | `off` serves this fleet's hosts over plain HTTP as well. Useful behind a load balancer that terminates TLS itself. |
| `SCAFO_ACCESS_LOG` | `/dev/stdout` | Where the access log goes. Point it at a file the node also reads to have the node count requests. |

Every variable is listed in the [module reference](/reference/caddy-modules#every-environment-variable).

## Several fleets behind one ingress

One Caddy can stand in front of several fleets at once. The typical case is an agency that runs one fleet per client: one ingress per machine, in front of one node per client, each client's installation with its own bucket and its own token.

Each fleet gets a name inside the variable: `SCAFO_FLEET_<NAME>_ENDPOINT`, `SCAFO_FLEET_<NAME>_TOKEN`, and so on. `<NAME>` is uppercase letters, digits and underscores, and becomes the fleet's name lowercased with underscores turned into hyphens: `SCAFO_FLEET_CLIENT_A_ENDPOINT` is fleet `client-a`.

```sh
docker run -p 80:80 -p 443:443 -p 443:443/udp \
  -e SCAFO_FLEET_CLIENT_A_ENDPOINT=http://10.0.0.1:7400 \
  -e SCAFO_FLEET_CLIENT_A_TOKEN=... \
  -e SCAFO_FLEET_CLIENT_B_ENDPOINT=http://10.0.0.2:7400 \
  -e SCAFO_FLEET_CLIENT_B_TOKEN=... \
  -e SCAFO_FLEET_CLIENT_B_HTTP_REDIRECT=off \
  ghcr.io/sandros94/scafo-caddy:0.2.1
```

Every per-fleet setting takes the same form, so each fleet has its own poll interval, its own `prefer` and its own redirect choice. Above, `client-b`'s hosts are also served over plain HTTP, and `client-a`'s are still redirected.

::warning
**The single and the named forms cannot be mixed.** `SCAFO_FLEET_TOKEN` beside `SCAFO_FLEET_A_TOKEN` fails startup with an error naming both variables, and so does `SCAFO_FLEET_ENDPOINT` beside `SCAFO_FLEET_DEFAULT_ENDPOINT`, which would be the same fleet twice.
::

Nothing is guessed either. A `SCAFO_FLEET_*` variable this build does not recognize fails startup, naming it and listing the ones it does: a misspelled variable that was quietly ignored would be a fleet that quietly does not exist. A fleet with an endpoint and no token fails too.

### How a request finds its fleet

Every fleet has its own poller and its own cached routing document, and a request is resolved against all of them at once. The host, lowercased and without its port, is tried in this order, and the first match answers:

1. a route naming it exactly, in any fleet;
2. a fleet API host naming it exactly;
3. a wildcard route covering it.

So an exact route beats a wildcard even across fleets. When two fleets publish the same host, the one that published it **first** keeps it for as long as it goes on publishing it, and the conflict is logged once per change. The host never flaps between the two; the conflict is something to fix.

A fleet whose node is unreachable affects only its own hosts. Its last document keeps answering, and the other fleets are untouched.

### Certificates never cross fleets

Every certificate, private key and issuance lock is stored with the fleet that owns its name, written through that fleet's node with that fleet's token. One client's private key never lands in another client's bucket, and a name no fleet owns is written nowhere at all: it gets no default fleet.

What names no domain belongs to no fleet, so it stays on the ingress's own disk, under Caddy's data directory. That includes the **ACME account**: it is the ingress's, not any fleet's, and putting it in one fleet's bucket would let that fleet impersonate the ingress to its certificate authority. Keep `/data` on a volume.

### Reaching nodes over a network

With one node per client, each node's API is usually reached over a shared private network rather than loopback:

```yaml [compose.yaml]
services:
  caddy:
    image: ghcr.io/sandros94/scafo-caddy:0.2.1-dns
    ports:
      - "80:80"
      - "443:443"
      - "443:443/udp"
    environment:
      SCAFO_FLEET_CLIENT_A_ENDPOINT: http://node-a:7400
      SCAFO_FLEET_CLIENT_A_TOKEN: ${TOKEN_A:?}
      SCAFO_FLEET_CLIENT_B_ENDPOINT: http://node-b:7400
      SCAFO_FLEET_CLIENT_B_TOKEN: ${TOKEN_B:?}
      SCAFO_DNS_PROVIDER: cloudflare
      SCAFO_DNS_API_TOKEN: ${CLOUDFLARE_API_TOKEN:?}
      SCAFO_DNS_ZONES: apps.example.com
      SCAFO_ACME_DELEGATION: acme.example.com
    volumes:
      - caddy-data:/data

volumes:
  caddy-data:
```

The token then travels over that network: keep it private, or use `https` endpoints.

## Wildcard certificates

A fleet's routing document lists the bases it wants a `*.<base>` certificate for: its platform domains and every custom domain [claimed with its subdomains](/docs/custom-domains#claiming-the-subdomains-too). Give the ingress a DNS provider and it obtains those wildcards itself, over the ACME DNS-01 challenge:

```sh
  -e SCAFO_DNS_PROVIDER=cloudflare \
  -e SCAFO_DNS_API_TOKEN=... \
  -e SCAFO_DNS_ZONES=apps.example.com \
  -e SCAFO_ACME_DELEGATION=acme.example.com \
  -e SCAFO_ACME_EMAIL=ops@example.com
```

`SCAFO_DNS_<FIELD>`, lowercased, is one of the provider module's own settings: `SCAFO_DNS_API_TOKEN` is `api_token`. This needs the **`-dns` image flavour**, the only one with DNS provider modules compiled in. With no DNS provider configured, nothing is managed, and only the names a fleet routes exactly are covered, on demand, one certificate each.

A name covered only by a wildcard route is **never** issued on demand. Anyone can point a name at the ingress, and a certificate per arbitrary subdomain would let a stranger exhaust a customer's weekly issuance limit at the certificate authority. Those names are served over HTTPS through the managed wildcard only.

::read-more{to="/docs/wildcard-certificates"}
The two situations, the records a customer creates, and why listing your zones matters.
::

## The Caddyfile form

When the default cannot do what you need — another site, your own error page, headers, rate limiting — mount a Caddyfile of your own over `/etc/caddy/Caddyfile`. It **replaces** the default entirely. The module repository's `examples/Caddyfile` is the default written out in full, and the place to start.

Fleets are declared in the `scafo` global option, once. No other directive takes an endpoint or a token:

```caddyfile [Caddyfile]
{
	scafo {
		fleet default {
			endpoint http://127.0.0.1:7400
			token    {env.SCAFO_INGRESS_TOKEN}
		}
	}

	# Certificates live in the bucket of the fleet that owns the name.
	storage scafo

	# A name gets a certificate on demand only when a fleet routes it exactly.
	on_demand_tls {
		permission scafo
	}
}

(scafo_proxy) {
	reverse_proxy {
		dynamic scafo

		lb_policy       first
		lb_retries      2
		lb_try_duration 5s

		fail_duration 30s
		max_fails     1
	}
}

(scafo_errors) {
	handle_errors {
		@missing vars scafo.route unknown denied
		handle @missing {
			respond "No deployment answers to {http.vars.scafo.host}." 404
		}
		respond "{err.status_code} {err.status_text}" {err.status_code}
	}
}

# Plain HTTP: served as-is for a fleet with `http_redirect off`, redirected
# to HTTPS for everything else.
http:// {
	@plain scafo_plain_http
	handle @plain {
		import scafo_proxy
	}
	handle {
		redir https://{host}{uri} 308
	}
	import scafo_errors
}

# Everything over HTTPS, whichever fleet serves it.
https:// {
	tls {
		on_demand
	}
	import scafo_proxy
	import scafo_errors
}
```

::important
**Writing a fleet down replaces the environment, it does not add to it.** When the configuration declares at least one fleet, no `SCAFO_FLEET_*` variable is read. The same holds for DNS providers on their own. `acme_delegation`, `acme_email` and `acme_ca` are the exception: a value written in the configuration wins, and the environment fills in only what it left empty.
::

Several fleets are several `fleet` blocks, each named with a slug:

```caddyfile
scafo {
	fleet client-a {
		endpoint http://10.0.0.1:7400
		token    {env.TOKEN_A}
	}
	fleet client-b {
		endpoint      http://10.0.0.2:7400
		token_file    /run/secrets/client-b
		http_redirect off
	}
}
```

`storage scafo`, `dynamic scafo` and `permission scafo` take **no options**. An `endpoint` or a `token` inside one of them is refused at startup, with an error saying that fleets are configured in the `scafo` global option.

::important
Use `{env.VAR}` for a secret. It is resolved while the app starts and never written back: `caddy adapt` and the admin API's `GET /config/` both return the placeholder verbatim. The Caddyfile's other form, `{$VAR}`, is substituted before parsing and bakes the secret into the adapted JSON.
::

### What each directive does

| Directive | Effect |
|---|---|
| `lb_policy first` | Uses the head of the node's upstream order, which is a priority and never a weighting. With `prefer` at its default, `self`, the head is the process on this very machine. |
| `max_fails 1` | **Required with `first`.** `first` picks the first upstream not already marked down, and a failed dial marks one down only after `max_fails` failures, so a higher count sends every retry back to the same dead process and the request fails before the mark is set. |
| `fail_duration 30s` | How long a failed upstream is skipped; it lets a recovered process back in. |
| `lb_retries 2`, `lb_try_duration 5s` | Bound how long one request keeps trying. A dynamic upstream that resolved to nothing resolves to nothing again, so a larger budget only lengthens the wait. |
| `handle_errors` | Turns *no upstream* into a `404` when no fleet serves the host. |
| `scafo_plain_http` | Matches a host whose fleet has `http_redirect off`. A host no fleet owns does not match, so it is redirected like everything else. |

### Request variables

Caddy's answer to *no upstreams* is `503`. The dynamic-upstreams module leaves the status alone and records why in request variables, for a `handle_errors` route to act on:

| `scafo.route` | Meaning |
|---|---|
| `ok` | Some fleet named upstreams for this request; a `503` here means they were all unreachable. |
| `unknown` | No fleet serves this host. |
| `denied` | A fleet API host and a path the ingress must answer itself rather than proxy. |
| `unavailable` | Not one fleet has a routing document yet. A cold start, not a missing deployment. |

`scafo.host` carries the host the lookup was keyed by, `scafo.matched` the routing entry that answered it, and `scafo.fleet` the name of the fleet that answered.

::note
`denied` is answered the same way as `unknown` on purpose: a distinct status would confirm that the path exists. The denial is enforced whatever the error route says.
::

## JSON

The same configuration in Caddy's JSON, trimmed to the HTTPS server. The `scafo` app sits under `apps` beside `http` and `tls`:

```json
{
  "storage": { "module": "scafo" },
  "apps": {
    "scafo": {
      "fleets": [
        {
          "name": "default",
          "endpoint": "http://127.0.0.1:7400",
          "token": "{env.SCAFO_INGRESS_TOKEN}"
        }
      ]
    },
    "tls": {
      "automation": {
        "policies": [{ "on_demand": true }],
        "on_demand": {
          "permission": { "module": "scafo" }
        }
      }
    },
    "http": {
      "servers": {
        "scafo": {
          "listen": [":443"],
          "tls_connection_policies": [{}],
          "routes": [
            {
              "handle": [
                {
                  "handler": "reverse_proxy",
                  "dynamic_upstreams": { "source": "scafo" },
                  "load_balancing": {
                    "selection_policy": { "policy": "first" },
                    "retries": 2,
                    "try_duration": "5s"
                  },
                  "health_checks": {
                    "passive": { "fail_duration": "30s", "max_fails": 1 }
                  }
                }
              ]
            }
          ],
          "errors": {
            "routes": [
              {
                "match": [{ "vars": { "scafo.route": ["unknown", "denied"] } }],
                "handle": [
                  {
                    "handler": "static_response",
                    "status_code": 404,
                    "body": "No deployment answers to this name.\n"
                  }
                ]
              }
            ]
          }
        }
      }
    }
  }
}
```

Every `fleet` option keeps its Caddyfile name in JSON. The plain-HTTP matcher is `{ "scafo_plain_http": {} }` inside a route's `match`.

## Start order

::warning
Start the node before Caddy and stop it after. A Caddy that starts while every node is down has an empty certificate cache and no routing document: TLS fails and every request is `unavailable` until a node answers.
::

A node that does not answer at startup does not keep Caddy from starting: the failure is logged and the poller carries on. Only a configuration error fails startup — a missing, unreadable or empty token aborts with an error naming the option.

A **running** Caddy is not affected by node downtime. Handshakes come from the in-memory certificate cache and routes from the last document that parsed. Only background work fails and is retried: renewals, OCSP staple refreshes, a certificate for a new name, a route that changed since the last successful poll. With several fleets, all of that stays confined to the fleet whose node is down.

| Platform | How |
|---|---|
| Compose | Share the node's network namespace where there is one node, and `depends_on` it. |
| systemd | Order the Caddy unit `After=` the node's, with `Restart=on-failure` and `StartLimitIntervalSec=0`, so the first boot survives the moment before the node has written its token. |
| Kubernetes | Run the node as a sidecar in the Caddy pod with a startup probe. |

A **reload** is graceful: each fleet's poller is carried across it, so the last document keeps answering throughout, even while a node is down.

::warning
When you mount your own Caddyfile, mount its **directory**, not the file. An editor replaces the file's inode, and a file bind mount keeps pointing at the old one, so `caddy reload` would reload a stale configuration.
::

## Access log

Point the access log at a file the node is told to read, and the node counts requests per application:

```yaml
caddy:
  environment:
    SCAFO_ACCESS_LOG: /var/log/scafo-ingress/access.log
  volumes:
    - ingress-log:/var/log/scafo-ingress
```

```sh
scafo sail --observability --ingress-access-log /var/log/scafo-ingress/access.log
```

Both processes need the path: Caddy writes it, the node reads it. In Compose that is a shared volume mounted into both services. The image writes the log as JSON with mode `0644`, so a node running as another user can read it. `--ingress-access-log` is rejected with `invalid-input` when observability is off.

::read-more{to="/docs/observability"}
What the node does with those lines.
::

## Rotating the ingress token

The ordering rule is **the node first, then Caddy**: a reload refills the certificate cache from storage, so Caddy must find a node that already accepts the token it is about to present. Serving is unaffected throughout — a window where Caddy holds a token the node rejects stalls renewals and route updates, not handshakes and requests, and only for that fleet.

::steps

### Expand

Set `SCAFO_INGRESS_TOKEN="new,old"` on the node — new one first — and restart it. Both are accepted.

### Migrate

Give each Caddy the new token and reload it, one at a time, on your own schedule. Reloads are graceful.

### Contract

Set `SCAFO_INGRESS_TOKEN="new"` on the node and restart it. The old token stops working.

::

A single node with a single Caddy can skip the expand step: set the new value on both, restart the node, reload Caddy.

::read-more{to="/reference/caddy-modules"}
Every module, option and environment variable, in full.
::