Caddy with the scafo modules

#What it is

Upstream Caddy cannot route for Scafo. scafo-caddy is Caddy with five modules compiled in, and all of them read one place: the scafo app, which says where each fleet's node is and which token opens it.

A fleet here is one Scafo installation: one node endpoint, one ingress token, one routing document, one bucket. Caddy holds the endpoint and the token and nothing else. The node holds the bucket credentials and does the reading and writing, so there is no path from a compromised Caddy to the rest of the bucket, and rotating the token never involves the storage provider.

The token is the node's ingress token: SCAFO_INGRESS_TOKEN on the node, or the one the node mints into its state directory (/var/lib/scafo/ingress-token) when that is unset. It opens /v1/ingress/routes and /v1/storage on that node and nothing else.

#One fleet, from the environment

The published image needs no Caddyfile and no configuration file. It ships a default Caddyfile that declares an empty scafo block, and the app fills itself in from the environment. This is the intended way to run it.

docker run -p 80:80 -p 443:443 -p 443:443/udp \
  -e SCAFO_FLEET_ENDPOINT=http://127.0.0.1:7400 \
  -e SCAFO_FLEET_TOKEN=... \
  ghcr.io/sandros94/scafo-caddy:0.2.1

That is the whole configuration. The fleet is named default. Its routing document is polled every five seconds, certificates go into its bucket, and every name it routes exactly gets a certificate on demand.

With SCAFO_FLEET_TOKEN_FILE instead of SCAFO_FLEET_TOKEN, the token is read from a file once at startup. Give exactly one of the two.

#Beside a node in Compose

The usual shape is Caddy sharing the node's network namespace, which is what makes 127.0.0.1:7400 a real loopback address:

services:
  scafo-node:
    image: ghcr.io/sandros94/scafo:latest
    environment:
      SCAFO_INGRESS_TOKEN: ${INGRESS_TOKEN:?set INGRESS_TOKEN in .env}
      # ...the node's own bucket credentials and settings
    ports:
      - "80:80"
      - "443:443"
      - "443:443/udp"
    volumes:
      - state:/var/lib/scafo

  caddy:
    image: ghcr.io/sandros94/scafo-caddy:0.2.1
    network_mode: "service:scafo-node"
    depends_on:
      - scafo-node
    environment:
      SCAFO_FLEET_ENDPOINT: http://127.0.0.1:7400
      SCAFO_FLEET_TOKEN: ${INGRESS_TOKEN:?set INGRESS_TOKEN in .env}
    volumes:
      - caddy-data:/data

volumes:
  state:
  caddy-data:

Keep /data. Certificates live in the bucket, but the ACME account and Caddy's own housekeeping live there, and losing it means registering with the certificate authority again.

#What the default configuration does

#Settings you may want

Every variable is listed in the module reference.

#Several fleets behind one ingress

One Caddy can stand in front of several fleets at once. The typical case is an agency that runs one fleet per client: one ingress per machine, in front of one node per client, each client's installation with its own bucket and its own token.

Each fleet gets a name inside the variable: SCAFO_FLEET_<NAME>_ENDPOINT, SCAFO_FLEET_<NAME>_TOKEN, and so on. <NAME> is uppercase letters, digits and underscores, and becomes the fleet's name lowercased with underscores turned into hyphens: SCAFO_FLEET_CLIENT_A_ENDPOINT is fleet client-a.

docker run -p 80:80 -p 443:443 -p 443:443/udp \
  -e SCAFO_FLEET_CLIENT_A_ENDPOINT=http://10.0.0.1:7400 \
  -e SCAFO_FLEET_CLIENT_A_TOKEN=... \
  -e SCAFO_FLEET_CLIENT_B_ENDPOINT=http://10.0.0.2:7400 \
  -e SCAFO_FLEET_CLIENT_B_TOKEN=... \
  -e SCAFO_FLEET_CLIENT_B_HTTP_REDIRECT=off \
  ghcr.io/sandros94/scafo-caddy:0.2.1

Every per-fleet setting takes the same form, so each fleet has its own poll interval, its own prefer and its own redirect choice. Above, client-b's hosts are also served over plain HTTP, and client-a's are still redirected.

Nothing is guessed either. A SCAFO_FLEET_* variable this build does not recognize fails startup, naming it and listing the ones it does: a misspelled variable that was quietly ignored would be a fleet that quietly does not exist. A fleet with an endpoint and no token fails too.

#How a request finds its fleet

Every fleet has its own poller and its own cached routing document, and a request is resolved against all of them at once. The host, lowercased and without its port, is tried in this order, and the first match answers:

So an exact route beats a wildcard even across fleets. When two fleets publish the same host, the one that published it first keeps it for as long as it goes on publishing it, and the conflict is logged once per change. The host never flaps between the two; the conflict is something to fix.

A fleet whose node is unreachable affects only its own hosts. Its last document keeps answering, and the other fleets are untouched.

#Certificates never cross fleets

Every certificate, private key and issuance lock is stored with the fleet that owns its name, written through that fleet's node with that fleet's token. One client's private key never lands in another client's bucket, and a name no fleet owns is written nowhere at all: it gets no default fleet.

What names no domain belongs to no fleet, so it stays on the ingress's own disk, under Caddy's data directory. That includes the ACME account: it is the ingress's, not any fleet's, and putting it in one fleet's bucket would let that fleet impersonate the ingress to its certificate authority. Keep /data on a volume.

#Reaching nodes over a network

With one node per client, each node's API is usually reached over a shared private network rather than loopback:

services:
  caddy:
    image: ghcr.io/sandros94/scafo-caddy:0.2.1-dns
    ports:
      - "80:80"
      - "443:443"
      - "443:443/udp"
    environment:
      SCAFO_FLEET_CLIENT_A_ENDPOINT: http://node-a:7400
      SCAFO_FLEET_CLIENT_A_TOKEN: ${TOKEN_A:?}
      SCAFO_FLEET_CLIENT_B_ENDPOINT: http://node-b:7400
      SCAFO_FLEET_CLIENT_B_TOKEN: ${TOKEN_B:?}
      SCAFO_DNS_PROVIDER: cloudflare
      SCAFO_DNS_API_TOKEN: ${CLOUDFLARE_API_TOKEN:?}
      SCAFO_DNS_ZONES: apps.example.com
      SCAFO_ACME_DELEGATION: acme.example.com
    volumes:
      - caddy-data:/data

volumes:
  caddy-data:

The token then travels over that network: keep it private, or use https endpoints.

#Wildcard certificates

A fleet's routing document lists the bases it wants a *.<base> certificate for: its platform domains and every custom domain claimed with its subdomains. Give the ingress a DNS provider and it obtains those wildcards itself, over the ACME DNS-01 challenge:

  -e SCAFO_DNS_PROVIDER=cloudflare \
  -e SCAFO_DNS_API_TOKEN=... \
  -e SCAFO_DNS_ZONES=apps.example.com \
  -e SCAFO_ACME_DELEGATION=acme.example.com \
  -e SCAFO_ACME_EMAIL=ops@example.com

SCAFO_DNS_<FIELD>, lowercased, is one of the provider module's own settings: SCAFO_DNS_API_TOKEN is api_token. This needs the -dns image flavour, the only one with DNS provider modules compiled in. With no DNS provider configured, nothing is managed, and only the names a fleet routes exactly are covered, on demand, one certificate each.

A name covered only by a wildcard route is never issued on demand. Anyone can point a name at the ingress, and a certificate per arbitrary subdomain would let a stranger exhaust a customer's weekly issuance limit at the certificate authority. Those names are served over HTTPS through the managed wildcard only.

#The Caddyfile form

When the default cannot do what you need — another site, your own error page, headers, rate limiting — mount a Caddyfile of your own over /etc/caddy/Caddyfile. It replaces the default entirely. The module repository's examples/Caddyfile is the default written out in full, and the place to start.

Fleets are declared in the scafo global option, once. No other directive takes an endpoint or a token:

{
	scafo {
		fleet default {
			endpoint http://127.0.0.1:7400
			token    {env.SCAFO_INGRESS_TOKEN}
		}
	}

	# Certificates live in the bucket of the fleet that owns the name.
	storage scafo

	# A name gets a certificate on demand only when a fleet routes it exactly.
	on_demand_tls {
		permission scafo
	}
}

(scafo_proxy) {
	reverse_proxy {
		dynamic scafo

		lb_policy       first
		lb_retries      2
		lb_try_duration 5s

		fail_duration 30s
		max_fails     1
	}
}

(scafo_errors) {
	handle_errors {
		@missing vars scafo.route unknown denied
		handle @missing {
			respond "No deployment answers to {http.vars.scafo.host}." 404
		}
		respond "{err.status_code} {err.status_text}" {err.status_code}
	}
}

# Plain HTTP: served as-is for a fleet with `http_redirect off`, redirected
# to HTTPS for everything else.
http:// {
	@plain scafo_plain_http
	handle @plain {
		import scafo_proxy
	}
	handle {
		redir https://{host}{uri} 308
	}
	import scafo_errors
}

# Everything over HTTPS, whichever fleet serves it.
https:// {
	tls {
		on_demand
	}
	import scafo_proxy
	import scafo_errors
}

Several fleets are several fleet blocks, each named with a slug:

scafo {
	fleet client-a {
		endpoint http://10.0.0.1:7400
		token    {env.TOKEN_A}
	}
	fleet client-b {
		endpoint      http://10.0.0.2:7400
		token_file    /run/secrets/client-b
		http_redirect off
	}
}

storage scafo, dynamic scafo and permission scafo take no options. An endpoint or a token inside one of them is refused at startup, with an error saying that fleets are configured in the scafo global option.

#What each directive does

#Request variables

Caddy's answer to no upstreams is 503. The dynamic-upstreams module leaves the status alone and records why in request variables, for a handle_errors route to act on:

scafo.host carries the host the lookup was keyed by, scafo.matched the routing entry that answered it, and scafo.fleet the name of the fleet that answered.

#JSON

The same configuration in Caddy's JSON, trimmed to the HTTPS server. The scafo app sits under apps beside http and tls:

{
  "storage": { "module": "scafo" },
  "apps": {
    "scafo": {
      "fleets": [
        {
          "name": "default",
          "endpoint": "http://127.0.0.1:7400",
          "token": "{env.SCAFO_INGRESS_TOKEN}"
        }
      ]
    },
    "tls": {
      "automation": {
        "policies": [{ "on_demand": true }],
        "on_demand": {
          "permission": { "module": "scafo" }
        }
      }
    },
    "http": {
      "servers": {
        "scafo": {
          "listen": [":443"],
          "tls_connection_policies": [{}],
          "routes": [
            {
              "handle": [
                {
                  "handler": "reverse_proxy",
                  "dynamic_upstreams": { "source": "scafo" },
                  "load_balancing": {
                    "selection_policy": { "policy": "first" },
                    "retries": 2,
                    "try_duration": "5s"
                  },
                  "health_checks": {
                    "passive": { "fail_duration": "30s", "max_fails": 1 }
                  }
                }
              ]
            }
          ],
          "errors": {
            "routes": [
              {
                "match": [{ "vars": { "scafo.route": ["unknown", "denied"] } }],
                "handle": [
                  {
                    "handler": "static_response",
                    "status_code": 404,
                    "body": "No deployment answers to this name.\n"
                  }
                ]
              }
            ]
          }
        }
      }
    }
  }
}

Every fleet option keeps its Caddyfile name in JSON. The plain-HTTP matcher is { "scafo_plain_http": {} } inside a route's match.

#Start order

A node that does not answer at startup does not keep Caddy from starting: the failure is logged and the poller carries on. Only a configuration error fails startup — a missing, unreadable or empty token aborts with an error naming the option.

A running Caddy is not affected by node downtime. Handshakes come from the in-memory certificate cache and routes from the last document that parsed. Only background work fails and is retried: renewals, OCSP staple refreshes, a certificate for a new name, a route that changed since the last successful poll. With several fleets, all of that stays confined to the fleet whose node is down.

A reload is graceful: each fleet's poller is carried across it, so the last document keeps answering throughout, even while a node is down.

#Access log

Point the access log at a file the node is told to read, and the node counts requests per application:

caddy:
  environment:
    SCAFO_ACCESS_LOG: /var/log/scafo-ingress/access.log
  volumes:
    - ingress-log:/var/log/scafo-ingress

scafo sail --observability --ingress-access-log /var/log/scafo-ingress/access.log

Both processes need the path: Caddy writes it, the node reads it. In Compose that is a shared volume mounted into both services. The image writes the log as JSON with mode 0644, so a node running as another user can read it. --ingress-access-log is rejected with invalid-input when observability is off.

#Rotating the ingress token

The ordering rule is the node first, then Caddy: a reload refills the certificate cache from storage, so Caddy must find a node that already accepts the token it is about to present. Serving is unaffected throughout — a window where Caddy holds a token the node rejects stalls renewals and route updates, not handshakes and requests, and only for that fleet.

A single node with a single Caddy can skip the expand step: set the new value on both, restart the node, reload Caddy.