Caddy modules

Five modules in one repository and one binary. The scafo app is where the fleets are configured, once; the other four take no options and read it. A fleet is one Scafo installation: one node endpoint, one bearer token, one routing document, one bucket. One Caddy can stand in front of one fleet or several, and both are the same configuration and the same code path.

Caddy holds an endpoint URL and a bearer token per fleet, which authorize that node's /v1/storage and /v1/ingress/routes and nothing else. The node holds its bucket credentials and does the reading and writing, so there is no path from a compromised Caddy to the rest of any bucket.

Module path: github.com/sandros94/scafo-ingress. License: MIT.

#Building

xcaddy build --with github.com/sandros94/scafo-ingress            # every module here
xcaddy build --with github.com/sandros94/scafo-ingress/storage    # the storage one and the app
xcaddy build --with github.com/sandros94/scafo-ingress/ingress    # the HTTP ones and the app
xcaddy build --with github.com/sandros94/scafo-ingress/permission # the permission one and the app

The repository root imports every module in it, so naming the root compiles them all in; naming a single package compiles that module and the scafo app it reads, and nothing else.

#Dockerfile build arguments

The repository's Dockerfile does the same inside the official caddy:<version>-builder image and copies the result into the matching runtime image.

docker build -t my-caddy .
docker build -t my-caddy --build-arg CADDY_VERSION=2.10.0 .
docker build -t my-caddy --build-arg DNS_PROVIDERS=github.com/caddy-dns/gandi@v1.0.3 .

One Dockerfile and one xcaddy invocation stand behind both published flavours, so a custom build is the published build with a different argument.

#Images, tags and binaries

Published for linux/amd64 and linux/arm64, in two flavours. Lean is Caddy with only the modules above. -dns is the same build with a curated set of DNS-01 provider plugins compiled in, for the managed wildcard certificates — a Caddy plugin can only arrive at compile time.

Each binary ships a .sha256 alongside it and is the same binary that is inside the image of its flavour, lifted out of that build's builder stage. Every release also carries one caddy-version file naming the Caddy both flavours embed.

#DNS providers in the -dns flavour

Exact pins, from dns-providers.txt, which is the whole list.

The last path element, minus any /vN suffix, is the Caddy module name. The release build asserts that every entry registered under that name, so a provider that breaks the convention fails the build rather than going missing quietly. The DigitalOcean pin is a Go pseudo-version because that repository has never cut a tag.

A provider that is not in the list is added the same way the curated ones arrive — --with github.com/caddy-dns/gandi@v1.0.3 to xcaddy, or the same string in DNS_PROVIDERS.

#The scafo app

The app is configured in one of two ways, and never both at once:

• From the environment. The published image ships a default Caddyfile that declares an empty scafo block, and the app fills itself in from the process environment. No Caddyfile to write, no volume to bind. See every environment variable.
• In the configuration, as below. When the configuration declares at least one fleet, the environment is not consulted for fleets at all. The same rule applies to DNS providers on their own. The two can never be half-merged.

scafo is a global option, so it goes in the leading braces block:

{
	scafo {
		fleet client-a {
			endpoint http://10.0.0.1:7400
			token    {env.TOKEN_A}
		}
		fleet client-b {
			endpoint      http://10.0.0.2:7400
			token_file    /run/secrets/client-b
			http_redirect off
		}

		dns cloudflare {
			api_token {env.CLOUDFLARE_API_TOKEN}
		}

		acme_delegation acme.example.com
		acme_email      ops@example.com
	}
}

In JSON every option keeps its Caddyfile name, and a fleet's name is its name member.

#scafo options

A value written in the configuration wins over the environment for acme_delegation, acme_email and acme_ca; the environment fills in only what the configuration left empty.

#fleet options

#DNS providers

The dns block mirrors the environment form exactly, and every field value is a string:

dns cloudflare {
	api_token {env.CLOUDFLARE_API_TOKEN}
}

dns other cloudflare {
	zones     a.example b.example
	api_token {env.OTHER_CLOUDFLARE_API_TOKEN}
}

A wildcard base is solved by the named provider whose zones entry is its longest suffix, matched at a label boundary; otherwise by the default provider, the one with no zones of its own; otherwise, when zones are declared somewhere and acme_delegation is set, by the delegation. When no zone is declared anywhere, the default provider takes everything.

#Placeholders

Every value accepts Caddy's global placeholders, most usefully {env.VAR}. The app is provisioned before it serves anything, so request-scoped placeholders are not available — only environment and system values.

The endpoint is expected to be loopback or an equally private network path. Do not expose a node's API publicly on the strength of the token alone; a node reached over a shared network carries the token over that network, so keep it private or use https.

#caddy.storage.scafo

storage is a global option. It takes no options: the fleets are the app's.

{
	scafo {
		fleet default {
			endpoint http://127.0.0.1:7400
			token    {env.SCAFO_INGRESS_TOKEN}
		}
	}
	storage scafo
}

An option inside storage scafo is refused at startup with an error that says where fleets are configured instead.

#Where each key goes

One Caddy has one certificate keyspace, and every key in it that names a domain is routed to the fleet that owns that domain — through that fleet's node, with that fleet's token. A certificate never crosses a fleet. Ownership follows the same resolution routing uses, plus one rule for a *.base certificate: it belongs to the fleet whose document carries *.base as a route, answers on base as its platform apex, or asks for base in its wildcards.

Keys that name no domain have no fleet, so they live on the ingress's own disk, in Caddy's default file storage under its data directory. The ACME account is the important one: it is the ingress's, not any fleet's, and putting it in one fleet's bucket would let that fleet impersonate the ingress to its certificate authority. Keep the data directory (/data in the image) on a volume.

Two consequences:

• CertMagic's storage check writes and reads back an rw_test_<random> key before an issuance. It names no domain, so it checks the local disk rather than a fleet's bucket; a fleet's bucket is exercised by the certificate writes themselves.
• The renewal lock CertMagic takes while fetching ARI names no domain, so it is taken locally: a lock between the goroutines of this ingress rather than between several of them.

A domain nobody owns gets no default fleet. Falling back to "the first one" would put one customer's private key in another customer's bucket. A read of such a key is reported as missing, and a write is refused with an error naming the domain.

A listing of a prefix that names a domain is sent to that fleet alone. A listing whose children name domains (certificates/<issuer>, ocsp) is sent to every fleet and merged, keeping a key only if it came back from the fleet that owns its domain — so a stale key left in one fleet's bucket for a name another fleet owns is never loaded, renewed or deleted through the wrong node.

Each fleet's storage requests use a dedicated HTTP client bounded by that fleet's storage_timeout. There are no retries other than the lock poll.

#http.reverse_proxy.upstreams.scafo

It goes inside a reverse_proxy block as dynamic scafo, and takes no options.

reverse_proxy {
	dynamic scafo
	lb_policy first
}

An option inside dynamic scafo is refused at startup with an error that says where fleets are configured instead. prefer, poll_interval and the timeouts are per fleet, so two site blocks cannot disagree about a fleet.

#Resolving a host

Every fleet has its own poller and its own cached routing document; a request is resolved against all of them at once, so it costs a map lookup and no round trip. The request's host, lowercased and without its port, is tried in this order, and the first match answers:

An exact route beats a wildcard across fleets, just as it does within one. When two fleets publish the same host, the one that published it first keeps it for as long as it goes on publishing it, and the conflict is logged once per change rather than once per poll. Two fleets claiming a host in the very same refresh have no "first", so the order they were configured in decides: declaration order in a Caddyfile, alphabetical by name when they came from the environment.

A fleet whose node is unreachable affects only its own hosts: its last document keeps answering.

#Request variables

The module cannot choose a status code — Caddy's answer to no upstreams is 503 — so it records why in four request variables and leaves the answer to a handle_errors route.

scafo.host is always the request's own host; scafo.matched tells a wildcard's traffic apart from a route named for one application, and scafo.fleet tells one installation's traffic from another's.

These are request variables. A vars matcher branches on them, and any handler that expands request placeholders — respond, header, rewrite — can read {http.vars.scafo.fleet}. Caddy's access-log encoder does not run in a request context, so a fleet name reaches the log only by way of something that does, such as a response header the log already records.

Turning the first two into a 404 takes built-in directives only:

handle_errors {
	@missing vars scafo.route unknown denied
	handle @missing {
		respond "No deployment answers to {http.vars.scafo.host}." 404
	}
	respond "{err.status_code} {err.status_text}" {err.status_code}
}

denied is answered the same way as unknown on purpose: a distinct status would confirm that the path exists.

#The pollers

One poller per fleet, not one per site block: however many handlers use dynamic scafo, a node is asked once per poll_interval. A fetch whose bytes are identical to the last one's is dropped rather than republished, so the cross-fleet view is rebuilt, and a conflict logged, only when something actually changed. A document whose v is not 1 is refused whole, and the last one understood keeps answering.

Each fetch is bounded by the fleet's timeout, and the interval is waited between attempts rather than on a schedule, so a slow answer delays the next request instead of stacking up behind it.

#Load balancing

Each node returns a route's upstreams in priority order, never as a weighting. lb_policy first takes that at its word: with prefer self, the head of the list is the process on this very node, so a request that arrives here stays here whenever it can.

reverse_proxy {
	dynamic scafo

	lb_policy       first
	lb_retries      2
	lb_try_duration 5s

	fail_duration 30s
	max_fails     1
}

#tls.permission.scafo

The on-demand TLS permission. It goes in the on_demand_tls global option and takes no options.

{
	on_demand_tls {
		permission scafo
	}
}

A name is approved only when some fleet's current routing document names it as an exact route host or as a fleet API host. The answer comes from the documents already in memory, so a handshake costs a map lookup and no HTTP call — which matters, because this runs on the handshake path of every name the ingress has no certificate for, and a stranger decides how often that is.

• A name covered only by a *. entry is refused. An on-demand certificate per arbitrary subdomain is a door anyone can knock on: point a thousand names at the ingress and each handshake asks the CA for another certificate under a customer's domain, until that customer's weekly issuance limit is gone. The wildcard those names are covered by is obtained once, through DNS-01, and covers all of them.
• A name in no document is refused, and so is every name while no document has arrived yet. "I do not know" is not "yes", and a cold start must not become an issuance for whatever connects first.

#http.matchers.scafo_plain_http

A request matcher with no options. It matches a host whose fleet is configured with http_redirect off; a host no fleet owns does not match. Caddy decides HTTP-to-HTTPS redirects per site block from the host names written in the configuration, and this ingress writes none down — it serves whatever the routing documents say, and that changes without a reload — so the decision is made per request instead:

http:// {
	@plain scafo_plain_http
	handle @plain {
		reverse_proxy {
			dynamic scafo
		}
	}
	handle {
		redir https://{host}{uri} 308
	}
}

The default is the redirect, and only a fleet that asked for plain HTTP gets it. The ACME HTTP challenge is unaffected either way: Caddy answers it in the server itself, before any route or matcher runs.

#Every environment variable

Read by the scafo app when the configuration declares no fleet, which is how the published image is meant to run. <NAME> is [A-Z0-9_]+, and becomes a fleet or provider name lowercased with underscores turned into hyphens: SCAFO_FLEET_CLIENT_A_ENDPOINT is fleet client-a.

#Fleets

Durations are Caddy durations (5s, 1m30s) and must be positive; 0s is an error rather than "use the default".

#DNS providers

ZONES is reserved and cannot be a provider field name. A SCAFO_DNS_* variable with no provider declared anywhere is an error naming every one of them. Without SCAFO_DNS_<NAME>_PROVIDER, SCAFO_DNS_BN_API_KEY could not be told apart from a default-provider field called bn_api_key — declaring the name is what carves out the namespace.

#ACME and the image

#The image's default Caddyfile

What the published image runs at /etc/caddy/Caddyfile, with its comments shortened. Mounting a Caddyfile of your own over it replaces it entirely; the module repository's examples/Caddyfile is the same file with the scafo block written out, and the place to start.

{
	# Every fleet, DNS provider and ACME setting comes from the environment.
	scafo {
	}

	# Certificates live in the bucket of the fleet that owns each name. The
	# ACME account and Caddy's own housekeeping stay under the data directory.
	storage scafo

	# Only a name some fleet's routing document names exactly gets a
	# certificate on demand.
	on_demand_tls {
		permission scafo
	}
}

(scafo_log) {
	log {
		output file {$SCAFO_ACCESS_LOG:/dev/stdout} {
			mode 0644
		}
		format json
	}
}

(scafo_proxy) {
	reverse_proxy {
		dynamic scafo

		lb_policy first
		lb_retries 2
		lb_try_duration 5s

		fail_duration 30s
		max_fails 1
	}
}

(scafo_errors) {
	handle_errors {
		@missing vars scafo.route unknown denied
		handle @missing {
			respond <<HTML
				<!DOCTYPE html>
				<html lang="en">
				<head><meta charset="utf-8"><title>Not found</title></head>
				<body style="font: 16px/1.6 system-ui, sans-serif; max-width: 32rem; margin: 12vh auto; padding: 0 1.5rem">
				<h1 style="font-size: 1.25rem">Nothing is served here</h1>
				<p>No deployment answers to <code>{http.vars.scafo.host}</code>.</p>
				<p style="color: #666">scafo</p>
				</body>
				</html>
				HTML 404
		}
		respond "{err.status_code} {err.status_text}" {err.status_code}
	}
}

http:// {
	import scafo_log

	@plain scafo_plain_http
	handle @plain {
		import scafo_proxy
	}
	handle {
		redir https://{host}{uri} 308
	}

	import scafo_errors
}

https:// {
	import scafo_log

	tls {
		on_demand
	}

	import scafo_proxy
	import scafo_errors
}

#Operational notes

A node must be reachable when Caddy starts for its fleet to start already routing, but its absence does not keep Caddy from starting. A running Caddy is not affected by node downtime: TLS handshakes are served from the in-memory certificate cache, requests are routed from the last document that parsed, and only background work fails and is retried later — renewals, OCSP staple refreshes, a certificate for a new name, a route that has changed since the last successful poll — and only for that node's fleet.

A Caddy that starts while every node is down has an empty certificate cache and no routing document, so TLS fails and every request is unavailable until one answers:

• Start the nodes before Caddy and stop them after, so Caddy never outlives them.
• Compose: share a node's network namespace where there is one node, and depends_on it.
• systemd: order the Caddy unit After= the node's, with Restart=on-failure and StartLimitIntervalSec=0.
• Kubernetes: run a node as a sidecar in the Caddy pod with a startup probe.

Several Caddy instances may share one fleet's bucket; coordination relies on that node's lock endpoints, and expiring locks left behind by a crashed holder is the node's job. The ARI renewal lock is local to each ingress rather than shared.

#Rotating the token

The ordering rule: the node first, then Caddy. A Caddy reload refills the certificate cache from storage, so it must find a node that already accepts the token it is about to present. Serving is unaffected throughout, and only that fleet's renewals and route updates stall while Caddy holds a token the node rejects.

A single node and a single Caddy can skip the expand step: set the new value on both, restart the node, reload Caddy.