VPS mesh

No orchestrator joins the hosts: the bucket and prefix are the fleet, and each node's tunnel record in the registry is what the next node builds its interface from. roll.sh drives the whole thing over ssh.

With SCAFO_TRANSPORT=wireguard the node's API, its advertised address and the address runtime processes bind all default to its tunnel address. The node API is therefore private without being loopback-only: the colocated Caddy reaches it over the tunnel interface, a peer's Caddy reaches that host's application processes the same way, and nothing on the public interfaces answers on port 7400.

Both Caddys share one certificate store in the bucket, through their own node's /v1/storage, and read the fleet's routing from /v1/ingress/routes with prefer self. Either can serve any host: with apex and wildcard DNS pointing at both machines, the one a request lands on serves it locally when it can and proxies to the holder over the tunnel when it cannot.

#Prerequisites

• Two or more Ubuntu hosts with docker and wireguard-tools installed, and an ssh account with passwordless sudo on each.
• An S3-compatible bucket honouring If-Match / If-None-Match conditional writes and Range reads, reachable from every host. It is not created here.
• A DNS zone whose provider is one of the ones the -dns image carries (bunny, cloudflare, digitalocean, hetzner, ovh, route53) and an API credential for it. Any other provider needs its own build.
• Apex and wildcard A records, apps.example.com and *.apps.example.com, pointing at every host.
• Firewall: TCP 80 and 443 plus UDP 443 open to the world, UDP 51820 open between the hosts. Port 7400 stays on the tunnel and needs no rule.

#Configure

Three things are yours to fill in: .env, Caddyfile.template, and nothing else. The ingress token is minted per host by roll.sh.

#.env

Beside roll.sh by default, or anywhere named through ENV_FILE. It holds the bucket credentials, the DNS credential and the fleet's shape:

# `id:address:tunnel-address[:labels]` per host; address is what peers and ssh reach.
HOSTS="vps-a:203.0.113.10:10.83.0.1/24 vps-b:203.0.113.11:10.83.0.2/24:provider=ovh"
SSH_USER=ubuntu

# Platform apex, such as apps.example.com.
DOMAIN=

# Exact released tags; this recipe needs the `-dns` flavour of scafo-caddy.
IMAGE=ghcr.io/sandros94/scafo:
CADDY_IMAGE=ghcr.io/sandros94/scafo-caddy:

# The bucket. It must support conditional writes, and nothing creates it for you.
SCAFO_S3_ENDPOINT=
SCAFO_S3_BUCKET=
SCAFO_S3_ACCESS_KEY=
SCAFO_S3_SECRET_KEY=
SCAFO_S3_REGION=us-east-1
SCAFO_S3_PREFIX=scafo

# Variables copied into each host's ingress.env for the Caddyfile's `dns` line.
DNS_ENV="BUNNY_API_KEY"
BUNNY_API_KEY=

Set every blank value and complete the two image tags; roll.sh refuses to start with IMAGE or CADDY_IMAGE empty.

#Caddyfile.template

The ACME contact address and the dns line naming the zone's provider and the variable above are yours to edit. @DOMAIN@, @NODE@ and @LOGS@ are filled in per host during the roll; leave them. The scafo block declares this host's node as the one fleet the Caddy serves, and every other scafo directive reads it.

{
	# ACME contact address. @DOMAIN@, @NODE@ and @LOGS@ are filled in per host by
	# roll-node.sh; this line and the `dns` one below are yours to edit.
	email you@example.com
	# The fleet, declared once: every scafo module below reads it from here.
	# `{env.…}` is resolved at provisioning, so the token never lands in the
	# adapted configuration.
	scafo {
		fleet default {
			endpoint http://@NODE@
			token {env.SCAFO_INGRESS_TOKEN}
			prefer self
		}
	}
	# Certificates and locks go to the fleet's bucket through the node. The ACME
	# account is the ingress's own and stays under /data.
	storage scafo
	# A customer domain is issued for only once the fleet's routing names it.
	on_demand_tls {
		permission scafo
	}
}

(scafo) {
	log {
		# Readable by the node, which runs as another user and counts requests from
		# it: Caddy creates its log files 0600 otherwise.
		output file @LOGS@/access.log {
			mode 0644
		}
		format json
	}
	reverse_proxy {
		dynamic scafo
		# The node returns upstreams in priority order, this node's own first, so
		# `first` keeps a request on the machine it arrived at. `max_fails 1` is
		# what lets a retry move on: `first` skips only upstreams already marked
		# down, and a higher count would send every retry back to the dead one.
		lb_policy first
		lb_retries 2
		fail_duration 30s
		max_fails 1
	}
	handle_errors {
		@missing vars scafo.route unknown denied
		handle @missing {
			respond "scafo: no deployment answers to {http.vars.scafo.host}" 404
		}
		respond "{err.status_code} {err.status_text}" {err.status_code}
	}
}

# The platform domain and every deployment under it, on one DNS-01 wildcard.
# The `-dns` image carries bunny, cloudflare, digitalocean, hetzner, ovh and
# route53; name the zone's provider and the variable ingress.env holds its
# credential in, which is `DNS_ENV` in the roll script's env file.
*.@DOMAIN@, @DOMAIN@ {
	tls {
		dns bunny {env.BUNNY_API_KEY}
	}
	import scafo
}

# Customer domains pointed at this fleet, issued per name once the fleet's
# routing document names them.
https:// {
	tls {
		on_demand
	}
	import scafo
}

The platform domain and everything under it come from that one DNS-01 wildcard, which is what the -dns image flavour is for. Custom domains an application claims are issued per name through on-demand TLS, and only once the fleet's routing names them exactly; Caddy answers that from the routing it already holds. The ACME account stays in each host's scafo-caddy-data volume, not in the bucket.

#Roll

./roll.sh

#What roll.sh does, per host

#What roll-node.sh does, on the host

#The peer refresh

After every selected host has been rolled, roll.sh runs on every host in HOSTS:

sudo docker run --rm --network host --env-file /etc/scafo/node.env \
  -v /var/lib/scafo:/var/lib/scafo $IMAGE wg peers --stdout \
  | sudo tee /etc/wireguard/wg0.conf >/dev/null \
  && sudo bash -c 'wg syncconf wg0 <(wg-quick strip wg0)' && sudo wg show wg0 peers

scafo wg peers re-renders the interface from the nodes registered right now, which is how a host rolled first learns about a host rolled after it. Every node publishes its public key and endpoint in the registry, so this is a read of what is registered, not an exchange between the hosts.

#Verify

curl -sS https://apps.example.com/.well-known/scafo/health
curl -sS -o /dev/null -w '%{http_code}\n' https://nobody.apps.example.com/   # 404
ssh ubuntu@203.0.113.10 sudo wg show wg0

The health document comes from whichever host DNS resolved to; ask each address in turn with --resolve apps.example.com:443:203.0.113.11 to see both. An unclaimed host is a 404, not a wildcard fallback: the routing document names only hosts backed by a running process. wg show lists the other hosts as peers, each with a recent handshake.

#Deploy

Follow first deploy from your own machine, never from a host on the mesh. The bucket values it asks for are the ones in the .env beside roll.sh, prefix included; the endpoint is https:// plus DOMAIN, which either host's Caddy routes to its own node API.

scafo remote add prod --endpoint https://apps.example.com --register
scafo deploy ./app --name hello --replicas all
curl -sS https://hello.apps.example.com/

scafo fleet nodes lists every host in HOSTS by the id it was given. Without the apex, the API answers only on each host's tunnel address: see without the apex.

#Adding a host

Add its entry to HOSTS, with a free tunnel address in the same /24, point its A records at the machine, and roll that host alone:

./roll.sh vps-c

Named hosts are the only ones rolled, but the peer refresh at the end runs everywhere, which is what tells the existing hosts about the new one. Placement moves applications onto it by rendezvous hashing as it joins; --replicas all applications start there on the next reconcile.

#Re-rolling

./roll.sh with no arguments rolls every host in HOSTS, in the order they are listed, one after another. Naming hosts rolls those alone. A typo in an argument is rejected before anything is rolled. Because a roll is destructive per host, re-roll one host at a time when the fleet is serving: the other hosts keep answering for its applications as placement moves them.

#Caveats

• Caddy starts after the node is healthy, never beside it. The wait loop in roll-node.sh is why the script is sequential.
• The Caddyfile's directory is mounted, not the file. An editor replaces the file's inode, and a file bind mount keeps pointing at the old one, so caddy reload would reload a stale configuration.
• lb_policy first needs max_fails 1. first picks the first upstream not already marked down, and a dial failure marks one down only after max_fails failures, so with a higher count every retry returns to the same dead process and the request fails before the mark is ever set.
• A node's ingress token is that node's. roll.sh mints a fresh one per host per roll, and rolling one host does not disturb another's.
• One bucket and prefix is one fleet. Bucket credentials are fleet-root authority and live in /etc/scafo/node.env, mode 0700 and root-owned, never in Caddy's environment.
• Allow 50 seconds of stop grace. --stop-timeout 50 covers node drain plus the runtime's shutdown bound; raise it past CELLD_SHUTDOWN_TOTAL_MS + 10s if that is increased.