Known traps¶

Environment-specific behaviour that has cost, or would cost, real time. Each one is handled in the Ansible roles. This page exists so you know why the role looks the way it does, and so a symptom is diagnosable at 11pm.

Symptom to cause¶

Symptom	Cause	Section
Hardened the host, port 52843 never comes up	Ubuntu 24.04 socket-activates SSH, so `Port` in sshd_config is ignored	1
fail2ban running but never bans anything	No `/var/log/auth.log` on 24.04; sshd logs to the journal	2
Disk full, every tenant down at once	Docker's default log driver grows without limit	3
Tenant provisioning fails: "no available address range"	Docker's default network pool runs out at ~31 networks	4
Wildcard certificate never issues	Official Caddy image has no Cloudflare DNS module	5
Certificate requests suddenly all fail for a week	Let's Encrypt production rate limit hit while iterating	6
Client data gone after a disk cleanup	`docker volume prune` deleted the tenant databases	7
Host fully patched, containers months out of date	`unattended-upgrades` does not touch container images	8
Vanity domain returns a TLS error	CNAME resolved before Caddy knew the hostname	9
Locked out of the server	Phase 3 run before verifying Tailscale	10
Host unreachable months after setup, nothing changed	Tailscale node key expired	11

1. Ubuntu 24.04 ignores your SSH port¶

Ubuntu 24.04 socket-activates SSH. While ssh.socket is enabled, sshd ignores the Port directive entirely, because the socket unit decides the port. You harden the box, reload, and the new port never appears.

Handled by: the hardening role detects ssh.socket, disables it, and enables ssh.service.

If you hit it manually: systemctl disable --now ssh.socket && systemctl enable --now ssh.service

2. fail2ban silently bans nothing¶

Ubuntu 24.04 has no /var/log/auth.log. sshd logs to the journal. A fail2ban jail with the default file backend watches a file that does not exist. It runs, reports healthy, and bans nobody.

Handled by: backend = systemd in jail.local.

Verify: fail2ban-client status sshd should show a non-zero failure count on a box that has been reachable for a day.

3. Container logs fill the disk¶

Docker's default json-file driver has no size limit. One chatty container writes until the disk is full, and a full disk takes down every tenant on the box simultaneously. This is the most common self-inflicted outage on a Docker host.

Handled by: log-opts in daemon.json, at 10 MB by 3 files per container.

Check: docker system df and df -h in the weekly glance.

4. Docker runs out of networks¶

Docker's default address pool provides roughly 31 networks. Every tenant gets one (ADR-0023), so this is a ceiling you would actually reach. The failure surfaces during tenant provisioning as a confusing address-range error, long after the decision that caused it.

Handled by: default-address-pools in daemon.json, using 10.201.0.0/16 in /24s, giving 256.

Changing this later is disruptive

Address pools are read at daemon start and existing networks keep their subnets. Widening the pool after networks exist means recreating them, which means downtime for every tenant. It is set correctly at build time for exactly this reason.

5. Caddy cannot do DNS challenges out of the box¶

Caddy modules are compiled in, not loaded at runtime. The official caddy image has no Cloudflare DNS provider, and the DNS challenge is the only way to obtain a wildcard certificate. Without it, *.app.octopodia.com.au never issues, and the failure looks like a token problem.

Handled by: the caddy role builds a pinned image from Caddy's official builder with caddy-dns/cloudflare. We build rather than pull a third-party prebuild: same result, no extra party to trust.

6. Let's Encrypt locks you out for a week¶

Production Let's Encrypt allows roughly 5 failures per hostname per hour, and the resulting lockout lasts a week. Iterating on a Caddyfile against production will burn through that in one afternoon.

Handled by: caddy_staging_acme: true by default. Staging certificates show as untrusted in the browser, which is expected and correct while iterating.

When the config is settled:

Set caddy_staging_acme: false
Delete /opt/caddy/data so real certificates are requested
Re-run playbooks/edge.yml

7. `docker volume prune` deletes client data¶

docker system prune reclaims space from stopped containers and unused images. It is safe. Adding --volumes, or running docker volume prune, deletes the tenant databases, because volumes are where Postgres lives.

Not handled by anything. There is no guard. Do not run it.

Safe: docker system prune · docker image prune Never: docker volume prune · docker system prune --volumes

8. `unattended-upgrades` does not patch containers¶

OS packages and the kernel patch themselves nightly. Container images never do. They are pinned deliberately (ADR-0023 §2), which is the point and also the risk. Metabase in particular has a CVE history including pre-auth remote code execution.

If nobody bumps versions, the host stays patched and the containers rot.

Handled by: nothing automatic, by design. It is a monthly maintenance task and OI-020.

9. Vanity domains need Caddy configured first¶

When a client CNAMEs portal.theirdomain.com.au to their tenant hostname, Caddy must already know that hostname or the certificate request fails, and the client sees a TLS error on the name they just announced internally.

Order: Caddy config first, DNS second. Always (ADR-0022 §5).

10. The only way to lock yourself out¶

The phased hardening process cannot lock you out if you follow it. Public SSH stays open through phases 1 and 2 precisely so Tailscale can be installed and proven before anything closes.

The one way it fails is setting ssh_public_access: false before verifying that both admins can reach the host over its Tailscale address.

If it happens: the Vultr web console works regardless. It is a serial terminal into the VM and depends on neither SSH nor Tailscale. Find it before you start hardening, not during.

See host setup §3.

11. Tailscale node keys expire, and take the host with them¶

This is the most likely way to lose administrative access to a running host, and it fails months after the change that caused it.

Tailscale node keys expire by default, typically after 180 days. When one expires the host silently drops off the tailnet. Nothing crashes, no alert fires, the site keeps serving traffic on 443. You simply cannot reach it, and the last change anyone made was half a year ago, so nobody connects the two.

Under ADR-0031 there is no public SSH port, so there is no second path. The only remaining way in is the provider's web console.

Handled by: nothing automatic. The Ansible role cannot set this, because it is a tailnet-side property rather than a host-side one.

You must do it manually, per machine:

Tailscale admin console → Machines → the host → Disable key expiry.

Do it immediately after the host joins, and verify it before running phase 3. The tailscale role prints a reminder for this reason.

Check the whole fleet occasionally: the Machines list shows an expiry date per node. Any ABOS host showing one is a scheduled outage.

Tracked as OI-056.

12. A file in `group_vars/` only loads for the group it is named after¶

group_vars/vault.yml is not "the vault for all hosts". Ansible reads a file directly in group_vars/ only for the group whose name matches the file, so vault.yml applies to a group called vault, which does not exist. Every secret in it is silently undefined, and the first playbook fails on a missing variable with an error that points nowhere near the cause.

Handled by: the layout. Shared variables live in a folder named for the group, so every file inside applies to every host:

infra/inventory/group_vars/all/main.yml    plain variables
infra/inventory/group_vars/all/vault.yml   secrets, encrypted

Found during the first real use of the vault, October 2026.

13. An encrypted vault file must start with `$ANSIBLE_VAULT`¶

The first line of an encrypted vault file must be $ANSIBLE_VAULT;1.1;AES256 (or 1.2 with a vault ID). Anything above it, even a comment or a blank line, and Ansible reports "Input is not vault encrypted data".

It happens when the encrypted text is pasted into an existing file rather than replacing it. Comments are not lost by removing them from the top: if they were in the file when it was encrypted, they are inside the encrypted content, and ansible-vault edit shows them.

Windows line endings break it too. The repo forces LF on every text file, and vault files are named explicitly in .gitattributes so the reason is written down.

The safe round trip, since octo-ctl has read-only access to the repo:

Encrypt or edit on octo-ctl
cat the file and copy all of it
On the PC, replace the file's entire contents, and commit
Back on octo-ctl, pull and run ansible-vault view on the pulled copy to prove it decrypts

Only after step 4 passes is it safe to delete a secret from anywhere else it was held.

14. An inventory folder loads every file in it, examples included¶

If ansible.cfg points at the inventory/ folder, Ansible reads every file inside it, including hosts.example.yml and the phase examples. A playbook targeting all then also runs against the example hosts.

Harmless while the examples hold placeholder addresses. Dangerous the day one holds a real-looking IP, because that is how someone hardens the wrong machine.

Handled by: ansible.cfg points at the real file, ./inventory/hosts.yml, never the folder. group_vars/ beside it still loads.

The check: ansible all --list-hosts should list only real hosts. Found on the first real run, October 2026.

15. A secret in `group_vars/all` belongs to every host¶

Every file in group_vars/all/ applies to every host in the inventory (trap 12). With one host that is harmless. The day a second host is added, it silently receives the first host's machine secrets: the same n8n encryption key, the same Cloudflare tunnel token (so its cloudflared joins the first host's tunnel), and the first host's alerts@ client secret, so its alerts send under lite-dev's identity and revoking one host no longer leaves the others sending (ADR-0036 §3).

The layout, decided 13 October 2026, and in its own repository since ADR-0042 (16 October 2026):

group_vars/all/vault.yml        only what every host genuinely shares (the tailnet auth key), core
host_vars/<host>/vault.yml      everything one machine holds, core
tenants/<slug>/vault.yml        everything one tenant holds, the tenant's own vault ID

These are in the inventory repository, octopodia-abos-inventory. A host's own file holds vault_abos_alerts_client_secret, vault_cloudflared_tunnel_token, and on the host that runs the Console, the registry passwords and the Console's copies of tenant tokens. A tenant's file holds its n8n encryption key and its own copies of its tokens, because several tenants can share a host. host_vars/ beside hosts.yml loads like group_vars/ does; tenants/<slug>/ is loaded by the playbooks for the tenants a host lists in abos_tenants.

The check: infra/scripts/inventory_check.py refuses any secret in group_vars/all other than the tailnet key, and ansible localhost -m debug -a var=vault_abos_alerts_client_secret must say VARIABLE IS NOT DEFINED (the implicit localhost gets only what every host gets). lite-dev's secrets are moved by the inventory repository runbook (OI-124).

Found when lite-dev's alerts secret was first added to the vault, October 2026.

16. A container cannot resolve tailnet names¶

A container on a Docker network asks Docker's own DNS, which forwards to the host's resolvers and knows nothing of the tailnet. Our hosts do not take MagicDNS either (roles/tailscale, tailscale_accept_dns: false). So https://lite-dev.tail34193f.ts.net/... works from the host and fails inside a container with Name or service not known (gaierror -2).

The fix, decided 15 October 2026: the container gets each tailnet name it calls in its /etc/hosts (Compose extra_hosts), with the IP read from the host's tailscale status --json at every playbook run. The name in the URL stays the real one, so TLS still checks the ts.net certificate; a stale IP fails closed on that check rather than reaching the wrong machine. The Console role does this for every tenant's detail_url, and stops if the name is not a node the host can see (a typo, a host off the tailnet, or a tailnet policy that hides it).

Not dns: 100.100.100.100 in the Compose file: it would send every lookup the container makes (Cloudflare Access keys, Microsoft Graph for the alerts) through tailscaled, so a tailscaled restart or MagicDNS turned off in the admin console would break sign-in and alerts as well, and it quietly undoes the host's deliberate accept-dns false.

Found when the Console's container first called Junior: detail on lite-dev, October 2026.