Known traps¶
Environment-specific behaviour that has cost, or would cost, real time. Each one is handled in the Ansible roles. This page exists so you know why the role looks the way it does, and so a symptom is diagnosable at 11pm.
Symptom to cause¶
| Symptom | Cause | Section |
|---|---|---|
| Hardened the host, port 52843 never comes up | Ubuntu 24.04 socket-activates SSH, so Port in sshd_config is ignored |
1 |
| fail2ban running but never bans anything | No /var/log/auth.log on 24.04; sshd logs to the journal |
2 |
| Disk full, every tenant down at once | Docker's default log driver grows without limit | 3 |
| Tenant provisioning fails: "no available address range" | Docker's default network pool runs out at ~31 networks | 4 |
| Wildcard certificate never issues | Official Caddy image has no Cloudflare DNS module | 5 |
| Certificate requests suddenly all fail for a week | Let's Encrypt production rate limit hit while iterating | 6 |
| Client data gone after a disk cleanup | docker volume prune deleted the tenant databases |
7 |
| Host fully patched, containers months out of date | unattended-upgrades does not touch container images |
8 |
| Vanity domain returns a TLS error | CNAME resolved before Caddy knew the hostname | 9 |
| Locked out of the server | Phase 2 run before verifying phase 1 | 10 |
1. Ubuntu 24.04 ignores your SSH port¶
Ubuntu 24.04 socket-activates SSH. While ssh.socket is enabled, sshd ignores the Port directive entirely, because the socket unit decides the port. You harden the box, reload, and the new port never appears.
Handled by: the hardening role detects ssh.socket, disables it, and enables ssh.service.
If you hit it manually: systemctl disable --now ssh.socket && systemctl enable --now ssh.service
2. fail2ban silently bans nothing¶
Ubuntu 24.04 has no /var/log/auth.log. sshd logs to the journal. A fail2ban jail with the default file backend watches a file that does not exist. It runs, reports healthy, and bans nobody.
Handled by: backend = systemd in jail.local.
Verify: fail2ban-client status sshd should show a non-zero failure count on a box that has been reachable for a day.
3. Container logs fill the disk¶
Docker's default json-file driver has no size limit. One chatty container writes until the disk is full, and a full disk takes down every tenant on the box simultaneously. This is the most common self-inflicted outage on a Docker host.
Handled by: log-opts in daemon.json, at 10 MB by 3 files per container.
Check: docker system df and df -h in the weekly glance.
4. Docker runs out of networks¶
Docker's default address pool provides roughly 31 networks. Every tenant gets one (ADR-0023), so this is a ceiling you would actually reach. The failure surfaces during tenant provisioning as a confusing address-range error, long after the decision that caused it.
Handled by: default-address-pools in daemon.json, using 10.201.0.0/16 in /24s, giving 256.
Changing this later is disruptive
Address pools are read at daemon start and existing networks keep their subnets. Widening the pool after networks exist means recreating them, which means downtime for every tenant. It is set correctly at build time for exactly this reason.
5. Caddy cannot do DNS challenges out of the box¶
Caddy modules are compiled in, not loaded at runtime. The official caddy image has no Cloudflare DNS provider, and the DNS challenge is the only way to obtain a wildcard certificate. Without it, *.app.octopodia.com.au never issues, and the failure looks like a token problem.
Handled by: the caddy role builds a pinned image from Caddy's official builder with caddy-dns/cloudflare. We build rather than pull a third-party prebuild: same result, no extra party to trust.
6. Let's Encrypt locks you out for a week¶
Production Let's Encrypt allows roughly 5 failures per hostname per hour, and the resulting lockout lasts a week. Iterating on a Caddyfile against production will burn through that in one afternoon.
Handled by: caddy_staging_acme: true by default. Staging certificates show as untrusted in the browser, which is expected and correct while iterating.
When the config is settled:
- Set
caddy_staging_acme: false - Delete
/opt/caddy/dataso real certificates are requested - Re-run
playbooks/edge.yml
7. docker volume prune deletes client data¶
docker system prune reclaims space from stopped containers and unused images. It is safe. Adding --volumes, or running docker volume prune, deletes the tenant databases, because volumes are where Postgres lives.
Not handled by anything. There is no guard. Do not run it.
Safe: docker system prune · docker image prune
Never: docker volume prune · docker system prune --volumes
8. unattended-upgrades does not patch containers¶
OS packages and the kernel patch themselves nightly. Container images never do. They are pinned deliberately (ADR-0023 §2), which is the point and also the risk. Metabase in particular has a CVE history including pre-auth remote code execution.
If nobody bumps versions, the host stays patched and the containers rot.
Handled by: nothing automatic, by design. It is a monthly maintenance task and OI-020.
9. Vanity domains need Caddy configured first¶
When a client CNAMEs portal.theirdomain.com.au to their tenant hostname, Caddy must already know that hostname or the certificate request fails, and the client sees a TLS error on the name they just announced internally.
Order: Caddy config first, DNS second. Always (ADR-0022 §5).
10. The only way to lock yourself out¶
The two-phase SSH process cannot lock you out if you follow it. The one way it fails is setting ssh_keep_port_22: false before verifying that both admins can log in on the new port.
If it happens: the Vultr web console works regardless. It is a serial terminal into the VM and does not depend on SSH. Find it before you start hardening, not during.
See host setup §3.