Host maintenance¶

Ongoing care of a running host. For standing up a new one, see host setup. For adding a client to an existing one, see adding a tenant.

Partly pending

Sections marked ⏳ describe decided behaviour whose Ansible role is not yet written. They are here so the intent is recorded, not because the command works today.

The rhythm¶

flowchart LR
    D["<b>Automatic</b><br/>OS security patches<br/>nightly backups"]
    W["<b>Weekly</b><br/>glance at disk,<br/>backup success,<br/>Sentry"]
    M["<b>Monthly</b><br/>container image bumps<br/>review pinned versions"]
    Q["<b>Six-monthly</b><br/><b>restore test</b><br/>credential audit"]
    D --> W --> M --> Q

Only the six-monthly item is genuinely load-bearing. Everything else is a glance.

What patches itself, and what does not¶

Layer	Patched by	Frequency
OS packages, kernel	`unattended-upgrades`	Automatic, nightly
Reboot for kernel updates	`unattended-upgrades`	Automatic, 04:00 AEST
Container images	Nobody but you	Manual, deliberate

unattended-upgrades does not touch container images

Caddy, Postgres, n8n, Metabase and the brain all run from pinned images (ADR-0023 §2). A pinned image never updates on its own. That is the point, and it is also the risk. Metabase in particular has a CVE history including pre-auth remote code execution (OI-020).

If nobody bumps versions, the host stays patched and the containers rot.

Bumping an image ⏳¶

Change the pinned tag in the role's defaults/main.yml.
Open a PR. The version bump is the reviewable artefact.
Run the relevant playbook.
Confirm the container comes back healthy before closing the PR.

Do this per component, not all at once. A batch bump makes a failure ambiguous.

Weekly glance¶

Check	Command	Looking for
Disk	`df -h`	Under 70%. The landing layer grows and is never pruned
Containers	`docker ps`	Everything `Up`, nothing restart-looping
Memory	`free -h`	Swap in use means the box is undersized
Backups	Object storage bucket	Last night's dump present, per tenant
Errors	Sentry	Anything new since last week
Tailnet	`tailscale status`	Host connected, and no expiry warning

Backups and the restore test¶

Posture is set by ADR-0026: tiered by recoverability, per-tenant dumps to per-tenant paths, encrypted, AU-resident.

flowchart LR
    A["Nightly dump"] --> B["Encrypted with age,<br/>streamed to AU object storage"]
    B --> C{"Six-monthly:<br/>restore into a<br/>scratch container"}
    C -->|"tables present,<br/>counts plausible"| D(["Backup proven ✅"])
    C -->|"mismatch"| E["Backup is theatre.<br/>Fix before anything else"]

The restore test is the deliverable, not the dump

An untested restore is a hopeful cron job. The first restore must be proven before the first paying client goes live (OI-014).

Retention is 30 daily plus 12 monthly. The approval log carries a floor no client can request below: it is the evidence behind "no review, no go", and it exists nowhere else.

ansible-playbook playbooks/verify-restore.yml -e tenant=<slug> --vault-id core@prompt

The host cannot read its own backups, by design

Backups are encrypted to an age public key. The private key lives in Bitwarden and nowhere else, so a compromised host can write backups it cannot read (ADR-0026, OI-054).

The restore playbook prompts for that private key and deletes it afterwards. Lose it and every backup is landfill.

Certificates¶

Caddy renews automatically via the Cloudflare DNS challenge. There is nothing to do, but if a certificate ever fails to renew, the usual cause is the scoped Cloudflare API token having expired or been rotated without updating the vault.

Check: docker logs caddy | grep -i certificate

Resizing the host¶

When the sizing worksheet says you have outgrown the box:

Snapshot first.
Resize in the Vultr console. This reboots the instance.
Confirm every container returns.
Update the sizing worksheet with what the box actually uses.

Resizing is a migration trigger

Per ADR-0025 §3, resizing rather than extending is one of three triggers to move the control plane onto its own host. Do not resize past it without making that decision deliberately.

Credential hygiene¶

Check	When
Confirm both partners hold every vault password	On every tenant added
Confirm MFA recovery codes are in Bitwarden	Six-monthly
Rotate the Cloudflare API token	On any suspicion, or personnel change
Review the credential-name list in the repo	Six-monthly
Tailscale Machines list: no ABOS host shows an expiry date	Six-monthly. Any host showing one is a scheduled outage
Renew the Entra client secret for Cloudflare Access	Before its expiry date. See the warning below

Remember the boundary: human logins in Bitwarden, machine tokens in Ansible Vault (ADR-0027 §7). The same provider legitimately appears in both.

The Entra client secret expires, and there is no fallback

docs.octopodia.com.au sits behind Cloudflare Access with Microsoft Entra ID as the only identity provider. One-time PIN is not enabled as a fallback.

When the Azure app registration's client secret expires, every person loses access to the documentation at once, with an error that gives no hint of the cause. Nobody will connect an outage to a secret created two years earlier.

Mitigations, all of which should be in place:

Secret and its expiry date recorded in Bitwarden
Calendar reminder one month before expiry
To renew: Azure portal, App registrations, Cloudflare Access, Certificates and secrets, New client secret, copy the Value, then paste it into Cloudflare Zero Trust, Integrations, Identity providers
If already locked out, an account owner can re-enable One-time PIN from the Cloudflare dashboard, which does not depend on Entra

When something is wrong¶

Start with Known traps, which maps symptoms to causes for the non-obvious ones. The table below is the first-look list.

Symptom	Look at
Site down, SSH works	`docker ps`, then `docker logs caddy`
Cannot SSH at all, site still serving	Tailscale node key expiry (known traps §11). Check the Machines list, then use the Vultr web console
SSH refused	Vultr web console. Check `tailscale status`, `ufw status`, `systemctl status ssh`
One tenant down, others fine	That tenant's stack only. The isolation boundary is working
Everything down	Host level: disk full, OOM, or a reboot in progress
Disk full	Landing layer or Docker image cruft. `docker system prune`, and never prune volumes

Never run docker volume prune

Volumes are the tenant databases. docker system prune without --volumes is safe. Adding that flag deletes client data. There is no guard and no undo. See Known traps §7.