Host maintenance¶

Ongoing care of a running host. For standing up a new one, see host setup; for adding a client to an existing one, see adding a tenant.

Partly pending

Sections marked ⏳ describe decided behaviour whose Ansible role is not yet written. They are here so the intent is recorded, not because the command works today.

The rhythm¶

flowchart LR
    D["<b>Automatic</b><br/>OS security patches<br/>nightly backups"]
    W["<b>Weekly</b><br/>glance at disk,<br/>backup success,<br/>Sentry"]
    M["<b>Monthly</b><br/>container image bumps<br/>review pinned versions"]
    Q["<b>Six-monthly</b><br/><b>restore test</b><br/>credential audit"]
    D --> W --> M --> Q

Only the six-monthly item is genuinely load-bearing. Everything else is a glance.

What patches itself, and what does not¶

Layer	Patched by	Frequency
OS packages, kernel	`unattended-upgrades`	Automatic, nightly
Reboot for kernel updates	`unattended-upgrades`	Automatic, 04:00 AEST
Container images	Nobody — you	Manual, deliberate

unattended-upgrades does not touch container images

Caddy, Postgres, n8n, Metabase and the brain all run from pinned images (ADR-0023 §2). A pinned image never updates on its own — that is the point, and it is also the risk. Metabase in particular has a CVE history including pre-auth remote code execution (OI-020).

If nobody bumps versions, the host stays patched and the containers rot.

Bumping an image ⏳¶

Change the pinned tag in the role's defaults/main.yml.
Open a PR — the version bump is the reviewable artefact.
Run the relevant playbook.
Confirm the container comes back healthy before closing the PR.

Do this per component, not all at once. A batch bump makes a failure ambiguous.

Weekly glance¶

Check	Command	Looking for
Disk	`df -h`	Under 70%. The landing layer grows and is never pruned
Containers	`docker ps`	Everything `Up`, nothing restart-looping
Memory	`free -h`	Swap in use means the box is undersized
Backups	Object storage bucket	Last night's dump present, per tenant
Errors	Sentry	Anything new since last week
Bans	`fail2ban-client status sshd`	Volume is normal; a sudden spike is not

Backups and the restore test¶

Posture is set by ADR-0026 — tiered by recoverability, per-tenant dumps to per-tenant paths, encrypted, AU-resident.

flowchart LR
    A["Nightly dump"] --> B["Encrypted<br/>to AU object storage"]
    B --> C{"Six-monthly:<br/>restore into a<br/>scratch container"}
    C -->|"row counts match"| D(["Backup proven ✅"])
    C -->|"mismatch"| E["Backup is theatre.<br/>Fix before anything else"]

The restore test is the deliverable, not the dump

An untested restore is a hopeful cron job. The first restore must be proven before the first paying client goes live (OI-014).

Retention is 30 daily plus 12 monthly. The approval log carries a floor no client can request below — it is the evidence behind "no review, no go", and it exists nowhere else.

Certificates¶

Caddy renews automatically via the Cloudflare DNS challenge. There is nothing to do — but if a certificate ever fails to renew, the usual cause is the scoped Cloudflare API token having expired or been rotated without updating the vault.

Check: docker logs caddy | grep -i certificate

Resizing the host¶

When the sizing worksheet says you have outgrown the box:

Snapshot first.
Resize in the Vultr console — this reboots the instance.
Confirm every container returns.
Update the sizing worksheet with what the box actually uses.

Resizing is a migration trigger

Per ADR-0025 §3, resizing rather than extending is one of three triggers to move the control plane onto its own host. Do not resize past it without making that decision deliberately.

Credential hygiene¶

Task	When
Confirm both partners hold every vault password	On every tenant added
Confirm MFA recovery codes are in Bitwarden	Six-monthly
Rotate the Cloudflare API token	On any suspicion, or personnel change
Review the credential-name list in the repo	Six-monthly

Remember the boundary: human logins in Bitwarden, machine tokens in Ansible Vault (ADR-0027 §7). The same provider legitimately appears in both.

When something is wrong¶

Start with Known traps — it maps symptoms to causes for the non-obvious ones. The table below is the first-look list.

Symptom	Look at
Site down, SSH works	`docker ps`, then `docker logs caddy`
SSH refused	Vultr web console. Check `ufw status` and `systemctl status ssh`
One tenant down, others fine	That tenant's stack only — the isolation boundary is working
Everything down	Host-level: disk full, OOM, or a reboot in progress
Disk full	Landing layer or Docker image cruft. `docker system prune` — never prune volumes

Never run docker volume prune

Volumes are the tenant databases. docker system prune without --volumes is safe; adding that flag deletes client data. There is no guard and no undo — see Known traps §7.