Skip to content

Host setup — first time

How to stand up and harden a new host, top to bottom on the day. Run this again unchanged for any additional host (a second shared box, or a dedicated one for a client) — nothing here is specific to the first.

Once it is running: host maintenance · adding a tenant

Read Known traps first — or at least know it exists

Ubuntu 24.04, Docker and Caddy each have a non-obvious behaviour that will cost you an evening if you meet it cold. All are handled in the roles; that page explains why the roles look the way they do, and turns a symptom into a diagnosis.

The hardening step is sequenced so you cannot lock yourself out — do not skip the two-phase SSH process.

Baseline decisions: ADR-0021 (host baseline), ADR-0022 (edge routing, TLS, hostnames), ADR-0023 (container isolation), ADR-0024 (Ansible layout), ADR-0025 (control-plane co-location), ADR-0026 (backups), ADR-0027 (secrets and credential custody). This runbook executes them; it doesn't re-decide. (These superseded ADR-0014, which bundled all of the above into one document.)

0. Before you start — have these ready

Do the Bitwarden step first. You are about to generate a Vault password and two SSH keypairs. If the shared store doesn't exist yet, they land in someone's personal manager "temporarily" and stay there — which is the exact bus-factor failure ADR-0027 exists to remove.

  • Bitwarden Teams set up, shared collection created, both partners admins (ADR-0027 §6, OI-015). MFA on both accounts, and Bitwarden's own recovery codes stored outside Bitwarden.
  • Vultr account with billing set up.
  • Two SSH keypairs generated locally — one on El's machine, one on Lachlan's. If not done: ssh-keygen -t ed25519 -C "el@laptop" (repeat for Lachlan). You need the public keys (~/.ssh/id_ed25519.pub). Private keys are never shared (ADR-0021 §4) — a shared key can't be revoked for one person without locking out both, and destroys attribution.
  • core Vault password created and put in Bitwarden immediately. Per-tenant vault passwords come later, at tenant provisioning (ADR-0027 §3).
  • Ansible installed on the machine you'll run from (via pipx, per the dev container setup).
  • Scoped Cloudflare API token — DNS edit on octopodia.com.au only, never the global API key (ADR-0022 §9, OI-042). A global key in a playbook can delete records, which is the one thing we deliberately never automate. This token goes into Ansible Vault under core, not Bitwarden — Ansible consumes it, so it is a machine credential (ADR-0027 §7). Not needed for hardening; needed for Caddy.
  • Provider root logins in Bitwarden: Vultr, Cloudflare, Micron21 (registrar), Microsoft 365, GitHub org owner, Railway — with their MFA recovery codes.

Not needed yet: Vultr object storage for backups (OI-036) — that arrives with the backup role, several steps later.

Locked values (store the port in your password manager now):

Thing Value
Provider/region Vultr — Melbourne
OS Ubuntu 24.04 LTS
Host name bos1
SSH port 52843
Admin users el, lachlan
Domain octopodia.com.au (subdomains)

0b. Naming — read this before touching DNS

The single idea: a host is not a service. The box gets its own name; the things running on it get theirs. That is what makes a client migration invisible later (OI-047).

Host names — the machine itself. Ops vocabulary only; a client never sees one.

Situation Name
Shared host bos1, then bos2, …
Dedicated host for a client {tenant}-bos1 — e.g. greenhills-bos1

Uses the tenant slug from ADR-0022, which is immutable — one vocabulary, not two. No region label while everything is in one region (ADR-0008); add bos-syd1-style names only if that ever changes.

The rule: one A record per host. Everything else is a CNAME to it. Rebuild or resize the box and you edit exactly one record; inventory, SSH config and every service name keep working.

Name Type Points at Who uses it
bos1.octopodia.com.au A the IP Ops only — SSH, Ansible inventory. Not proxied (grey cloud).
console.octopodia.com.au CNAME → bos1 the box You. BOS Console, plus /health for monitoring.
*.app.octopodia.com.au CNAME → bos1 the box Clients. Their review app, plus /insights for Metabase.
docs.octopodia.com.au CNAME Cloudflare Pages You. Deliberately off-box (ADR-0028) — a runbook must survive the outage it is used in.

Why a client never sees bos1: they get greenhills.app.octopodia.com.au, which is independent of where it runs. Moving that tenant to dedicated hardware is an ansible_host change plus a CNAME edit — no new URL, no new certificate for them to trust, no email announcing a change.

Why monitoring has no name of its own: it sits at console.octopodia.com.au/health, same audience and same Access policy as the Console. Fewer names to remember. But uptime checking lives off the box — an on-box monitor cannot tell you the box is down (OI-048).

Do not confuse three things called "console": BOS Console = your operator app. /health = infrastructure monitoring. Vultr web console = the browser terminal into the VM, used when SSH is broken (see step 1).

0c. What you're building

Before step 1, read the deployment topology — the diagram of containers, networks and ports, and the four boundaries it asserts. For what each component is and what it costs in RAM, see stack and sizing.

Every step below builds one part of it.

Step Builds
3 — hardening The host itself: SSH, firewall, fail2ban. No containers yet
4 — docker The runtime and the networks the containers sit on
5 — caddy The front door. Milestone: TLS works on a placeholder page
6 — core_stack Console, brain, registry
7 — tenant_stack One tenant: app, Postgres, n8n, n8n's Postgres
8 — backup The dotted lines to object storage. The restore test is the deliverable

Steps 6 and 7 are currently blocked — there is no brain or registry image to deploy yet. Steps 1–5 are buildable today; stop at the step 5 milestone.

1. Create the host on Vultr

  1. Deploy a new instance:
  2. Location: Melbourne (AU).
  3. OS: Ubuntu 24.04 LTS.
  4. Size: see the sizing worksheet. Short version: 4 GB for client-zero alone, and budget ~1.4 GB per automation tenant or ~3.4 GB per dashboard tenant thereafter. 2 vCPU is enough to start; no GPU, ever, on this box. Resize on demand — but note that resizing is itself a control-plane migration trigger (ADR-0025 §3).
  5. Attach at least one of your SSH public keys at creation time. This is the safe path — the key is on the box before it boots, so your very first login is already key-based.
  6. Note the public IP.
  7. Find the Vultr web console now (instance → "View Console"). This is your ultimate lifeline: it works even if SSH is completely broken. Knowing where it is turns hardening from scary into routine.

First login test: ssh root@<ip> (using the key you attached). If that works, you're ready to harden.

2. Point DNS (can run in parallel)

In Cloudflare, for octopodia.com.au — per the naming rules in step 0b:

  1. bos1 → A → <ip>. Grey cloud (DNS-only, not proxied) — this is the host record, used for SSH; there is no reason for it to be publicly proxied.
  2. console → CNAME → bos1.octopodia.com.au. Proxied.
  3. *.app → CNAME → bos1.octopodia.com.au. Proxied. This wildcard covers every tenant, so adding a client needs no new DNS record.
  4. docs → CNAME → the Cloudflare Pages target. Not the box.

Do not create per-tenant records by hand — *.app already covers them, and tenant records are created by the dns role at provisioning time (ADR-0024 §10).

Not needed for hardening — needed before Caddy. Noted here so it isn't forgotten.

3. Harden the host — the two-phase, no-lockout process

The golden rule: never break your own access. Verify the new way in works BEFORE you close the old way in. Keep one known-good session open the whole time.

The hardening role is built for this: it validates every SSH change with sshd -t before reloading (a broken config self-removes and aborts instead of applying), it reloads SSH rather than restarting (your session survives), and a ssh_keep_port_22 flag keeps port 22 alive during the transition so you can test 52843 safely first.

flowchart TB
    A["<b>PHASE 1</b><br/>ssh_keep_port_22: true<br/>inventory: root @ port 22"]
    B["Run host-hardening.yml"]
    C["Creates el + lachlan, installs keys<br/>sshd listens on <b>22 AND 52843</b><br/>firewall opens both"]
    D{"In a <b>NEW</b> terminal:<br/>ssh -p 52843 el@host → sudo whoami<br/>ssh -p 52843 lachlan@host → sudo whoami<br/><br/>Do BOTH print root?"}
    E["<b>STOP — nothing is lost</b><br/>Port 22 is still open<br/>Root session still alive<br/>Fix the keys, re-run Phase 1"]
    F["<b>PHASE 2</b><br/>ssh_keep_port_22: false<br/>inventory: el @ port 52843"]
    G["Re-run host-hardening.yml"]
    H(["Port 22 closed<br/>SSH is 52843 only ✅"])

    A --> B --> C --> D
    D -->|"either fails"| E
    E --> B
    D -->|"both succeed"| F --> G --> H

Keep your original session open the entire time

Not until the playbook finishes — until after you have logged in successfully on the new port in a separate terminal. That open session is what turns a mistake into an inconvenience.

If you lose it anyway, the Vultr web console still works (step 1). It is the one access path that does not depend on SSH.

What is open at each stage

Port 22 Port 52843 Root login Passwords
Before open closed allowed allowed
After Phase 1 still open open disabled disabled
After Phase 2 closed open disabled disabled

The whole design is that middle row: for a while, both doors are open. That overlap is the safety margin — do not shorten it by setting ssh_keep_port_22: false before verifying.

3a. Set your public keys

In infra/inventory/group_vars/all.yml:

el_ssh_pubkey: "ssh-ed25519 AAAA... el@laptop"
lachlan_ssh_pubkey: "ssh-ed25519 AAAA... lachlan@laptop"

(Public keys are not secret — plain vars is correct here.)

3b. PHASE 1 — run as root, keep port 22 open

Inventory (infra/inventory/hosts.yml, gitignored — copy from the example):

all:
  hosts:
    abos-prod:
      ansible_host: <ip>
      ansible_user: root
      ansible_port: 22

Confirm ssh_keep_port_22: true in roles/hardening/defaults/main.yml, then run:

cd infra
ansible-playbook playbooks/host-hardening.yml

This creates el + lachlan, installs their keys, opens the firewall (22 + 52843 + 80 + 443), and hardens sshd to listen on both 22 and 52843.

3c. ⚠️ VERIFY — in a NEW terminal, KEEP the root session open

Do not close your existing session. In a second terminal:

ssh -p 52843 el@<ip>
sudo whoami        # must print: root
exit
ssh -p 52843 lachlan@<ip>
sudo whoami        # must print: root
exit

Both must succeed. If either fails, stop — you still have your root session and port 22 open, so fix the keys/config and re-run phase 1. Nothing is closed yet.

3d. PHASE 2 — switch over and close port 22

Only once both users verified. Update the inventory:

      ansible_user: el
      ansible_port: 52843

Set ssh_keep_port_22: false in roles/hardening/defaults/main.yml, then re-run:

ansible-playbook playbooks/host-hardening.yml

This closes port 22, stops sshd listening on 22, and confirms root + password login are off (they already were).

3e. Confirm the lockdown

ssh -p 22 root@<ip>      # should now REFUSE / time out — good
ssh -p 52843 el@<ip>     # should still work

Host is hardened. ✅

4. What the hardening covers (checklist)

  • Key-only auth; passwords disabled.
  • Root SSH login disabled.
  • Per-person users (el, lachlan), each own key.
  • SSH on 52843 (22 closed after phase 2).
  • fail2ban enabled on 52843.
  • Firewall: default-deny inbound; only 52843 + 80 + 443 open.
  • Unattended security upgrades installed.

5. Next steps (separate roles — do NOT run ahead)

The remaining roles are built and run after hardening, in this order. Each is walked through before it's run (see the build-order brief):

  1. docker — Docker + Compose, network model, restart/limits.
  2. caddy — reverse proxy + wildcard TLS (needs the Cloudflare token).
  3. core_stack — LangGraph, registry, monitoring.
  4. tenant_stack — stand up client-zero (Octopodia's own inbox) as tenant one.
  5. Backups → Vultr AU object storage; secrets via Ansible Vault throughout.