Host setup: first time

How to stand up and harden a new host, top to bottom on the day. Run this again unchanged for any additional host, whether a second shared box or a dedicated one for a client. Nothing here is specific to the first.

The inventory moved, 16 October 2026

The live inventory is now its own private repository, octopodia-abos-inventory, cloned beside the code (ADR-0042). Paths below written as infra/inventory/... or inventory/... are now in that repository, and secrets no longer go in group_vars/all/vault.yml: a host's go in host_vars/<host>/vault.yml (vault ID core), a tenant's in tenants/<slug>/vault.yml (the tenant's own vault ID), each value added with ansible-vault encrypt_string. See the inventory repository runbook.

Once it is running: host maintenance · adding a tenant

Read Known traps first, or at least know it exists

Ubuntu 24.04, Docker, Caddy and Tailscale each have a non-obvious behaviour that will cost you an evening if you meet it cold. All are handled in the roles. That page explains why the roles look the way they do, and turns a symptom into a diagnosis.

The hardening step is sequenced so you cannot lock yourself out. Do not skip the three-phase process.

Baseline decisions: ADR-0021 (host baseline), ADR-0022 (edge routing, TLS, hostnames), ADR-0023 (container isolation), ADR-0024 (Ansible layout), ADR-0025 (control-plane co-location), ADR-0026 (backups), ADR-0027 (secrets and credential custody), ADR-0031 (Tailscale admin access). This runbook executes them, it does not re-decide them. (ADR-0021 to ADR-0027 superseded ADR-0014, which bundled all of it into one document.)

---

0. Before you start: have these ready

Do the Bitwarden step first. You are about to generate a vault password and two SSH keypairs. If the shared store does not exist yet, they land in someone's personal manager "temporarily" and stay there, which is the exact bus-factor failure ADR-0027 exists to remove.

• Bitwarden organisation and shared collection created, both partners Owners (ADR-0027 §6, OI-015). A Free organisation is sufficient at two users. MFA on both accounts, and Bitwarden's own recovery codes stored outside Bitwarden.
• Vultr account with billing set up.
• Two SSH keypairs generated locally, one on El's machine, one on Lachlan's. See Generating your SSH key and send that page to anyone who needs to make one. You need the public keys (~/.ssh/id_ed25519.pub), which go into infra/inventory/group_vars/all/main.yml. Private keys are never shared (ADR-0021 §4): a shared key cannot be revoked for one person without locking out both, and it destroys attribution.
• core vault password created and put in Bitwarden immediately. Per-tenant vault passwords come later, at tenant provisioning (ADR-0027 §3).
• The control machine, octo-ctl, with Ansible installed via pipx and the repo cloned from main only: playbooks that touch real hosts run from what has been merged, never from work in progress. Windows is not a supported control node. The vault password lives at ~/.config/abos/vault_pass (mode 600), with ANSIBLE_VAULT_PASSWORD_FILE pointing at it, so playbooks need no --vault-id flag there.
• Tailscale auth key: pre-authorised, reusable, carrying tag:server (OI-057). Needed for phase 2. Tagged nodes are owned by the tailnet rather than a person's account, so a host does not vanish when someone's account changes. Machine credential, so it goes in Ansible Vault under core, not Bitwarden (ADR-0027 §7). You may need to define tag:server in the tailnet ACL first (OI-058).
• Scoped Cloudflare API token: DNS edit on octopodia.com.au only, never the global API key (ADR-0022 §9, OI-042). A global key in a playbook can delete records, which is the one thing we deliberately never automate. Also Ansible Vault under core. Not needed for hardening, needed for Caddy.
• Provider root logins in Bitwarden: Vultr, Cloudflare, Micron21 (registrar), Microsoft 365, GitHub org owner, Railway, each with its MFA recovery codes.

Not needed yet: Vultr object storage and the age backup keypair (OI-036, OI-054). Those arrive with the backup role, several steps later.

Locked values (store the port in Bitwarden now):

Thing	Value
Provider/region	Vultr, Melbourne
OS	Ubuntu 24.04 LTS
Host name	bos1
SSH port	52843
Admin users	el, lachlan
Domain	octopodia.com.au (subdomains)

---

0b. Naming: read this before touching DNS

The single idea: a host is not a service. The box gets its own name, and the things running on it get theirs. That is what makes a client migration invisible later (OI-047).

Host names are the machine itself. Ops vocabulary only, and a client never sees one.

Situation	Name
Shared host	bos1, then bos2, …
Dedicated host for a client	{tenant}-bos1, for example greenhills-bos1

Uses the tenant slug from ADR-0022, which is immutable, so there is one vocabulary rather than two. No region label while everything is in one region (ADR-0008). Add bos-syd1-style names only if that ever changes.

The rule: one A record per host. Everything else is a CNAME to it. Rebuild or resize the box and you edit exactly one record. Inventory, SSH config and every service name keep working.

Name	Type	Points at	Who uses it
bos1.octopodia.com.au	A	the IP	Ops only. Not proxied (grey cloud)
console.octopodia.com.au	CNAME → bos1	the box	You. BOS Console, plus /health for monitoring
*.app.octopodia.com.au	CNAME → bos1	the box	Clients. Their review app, plus /insights for Metabase
docs.octopodia.com.au	CNAME	Cloudflare Pages	You. Deliberately off-box (ADR-0028): a runbook must survive the outage it is used in

Note that under ADR-0031 the host record is no longer the SSH target. Admin access is over Tailscale, and bos1.octopodia.com.au exists so that the service CNAMEs have something stable to point at.

Why a client never sees bos1: they get greenhills.app.octopodia.com.au, which is independent of where it runs. Moving that tenant to dedicated hardware is an ansible_host change plus a CNAME edit. No new URL, no new certificate for them to trust, no email announcing a change.

Why monitoring has no name of its own: it sits at console.octopodia.com.au/health, same audience and same Access policy as the Console. Fewer names to remember. But uptime checking lives off the box, because an on-box monitor cannot tell you the box is down (OI-048).

Do not confuse three things called "console": BOS Console is your operator app. /health is infrastructure monitoring. Vultr web console is the browser terminal into the VM, used when SSH is broken (see step 1).

---

0c. What you are building

Before step 1, read the deployment topology: the diagram of containers, networks and ports, and the four boundaries it asserts. For what each component is and what it costs in RAM, see stack and sizing.

Every step below builds one part of it.

Step	Builds
3, hardening	The host itself: SSH, firewall, fail2ban. No containers yet
3d, tailscale	Admin access over the tailnet, before public SSH closes
4, docker	The runtime and the networks the containers sit on
5, caddy	The front door. Milestone: TLS works on a placeholder page
6, core_stack	Console, brain, registry
7, tenant_stack	One tenant: app, Postgres, n8n, n8n's Postgres
8, backup	The dotted lines to object storage. The restore test is the deliverable

Steps 6 and 7 are currently blocked: there is no brain or registry image to deploy yet. Steps 1 to 5 are buildable today. Stop at the step 5 milestone.

---

1. Create the host on Vultr

1. Deploy a new instance:
2. Location: Melbourne (AU).
3. OS: Ubuntu 24.04 LTS.
4. Size: see the sizing worksheet. Short version: 4 GB for client-zero alone, then budget around 1.4 GB per automation tenant or 3.4 GB per dashboard tenant. 2 vCPU is enough to start, and no GPU, ever, on this box. Resize on demand, but note that resizing is itself a control-plane migration trigger (ADR-0025 §3).
5. Attach at least one of your SSH public keys at creation time. The key is then on the box before it boots, so your very first login is already key-based.
6. Note the public IP.
7. Find the Vultr web console now (instance, then "View Console"). This is your ultimate lifeline: it works even if SSH and Tailscale are both broken. Knowing where it is turns hardening from scary into routine.

First login test: ssh root@<ip> using the key you attached. If that works, you are ready to harden.

---

2. Point DNS (can run in parallel)

In Cloudflare, for octopodia.com.au, per the naming rules in step 0b:

1. bos1 → A → <ip>. Grey cloud, DNS-only, not proxied.
2. console → CNAME → bos1.octopodia.com.au. Proxied.
3. *.app → CNAME → bos1.octopodia.com.au. Proxied. This wildcard covers every tenant, so adding a client needs no new DNS record.
4. docs → CNAME → the Cloudflare Pages target. Not the box.

Do not create per-tenant records by hand. *.app already covers them, and tenant records are created by the dns role at provisioning time (ADR-0024 §10).

Once the host exists and has an IP, the dns role can do steps 1 to 3 for you:

ansible-playbook playbooks/dns.yml --vault-id core@prompt

Not needed for hardening, needed before Caddy. Noted here so it is not forgotten.

---

3. Harden the host: the three-phase, no-lockout process

The golden rule: never break your own access. Verify the new way in works BEFORE you close the old way in. Keep one known-good session open the whole time.

Administrative access ends up on Tailscale with no public SSH port at all (ADR-0031). Getting there safely takes three phases: public SSH stays open while Tailscale is installed and proven, and only closes once both admins have used it.

The hardening role is built for this. It validates every SSH change with sshd -t before reloading, so a broken config removes itself and aborts rather than applying. It reloads SSH rather than restarting, so your session survives. And ssh_public_access controls the phase.

flowchart TB
    A["<b>PHASE 1</b><br/>ssh_public_access: true<br/>inventory: root @ port 22"]
    B["Run host-hardening.yml"]
    C["Creates el + lachlan, installs keys<br/>sshd on <b>22 AND 52843</b><br/>public SSH still open"]
    D{"NEW terminal, both admins:<br/>ssh -p 52843 el@&lt;public-ip&gt;<br/>ssh -p 52843 lachlan@&lt;public-ip&gt;<br/><br/>Both print root?"}
    E["<b>STOP. Nothing is lost.</b><br/>Public SSH still open,<br/>root session still alive.<br/>Fix keys, re-run Phase 1"]
    F["<b>PHASE 2</b><br/>Run tailnet.yml"]
    G["Host joins the tailnet.<br/><b>Disable key expiry</b> in the<br/>Tailscale admin console"]
    H{"Both admins over Tailscale:<br/>ssh -p 52843 el@&lt;tailscale-ip&gt;<br/>ssh -p 52843 lachlan@&lt;tailscale-ip&gt;<br/><br/>Both print root?"}
    I["<b>STOP.</b> Public SSH is still<br/>open. Fix the tailnet, retry"]
    J["<b>PHASE 3</b><br/>ssh_public_access: false<br/>Re-run host-hardening.yml"]
    K(["No public SSH port.<br/>Admin access via Tailscale only ✅"])

    A --> B --> C --> D
    D -->|"either fails"| E
    E --> B
    D -->|"both succeed"| F --> G --> H
    H -->|"either fails"| I
    I --> F
    H -->|"both succeed"| J --> K

Keep your original session open the entire time

Not until the playbook finishes, but until after you have logged in successfully by the new route in a separate terminal. That open session is what turns a mistake into an inconvenience.

If you lose it anyway, the Vultr web console still works (step 1). It depends on neither SSH nor Tailscale.

What is open at each stage

	Public 22	Public 52843	Tailscale 52843	Root login	Passwords
Before	open	closed	n/a	allowed	allowed
After Phase 1	still open	open	n/a	disabled	disabled
After Phase 2	still open	still open	open	disabled	disabled
After Phase 3	closed	closed	open	disabled	disabled

The design is those middle rows: for a while both doors are open. That overlap is the safety margin. Do not shorten it by setting ssh_public_access: false before verifying Tailscale.

3a. Set both public keys

In infra/inventory/group_vars/all/main.yml, replacing the placeholders:

admin_users:
  - name: el
    key: "ssh-ed25519 AAAA... el@laptop"
  - name: lachlan
    key: "ssh-ed25519 AAAA... lachlan@laptop"

Public keys are not secret, so plain vars is correct. See Generating your SSH key.

3b. PHASE 1: harden, with public SSH still open

Inventory (infra/inventory/hosts.yml, gitignored, copy from the example):

all:
  hosts:
    bos1:
      ansible_host: <public-ip>
      dns_host_ip: <public-ip>
      ansible_user: root
      ansible_port: 22

ssh_public_access defaults to true in roles/hardening/defaults/main.yml, which is phase 1. Leave the default alone; phase 3 is set per host in the inventory (step 3f). Then:

cd infra
ansible-playbook playbooks/host-hardening.yml

Creates el and lachlan, installs their keys, hardens sshd to listen on both 22 and 52843, and opens the firewall.

3c. VERIFY, in a NEW terminal, keeping the root session open

ssh -p 52843 el@<public-ip>
sudo whoami        # must print: root
exit
ssh -p 52843 lachlan@<public-ip>
sudo whoami        # must print: root
exit

Both must succeed. If either fails, stop. You still have the root session and public SSH, so fix the keys and re-run phase 1. Nothing is closed.

3d. PHASE 2: join the tailnet

Requires a pre-authorised, reusable Tailscale auth key carrying tag:server, in the core vault (OI-057).

Switch the inventory first. Phase 1 disabled root login, so Ansible can no longer connect as root on 22. Connect as el on 52843, still over the public address (see inventory/hosts.phase2.example.yml):

      ansible_host: <public-ip>
      ansible_user: el
      ansible_port: 52843

ansible-playbook playbooks/tailnet.yml --vault-id core@prompt

The role prints the host's Tailscale address when it finishes.

Disable key expiry before going further

Tailscale admin console → Machines → bos1 → Disable key expiry.

Node keys expire by default, typically after 180 days. An expired key silently drops the host off the tailnet, and after phase 3 there is no public SSH port to fall back to. It fails months after the change that caused it, when nobody connects the two.

The Ansible role cannot set this: it is a tailnet-side property, not a host-side one. See known traps §11 and OI-056.

3e. VERIFY over Tailscale

ssh -p 52843 el@<tailscale-ip>
sudo whoami
exit
ssh -p 52843 lachlan@<tailscale-ip>
sudo whoami
exit

Both must succeed, from both people's machines. Public SSH is still open, so a failure here costs nothing.

3f. PHASE 3: close public SSH

Only once both verifications have passed. Update this host's inventory entry to reach it over Tailscale, and set ssh_public_access: false on the host itself:

      ansible_host: <tailscale-ip>
      dns_host_ip: <public-ip>      # DNS record stays public
      ansible_user: el
      ansible_port: 52843
      ssh_public_access: false      # this host only

Never change the role's default instead. The default stays true, so every new host starts in phase 1. Setting false per host means each host's phase lives with that host, and closing SSH on one can never accidentally close it on another that has not yet joined the tailnet. Then re-run:

ansible-playbook playbooks/host-hardening.yml

3g. Confirm the lockdown

ssh -p 52843 el@<public-ip>      # must REFUSE or time out
ssh -p 52843 el@<tailscale-ip>   # must work

Host is hardened. The only ports open to the internet are 80 and 443.

---

4. What the hardening covers

• Key-only authentication, passwords disabled
• Root SSH login disabled
• Per-person users (el, lachlan), each with their own key
• SSH on 52843, Tailscale interface only
• No public SSH port at all (ADR-0031)
• Public exposure limited to 80 and 443
• fail2ban enabled (near-redundant now, kept for phases 1 and 2)
• Unattended OS security upgrades
• Tailscale key expiry disabled in the admin console (manual, OI-056)

---

5. Next steps: separate roles, do not run ahead

1. docker: Docker and Compose, the network model, restart policy and limits
2. caddy: reverse proxy and wildcard TLS. Needs the Cloudflare token. Milestone: the placeholder page over a valid certificate
3. core_stack: brain, registry, Console
4. tenant_stack: client-zero, Octopodia's own inbox, as tenant zero
5. backup: encrypted to AU object storage. The restore test is the deliverable

Steps 3 and 4 are blocked: no brain or registry image exists yet.