Skip to content

ADR-0030: Multi-host composition, what is global and what is per host

Context

"Core stack" has been treated as a single unit built once per host. That was accurate while there was one host, and stops being accurate the moment there is a second, because the components inside it do not scale the same way.

Caddy must exist wherever traffic arrives. The registry must exist exactly once, or it is no longer the map. Bundling the two under one name invites the wrong answer to an obvious question: "do we need another core stack on the second box?"

Decision

The dividing line is global versus per host, not core versus tenant.

Component Scope Why
Registry Exactly one, globally It is the map (ADR-0006). Two registries is two sources of truth, and no reconciliation pass can repair that
BOS Console Exactly one, globally The operator surface over the single registry
Caddy One per host It terminates TLS for the tenants on that host, and traffic arrives at a box
Brain One replica per host Stateless, so copies are cheap and each host serves its own tenants without a network hop per call
Tenant stack One per tenant Unchanged (ADR-0023)

The brain is replicated, not forked

One image, one pinned version, one role. Ansible deploys it to every host in inventory, in the same way common and hardening already apply to every host. There is no second thing to maintain.

Replicas are only safe while the brain holds no state that outlives a request. This is a hard constraint rather than a description:

  • Config is read from the registry, never cached to disk.
  • Work items, approvals and pgvector memory live in the tenant's Postgres.
  • LangGraph checkpoints go to the tenant's Postgres, never to local disk or memory. A work item resumed on a different host must find its state, or the brain has quietly become stateful and the replicas diverge.

Version skew is the real cost

Two hosts running different engine versions is a genuinely difficult class of bug: behaviour differs by which box a tenant happens to sit on, and nothing looks broken.

  • Deploy the brain to all hosts in one playbook run, never one host at a time.
  • The reconciliation pass (ADR-0006) reports the brain version per host, so skew is visible rather than inferred.

A second host triggers two things already decided

  1. The control plane moves to its own host. ADR-0025 §3 already lists "the first dedicated-box tenant" as a migration trigger, and a second box is usually that. In practice the registry does not stay co-located on bos1 while bos2 reaches across to it.
  2. Private networking becomes mandatory. Brain-to-registry is a Docker network on one host and a real network hop across two, and it must not traverse the public internet. Tailscale is the mechanism, and it is now adopted from the start (ADR-0031), which removes what would otherwise have been a prerequisite.

Consequences

The answer to "does the second box need another core stack?" is now precise: another Caddy and another brain replica, yes, and another registry or Console, never.

Because the brain is deployed by the same role to every host, host count is an inventory concern rather than an architectural one, which is what makes moving a tenant to dedicated hardware a one-line change (ADR-0024 §1).

The statelessness constraint is now load-bearing rather than incidental. Any future feature that wants to keep something in the brain between requests is a breaking change to this ADR and must be raised as one.

Cross-host operation adds a private network to secure and monitor. Since ADR-0031 adopts Tailscale from the start, that cost is largely already paid.

Alternatives considered

  • A registry per host, reconciled, is rejected. Two sources of truth for tenant identity and region is precisely the failure ADR-0006 exists to prevent, and reconciliation cannot arbitrate a genuine conflict.
  • One brain shared across hosts is rejected. Every call from a tenant on the other host becomes a network hop, and the brain becomes a single point of failure for hosts it does not live on, for no saving, since it is stateless and cheap.
  • Caddy on one host fronting both is rejected. All traffic for both hosts would route through one box, recreating a single point of failure at the edge and adding a hop to every request.