Split deployment

The self-hosting guide puts everything on one machine, which is the right answer until it is not. This page is the other shape: the control plane on a container host, the worker fleet on machines you own, and the durable pieces in a cloud region next to the control plane.

It is the deployment Warmbly's own architecture describes. The control plane owns relational state and runs where a container host is convenient. The execution plane is a fleet of machines you rent, because a worker's value is being a distinct machine somewhere.

container host          machines you own        cloud region
──────────────          ────────────────        ────────────
backend                 worker                  Postgres
consumer                worker                  KMS key
realtime                nats + redis            S3 bucket
tracking                                        SES
forms
web / admin

Everything here is env-var configuration and manifests in deploy/split-cloud/. No code changes, no fork.

What goes where, and why

The consumer is the one people put in the wrong place. It is not a worker: it updates relational state, and it needs the database DSN a worker is deliberately never given. Running it beside the backend saves a database connection crossing the internet for every event.

Keep the three regions close

The backend reaches Postgres and Redis on every request, so the round trips between the three providers are the latency floor of the whole product. Pick a container-host region, a cloud region, and a datacentre for the machines that are near each other, and confirm it before you build anything on top.

Two things that are not optional

Blobs have to be object storage. A worker reads the message body the backend wrote. With BLOB_PROVIDER=filesystem it has neither the disk nor the permissions, and the failure arrives at the last step of a send with everything else looking healthy. warmbly join warns about this and starts the node anyway; the warning is the whole warning you get.

Nodes carry no cloud credentials

A machine in the fleet needs two privileged things: opening the sealed data key for an organization, and reading and writing message bodies. Both would normally mean an AWS credential on every box, in a file warmbly join rewrites on each run.

It does not work that way. When the control plane runs KMS_PROVIDER=aws or BLOB_PROVIDER=s3, a joining node is handed the brokered form instead:

A brokered provider authenticates with the internal API token the node already holds and asks the instance to perform the one privileged operation: unwrap this key, sign this blob operation. The instance token is scoped to one deployment and revocable from it, which an IAM access key on a rented machine is not.

Blob bytes are not proxied. The control plane signs a URL and the node transfers directly against the object store, so a mailbox sync costs the backend one small request per object rather than the bytes.

The cost is one HTTPS call per DEK open, which Redis caches, and one per blob operation. If you would rather a node talk to AWS directly, put KMS_PROVIDER, BLOB_PROVIDER and the credentials in /etc/warmbly/node.local.env, which a re-join does not overwrite.

Two limits are worth knowing. A signed URL covers one verb on one key, and only for the prefixes a node reaches: transport bodies, campaign attachments, and the mailbox bodies a sync stores. Avatars, form assets and workspace export archives cannot be signed for at all, so the credential a node holds is not a credential for the bucket.

That credential is NODE_BROKER_TOKEN. It falls back to INTERNAL_API_TOKEN, which is what a single-machine install uses, but set it to its own value here: the tracking and forms services are internet-facing and carry the shared token, and there is no reason for what they hold to be enough to open a data key.

Building it

scripts/aws-bootstrap.sh --domain example.com --region eu-central-1

cp -r deploy/split-cloud/bus /opt/warmbly/bus
cd /opt/warmbly/bus
printf 'NATS_TOKEN=%s\nREDIS_PASSWORD=%s\n' \
  "$(openssl rand -hex 32)" "$(openssl rand -hex 32)" > .env
docker compose up -d

PRIMARY_DB=postgres://warmbly:[email protected]:5432/warmbly?sslmode=verify-full
REDIS=rediss://:<password>@bus.example.com:6380
NATS_URL=tls://<token>@bus.example.com:4222
ENCRYPTED_KEYS_BACKEND_URL=https://api.example.com
BLOB_PROVIDER=s3

warmblyctl fleet join-token

# on each machine, as root
curl -fsSL https://api.example.com/join.sh | sh -s -- \
  --url https://api.example.com \
  --token <join-token> \
  --role worker \
  --region eu-central

Anything the control plane cannot know

warmbly join writes /etc/warmbly/node.env from the control plane's answer and rewrites it on every join. Next to it, /etc/warmbly/node.local.env is created once and never written again, and the container reads it second, so a name repeated there wins.

That is the place for a value this instance does not hold: a DSN kept in a secret manager rather than the environment, credentials for infrastructure of your own, a per-machine tuning knob.

printf 'PRIMARY_DB=%s\n' "postgres://..." >> /etc/warmbly/node.local.env
systemctl restart warmbly-consumer

Sizing the fleet

A worker's capacity is 16 cold-mailbox equivalents, and each mailbox declares its own weight: an SMTP or IMAP mailbox costs 1.0, a Gmail or Outlook mailbox 0.05, a warmup-only mailbox 0.4.

So one worker holds 16 SMTP mailboxes or around 300 OAuth ones. OAuth-heavy customers cost almost nothing in machines; SMTP and IMAP customers are what drive the count.

Give NATS and Redis their own machine once you have more than one worker. Until then the control plane depends on a box that also runs a worker, and restarting that worker takes the bus with it.

What it costs

A starting instance, with one machine running the bus and the first worker:

Two things move that number. A cloud account under twelve months old gets the smallest database instance free, which takes roughly $14 off. Turning on a standby doubles the database line, so leave it off at the start and take snapshots instead.

The line that grows quietly is egress. Every query result and every message body a node reads crosses the internet, billed past the first 100 GB a month. It is invisible at launch and becomes the second-largest cloud line once mailbox sync is moving real volume.

Where things run out

One bus machine is one failure domain. Nothing sends while it is down. Snapshots and a documented rebuild are the honest answer at this size; NATS clustering is the answer when the instance is worth more than the afternoon it costs.

The database is reachable from the internet. That is what makes a container host and a fleet able to share it. Restrict the security group to the addresses that need it, force TLS, and treat the master password as the credential it is.

Auto-update covers nodes, not the control plane. The backend is what tells every node which version to be, so it upgrades the way the rest of your container host does. Upgrade it, and the fleet follows.