Architecture

Warmbly is split into two planes:

• Control plane: backend API, consumer, tracking, realtime, web. Runs in one region on a container host (Railway in production).
• Execution plane: a fleet of worker processes, one per VPS, spread across many providers and IPs. Each VPS runs the worker as a systemd-supervised Docker container.

The boundary exists for two reasons. First, cold-mail deliverability lives at the IP level, so workers must be spread across distinct machine-level network identities. Second, the control plane owns relational state and the worker fleet should remain disposable.

Services

Data flow

The frontend talks to the backend over REST + JWT, and to the realtime service over WebSocket. Backend writes business state to Postgres. Backend, tracking, and workers publish to the event bus: NATS JetStream with JSON encoding by default, Kafka with Avro and Schema Registry as an opt-in build. The consumer reads the bus and updates Postgres (analytics, suppression, deliverability). Workers subscribe to a topic named for their worker UUID (w.<uuid>) and publish results to jobs.worker-events; see the event system for the full topic map.

Realtime fanout is a separate channel: backend and consumer publish JSON events over Redis pub/sub (or Google Cloud Pub/Sub when PUBSUB_ENABLED=true); the Elixir realtime service subscribes and pushes to connected WebSocket clients.

Object storage: encrypted email bodies (EMSG format) live in the blob store (filesystem by default, S3-compatible opt-in).

Data stores

Encryption model

Warmbly uses envelope encryption end-to-end for sensitive data.

KMS holds the master key. Each organization gets a 32-byte data encryption key (DEK), generated by KMS and stored encrypted in the organization_encrypted_keys Postgres table keyed by organization ID (workers reach it over the backend's internal API rather than touching Postgres). The DEK is decrypted only at the moment of use; cached in Redis with a TTL to amortize cost.

Application-layer secrets are sealed with AES-256-GCM under the DEK and base64-encoded. This applies to:

• email account credentials (IMAP/SMTP passwords, OAuth tokens)
• email body content stored in S3 (EMSG format)

Instance-level secrets that are not owned by any organization are encrypted under a platform DEK (key ID = uuid.Nil): the same envelope as organization secrets, the same trust boundary, a different identity. See internal/app/cipher/.

Mailbox credentials are the one exception to the per-organization DEK. They are sealed with CREDENTIALS_ENCRYPTION_KEY (KeyDomainInstance) because a worker reads them without an organization context.

Worker model

Every Warmbly process that runs on a machine you own is a node: a worker sends and syncs mail, a consumer processes events. Both share one registry (fleet_nodes) and one lifecycle. workers is the placement extension over that registry and holds only account_count, health_state and load_score; workers.id is the node id.

The fleet is pull-based. A node joins by running one command with the instance join token, then heartbeats forever, and nothing is ever pushed to it:

1. The operator issues a join token with warmblyctl fleet join-token.
2. On the machine, curl -fsSL https://api.example.com/join.sh | sh -s -- --url ... --token ... --role worker enrols the node. The script is embedded in the backend and served at GET /join.sh, so a self-hosted fleet never depends on a vendor host and always gets a script matching its own backend.
3. The backend returns a node id, the config rendered from its own environment, and the version to run. The script writes /etc/warmbly/node.env, installs a systemd service and an update timer, and starts the node container.
4. The node subscribes to the event bus topic named for its id and heartbeats, pacing its own beat at a third of models.NodeLivenessWindow from the value the server returns.

The heartbeat reply carries exactly one instruction: desired_version. The node writes it to a file and a systemd timer pulls and restarts, so the process being replaced is never the process doing the replacing. An empty desired_version means "no opinion", never "downgrade to nothing".

The machine needs no inbound port, no SSH key and no cloud account. There is no restart, logs, reboot or uninstall action in the dashboard, because nothing reaches into a machine. The operator surface is warmblyctl fleet (join-token, list, show, remove, pin, version, channel) and the admin panel's Fleet section.

Why per-VPS instead of a Kubernetes DaemonSet

A k8s DaemonSet was the previous shape. It was removed because k8s nodes typically NAT all pods through a small set of egress IPs, and pods churn while IPs do not. What a mailbox provider remembers is the address an account signs in from, so the unit of identity has to be the IP, not the pod: a mailbox whose client address changes every deploy collects sign-in risk challenges for no reason. The worker also has no Postgres dependency, so cluster-level service discovery and RBAC buy us nothing.

Node identity

A node's id is assigned at enrolment and persisted on the machine, so re-running the join command re-joins under the same identity and the node keeps its history and its mailboxes.

A machine with several public addresses can instead run one node per address: with WORKER_BIND_IP set and no WORKER_ID, the worker derives UUIDv5(URL_namespace, public_ipv4). Same IP, same worker, with no state at the control plane needed to recover it, which lets each address remain a stable provider-facing login identity.

Worker placement

There is one kind of worker. You install it, it heartbeats, and the control plane decides what runs on it. Workers carry no tier, type, risk pool or egress category, and nothing about a mailbox has to match anything about a machine.

That follows from where the sending identity actually lives. A worker never talks to a recipient's MX. It authenticates to the customer's own mailbox provider (Gmail, Microsoft Graph, or a customer SMTP host) and that provider delivers the message from its own outbound pool. Two consequences shape the whole model:

• The worker's IP is invisible to recipient spam filtering. Google strips the submitting client's IP from outgoing messages and Microsoft dropped X-Originating-IP years ago. So co-locating a spam-prone mailbox next to a healthy one cannot contaminate the healthy one's sending reputation, and segregating workers by customer tier buys nothing.
• The worker's IP is very visible to the mailbox provider. It is what drives sign-in risk challenges, per-IP authentication throttles (454 4.7.0) and per-IP rate limits (421 4.7.28).

So the levers invert: IP stability per mailbox beats IP diversity, and a migration is a cost rather than a win.

Choosing a worker

Placement scores every live worker and takes the best (internal/app/worker/placement.go). Hard constraints are only about whether the work can be done at all: the worker has to be heartbeating and in healthy or watch. Everything else is a preference:

Capacity is a per-worker assigned-mailbox planning target because a small shared VM and a dedicated host do not have the same headroom. It defaults to a conservative 100 and can be set per machine with WARMBLY_WORKER_CAPACITY. Every assigned mailbox counts as one. The value guides placement; it is not a tested maximum or a mailbox-provider sending quota. Provider limits remain attached to the mailbox, tenant, or API project.

Capacity is a target, not a ceiling. Nothing refuses a placement for being over it. Going over costs more than any bonus a candidate can earn, so stickiness alone cannot keep a mailbox on an over-target worker. Once no worker has room, the least-overloaded one wins with region, blast radius and provider crowding all applied. A new worker keeps its full displayed target while node youth slows how quickly it fills. Recipient bounces and complaints remain mailbox and campaign signals; they do not shrink machine capacity or quarantine a worker.

Moving a mailbox

Rotation (internal/app/worker/rotation.go) is deliberately reluctant. Mailbox outcomes never mark a machine unhealthy. The loop responds to liveness, workload, isolated-egress drift, and measurable provider or workspace concentration:

The externally managed health labels describe machine or operator state. Mailbox delivery outcomes do not set them.

A worker that stops answering is a separate path (internal/app/consumer/dead_worker.go), and it waits before acting. The heartbeat key lives three minutes, which a version rollout can outlast while the container is replaced, so a missing key alone is not death: the worker also has to have been absent from the registry for MailboxEvacuationGrace (10 minutes) before its mailboxes are moved. Both signals are required, because during a cache outage every heartbeat key disappears at once while the registry stays current, and moving a mailbox is a real cost rather than a neutral rebalance: it changes the address the mailbox's provider sees and can earn a sign-in challenge.

Isolated egress

Plans that reserve egress bind an organization to a worker (dedicated_worker_assignments). It is a strong placement preference, not a pin: the worker carries no marking, so if it dies the organization's mailboxes place normally instead of stranding, and the rotation loop pulls them onto a replacement once one exists. Mailboxes belonging to other tenants drift off a reserved worker on the same loop.

Node configuration

Nodes never carry hardcoded credentials, and there is nothing to keep in sync by hand. The join endpoint renders /etc/warmbly/node.env from the backend's own environment (nodeEnvKeys in internal/api/handler/fleet_nodes.go), so a node is configured with exactly the infrastructure the control plane uses: event bus, Redis, KMS, blob storage, the internal API token, and the mailbox OAuth clients.

PRIMARY_DB is deliberately excluded. A worker reaches relational data through the internal API and nothing else, and shipping a DSN here would quietly undo that boundary.

Schema: fleet_nodes is the registry every role shares, and workers is the placement extension over it (internal/infrastructure/db/migrations/000141_fleet_nodes.up.sql); 000142_drop_push_provisioning.up.sql removed the push path it replaced.

Auto-update from GitHub releases

The fleet targets one release channel, held in admin_settings under fleet.release:

• stable: latest non-prerelease GitHub Release
• dev: latest release (including prereleases)

internal/app/releases resolves the head of that channel and writes the tag. It updates nothing itself. The heartbeat reply carries desired_version; the node writes it to a file and a systemd timer (warmbly-node-update, installed by the join script) pulls and restarts, so the process being replaced is never the process doing the replacing.

An empty desired_version means "no opinion" and must never be read as "downgrade to nothing": a node that cannot be told what to run keeps running what it has. A per-node pinned_version overrides the fleet target, for canarying or holding a machine back.

The backend is deliberately excluded from this. It is what tells everyone else their version, and a self-update that goes wrong leaves nothing to recover with.

Trigger model is push-driven, not poll-driven: a one-shot check on backend boot, a GitHub webhook (POST /webhooks/github/releases, HMAC-validated) on every release event, and an admin "Check now" button as a manual fallback. All configuration is env-driven (RELEASES_GITHUB_REPO, RELEASES_WORKER_IMAGE_REPO, RELEASES_WEBHOOK_SECRET, RELEASES_ENABLED) so self-hosters can point at their own fork and registry.

See internal/app/releases/service.go.

Worker safety policy

Cold-email throughput is mailbox-first, not worker-first. A worker's safe outbound volume is the sum of its assigned mailbox budgets, not a flat global cap. Defaults in internal/config/constants.go:

• default cold campaign cap per mailbox: 50/day
• default minimum gap per mailbox: 600s
• default warmup start per mailbox: 10/day
• default warmup ceiling per mailbox: 40/day
• default warmup ramp: +1/day

Pool model: warmup traffic is segregated into free and premium pools (the warmup_pools tables in internal/infrastructure/db/migrations/000001_baseline.up.sql). Pool membership is a property of the mailbox, independent of which worker happens to be sending for it.

Warmup content generation is an offline control-plane workload. The autonomous controller runs every six hours, derives a shared-bank target from seven-day send demand, maintains a 200-thread floor, scales up to 5,000 threads, and submits at most 250 replacements per batch within a 1,000-thread daily cap. It deliberately does not split generation by customer mailbox tags, which would create unbounded queues and smaller, more repetitive content cohorts. Scheduled top-ups use gpt-5-mini through durable provider batch jobs rather than synchronous model calls, with a database uniqueness guard that prevents duplicate in-flight scheduled batches when several backend replicas are running. Content with at least 20 sampled sends is automatically archived when it has at least 3 spam placements and a placement rate of 15% or more. The send path atomically selects and increments a least-used active conversation through an indexed query, then falls back to the static reviewed library if no generated content is available. Model availability therefore never gates a warmup send.

Anti-abuse layers

There is no single ML fraud engine. Layered controls instead:

• CAPTCHA (Cloudflare Turnstile) on auth-sensitive flows
• Per-user, per-category API rate limiting (Redis-backed)
• Per-WebSocket join/message/event rate limiting (realtime service)
• Warmup-token verification: single-use, recipient-bound tokens, so warmup mail cannot be replayed or redirected
• Tracking event deduplication (in-memory + persistent)
• Idempotent deliverability event processing
• Worker-side mailbox-sync fair use: the sync governor (internal/app/worker/wmail/governor.go) with priority, live and backfill lanes, deferral instead of dropping, and flood or chronic-overage escalation
• Suppression lists for bounced/complained/unsubscribed recipients
• Admin ban + manual override surface

See the event system reference for the Kafka events, and the codebase's internal/app/consumer/ for the event handlers.

Source anchors

These files are the fastest way to rebuild context:

• README.md
• docs/content/docs/development/deployment-guide.mdx
• cmd/worker/main.go
• internal/app/worker/assignment.go
• internal/app/worker/placement.go
• internal/app/fleetnode/service.go
• internal/app/releases/service.go
• internal/app/cipher/cipher.go
• internal/tasks/email_task.go
• internal/repository/pg_worker.go
• internal/repository/pg_worker_placement.go
• internal/repository/pg_warmup.go