Architecture

Warmbly is split into two planes:

• Control plane: backend API, consumer, tracking, realtime, web. Runs in one region on a container host (Railway in production).
• Execution plane: a fleet of worker processes, one per VPS, spread across many providers and IPs. Each VPS runs the worker as a systemd-supervised Docker container.

The boundary exists for two reasons. First, cold-mail deliverability lives at the IP level, so workers must be spread across distinct machine-level network identities. Second, the control plane owns relational state and the worker fleet should remain disposable.

Services

Data flow

The frontend talks to the backend over REST + JWT, and to the realtime service over WebSocket. Backend writes business state to Postgres. Backend, tracking, and workers publish to the event bus: NATS JetStream with JSON encoding by default, Kafka with Avro and Schema Registry as an opt-in build. The consumer reads the bus and updates Postgres (analytics, suppression, deliverability). Workers subscribe to a topic named for their worker UUID (w.<uuid>) and publish results to jobs.worker-events; see the event system for the full topic map.

Realtime fanout is a separate channel: backend and consumer publish JSON events over Redis pub/sub (or Google Cloud Pub/Sub when PUBSUB_ENABLED=true); the Elixir realtime service subscribes and pushes to connected WebSocket clients.

Object storage: encrypted email bodies (EMSG format) live in the blob store (filesystem by default, S3-compatible opt-in).

Data stores

Encryption model

Warmbly uses envelope encryption end-to-end for sensitive data.

KMS holds the master key. Each organization gets a 32-byte data encryption key (DEK), generated by KMS and stored encrypted in the organization_encrypted_keys Postgres table keyed by organization ID (workers reach it over the backend's internal API rather than touching Postgres). The DEK is decrypted only at the moment of use; cached in Redis with a TTL to amortize cost.

Application-layer secrets are sealed with AES-256-GCM under the DEK and base64-encoded. This applies to:

• email account credentials (IMAP/SMTP passwords, OAuth tokens)
• email body content stored in S3 (EMSG format)
• worker SSH private keys (since admins drive workers over SSH)
• AWS credential rows and worker-profile secrets (Kafka SASL passwords, Schema Registry secrets, Redis URLs) used to configure remote workers

Worker-related secrets are encrypted under a platform DEK (key ID = uuid.Nil): the same envelope as organization secrets, the same trust boundary, a different identity. See internal/app/cipher/ and internal/app/worker_orchestrator/orchestrator.go.

Worker model

Workers are added and managed from the admin dashboard. The flow:

1. Admin fills out host/port/user. Backend generates an ed25519 keypair, encrypts the private key, stores the row in pending state.
2. Admin pastes the generated public key into the VPS's ~/.ssh/authorized_keys.
3. Admin clicks Test. Backend opens an SSH session and runs true. First success pins the host SHA256 fingerprint (trust-on-first-use).
4. Admin clicks Install. Backend uploads scripts/install-worker.sh and a per-worker env file, runs the installer, which installs Docker if missing, generates a deterministic UUID from the VPS's public IPv4 (UUIDv5, URL namespace), writes a systemd unit, and starts the worker container with --hostname <uuid>.
5. The worker reads its identity from os.Hostname(), subscribes to the event bus topic named for that UUID, and heartbeats to the backend's internal API every 90 seconds.

From then on, every lifecycle operation (restart, update image, uninstall, rotate keys, system updates, reboot, tail logs, fetch live status) happens via SSH from the dashboard.

Why per-VPS instead of a Kubernetes DaemonSet

A k8s DaemonSet was the previous shape. It was removed because k8s nodes typically NAT all pods through a small set of egress IPs, defeating the IP-diversity goal. Pods churn but IPs accumulate reputation, so the unit of identity needs to be the IP, not the pod. The worker also has no Postgres dependency, so cluster-level service discovery and RBAC buy us nothing.

Identity from IP

Worker UUID is UUIDv5(URL_namespace, public_ipv4). Properties:

• same IP → same worker (reputation persists across reinstalls)
• new IP → new worker (fresh identity, no inherited history)
• deterministic, no state needed at the control plane to recover it

Credentials and profiles

Workers never carry hardcoded credentials. Two reusable entities in the admin dashboard:

• AWS credentials: a named keypair; the secret access key is stored as ciphertext from the cipher service.
• Worker profile: a named bundle of event bus settings (NATS URL or Kafka bootstrap + SASL + Schema Registry), Redis URL, backend internal API URL and token, worker image, and release channel, with an optional foreign key to one AWS credentials row.

Workers reference one profile. When the admin edits the profile, the backend marks any assigned worker whose config_applied_at is older than profile.updated_at as having stale config. The dashboard shows a warning; one click rewrites /etc/warmbly/worker.env over SSH and restarts the worker.

Schema: the workers, worker_profiles, and aws_credentials tables in internal/infrastructure/db/migrations/000001_baseline.up.sql.

Auto-update from GitHub releases

Each profile picks a release channel:

• pinned: admin sets the image tag manually
• stable: latest non-prerelease GitHub Release
• dev: latest release (including prereleases)

Trigger model is push-driven, not poll-driven:

• One-shot check on backend boot, populates the dashboard.
• GitHub webhook (POST /webhooks/github/releases, HMAC-validated) on every release event.
• Admin "Check now" button as a manual fallback.

When auto_update is on and a new tag is resolved, the backend records the new image on the profile, then rolls each assigned worker by SSHing in, re-running the installer with --update --image <new>, which regenerates the systemd unit, pulls the image, and restarts. Workers' running version is recorded in workers.image_version so the dashboard can show a v1.2.3 → v1.2.4 diff.

All configuration is env-driven (RELEASES_GITHUB_REPO, RELEASES_WORKER_IMAGE_REPO, RELEASES_WEBHOOK_SECRET, RELEASES_ENABLED) so self-hosters can point at their own fork and registry.

See internal/app/releases/service.go.

Worker safety policy

Cold-email throughput is mailbox-first, not worker-first. A worker's safe outbound volume is the sum of its assigned mailbox budgets, not a flat global cap. Defaults in internal/config/constants.go:

• default cold campaign cap per mailbox: 50/day
• default minimum gap per mailbox: 600s
• default warmup start per mailbox: 10/day
• default warmup ceiling per mailbox: 40/day
• default warmup ramp: +1/day

Pool model: warmup traffic is segregated into free and premium pools (the warmup_pools tables in internal/infrastructure/db/migrations/000001_baseline.up.sql). Dedicated-worker customers still pick a pool explicitly; tier and pool are orthogonal.

Warmup content generation is an offline control-plane workload. The autonomous controller runs every six hours, derives a shared-bank target from seven-day send demand, maintains a 200-thread floor, scales up to 5,000 threads, and submits at most 250 replacements per batch within a 1,000-thread daily cap. It deliberately does not split generation by customer mailbox tags, which would create unbounded queues and smaller, more repetitive content cohorts. Scheduled top-ups use gpt-5-mini through durable provider batch jobs rather than synchronous model calls, with a database uniqueness guard that prevents duplicate in-flight scheduled batches when several backend replicas are running. Content with at least 20 sampled sends is automatically archived when it has at least 3 spam placements and a placement rate of 15% or more. The send path atomically selects and increments a least-used active conversation through an indexed query, then falls back to the static reviewed library if no generated content is available. Model availability therefore never gates a warmup send.

Anti-abuse layers

There is no single ML fraud engine. Layered controls instead:

• CAPTCHA (Cloudflare Turnstile) on auth-sensitive flows
• Per-user, per-category API rate limiting (Redis-backed)
• Per-WebSocket join/message/event rate limiting (realtime service)
• Warmup-token verification with invalid-attempt counting and auto-blocking
• Tracking event deduplication (in-memory + persistent)
• Idempotent deliverability event processing
• Worker-side mailbox-sync rate limiting (internal/app/worker/wmail/ratelimit.go)
• Suppression lists for bounced/complained/unsubscribed recipients
• Admin ban + manual override surface

See the event system reference for the Kafka events, and the codebase's internal/app/consumer/ for the event handlers.

Source anchors

These files are the fastest way to rebuild context:

• README.md
• docs/content/docs/development/deployment-guide.mdx
• cmd/worker/main.go
• internal/app/worker/assignment.go
• internal/app/worker_orchestrator/orchestrator.go
• internal/app/releases/service.go
• internal/app/cipher/cipher.go
• internal/tasks/email_task.go
• internal/repository/pg_worker.go
• internal/repository/pg_worker_ssh.go
• internal/repository/pg_credentials.go
• internal/repository/pg_warmup.go