Deployment

Production runbook for the backend - build it, migrate the stores, run each process, and harden for production traffic. The dashboard has its own build and deploy; see Dashboard.

Before deploying: meet all Requirements and set every variable in Configuration. The make commands below run from the backend repo root.

1. Build

You can ship the backend as binaries (systemd, bare processes) or container images (Kubernetes, Compose). Pick one.

Option A: binaries

make build

make build compiles the following into bin/. bin/pug is the unified CLI; every other binary is a single-purpose build of one process: an alternative to the matching pug sub-command for deployment tools that prefer one binary per unit.

Run the binaries from a directory that holds schema/. The workers and nats migrate read schema/nats/*.yaml, and the Postgres and ClickHouse migrate runners read schema/{postgres,clickhouse}/migrations, all relative to the working directory. Ship the repo’s schema/ folder with the binaries and start each unit from the folder above it (systemd WorkingDirectory=). The NATS files can also be pointed at directly with NATS_STREAMS_CONFIG and NATS_CONSUMERS_CONFIG; the migration folders can’t.

There is no make migrate or make dev target. Migrations run via the pug CLI or the standalone migrate binaries (below); local all-in-one runs via the pug dev command. See the CLI reference.

Option B: container images

Each image’s entrypoint is one single-purpose binary, so you run the container with no arguments. They run as distroless nonroot, with /app as the working directory and whatever schema/ files the role needs already inside.

Published images

Every release publishes one image per role to the GitHub Container Registry, for linux/amd64 and linux/arm64:

docker pull ghcr.io/pug-sh/pug-server:0.0.36

Each image is tagged with its full version (0.0.36), its minor line (0.0) and latest. Pin the full version in production, and roll every role to the same one.

Build your own

The repo ships one parameterized Dockerfile that builds any of the images above. Two build args select the role, from the CMD and Target columns:

• --build-arg CMD=<path under ./cmd>: which main to compile.
• --target <app|worker|migrate>: the runtime shape (what, besides the binary, lands in the image).

The three targets exist because the roles need different files at runtime:

# API server
docker build --build-arg CMD=server            --target app     -t pug-server .

# a worker (swap the CMD path for any worker in the table)
docker build --build-arg CMD=workers/events    --target worker  -t pug-worker-events .

# a migration runner
docker build --build-arg CMD=migrate/postgres  --target migrate -t pug-migrate-postgres .

A plain docker build produces an image for the host’s architecture. The Dockerfile cross-compiles, so docker buildx build --platform linux/amd64,linux/arm64 ... builds both without QEMU.

2. Migrations

Run migrations once against each store before the first boot, and again on every upgrade. They are idempotent - re-running them on an up-to-date store is safe.

# unified CLI
./bin/pug postgres migrate
./bin/pug nats migrate
./bin/pug clickhouse migrate

# or the standalone binaries / migrate images
./bin/pug-migrate-postgres
./bin/pug-migrate-nats
./bin/pug-migrate-clickhouse

All migration commands read connection details from the environment (DATABASE_URL, NATS_URL, CLICKHOUSE_URL). postgres migrate and clickhouse migrate apply every pending migration.

To roll back, use the unified CLI: --direction down --num N undoes N steps. Always pass --num - --direction down on its own rolls back every migration, dropping every table. The standalone binaries and migrate images only migrate up and take no flags.

nats migrate creates or updates the JetStream streams and consumers to match schema/nats/, with the replica count from NATS_STREAM_REPLICAS (see NATS JetStream HA). It never deletes a stream.

3. Run

Production topology: one process per role, each as its own replica. pug dev runs every component in a single process with no independent scaling or restart boundaries - acceptable for a small low-volume box (below), not for production traffic at any real volume.

API server

./bin/pug server        # or ./bin/pug-server, or the pug-server image

Listens on :3000 (override with PUG_SERVER_PORT). Place a TLS-terminating load balancer in front - see Production hardening.

Workers

There are 6 worker processes in a normal deployment (the demo worker is only for public demos). Run each as its own replica, using whichever form fits your tooling:

Scale the events worker horizontally for high ingest - run multiple replicas (see Horizontal scaling):

./bin/pug worker events        # or ./bin/pug-worker-events

Scheduled jobs

Two jobs run on a schedule. Each runs one pass and exits - non-zero when the pass failed - so a scheduler runs it: a Kubernetes CronJob, a systemd timer, or cron. Nothing starts them for you, pug dev included.

Each pass takes a Postgres advisory lock, so a run that overlaps the previous one exits cleanly instead of doing the work twice. On Kubernetes, set concurrencyPolicy: Forbid and an activeDeadlineSeconds shorter than the interval - Pug Cloud uses 270 seconds for purge and 3000 for usage. Give the jobs the server’s environment; cron usage also reads PUG_USAGE_RESCAN_DAYS.

The two billing jobs, pug-cron-billing-reconcile and pug-cron-billing-meter, belong to Pug Cloud’s billing, which is off by default. A self-hosted instance doesn’t schedule them.

Single node

Running a single small node? pug dev (alias pug start) runs the server and all workers in one process. It’s built for local dev, but it’s a legitimate way to stand up a low-volume single-box instance - you just give up independent scaling and restarts. Schedule the jobs alongside it. See the CLI reference.

4. Production hardening

TLS termination

pug server listens on plain HTTP (:3000). Terminate TLS at a load balancer (nginx, Caddy, AWS ALB, Cloudflare) and proxy to port 3000. Do not expose port 3000 directly to the internet - the SDKs and dashboard send credentials (x-api-key, Authorization JWT), so TLS is required.

Secrets management

Do not put secrets in unencrypted files. Inject the following via your secrets manager (Vault, AWS Secrets Manager, Kubernetes Secrets, etc.):

Ending sessions

A dashboard session is a refresh token stored in Postgres, on a 90-day sliding expiry; the JWT it mints lasts 24 hours. When a JWT is rejected, the dashboard quietly trades the refresh token for a new pair, so rotating PUG_JWT_SECRET_KEY only kills the JWTs already issued. To sign everyone out, revoke the refresh tokens, then rotate the key so the outstanding JWTs die too:

update refresh_tokens set revoked_at = now() where revoked_at is null;

CORS origins

Set PUG_CORS_ORIGINS to the exact origin(s) your dashboard runs on (e.g. https://app.example.com), comma-separated with no spaces. Unset, it defaults to *, which lets any website call the dashboard API from a visitor’s browser; the server logs a warning when it starts that way. The dashboard keeps its token in an Authorization header, not a cookie, so * doesn’t break sign-in - it only removes a layer of defence. The SDK ingest services accept any origin regardless.

NATS JetStream HA

Run a 3-node NATS cluster with JetStream on each node - a single node is a single point of failure for all ingestion. Start each with --jetstream --store_dir /data/nats, give the cluster a name and configure its routes. Then set NATS_STREAM_REPLICAS=3 and re-run pug nats migrate, so one node’s loss doesn’t lose data.

pug nats migrate refuses a replica count above 1 against a server that reports no cluster name, since a standalone server only loses the streams at its next restart. It also owns the stream settings: one changed by hand (nats stream edit) is reset on its next run.

ClickHouse HA and storage

A single ClickHouse node is fine for low-to-medium volumes. For higher volumes or fault tolerance:

• Replication: Pug’s migrations create plain MergeTree-family tables with no ON CLUSTER, so a self-managed replicated cluster isn’t supported yet. For fault tolerance today, use ClickHouse Cloud, which replicates every table, or back up the node’s storage.
• Tiered storage: an S3-compatible object store as the “cold” tier for high-retention deployments.
• ulimits: ensure nofile >= 262144 on all ClickHouse nodes (required).

Horizontal scaling

The events worker is the primary bottleneck at high ingest rates. Its replicas share one durable JetStream consumer, so each one you add takes a share of the stream. Other workers process lower-volume streams; start with one replica each and scale as needed.

Monitoring

Point OTEL_EXPORTER_OTLP_ENDPOINT at your collector’s OTLP/gRPC endpoint (usually port 4317) to export traces, metrics, and logs. Export is plaintext unless you set OTEL_EXPORTER_OTLP_INSECURE=false, which a TLS endpoint needs. OTLP over HTTP isn’t supported, so a backend that only accepts HTTP needs an OpenTelemetry Collector in between. With no endpoint set, Pug logs to stdout as text. See Telemetry in Configuration.

Health checks

The server and every worker except demo expose Kubernetes-style probes. Wire your orchestrator’s liveness/readiness probes (or load-balancer health checks) to these:

The worker probe address is PUG_WORKER_HEALTH_ADDR (default :8090; set to off to disable). A worker opens it only once its first JetStream consumer exists, so a probe sent earlier gets a refused connection rather than a 503 - give the probes a startup allowance (initialDelaySeconds, or a startupProbe). When several worker binaries share a host, give each a distinct port to avoid a bind clash. pug dev disables worker probes (it’s for local use). The demo worker has no probes.

Backing services keep their own health endpoints - probe those for infrastructure liveness:

Further reading

• Dashboard: build and serve the UI
• Connect your SDKs: point your apps at this instance
• CLI reference: every pug command and flag
• Configuration: all environment variables