Observability — Prometheus + Grafana (Staging)

Single-node Prometheus + Grafana for the staging environment. Configuration is versioned in the repo under dev-stack/staging/observability/ and deployed with Docker Compose — never hand-clicked on the host (ZAF-675).

The rule behind the split: operational/telemetry data → Prometheus; product data → TimescaleDB. There is no third time-series store (no InfluxDB). The full decision and sizing analysis live on ZAF-672 (plan, rev 2); the instrumentation that exposes /metrics on each Rust service is ZAF-674 (lo-metrics).

Tiering. Like the message brokers (ZAF-655), staging runs a single node. The multi-node / HA build is production on k3s (ZAF-61) and is tracked separately — this stack is not k3s-gated.

What ships

dev-stack/staging/
├── observability.docker-compose.yml        # Prometheus + Alertmanager + Grafana
├── .env.observability.example              # Grafana admin credentials template
└── observability/
    ├── prometheus/
    │   ├── prometheus.yml                   # scrape config (30 s interval) + alerting
    │   ├── rules/recording_rules.yml        # 5-min + 1-h rollups
    │   ├── rules/alerting_rules.yml          # alert rules (ZAF-687 + ZAF-676)
    │   └── targets/crawler-twitch.json      # file_sd for the crawler host
    ├── alertmanager/
    │   ├── alertmanager.yml                  # severity routing (page | ticket)
    │   └── secrets/                          # webhook URLs (git-ignored, host-provisioned)
    └── grafana/
        ├── provisioning/
        │   ├── datasources/datasource.yml   # Prometheus datasource
        │   └── dashboards/provider.yml      # dashboard provider ("Lumio" folder)
        └── dashboards/                       # versioned dashboard JSON

Retention (why the numbers are acceptance criteria)

The founder's objection on ZAF-672 was long-term query performance. Prometheus keeps only 15 days by default; anything not retained in the first months is gone forever. So the long window is present in the first config, not a later phase:

• --storage.tsdb.retention.time=400d — ≥ 400 days.
• --storage.tsdb.retention.size=90GB — disk-safety guard below the ≥ 100 GB volume. With the sizing estimate below, time is the binding limit.
• The Prometheus data volume must be backed by ≥ 100 GB of persistent disk.

Sizing (plan §4a): ~10 Rust services, ~15 000 active series, 30-s scrape, ~2 B/sample → ~86 MB/day ≈ 31 GB/year; 13 months ≈ 35 GB. The 400-day window fits comfortably inside 100 GB.

To pin the TSDB to a dedicated disk, back the staging-prometheus-data named volume with a driver_opts bind (documented inline in the compose file) and chown 65534:65534 the target directory.

Scrape config

• Interval: 30 s (not 15 s) — trend/alarm-adequate and half the long-term storage cost.
• Targets are each Rust service's internal /metrics (served by lo-metrics on the internal health server). Every series already carries the constant labels service and env, so the scrape does not re-label service; it adds only job (= service name) and instance.
• Cardinality rule (hard): the /metrics exposition never carries account_id / channel_id / user_id / session_id / raw URL paths. Per-channel stream history stays in channel_history_stats, never Prometheus.

Internal metrics ports — reconciled against the actual ZAF-674 wiring (ZAF-679). The port comes from config/default.toml where a bot overrides its src/config.rs struct default, so the two disagree; the TOML value wins.

Service	Target	Source of port	Status in staging
api	staging-api:9100	apps/api/src/config.rs MetricsConfig default 127.0.0.1:9100; served by lo_metrics::serve	needs LUMIO__METRICS__HOST=0.0.0.0 in .env.api (loopback is unreachable from the Prometheus container)
twitch-bot	staging-twitch-bot:3001	apps/twitch-bot/config/default.toml [health] port (overrides 9090)	up when the bot runs — /metrics on the health server (src/health.rs)
youtube-bot	staging-youtube-bot:3002	config/default.toml [health] port	"
kick-bot	staging-kick-bot:3003	config/default.toml [health] port	"
trovo-bot	staging-trovo-bot:3004	config/default.toml [health] port	"
discord-bot	staging-discord-bot:3005	config/default.toml [health] port	"
crawler-twitch	file_sd (empty)	health server :8080 (ZAF-595 §7)	operator-maintained, empty in staging — see below

The scrape set is exactly the six jobs above (plus Prometheus itself) — the services that actually run in staging. There are no static jobs for services that staging does not deploy, so there are no dead up == 0 targets by construction.

Instrumented but deliberately NOT scraped (ZAF-684). Several Rust services carry lo-metrics /metrics yet get no staging job on purpose. "Instrumented" is not "deployed": a static job pointed at a container staging never starts would be a permanent up == 0 target — precisely the silent failure the acceptance criterion forbids. Staging runs only api + admin + id + web + the five bots + infra + observability; nothing else.

• bot-module-worker, automation-worker — no staging compose exists. Add a job only when one lands under dev-stack/staging/.
• innertube-proxy — runs deliberately outside the cluster on a trusted-IP host (AGENTS.md; developer-guide/innertube-proxy.md), so it is never on the lumio-staging network.
• When any of these later gets a staging compose, the scrape port is already fixed at :9100 — its own internal metrics server, same layout as apps/api — assigned to the Rust backend in ZAF-683 (which moves /metrics off the 3100/8090/8091 payload port). Nothing needs coordinating first.
• crawler-twitch is not built by release-images.yml (absent from RUST_APPS) and has no staging compose: by design it runs on its own trusted-IP host (ZAF-595 §4), off the lumio-staging network. Its file_sd file (targets/crawler-twitch.json) is therefore operator-maintained and ships empty — an operator adds the real host:port out-of-band. An empty file_sd yields no target and no up series — a deliberate, documented absence, not a dead placeholder.

Watch the job:up:min recording rule (group target_liveness) or query min by (job) (up) to see, at a glance, whether any job's target is down — before anyone builds an alert on the metrics it should be producing (ZAF-676).

Recording rules (5-min + 1-h rollups)

A 400-day query over raw 30-s samples is slow. The rules in recording_rules.yml precompute the rates/quantiles the dashboards use so a year-scale query runs over a handful of precomputed series:

• target_liveness — job:up:min = min by (job) (up), one series per scrape job, so a dead target is visible before anyone alerts on it (ZAF-676).
• http_rollup_5m / http_rollup_1h — request rate, 5xx error ratio, and duration p50/p90/p99 per (service, env), plus per-endpoint p99.
• crawler_rollup_5m / crawler_rollup_1h — reputation (429 / LOGIN_REQUIRED / ban), reconnects, writer throughput/errors, drop-policy activations.

Raw samples are untouched — rollups are additional series, not a downsample-and-delete. Retention (400 d) applies to raw and rollup alike. The 1-h group evaluates hourly (not every 30 s), which is what makes the long-window series cheap.

Alerting (Prometheus rules + Alertmanager)

rules/alerting_rules.yml arms the ZAF-595 §7 alarms (ZAF-687) plus the WebSocket / chat / bot degradation alarms (ZAF-676). Prometheus evaluates them and pushes firing alerts to Alertmanager (staging-alertmanager:9093), which groups, deduplicates and routes them by a severity label:

severity	Meaning	Route
page	Reputation-fatal or data-losing — the crawler is banned / login-gated / blind / not writing. Wakes a human.	pager receiver, re-notify hourly
ticket	Degraded-but-working — coverage / latency / throttle pressure. Looked at next business hours.	ticket receiver, re-notify 4-hourly

Every alert also carries a component label (crawler | http | target | websocket | chat | bot). A page inhibits the lower-severity ticket for the same incident.

Rule groups (31 alerts):

• target_liveness_alerts — up == 0 per job (API + crawler paged, other services ticket) and ServiceCrashLooping (changes(process_start_time_seconds)). A dead target is the most expensive silent failure (ZAF-676).
• crawler_reputation_alerts (PAGED) — CrawlerLoginRequired, CrawlerBanEvents, CrawlerHelix429Sustained, CrawlerHelixAuthFailures. These are the §7 first-class reputation alarms.
• crawler_connection_alerts — no IRC connection, joined-but-silent (a wedged read path), reconnect storm, JOIN-throttle saturation, drop-policy activation.
• crawler_pipeline_alerts — writer errors (paged), write-batch latency, writer backlog.
• crawler_coverage_alerts — discovery staleness (paged), discovery errors, empty roster.
• http_alerts — API 5xx error ratio (paged) + p99 latency, and a ticket-level 5xx alarm for the other services. Reuses the recording rules so the alert and the dashboard read the same precomputed series.
• websocket_leak_alerts (ZAF-676 §3) — WsSessionLeakSuspected (live sessions accumulate without draining — the ZAF-396 wedged-session leak), WsIdleReapStorm (idle-reaper force-closing wedged sessions), WsBroadcastLagging (slow-consumer backpressure). Uses the ws_* series from ZAF-680.
• chat_buffer_alerts (ZAF-676 §4) — ChatBufferBacklog (persist buffer not draining), ChatFlushSlow (p99 flush latency), ChatMessagesDropped (paged — dead_letter / lost = chat data loss). Uses chat_buffer_depth / chat_flush_duration_seconds / chat_messages_dropped_total from ZAF-680.
• bot_reconnect_alerts (ZAF-676 §5) — per-platform BotReconnectLoop (outer connect-loop flapping), BotSyncStale (up but no successful channel-sync), BotSyncFailing (consecutive sync errors). Uses bot_reconnects_total{platform} and the bot_sync_*{platform} gauges from ZAF-680.

Each of the three ZAF-676 groups carries an explicit runbook annotation (what the alert means + the first handgriff, ZAF-676 §7); the older groups fold the first action into description.

Tuned quiet-under-normal-load (the ZAF-635 go-live entry criterion): the reputation counters sit at 0 when healthy, so a sustained non-zero rate (never a single sample) is the signal, and for: windows outlast normal transients.

What is needed before delivery is real

The metric surface these rules reference is now on next: ZAF-674 landed lo-metrics + the crawler_* / http_* rework (the IRC NOTICE classifier that increments crawler_login_required_total / crawler_ban_events_total), and ZAF-680 added the ws_* / chat_* / bot_* domain series the ZAF-676 groups alert on. So the alerts have a real signal once the services deploy to staging.

Two steps remain, both gated on the staging-deploy boundary (host access + DevOps-Leader go/no-go — the ZAF-675 deploy gate):

1. Deploy the stack + instrumented services to staging. The rules go live when Prometheus scrapes services built from a next that includes ZAF-674 / ZAF-680.
2. Provision the delivery secret (ZAF-676 §6). The pager / ticket receivers read their webhook URL from a file (url_file), not an env var (Alertmanager does not expand env vars) and not a literal (no secret in git). Drop pager_webhook_url / ticket_webhook_url into alertmanager/secrets/ on the host from the board-gated secret path, then POST /-/reload — a low-threshold Slack/Discord/Mattermost webhook is enough for staging. See alertmanager/secrets/README.md. Until the files exist the config still loads and alerts are visible in the UI (127.0.0.1:9093); they simply are not delivered off-box.

Deliberately NOT alarmed yet (no emitting metric)

An alert on a series nothing emits is a permanently-silent lie, so these are tracked here instead of armed:

• Kafka consumer-lag — the crawl-job / result bus is in-memory single-node today (apps/crawler-twitch/src/bus.rs); the Kafka/RabbitMQ impls are gated (ZAF-655 / ZAF-392). No consumer group, no lag series. Add the lag alarm when the Kafka bus lands.
• Opt-out enforcer errors (crawler_optout_refresh_errors_total / _erasure_errors_total) — the enforcer and those metrics are not on next yet (ZAF-635 / ZAF-686). Add the alarm in the PR that emits the counters.
• Write-amplification ratio (rows_upserted vs rows_skipped_noop) — crawler_rows_skipped_noop_total is registered at 0; its increment site ships with the writer's DO UPDATE … WHERE no-op guard, which is not wired yet. Until then write health is covered by CrawlerWriterErrors + CrawlerWriteLatencyHigh.

Dashboards

Provisioned into the Lumio folder in Grafana:

• Lumio — API — request rate, error ratio, latency p50/p90/p99, per-endpoint p99, process memory/CPU, scrape-target health.
• Lumio — Crawler reputation & pipeline — 429/LOGIN_REQUIRED/ban rates, reconnects, roster size, discovery freshness, writer throughput/latency, pipeline backlog + drop policy.
• Lumio — Realtime (WS / Chat / Bots) — WS active sessions, chat buffer lag, bot reconnects. These panels expect series named ws_active_sessions, chat_buffer_pending, chat_flush_lag_seconds, and bot_reconnects_total (label platform) from ZAF-674; they show no data until each service emits them.

Runbook

Both UIs bind to 127.0.0.1 only — reach them over an SSH tunnel; never expose them publicly (metric names + route lists are recon material).

Deploy

cd dev-stack/staging
cp .env.observability.example .env.observability   # set a real GF_SECURITY_ADMIN_PASSWORD

# The lumio-staging network is created by the infra stack:
docker compose -f infra.docker-compose.yml up -d   # if not already running

docker compose -f observability.docker-compose.yml up -d

Verify:

curl -s http://127.0.0.1:9090/-/healthy                 # Prometheus
curl -s http://127.0.0.1:9090/api/v1/rules | jq '.data.groups[].name'
curl -s http://127.0.0.1:9090/api/v1/alerts | jq '.data.alerts[] | {alertname:.labels.alertname, state}'
curl -s http://127.0.0.1:9090/api/v1/alertmanagers | jq '.data.activeAlertmanagers'
curl -s http://127.0.0.1:9090/api/v1/targets | jq '.data.activeTargets[] | {job:.labels.job, health}'
curl -s http://127.0.0.1:9093/-/healthy                 # Alertmanager
curl -s http://127.0.0.1:3009/api/health                # Grafana

Grafana UI: http://127.0.0.1:3009 (via tunnel), login with the admin credentials from .env.observability.

Reload config / rules (no restart)

Prometheus runs with --web.enable-lifecycle:

docker compose -f observability.docker-compose.yml restart prometheus   # simplest
# or hot-reload:
curl -s -X POST http://127.0.0.1:9090/-/reload

Grafana re-reads provisioned datasources/dashboards on restart (dashboards also poll every 30 s).

Rollback

Config is versioned, so rollback is a git revert of the config change plus a recreate:

git revert <commit>            # or check out the previous config
docker compose -f observability.docker-compose.yml up -d --force-recreate

The TSDB and Grafana state persist in named volumes, so a config rollback does not lose collected metrics. To wipe everything (destructive — loses history):

docker compose -f observability.docker-compose.yml down -v

Validate config before deploy

# Prometheus config + recording + alerting rules:
docker run --rm --entrypoint promtool \
  -v "$PWD/observability/prometheus:/etc/prometheus:ro" \
  prom/prometheus:v3.1.0 check config /etc/prometheus/prometheus.yml

# Alertmanager routing config:
docker run --rm --entrypoint amtool \
  -v "$PWD/observability/alertmanager:/etc/alertmanager:ro" \
  prom/alertmanager:v0.28.0 check-config /etc/alertmanager/alertmanager.yml

Explicitly out of scope

• No InfluxDB / second time-series store. If series counts later explode, the path is a remote_write block to VictoriaMetrics — no code, no metric rename.
• No OpenTelemetry / tracing collector.
• No product/customer data in Prometheus (per-channel cardinality is forbidden).
• No delivery secret in the repo — the Alertmanager receivers reference a webhook URL by file (url_file); the actual Slack/Discord/PagerDuty URL is provisioned on the host via the board-gated secret path, never committed (see Alerting → What is needed before delivery is real).
• No production pager (persistent volume, on-call rotation, PagerDuty/Opsgenie) — that is stage 3, gated on ZAF-61 / k3s.