Operational Metrics (Prometheus)

Every Rust service exposes a Prometheus /metrics endpoint backed by the shared crates/lo-metrics crate. This is stage 1 of the observability plan (ZAF-672): the instrumentation and the exposition format only. The production scraping, dashboards, alerting and retention backend is stage 2 and is intentionally not covered here — but a throwaway local Prometheus + Grafana for developers is (see Viewing metrics locally).

Metrics are operational telemetry, distinct from the product time-series in TimescaleDB (channel_history*) and from Sentry (which catches crashes, not degradation). They emit no audit event — the audit log is a security record, not a telemetry stream.

What each service exposes

lo-metrics registers the same baseline in every service, plus any service-specific series:

Metric	Type	Labels	Meaning
http_requests_total	counter	method, endpoint, status	HTTP requests handled
http_request_duration_seconds	histogram	method, endpoint	Request latency (buckets 5 ms → 10 s)
process_cpu_seconds_total	counter	—	Process CPU time (Linux)
process_resident_memory_bytes	gauge	—	Resident memory (Linux)
process_open_fds / process_max_fds	gauge	—	Open / max file descriptors (Linux)
process_start_time_seconds	gauge	—	Process start time (Linux)

Every series additionally carries two constant labels applied by the shared registry:

• service — the logical service name (api, twitch-bot, crawler-twitch, …).
• env — the deployment environment (staging, prod, …), sourced from the service's Sentry environment.

process_* metrics come from the prometheus process collector and are Linux-only (every Lumio Rust build target is Linux).

The API's live gauges

apps/api registers a pull-time collector that reads the running AppState on every scrape and exposes it as these gauges. Being live values, they are read at scrape time from a prometheus::core::Collector, not eagerly updated; every read is lock-free (atomic WsServer counters and sqlx pool counters), so a scrape never contends with the WebSocket broadcast hot path.

Metric	Type	Labels	Meaning
lumio_build_info	gauge	version	Constant 1; running build version in the label
lumio_uptime_seconds	gauge	—	Seconds since the API process started
lumio_websocket_sessions	gauge	—	Active WebSocket sessions
lumio_websocket_channels	gauge	—	Active WebSocket broadcast channels
lumio_db_pool_connections	gauge	pool	Connections owned by the pool (postgres, timescaledb)
lumio_db_pool_idle	gauge	pool	Idle connections available for checkout

The WebSocket, version and uptime values are also served as JSON on GET /v1/health. These were ZAF-646's operational gauges; ZAF-674 retired their hand-rolled public /metrics, and this collector re-registered them onto the shared registry (ZAF-685) so they render on the internal endpoint again.

The crawler's series

apps/crawler-twitch registers ~30 crawler_* counters/gauges into the same registry (discovery/roster coverage, Helix + reputation alarms, IRC/JOIN health, bus + writer pipeline, drop policy). These names are an arming precondition for live global ingest (ZAF-595 §7) and are stable — a rename silently breaks that promise, so it must be called out explicitly in the PR that makes it. Before ZAF-674 these were hand-rolled AtomicU64 counters rendered to text by hand; they now live in the shared registry with no name change.

The API's domain series (ZAF-680)

apps/api registers domain series into the same registry so ZAF-676 can alarm on signals the HTTP status class structurally cannot see. All labels are bounded enumerations — never an id.

Metric	Type	Labels	Meaning
ws_sessions_active	gauge	—	Live WebSocket sessions (was JSON-only on /v1/health, ZAF-646)
ws_channels_active	gauge	—	Live broadcast channels with ≥1 subscriber
ws_sessions_opened_total	counter	—	Sessions opened since start (churn numerator)
ws_sessions_closed_total	counter	—	Sessions closed since start. ws_sessions_active high while this rate → 0 is the ZAF-396 leak ("session steht seit 72 h")
ws_outbound_dropped_total	counter	—	Forwarders dropped because the session outbound queue was full (the wedged/slow-consumer path)
ws_broadcast_lagged_total	counter	—	Subscribers that fell behind the broadcast ring and lost messages
ws_idle_reaped_total	counter	—	Sessions torn down by the idle (client-timeout) reaper
graphql_requests_total	counter	op_type (query/mutation), outcome (ok/error)	GraphQL outcome — GraphQL answers HTTP 200 on error, so status can't see a GraphQL failure
db_pool_connections	gauge	pool (primary/tsdb), state (total/idle)	Live sqlx pool connections, sampled every 15 s
db_pool_max_connections	gauge	pool	Configured max_connections ceiling per pool

The WebSocket counters were the quietest paths in the layer before ZAF-680: ws_outbound_dropped_total had neither log nor metric, ws_broadcast_lagged_total had only a warn!, and ws_idle_reaped_total only an info!.

Chat-domain series (ZAF-680 §2)

crates/lo-chat registers a ChatMetrics handle (defined once, shared by the apps/api chat-flush worker and the ProfileService) so chat-pipeline lag, flush health and ProfileService behaviour become scrapeable. They feed ZAF-676 §4. All labels are bounded — never an account/user/channel id.

Metric	Type	Labels	Meaning
chat_buffer_depth	gauge	—	Messages buffered in Redis at the last flush cycle, summed across accounts (the lag signal ChatBuffer::push's warn! never exported)
chat_pending_accounts	gauge	—	Accounts with a non-empty buffer at the last flush cycle
chat_flush_duration_seconds	histogram	—	Per-account TimescaleDB batch-insert (flush) duration
chat_messages_flushed_total	counter	—	Messages successfully inserted into TimescaleDB
chat_messages_dropped_total	counter	reason (dead_letter/lost)	Messages that never reached TimescaleDB — parked after the retry ceiling (dead_letter, recoverable) or lost when insert + dead-letter + re-queue all failed (lost)
profile_service_cache_total	counter	result (hit/miss)	ProfileService Redis cache lookups (was a debug! that never fires normally)
profile_service_circuit_trips_total	counter	platform	ProfileService circuit-breaker trips. A counter, not a live gauge: the breaker opens on an explicit event but closes silently via Redis TTL expiry, so only the trip is observable without per-account state

Platform-bot series (ZAF-680 §3)

Each platform bot registers the lo-bot-commands metric handles for the constructs it actually has — a bot never registers a series it cannot drive (no 0-by-construction line). They feed ZAF-676 §5. The single bounded platform label is the one dimension the cardinality rule allows.

Metric	Type	Labels	Bots	Meaning
bot_reconnects_total	counter	platform	twitch, kick, trovo (outer retry loop); discord (gateway Resume)	Reconnect / leader-handover churn. youtube-bot runs Bot::run() once (no restart loop) — its gRPC re-opens are by-design ~10 s cycling, not churn — so it registers no reconnect series
bot_leader	gauge (0/1)	platform	twitch, youtube, kick, trovo	Whether this replica holds the leader lock. Two replicas both reporting 1 = split-brain (LeaderElection's is_leader was exported nowhere before)
bot_sync_consecutive_failures	gauge	platform	kick, trovo, youtube	Consecutive channel-sync failures (0 when healthy)
bot_sync_attempts_total	gauge	platform	kick, trovo, youtube	Total channel-sync attempts (success + failure)
bot_sync_last_success_unix	gauge	platform	kick, trovo, youtube	Unix timestamp of the last successful channel sync (0 = never)
bot_sync_channels_active	gauge	platform	kick, trovo, youtube	Channels active after the last successful sync
bot_sync_channels_skipped	gauge	platform	kick, trovo, youtube	Channels dropped in the last sync for missing provisioning data

The bot_sync_* gauges mirror the SyncHealth atomics (kick/trovo/youtube) that previously surfaced only as JSON on /ready. twitch-bot has no SyncHealth (it joins via anonymous IRC, not the fetch_channels loop) and discord-bot has neither leader election nor a channel-sync loop, so each registers only the series that applies to it.

The cardinality rule (hard, day 1)

A Prometheus series is created per unique label-value combination. High-cardinality labels are the way to kill Prometheus — and across tens of thousands of crawled channels a per-channel label would be the normal case, not an edge case.

Never use as a label:

• account_id, channel_id, user_id, session_id
• raw URL paths (/v1/users/abc-123)

Allowed labels:

• service, env, platform
• endpoint — the template route (/v1/users/{id}), never the concrete path
• status class (2xx, 3xx, 4xx, 5xx) — plus 429, broken out of the 4xx class because rate-limiting is operational backpressure, not a client error, and needs a dedicated alarm (ZAF-680)
• outcome
• method — the bounded HTTP verb
• state, pool, op_type — bounded enumerations (see the API/WebSocket series below)

The RequestMetrics middleware enforces the route rule by labelling on Actix's match_pattern() (the template), so /v1/users/alice and /v1/users/bob collapse to a single endpoint="/v1/users/{id}" series. Requests that match no route are bucketed as endpoint="<unmatched>" rather than leaking the raw path.

/metrics is not publicly exposed

/metrics is an operations endpoint — metric names and route lists are recon material — so it is never reachable via the public ingress. Two service shapes, one rule: the scrape endpoint never sits unauthenticated next to payload routes.

A. Health-only internal server (bots, crawler). These services' only HTTP surface is an internal /health server that the public ingress does not route and that carries no payload routes; /metrics is mounted next to /health there. The effective bind ports come from each service's config/default.toml (the TOML layer overrides the Rust struct default), so read that, not the struct default, when configuring a scraper:

Service	/metrics bind (default.toml)
twitch-bot	0.0.0.0:3001
youtube-bot	0.0.0.0:3002
kick-bot	0.0.0.0:3003
trovo-bot	0.0.0.0:3004
discord-bot	0.0.0.0:3005
crawler-twitch	0.0.0.0:8080

B. Separate internal metrics server (apps/api, workers, innertube-proxy). These services serve payload / handler routes on their main port, so /metrics must not ride it — a co-located scrape endpoint would be unauthenticated next to those routes (apps/api's /v1/health is on the public ingress; the workers' payload is system-key-gated per handler; the proxy's POST /player is bearer-gated per handler on a publicly published port). Each therefore serves /metrics from a separate internal server bound to loopback by default (lo_metrics::serve), with the RequestMetrics middleware still recording latency/status on the main server:

Service	/metrics bind (default)	Host / port override
apps/api	127.0.0.1:9100	LUMIO__METRICS__HOST / __PORT
bot-module-worker	127.0.0.1:9100	BOT_MODULE_WORKER__METRICS__HOST / __PORT
automation-worker	127.0.0.1:9100	AUTOMATION_WORKER__METRICS__HOST / __PORT
innertube-proxy	127.0.0.1:9100	METRICS_ADDR (single host:port)

9100 is the single dedicated metrics port for every service that runs a separate metrics server — apps/api set the pattern (ZAF-683). One port for all is collision-free because each service runs in its own container (the same way the bots would share 3001–3005 if they could); only a scraper needs to know one port, not four. For local development that runs several of these as processes on one host, override the port per process via [metrics] port (workers / apps/api) or METRICS_ADDR (innertube-proxy).

apps/api configures this under [metrics]; the workers under their own [metrics] section; innertube-proxy via the METRICS_ADDR env var:

[metrics]
enabled = true
host = "127.0.0.1"  # loopback — internal only; a scraper overrides this
port = 9100

Reaching a loopback endpoint from another container

Loopback (127.0.0.1) is the safe default, but a Prometheus running in a different container cannot reach another container's loopback. To let a scraper in, bind the metrics server to 0.0.0.0 via the host override — e.g. LUMIO__METRICS__HOST=0.0.0.0 for apps/api, BOT_MODULE_WORKER__METRICS__HOST=0.0.0.0 for the worker, or METRICS_ADDR=0.0.0.0:9100 for the proxy — and then publish that metrics port only to the scraper's network, never on the public ingress. The security here is carried by network isolation (an unpublished/internal port), not by the bind address: 0.0.0.0 is the intended, supported way to expose the scrape endpoint to an in-cluster Prometheus.

Fail-open

No request-handling error path depends on metric collection (the same discipline as GeoIP in the audit path). The RequestMetrics middleware only records after the inner service resolves, with fixed label arity, so it can neither fail nor delay a request. Registry construction and metric registration ignore duplicate errors so they never abort service startup, and rendering a scrape returns an empty body rather than surfacing an encode error.

Viewing metrics locally (dev stack)

The scrape endpoints are useless on a laptop without something to read them. A throwaway Prometheus + Grafana ships in the dev stack for exactly this:

just observability-up      # Grafana http://localhost:3009 · Prometheus http://localhost:9090
just observability-down    # stop it (add `-v` on the compose to wipe the volumes)

This is not the staging stack (dev-stack/staging/, ZAF-675) — it is a separate, minimal dev-stack/observability.docker-compose.yml:

• Host networking (Linux). Local services run as processes on your host, not in containers, so Prometheus scrapes localhost:<port> directly. A service bound to its default 127.0.0.1 is reachable as-is — you do not need to set LUMIO__METRICS__HOST=0.0.0.0 the way the containerised staging stack does. On Docker Desktop (macOS/Windows), host networking behaves differently; use the staging pattern (bridge network + host.docker.internal) instead.
• Targets are api (:9100) and the five bots (:3001–:3005). Anything you are not currently running shows as up == 0 — expected on a dev box, and not the "silent dead target" problem the staging config guards against. The crawler and the workers/proxy (which share :9100 with the api) are commented out in dev-stack/observability/prometheus.yml with instructions to enable them.
• Grafana loads the same three dashboards as staging (single-sourced from dev-stack/staging/observability/grafana/dashboards) and runs with anonymous admin + no login — a local-only convenience that is explicitly disabled in staging. Both UIs bind to 127.0.0.1 only.

Prefer a one-off check without the stack? curl -s localhost:9100/metrics (api) or curl -s localhost:3001/metrics (twitch-bot) prints the raw exposition.

Adding metrics to a service

// Build one registry per process (service name + environment label).
let metrics = lo_metrics::Metrics::new("my-service", &settings.sentry.environment);

// Shape A — a health-only internal server (no payload routes): mount /metrics
// next to /health on that server.
App::new()
    .wrap(lo_metrics::RequestMetrics::new(&metrics))   // http_* series
    .service(lo_metrics::metrics_route(metrics.clone())) // GET /metrics

If the server also carries payload / handler routes (shape B above), do not .service(metrics_route(..)) it there — that puts an unauthenticated /metrics next to them. Keep the RequestMetrics middleware on the payload server, but serve the scrape endpoint from a separate internal server:

// Loopback-bound scrape endpoint on its own port; spawn alongside the main server.
if cfg.metrics.enabled {
    if let Ok(server) = lo_metrics::serve(metrics.clone(), &cfg.metrics.host, cfg.metrics.port) {
        tokio::spawn(server);
    }
}

To declare a service-specific series, register it into the shared registry so it renders alongside everything else:

use lo_metrics::IntCounter;

let jobs_total = IntCounter::new("myservice_jobs_total", "Jobs processed")
    .expect("valid metric");
let _ = metrics.registry().register(Box::new(jobs_total.clone()));
jobs_total.inc();

Adding a metrics dependency pulls a new prometheus (+ procfs) into the graph: hand-add it to the affected BUILD.bazel and run just bazel-vendor, or CI's Vendor Check goes red.

Out of scope (stage 2+)

Scraper, Grafana, alert rules, retention (ZAF-672 stage 2/3). OpenTelemetry / distributed tracing is deliberately excluded — the door stays open via tracing-opentelemetry, but an OTel collector is another stateful hop.