Stream History: adding a platform

Stream History records each stream session (a session may span several simultaneous platform broadcasts = multistream) and its aggregate metrics. It is built platform-neutral: a platform is data, not schema. Adding a new platform (Kick, Trovo, a future one) is a small, bounded change — never a migration, and never a change to the sampler, finalizer, API, or charts.

This guide describes the backend adapter layer (apps/api/src/services/history/) and the exact steps to add a platform.

Design principle: platform is data, not schema

• Database (apps/api/tsdb_migrations): the three history tables (channel_history, channel_history_streams, channel_history_stats) use platform TEXT everywhere — no SQL enum, no per-platform columns. Anything platform-specific lives in the JSONB platform_metrics with namespaced keys.
• Backend: one HistoryPlatformAdapter per platform, held in a registry the sampler/finalizer iterate over. There is no match platform outside the adapters. An event whose platform: prefix has no adapter is logged and ignored — never a panic, never data loss.

The adapter trait

apps/api/src/services/history/platforms/mod.rs:

pub trait HistoryPlatformAdapter: Send + Sync {
    /// Platform key, matching the `platform:` prefix of its event types.
    fn platform(&self) -> &'static str;

    /// Whether this platform drives `channel_status` (and can start a session).
    fn drives_sessions(&self) -> bool;

    /// Map one `platform_event_logs` event onto normalized metric deltas.
    fn classify(&self, event: &PlatformEventRow) -> Vec<MetricDelta>;

    /// Platform-specific live values for the 60s sample (default: none).
    fn sample_metrics(&self, status: &ChannelStatusSnapshot) -> serde_json::Value;
}

classify returns a list of MetricDelta, each a single normalized contribution the finalizer folds into the session aggregates:

MetricDelta	Meaning
Follower	A new follower, or the platform's follower equivalent
PaidSub	A new paid subscription / membership (not a gift, not Twitch Prime)
GiftedSubs(n)	n gifted subs / memberships
Donation { cents, currency }	A genuine viewer donation (e.g. a StreamElements tip) in integer cents of its own currency — no FX conversion. Platform-native monetization (bits, Super Chats) is not a donation — see below
PlatformCounter { key, by }	Increment a namespaced counter in platform_metrics (e.g. Twitch bits, YouTube superchats)

Normalization conventions

• Follower-equivalent — the platform's "new follower" signal. Twitch / Kick / Trovo use their follower event. YouTube has no real new-subscriber event (youtube:subscribe is a repurposed video-upload signal), so its follower equivalent is derived from a channel subscriberCount delta, not from classify.
• Paid-sub-equivalent — new paid recurring support. Twitch paid subs/resubs and YouTube new memberships count; Twitch Prime does not (a platform counter only); gifts are GiftedSubs.
• Currency — donation amounts are normalized to integer cents of the event's own currency. Virtual currencies (Twitch bits, Trovo spells) are platform counters, not donations.
• Bits / Super Chats / Super Stickers are a distinct monetization category, not donations (ZAF-583 §5 field-visibility ruling (a), founder 2026-08-20). Twitch cheer (bits) and YouTube superchat / supersticker emit only namespaced PlatformCounters — a count (bits, superchats, superstickers) plus, for Super Chats/Stickers, an account-only *_amount_cents total — and never a Donation. They must never fold into donations_total_cents: donations stay never-public, whereas these categories are public as counts only, never identities.
• Donations are StreamElements tips only. streamelements:tip is the only event that emits a Donation; every report/email/export surface labels the metric "Donations (StreamElements)" so it is not confused with bits or Super Chats (ZAF-745). Sessions finalized before the ZAF-601 fix (2026-08-21) folded Super Chat/Sticker money into donations; a boot-time re-fold backfill (backfill::backfill_monetization_split) re-derives them from the retained event log and rewrites the tips-only total (idempotent, best-effort).
• Raids — an incoming raid is a PlatformCounter pair: raids (count) + raid_viewers (incoming audience). Twitch (twitch:raid, viewers in raw) and Trovo (trovo:raid, amount in raw) emit it; outgoing twitch:raid_done is not counted. Kick's webhook catalogue exposes no host/raid event, so Kick contributes none. Raids are not monetization — never a Donation (ZAF-745).
• Platform-aware terminology — on YouTube, followers are Subscribers and paid subs are Members (de: Abonnenten / Mitglieder). Per-platform report and email sections use the platform-correct term; single-platform YouTube sessions read Subscribers/Members in the aggregate cards/rows/prose too, while multi-platform sessions keep neutral aggregate labels (ZAF-745).

The registry

HistoryAdapterRegistry::with_builtins() seeds every adapter. Built-ins today: twitch, youtube, streamelements, kick, trovo. Twitch and YouTube drive sessions; the rest have drives_sessions = false (their events are counted inside a session another platform started).

pub fn with_builtins() -> Self {
    let mut reg = Self::new();
    reg.register(Box::new(twitch::TwitchAdapter));
    reg.register(Box::new(youtube::YouTubeAdapter));
    reg.register(Box::new(streamelements::StreamElementsAdapter));
    reg.register(Box::new(kick::KickAdapter));
    reg.register(Box::new(trovo::TrovoAdapter));
    reg
}

Session lifecycle

apps/api/src/services/history/session.rs hooks into the channel-status service (set_online / set_offline / set_broadcast_offline), so every online/offline transition — from the workers and from startup recovery — feeds history exactly once. History is analytics: the hooks are best-effort and never fail the live status path.

• online — find or open the account's session; attach this platform's stream row. A reconnect within the 5-minute grace window continues the same stream (no duplicate); a second distinct platform promotes the session to multi.
• offline — end the matching stream row(s). The session stays open through its grace window.
• grace close / orphan recovery — the grace window is derived from TSDB (max(stream.ended_at) + 5min < now), so correctness never depends on Redis. Startup recovery closes sessions left open by a restart mid-stream-end.

Channel identity (frozen at open)

Each channel_history_streams row carries the streaming channel's identity, frozen at stream open so a recorded session is channel-attributable without a live JOIN back to the main-DB channel_connections (a separate Postgres instance that is gone once the account is dissolved). Three nullable columns:

Column	Source	Meaning
platform_channel_id	channel_connections.platform_channel_id	stable platform channel id — the join key for the channel page
channel_login	channel_connections.channel_name	handle / slug as-of-session (the URL spelling)
channel_display_name	— (no source today)	display name as-of-session; always NULL for now

Capture rules:

• Resolved once, at the online transition in channel_status::set_online, from the same connections::get_connection(pool, account_id, platform) lookup the startup verifiers use, and threaded into the session hook.
• Written exactly once by db::create_stream. db::reopen_stream (a grace-window reconnect) deliberately does not touch these columns — the stream keeps the identity captured when it first opened, so a later handle rename never rewrites history.
• Honest NULL, never guessed — a momentarily-unreadable connection writes NULL rather than a placeholder, and never blocks the (best-effort) status path. channel_connections exposes a single channel_name (a handle/login/slug on every platform, never a display name), so it maps to channel_login only; channel_display_name has no source and stays NULL.
• Public addressability — a partial-unique index on (platform, broadcast_id) WHERE broadcast_id IS NOT NULL makes the platform-native broadcast id the URL key; (platform, platform_channel_id, started_at DESC) lists a channel's streams newest-first.

Legacy rows recorded before this capture landed are filled by an idempotent startup backfill (services::history::backfill::backfill_stream_identity) that reads the current channel_connections and UPDATEs rows whose platform_channel_id is still NULL; the rest stay NULL (unresolvable, never guessed). It logs the coverage ratio (filled/total) on every boot.

Provenance (source)

Each channel_history_streams row also carries a provenance flag recording how the row was discovered — the basis for the public "Lumio channel" badge on the future stats surface:

source	Meaning
first_party	Recorded from a connected Lumio channel (the default, and the only writer today). This value is the public "Lumio channel" badge.
external_crawler	Materialized by the future crawler-discovery path. No writer yet — that writer ships with the crawler issue.

• source TEXT NOT NULL DEFAULT 'first_party'. The constant default is retroactively correct for every existing row: all pre-crawler data is, by definition, first-party, so the column is added with no backfill.
• Write path unchanged. The first-party path (db::create_stream) simply inherits the column default; the only non-default writer ('external_crawler') belongs to the crawler-discovery path, not the recording path here.

Sampler & finalizer

One global worker (apps/api/src/workers/history_sampler.rs) on a 60-second tick runs both jobs. It knows no platform names — it iterates the registry.

• Sampler — for every open session it writes one channel_history_stats sample per live platform: the viewer count and live values come from the Redis status snapshot (see below), platform-specific values from the adapter's sample_metrics. Cumulative counters (messages / followers / subs / donations) are counted incrementally since the last sample and added onto the interim totals held in a per-session Redis sampler-state key, so per-tick cost stays constant on a marathon stream instead of re-counting from session start.
• Finalizer — once the grace window closes a session, it computes the aggregates once and persists them, flipping finalized = true:
  • messages / unique + top-10 chatters from platform_chat_messages, each entry { user_id, name, platform, avatar_url, count } — avatar_url is a representative avatar taken from the chatter's messages and is null when the platform's ingestion captured none;
  • followers, paid subs, gifted subs, Twitch tier breakdown + bits, YouTube memberships / superchats / superstickers, incoming raids (raids + raid_viewers, Twitch/Trovo), and donations — all via classify;
  • donations are kept per currency (no FX conversion): the denormalized donations_total_cents / donations_currency columns hold the dominant currency for the list preview, the full map lives in platform_metrics.donations_by_currency. Bits and Super Chats / Super Stickers are excluded from all of these — they are their own category (counts + account-only *_amount_cents) under platform_metrics.twitch / platform_metrics.youtube, never donations (§5 ruling (a); see Normalization conventions);
  • emote count + top-10 from the messages' emotes JSON, each entry { emote, provider, url, count } — url is a representative image URL lifted from the stored emote objects. All four platforms can appear here: Twitch builds the static-cdn.jtvnw.net URL from the fragment's emote id, YouTube resolves it from the InnerTube emote cache, Kick derives each emote's name from content[s..e] at webhook ingest (with a best-effort catalog URL), and Trovo enriches messages from the durable catalog (ZAF-578). Because Kick emotes now carry a name, they survive the aggregate's name IS NOT NULL filter and rank normally — closing the old disagreement where emote_total counted them but top_emotes dropped them. A top_emotes entry that is still missing a url (e.g. a Kick emote with no catalog row, or a report finalized before URLs existed) is resolved from the catalog at read time in service::build_report_for /build_export (lo_chat::resolve_emote_urls, keyed on (provider, name)); an entry that already carries a url keeps its frozen snapshot. The resolution is best-effort and fail-open — an unavailable catalog degrades to the shorthand plus a provider icon, never a failed report — and it covers the CSV/JSON exports, the public share page and the summary e-mail, which all reuse the same report builder. Emote counts are per-platform, not cross-platform, comparable (ZAF-570): both aggregates read the same emotes array, so emote_total (SUM(jsonb_array_length)) and top_emotes (COUNT(*) over jsonb_array_elements) never disagree within a channel. Across platforms the magnitudes differ in spirit because the writers count differently: Twitch ingest stores one entry per emote occurrence (one per emote fragment, so Kappa Kappa → 2), while the Kick / Trovo / YouTube enrichers dedupe to one entry per shortcode per message (normalize_kick_emotes / enrich_message_emotes_from_sets, matched by name, so Kappa Kappa → 1). This is deliberately not normalised: the deduped shape is exactly what the chat renderer consumes (it matches emotes by name, so per-occurrence duplicates would be redundant), and rewriting either writer would retroactively change already-frozen report totals. Per-platform ranking and trends stay correct; only absolute totals are not directly comparable between platforms;
  • cross-platform peak/avg viewers from the samples. A running (not-yet-finalized) session's aggregates are computed on the fly with the same code, cached briefly in Redis.

YouTube follower equivalent

YouTube has no real new-subscriber event, so the follower count comes from the channel subscriberCount: the sampler captures it at session start and the finalizer re-reads it at close (two Data-API calls per session, or zero via InnerTube). The delta is added to new_followers and marked approximate in platform_metrics.youtube (followers_approx), because YouTube rounds public subscriber counts to three significant figures. youtube:subscribe is a repurposed upload signal and is ignored for follower counting.

channel_status slim-down

Live counters (viewer count, YouTube likes/views) now live in the Redis status snapshot and the 60-second history sample, not in per-poll Postgres writes. channel_status is written only on real state transitions (online / offline / upcoming / live_chat_id / title / category change). The dashboard endpoints (channelStatus GraphQL query and GET /v1/channel-status) overlay the live counters from the Redis snapshot, falling back to the latest history sample when the cache is cold — so they never show a frozen value. See Channel status.

Steps to add a platform

1. Emit events in the platform worker under the platform:action convention (e.g. kick:follower) into platform_event_logs — this is the ingest side.
2. Add an adapter apps/api/src/services/history/platforms/<platform>.rs implementing HistoryPlatformAdapter. Map its events in classify; expose any live values in sample_metrics. Set drives_sessions — false until the platform also drives channel_status.
3. Register it in HistoryAdapterRegistry::with_builtins().
4. Unit-test classify with representative payloads (positive and unknown- action cases), following the existing adapter tests.
5. Frontend (when the UI ships): add metric-descriptor entries + icon + i18n (en + de) for any new platform_metrics keys; unknown keys render with a generic fallback.

To make a platform a session driver later, wire its channel_status online/offline events and flip drives_sessions to true — a pure adapter change, no migration and no schema change.

Retention

There is no retention sweep on the stream-history tables (channel_history, channel_history_streams, channel_history_stats). Only chat messages are pruned, by the chat_retention worker (gated on the chat_retention toggle). Stream history is durable product value and is retained indefinitely — this is a deliberate decision, not an oversight: the aggregates and viewer curves are the whole point of the feature, and the future public stats surface assumes an open-ended history of (opt-in) sessions. If a history retention policy is ever introduced it must be an explicit, separately-gated decision.