Stream History: adding a platform

Stream History records each stream session (a session may span several simultaneous platform broadcasts = multistream) and its aggregate metrics. It is built platform-neutral: a platform is data, not schema. Adding a new platform (Kick was the last one) is a small, bounded change - never a migration, and never a change to the sampler, finalizer, API, or charts.

This guide describes the backend adapter layer (apps/api/src/services/history/) and the exact steps to add a platform.

Design principle: platform is data, not schema

• Database (apps/api/tsdb_migrations): the three history tables (channel_history, channel_history_streams, channel_history_stats) use platform TEXT everywhere - no SQL enum, no per-platform columns. Anything platform-specific lives in the JSONB platform_metrics with namespaced keys.
• Backend: one HistoryPlatformAdapter per platform, held in a registry the sampler/finalizer iterate over. There is no match platform outside the adapters. An event whose platform: prefix has no adapter is logged and ignored - never a panic, never data loss.

The adapter trait

apps/api/src/services/history/platforms/mod.rs:

pub trait HistoryPlatformAdapter: Send + Sync {
    /// Platform key, matching the `platform:` prefix of its event types.
    fn platform(&self) -> &'static str;

    /// Whether this platform drives `channel_status` (and can start a session).
    fn drives_sessions(&self) -> bool;

    /// Map one `platform_event_logs` event onto normalized metric deltas.
    fn classify(&self, event: &PlatformEventRow) -> Vec<MetricDelta>;

    /// Platform-specific live values for the 60s sample (default: none).
    fn sample_metrics(&self, status: &ChannelStatusSnapshot) -> serde_json::Value;
}

classify returns a list of MetricDelta, each a single normalized contribution the finalizer folds into the session aggregates:

MetricDelta	Meaning
Follower	A new follower, or the platform's follower equivalent
PaidSub	A new paid subscription / membership (not a gift, not Twitch Prime)
GiftedSubs(n)	n gifted subs / memberships
Donation { cents, currency }	A genuine viewer donation (e.g. a StreamElements tip) in integer cents of its own currency - no FX conversion. Platform-native monetization (bits, Super Chats) is not a donation - see below
PlatformCounter { key, by }	Increment a namespaced counter in platform_metrics (e.g. Twitch bits, YouTube superchats)

Normalization conventions

• Follower-equivalent - the platform's "new follower" signal. Twitch and Kick use their follower event. YouTube has no real new-subscriber event (youtube:subscribe is a repurposed video-upload signal), so its follower equivalent is derived from a channel subscriberCount delta, not from classify.
• Paid-sub-equivalent - new paid recurring support. Twitch and Kick paid subs/resubs and YouTube new memberships count; Twitch Prime does not (a platform counter only); gifts are GiftedSubs. A resub (twitch:resub, kick:resub) additionally increments the resubs platform counter.
• Currency - donation amounts are normalized to integer cents of the event's own currency. Virtual currencies (Twitch bits, Kick KICKs) are platform counters, not donations.
• Bits / Super Chats / Super Stickers are a distinct monetization category, not donations (ZAF-583 §5 field-visibility ruling (a), founder 2026-08-20). Twitch cheer (bits) and YouTube superchat / supersticker emit only namespaced PlatformCounters - a count (bits, superchats, superstickers) plus, for Super Chats/Stickers, an account-only *_amount_cents total - and never a Donation. They must never fold into donations_total_cents: donations stay never-public, whereas these categories are public as counts only, never identities. Kick's kick:kicks belongs to the same category: one PlatformCounter { key: "kicks", by: <amount> }, never a Donation.
• Donations are StreamElements tips only. streamelements:tip is the only event that emits a Donation; every report/email/export surface labels the metric "Donations (StreamElements)" so it is not confused with bits or Super Chats (ZAF-745). Sessions finalized before the ZAF-601 fix (2026-08-21) folded Super Chat/Sticker money into donations; a boot-time re-fold backfill (backfill::backfill_monetization_split) re-derives them from the retained event log and rewrites the tips-only total (idempotent, best-effort).
• Raids - an incoming raid is a PlatformCounter pair: raids (count) + raid_viewers (incoming audience). Twitch (twitch:raid, viewers in raw) and not counted. Kick's webhook catalogue exposes no host/raid event, so Kick contributes none. Raids are not monetization - never a Donation (ZAF-745).
• Platform-aware terminology follows the scope of the number. A Twitch-only or Kick-only scope says Followers / Subs (de: Follower / Subs). A YouTube-only scope says Subscribers / Members (de: Abonnenten / Mitglieder), with ≈ on rounded YouTube subscriber values. A mixed scope says Followers & Subscribers / Subs & Members (de: Follower & Abonnenten / Subs & Mitglieder); the per-platform breakdown keeps each native term. Apply the same rule after a platform filter, even when the underlying session is a multistream. Do not call paid subs “Abo” in German: it collides with YouTube's free Abonnenten. The Twitch tier chart covers Twitch tiers only; YouTube members sit separately.
• Gift tiers and bundle counting - Twitch gift tiers (gift_subs_t1/t2/t3) are priced separately. A community gift consists of a header and recipient rows tied by community_gift_id; the classifier counts the header and skips bundled recipient rows, so N gifts count N times, not 2N. A targeted gift without that id counts normally. The boot backfill corrects gifted_subs, twitch.gift_subs, and the platform breakdown while event logs remain retained; older frozen sessions without those logs keep the assumed tier-1 note. The channel.subscribe EventSub recipient signal is not stored as a PaidSub, so new_paid_subs does not double-count it.
• Twitch and Kick ads - twitch:ad_break and kick:ad_break yield ad_breaks, ad_breaks_automatic, and ad_seconds. Finalization freezes the break list and viewer spots. Estimated gross impressions are viewers at break start × ceil(duration_seconds / 30); the editable ad-free share applies at read time. Missing CPM leaves an unpriced cpm_missing line. Kick's feed comes from the ad-break status poller, while YouTube has no ad-break feed.
• Super Chat currencies - YouTube Super Chats and Stickers accumulate superchats_amount_by_currency / superstickers_amount_by_currency. Their legacy combined *_amount_cents fields remain for compatibility but can mix currencies. Conversion and totals use the per-currency values; old mixed rows are labelled mixed_currencies rather than given a false exchange rate.

Kick

The Kick adapter reads the canonical top-level fields the Kick webhook receiver adds at ingest (lo_kick_api::normalize_event, see Webhooks):

Event	Deltas
kick:follower	Follower
kick:subscribe	PaidSub
kick:resub	PaidSub + PlatformCounter { key: "resubs", by: 1 }
kick:gift	GiftedSubs(n) + PlatformCounter { key: "gift_subs", by: n }, where n = giftCount (the number of giftees; a gift to 5 viewers counts 5). A row stored raw without canonical fields falls back to counting giftees, then to 1
kick:kicks	PlatformCounter { key: "kicks", by: kicks } (nothing when the amount is 0)
kick:reward, kick:reward_status, kick:stream_*	none

The separate Kick ad poller (services/kick_ads.rs) calls GET /public/v1/ads/ad-break-status only when a Kick stream is live and the stored grant includes ads:read, plus one final call. The first live call is due five minutes after stream start. Subsequent calls use half of limits.period_seconds, clamped to 5–60 minutes (five minutes when the provider gives no window). A Redis lock permits one poller per session. Provider 429/5xx and transport errors use exponential backoff and Retry-After; after three failures polling stops for that stream. A 403 stops immediately and removes ads:read from the stored grant. Each newly observed break becomes a kick:ad_break event with duration_seconds, source, status, started_at and ad_break_id; the event is not a webhook. The poller uses get_fresh_connection_token and records each outgoing call in platform_api_requests_total.

drives_sessions is true. Kick's live status runs through the same channel_status service as Twitch (services/kick_live_status.rs): the livestream.status.updated webhook calls set_online / set_offline, the Kick worker's start reconcile and the 60-s poll while live (workers/kick_live_status.rs) correct a missed webhook. set_online fires the on_platform_online hook, so a Kick-only stream opens its own session; alongside Twitch or YouTube it adds a kick stream row to the same session and promotes it to multi. The poll writes the viewer count into the Redis status snapshot, and the 60-s sampler turns it into the Kick stream's peak and average viewers. The Kick broadcast_id is the stream start time (started_at in the webhook, stream.start_time in the API), normalised to UTC seconds.

The registry

HistoryAdapterRegistry::with_builtins() seeds every adapter. Built-ins today: twitch, youtube, streamelements, kick. Twitch, YouTube and Kick drive sessions; StreamElements has drives_sessions = false (its events are counted inside a session another platform started).

pub fn with_builtins() -> Self {
    let mut reg = Self::new();
    reg.register(Box::new(twitch::TwitchAdapter));
    reg.register(Box::new(youtube::YouTubeAdapter));
    reg.register(Box::new(streamelements::StreamElementsAdapter));
    reg.register(Box::new(kick::KickAdapter));
    reg
}

Session lifecycle

apps/api/src/services/history/session.rs hooks into the channel-status service (set_online / set_offline / set_broadcast_offline), so every online/offline transition - from the workers and from startup recovery - feeds history exactly once. History is analytics: the hooks are best-effort and never fail the live status path.

• online - find or open the account's session; attach this platform's stream row. A reconnect within the 5-minute grace window continues the same stream (no duplicate); a second distinct platform promotes the session to multi.
• offline - end the matching stream row(s). The session stays open through its grace window.
• grace close / orphan recovery - the grace window is derived from TSDB (max(stream.ended_at) + 5min < now), so correctness never depends on Redis. Startup recovery closes sessions left open by a restart mid-stream-end.

Channel identity (frozen at open)

Each channel_history_streams row carries the streaming channel's identity, frozen at stream open so a recorded session is channel-attributable without a live JOIN back to the main-DB channel_connections (a separate Postgres instance that is gone once the account is dissolved). Three nullable columns:

Column	Source	Meaning
platform_channel_id	channel_connections.platform_channel_id	stable platform channel id - the join key for the channel page
channel_login	channel_connections.channel_name	handle / slug as-of-session (the URL spelling)
channel_display_name	- (no source today)	display name as-of-session; always NULL for now

Capture rules:

• Resolved once, at the online transition in channel_status::set_online, from the same connections::get_connection(pool, account_id, platform) lookup the startup verifiers use, and threaded into the session hook.
• Written exactly once by db::create_stream. db::reopen_stream (a grace-window reconnect) deliberately does not touch these columns - the stream keeps the identity captured when it first opened, so a later handle rename never rewrites history.
• Honest NULL, never guessed - a momentarily-unreadable connection writes NULL rather than a placeholder, and never blocks the (best-effort) status path. channel_connections exposes a single channel_name (a handle/login/slug on every platform, never a display name), so it maps to channel_login only; channel_display_name has no source and stays NULL.
• Public addressability - a partial-unique index on (platform, broadcast_id) WHERE broadcast_id IS NOT NULL makes the platform-native broadcast id the URL key; (platform, platform_channel_id, started_at DESC) lists a channel's streams newest-first.

Legacy rows recorded before this capture landed are filled by an idempotent startup backfill (services::history::backfill::backfill_stream_identity) that reads the current channel_connections and UPDATEs rows whose platform_channel_id is still NULL; the rest stay NULL (unresolvable, never guessed). It logs the coverage ratio (filled/total) on every boot.

Provenance (source)

Each channel_history_streams row also carries a provenance flag recording how the row was discovered - the basis for the public "Lumio channel" badge on the future stats surface:

source	Meaning
first_party	Recorded from a connected Lumio channel (the default, and the only writer today). This value is the public "Lumio channel" badge.
external_crawler	Materialized by the future crawler-discovery path. No writer yet - that writer ships with the crawler issue.

• source TEXT NOT NULL DEFAULT 'first_party'. The constant default is retroactively correct for every existing row: all pre-crawler data is, by definition, first-party, so the column is added with no backfill.
• Write path unchanged. The first-party path (db::create_stream) simply inherits the column default; the only non-default writer ('external_crawler') belongs to the crawler-discovery path, not the recording path here.

Sampler & finalizer

One global worker (apps/api/src/workers/history_sampler.rs) on a 60-second tick runs both jobs. It knows no platform names - it iterates the registry.

• Sampler - for every open session it writes one channel_history_stats sample per live platform: the viewer count and live values come from the Redis status snapshot (see below), platform-specific values from the adapter's sample_metrics. Cumulative counters (messages / followers / subs / donations) are counted incrementally since the last sample and added onto the interim totals held in a per-session Redis sampler-state key, so per-tick cost stays constant on a marathon stream instead of re-counting from session start.
• Finalizer - once the grace window closes a session, it computes the aggregates once and persists them, flipping finalized = true:
  • messages / unique + top-10 chatters from platform_chat_messages, each entry { user_id, name, platform, avatar_url, count } - avatar_url is a representative avatar taken from the chatter's messages and is null when the platform's ingestion captured none;
  • followers, paid subs, gifted subs, Twitch tier breakdown + bits, Kick kicks / resubs / gift_subs, YouTube memberships / superchats / superstickers, incoming raids (raids + raid_viewers, Twitch), and donations - all via classify;
  • donations are kept per currency (no FX conversion): the denormalized donations_total_cents / donations_currency columns hold the dominant currency for the list preview, the full map lives in platform_metrics.donations_by_currency. Bits and Super Chats / Super Stickers are excluded from all of these - they are their own category (counts + account-only *_amount_cents) under platform_metrics.twitch / platform_metrics.youtube, never donations (§5 ruling (a); see Normalization conventions);
  • emote count + top-10 from the messages' emotes JSON, each entry { emote, provider, url, count } - url is a representative image URL lifted from the stored emote objects. All supported chat platforms can appear here: Twitch builds the static-cdn.jtvnw.net URL from the fragment's emote id, YouTube resolves it from the InnerTube emote cache, and Kick builds https://files.kick.com/emotes/{id}/fullsize from the [emote:ID:NAME] marker Kick puts in the message content (ZAF-1622), so every Kick entry carries a name and a url. A top_emotes entry that is still missing a url (e.g. a report finalized before URLs existed) is resolved from the catalog at read time in service::build_report_for /build_export (lo_chat::resolve_emote_urls, keyed on (provider, name)); an entry that already carries a url keeps its frozen snapshot. The resolution is best-effort and fail-open - an unavailable catalog degrades to the shorthand plus a provider icon, never a failed report - and it covers the CSV/JSON exports and the public share page, which reuse the report builder. The summary e-mail uses report data but does not display emote counts or top-emote lists. Emote counts are per-platform, not cross-platform, comparable (ZAF-570): both aggregates read the same emotes array, so emote_total (SUM(jsonb_array_length)) and top_emotes (COUNT(*) over jsonb_array_elements) never disagree within a channel. Across platforms the magnitudes differ in spirit because the writers count differently: Twitch and Kick ingest store one entry per emote occurrence (Twitch: one per emote fragment; Kick: one per [emote:ID:NAME] marker, so Kappa Kappa → 2), while the YouTube enricher dedupes to one entry per shortcode per message (enrich_message_emotes_from_sets, matched by name, so Kappa Kappa → 1). Stored Kick rows were rewritten to the per-occurrence shape by the TimescaleDB migration 20260930000001_kick_chat_emote_markup_backfill; the frozen totals of Kick reports finalised before it are not recomputed and keep their deduped counts. The YouTube asymmetry is deliberately not normalised: the deduped shape is exactly what the chat renderer consumes, and rewriting that writer would retroactively change already-frozen report totals. Per-platform ranking and trends stay correct; only absolute totals are not directly comparable between YouTube and Twitch/Kick;
  • cross-platform peak/avg viewers from the samples. A running (not-yet-finalized) session's aggregates are computed on the fly with the same code, cached briefly in Redis.

YouTube follower equivalent

YouTube has no real new-subscriber event, so the follower count comes from the channel subscriberCount: the sampler captures it at session start and the finalizer re-reads it at close (two Data-API calls per session, or zero via InnerTube). The delta is added to new_followers and marked approximate in platform_metrics.youtube (followers_approx), because YouTube rounds public subscriber counts to three significant figures. youtube:subscribe is a repurposed upload signal and is ignored for follower counting.

channel_status slim-down

Live counters (viewer count, YouTube likes/views) now live in the Redis status snapshot and the 60-second history sample, not in per-poll Postgres writes. channel_status is written only on real state transitions (online / offline / upcoming / live_chat_id / title / category change). The dashboard endpoints (channelStatus GraphQL query and GET /v1/channel-status) overlay the live counters from the Redis snapshot, falling back to the latest history sample when the cache is cold - so they never show a frozen value. See Channel status.

Steps to add a platform

1. Emit events in the platform worker under the platform:action convention (e.g. kick:follower) into platform_event_logs - this is the ingest side.
2. Add an adapter apps/api/src/services/history/platforms/<platform>.rs implementing HistoryPlatformAdapter. Map its events in classify; expose any live values in sample_metrics. Set drives_sessions - false until the platform also drives channel_status.
3. Register it in HistoryAdapterRegistry::with_builtins().
4. Unit-test classify with representative payloads (positive and unknown- action cases), following the existing adapter tests.
5. Frontend (when the UI ships): add metric-descriptor entries + icon + i18n (en + de) for any new platform_metrics keys; unknown keys render with a generic fallback.

To make a platform a session driver later, wire its channel_status online/offline events and flip drives_sessions to true - a pure adapter change, no migration and no schema change.

Retention

There is no retention sweep on the stream-history tables (channel_history, channel_history_streams, channel_history_stats). Only chat messages are pruned, by the chat_retention worker (gated on the chat_retention toggle). Stream history is durable product value and is retained indefinitely - this is a deliberate decision, not an oversight: the aggregates and viewer curves are the whole point of the feature, and the future public stats surface assumes an open-ended history of (opt-in) sessions. If a history retention policy is ever introduced it must be an explicit, separately-gated decision.