Data layer - PostgreSQL, TimescaleDB, extensions PostgreSQL, Redis (Staging)
The staging data layer is defined in dev-stack/staging/infra.docker-compose.yml.
That file is a reference: the staging host runs its own copy of it, so a
change here reaches staging only when the host copy is updated (see
Applying a change on the staging host).
The local dev-stack/db.docker-compose.yml uses the same flags, with default
memory sizes and without the access log (local data is not
protected customer data).
Memory sizing
PostgreSQL and TimescaleDB use about 25 % of the RAM they may use for
shared_buffers and about 70 % for effective_cache_size. Redis gets a container
limit and a maxmemory of about 70 % of that limit. The remaining 30 % covers the
fork for an AOF rewrite and allocator overhead.
The sizes are compose variables with defaults. Compose reads them from the shell
or the project .env file, not from env_file: .env.infra:
| Variable | Default | Setting |
|---|---|---|
PG_SHARED_BUFFERS | 1GB | PostgreSQL shared_buffers |
PG_EFFECTIVE_CACHE_SIZE | 3GB | PostgreSQL effective_cache_size |
TSDB_SHARED_BUFFERS | 1GB | TimescaleDB shared_buffers |
TSDB_EFFECTIVE_CACHE_SIZE | 3GB | TimescaleDB effective_cache_size |
REDIS_MEM_LIMIT | 2g | Redis container mem_limit |
REDIS_MAXMEMORY | 1400mb | Redis --maxmemory |
The defaults assume each database may use about 4 GB of the host. On a larger host, raise the values for both databases. Keep the ratios when you do.
Redis maxmemory: one TimescaleDB outage hour
While TimescaleDB is down, the chat flush cannot drain the buffer, so every chat
message stays in Redis. Once Redis is full, the direct-insert fallback fails as
well, because it writes to the same TimescaleDB, and messages are lost. Size
REDIS_MAXMEMORY so that it holds the steady state plus at least one outage
hour of chat-buffer growth. Then set REDIS_MEM_LIMIT to about
REDIS_MAXMEMORY / 0.7.
The capacity model (ZAF-2180 §3.5) gives these values:
| Active channels | Steady state | Buffer growth per outage hour | Minimum maxmemory | REDIS_MAXMEMORY / REDIS_MEM_LIMIT |
|---|---|---|---|---|
| 500 | ~0.15 GB (scaled linearly) | ~0.36 GB | ~0.5 GB | 1400mb / 2g (the defaults) |
| 5 000 | ~1.5 GB (profiles ~1.1 GB, chatter hashes ~0.3 GB) | ~3.6 GB | ~5.1 GB | 5600mb / 8g |
Dead letters stay for 72 h, so add their current size from redis-cli --bigkeys
on top. With these sizes, RedisMemoryNearMaxmemory (80 %) fires well before
Redis starts rejecting writes. At 5 000 channels it fires about 50 minutes into
an outage.
Redis: noeviction
Redis runs with --maxmemory-policy noeviction. Some Redis data is durable and
has no other copy until it is flushed or expires:
- the chat buffer
lumio:chat:buffer:*(messages waiting for the batch insert into TimescaleDB) and its dead letters (72 h), - parked platform events
lumio:events:deadletter:*(inserts that failed while TimescaleDB was unavailable, 72 h; see Chat and event durability), - extension storage (some keys have no TTL),
- automation variables (30 days).
An allkeys-* policy would drop these keys without an error. With noeviction,
Redis answers every write with OOM command not allowed when used memory > 'maxmemory' when it is full. Reads and deletes still work, so the chat flush
(LRANGE + DEL) can still drain the buffer. What a full Redis does to each writer:
- Chat ingest - the buffer push fails and the message goes into TimescaleDB
with a direct insert (
buffer_or_persistincrates/lo-chat/src/buffer.rs). It is counted inchat_buffer_push_failures_total{outcome="direct_insert"}. - Platform events - unaffected while TimescaleDB is healthy (Redis is only
written when an insert fails). If both fail, the event is lost and counted in
events_persist_failures_total{outcome="lost"}. - Extension storage, automation variables, caches - the write fails, and the caller gets the error.
Do not switch to volatile-lru until the extension-storage keys without a TTL
are stored in PostgreSQL. With volatile-lru, Redis evicts only keys that have
a TTL. That still includes the chat and platform-event dead-letter lists.
Alarms
oliver006/redis_exporter (container staging-redis-exporter) exports Redis
INFO to Prometheus as job redis. The redis_alerts group in
observability/prometheus/rules/alerting_rules.yml uses it:
| Alert | Fires when | Severity |
|---|---|---|
RedisMemoryNearMaxmemory | redis_memory_used_bytes / redis_memory_max_bytes > 0.8 for 10 min | ticket |
RedisRejectingWrites | rate(redis_errors_total{err="OOM"}[5m]) > 0 for 5 min | page |
RedisMaxmemoryUnset | redis_memory_max_bytes == 0 for 15 min (the running Redis has no --maxmemory) | ticket |
If the exporter is down, ServiceTargetDown fires for job redis.
To give Redis more memory, raise REDIS_MEM_LIMIT and REDIS_MAXMEMORY together
and recreate the container. Before that, run redis-cli --bigkeys to find what
grew. A stuck chat flush also shows up as ChatBufferBacklog.
Command latency
The lo-cache client records redis_command_duration_seconds{operation}
(ZAF-1532) on the API's /metrics. The recording rule
service_operation:redis_command_duration_seconds:p99_5m holds the p99 per
command. The weekly read is the highest p99 per command over 7 days:
max_over_time(service_operation:redis_command_duration_seconds:p99_5m[7d])
The client times out after 2 000 ms (command_timeout_ms in
crates/lo-config/src/common.rs). Likely slow commands are HGETALL on chatter
hashes (up to 5 000 fields, about 1 MB), the chat flush worker's one
full-keyspace SCAN at startup, and any new unbounded LRANGE 0 -1. A p99 that stays near 2 s is an
infrastructure problem, not a timeout that is too tight.
PostgreSQL and TimescaleDB settings
Both databases start with these -c flags:
| Setting | Value | Why |
|---|---|---|
shared_buffers | 25 % of the database's RAM | The 128 MB default is too small for the working set |
effective_cache_size | 70 % of the database's RAM | Lets the planner choose index scans |
log_min_duration_statement | 1000 | Logs every statement that takes 1 s or longer |
shared_preload_libraries | pg_stat_statements (PostgreSQL); timescaledb,pg_textsearch,pg_stat_statements,pgaudit (TimescaleDB) | Needed for pg_stat_statements, and on TimescaleDB for the access log |
On TimescaleDB, shared_preload_libraries replaces the image's own list
(timescaledb,pg_textsearch). Both entries must stay in it.
All three databases also write their server log to daily files
(logging_collector and the connection log; on TimescaleDB also the pgaudit.*
flags) under TSDB_LOG_DIR, PG_LOG_DIR and EXT_PG_LOG_DIR, so docker logs
shows only the start-up lines. That log is the access log for protected customer
data: see
Access log for protected customer data.
TimescaleDB job workers
TimescaleDB also gets these flags:
| Setting | Value |
|---|---|
timescaledb.max_background_workers | 16 |
max_parallel_workers | 4 |
max_worker_processes | 24 |
tsdb_migrations create 20 background jobs. TimescaleDB adds its own
housekeeping jobs.
| Kind | Count | Tables |
|---|---|---|
| Compression policy | 9 | platform_event_logs, platform_chat_messages, audit_events, channel_history_stats, channel_history_stats_5m, channel_history_stats_1h, extension_errors, extension_install_logs, extension_events |
| Retention policy | 5 | extension_errors, extension_install_logs, extension_events, channel_history_stats, channel_history_stats_5m |
| Continuous-aggregate refresh | 2 | channel_history_stats_5m, channel_history_stats_1h |
Custom job (add_job) | 4 | anonymise_old_audit_events, purge_old_audit_events, prune_old_csp_reports, recompress_stale_orderby_chunks (deschedules itself once no chunk with an old compression order is left) |
Each running job takes one background worker, and each database with the
extension adds one scheduler. The default of 16 background workers is enough.
Background workers are taken from max_worker_processes. Parallel queries and
the launcher take from it too, and its default is 8. So the jobs had less than
8 slots, shared with parallel queries. 24 = 16 background workers + 4 parallel
workers + the launcher + headroom.
pg_stat_statements
The library is preloaded, but the view exists only after you create the extension. Run this once in each database as the database owner:
CREATE EXTENSION IF NOT EXISTS pg_stat_statements;
Run it in lumio (PostgreSQL) and lumio_tsdb (TimescaleDB). The slowest
statements by total time:
SELECT calls, round(total_exec_time) AS total_ms, round(mean_exec_time, 1) AS mean_ms, query
FROM pg_stat_statements ORDER BY total_exec_time DESC LIMIT 20;
Read-only access for anyone else (for example a pg_read_all_stats role) is a
separate decision. Enabling the extension does not grant it.
Extensions PostgreSQL: connection budget
Extension data (ext_{extension_id} schemas) lives in a separate PostgreSQL,
extensions-postgres (lumio_extensions, host port 5434). It runs
postgres:18-alpine with max_connections set explicitly to 100
(EXT_PG_MAX_CONNECTIONS). PostgreSQL keeps 3 of them for superusers, so
clients can open 97.
Every client keeps its own pool:
| Client | Pool setting | Per replica |
|---|---|---|
| API | [extensions.database] max_connections (LUMIO__EXTENSIONS__DATABASE__MAX_CONNECTIONS) | 10 on staging (staging.toml), 20 in production (production.toml) |
| bot-module-worker | its extensions-database pool (ZAF-2409) | 5 |
API DDL pool (lumio_ext, ZAF-2513) | - | 2 |
Headroom (psql, backups, migrations) | - | 5 |
All pools at their maximum must fit:
P_api · N_api + 5 · M_worker + 2 + 5 ≤ 97
N_api and M_worker are the replica counts, P_api the API pool size.
- Staging with one API and one bot-module-worker replica:
10 + 5 + 2 + 5 = 22. - Production at
P_api = 20fits at most 3 API replicas (3 · 20 + 5 · 6 + 7 = 97with up to 6 workers). With more than 3 API replicas, setLUMIO__EXTENSIONS__DATABASE__MAX_CONNECTIONS=10before you scale up.
When you add a replica or raise a pool, recompute the sum first. If the sum
exceeds 97, new connections fail with too many clients already and the
affected requests and handlers fail. Raise max_connections only together with
the memory each connection needs.
Current usage per client:
docker exec staging-extensions-postgres psql -U lumio_ext -d lumio_extensions -Atc \
"SELECT application_name, client_addr, count(*) FROM pg_stat_activity GROUP BY 1, 2 ORDER BY 3 DESC"
Extensions PostgreSQL: roles
Superusers and roles with BYPASSRLS ignore row-level security, FORCE included.
So no app connects to the extensions PostgreSQL as a superuser (ZAF-2509).
| Role | Attributes | Used by |
|---|---|---|
postgres | bootstrap superuser (the image's POSTGRES_USER) | operators only, never in an app URL |
lumio_ext | LOGIN, owner of lumio_extensions | the API's DDL pool (LUMIO__EXTENSIONS__DATABASE__DDL_URL): migrations, schema provisioning, purge paths |
lumio_ext_dml | NOLOGIN group | target of every DML grant; the extensions migration fails if it is missing |
lumio_ext_api | LOGIN, member of lumio_ext_dml | the API's server-function pool (LUMIO__EXTENSIONS__DATABASE__URL) |
lumio_ext_worker | LOGIN, member of lumio_ext_dml | the bot-module-worker's extensions pool |
- None of them except
postgresisSUPERUSER,BYPASSRLS,CREATEROLE,CREATEDBorREPLICATION, none is a member of another role that has one of these attributes, and none is a member ofpg_execute_server_program,pg_read_server_filesorpg_write_server_files. A migration cannot create roles, so the roles come fromdev-stack/initdb/extensions/10-roles.sh, not from a migration. - No login has a
rolesetting (ALTER ROLE … SET role), and no app URL carriesoptions=-c role=…. The startup check reads the session role, andRESET ROLEwould return to a login it never read, so a session role that differs from the login stops the service. - The DML logins are also not members of
pg_read_all_dataorpg_write_all_data, and not oflumio_extor any other owner of the database,public.schema_registryor anext_*schema. - The DML logins have no
CREATEand noTEMPORARYon the database and noCREATEonpublic. Their membership isWITH INHERIT TRUE, SET FALSE, so they use the group's grants but cannotSET ROLEto it. - Each DML login has
statement_timeout = '2s'. It is set on the login user, because settings of aNOLOGINgroup are not inherited. CONNECTION LIMITper login follows the budget above:lumio_ext_api= API pool × API replicas,lumio_ext_worker= 5 × worker replicas,lumio_ext= 2 × API replicas + 2 formigrateandpsql. On staging that is10,5and4(EXT_PG_*_CONN_LIMIT).- The script also creates the
pg_freespacemapextension inlumio_extensionsand grantspg_stat_scan_tablestolumio_extandlumio_ext_api. The storage quota pass needs both to read the free space map. The extension is not trusted, so only a superuser can create it, and a migration cannot.lumio_ext_workerdoes not get the grant.
The script runs automatically on an empty data volume. It is idempotent: run it again to change a password or a connection limit. It re-applies the attributes, grants and settings each time:
docker exec -i -e EXT_DDL_PASSWORD -e EXT_API_PASSWORD -e EXT_WORKER_PASSWORD \
-e EXT_DDL_CONN_LIMIT=4 -e EXT_API_CONN_LIMIT=10 -e EXT_WORKER_CONN_LIMIT=5 \
staging-extensions-postgres bash -s < dev-stack/initdb/extensions/10-roles.sh
Check the roles before a deploy
The API and the bot-module-worker check their roles at startup and refuse to
start when one breaks a rule above (ZAF-2509 A1). Memberships count even when
they are granted WITH INHERIT FALSE, SET TRUE, because SET ROLE restores
the rights. Run this check before you deploy a build with the check, and after
you change a role by hand:
docker exec staging-extensions-postgres psql -U postgres -d lumio_extensions -c "
SELECT r.rolname,
r.rolsuper OR r.rolbypassrls OR r.rolcreaterole OR r.rolcreatedb
OR r.rolreplication AS privileged_attribute,
ARRAY(SELECT p.rolname FROM pg_roles p
WHERE p.oid <> r.oid
AND (p.rolsuper OR p.rolbypassrls OR p.rolcreaterole OR p.rolcreatedb
OR p.rolreplication)
AND pg_has_role(r.oid, p.oid, 'MEMBER')) AS privileged_memberships,
ARRAY(SELECT c FROM pg_db_role_setting s, unnest(s.setconfig) c
WHERE s.setrole = r.oid
AND s.setdatabase IN (0, (SELECT oid FROM pg_database
WHERE datname = current_database()))
AND c LIKE 'role=%') AS role_settings,
ARRAY(SELECT s.rolname FROM pg_roles s
WHERE s.rolname IN ('pg_execute_server_program', 'pg_read_server_files',
'pg_write_server_files')
AND pg_has_role(r.oid, s.oid, 'MEMBER')) AS server_roles,
ARRAY(SELECT s.rolname FROM pg_roles s
WHERE s.rolname IN ('pg_read_all_data', 'pg_write_all_data')
AND pg_has_role(r.oid, s.oid, 'MEMBER')) AS all_data_roles,
has_database_privilege(r.oid, current_database(), 'CREATE')
OR has_database_privilege(r.oid, current_database(), 'TEMPORARY') AS db_create_temp,
ARRAY(SELECT DISTINCT o::regrole::text FROM (
SELECT datdba AS o FROM pg_database WHERE datname = current_database()
UNION SELECT relowner FROM pg_class
WHERE oid = to_regclass('public.schema_registry')
UNION SELECT nspowner FROM pg_namespace WHERE nspname LIKE 'ext\_%') x
WHERE pg_has_role(r.oid, o, 'MEMBER')) AS owner_memberships
FROM pg_roles r
WHERE r.rolname IN ('lumio_ext', 'lumio_ext_api', 'lumio_ext_worker')
ORDER BY 1"
Expected:
| Role | privileged_attribute | privileged_memberships | role_settings | server_roles | all_data_roles | db_create_temp | owner_memberships |
|---|---|---|---|---|---|---|---|
lumio_ext (DDL) | f | {} | {} | {} | not checked | not checked | not checked (it is the owner) |
lumio_ext_api, lumio_ext_worker (DML) | f | {} | {} | {} | {} | f | {} |
Any other value stops that service at startup. The query cannot see an
options=-c role=… in an app URL: check LUMIO__EXTENSIONS__DATABASE__URL
and LUMIO__EXTENSIONS__DATABASE__DDL_URL for it by hand. A t in privileged_attribute
means the role could ignore row-level security or create roles and databases. A volume created before ZAF-2512 used POSTGRES_USER=lumio_ext, so
lumio_ext is its bootstrap superuser: switch it with one of the two paths
below.
Switching an existing volume to the roles
Stop the API and the bot-module-worker first. A renamed role keeps its open sessions, so the old superuser would stay connected until they restart.
Count what the volume holds:
docker exec staging-extensions-postgres psql -U lumio_ext -d lumio_extensions -Atc \
"SELECT count(*) FROM pg_namespace WHERE nspname LIKE 'ext\_%'"
A. The volume holds no ext_* schema: recreate it. Back it up anyway
(see Backups). Stop the container, move its data directory aside,
set the EXT_PG_* passwords from .env.infra.example and run
docker compose -f infra.docker-compose.yml up -d extensions-postgres. The
script creates the roles on the empty volume.
B. The volume holds data: switch in place. The bootstrap superuser cannot
lose SUPERUSER, and REASSIGN OWNED refuses it. So it gets a new name, and a
new lumio_ext takes over its objects. Back up first.
# 1. A new bootstrap superuser, created by the old one.
docker exec staging-extensions-postgres psql -v ON_ERROR_STOP=1 -U lumio_ext -d postgres \
-c "CREATE ROLE postgres SUPERUSER LOGIN PASSWORD '<EXT_PG_BOOTSTRAP_PASSWORD>'"
# 2. Rename the old bootstrap superuser and take its login away.
docker exec staging-extensions-postgres psql -v ON_ERROR_STOP=1 -U postgres -d postgres \
-c "ALTER ROLE lumio_ext RENAME TO lumio_ext_legacy_bootstrap" \
-c "ALTER ROLE lumio_ext_legacy_bootstrap NOLOGIN"
# 3. Create lumio_ext, the group and the DML logins, and hand every object in
# lumio_extensions to lumio_ext.
docker exec -i -e POSTGRES_USER=postgres -e EXT_DDL_PASSWORD -e EXT_API_PASSWORD \
-e EXT_WORKER_PASSWORD -e EXT_DDL_CONN_LIMIT=20 -e EXT_API_CONN_LIMIT=10 \
-e EXT_WORKER_CONN_LIMIT=5 \
staging-extensions-postgres bash -s < dev-stack/initdb/extensions/10-roles.sh
If EXT_DDL_PASSWORD is the password the API URL already carries, the API
starts again unchanged and now connects as the non-superuser owner. Until the
API's URL moves to lumio_ext_api, its whole pool connects as lumio_ext, so
EXT_DDL_CONN_LIMIT must be at least LUMIO__EXTENSIONS__DATABASE__MAX_CONNECTIONS
(20 above). With the budget value 4, the fifth pool connection fails with
too many connections for role "lumio_ext". After the URL switch, lower it:
ALTER ROLE lumio_ext CONNECTION LIMIT 4, and keep the API pool at or below
lumio_ext_api's limit. Then set
POSTGRES_USER=postgres and POSTGRES_PASSWORD in the host compose to match:
the image reads them only on an empty volume, but the healthcheck uses
postgres.
Rollback for B: as postgres, rename lumio_ext to
lumio_ext_new, rename lumio_ext_legacy_bootstrap back to lumio_ext and
give it LOGIN again. It still holds SUPERUSER, so it can still read every
object.
Check the result
for u in lumio_ext_api lumio_ext_worker; do
docker exec -e PGPASSWORD=<password of $u> staging-extensions-postgres \
psql -h 127.0.0.1 -U "$u" -d lumio_extensions \
-c "SELECT current_user, current_setting('statement_timeout')" \
-c "CREATE SCHEMA ext_probe" -c "CREATE TEMP TABLE probe (x int)"
done
Expected per login: 2s, then permission denied for database lumio_extensions and permission denied to create temporary tables. Run the
role query from Check the roles before a deploy:
lumio_ext must match the DDL row of its expected table.
App URLs
The API needs both URLs, and they must differ (the API refuses to start otherwise, ZAF-2513):
| Variable | Role |
|---|---|
LUMIO__EXTENSIONS__DATABASE__URL | lumio_ext_api |
LUMIO__EXTENSIONS__DATABASE__DDL_URL | lumio_ext |
| bot-module-worker extensions URL (ZAF-2409) | lumio_ext_worker |
Point LUMIO__EXTENSIONS__DATABASE__URL at lumio_ext_api only with the API
release that reads DDL_URL. An older API provisions schemas through
LUMIO__EXTENSIONS__DATABASE__URL, which fails without CREATE. Until then it
stays on lumio_ext, which is no longer a superuser after the switch above.
Extensions PostgreSQL: storage quota measurement
Every 15 minutes the API's data-deletion worker sizes each extension schema
for the 1 GiB storage quota. It writes the file size to
schema_registry.size_bytes and the space PostgreSQL can reuse to
reclaimable_bytes. Only size_bytes - reclaimable_bytes counts against the
quota. A blocked extension is unblocked once that drops to 90 % of the quota.
The reusable space comes from the free space map when the pg_freespacemap
extension is installed and the API's role may call pg_freespace. That reads
1 byte per table page. pg_freespacemap is not a trusted extension, so a
superuser creates it once. 10-roles.sh (see the roles section above) does
this and grants pg_stat_scan_tables to lumio_ext and lumio_ext_api. By
hand, as the bootstrap superuser:
docker exec staging-extensions-postgres psql -U postgres -d lumio_extensions \
-c "CREATE EXTENSION IF NOT EXISTS pg_freespacemap" \
-c "GRANT pg_stat_scan_tables TO lumio_ext, lumio_ext_api"
The pass runs on the API's DDL pool (LUMIO__EXTENSIONS__DATABASE__DDL_URL,
role lumio_ext), which is why lumio_ext holds pg_stat_scan_tables.
Without the extension nothing counts as reusable and the file size alone
decides: deleted rows then unblock an extension only once the file shrinks.
ext_* tables force row-level security for their owner too, so no role can
scan every account's live rows as a fallback. A failed measurement also counts
nothing as reusable.
Check the state per extension:
docker exec staging-extensions-postgres psql -U lumio_ext -d lumio_extensions -Atc \
"SELECT schema_name, size_bytes, reclaimable_bytes, writes_blocked, size_measured_at FROM schema_registry ORDER BY size_bytes DESC NULLS LAST LIMIT 20"
Data directories
The two images keep their data in different places. A volume mounted at the wrong path leaves the data in the container's writable layer. Docker then deletes the data the next time it recreates the container, for example after an image pull.
| Service | Image | PGDATA | Mount the volume at |
|---|---|---|---|
| PostgreSQL | postgres:18-alpine | /var/lib/postgresql/18/docker | /var/lib/postgresql |
| Extensions PostgreSQL | postgres:18-alpine | /var/lib/postgresql/18/docker | /var/lib/postgresql |
| TimescaleDB | timescale/timescaledb-ha:pg18 | /home/postgres/pgdata/data | /home/postgres/pgdata |
postgres:18does not start if a volume is mounted at/var/lib/postgresql/data. It reports that path as an "unused mount/volume".timescaledb-haruns as uid 1000. A host directory mounted at/home/postgres/pgdatamust be owned by1000:1000.
Check where a running database keeps its data:
docker exec staging-tsdb psql -U lumio -d lumio_tsdb -Atc 'SHOW data_directory'
docker inspect staging-tsdb --format '{{json .Mounts}}'
The data_directory must be inside one of the listed mounts.
Backups
The daily encrypted off-host backup of all four stores, its retention, the restore test and the restore log are described in Backups. The notes below explain what a TimescaleDB backup must contain and how a manual restore works.
A TimescaleDB backup must contain the hypertable chunks (schema
_timescaledb_internal) and the continuous-aggregate materializations. Both are
ordinary tables in TimescaleDB's internal schemas, so pg_dump of the whole
database includes them. Never dump single hypertables.
pg_dump may warn about circular foreign keys in the TimescaleDB catalog. The
warning is expected. A restore needs the same TimescaleDB version as the dump,
and it must run between timescaledb_pre_restore() and
timescaledb_post_restore():
CREATE DATABASE lumio_tsdb;
\c lumio_tsdb
CREATE EXTENSION IF NOT EXISTS timescaledb;
SELECT timescaledb_pre_restore(); -- stops the background jobs
-- shell: pg_restore -U lumio -d lumio_tsdb < lumio_tsdb.dump
SELECT timescaledb_post_restore(); -- starts them again
Applying a change on the staging host
The host runs its own compose files, so these steps change staging.
- Find where the data is now. Run the two commands under
Data directories. If
data_directoryis not inside a mount, the data exists only in the container. Back it up before anything else, because recreating the container deletes it. - Check Redis memory. Run
docker exec staging-redis redis-cli -a "$REDIS_PASSWORD" INFO memory.used_memorymust be well below the newREDIS_MAXMEMORY. Otherwise Redis rejects writes as soon as it starts. - Back up the data layer:
/opt/lumio-backup/lumio-backup.sh backup(see Backups). - Update the host copy of
infra.docker-compose.ymlto match this file. Set the sizing variables in the project.env, andchown -R 1000:1000the TimescaleDB data directory. - Recreate the containers:
docker compose -f infra.docker-compose.yml up -d. - Check the result. Run
SHOW shared_buffers;andSHOW shared_preload_libraries;in both databases, andredis-cli CONFIG GET maxmemory-policy(expected:noeviction). Createpg_stat_statementsin both databases. In Prometheus,up{job="redis"}must be1.
Rollback: restore the previous host copy of the compose file and run step 5 again. Moving the data directory cannot be undone by a rollback: do it only after the backup in step 3 succeeded.