Best Practices on Strata Documentation

Web UI (Strata Console)

Mon, 01 Jan 0001 00:00:00 +0000

Strata Console (Web UI) — Operator Guide#

The Strata Console is an embedded read-only web UI for cluster operators. It ships in the same binary as the gateway (go:embed) and is served at /console/ on the gateway HTTP port. No separate process, no separate deploy.

This document is the Phase 1 (foundation) operator guide. Phase 2 (admin write actions) and Phase 3 (debug tooling — heatmaps, slow queries, OTel trace browser) ship in their own cycles.

Tracing

Mon, 01 Jan 0001 00:00:00 +0000

Tracing#

Strata emits OpenTelemetry spans across every meaningful tier of the request and worker paths. Each span carries the strata.component attribute so an operator can filter the entire gateway path or the entire worker path in one Jaeger query, and each worker iteration appears as a discrete trace so a slow / failing tick is easy to correlate with the meta + data ops it triggered.

The wire-up env vars + tail-sampler + ring buffer behaviour live in Monitoring. This page is the operator-facing reference for what spans exist, how they are named, and how to filter them.

GC + lifecycle tuning

Mon, 01 Jan 0001 00:00:00 +0000

GC + lifecycle tuning#

The gc and lifecycle workers run inside every gateway replica. The operator-facing knobs split into two layers:

Phase 1 — per-replica fan-out: STRATA_GC_CONCURRENCY / STRATA_LIFECYCLE_CONCURRENCY cap the goroutine count inside one replica’s worker.
Phase 2 — multi-replica sharding: STRATA_GC_SHARDS shards the leader-election space so N replicas process disjoint slices of the queue in parallel.

Both layers compose: a 3-replica deploy with STRATA_GC_SHARDS=3 and per-replica STRATA_GC_CONCURRENCY=64 runs three independent fan-outs against disjoint shards.

Quotas + billing

Mon, 01 Jan 0001 00:00:00 +0000

Quotas + billing#

Strata enforces hard per-bucket and per-user storage quotas at PUT-validate time and emits a nightly per-(bucket, storage class) usage aggregate that an external invoice generator can consume. The shape mirrors AWS / RGW so a RADOS Gateway tenant migrating to Strata sees the same QuotaExceeded 403 on overage.

The cycle stops at the usage feed. Invoice ledger / payment integration lives in a separate ops-layer service that consumes usage_aggregates.

Production hardening

Mon, 01 Jan 0001 00:00:00 +0000

Production hardening#

12-line checklist to run through before flipping a Strata replica into prod traffic. Each line links to the runbook section that explains the knob, the metric, and the failure mode it prevents. Every knob is opt-in and zero-by-default — running through the list flips a memory / lab deployment into a prod-ready shape without touching code.

#	Check	Runbook
1	HTTP server timeouts non-zero (`STRATA_HTTP_READ_HEADER_TIMEOUT=10s` / `STRATA_HTTP_READ_TIMEOUT=60s` / `STRATA_HTTP_WRITE_TIMEOUT=30m` / `STRATA_HTTP_IDLE_TIMEOUT=120s` / `STRATA_HTTP_MAX_HEADER_BYTES=1048576`). Defaults already match — only verify if you tuned them.	STRATA_HTTP_*
2	TLS terminated on the gateway (`STRATA_TLS_CERT_FILE` / `STRATA_TLS_KEY_FILE`, or `STRATA_TLS_CERT_DIR` for SNI multi-tenant) — or behind an ingress with `STRATA_TRUSTED_PROXIES` set to the ingress source CIDR.	TLS termination — shapes B/C
3	`STRATA_TLS_MIN_VERSION=TLS1.2` (default) and `STRATA_TLS_CIPHER_PROFILE=mozilla-modern` (default). Bump `MIN_VERSION=TLS1.3` if every client supports it.	TLS shapes
4	Cert hot-reload enabled (`STRATA_TLS_RELOAD_INTERVAL=60s`, default) so cert-manager / Vault PKI rotation is picked up without restart.	cert-manager recipe
5	Admin / console / metrics on a separate listener (`STRATA_ADMIN_LISTEN=127.0.0.1:9001` recommended; loopback or RFC1918 only). Optionally pin operator client certs via `STRATA_ADMIN_TLS_CLIENT_CA_FILE`.	Shape C — split admin listener
6	`STRATA_TRUSTED_PROXIES` set to the ingress / LB source CIDR. Default empty = `X-Forwarded-*` ignored. Required for the `Secure` cookie flag + audit-log client-IP fidelity behind any proxy.	Trusted proxies — README breaking change
7	Per-IP + per-key ingress rate limit on (`STRATA_RATE_LIMIT_PER_IP=N` and / or `STRATA_RATE_LIMIT_PER_KEY=N`). Default 0 = disabled. Refusal returns HTTP 429 + `<Code>SlowDown</Code>`.	STRATA_RATE_LIMIT_*
8	Cassandra mTLS (`STRATA_CASSANDRA_TLS_CA_FILE` + `STRATA_CASSANDRA_TLS_CERT_FILE` + `STRATA_CASSANDRA_TLS_KEY_FILE`). `SKIP_VERIFY` must be `false` (default).	Cassandra mTLS
9	TiKV mTLS (`STRATA_TIKV_TLS_CA_FILE` is required when any other TLS knob is set — the upstream silently downgrades on empty CA). PD endpoints accept `https://`.	TiKV mTLS
10	S3-upstream mTLS (`STRATA_S3_TLS_*` global default; per-cluster `tls` override on `STRATA_S3_CLUSTERS` JSON wins outright per cluster).	S3-upstream mTLS
11	RADOS cephx in place — `STRATA_RADOS_KEYRING` populated; `ms_cluster_mode=secure` set in `ceph.conf` if wire-level confidentiality is required. (No Strata-side TLS knob for RADOS.)	RADOS uses cephx
12	Prometheus alert on `sum(strata_backend_tls_skip_verify) > 0` (any backend with `SKIP_VERIFY=true`) AND on `rate(strata_ingress_rate_limit_refused_total[5m]) > N` (sustained client floods).	Monitoring — alert recipes

What this checklist closes#

Every line above closes a P0 gap from the 2026-05-25 prod-readiness audit:

S3 multi-cluster routing

Mon, 01 Jan 0001 00:00:00 +0000

S3 multi-cluster routing#

The S3 data backend supports routing per storage class to a distinct (cluster, bucket) pair. Two envs hold the full config — STRATA_S3_CLUSTERS (JSON array of bucket-less cluster specs) and STRATA_S3_CLASSES (JSON object mapping storage class names to {cluster, bucket} tuples). Adding or removing a cluster requires a gateway restart; multi-replica deployments hide per-instance downtime via rolling restart.

This page is the operator guide. For the conceptual S3 backend overview see S3 data backend.

Compliance

Mon, 01 Jan 0001 00:00:00 +0000

Compliance#

Strata implements S3 Object Lock with COMPLIANCE and GOVERNANCE retention modes plus legal hold. COMPLIANCE rows are immutable until their RetainUntilDate elapses — not even the bucket owner can shorten the retention window or delete the object early. The audit log records every COMPLIANCE-mode write and every expiry the lifecycle worker performs so auditors can grep audit_log for retention-policy events with a single action LIKE 'objectlock:%' clause.

This page is the operator workflow guide. For the audit-log shape see Monitoring — audit log; for the underlying retention semantics see the S3 Compatibility matrix.

Placement + rebalance

Mon, 01 Jan 0001 00:00:00 +0000

Placement + rebalance#

Once you run more than one RADOS / S3 cluster behind a single Strata deployment, you also need to control which cluster a bucket’s chunks land on and how to migrate old chunks when a new cluster joins or an old cluster is being retired. Strata ships both pieces:

A per-bucket placement policy (meta.Bucket.Placement — a {cluster: weight} map) that the chunk PUT path consults via a stable hash-mod router.
A leader-elected rebalance worker (strata server --workers=rebalance) that walks every bucket with a non-nil policy, compares the actual per-cluster chunk distribution to the policy’s target, and copies chunks A → B until the two match.

A bucket without a policy (Placement == nil) behaves exactly as before — chunks land on the storage class’s default cluster. No migration, no schema bump, no behavior change. The policy + worker are both opt-in.

Billing

Mon, 01 Jan 0001 00:00:00 +0000

Billing#

Strata does not ship an invoice generator. It ships the inputs an external invoice generator can consume: a daily per-(bucket, storage class) row in usage_aggregates whose byte_seconds field integrates the bucket-usage counter across the UTC day with the trapezoid rule.

This page documents the integration math, the sampling cadence, and the env knob that tunes accuracy vs. meta-backend cost. For the live quota counter, the QuotaExceeded shape, the reconcile worker, and the admin API surface, see Quotas + billing.