Strata Documentation

ADR-0001: Skip RADOS omap for bucket index

Mon, 01 Jan 0001 00:00:00 +0000

ADR-0001: Skip RADOS omap for bucket index#

Status#

Accepted — April 2026

Context#

Strata is positioned as a drop-in replacement for Ceph RGW. RGW stores the bucket index in a small set of RADOS objects, each carrying an omap (ordered key→value map) that lists the bucket’s objects in lex order. The omap is convenient — listing is a native ordered scan — but it has hard scale ceilings:

The omap of a single index object lives in a single placement group. All listing traffic and all index mutations for that shard land on one OSD, capped by that OSD’s IOPS budget.
RGW’s only mitigation is bucket-index resharding (radosgw-admin bucket reshard). Resharding rewrites the entire omap, must quiesce or rate-limit write traffic during the cut-over, and the shard count tops out before the largest production buckets do — beyond roughly 100M objects a single bucket exhausts the resharding ceiling and starts taking IOPS hits regardless.
The contract is opaque to the layer above. We cannot trade consistency for throughput, partition by a different key, or fan out across heterogeneous storage tiers without re-implementing the scan path.

We considered (a) keeping omap with aggressive resharding and (b) moving the index to a separate ordered store. (a) inherits RGW’s ceiling; the project goal is to lift it, not match it.

ADR-0002: Derive IsLatest at read time

Mon, 01 Jan 0001 00:00:00 +0000

ADR-0002: Derive `IsLatest` at read time#

Status#

Accepted — April 2026

Context#

S3 versioning requires every object row to expose an IsLatest bit: the most recent (un-deleted) version of a key carries IsLatest=true, every older version carries IsLatest=false. The naive implementation flips the bit on the previous head whenever a new PUT lands — UPDATE objects SET is_latest=false WHERE bucket_id=? AND key=? AND version_id=<prev> immediately after inserting the new version row.

ADR-0003: Single manifest blob column

Mon, 01 Jan 0001 00:00:00 +0000

ADR-0003: Single `manifest` blob column for object metadata#

Status#

Accepted — April 2026

Context#

Per-object metadata in Strata is rich and growing: chunk OIDs, chunk sizes, MD5 / SHA256 digests, SSE wrap context, multipart part boundaries (PartChunkCounts, PartChunks), backend routing hints (BackendRef.Cluster), and several smaller fields. A normalised schema would dedicate one column per field, possibly with side tables for the variable-length arrays (chunks, parts).

The cost of normalised columns on Cassandra is two-fold:

ADR-0004: One leader lease per background worker

Mon, 01 Jan 0001 00:00:00 +0000

ADR-0004: One leader lease per background worker#

Status#

Accepted — April 2026

(Reconsideration tracked under ROADMAP ## Consolidation & validation. If the supervisor collapse lands, this ADR will be marked Superseded by ADR-XXXX.)

Context#

strata server ships roughly ten background workers — gc, lifecycle, replicator, notify, access-log, inventory, audit-export, manifest-rewriter, rebalance, quota-reconcile, usage-rollup. Each of them must run on at most one replica at a time: GC must not double-delete, lifecycle must not duplicate transitions, etc. Some shape of singleton election is required regardless of the deploy footprint.

Auth

Mon, 01 Jan 0001 00:00:00 +0000

Auth#

The auth layer lives under internal/auth/. Every request enters through auth.Middleware, which verifies an AWS SigV4 signature, derives a stable identity, and stamps the result onto the request context. The router and handlers downstream never re-derive identity — they read it from auth.FromContext(ctx).Owner.

SigV4 verification#

internal/auth/sigv4.go implements the standard four-step canonicalisation:

Parse Authorization (or X-Amz-* query parameters for presigned URLs) to extract AccessKey, Scope, SignedHeaders, and Signature.
Look up the secret for the access key via the configured static credentials store (internal/auth/static.go) — no IdP federation in this cycle.
Build the canonical request string from Method, URL.Path, the sorted SignedHeaders, and the body hash (x-amz-content-sha256, which may be the literal sentinel STREAMING-AWS4-HMAC-SHA256-PAYLOAD for chunked uploads — see below).
Recompute the signature with the derived signing key and constant-time compare. Mismatch returns ErrSignatureInvalid (HTTP 403, AWS code SignatureDoesNotMatch).

The middleware MUST run before any URL rewriting. The signed canonical string includes the original Host header and the original URL.Path — if the router rewrites either before verification, the signature breaks. See the Router page for the order.

Binary consolidation

Mon, 01 Jan 0001 00:00:00 +0000

Migrating to the unified `strata` binary#

Strata used to ship eleven cmd/* binaries — one gateway plus a long tail of single-purpose worker daemons. The codebase now ships exactly one:

strata — the gateway plus every long-running background worker (selected at startup via STRATA_WORKERS= / --workers=), with one-shot operator commands available as strata admin <subcommand> against the same binary.

This guide is the operator-facing checklist for upgrading an existing deploy. There is no schema migration, no data migration, and no env-var rename — the move is purely a packaging change.

Drain a cluster

Mon, 01 Jan 0001 00:00:00 +0000

Drain a cluster#

Taking a data cluster out of rotation is a five-step workflow: preview impact → drain → watch progress → wait for deregister-ready → remove from env. Strata exposes each step through the admin API and the operator console so you can drive the workflow from the browser or from a shell script.

This page is the day-2 runbook. The Drain & rebalance concept covers the lifecycle states; the Placement + rebalance best-practice covers tuning the rebalance throughput and the cluster-state machine in depth.

GC + Lifecycle scaling

Mon, 01 Jan 0001 00:00:00 +0000

GC + Lifecycle worker scaling benchmark (Phase 1 + Phase 2)#

Phase 2 (US-006) lands multi-leader numbers below Phase 1’s single-leader baseline. The two-section structure is intentional — Phase 1 measured the per-leader concurrency cap, Phase 2 measures the multiplier from running N replicas in parallel under STRATA_GC_SHARDS=N.

Phase 1 — single-leader concurrency cap#

Quantifies the throughput curve for internal/gc.Worker and internal/lifecycle.Worker across the per-worker bounded-errgroup fan-out introduced in cycle ralph/gc-lifecycle-scale (US-001 / US-002). The harness lands as strata admin bench-gc + strata admin bench-lifecycle (US-003) and is wired into make bench-gc / make bench-lifecycle to drive the canonical lab-tikv stack.

Router

Mon, 01 Jan 0001 00:00:00 +0000

Router#

The router lives in internal/s3api/server.go. It is a single Server.ServeHTTP method that classifies every request along three axes:

Special prefix — /admin/... is the embedded operator console JSON API and bypasses the S3 dispatch entirely.
Bucket vs object scope — the URL path is split at the first slash into (bucket, key). Empty bucket → service-level (ListBuckets, IAM actions). Empty key → bucket-scoped. Both populated → object-scoped.
Query-string sub-resource — within bucket and object scope, the sub-operation is dispatched by query parameter (?cors, ?policy, ?lifecycle, ?uploads, ?uploadId=, ?tagging, …) plus the HTTP method. This is the AWS S3 wire shape; we mirror it.

Dispatch order#

ServeHTTP(w, r):
 1. extractAccessPointAlias(r.Host) # alias.<host> -> rewrite to /<bucket>/...
 2. extractVHostBucket(r.Host, ...) # *.s3.local -> rewrite to /<bucket>/...
 3. if path starts with /admin/ -> handleAdmin
 4. splitPath(r.URL.Path) -> (bucket, key)
 5. bucket == "" -> IAM action / ListBuckets / DLQ-audit listings
 key == "" -> handleBucket(...)
 default -> handleObject(...)

Auth middleware runs before the access-point and vhost rewrites because those rewrites only mutate r.URL.Path after SigV4 has already validated the original. See Auth for the ordering rationale.

S3 surface

Mon, 01 Jan 0001 00:00:00 +0000

S3 surface#

Strata speaks the AWS S3 REST API over HTTP. The goal is compatibility with the Amazon S3 client ecosystem — the same aws s3 / aws s3api / mc commands and the same SDK calls work against Strata with only the endpoint URL changed. Compatibility is measured against Ceph’s upstream s3-tests suite; the running pass rate lives on the S3 Compatibility page.

The rest of this page introduces each major operation family. None of the details below depend on which metadata or data backend you pick — the S3 surface is identical across backends.

ScyllaDB

Mon, 01 Jan 0001 00:00:00 +0000

ScyllaDB metadata backend#

ScyllaDB is supported as a drop-in replacement for Apache Cassandra. Strata’s metadata access path is gocql + CQL — no Cassandra-specific extensions are used — so the same internal/meta/cassandra backend talks to either cluster. The storetest.Run contract suite (35+ scenarios covering bucket/object/multipart LWT semantics, sharded objects listing, GC and notification queues, audit log, versioning null literal, access points, etc.) passes against both.

Compatibility status#

Code: zero gateway changes required to switch from Cassandra to ScyllaDB. Point STRATA_CASSANDRA_HOSTS (and friends) at a Scylla cluster and bring up the gateway as usual.
Schema: same DDL — internal/meta/cassandra/schema.go::tableDDL applies as-is. No ALTER differences.
Consistency: gateway uses LOCAL_QUORUM for reads/writes and LOCAL_SERIAL for LWT (internal/meta/cassandra/session.go). ScyllaDB honours both.
LWT: bucket creation, versioning toggle, multipart Complete, GC ack, storage-class transitions, lifecycle worker leases — all rely on IF NOT EXISTS / IF EXISTS. ScyllaDB 5.0 ships LWT on the Paxos protocol by default; ScyllaDB 5.4+ optionally enables Raft-backed LWT (raft_lwt-enabled tables) which dramatically reduces tail latency. The storetest contract is agnostic to the underlying coordination protocol; it only asserts the linearizable observable behaviour, which both Paxos and Raft modes provide.
Sharded objects listing: Strata fans out N=64 shard partitions in parallel (cassandra.Store.ListObjects). ScyllaDB’s per-shard CPU pinning is a natural fit for this access pattern — token-aware queries land on the right shard with a single hop.

Deployment notes#

Connection#

STRATA_CASSANDRA_HOSTS=scylla-1.example,scylla-2.example,scylla-3.example
STRATA_CASSANDRA_KEYSPACE=strata
STRATA_CASSANDRA_LOCAL_DC=datacenter1
STRATA_CASSANDRA_REPLICATION='{"class":"NetworkTopologyStrategy","datacenter1":"3"}'

STRATA_CASSANDRA_LOCAL_DC should match the Scylla dc you want token-aware routing to prefer. Multi-DC deployments work identically to Cassandra: NetworkTopologyStrategy keyspace + per-DC RF.

Single-node deployment

Mon, 01 Jan 0001 00:00:00 +0000

Single-node Strata#

A single-node deployment runs one strata process. It is the simplest shape Strata ships and the recommended starting point for labs, evaluation, and single-tenant pilots. The gateway is HTTP-stateless — durability lives in the metadata + data backends — so a single replica is not an HA shape: the box is the SPOF for HTTP traffic. For HA, see Multi-replica cluster.

Prerequisites#

One Linux host (or a macOS workstation for dev).
Docker (for the full stack) or Go 1.23+ (for the in-memory smoke).
2 vCPU + 1 GiB RSS baseline (see Sizing below).

The pure-memory path needs no other software. The Cassandra-backed path needs cassandra:5.0 reachable on :9042. The full stack (Cassandra + RADOS) is bootstrapped from the bundled compose file.

STRATA_* environment variables

Mon, 01 Jan 0001 00:00:00 +0000

`STRATA_*` environment variables#

Every STRATA_* knob, grouped by the layer that consumes it. CLI flags on strata server (--listen, --vhost-pattern, --log-level, --workers=) override the matching env var.

Web UI (Strata Console)

Mon, 01 Jan 0001 00:00:00 +0000

Strata Console (Web UI) — Operator Guide#

The Strata Console is an embedded read-only web UI for cluster operators. It ships in the same binary as the gateway (go:embed) and is served at /console/ on the gateway HTTP port. No separate process, no separate deploy.

This document is the Phase 1 (foundation) operator guide. Phase 2 (admin write actions) and Phase 3 (debug tooling — heatmaps, slow queries, OTel trace browser) ship in their own cycles.

PUT flow

Mon, 01 Jan 0001 00:00:00 +0000

PUT flow#

A single S3 PutObject traverses the gateway, the metadata store, and the data backend. The picture below names the components on the hot path so the prose that follows can stay focused on the interesting choices — streaming chunk decode, the manifest compare-and-set, and the failure modes when one tier wins a race against another.

Sequence diagram#

sequenceDiagram
 autonumber
 participant C as S3 client
 participant GW as s3api.Server
 participant Auth as auth.Middleware
 participant Meta as meta.Store
 participant Data as data.Backend
 C->>GW: PUT /bucket/key (+ SigV4 headers, streaming body)
 GW->>Auth: verify signature (canonical request, chain HMAC if streaming)
 Auth-->>GW: identity{Owner, AccessKeyID}
 GW->>Meta: LookupBucket(bucket) → policy + placement
 GW->>Data: PutChunks(stream, placement, classSpec)
 Data->>Data: split body into 4 MiB chunks, write to selected cluster
 Data-->>GW: manifest{chunks[], etag, size, BackendRef}
 GW->>Meta: SetObjectManifest(bucket, key, prevVersion, manifest) — compare-and-set
 alt CAS applied
 Meta-->>GW: applied=true, version=v
 GW-->>C: 200 OK, ETag, x-amz-version-id
 else CAS rejected (concurrent writer landed first)
 Meta-->>GW: applied=false
 GW->>Data: EnqueueChunkDeletion(losingChunks) — GC absorbs the orphan
 GW-->>C: 200 OK with the winning manifest (retry-safe)
 end

Step-by-step#

SigV4 verify. auth.Middleware canonicalises the request, computes the expected signature against the secret in auth.StaticStore, and rejects on mismatch. Streaming chunk uploads (aws-chunked) carry a chain HMAC validated incrementally by auth.streamingReader so a torn body fails before the manifest is touched. See Auth.

Multi-cluster routing

Mon, 01 Jan 0001 00:00:00 +0000

Multi-cluster routing#

Strata can front multiple data clusters (multiple RADOS pools, multiple upstream S3 endpoints, or a mix). Every PUT picks exactly one cluster before any chunks land. The picker is a thin layer over three inputs: the bucket’s placement policy, the per-cluster weights from the cluster_state table, and the storage-class spec inferred from the request headers.

Flowchart#

flowchart TD
 PUT["s3api.Server.putObject"] --> Spec["Resolve storage-class spec<br/>(default / standard-ia / glacier / @cluster)"]
 Spec --> Pin{"spec.Cluster != ''<br/>(@cluster suffix?)"}
 Pin -- yes --> PickPinned["clusterForPlacement(pinnedID)"]
 Pin -- no --> BPolicy{"bucket.Placement<br/>!= nil?"}
 BPolicy -- yes --> Effective["placement.EffectivePolicy(<br/>bucketPolicy, mode, weights, states)"]
 Effective --> EffectiveNonEmpty{"non-empty?"}
 EffectiveNonEmpty -- yes --> Wheel["placement.PickClusterExcluding(<br/>policy, drainSet)"]
 EffectiveNonEmpty -- no --> StrictCheck{"mode == strict?"}
 StrictCheck -- yes --> Refuse[/"503 DrainRefused<br/>(compliance pin)"/]
 StrictCheck -- no --> Default["Synthesised default policy<br/>from cluster weights"]
 Default --> Wheel
 BPolicy -- no --> Default
 Wheel --> Drain{"selected cluster<br/>draining?"}
 Drain -- yes --> Refuse
 Drain -- no --> Put["data.Backend.PutChunks(...)"]
 PickPinned --> DrainPin{"pinned cluster<br/>draining?"}
 DrainPin -- yes --> Refuse
 DrainPin -- no --> Put

The picker contract#

placement.PickCluster (and its drain-aware sibling PickClusterExcluding) lives in internal/data/placement/. The inputs are deterministic:

Docker Compose

Mon, 01 Jan 0001 00:00:00 +0000

Docker Compose deployment#

The bundled deploy/docker/docker-compose.yml is the canonical reference shape for a Strata stack on one host. Bare docker compose up -d brings up the TiKV-default 2-replica lab (PD + TiKV + ceph + ceph-b + strata-a + strata-b + nginx LB + prometheus + grafana). The Cassandra-backed regression lab lives under --profile cassandra so an operator can make up-cassandra to validate the Cassandra meta backend side by side.

Meta store

Mon, 01 Jan 0001 00:00:00 +0000

Meta store#

internal/meta/store.go defines meta.Store, the interface every metadata backend implements. The contract is intentionally narrow: only the operations the S3 surface needs, and only with the consistency primitives the backends can all support without bolting a coordinator in front.

Three production-eligible backends satisfy meta.Store:

Backend	When to pick	Notes
`memory`	Tests, smoke pass, single-process demos	In-process tree-map. Naturally ordered. No durability.
`cassandra`	Multi-replica, scale tested against the s3-tests suite	ScyllaDB drops in unchanged (CQL-compatible). Sharded objects table, fan-out + heap-merge listing.
`tikv`	Multi-replica, prefer ordered scans	Native KV via `tikv/client-go`. Implements `RangeScanStore` so `ListObjects` is a single scan.

A new backend MUST satisfy meta.Store and pass the contract suite at internal/meta/storetest/contract.go. The suite is shared across all backends and is the parity oracle.

Drain pipeline

Mon, 01 Jan 0001 00:00:00 +0000

Drain pipeline#

A draining cluster is one that the operator has flagged stop-write so it can be taken out of rotation safely. Strata models the lifecycle as a state machine on the cluster_state row; the rebalance worker migrates chunks off in the evacuating state; the deregister-ready gate keeps the operator honest before they drop the cluster from STRATA_RADOS_CLUSTERS.

State diagram#

stateDiagram-v2
 [*] --> live: cluster_state row created<br/>(weight default 100)
 [*] --> pending: env-only cluster<br/>(no chunks yet)
 pending --> live: POST /activate {weight: N}
 live --> live: PUT /weight {weight: N}
 live --> draining_readonly: POST /drain {mode: "readonly"}
 live --> evacuating: POST /drain {mode: "evacuate"}
 draining_readonly --> evacuating: POST /drain {mode: "evacuate"}<br/>(upgrade — no readonly→evacuate skip)
 draining_readonly --> live: POST /undrain
 evacuating --> live: POST /undrain
 evacuating --> removed: chunks_on_cluster == 0<br/>operator drops env entry
 removed --> [*]: row remains as tombstone

States#

State	Picker	Reads	Scan-on-tick	Operator entry
`pending`	excluded from default wheel; explicit policy still routes	works	no	`POST /admin/v1/clusters/{id}/activate {weight: N}` flips to `live`
`live` (weight > 0)	included proportional to weight	works	no	`PUT /admin/v1/clusters/{id}/weight` adjusts in place
`live` (weight == 0)	excluded from default wheel; explicit policy still routes	works	no	legal “drained but not draining” state — useful for staged decommission
`draining_readonly`	excluded	works	no — readonly is stop-write only	`POST /admin/v1/clusters/{id}/drain {mode: "readonly"}`
`evacuating`	excluded	works	yes — rebalance worker migrates chunks off	`POST /admin/v1/clusters/{id}/drain {mode: "evacuate"}` or upgrade from readonly
`removed`	excluded everywhere	n/a	n/a	operator-flow tombstone — set after deregister

Stop-write semantics: a draining cluster accepts reads, deletes, HEAD, multipart UploadPart / Complete / Abort, and listings. Only fresh PUTs are refused with 503 DrainRefused + Retry-After: 300.

Admin API surface

Mon, 01 Jan 0001 00:00:00 +0000

Admin API surface#

This is the operator index. Authoritative contract lives in the admin API OpenAPI document — rendered interactively at the Admin API viewer; raw bytes at /openapi.yaml.

Data backend

Mon, 01 Jan 0001 00:00:00 +0000

Data backend#

internal/data/backend.go defines the data.Backend interface every chunk store implements. The metadata layer keeps the per-object manifest (chunk list, sizes, content hash); the data backend is responsible for opaque fixed-size chunks only. The split is what lets us drop in different backing stores (RADOS, S3, in-memory) without touching the gateway.

Backends#

Backend	Build tag	Notes
`memory`	none	In-process map. Used by tests and the smoke pass.
`rados`	`ceph`	RADOS pools via `goceph` (cgo, librados). Requires `make build` with `-tags ceph` or the docker-built image. Multi-cluster routing supported.
`s3`	none	S3-over-S3 — Strata as a transparent gateway in front of an upstream S3 endpoint. Useful for migrating from MinIO / SeaweedFS / AWS without lifting and shifting data.

Selection is via STRATA_DATA_BACKEND (memory / rados / s3). RADOS requires the configured pool ([rados] classes) to exist; the Storage status page covers the operator-facing health surface.

GC + lifecycle Phase 2

Mon, 01 Jan 0001 00:00:00 +0000

Migrating to gc / lifecycle Phase 2 (sharded leader-election)#

Phase 1 (cycle ralph/gc-lifecycle-scale, commit 6561845) lifted the per-leader concurrency cap via bounded errgroup fan-out (STRATA_GC_CONCURRENCY / STRATA_LIFECYCLE_CONCURRENCY). The leader was still single-replica.

Phase 2 (cycle ralph/gc-lifecycle-scale-phase-2) shards the leader-election space so multiple replicas process disjoint slices of the queue in parallel:

gc gets gc-leader-0..N-1 lease keys driven by STRATA_GC_SHARDS (default 1, range [1, 1024]). Each replica races for one or more of the per-shard leases and drains only the entries it owns via Meta.ListGCEntriesShard. The legacy global gc-leader lease is retired.
lifecycle gets per-bucket leases (lifecycle-leader-<bucketID>) plus a distribution gate (fnv32a(bucketID) % STRATA_GC_SHARDS == myReplicaID, where myReplicaID = min(GCFanOut.HeldShards())). The legacy global lifecycle-leader lease is retired.
Backwards-compat: STRATA_GC_SHARDS=1 (default) reproduces Phase 1 behaviour byte-for-byte. Existing make smoke / make smoke-tikv / make smoke-lab-tikv continue to pass without configuration changes.

This guide is the operator-facing checklist for the schema cutover and the multi-replica rollout.

Meta-backend comparison

Mon, 01 Jan 0001 00:00:00 +0000

Meta-backend benchmarks: TiKV vs Cassandra#

This page captures hot-path latency / throughput numbers for Strata’s two production metadata backends (Cassandra and TiKV) and the in-tree memory reference. The numbers are operator-runnable on a single laptop docker stack via the harness in internal/meta/storetest/bench.go.

The headline operations (US-018):

Op	Why it matters
`CreateBucket`	LWT-equivalent create-if-not-exists
`GetObject`	Single Get / latest-version range-scan
`ListObjects` 100k page=1k	Listing throughput on a real-world bucket
`CompleteMultipartUpload`	LWT-equivalent status flip (`uploading→completing`)
`GetIAMAccessKey`	SigV4 verifier — runs on every request
`AuditAppend`	Audit log write amplification
`AuditSweepPartition`	Periodic retention sweep (TiKV emulates Cassandra TTL)
`RangeScanObjects` 100k	TiKV-only: native ordered scan vs Cassandra fan-out

Rig#

Single-laptop docker stack. Both backends run as 3-node clusters so LWT

Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Monitoring#

Strata exposes three observability surfaces and one operator console embedded in the gateway binary:

Prometheus metrics at /metrics (every replica).
Structured slog logs to stdout, JSON-shaped, correlated by request_id.
OpenTelemetry traces exported via OTLP/HTTP, sampled tail-first with an in-process ring buffer for failed-trace replay.
Audit log in the metadata backend (audit_log table or TiKV prefix), one row per state-changing request.

This page covers the wire-up; the Observability deep dive covers the implementation rationale.

Multi-cluster routing

Mon, 01 Jan 0001 00:00:00 +0000

Multi-cluster routing#

A single Strata gateway can fan writes across multiple data clusters. The clusters can be Ceph RADOS pools, S3-compatible buckets in different regions or providers, or a mix. The routing decision happens on every PutObject and works the same for chunked RADOS writes and S3-over-S3 pass-through.

The two layers#

Routing is decided by two independent inputs, evaluated in order:

Bucket placement policy. A bucket can carry an explicit Placement map — {cluster-a: 70, cluster-b: 30} — set via PUT /admin/v1/buckets/{name}/placement. When present, this policy wins. The gateway weighted-picks among the live clusters in the policy and never routes the bucket elsewhere.
Cluster weights. When the bucket has no explicit policy, the gateway synthesizes a default policy from per-cluster weights set via PUT /admin/v1/clusters/{id}/weight {weight: N}. Weights are integers in [0, 100]; clusters with weight 0 accept reads and explicit-policy writes but receive no new default-routed writes.

The two layers do not combine. If a bucket has an explicit placement, the cluster-weight wheel is ignored for that bucket.

Multi-replica cluster

Mon, 01 Jan 0001 00:00:00 +0000

Multi-replica Strata cluster — operator guide#

Strata’s gateway is stateless. Replicas don’t form a quorum among themselves; the storage layer (TiKV / RADOS) provides durability + consistency. A single replica is therefore a single point of failure for HTTP traffic — not for data. Running ≥2 replicas behind a load balancer is the minimum HA shape.

This page covers the bare-default 2-replica TiKV lab (two TiKV-backed replicas behind nginx at host :9999) — the reference shape for multi-replica deployments. Same wiring works for 3+ replicas in production.

Rebalance scaling

Mon, 01 Jan 0001 00:00:00 +0000

Rebalance worker scaling benchmark (Phase 2)#

Closes ROADMAP P2 “Rebalance worker not sharded — single goroutine bottleneck on large deploys”. Mirrors the gc + lifecycle Phase 2 shape (see GC + Lifecycle scaling for the same fan-out pattern measured against gc and lifecycle workers).

The single-leader rebalance worker scans every draining bucket per tick on one goroutine. Phase 2 (STRATA_REBALANCE_SHARDS=N) shards that scan by fnv32a(bucketID) % N; each replica races for one of the rebalance-leader-0..N-1 leases and processes only its own shard’s bucket subset. STRATA_REBALANCE_SHARDS=1 reproduces Phase 1 byte-for-byte.

TiKV

Mon, 01 Jan 0001 00:00:00 +0000

TiKV metadata backend#

TiKV is a first-class metadata backend for Strata, on equal footing with Cassandra (and ScyllaDB as the CQL drop-in). The full meta.Store contract in internal/meta/storetest/contract.go runs against a real PD+TiKV cluster on every PR via .github/workflows/ci-tikv.yml.

The fundamental shape difference vs Cassandra is the objects table: TiKV keys are a flat ordered byte space, so ListObjects is a single ordered range scan instead of Cassandra’s 64-way fan-out + heap-merge. The gateway discovers this at runtime via the optional meta.RangeScanStore interface (internal/meta/store.go) and dispatches to the native scan path automatically — no operator action required.

Alerts

Mon, 01 Jan 0001 00:00:00 +0000

Alerts#

Strata ships a curated alert rule set in deploy/prometheus/alerts.yml. The file declares 47 rules — 17 SLO recording rules, 18 single-window alert rules, and 12 multi-window burn-rate alerts (Google SRE workbook ch.5). This page is the operator-facing companion: what each alert means, when it fires, and what to do.

Validate the file via make promtool-check. CI installs promtool and runs the target on every push; locally the target degrades to WARN when the binary is missing (mirrors make helm-lint). Lab Prometheus auto-loads the file via the rule_files: [alerts.yml] directive in deploy/prometheus/prometheus.yml.

SLO / SLI

Mon, 01 Jan 0001 00:00:00 +0000

SLO / SLI#

Strata ships three production SLOs as a starting point. Operators can re-tune each target by editing one line in deploy/prometheus/alerts.yml — every single-window alert and multi-burn-rate alert from /operate/alerts references the same recording rule so the change ripples cleanly.

SLO	Target	Window	SLI source
Availability	99.9%	30 days	`strata:availability:ratio_rate5m` (recording rule)
Latency p99	GET/PUT < 500 ms, LIST < 2 s, multipart Complete < 1 s	30 days	`strata:latency_get_put:p99_rate5m` + per-op equivalents
Durability	0 non-OK terminal GC acks	90 days	`strata:durability:error_rate5m` (always-on; no inventory dep)

Availability#

99.9% of S3 requests over a rolling 30-day window return a non-5xx status. Admin paths (bucket="_admin") excluded — admin error rate is tracked separately under the per-tenant dashboard.

Profiling

Mon, 01 Jan 0001 00:00:00 +0000

Profiling#

Strata exposes the Go runtime’s standard /debug/pprof/* endpoints when STRATA_PPROF_ENABLED=true. Profiles are protected by the same auth chain that guards /admin/v1/* (session cookie or SigV4). The endpoints are opt-in — defense-in-depth — because the heap profile can leak the contents of in-flight buffers in error paths.

Quick start#

# Boot Strata with pprof attached to the admin listener.
STRATA_PPROF_ENABLED=true \
STRATA_ADMIN_LISTEN=127.0.0.1:9001 \
STRATA_AUTH_MODE=required \
STRATA_STATIC_CREDENTIALS=AKADMIN:SKADMIN:admin \
 strata server

# Capture a 30s CPU profile (SigV4 from go tool pprof via aws-cli sigv4 wrapper
# OR from a pre-signed URL; for loopback dev use the admin session cookie).
go tool pprof -seconds=30 -http=:7070 \
 -url 'http://127.0.0.1:9001/debug/pprof/profile?seconds=30'

Configuration#

Env var	TOML key	Default	Description
`STRATA_PPROF_ENABLED`	`pprof.enabled`	`false`	Master switch. `true` registers `/debug/pprof/*`.
`STRATA_PPROF_LISTEN`	`pprof.listen`	empty	Optional dedicated listener (e.g. `127.0.0.1:9002`). Empty → attach to `admin_listen.listen`. One of the two MUST be set when enabled — pprof never attaches to the S3 hot path.
`STRATA_PPROF_BLOCK_RATE`	`pprof.block_rate`	`0`	`runtime.SetBlockProfileRate(N)` argument. `0` keeps block profile data empty.
`STRATA_PPROF_MUTEX_RATE`	`pprof.mutex_rate`	`0`	`runtime.SetMutexProfileFraction(N)` argument. `0` keeps mutex profile data empty.

STRATA_PPROF_ENABLED=true with neither STRATA_PPROF_LISTEN nor STRATA_ADMIN_LISTEN set fails fast at boot — the gateway refuses to silently expose profiling on the public S3 listener.

Workers

Mon, 01 Jan 0001 00:00:00 +0000

Workers#

Background loops run inside the same cmd/strata server binary as the gateway. STRATA_WORKERS= (or --workers=) selects which to run; an empty list runs the gateway only. Each worker is leader-elected, panic- recovered, and supervised — one worker’s panic or lease loss never affects the gateway or sibling workers.

Registry shape#

Each worker has a per-worker file under cmd/strata/workers/<name>.go that calls workers.Register from init():

func init() {
 workers.Register(workers.Worker{
 Name: "gc",
 Build: func(deps Dependencies) (Runner, error) { ... },
 SkipLease: true, // gc fan-out manages its own per-shard leases
 })
}

Build constructs the per-worker runner from the shared workers.Dependencies struct (Logger, Meta, Data, Tracer, Locker, Region, EmitLeader). Per-worker tunables (STRATA_GC_INTERVAL, STRATA_LIFECYCLE_*, …) are read inside Build directly so the dependency surface stays small.

Worker + leader election

Mon, 01 Jan 0001 00:00:00 +0000

Worker + leader election#

Every background worker shares a single supervisor shape: one goroutine per worker, leader-elected on a per-name lease, panic recovered with exponential backoff, heartbeat chip wired to the operator console. This page draws the lifecycle end-to-end and points at the per-worker carve-outs.

Lifecycle flowchart#

flowchart TD
 Boot["cmd/strata server"] --> Resolve["workers.Resolve(STRATA_WORKERS)"]
 Resolve -->|valid| Build["workers.Supervisor.Run(ctx, workers)"]
 Resolve -->|unknown name| Exit[/"exit 2"/]
 Build --> Spawn["spawn 1 goroutine per worker"]
 Spawn --> LeaseGate{"worker.SkipLease?"}
 LeaseGate -- no --> Acquire["leader.Session.Acquire(<name>-leader)"]
 LeaseGate -- yes --> RunnerBuild["Build(deps) → Runner"]
 Acquire -- acquired --> Emit["deps.EmitLeader(name, true)"]
 Emit --> RunnerBuild
 Acquire -- lost / partition --> Acquire
 RunnerBuild --> Run["runner.Run(ctx)"]
 Run -- clean return --> Release["release lease<br/>deps.EmitLeader(name, false)"]
 Run -- panic --> PanicMetric["strata_worker_panic_total{worker, shard}<br/>++"]
 PanicMetric --> Release
 Release --> Backoff{"healthy ≥ 5m?"}
 Backoff -- yes --> Spawn
 Backoff -- no --> Wait["sleep 1s → 5s → 30s → 2m"]
 Wait --> Spawn

Supervisor responsibilities#

workers.Supervisor.Run(ctx, []workers.Worker) is the entrypoint:

Compose collapse

Mon, 01 Jan 0001 00:00:00 +0000

Migrating to the single-strata compose shape#

docker compose up -d used to bring up two strata services in parallel on the same Cassandra metadata: a single-cluster strata (port 9999, knew only the default RADOS cluster) and a profile-gated strata-multi (port 9998, knew default + cephb). Both raced for the same global worker leases (gc-leader-N, rebalance-leader-N, lifecycle-leader-<bucketID>, etc.) on Cassandra. Whichever container started first won — and when the single-cluster winner popped a GC entry for cephb, deletion failed forever (no cephb connection), the queue piled up, deregister_ready stuck false, and drain wedged.

Drain & rebalance

Mon, 01 Jan 0001 00:00:00 +0000

Drain & rebalance#

Taking a data cluster out of rotation is a multi-step process: stop new writes, move existing bytes off, then remove the cluster from the configuration. Strata models this as an explicit lifecycle so operators can follow progress and abort partway through if needed.

Lifecycle#

stateDiagram-v2
 [*] --> pending: cluster added to env
 pending --> live: activate
 live --> draining_readonly: drain mode=readonly
 live --> evacuating: drain mode=evacuate
 draining_readonly --> evacuating: upgrade
 draining_readonly --> live: undrain
 evacuating --> live: undrain
 evacuating --> removed: deregister
 removed --> [*]

pending — a new cluster ID was added to STRATA_RADOS_CLUSTERS or STRATA_S3_CLUSTERS but the gateway has not yet been told to route default traffic to it. Reads and explicit-policy writes work; default routing skips it.
live — normal operating state. The cluster participates in the default-routing weight wheel.
draining_readonly — operator-initiated stop-write drain. New PUTs refuse with 503 DrainRefused. Reads, deletes, and in-flight multipart sessions keep working. The rebalance worker does not actively move data — use this for maintenance windows.
evacuating — operator-initiated decommission drain. Same write refusal as draining_readonly, plus the rebalance worker scans the cluster and migrates chunks to the remaining clusters honoring each bucket’s placement policy. A live progress chip in the operator console shows bytes moved, estimated time remaining, and per-bucket breakdown.
removed — cluster is gone. Excluded from every code path.

Drain refusal semantics#

While a cluster is draining, the gateway answers PUTs that would have landed on it with 503 DrainRefused and a Retry-After: 300 header. This is PUT only — reads, deletes, HEAD, multipart UploadPart / CompleteMultipartUpload / AbortMultipartUpload, and ListObjects continue working against the draining cluster. The intent is stop-write, not stop-read.

Parallel chunk PUT + GET (RADOS)

Mon, 01 Jan 0001 00:00:00 +0000

Parallel chunk PUT + GET (RADOS)#

Closes the two open ROADMAP.md P2 entries:

Parallel chunk upload in PutChunks — sequential for { ReadFull; writeChunk } loop replaced with a bounded worker pool that dispatches chunk writes concurrently. Manifest order + ETag (MD5 of the byte stream) are preserved.
Parallel chunk read / prefetch in GetChunks — sequential per-chunk fetch replaced with a bounded prefetch reader. The next chunk is fetched while the current one drains to the wire. Memory-bounded; aborts on client cancel.

The S3 data backend (internal/data/s3/backend.go) is unaffected: the AWS SDK manager.Uploader already parallelises multipart uploads. Memory backend is tests-only. Multi-cluster manifests (US-044) automatically benefit since chunk-write dispatch is cluster-agnostic — the worker pool dials the right ioctx per chunk via the existing resolveClass / ioctx helpers.

Per-bucket signing key rotation

Mon, 01 Jan 0001 00:00:00 +0000

Per-bucket signing key rotation#

Strata supports KMS-backed per-bucket SigV4 signing keys (US-001 / US-002 of ralph/auth-dx-trailer-lima). Each opted-in bucket carries a 32-byte DEK wrapped under a KMS CMK (AWS KMS / Vault Transit / LocalHSMProvider) plus a creation timestamp. The auth middleware unwraps on cache miss and uses hex(DEK) as the SigV4 secret in place of the IAM access-key secret.

This page covers the day-2 rotation runbook. The KMS provider reference lists the per-provider env vars; the admin API reference documents the three rotation endpoints.

RGW comparison

Mon, 01 Jan 0001 00:00:00 +0000

Benchmarks vs Ceph RGW#

This page captures the side-by-side numbers backing the README’s “drop-in RGW replacement” claim. Closes ROADMAP P2 “Benchmarks vs RGW” via cycle ralph/rgw-benchmarks (US-001..US-012); the post-fix rerun landed via ralph/p1-fixes (US-001..US-009) after the per-bucket bucket_stats LWT saturation was removed.

Headline conclusion#

Bucket-index claim — VERIFIED (provisionally) at the concurrency RGW can sustain. Two independent regimes surface the win:

1 KiB PUT @ c=8 (the canonical “many small writes” hot path): Strata p99 ≈ 211 ms vs RGW p99 11–22 s (runs varied; one run wedged at 364 s). Strata is ~50× faster at p99 and stays flat across the 3-run set; RGW’s omap-index serialisation surfaces immediately past c=1 on the single-OSD lab. Same data tier (single-OSD ceph-a memstore), same make up-bench-rgw shape — the delta is gateway-side, not OSD.
ListObjects 100k-key @ c=8: Strata sustains ~208 k ops/s @ p99 ~640 ms across 3 runs, zero errors. RGW could not be measured at this scale on the lima reference dev box — the per-bucket pools default.rgw.buckets.index saturate the memstore OSD before the seed phase completes. The very fact that RGW cannot complete the workload at the same lab shape reinforces the bucket-index thesis: Strata’s sharded fan-out (s/B/<bid>/bs/<shard> × 8 shards on TiKV; sharded objects table on Cassandra) absorbs the same concurrent write pressure RGW’s omap index falls over on.

RGW wins at c=1 small PUT as expected: Strata p99 ≈ 12 ms vs RGW p99 ≈ 2 ms. The user-space SigV4 + policy-verify hop Strata adds vs RGW’s in-process auth costs ~10 ms on the single-threaded hot path; the gap closes as soon as concurrency arrives because the gateway-side per-request cost stops dominating relative to backend serialisation.

S3 API operations

Mon, 01 Jan 0001 00:00:00 +0000

S3 API operations#

Operator + SDK-author index of every S3 operation Strata answers. The router dispatches ServeHTTP → handleBucket (bucket-scope) → handleObject (object-scope); leaf handlers live alongside each feature in the s3api package. The Handler file:line column is intentionally line-anchored — a lint test asserts every dispatched method shows up here (see the maintainer note above this section). AWS divergence column is empty when the operation is full-compat; one-liner otherwise.

S3-over-S3 data backend

Mon, 01 Jan 0001 00:00:00 +0000

S3 Data Backend#

The S3-over-S3 data backend stores object bytes in any S3-compatible bucket (AWS S3, MinIO, Ceph RGW, Garage). One Strata object = one backend S3 object under the key <bucket-uuid>/<object-uuid>. RADOS remains the default and recommended data backend; this page covers the S3 alternative.

When to choose S3 over RADOS#

Question	Pick RADOS if	Pick S3 if
Existing infra investment	Operator already runs Ceph	Operator already runs MinIO / has AWS account / wants Garage
Tail latency budget	< 5 ms p99 GET on hot data is critical	50–200 ms p99 GET is acceptable (HTTPS round-trip floor)
Bytes-stored cost model	Self-hosted hardware, want bare-metal economics	Cloud storage tiers (Standard-IA, Glacier) carry the lifecycle
Availability boundary	Single-DC, hardware-controlled	Multi-AZ via the cloud provider’s bucket SLA
Encryption-at-rest mandate	Strata-side envelope encryption is enough	Want backend SSE / KMS independent of Strata
Operational complexity	Team owns Ceph operations	Team owns S3 operations (or has zero ops via AWS)

The two backends are equal-tier alternatives — pick by what your operations team already runs. Mixing within a single Strata deployment is not supported (STRATA_DATA_BACKEND is a single-value flag).

Scaling

Mon, 01 Jan 0001 00:00:00 +0000

Scaling#

Strata replicas are stateless gateways — they own no on-disk state, so sizing is a function of three signals: peak request rate (RPS), the working-set bytes the gateway holds in flight (RAM), and the cores the SigV4 + crypto path consumes (CPU). The metadata + data tiers (Cassandra / TiKV / RADOS) follow their own upstream sizing guides; this page covers the gateway tier and points operators at the upstream references.

Storage status

Mon, 01 Jan 0001 00:00:00 +0000

Storage status — operator guide#

The Storage page (/console/storage) and the Cluster Overview Storage hero card surface the live health of the meta + data backends and the per-storage-class object distribution. This page covers the env vars the UI reads through, what each warning means, and how to interpret the RADOS / TiKV / Cassandra-specific signals.

The page is fed by three small admin endpoints:

Endpoint	Returns	Polled by
`GET /admin/v1/storage/meta`	`MetaHealthReport` (Cassandra peers / TiKV PD stores / memory)	`/storage` Meta tab @ 30 s
`GET /admin/v1/storage/data`	`DataHealthReport` (RADOS pool stats / S3 reachability / memory)	`/storage` Data tab @ 30 s
`GET /admin/v1/storage/classes`	`{classes:[…], pools_by_class:{…}}`	`/storage` Data tab @ 30 s, Cluster Overview hero @ 60 s
`GET /admin/v1/storage/health`	aggregate `{ok, warnings, source}`	`<StorageDegradedBanner>` @ 30 s on every authed page

Schema details live in internal/adminapi/openapi.yaml.

RADOS ReadOp / WriteOp batching

Mon, 01 Jan 0001 00:00:00 +0000

RADOS ReadOp / WriteOp batching#

Closes the ROADMAP.md P3 entry ReadOp / WriteOp batching in RADOS (US-003 of ralph/storage-correctness).

What batching does#

internal/data/rados/ops.go (build tag ceph) defines two helpers that bundle a chunk Write / Read with N xattr ops into a single librados operation:

writeChunkBatched(ioctx, oid, body, xattrs) — builds a goceph.WriteOp, appends WriteFull(body) + SetXattr(k, v) per entry of xattrs, then Operate()s once. xattrs nil ⇒ byte-identical to the legacy writeChunk in backend.go.
readChunkBatched(ioctx, oid, off, length, wantXattrs) — builds a goceph.ReadOp with Read(off, buf), then Operate()s once. When wantXattrs is true a follow-up ioctx.ListXattrs runs to populate the xattrs map — go-ceph v0.39 does not expose rados_read_op_getxattrs, so xattrs sit on a second client round-trip. Today no caller requests xattrs.

writeChunk (legacy) and writeChunkBatched(_, _, _, nil) produce the same on-wire librados WriteOp{ WriteFull }. The win materialises only when xattrs are added to the PUT hot path — a future cycle. The helpers exist now so the call sites are wired and the bench shape is established.

Sharding

Mon, 01 Jan 0001 00:00:00 +0000

Sharding#

Sharding is the single biggest divergence from Ceph RGW: every bucket’s metadata is split across N partitions instead of living in a single bucket-index object. The split avoids RGW’s bucket-index ceiling at large object counts and lets ListObjects scale linearly with N.

Objects table partition key#

The cassandra objects table is partitioned by (bucket_id, shard):

PRIMARY KEY ((bucket_id, shard), key, version_id)
WITH CLUSTERING ORDER BY (key ASC, version_id DESC)

Where shard = fnv32a(key) % N and N is per-bucket (STRATA_BUCKET_SHARDS at bucket creation, default 64). N must be a power of two — meta.IsValidShardCount(n) enforces it. Power-of-two constraint matters for the reshard worker below: when N doubles, every old shard either stays under the new modulo or splits cleanly into two new ones, never three.

Tracing

Mon, 01 Jan 0001 00:00:00 +0000

Tracing#

Strata emits OpenTelemetry spans across every meaningful tier of the request and worker paths. Each span carries the strata.component attribute so an operator can filter the entire gateway path or the entire worker path in one Jaeger query, and each worker iteration appears as a discrete trace so a slow / failing tick is easy to correlate with the meta + data ops it triggered.

The wire-up env vars + tail-sampler + ring buffer behaviour live in Monitoring. This page is the operator-facing reference for what spans exist, how they are named, and how to filter them.

RADOS conn-pool sizing

Mon, 01 Jan 0001 00:00:00 +0000

RADOS conn-pool sizing#

Closes the ROADMAP.md P3 entry Connection pool tuning (US-004 of ralph/storage-correctness).

What the pool does#

A single librados *Conn serialises ops through one cephx session and a per-conn thread pool. Write-heavy workloads that saturate that single session see contention even though the OSD layer has headroom. The connPool in internal/data/rados/pool.go (build tag ceph) holds N pre-connected *goceph.Conn instances per cluster and serves them round-robin via an atomic counter; per-conn IOContexts are cached lazily by (pool, namespace).

OTel ring-buffer bytes budget

Mon, 01 Jan 0001 00:00:00 +0000

OTel ring-buffer bytes budget#

Closes the ROADMAP.md P3 entry OTel ring-buffer eviction tuning under burst load (US-005 of ralph/storage-correctness).

What the ring buffer does#

internal/otel/ringbuf.RingBuffer is an in-process SpanProcessor that retains every finished span under an LRU + bytes-budget. The /admin/v1/diagnostics/trace/{requestID} admin endpoint reads from the ring so operators can debug a request without a Jaeger / Tempo deployment. Tail sampling on the exporter side still drops most spans on the wire — the ring keeps the full payload locally.

Admin API viewer

Mon, 01 Jan 0001 00:00:00 +0000

Admin API viewer#

Interactive Redoc render of the canonical Admin-API OpenAPI contract. The YAML is copied into the Hugo static dir at build time by make docs-openapi-copy (a prerequisite of make docs-build / make docs-serve), so the viewer always reflects the contract at the same SHA as the rest of the docs site.

For a flat index of every admin endpoint with method, path, audit verb, and a one-line summary, see Admin API surface. For the raw contract, fetch /openapi.yaml.

Backup + restore

Mon, 01 Jan 0001 00:00:00 +0000

Backup + restore#

Strata splits backup responsibility along the same tiers the gateway splits state into:

Metadata tier (Cassandra / TiKV): the source of truth for bucket / object / IAM rows. Backed up via the upstream tooling for each backend.
Data tier (RADOS / S3-over-S3): chunk bytes. Backed up via Ceph pool snapshots or the upstream S3 service’s native versioning / replication.
Cross-region replication (replicator worker): ships object PUT / DELETE events to a peer Strata cluster in near-real time.
Inventory worker: writes manifest.json + CSV.gz pairs that document every object in a bucket — useful as an audit ledger for external backup tooling.

This page is a backup-strategy overview. Implementation details live in Architecture — Workers (the inventory + replicator workers) and the upstream docs for Ceph / Cassandra / TiKV.

Drain progress physical chunks

Mon, 01 Jan 0001 00:00:00 +0000

Migrating to the physical-chunk drain progress#

GET /admin/v1/clusters/<id>/drain-progress and the operator console <DrainProgressBar> used to surface chunks_on_cluster — the manifest-derived chunk count — as the sole headline metric. After the ralph/drain-progress-physical cycle (US-001..US-003) the response gains three additive fields and the UI renders a 3-state machine that distinguishes manifest progress from physical pool state.

This page documents the wire shape change, the back-compat fallback for backends without the new probe, and the operator-visible state machine. There is no schema migration; no data migration; the change is purely additive on the /drain-progress JSON shape.

GC + lifecycle tuning

Mon, 01 Jan 0001 00:00:00 +0000

GC + lifecycle tuning#

The gc and lifecycle workers run inside every gateway replica. The operator-facing knobs split into two layers:

Phase 1 — per-replica fan-out: STRATA_GC_CONCURRENCY / STRATA_LIFECYCLE_CONCURRENCY cap the goroutine count inside one replica’s worker.
Phase 2 — multi-replica sharding: STRATA_GC_SHARDS shards the leader-election space so N replicas process disjoint slices of the queue in parallel.

Both layers compose: a 3-replica deploy with STRATA_GC_SHARDS=3 and per-replica STRATA_GC_CONCURRENCY=64 runs three independent fan-outs against disjoint shards.

Kubernetes

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes deployment#

Strata on Kubernetes is a stateless gateway tier in front of external metadata (TiKV or Cassandra) and object-data (RADOS or S3) clusters. Pods are interchangeable — no PVCs, no per-pod storage, no quorum among gateways. Scale horizontally with kubectl scale; durability lives in the storage tier.

This page walks through the worked example committed under deploy/k8s/ — a 3-replica Deployment + Service + ConfigMap + Secret + Ingress aimed at a production-shaped cluster. A Helm chart ships alongside at deploy/helm/strata/.

Observability

Mon, 01 Jan 0001 00:00:00 +0000

Observability#

Strata ships three correlated observability surfaces: structured logs, an audit log table for state-changing requests, and OTel tracing with a ring-buffer trace browser embedded in the operator console. All three key off the same request_id so an operator can pivot from one to the next without rebuilding context.

Structured logs (slog)#

internal/logging is the canonical setup. The cmd/strata binary (both strata server and strata admin) calls logging.Setup() first thing to install a JSON-handler *slog.Logger driven by STRATA_LOG_LEVEL (DEBUG/INFO/WARN/ERROR; default INFO).

Workers

Mon, 01 Jan 0001 00:00:00 +0000

Workers#

A Strata deployment runs the same strata binary in two modes: the gateway handles S3 traffic, and one or more workers handle the background work that does not belong on the request path. Workers run inside the same binary you already deploy — set STRATA_WORKERS=gc,lifecycle,… on a gateway replica and it spawns those loops alongside the HTTP listener.

Every worker is leader-elected: across N gateway replicas, one replica acquires the lease for a given worker and runs it; the others stand by and take over if the leader’s lease expires. Leases are scoped per-worker, so different replicas can lead different workers — gc on replica A, lifecycle on replica B. Workers panic-restart on exponential backoff so a single iteration error never takes a worker offline permanently.

Reconcile + rebuild-index

Mon, 01 Jan 0001 00:00:00 +0000

Reconcile + rebuild-index#

Strata splits the S3-key→chunk map (meta tier: TiKV / Cassandra) from the chunk bytes (data tier: RADOS / S3-over-S3). That split is the scale advantage — but it drops the safety net RGW gets by colocating its bucket index with the data. After you restore one tier from a backup that is even slightly out of step with the other, the two tiers diverge silently:

Orphan chunks — data has a chunk no manifest references. GC walks meta→data, so it can never see them; they leak storage forever.
Dangling manifests — meta has a manifest pointing at a chunk the data tier no longer holds. The client discovers it as a 5xx on GET, not on the restore.
Lost meta backup — the bytes are intact in RADOS but unreadable, because the key→chunk map is gone.

This page is the post-restore runbook for realigning the two tiers. It pairs with Backup + restore (how each tier is snapshotted) — run reconcile after the restore drill, before you promote traffic.

Bench runner setup (self-hosted)

Mon, 01 Jan 0001 00:00:00 +0000

Bench runner setup (self-hosted)#

The weekly RGW comparison bench (.github/workflows/bench-rgw.yml) runs on a self-hosted GitHub Actions runner labelled [self-hosted, strata-bench]. The runner is operator-provisioned once; afterwards the cron + manual-dispatch triggers fire on schedule with zero human input. Auto-PRs land against main carrying refreshed numbers and the runner’s specs.

This page covers the one-time provision. Closes ROADMAP P3 “bench-rgw lima envelope fix” via US-005 of cycle ralph/auth-dx-trailer-lima.

Required hardware#

The bench cycle exercises an 11-workload sweep with multipart uploads up to 25 GiB transient and a 100 k-key list seed. The lima dev box does not have the RAM envelope for the full sweep (rgw-comparison.md Limitations). A dedicated Linux box clears the blocker.

Capacity planning

Mon, 01 Jan 0001 00:00:00 +0000

Capacity planning#

Capacity on a Strata deploy splits across three independently-scaled tiers:

Gateway tier: stateless replicas. Capacity = peak RPS / SigV4 + routing budget per replica. Scale horizontally; see Scaling.
Metadata tier: Cassandra / TiKV. Capacity = total object-row count × per-row overhead, plus the compare-and-set or pessimistic-txn rate during writes.
Data tier: RADOS / S3-over-S3. Capacity = total object bytes × pool replication factor, plus chunk fan-out overhead.

This page covers the math; pair with Backup + restore for the backup overhead and GC + lifecycle tuning for the drain rate.

Quotas + billing

Mon, 01 Jan 0001 00:00:00 +0000

Quotas + billing#

Strata enforces hard per-bucket and per-user storage quotas at PUT-validate time and emits a nightly per-(bucket, storage class) usage aggregate that an external invoice generator can consume. The shape mirrors AWS / RGW so a RADOS Gateway tenant migrating to Strata sees the same QuotaExceeded 403 on overage.

The cycle stops at the usage feed. Invoice ledger / payment integration lives in a separate ops-layer service that consumes usage_aggregates.

TiKV-default lab

Mon, 01 Jan 0001 00:00:00 +0000

Migrating to the TiKV-default 2-replica lab#

docker compose up -d used to bring up a single Cassandra-backed strata service on :9999 plus a ceph + ceph-b multi-cluster RADOS pair. TiKV was opt-in via --profile tikv (strata-tikv) or --profile lab-tikv / --profile lab-tikv-3 for 2- and 3-replica labs, and a parallel 3-replica Cassandra lab lived under --profile lab-cassandra-3. Four distinct profile shapes plus the always-on bare default — each rarely-used, each one a maintenance surface to keep green across CI + smoke + bench scripts.

TLS termination + backend mTLS

Mon, 01 Jan 0001 00:00:00 +0000

TLS termination + backend mTLS#

This is the operator playbook for putting Strata on the wire securely. The gateway can terminate TLS itself (recommended for prod) or sit behind a load balancer that handles termination upstream. Backend connections (Cassandra, TiKV, S3-upstream) each take an opt-in mutual-TLS bundle so a network intruder cannot impersonate the gateway. RADOS is the one exception: it authenticates via Ceph’s native cephx protocol and ignores the TLS layer entirely (see RADOS uses cephx below).

Image verification

Mon, 01 Jan 0001 00:00:00 +0000

Image verification#

Every Strata image published to ghcr.io/danchupin/strata is signed, attested, and inventoried by the release pipeline (.github/workflows/release-image.yml). This page is the operator playbook for verifying that supply-chain evidence before the image reaches a node, plus two Kubernetes admission-control shapes that make the check mandatory at deploy time.

The pipeline produces four pieces of evidence, all anchored to the same image digest:

Evidence	Producer	Story	Verified with
cosign signature (keyless OIDC)	`sigstore/cosign`	US-008	`cosign verify`
SLSA L3 provenance	`slsa-github-generator`	US-006	`cosign verify-attestation`
SPDX 2.3 SBOM	`anchore/sbom-action` (syft)	US-007	`grype` / `trivy sbom`
License inventory	`go-licenses`	US-009	`go-licenses report`

What signing buys you#

Keyless signing gives you a non-repudiable provenance trail from source to running image: the GitHub Actions OIDC identity that built the image is the signer, Fulcio issues a short-lived certificate bound to that identity, and the signature lands in the Rekor public transparency log. Because the signature covers the image digest (not a mutable tag), a verified image is tamper-proof — repush a different layer and the signature no longer matches. There is no maintainer-held private key to leak or rotate; the trust root is GitHub’s OIDC issuer plus the public Sigstore infrastructure. The Rekor entry is append-only and publicly queryable, so a forensic investigation can prove exactly which workflow run, at which commit, produced a given digest — even years later.

Pre-prod validation walkthrough

Mon, 01 Jan 0001 00:00:00 +0000

Pre-prod validation walkthrough#

This is the release-owner runbook: a single documented pass that brings up both production labs and exercises the whole hardened object path so the GO/NO-GO decision rests on observed behaviour, not story-by-story claims. It is the operator-facing companion to the per-feature smokes — it composes them into one walkthrough and records a pass/fail-per-leg readiness note.

The composite is scripts/smoke-architecture-hardening.sh, wired as make smoke-architecture-hardening. Each leg is independent: it WARN-skips when its lab is down and only fails the run on a real assertion failure. Set REQUIRE_LAB=1 to promote a down lab to a hard fail (use this in CI where the labs are guaranteed up).

Production hardening

Mon, 01 Jan 0001 00:00:00 +0000

Production hardening#

12-line checklist to run through before flipping a Strata replica into prod traffic. Each line links to the runbook section that explains the knob, the metric, and the failure mode it prevents. Every knob is opt-in and zero-by-default — running through the list flips a memory / lab deployment into a prod-ready shape without touching code.

#	Check	Runbook
1	HTTP server timeouts non-zero (`STRATA_HTTP_READ_HEADER_TIMEOUT=10s` / `STRATA_HTTP_READ_TIMEOUT=60s` / `STRATA_HTTP_WRITE_TIMEOUT=30m` / `STRATA_HTTP_IDLE_TIMEOUT=120s` / `STRATA_HTTP_MAX_HEADER_BYTES=1048576`). Defaults already match — only verify if you tuned them.	STRATA_HTTP_*
2	TLS terminated on the gateway (`STRATA_TLS_CERT_FILE` / `STRATA_TLS_KEY_FILE`, or `STRATA_TLS_CERT_DIR` for SNI multi-tenant) — or behind an ingress with `STRATA_TRUSTED_PROXIES` set to the ingress source CIDR.	TLS termination — shapes B/C
3	`STRATA_TLS_MIN_VERSION=TLS1.2` (default) and `STRATA_TLS_CIPHER_PROFILE=mozilla-modern` (default). Bump `MIN_VERSION=TLS1.3` if every client supports it.	TLS shapes
4	Cert hot-reload enabled (`STRATA_TLS_RELOAD_INTERVAL=60s`, default) so cert-manager / Vault PKI rotation is picked up without restart.	cert-manager recipe
5	Admin / console / metrics on a separate listener (`STRATA_ADMIN_LISTEN=127.0.0.1:9001` recommended; loopback or RFC1918 only). Optionally pin operator client certs via `STRATA_ADMIN_TLS_CLIENT_CA_FILE`.	Shape C — split admin listener
6	`STRATA_TRUSTED_PROXIES` set to the ingress / LB source CIDR. Default empty = `X-Forwarded-*` ignored. Required for the `Secure` cookie flag + audit-log client-IP fidelity behind any proxy.	Trusted proxies — README breaking change
7	Per-IP + per-key ingress rate limit on (`STRATA_RATE_LIMIT_PER_IP=N` and / or `STRATA_RATE_LIMIT_PER_KEY=N`). Default 0 = disabled. Refusal returns HTTP 429 + `<Code>SlowDown</Code>`.	STRATA_RATE_LIMIT_*
8	Cassandra mTLS (`STRATA_CASSANDRA_TLS_CA_FILE` + `STRATA_CASSANDRA_TLS_CERT_FILE` + `STRATA_CASSANDRA_TLS_KEY_FILE`). `SKIP_VERIFY` must be `false` (default).	Cassandra mTLS
9	TiKV mTLS (`STRATA_TIKV_TLS_CA_FILE` is required when any other TLS knob is set — the upstream silently downgrades on empty CA). PD endpoints accept `https://`.	TiKV mTLS
10	S3-upstream mTLS (`STRATA_S3_TLS_*` global default; per-cluster `tls` override on `STRATA_S3_CLUSTERS` JSON wins outright per cluster).	S3-upstream mTLS
11	RADOS cephx in place — `STRATA_RADOS_KEYRING` populated; `ms_cluster_mode=secure` set in `ceph.conf` if wire-level confidentiality is required. (No Strata-side TLS knob for RADOS.)	RADOS uses cephx
12	Prometheus alert on `sum(strata_backend_tls_skip_verify) > 0` (any backend with `SKIP_VERIFY=true`) AND on `rate(strata_ingress_rate_limit_refused_total[5m]) > N` (sustained client floods).	Monitoring — alert recipes

What this checklist closes#

Every line above closes a P0 gap from the 2026-05-25 prod-readiness audit:

S3 multi-cluster routing

Mon, 01 Jan 0001 00:00:00 +0000

S3 multi-cluster routing#

The S3 data backend supports routing per storage class to a distinct (cluster, bucket) pair. Two envs hold the full config — STRATA_S3_CLUSTERS (JSON array of bucket-less cluster specs) and STRATA_S3_CLASSES (JSON object mapping storage class names to {cluster, bucket} tuples). Adding or removing a cluster requires a gateway restart; multi-replica deployments hide per-instance downtime via rolling restart.

This page is the operator guide. For the conceptual S3 backend overview see S3 data backend.

Compliance

Mon, 01 Jan 0001 00:00:00 +0000

Compliance#

Strata implements S3 Object Lock with COMPLIANCE and GOVERNANCE retention modes plus legal hold. COMPLIANCE rows are immutable until their RetainUntilDate elapses — not even the bucket owner can shorten the retention window or delete the object early. The audit log records every COMPLIANCE-mode write and every expiry the lifecycle worker performs so auditors can grep audit_log for retention-policy events with a single action LIKE 'objectlock:%' clause.

This page is the operator workflow guide. For the audit-log shape see Monitoring — audit log; for the underlying retention semantics see the S3 Compatibility matrix.

Placement + rebalance

Mon, 01 Jan 0001 00:00:00 +0000

Placement + rebalance#

Once you run more than one RADOS / S3 cluster behind a single Strata deployment, you also need to control which cluster a bucket’s chunks land on and how to migrate old chunks when a new cluster joins or an old cluster is being retired. Strata ships both pieces:

A per-bucket placement policy (meta.Bucket.Placement — a {cluster: weight} map) that the chunk PUT path consults via a stable hash-mod router.
A leader-elected rebalance worker (strata server --workers=rebalance) that walks every bucket with a non-nil policy, compares the actual per-cluster chunk distribution to the policy’s target, and copies chunks A → B until the two match.

A bucket without a policy (Placement == nil) behaves exactly as before — chunks land on the storage class’s default cluster. No migration, no schema bump, no behavior change. The policy + worker are both opt-in.

Billing

Mon, 01 Jan 0001 00:00:00 +0000

Billing#

Strata does not ship an invoice generator. It ships the inputs an external invoice generator can consume: a daily per-(bucket, storage class) row in usage_aggregates whose byte_seconds field integrates the bucket-usage counter across the UTC day with the trapezoid rule.

This page documents the integration math, the sampling cadence, and the env knob that tunes accuracy vs. meta-backend cost. For the live quota counter, the QuotaExceeded shape, the reconcile worker, and the admin API surface, see Quotas + billing.

Mon, 01 Jan 0001 00:00:00 +0000

Phase 2 — gc + lifecycle multi-leader bench artifacts#

JSONL captured by the bench harness in in-process simulation mode: both meta and data backends are pure-memory; the multi-shard / multi- replica race happens inside one strata admin process via the --shards (gc) / --replicas (lifecycle) flags wired in US-006.

Files#

File	What	How to regenerate
`sim-bench-gc-shards1.jsonl`	gc bench, single-leader (Phase 1 shape), N=50000, c∈{1,4,16,64,256}	`for c in 1 4 16 64 256; do STRATA_META_BACKEND=memory STRATA_DATA_BACKEND=memory ./bin/strata admin bench-gc --entries=50000 --concurrency=$c --shards=1; done \| jq -c .`
`sim-bench-gc-shards3.jsonl`	gc bench, 3-shard (Phase 2 shape), N=50000	…`--shards=3`
`sim-bench-lifecycle-replicas1.jsonl`	lifecycle bench, single-replica (Phase 1), N=10000, 1 bucket	`for c in …; do … bench-lifecycle --objects=10000 --concurrency=$c --replicas=1 --buckets=1; done`
`sim-bench-lifecycle-replicas3.jsonl`	lifecycle bench, 3-replica (Phase 2), N=10000, 9 buckets	…`--replicas=3 --buckets=9`

Caveats#

Memory meta + memory data put the worker on a fast path with effectively zero per-op latency. Concurrency speedup that the production lab observes on TiKV (real pessimistic-txn round-trips) does NOT manifest here — the in-process Go RWMutex is the bottleneck. Treat these numbers as a noise floor that pins the shape of multi-leader coordination overhead, not as a forecast of production throughput.
For the canonical Phase 2 numbers operators should rerun make bench-gc-multi / make bench-lifecycle-multi against the 3-replica lab brought up via make up-lab-tikv-3 (see docs/site/content/architecture/benchmarks/gc-lifecycle.md’s “Phase 2 — multi-leader” section).
Every JSON object on a line is one bench level (one (--concurrency, --shards|--replicas) combination). Schema matches cmd/strata/admin/ bench_common.go::benchResult.

Strata Documentation

ADR-0001: Skip RADOS omap for bucket index

ADR-0001: Skip RADOS omap for bucket index#

Status#

Context#

ADR-0002: Derive IsLatest at read time

ADR-0002: Derive IsLatest at read time#

Status#

Context#

ADR-0003: Single manifest blob column

ADR-0003: Single manifest blob column for object metadata#

Status#

Context#

ADR-0004: One leader lease per background worker

ADR-0004: One leader lease per background worker#

Status#

Context#

Auth

Auth#

SigV4 verification#

Binary consolidation

Migrating to the unified strata binary#

Drain a cluster

Drain a cluster#

GC + Lifecycle scaling

GC + Lifecycle worker scaling benchmark (Phase 1 + Phase 2)#

Phase 1 — single-leader concurrency cap#

Router

Router#

Dispatch order#

S3 surface

S3 surface#

ScyllaDB

ScyllaDB metadata backend#

Compatibility status#

Deployment notes#

Connection#

Single-node deployment

Single-node Strata#

Prerequisites#

STRATA_* environment variables

STRATA_* environment variables#

Web UI (Strata Console)

Strata Console (Web UI) — Operator Guide#

PUT flow

PUT flow#

Sequence diagram#

Step-by-step#

Multi-cluster routing

Multi-cluster routing#

Flowchart#

The picker contract#

Docker Compose

Docker Compose deployment#

Meta store

Meta store#

Drain pipeline

Drain pipeline#

State diagram#

States#

Admin API surface

Admin API surface#

Data backend

Data backend#

Backends#

GC + lifecycle Phase 2

Migrating to gc / lifecycle Phase 2 (sharded leader-election)#

Meta-backend comparison

Meta-backend benchmarks: TiKV vs Cassandra#

Rig#

Monitoring

Monitoring#

Multi-cluster routing

Multi-cluster routing#

The two layers#

Multi-replica cluster

Multi-replica Strata cluster — operator guide#

Rebalance scaling

Rebalance worker scaling benchmark (Phase 2)#

TiKV

ADR-0002: Derive `IsLatest` at read time#

ADR-0003: Single `manifest` blob column for object metadata#

Migrating to the unified `strata` binary#

`STRATA_*` environment variables#