Operate on Strata Documentation

Drain a cluster

Mon, 01 Jan 0001 00:00:00 +0000

Drain a cluster#

Taking a data cluster out of rotation is a five-step workflow: preview impact → drain → watch progress → wait for deregister-ready → remove from env. Strata exposes each step through the admin API and the operator console so you can drive the workflow from the browser or from a shell script.

This page is the day-2 runbook. The Drain & rebalance concept covers the lifecycle states; the Placement + rebalance best-practice covers tuning the rebalance throughput and the cluster-state machine in depth.

Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Monitoring#

Strata exposes three observability surfaces and one operator console embedded in the gateway binary:

Prometheus metrics at /metrics (every replica).
Structured slog logs to stdout, JSON-shaped, correlated by request_id.
OpenTelemetry traces exported via OTLP/HTTP, sampled tail-first with an in-process ring buffer for failed-trace replay.
Audit log in the metadata backend (audit_log table or TiKV prefix), one row per state-changing request.

This page covers the wire-up; the Observability deep dive covers the implementation rationale.

Alerts

Mon, 01 Jan 0001 00:00:00 +0000

Alerts#

Strata ships a curated alert rule set in deploy/prometheus/alerts.yml. The file declares 47 rules — 17 SLO recording rules, 18 single-window alert rules, and 12 multi-window burn-rate alerts (Google SRE workbook ch.5). This page is the operator-facing companion: what each alert means, when it fires, and what to do.

Validate the file via make promtool-check. CI installs promtool and runs the target on every push; locally the target degrades to WARN when the binary is missing (mirrors make helm-lint). Lab Prometheus auto-loads the file via the rule_files: [alerts.yml] directive in deploy/prometheus/prometheus.yml.

SLO / SLI

Mon, 01 Jan 0001 00:00:00 +0000

SLO / SLI#

Strata ships three production SLOs as a starting point. Operators can re-tune each target by editing one line in deploy/prometheus/alerts.yml — every single-window alert and multi-burn-rate alert from /operate/alerts references the same recording rule so the change ripples cleanly.

SLO	Target	Window	SLI source
Availability	99.9%	30 days	`strata:availability:ratio_rate5m` (recording rule)
Latency p99	GET/PUT < 500 ms, LIST < 2 s, multipart Complete < 1 s	30 days	`strata:latency_get_put:p99_rate5m` + per-op equivalents
Durability	0 non-OK terminal GC acks	90 days	`strata:durability:error_rate5m` (always-on; no inventory dep)

Availability#

99.9% of S3 requests over a rolling 30-day window return a non-5xx status. Admin paths (bucket="_admin") excluded — admin error rate is tracked separately under the per-tenant dashboard.

Profiling

Mon, 01 Jan 0001 00:00:00 +0000

Profiling#

Strata exposes the Go runtime’s standard /debug/pprof/* endpoints when STRATA_PPROF_ENABLED=true. Profiles are protected by the same auth chain that guards /admin/v1/* (session cookie or SigV4). The endpoints are opt-in — defense-in-depth — because the heap profile can leak the contents of in-flight buffers in error paths.

Quick start#

# Boot Strata with pprof attached to the admin listener.
STRATA_PPROF_ENABLED=true \
STRATA_ADMIN_LISTEN=127.0.0.1:9001 \
STRATA_AUTH_MODE=required \
STRATA_STATIC_CREDENTIALS=AKADMIN:SKADMIN:admin \
 strata server

# Capture a 30s CPU profile (SigV4 from go tool pprof via aws-cli sigv4 wrapper
# OR from a pre-signed URL; for loopback dev use the admin session cookie).
go tool pprof -seconds=30 -http=:7070 \
 -url 'http://127.0.0.1:9001/debug/pprof/profile?seconds=30'

Configuration#

Env var	TOML key	Default	Description
`STRATA_PPROF_ENABLED`	`pprof.enabled`	`false`	Master switch. `true` registers `/debug/pprof/*`.
`STRATA_PPROF_LISTEN`	`pprof.listen`	empty	Optional dedicated listener (e.g. `127.0.0.1:9002`). Empty → attach to `admin_listen.listen`. One of the two MUST be set when enabled — pprof never attaches to the S3 hot path.
`STRATA_PPROF_BLOCK_RATE`	`pprof.block_rate`	`0`	`runtime.SetBlockProfileRate(N)` argument. `0` keeps block profile data empty.
`STRATA_PPROF_MUTEX_RATE`	`pprof.mutex_rate`	`0`	`runtime.SetMutexProfileFraction(N)` argument. `0` keeps mutex profile data empty.

STRATA_PPROF_ENABLED=true with neither STRATA_PPROF_LISTEN nor STRATA_ADMIN_LISTEN set fails fast at boot — the gateway refuses to silently expose profiling on the public S3 listener.

Per-bucket signing key rotation

Mon, 01 Jan 0001 00:00:00 +0000

Per-bucket signing key rotation#

Strata supports KMS-backed per-bucket SigV4 signing keys (US-001 / US-002 of ralph/auth-dx-trailer-lima). Each opted-in bucket carries a 32-byte DEK wrapped under a KMS CMK (AWS KMS / Vault Transit / LocalHSMProvider) plus a creation timestamp. The auth middleware unwraps on cache miss and uses hex(DEK) as the SigV4 secret in place of the IAM access-key secret.

This page covers the day-2 rotation runbook. The KMS provider reference lists the per-provider env vars; the admin API reference documents the three rotation endpoints.

Scaling

Mon, 01 Jan 0001 00:00:00 +0000

Scaling#

Strata replicas are stateless gateways — they own no on-disk state, so sizing is a function of three signals: peak request rate (RPS), the working-set bytes the gateway holds in flight (RAM), and the cores the SigV4 + crypto path consumes (CPU). The metadata + data tiers (Cassandra / TiKV / RADOS) follow their own upstream sizing guides; this page covers the gateway tier and points operators at the upstream references.

Backup + restore

Mon, 01 Jan 0001 00:00:00 +0000

Backup + restore#

Strata splits backup responsibility along the same tiers the gateway splits state into:

Metadata tier (Cassandra / TiKV): the source of truth for bucket / object / IAM rows. Backed up via the upstream tooling for each backend.
Data tier (RADOS / S3-over-S3): chunk bytes. Backed up via Ceph pool snapshots or the upstream S3 service’s native versioning / replication.
Cross-region replication (replicator worker): ships object PUT / DELETE events to a peer Strata cluster in near-real time.
Inventory worker: writes manifest.json + CSV.gz pairs that document every object in a bucket — useful as an audit ledger for external backup tooling.

This page is a backup-strategy overview. Implementation details live in Architecture — Workers (the inventory + replicator workers) and the upstream docs for Ceph / Cassandra / TiKV.

Reconcile + rebuild-index

Mon, 01 Jan 0001 00:00:00 +0000

Reconcile + rebuild-index#

Strata splits the S3-key→chunk map (meta tier: TiKV / Cassandra) from the chunk bytes (data tier: RADOS / S3-over-S3). That split is the scale advantage — but it drops the safety net RGW gets by colocating its bucket index with the data. After you restore one tier from a backup that is even slightly out of step with the other, the two tiers diverge silently:

Orphan chunks — data has a chunk no manifest references. GC walks meta→data, so it can never see them; they leak storage forever.
Dangling manifests — meta has a manifest pointing at a chunk the data tier no longer holds. The client discovers it as a 5xx on GET, not on the restore.
Lost meta backup — the bytes are intact in RADOS but unreadable, because the key→chunk map is gone.

This page is the post-restore runbook for realigning the two tiers. It pairs with Backup + restore (how each tier is snapshotted) — run reconcile after the restore drill, before you promote traffic.

Capacity planning

Mon, 01 Jan 0001 00:00:00 +0000

Capacity planning#

Capacity on a Strata deploy splits across three independently-scaled tiers:

Gateway tier: stateless replicas. Capacity = peak RPS / SigV4 + routing budget per replica. Scale horizontally; see Scaling.
Metadata tier: Cassandra / TiKV. Capacity = total object-row count × per-row overhead, plus the compare-and-set or pessimistic-txn rate during writes.
Data tier: RADOS / S3-over-S3. Capacity = total object bytes × pool replication factor, plus chunk fan-out overhead.

This page covers the math; pair with Backup + restore for the backup overhead and GC + lifecycle tuning for the drain rate.

TLS termination + backend mTLS

Mon, 01 Jan 0001 00:00:00 +0000

TLS termination + backend mTLS#

This is the operator playbook for putting Strata on the wire securely. The gateway can terminate TLS itself (recommended for prod) or sit behind a load balancer that handles termination upstream. Backend connections (Cassandra, TiKV, S3-upstream) each take an opt-in mutual-TLS bundle so a network intruder cannot impersonate the gateway. RADOS is the one exception: it authenticates via Ceph’s native cephx protocol and ignores the TLS layer entirely (see RADOS uses cephx below).

Image verification

Mon, 01 Jan 0001 00:00:00 +0000

Image verification#

Every Strata image published to ghcr.io/danchupin/strata is signed, attested, and inventoried by the release pipeline (.github/workflows/release-image.yml). This page is the operator playbook for verifying that supply-chain evidence before the image reaches a node, plus two Kubernetes admission-control shapes that make the check mandatory at deploy time.

The pipeline produces four pieces of evidence, all anchored to the same image digest:

Evidence	Producer	Story	Verified with
cosign signature (keyless OIDC)	`sigstore/cosign`	US-008	`cosign verify`
SLSA L3 provenance	`slsa-github-generator`	US-006	`cosign verify-attestation`
SPDX 2.3 SBOM	`anchore/sbom-action` (syft)	US-007	`grype` / `trivy sbom`
License inventory	`go-licenses`	US-009	`go-licenses report`

What signing buys you#

Keyless signing gives you a non-repudiable provenance trail from source to running image: the GitHub Actions OIDC identity that built the image is the signer, Fulcio issues a short-lived certificate bound to that identity, and the signature lands in the Rekor public transparency log. Because the signature covers the image digest (not a mutable tag), a verified image is tamper-proof — repush a different layer and the signature no longer matches. There is no maintainer-held private key to leak or rotate; the trust root is GitHub’s OIDC issuer plus the public Sigstore infrastructure. The Rekor entry is append-only and publicly queryable, so a forensic investigation can prove exactly which workflow run, at which commit, produced a given digest — even years later.

Pre-prod validation walkthrough

Mon, 01 Jan 0001 00:00:00 +0000

Pre-prod validation walkthrough#

This is the release-owner runbook: a single documented pass that brings up both production labs and exercises the whole hardened object path so the GO/NO-GO decision rests on observed behaviour, not story-by-story claims. It is the operator-facing companion to the per-feature smokes — it composes them into one walkthrough and records a pass/fail-per-leg readiness note.

The composite is scripts/smoke-architecture-hardening.sh, wired as make smoke-architecture-hardening. Each leg is independent: it WARN-skips when its lab is down and only fails the run on a real assertion failure. Set REQUIRE_LAB=1 to promote a down lab to a hard fail (use this in CI where the labs are guaranteed up).