Benchmarks on Strata Documentation

GC + Lifecycle scaling

Mon, 01 Jan 0001 00:00:00 +0000

GC + Lifecycle worker scaling benchmark (Phase 1 + Phase 2)#

Phase 2 (US-006) lands multi-leader numbers below Phase 1’s single-leader baseline. The two-section structure is intentional — Phase 1 measured the per-leader concurrency cap, Phase 2 measures the multiplier from running N replicas in parallel under STRATA_GC_SHARDS=N.

Phase 1 — single-leader concurrency cap#

Quantifies the throughput curve for internal/gc.Worker and internal/lifecycle.Worker across the per-worker bounded-errgroup fan-out introduced in cycle ralph/gc-lifecycle-scale (US-001 / US-002). The harness lands as strata admin bench-gc + strata admin bench-lifecycle (US-003) and is wired into make bench-gc / make bench-lifecycle to drive the canonical lab-tikv stack.

Meta-backend comparison

Mon, 01 Jan 0001 00:00:00 +0000

Meta-backend benchmarks: TiKV vs Cassandra#

This page captures hot-path latency / throughput numbers for Strata’s two production metadata backends (Cassandra and TiKV) and the in-tree memory reference. The numbers are operator-runnable on a single laptop docker stack via the harness in internal/meta/storetest/bench.go.

The headline operations (US-018):

Op	Why it matters
`CreateBucket`	LWT-equivalent create-if-not-exists
`GetObject`	Single Get / latest-version range-scan
`ListObjects` 100k page=1k	Listing throughput on a real-world bucket
`CompleteMultipartUpload`	LWT-equivalent status flip (`uploading→completing`)
`GetIAMAccessKey`	SigV4 verifier — runs on every request
`AuditAppend`	Audit log write amplification
`AuditSweepPartition`	Periodic retention sweep (TiKV emulates Cassandra TTL)
`RangeScanObjects` 100k	TiKV-only: native ordered scan vs Cassandra fan-out

Rig#

Single-laptop docker stack. Both backends run as 3-node clusters so LWT

Rebalance scaling

Mon, 01 Jan 0001 00:00:00 +0000

Rebalance worker scaling benchmark (Phase 2)#

Closes ROADMAP P2 “Rebalance worker not sharded — single goroutine bottleneck on large deploys”. Mirrors the gc + lifecycle Phase 2 shape (see GC + Lifecycle scaling for the same fan-out pattern measured against gc and lifecycle workers).

The single-leader rebalance worker scans every draining bucket per tick on one goroutine. Phase 2 (STRATA_REBALANCE_SHARDS=N) shards that scan by fnv32a(bucketID) % N; each replica races for one of the rebalance-leader-0..N-1 leases and processes only its own shard’s bucket subset. STRATA_REBALANCE_SHARDS=1 reproduces Phase 1 byte-for-byte.

Parallel chunk PUT + GET (RADOS)

Mon, 01 Jan 0001 00:00:00 +0000

Parallel chunk PUT + GET (RADOS)#

Closes the two open ROADMAP.md P2 entries:

Parallel chunk upload in PutChunks — sequential for { ReadFull; writeChunk } loop replaced with a bounded worker pool that dispatches chunk writes concurrently. Manifest order + ETag (MD5 of the byte stream) are preserved.
Parallel chunk read / prefetch in GetChunks — sequential per-chunk fetch replaced with a bounded prefetch reader. The next chunk is fetched while the current one drains to the wire. Memory-bounded; aborts on client cancel.

The S3 data backend (internal/data/s3/backend.go) is unaffected: the AWS SDK manager.Uploader already parallelises multipart uploads. Memory backend is tests-only. Multi-cluster manifests (US-044) automatically benefit since chunk-write dispatch is cluster-agnostic — the worker pool dials the right ioctx per chunk via the existing resolveClass / ioctx helpers.

RGW comparison

Mon, 01 Jan 0001 00:00:00 +0000

Benchmarks vs Ceph RGW#

This page captures the side-by-side numbers backing the README’s “drop-in RGW replacement” claim. Closes ROADMAP P2 “Benchmarks vs RGW” via cycle ralph/rgw-benchmarks (US-001..US-012); the post-fix rerun landed via ralph/p1-fixes (US-001..US-009) after the per-bucket bucket_stats LWT saturation was removed.

Headline conclusion#

Bucket-index claim — VERIFIED (provisionally) at the concurrency RGW can sustain. Two independent regimes surface the win:

1 KiB PUT @ c=8 (the canonical “many small writes” hot path): Strata p99 ≈ 211 ms vs RGW p99 11–22 s (runs varied; one run wedged at 364 s). Strata is ~50× faster at p99 and stays flat across the 3-run set; RGW’s omap-index serialisation surfaces immediately past c=1 on the single-OSD lab. Same data tier (single-OSD ceph-a memstore), same make up-bench-rgw shape — the delta is gateway-side, not OSD.
ListObjects 100k-key @ c=8: Strata sustains ~208 k ops/s @ p99 ~640 ms across 3 runs, zero errors. RGW could not be measured at this scale on the lima reference dev box — the per-bucket pools default.rgw.buckets.index saturate the memstore OSD before the seed phase completes. The very fact that RGW cannot complete the workload at the same lab shape reinforces the bucket-index thesis: Strata’s sharded fan-out (s/B/<bid>/bs/<shard> × 8 shards on TiKV; sharded objects table on Cassandra) absorbs the same concurrent write pressure RGW’s omap index falls over on.

RGW wins at c=1 small PUT as expected: Strata p99 ≈ 12 ms vs RGW p99 ≈ 2 ms. The user-space SigV4 + policy-verify hop Strata adds vs RGW’s in-process auth costs ~10 ms on the single-threaded hot path; the gap closes as soon as concurrency arrives because the gateway-side per-request cost stops dominating relative to backend serialisation.

RADOS ReadOp / WriteOp batching

Mon, 01 Jan 0001 00:00:00 +0000

RADOS ReadOp / WriteOp batching#

Closes the ROADMAP.md P3 entry ReadOp / WriteOp batching in RADOS (US-003 of ralph/storage-correctness).

What batching does#

internal/data/rados/ops.go (build tag ceph) defines two helpers that bundle a chunk Write / Read with N xattr ops into a single librados operation:

writeChunkBatched(ioctx, oid, body, xattrs) — builds a goceph.WriteOp, appends WriteFull(body) + SetXattr(k, v) per entry of xattrs, then Operate()s once. xattrs nil ⇒ byte-identical to the legacy writeChunk in backend.go.
readChunkBatched(ioctx, oid, off, length, wantXattrs) — builds a goceph.ReadOp with Read(off, buf), then Operate()s once. When wantXattrs is true a follow-up ioctx.ListXattrs runs to populate the xattrs map — go-ceph v0.39 does not expose rados_read_op_getxattrs, so xattrs sit on a second client round-trip. Today no caller requests xattrs.

writeChunk (legacy) and writeChunkBatched(_, _, _, nil) produce the same on-wire librados WriteOp{ WriteFull }. The win materialises only when xattrs are added to the PUT hot path — a future cycle. The helpers exist now so the call sites are wired and the bench shape is established.

RADOS conn-pool sizing

Mon, 01 Jan 0001 00:00:00 +0000

RADOS conn-pool sizing#

Closes the ROADMAP.md P3 entry Connection pool tuning (US-004 of ralph/storage-correctness).

What the pool does#

A single librados *Conn serialises ops through one cephx session and a per-conn thread pool. Write-heavy workloads that saturate that single session see contention even though the OSD layer has headroom. The connPool in internal/data/rados/pool.go (build tag ceph) holds N pre-connected *goceph.Conn instances per cluster and serves them round-robin via an atomic counter; per-conn IOContexts are cached lazily by (pool, namespace).

OTel ring-buffer bytes budget

Mon, 01 Jan 0001 00:00:00 +0000

OTel ring-buffer bytes budget#

Closes the ROADMAP.md P3 entry OTel ring-buffer eviction tuning under burst load (US-005 of ralph/storage-correctness).

What the ring buffer does#

internal/otel/ringbuf.RingBuffer is an in-process SpanProcessor that retains every finished span under an LRU + bytes-budget. The /admin/v1/diagnostics/trace/{requestID} admin endpoint reads from the ring so operators can debug a request without a Jaeger / Tempo deployment. Tail sampling on the exporter side still drops most spans on the wire — the ring keeps the full payload locally.

Mon, 01 Jan 0001 00:00:00 +0000

Phase 2 — gc + lifecycle multi-leader bench artifacts#

JSONL captured by the bench harness in in-process simulation mode: both meta and data backends are pure-memory; the multi-shard / multi- replica race happens inside one strata admin process via the --shards (gc) / --replicas (lifecycle) flags wired in US-006.

Files#

File	What	How to regenerate
`sim-bench-gc-shards1.jsonl`	gc bench, single-leader (Phase 1 shape), N=50000, c∈{1,4,16,64,256}	`for c in 1 4 16 64 256; do STRATA_META_BACKEND=memory STRATA_DATA_BACKEND=memory ./bin/strata admin bench-gc --entries=50000 --concurrency=$c --shards=1; done \| jq -c .`
`sim-bench-gc-shards3.jsonl`	gc bench, 3-shard (Phase 2 shape), N=50000	…`--shards=3`
`sim-bench-lifecycle-replicas1.jsonl`	lifecycle bench, single-replica (Phase 1), N=10000, 1 bucket	`for c in …; do … bench-lifecycle --objects=10000 --concurrency=$c --replicas=1 --buckets=1; done`
`sim-bench-lifecycle-replicas3.jsonl`	lifecycle bench, 3-replica (Phase 2), N=10000, 9 buckets	…`--replicas=3 --buckets=9`

Caveats#

Memory meta + memory data put the worker on a fast path with effectively zero per-op latency. Concurrency speedup that the production lab observes on TiKV (real pessimistic-txn round-trips) does NOT manifest here — the in-process Go RWMutex is the bottleneck. Treat these numbers as a noise floor that pins the shape of multi-leader coordination overhead, not as a forecast of production throughput.
For the canonical Phase 2 numbers operators should rerun make bench-gc-multi / make bench-lifecycle-multi against the 3-replica lab brought up via make up-lab-tikv-3 (see docs/site/content/architecture/benchmarks/gc-lifecycle.md’s “Phase 2 — multi-leader” section).
Every JSON object on a line is one bench level (one (--concurrency, --shards|--replicas) combination). Schema matches cmd/strata/admin/ bench_common.go::benchResult.