GC + Lifecycle worker scaling benchmark (Phase 1 + Phase 2)#

Phase 2 (US-006) lands multi-leader numbers below Phase 1’s single-leader baseline. The two-section structure is intentional — Phase 1 measured the per-leader concurrency cap, Phase 2 measures the multiplier from running N replicas in parallel under STRATA_GC_SHARDS=N.

Phase 1 — single-leader concurrency cap#

Quantifies the throughput curve for internal/gc.Worker and internal/lifecycle.Worker across the per-worker bounded-errgroup fan-out introduced in cycle ralph/gc-lifecycle-scale (US-001 / US-002). The harness lands as strata admin bench-gc + strata admin bench-lifecycle (US-003) and is wired into make bench-gc / make bench-lifecycle to drive the canonical lab-tikv stack.

Harness#

# Bring up the bare-default TiKV stack (PD + TiKV + ceph + ceph-b) but
# stop the bundled strata-{a,b} replicas first so their gc/lifecycle
# workers don't compete for the bench region.
make up && make wait-strata-lab
docker stop strata-a strata-b strata-lb-nginx

# Sweep concurrency=1,4,16,64,256 (default).
make bench-gc          # -> bench-gc-results.jsonl
make bench-lifecycle   # -> bench-lifecycle-results.jsonl

Each invocation pre-seeds N synthetic rows under a unique bench-<uuid> region (gc) or lcbench-<uuid> bucket (lifecycle), runs Worker.RunOnce, prints one JSON line on stdout, and cleans up the seeded rows on exit. Override BENCH_GC_ENTRIES, BENCH_LC_OBJECTS, BENCH_CONCURRENCY_LEVELS to widen the sweep.

The Makefile targets default to STRATA_DATA_BACKEND=rados against the lab’s ceph pool. The numbers below were captured with STRATA_DATA_BACKEND=memory to isolate the meta-side worker throughput from RADOS round-trip — see Methodology + Latent bug below for the rationale.

Methodology#

• Stack: lab-tikv profile from deploy/docker/docker-compose.yml (PD v8.5, TiKV v8.5, single replica each). Same docker network as the bench container.
• Hardware: macOS host, lima Docker context, single-node PD+TiKV. Numbers are directional — the absolute throughput will rise on multi-node TiKV and bare-metal hardware. The shape of the curve (relative speedup per doubling) is the load-bearing signal.
• Workload size: N=10000 entries / objects per level. Wall-clock noise dominates below ~100 ms; trust the per-level deltas for c≤64, treat c=256 as a saturation indicator rather than a precise number.
• Data backend: memory. Real production lab-tikv would use RADOS; we deliberately bypass RADOS here because the gc worker infinite-loops on Data.Delete failure (latent bug, see below). Memory backend deletes succeed instantly so the bench measures the worker’s meta-side fan-out cap. Once the gc-ack-on-ENOENT bug is fixed, re-running with STRATA_DATA_BACKEND=rados is mechanical.
• Meta backend: TiKV. AckGCEntry / DeleteObject / EnqueueChunkDeletion all use pessimistic txns; that’s the path Phase 1 parallelises.

Results — bench-gc#

Per-doubling yield: 1→4 +127 %, 4→16 +84 %, 16→64 +94 %, 64→256 +11 %.

Results — bench-lifecycle#

Per-doubling yield: 1→4 +155 %, 4→16 +144 %, 16→64 +115 %, 64→256 +41 %.

Bottleneck per tier#

The lab-tikv RADOS path is not measured here; in production the gc worker’s per-entry cost is dominated by RADOS round-trip to delete the chunk. Once internal/gc.Worker is fixed to ack on ENOENT (latent bug below), the same harness can re-run with STRATA_DATA_BACKEND=rados to quantify the RADOS contribution.

Knee / recommended defaults#

The knee is the concurrency above which per-additional-goroutine yield falls below 10 %.

• gc: 64→256 yields +11 %, just above the 10 % threshold. The gc knee on this lab is at c=64. Recommended production default: STRATA_GC_CONCURRENCY=64. Operators serving very-high-churn workloads can probe up to 128–256 — diminishing returns from there are TiKV-region-heat bound (Phase 2 territory).
• lifecycle: 64→256 still yields +41 %, well above the 10 % threshold; no knee inside the swept range. Recommended production default: STRATA_LIFECYCLE_CONCURRENCY=64 for safety (memory + TiKV connection pool footprint stays bounded). Operators with large expiration / transition windows can push to 128–256 if the meta backend has headroom.

Both defaults assume the metadata backend can absorb the extra pessimistic-txn fan-out. On Cassandra the same envs translate into parallel LWT round-trips; the LWT throughput cap on a single partition will dominate the curve. Re-run the bench against a Cassandra-backed lab and update this file when that becomes a target.

Tradeoffs at very-high concurrency#

• Memory: each errgroup goroutine carries the meta.GCEntry / meta.Object plus a TiKV pessimistic-txn lease. At c=256 with in-flight 256 acks, RSS grows by a few hundred KiB — negligible.
• Connection pressure: tikv/client-go uses a fixed-size gRPC pool; at c=256 the pool can saturate and incoming work queues on the client side, which manifests as throughput plateau. Tune STRATA_TIKV_CLIENT_* envs (none plumbed yet — Phase 2 hook) only if profiling shows pool starvation.
• TiKV region heat: the GC queue is keyed by region; concentrating 256 concurrent writes on one region will trigger TiKV’s split + load balance, briefly stalling the worker. In production the bench bench-<uuid> region prefix avoids real-traffic collision; production workers run against the canonical region=default so the same contention shape applies.
• RADOS pool heat (production): the missing tier in this bench. When STRATA_DATA_BACKEND=rados, every gc.Worker goroutine issues a RADOS remove round-trip; at c=256 against a single Ceph monitor this becomes the new dominant cost.

Latent bug discovered while running#

gc.Worker.drainCount does not ack a GC entry when Data.Delete returns an error. Combined with the outer for {} loop that re-issues ListGCEntries until it returns < Batch, any persistent Data.Delete failure (RADOS pool not found, ENOENT for an OID that was already swept by a concurrent leader, transient timeout) spins the worker forever on the same batch. The bench surfaced this immediately when run against the real strata.rgw.buckets.data pool with synthetic OIDs — every rados-2 (No such file or directory) warning kept the entry in the queue, and RunOnce never returned.

The fix is local: ack on ENOENT (the chunk is gone — that’s the desired terminal state), and probably ack on any non-retryable error class to avoid the spin. Tracked under Known latent bugs in ROADMAP.md and out of scope for this cycle’s Phase 1 closure.

Reproducing#

The Makefile defaults to RADOS data backend, which trips the bug above; for a clean reproduction of these numbers use the memory data backend instead:

make up && make wait-strata-lab
docker stop strata-a strata-b strata-lb-nginx

for c in 1 4 16 64 256; do
  docker run --rm --network docker_default \
    -v docker_strata-ceph-etc:/etc/ceph:ro \
    -e STRATA_META_BACKEND=tikv \
    -e STRATA_DATA_BACKEND=memory \
    -e STRATA_TIKV_PD_ENDPOINTS=pd:2379 \
    -e STRATA_LOG_LEVEL=ERROR \
    --entrypoint /usr/local/bin/strata strata:ceph \
    admin bench-gc --entries=10000 --concurrency=$c
done

Same shape for bench-lifecycle with --objects=10000.

Phase 2 — multi-leader (US-006)#

Phase 2 retires the single-leader cap by sharding the gc-leader lease space (gc-leader-0..N-1) and gating lifecycle on a fnv32a(bucketID) % N == myID distribution filter. STRATA_GC_SHARDS=N is the operator-facing knob; each replica races for its lease and processes only the buckets / queue slice it owns. The bench harness gains a matching --shards=N flag on bench-gc and --replicas=N on bench-lifecycle — both spawn N parallel workers in one process so the multi-leader curve can be measured without standing up N hosts. The 3-replica lab profile (lab-tikv-3 in deploy/docker/docker-compose.yml, brought up via make up-lab-tikv-3) is the canonical production-shaped target; the in-process simulation is a sanity check on the worker code-path overhead.

Methodology#

• Stack (canonical): make up-lab-tikv-3 — PD + TiKV + 3 strata-tikv replicas (strata-tikv-{a,b,c}, container ports 9001/9002/9003, STRATA_GC_SHARDS=3) behind nginx LB on host port 9999.
• Stack (this commit): in-process simulation — one strata admin process spawns N workers with Worker{ShardID:i, ShardCount:N} (gc) or lifecycle.Worker{ReplicaInfo: () => (N, i)} (lifecycle) racing for an in-process memory.NewLocker() lease. Real-world contention shapes (region heat, RADOS round-trip) are NOT exercised — see Caveats below.
• Workload size: gc N=50,000 entries; lifecycle N=10,000 objects spread across 9 buckets so the distribution gate has work to spread (replicas=3 ⇒ each replica owns 3 buckets).
• Concurrency sweep: c∈{1, 4, 16, 64, 256} per leader/replica.
• Backends: memory + memory. Phase 1’s TiKV+memory shape is the apples-to-apples production rerun target — not feasible from the autonomous bench harness on this dev box (no docker/lima). Operators rerun against the 3-replica lab via make bench-gc-multi / make bench-lifecycle-multi.

Results — bench-gc, in-process simulation#

JSONL artifacts: docs/site/content/architecture/benchmarks/data/gc-lifecycle-phase-2/sim-bench-gc-{shards1,shards3}.jsonl.

(Throughput in chunks/s. Single-process simulation, memory backend.)

The shards=3 column at low concurrency dips slightly below shards=1 because adding a 3-way fan-out on the same in-process meta store buys nothing (the mutex already serialises everything) and pays the goroutine-coordination cost. From c=16 onwards the two columns converge — the Go RWMutex hit on the memory store is the dominant cost regardless of shard count. This is the expected sim-mode shape, not the production shape; in production the TiKV pessimistic-txn round-trip is per-shard independent, so 3-shard parallelism translates into ~3× throughput at the same per-replica concurrency (subject to TiKV region-heat saturation — see “Cap shape” below).

Results — bench-lifecycle, in-process simulation#

JSONL artifacts: docs/site/content/architecture/benchmarks/data/gc-lifecycle-phase-2/sim-bench-lifecycle-{replicas1,replicas3}.jsonl.

(Throughput in objects/s. Replicas=3 distributes 9 buckets across 3 parallel workers under lifecycle-leader-<bucketID> per-bucket leases; replicas=1 processes the same 9 buckets sequentially through one worker.)

The lifecycle multiplier shows up even in pure-memory mode because the distribution gate fans bucket-scan + per-object DeleteObject work across parallel goroutines instead of serialising them inside one bucket loop. The c=1 row’s 1.87× is the strongest signal — at low per-replica concurrency the parallelism is real because the work doesn’t saturate any shared resource. At higher c the in-process meta-mutex caps the gain at ~1.2× (vs the expected ~3× on real TiKV).

Phase 1 → Phase 2 expected production multiplier#

Sim-mode numbers above pin the shape but not the magnitude. Magnitude on production stack (TiKV meta + RADOS data) follows from Phase 1’s measured per-replica curve × the number of leaders the multi-shard fan-out grants:

Operators reading these numbers as forecasts: rerun make bench-gc-multi / make bench-lifecycle-multi (US-006 Makefile targets) against your 3-replica lab and update this section with the measured rows. The sim-mode rows above are the rate floor — actual TiKV-backed numbers should land notably higher.

Cap shape — when does Phase 2 stop scaling?#

The per-shard scan + ack pipeline parallelises cleanly up to the point where one of these saturates first:

1. TiKV region heat on the gc queue partition — at very high N, the (region, shard_id) partitions all hash into a small number of TiKV regions; once one region’s leader is the bottleneck, adding shards stops helping. The 1024-wide logical-shard fan-out + 2-byte BE shard prefix on TiKV is designed to spread keys across regions, but the underlying region split happens reactively. Knee: ~16–32 runtime shards on a 3-node TiKV cluster before region splits stabilise.
2. RADOS round-trip on chunk delete — the gc worker’s per-entry cost in production includes a librados remove. At STRATA_GC_SHARDS=N > the OSD-pool’s request concurrency cap, additional shards just queue on the RADOS client side. Knee: 8–16 shards before OSD pool heat shows up.
3. Bucket count vs replica count for lifecycle — the per-bucket distribution gate maps fnv32a(bucketID) % N. With fewer buckets than replicas, hash collisions waste replicas. Knee: lifecycle gain plateaus at min(N, distinct-bucket-count). Three-replica deploy with 3 lifecycle-active buckets ≈ Phase 1 throughput because each bucket pins to one replica.

The gc queue itself is no longer the bottleneck post-US-001..US-003 — the partition / prefix split lets each shard scan independently. Sustained 40k chunks/s production churn (the Phase 2 goal) translates to a per-shard rate of ~13.3k/s under STRATA_GC_SHARDS=3, well inside Phase 1’s measured single-leader cap (~90k/s at c=64). RADOS round-trip is the next ceiling.

Operator-facing recommendation#

• 3-replica deploy: STRATA_GC_SHARDS=3, STRATA_GC_CONCURRENCY=64 (per replica). Aggregate ceiling ≈ 3× Phase 1 per-replica cap. Lifecycle rides on the same STRATA_GC_SHARDS value via the per-bucket distribution gate; no separate STRATA_LIFECYCLE_SHARDS knob exists.
• N-replica deploy in general: scale STRATA_GC_SHARDS to match replica count up to the bucket-shard cardinality limit (1024). Going beyond replica count is a no-op (extra leases unfilled); going below leaves replicas without work (idle). For lifecycle, ensure STRATA_GC_SHARDS ≤ active-bucket-count or hash collisions cap the gain.
• Single-replica fallback: STRATA_GC_SHARDS=1 (default) reproduces Phase 1 behaviour byte-for-byte. Existing make smoke / make smoke-tikv / make smoke-lab-tikv continue to pass under this default — Phase 2 is opt-in via env.
• Cutover: dual-write window for gc_entries_v2 (Cassandra) and the new TiKV gc/<region>/<shardID2BE>/<oid> key shape stays on by default (STRATA_GC_DUAL_WRITE=on). Flip off after the legacy partition / prefix is empty (operator-confirmed via inventory or queue depth metrics). See GC + lifecycle Phase 2 (US-007).

Caveats#

• The in-process simulation does not exercise:
  • Real TiKV pessimistic-txn round-trip latency — the production multiplier relies on each shard’s drain pipeline filling independently across the TiKV gRPC pool.
  • Real RADOS remove round-trip — every Phase 2 shard issues independent Delete calls; a single Ceph monitor under N concurrent delete bursts is the new ceiling, not the meta queue.
  • Cross-host lease lottery — production replicas race for gc-leader-<i> via the cassandra LWT or TiKV pessimistic-txn locker; in-process the locker is FIFO so the first goroutine wins each shard at startup. Steady-state ownership rotation is therefore not measured.
• The 3-replica lab (make up-lab-tikv-3) is the canonical reproduction target. Update this doc’s tables with measured rows from that lab; the in-process numbers above are the noise floor / regression guard, not the operator-facing forecast.