Parallel chunk PUT + GET (RADOS)#

Closes the two open ROADMAP.md P2 entries:

• Parallel chunk upload in PutChunks — sequential for { ReadFull; writeChunk } loop replaced with a bounded worker pool that dispatches chunk writes concurrently. Manifest order + ETag (MD5 of the byte stream) are preserved.
• Parallel chunk read / prefetch in GetChunks — sequential per-chunk fetch replaced with a bounded prefetch reader. The next chunk is fetched while the current one drains to the wire. Memory-bounded; aborts on client cancel.

The S3 data backend (internal/data/s3/backend.go) is unaffected: the AWS SDK manager.Uploader already parallelises multipart uploads. Memory backend is tests-only. Multi-cluster manifests (US-044) automatically benefit since chunk-write dispatch is cluster-agnostic — the worker pool dials the right ioctx per chunk via the existing resolveClass / ioctx helpers.

Tuning knobs#

Both are read once at rados.Backend construction time (New), not per request. Restart the gateway to apply a change.

Memory budget per request#

• PUT: concurrency × chunk_size. Default 32 × 4 MiB = 128 MiB inflight per upload at saturation. The per-call hot path also allocates one 4 MiB read buffer in the dispatcher goroutine.
• GET: prefetch × chunk_size. Default 4 × 4 MiB = 16 MiB inflight per reader. The semaphore acquires a slot before launching a fetcher and the reader releases it after consuming the body, so peak buffered-but-unconsumed bytes never exceeds prefetch × chunk_size.

When to raise#

• Large objects + high per-OSD latency. RADOS p99 write latency of 20–50 ms on multi-MiB OSDs dominates wall-clock. Raising STRATA_RADOS_PUT_CONCURRENCY to 64–128 closes that gap. Same for STRATA_RADOS_GET_PREFETCH=8..16 for streaming GETs on large objects.
• Multi-cluster manifests (US-044). Each chunk’s ioctx may sit on a different cluster; the worker pool naturally distributes load. Higher concurrency lets the gateway saturate multiple clusters in parallel.

When to lower#

• Small-object workloads. Sub-chunk objects (single chunk) see no benefit from concurrency; lower the knobs to reduce per-request allocator pressure and goroutine-spawn overhead.
• Memory pressure. Each gateway replica caps memory at replicas × inflight_requests × concurrency × chunk_size. With default 4 MiB chunks and STRATA_RADOS_PUT_CONCURRENCY=32, a single uploader pins 128 MiB.

Reader-side bottleneck (PUT)#

The PutChunks path is gated by a single dispatcher goroutine that owns the MD5 hasher (required to preserve the byte-stream-MD5 ETag invariant) and the input stream. On synthetic benches with ≤ 5 ms per-OSD latency the dispatcher cost (read + hash + handoff) is comparable to the worker cost, so concurrency gain is modest. On production traffic with 20–100 ms per-OSD p99, the gain scales close to linearly until concurrency × dispatcher_throughput exceeds the byte stream’s incoming bandwidth. ETag and manifest order remain exact under all concurrency settings.

Benchmarks#

Synthetic harness in internal/data/rados/backend_bench_test.go. The fake chunk-put / chunk-get callbacks inject a 5 ms per-op time.Sleep (deliberate lower bound for RADOS round-trip on a healthy local OSD); the bench reports wall-clock for a full PutChunks / GetChunks invocation against in-memory chunk buffers, no librados, no kernel network stack.

go test -bench='^Benchmark(Put|Get)Chunks' -benchtime=5x -run='^$' \
  -benchmem ./internal/data/rados/...

Apple M3 Pro, Go 1.25, 5 iterations per case:

Interpreting the numbers#

• GET speedup matches the headline. The prefetch reader exposes the full depth=4 pipeline benefit because each chunk fetch is independent. Throughput is bounded by chunk_size / per_op_latency × depth = 4 MiB / 5 ms × 4 = 3.2 GB/s ceiling; the 2.7–3 GB/s observed sits in that range (the gap is reader copy + sem release).
• PUT speedup is modest at low per-OSD latency. The synthetic 5 ms perOp is comparable to the dispatcher’s per-chunk MD5 + alloc cost on the input stream, so concurrency cannot hide the dispatcher path. As per-OSD latency grows (production: 20–100 ms p99), the dispatcher’s relative cost shrinks and the worker pool starts to dominate — operators see the headline near-linear-to-concurrency speedup on slow OSDs. Bench harness is conservative on this front; raise benchPerOp in the bench file to model a slower OSD locally.

Behaviour invariants#

Both schedulers (internal/data/rados/parallel.go, internal/data/rados/prefetch.go) live in tag-free files so unit tests exercise them on macOS without librados.

• Source-byte order. Manifest chunks emit in source order regardless of worker completion order (putChunksParallel). Prefetch reader emits bytes in source order regardless of fetch completion order (prefetchReader).
• ETag = MD5 of byte stream. The dispatcher owns the hasher and feeds it in source order before dispatching the chunk to the pool. No per-worker partial-hashes.
• Error semantics. On first worker error, gctx cancels, in-flight workers drain, the partial manifest is returned to the caller, and the caller runs cleanupManifest over OIDs that landed in RADOS. No leaked partial-write OIDs.
• Reader / client cancel. ctx.Err() is honoured at chunk-read boundary and at worker dispatch (PUT) / at the future channel (GET). Prefetch goroutines exit within 500 ms of radosReader.Close or parent-ctx cancel (no goroutine leak — verified by runtime.NumGoroutine() baseline test).
• Per-chunk observability. ObserveOp + tracer spans fire from the worker goroutine, so duration metrics + spans reflect the actual OSD Write / Read time, not wait-on-channel time. Multi-cluster manifests are handled by the worker resolving the per-chunk ioctx via b.ioctx(ctx, c.Cluster, c.Pool, c.Namespace) — no special-case multi-cluster code path.

Reproducing on RADOS#

The synthetic bench is a wall-clock floor — real-world make up-all numbers sit higher (network + librados + OSD i/o) but exhibit the same scaling shape. To re-run against a live cluster:

make up-all
go test -tags ceph -run TestRADOSBackendIntegration ./internal/data/rados/...

…then drive PUT / GET via aws s3 cp against the gateway and tune STRATA_RADOS_PUT_CONCURRENCY / STRATA_RADOS_GET_PREFETCH for the workload.