Sharding#

Sharding is the single biggest divergence from Ceph RGW: every bucket’s metadata is split across N partitions instead of living in a single bucket-index object. The split avoids RGW’s bucket-index ceiling at large object counts and lets ListObjects scale linearly with N.

Objects table partition key#

The cassandra objects table is partitioned by (bucket_id, shard):

PRIMARY KEY ((bucket_id, shard), key, version_id)
WITH CLUSTERING ORDER BY (key ASC, version_id DESC)

Where shard = fnv32a(key) % N and N is per-bucket (STRATA_BUCKET_SHARDS at bucket creation, default 64). N must be a power of two — meta.IsValidShardCount(n) enforces it. Power-of-two constraint matters for the reshard worker below: when N doubles, every old shard either stays under the new modulo or splits cleanly into two new ones, never three.

Per-shard partition shape:

Per-shard size is bounded by your bucket’s per-shard fanout target (typically <= 100k keys per shard). The meta-backend benchmark covers the per-shard read/write profile.

ListObjects fan-out + heap merge#

ListObjects queries N partitions concurrently and merges results by clustering order. The merge logic is in internal/meta/cassandra/store.go:

• cursorHeap — min-heap by key. Each cursor advances one row at a time within its shard partition. The top of the heap is the next globally-ordered key.
• versionHeap — heap by (key, version_id DESC) for ListObjectVersions. Latest version of a key emerges first; null- versioned rows sort last by encoded sentinel.

Pagination cookies carry per-shard cursor positions so the next page resumes exactly where the previous left off. This is the N-way-fan-out path the gateway uses when the meta backend does NOT implement RangeScanStore.

Bounded fan-out + read consistency (US-012)#

The fan-out is bounded, not one-goroutine-per-shard. A worker pool (runBoundedFanOut, sized by STRATA_CASSANDRA_LIST_CONCURRENCY, default 16, clamped [1, 256]) caps how many shard partitions a single listing queries concurrently, so the gocql connection pool and goroutine count stay bounded no matter how high N (per-bucket shard count) climbs or how many listings run at once. The earlier unbounded fan-out spawned N goroutines + N connection checkouts per request and exploded under high-N × concurrent-lists.

Memory. Each shard cursor buffers at most one bounded gocql page (listPageSize = 256 rows) regardless of the request limit (up to 1000); the heap-merge auto-pages a cursor via advance() so a small page is purely a fetch-granularity choice, never a correctness/truncation one (truncation is decided by counting emitted rows). Resident listing memory is therefore N × listPageSize rows of buffered page plus the N heap heads — and the concurrently-fetching burst is capped at listConcurrency × listPageSize — not the old N × (limit+1).

Consistency. The listing queries pin LOCAL_QUORUM explicitly rather than inheriting the cluster/session default, so the read-after-write guarantee a client expects (a LOCAL_QUORUM write is visible to the very next LOCAL_QUORUM list) does not depend on an operator not having lowered a Cassandra default.

RangeScanStore short-circuit#

Backends with a globally-ordered keyspace skip the fan-out. meta.RangeScanStore is the optional capability — see Meta store. TiKV implements it because its byte-string key encoding (FoundationDB-style stuffing for variable segments, big-endian fixed-width integer fields, inverted-timestamp version suffix) gives a globally lex-ordered keyspace. ListObjects under TiKV is one continuous scan.

The dispatch decision is at internal/s3api/server.go::listObjects:

if rs, ok := s.Meta.(meta.RangeScanStore); ok {
    return rs.ScanObjects(ctx, bucketID, opts)
}
return s.Meta.ListObjects(ctx, bucketID, opts)

GC fan-out#

GC entries (chunks scheduled for deletion) are partitioned across 1024 logical shards in the meta store. The gc worker’s runtime shard count (STRATA_GC_SHARDS, default 1, range [1, 1024]) modulates how many of those logical shards a single replica owns:

entry belongs to runtime shard i  iff  entry.LogicalShardID % STRATA_GC_SHARDS == i

Per-shard gc-leader-<shardID> leases let one replica drain multiple shards in parallel and lose only one shard’s lease on a panic. Multi- replica deployments scale linearly: with 3 replicas and STRATA_GC_SHARDS=3, each replica owns ~1/3 of the GC queue.

The lifecycle worker reuses the same shard distribution: per-bucket lifecycle-leader-<bucketID> leases are gated by fnv32a(bucketID) % STRATA_GC_SHARDS == min(GCFanOut.HeldShards()) so lifecycle work distributes in lockstep with the gc fan-out.

Online reshard#

internal/reshard is a per-bucket online shard-resize worker (US-045). It drains the source shards, rewrites every key under the new modulo, and flips the bucket’s shardCount once the rewrite catches up. As of US-005 it runs asynchronously as a leader-elected background worker (STRATA_WORKERS=…,reshard, lease reshard-leader) — the admin endpoint only queues the job and returns immediately.

Operator runbook (async trigger + progress)#

1. Enable the worker on at least one replica: STRATA_WORKERS=gc,lifecycle,rebalance,reshard. Tunables: STRATA_RESHARD_INTERVAL (default 30s, range [1s,1h]) and STRATA_RESHARD_BATCH_LIMIT (default 500).
2. Trigger (queues the job, returns 202 immediately — never blocks on the migration): POST /admin/bucket/reshard?bucket=<name>&target=<power-of-two> → {"ok":true,"state":"queued","source":64,"target":128}. Stamps the admin:BucketReshard audit row. A second trigger while a job is in flight returns 409 OperationAborted.
3. Watch progress: GET /admin/bucket/reshard?bucket=<name> → state is queued (no rows moved yet), running (last_key watermark advancing), or idle (no job in flight — shard_count reports the live count, the signal the reshard converged).
4. CLI: strata admin bucket reshard --bucket <name> --target 128 queues and prints the job; add --wait to poll progress until the worker drains the job.

Crash-safe + resumable. The worker persists a LastKey watermark after each batch. A crash (or leader handover) mid-job resumes from that watermark on the next tick; MigrateReshardKey is idempotent, so re-walking the partial batch never double-moves or strands a key. CompleteReshard (the shard_count flip) fires only after the walk drains every key — cleanup-before-flip, so a post-flip listing never double-emits a moved key. The make smoke-reshard harness (scripts/smoke-reshard.sh) exercises the full path against the Cassandra lab: async trigger, concurrent PUT/GET/DELETE under load, and a docker restart mid-job crash-resume leg.

Reshard only does physical work on the Cassandra fan-out backend. The objects table is partitioned by (bucket_id, shard), so changing shardCount moves a key to a different partition — that is the rewrite the worker performs. Cassandra implements meta.ReshardMigrator; the worker drives MigrateReshardKey per key.

The power-of-two constraint matters here: doubling from N=64 to N=128 means every old shard either stays in place (keys whose fnv32a(key) % 128 < 64) or moves to its new sibling shard (+ 64). No three-way splits. The reshard worker exploits this — it reads each old shard once and either keeps the row in place or writes it to the sibling, never to two destinations.

TiKV and memory need no resharding. TiKV addresses objects through a single globally-ordered range scan and the memory backend through a flat map — a key’s physical placement does not depend on shardCount, so there is nothing to move. These backends do not implement meta.ReshardMigrator; StartReshard queues a job that the worker completes at once with zero rows moved — no ListObjectVersions walk, no watermark writes (US-004 immediate-complete no-op). A direct API/CLI caller gets success, not an error, and the full key set stays readable before, during, and after. The web console (US-006) disables the Reshard action on these backends — offering a button that does nothing is worse UX than hiding it; API = no-op success, UI = disabled, both consistent with “nothing to reshard”.

Per-bucket / per-shard observability#

bucketstats.Sampler (see Storage status) emits per-(bucket, shard) gauges (strata_bucket_shard_bytes, strata_bucket_shard_objects) so operators can spot hotshards before they take down a partition. Cardinality is capped at STRATA_BUCKETSTATS_TOPN (default 100) — the cluster-wide totals are unaffected by the cap.

Source#

• internal/meta/store.go — IsValidShardCount, RangeScanStore.
• internal/meta/cassandra/store.go — shardOf, cursorHeap, versionHeap, ListObjects fan-out + merge.
• internal/meta/tikv/keys.md — TiKV byte-level key encoding.
• internal/gc/fanout.go — FanOut, runtime shard ownership, panic metrics.
• internal/lifecycle/distribute.go — per-bucket distribution gate.
• internal/reshard/ — online reshard worker.