Placement + rebalance#

Once you run more than one RADOS / S3 cluster behind a single Strata deployment, you also need to control which cluster a bucket’s chunks land on and how to migrate old chunks when a new cluster joins or an old cluster is being retired. Strata ships both pieces:

• A per-bucket placement policy (meta.Bucket.Placement — a {cluster: weight} map) that the chunk PUT path consults via a stable hash-mod router.
• A leader-elected rebalance worker (strata server --workers=rebalance) that walks every bucket with a non-nil policy, compares the actual per-cluster chunk distribution to the policy’s target, and copies chunks A → B until the two match.

A bucket without a policy (Placement == nil) behaves exactly as before — chunks land on the storage class’s default cluster. No migration, no schema bump, no behavior change. The policy + worker are both opt-in.

For the conceptual S3 multi-cluster overview see S3 multi-cluster routing. This page is the operator runbook.

Operator workflow#

The end-to-end “add a new cluster and drain an old one” workflow is four steps:

1. Register the new cluster in STRATA_RADOS_CLUSTERS / STRATA_S3_CLUSTERS (env-driven — rolling-restart each replica).
2. Set placement on the buckets that should start using the new cluster:

   curl -X PUT http://strata/admin/v1/buckets/<name>/placement \
        -H 'content-type: application/json' \
        -d '{"placement":{"oldc":1,"newc":3}}'

   New PUTs split per the weights immediately. Existing chunks stay on oldc until the rebalance worker moves them.
3. Drain the cluster you want to retire (when you also want the leftover chunks moved off):

   curl -X POST http://strata/admin/v1/clusters/oldc/drain

   This marks oldc as draining in the cluster_state table. placement.PickClusterExcluding will skip oldc on the PUT hot path (drain is treated as weight=0 even when the policy still lists it). The rebalance worker treats draining as the source of moves but refuses to ever pick a draining cluster as a target. The admin handler invalidates the drain cache so the flip takes effect on the next PUT without waiting out the 30 s TTL.
4. Let the rebalance worker converge. It runs on the STRATA_REBALANCE_INTERVAL cadence (default 1h) and is leader- elected on the rebalance-leader lease — exactly one replica moves chunks at any given moment. Watch strata_rebalance_planned_moves_total go to zero and the _chunks_moved_total{from=oldc} counter plateau, then deregister the cluster by dropping it from STRATA_RADOS_CLUSTERS / STRATA_S3_CLUSTERS and rolling-restarting.

Undrain (if you change your mind mid-workflow):

curl -X POST http://strata/admin/v1/clusters/oldc/undrain

This deletes the cluster_state row (absence == live) and invalidates the drain cache.

Placement policy shape#

The policy is a {cluster: weight} JSON object:

{"placement": {"hot": 1, "warm": 3, "cold": 0}}

Validation runs in meta.ValidatePlacement:

A zero-weight entry is legal — it pins the cluster in the policy without letting new chunks land there. Useful for “decommissioning soon, drained but not yet deregistered”.

Strict vs Weighted placement (per-bucket mode)#

Each bucket carries a PlacementMode ∈ {weighted (default), strict} alongside its Placement policy. The mode controls what placement.EffectivePolicy returns when every cluster named in the bucket’s policy is draining — the resolution order is identical to the PUT hot path and the rebalance worker classifier (US-001..US-005 of ralph/effective-placement, closes the P2 ROADMAP entry):

mode == "" (legacy buckets) is coerced to weighted everywhere via meta.NormalizePlacementMode. Setting mode = strict on a nil/empty policy is a no-op — strict only has meaning when the bucket has an explicit Placement to pin.

When to use strict:

• Data sovereignty. Bucket data must NOT silently fall back to a cluster in a different region / jurisdiction. Strict refuses the PUT with 503 DrainRefused rather than land bytes on the cluster-weights default target.
• Replication design. A bucket whose Placement is part of a multi-cluster replication scheme — falling back to weights would break the replication invariant.
• Hot/cold separation. A bucket pinned to hot=1 should not silently route to cold clusters during a hot-cluster drain.

When to use weighted (default):

• Typical operator deploys. A drain should be a paperwork-free operation: pick a cluster, click Drain, let auto-fallback to cluster.weights cover the bucket policy gap, deregister.
• Buckets without compliance constraints. Most user-facing workloads — the cluster weights wheel was designed to be the routing fallback for exactly this case.

Migration note. Existing buckets get mode = weighted by default — no operator action required. Pre-cycle behaviour matched mode = strict (no fallback), which is why the cycle ships a console toggle + bulk-fix flow so operators can opt into strict explicitly on the buckets that need it. The smoke harness scripts/smoke-effective-placement.sh walks all four scenarios (weighted auto-fallback, strict blocks drain, flip strict→weighted resolves stuck, all-clusters-drained → 503) end-to-end against the running multi-cluster compose profile; run via make smoke-effective-placement.

Console surfaces (UI half of US-006):

• BucketDetail Placement tab gains a Strict placement switch — flip ON opens a confirmation dialog (“may block drain workflows if this bucket’s clusters become unavailable”); flip OFF is one-click. Header renders a small strict badge when the bucket has a non- empty Placement AND mode = strict.
• <BulkPlacementFixDialog> filters its row list to compliance-locked buckets (placement_mode = strict) only — weighted stuck buckets auto-resolve post-EffectivePolicy and never reach the dialog. Per- row default suggestion is “Flip to weighted (auto-fallback to cluster weights)” so the operator can resolve a strict-stuck row without manually editing the policy cluster list.
• <ConfirmDrainModal> amber stuck-row reads <N> compliance-locked buckets need fix; Submit blocks while the count is > 0 even when the typed-confirm matches.

Routing — stable hash-mod#

placement.PickCluster is the chunk PUT router:

1. Empty / nil policy → return "" so the caller falls back to the class’s $defaultCluster.
2. Sort cluster ids lex (sort.Strings(slices.Collect(maps.Keys(policy)))) so the walk is deterministic regardless of Go’s random map order.
3. Compute fnv32a("<bucketID>/<key>/<chunkIdx>") % sum(weights) and walk the weight wheel.

Determinism guarantee: the same (bucketID, key, chunkIdx) always maps to the same cluster across retries, gateway restarts, and policy edits that don’t change the weight wheel. Adding a fourth cluster to a {a:1, b:1, c:1} policy moves only ~1/4 of chunks, not all of them — the wheel grows but the spokes are stable.

The drain-aware variant is PickClusterExcluding:

PickClusterExcluding(bucketID, key, chunkIdx, policy, draining map[string]bool) string

Entries in draining are treated as weight=0; if every cluster in the policy is draining, the function returns "" and the caller falls back to $defaultCluster.

STRATA_REBALANCE_* env knobs#

All three are env-only, read at worker Build time — no flags. Restart the replica that owns the rebalance-leader lease to pick up new values (or rolling-restart, the lease re-acquires on the next replica).

Bandwidth tuning#

STRATA_REBALANCE_RATE_MB_S is the single knob that decides whether migration coexists with user traffic on the same network or starves it. Pick the value from the network share calculator below; verify it live on the Cluster Overview rebalance card after a rolling restart.

Token-bucket model#

Each replica’s rebalance worker has a token bucket sized at STRATA_REBALANCE_RATE_MB_S MB/s. A chunk move consumes chunkSize × 2 tokens (read from source + write to target — both legs debit the same bucket). With N replicas, the aggregate ceiling is N × rate_mb_s; the effective forward (net new data written on the target cluster) is approximately aggregate / 2, because half of the token spend is read-side traffic that never crosses to the target.

Implications:

• Per-replica rate is what each replica’s NIC actually carries on migration traffic — pick it relative to the per-host pipe.
• Aggregate is what the target cluster’s pool absorbs on writes; effective forward is what the dashboard “data moved” counter climbs by per second.
• The rebalance leader is a single replica per shard (see Multi-leader scaling) — only the leader-holding replicas debit their bucket. Idle replicas contribute nothing until they win a shard lease.

Network share calculator#

Pick STRATA_REBALANCE_RATE_MB_S to keep migration ≤ ~16 % of the per-host pipe (one-sixth headroom rule — leaves the remaining ~84 % for user traffic + protocol overhead + the inevitable bursts). Table below assumes a 2-replica deploy (bare-default lab shape); scale per-replica values down proportionally for more replicas if the target cluster’s write bandwidth is the bottleneck.

The default 100 MB/s is calibrated for a 10 GbE host. Operators on 1 GbE hosts who leave the default will see migration saturate the NIC — clients on the same hosts will see latency spikes. The first operator-facing symptom is usually a P99 GET-latency cliff that disappears when the rebalance worker stops; if you see that, drop RATE_MB_S to the table value for your link and rolling-restart.

Tuning workflows#

Low-bandwidth lab (1 GbE virtual NIC). The bare compose lab runs on the host’s loopback, so the 10 MB/s setting is a test-rig preset, not a production recommendation:

STRATA_REBALANCE_RATE_MB_S=10 make up-all

The scripts/smoke-drain-progress-ui.sh smoke harness goes even lower (RATE_MB_S=1) to deliberately widen the Migrating phase past the 3 s poll cadence — that value is smoke-only and would stall real migration; never run prod gateways at it.

Production 10 GbE rolling restart. Set the env on every replica in the Deployment / DaemonSet manifest and roll. The leader re-acquires on the next replica; the new rate takes effect on the next rebalance tick:

# k8s Deployment fragment — apply to every strata replica
env:
  - name: STRATA_REBALANCE_RATE_MB_S
    value: "100"

Live observability + scripted health-check#

Two surfaces verify the value once the rolling restart completes:

• Cluster Overview rebalance card (operator console) — renders the per-replica rate, aggregate (× replicas), effective forward, cadence, inflight, and shard count plus a live 1 m observed-MB/s row driven by Prometheus. See the Drain progress states section for the matching <DrainProgressBar> per-cluster bandwidth indicator on the Migrating chip.

• Scripted health-check — curl the read-only admin endpoint from any pod or operator workstation that can reach the gateway:

  curl -s http://gateway:9999/admin/v1/rebalance-config | jq .
  # → {"interval_seconds":300,"rate_mb_s":100,"inflight":4,"shards":1,"replicas_count":2}

  The endpoint is audit-stamped admin:GetRebalanceConfig and the matching companion GET /admin/v1/gc-config exposes the GC tunables read by the Awaiting GC chip ETA formula. Both are env-static — restart picks up new values; no PUT counterpart.

Safety rails#

The rebalance worker won’t dispatch a move when:

• Target is draining. The PickClusterExcluding policy filter rejects this at plan time; the post-filter in Worker.applySafetyRails is defense-in-depth for the race between scan emission and a drain flip. Bumps strata_rebalance_refused_total{reason="target_draining",target}.
• Target is > 90 % full (RADOS only). The worker type-asserts data.ClusterStatsProbe against the data backend (RADOS implements; S3 + memory don’t). Per-tick fill probe is cached so a fan-out of N moves into K targets costs K probes per iteration, not N. S3-only deployments treat data.ErrClusterStatsNotSupported as “OK to proceed” with one WARN per iteration. Bumps strata_rebalance_refused_total{reason="target_full",target}.

Both rails are post-filter — the plan is built first, refused moves get logged + metricked + skipped, and the rest of the plan still executes.

Movers#

The worker dispatches the plan through a MoverChain that partitions by target-cluster ownership:

After every move the mover issues a per-object manifest CAS via meta.Store.SetObjectStorage(... expectedClass=currentClass). A pre- CAS sanity check inside buildUpdatedManifest (RADOS) / buildUpdatedBackendManifest (S3) verifies the live chunk locator still matches the planned SrcRef — a concurrent client write that rewrote the chunk between scan and Move is caught BEFORE the compare-and-set so the rebalance doesn’t clobber a newer locator. If the live row diverges, the new target chunks go to the GC queue and strata_rebalance_cas_conflicts_total{bucket} bumps.

On CAS success the OLD chunks are enqueued via meta.Store.EnqueueChunkDeletion and the existing gc worker collects them per STRATA_GC_GRACE.

Metric family#

from / to carry the cluster id from STRATA_RADOS_CLUSTERS / STRATA_S3_CLUSTERS.

Trace shapes#

Iteration parent: worker.rebalance.tick with strata.component=worker + strata.worker=rebalance + strata.iteration_id=<atomic.uint64>. Sub-ops:

• rebalance.scan_bucket — one per bucket scanned per tick. Attrs: strata.rebalance.bucket, bucket_id, planned-moves count.
• rebalance.move_chunk — one per chunk move. Attrs: strata.rebalance.{bucket,key,from,to,chunk_idx}. Spans get RecordError + SetStatus(Error) on failure; the iteration parent’s sticky-err accumulator flips it to Error so the tail- sampler exports the full iteration regardless of STRATA_OTEL_SAMPLE_RATIO.

Filter recipe — “what did the last rebalance tick move across all clusters”:

strata.component=worker
strata.worker=rebalance

Troubleshooting#

_planned_moves_total stays high, _chunks_moved_total stays flat#

The plan keeps being built but no moves complete. Likely causes:

• Safety rails refusing. Check strata_rebalance_refused_total{reason} — if target_full dominates, the target cluster crossed 90 % fill; either grow the cluster or re-route to a different cluster id via the policy. If target_draining dominates, an operator drained the cluster you were trying to fill — un-drain or change the policy.
• Token bucket starved. Raise STRATA_REBALANCE_RATE_MB_S. Both read + write debit the same bucket so the wall-clock throughput is RATE_MB_S MiB/s on the busier leg.
• CAS conflict storm. Watch strata_rebalance_cas_conflicts_total{bucket}. Steady conflicts on one bucket means concurrent client traffic keeps winning the compare-and-set — this is correct behavior (client always wins) and the chunks will re-plan next tick. If conflicts grow unboundedly, you have hot keys being rewritten faster than the rebalance loop can converge; pause the rebalance via STRATA_REBALANCE_INTERVAL=24h until traffic settles.

Target-full refusals#

The 90 % fill ceiling is a hard-coded constant on the rebalance worker. Operators who want a different threshold should grow the cluster (easier) or open a tracking issue. S3-side has no fill probe — the worker proceeds. If your S3 backend has a quota, monitor it externally.

Drain cache TTL surprises#

The drain sentinel is cached in-process for 30 s (placement.DefaultDrainCacheTTL). The drain / undrain admin handlers Invalidate() the cache so the flip takes effect on the next PUT — operators never wait the TTL. Multi-replica deployments need to invalidate on every replica; the admin handler runs locally so an external load balancer must hit each replica’s drain endpoint, or you can rely on the 30 s TTL for the replicas you didn’t hit. For zero-downtime drains the safer path is a rolling drain — drain via one admin endpoint and wait 30 s before expecting cluster-wide quiescence.

Rebalance worker never picks up the lease#

The lease is rebalance-leader on whatever meta.Locker you configured (Cassandra-backed lease or in-process memory locker for dev). Check strata server logs for the leader_for=rebalance heartbeat chip; if absent, the worker isn’t running. Common cause: STRATA_WORKERS=... doesn’t include rebalance. Set STRATA_WORKERS=rebalance (or include it in your comma-separated list) on the replica you expect to lead.

What’s NOT supported#

• Per-version rebalance. The scan walks only current versions via meta.Store.ListObjects — non-current versions keep their original cluster. The version-DESC clustering means walking every version would dominate the scan budget for almost no operator value. If you need to rebalance a tombstoned version, restore it first.
• Cross-backend moves. RADOS chunks can’t be moved into an S3 cluster and vice-versa — the mover chain partitions by target- cluster owner, and a RADOS source / S3 target pair has no owner. Use a class re-route + lifecycle transition instead.
• Per-cluster pool overrides on RADOS. Placement routes the cluster id; the pool + namespace come from the class spec. If you need a per-cluster pool override, register a second class on the target cluster (same pattern as the S3 backend’s bucketOnCluster).

Cluster lifecycle (register → activate → ramp)#

ralph/cluster-weights (US-001..US-005) folded a 5th state — pending — into the cluster machine and added a per-cluster weight int (0..100) field. Together they enable a safe gradual-activation flow for new clusters: register the new cluster via env, validate it without taking client traffic, ramp the routing share 10% → 25% → 50% → 100% under observation.

Two weight layers — never combined#

Strata has two weight knobs that look alike. They serve different scopes and are never multiplied together:

The picker order inside placement.PickCluster is:

1. bucket.Placement != nil → use bucket policy (short-circuit before weights).
2. Class env spec carries @cluster pin (e.g. STANDARD=hot@cephb) → use that cluster.
3. Synthesise default policy from live cluster weights (skipping pending / draining_readonly / evacuating / removed). All-zero-live → policy is empty → caller falls back to class spec.Cluster.

Walkthrough table — adding cephc to a live deploy#

Choosing the initial weight#

Negative paths#

• Activate on a live cluster → 409 InvalidTransition. Use PUT /weight for in-place adjustments.
• PUT /weight on a pending cluster → 409 InvalidTransition. Must POST /activate first.
• All live clusters have weight=0 → synthesised default policy is empty → PickCluster returns "" → caller falls back to class spec.Cluster (or 503 <Code>DrainRefused</Code> if the fallback cluster is draining).
• Boot reconcile sees an existing-live cluster (chunks via the bucket usage stats) → auto-creates state=live weight=100, not pending. Zero operator action required during upgrade from older strata versions.

cluster_state schema reminder#

cluster_state rows carry state + mode + weight (added in ralph/cluster-weights US-001). Absence still means “live” — the boot reconcile materialises the row when needed. Memory + Cassandra + TiKV backends carry the same three fields; the TiKV value is a 3-segment byte string state\x00mode\x00<decimal-weight> so older 1- and 2-segment values decode as weight=0 on read (forward-compat with mid-upgrade clusters).

Drain lifecycle#

Drain is the operator hook that takes a cluster out of the write hot path — either temporarily for maintenance, or permanently to deregister. The 4-state machine + mode picker shipped in ralph/drain-transparency (US-001..US-008) separates the two intents; the old single-mode drain from ralph/drain-lifecycle is now a special case (mode=evacuate).

Stop-writes vs. evacuate — pick the right mode#

POST /admin/v1/clusters/<id>/drain requires a body {"mode":"…"}. There is no default — the operator must pick:

Upgrade path: draining_readonly → evacuating via a second POST /drain {"mode":"evacuate"} — the modal renders the readonly radio hidden and the title flips to “Upgrade to evacuate”. There is no downgrade (evacuate → readonly); use undrain → re-drain readonly if you really need it.

Drain is unconditionally strict in both modes (US-007). RADOS + S3 PutChunks refuse to fall back to a draining_readonly or evacuating cluster — data.ErrDrainRefused → HTTP 503 <Code>DrainRefused</Code> with Retry-After: 300.

Pre-drain operator checklist#

Walk this list before submitting the ConfirmDrainModal. Each item has a console surface to drive it.

1. Open the Pools matrix. /console/storage → Data tab → Pools table. The matrix renders #clusters × #distinct-pools rows. A 0 B row means the class is routed elsewhere and drain is a no-op.
2. Click “Show affected buckets” on the cluster card. The <BucketReferencesDrawer> now consumes /drain-impact (US-001 drain-cleanup) and renders the same three categorized sections the modal shows — Migrating (green), Stuck — single-policy (amber), Stuck — no policy (amber) — so nil-policy buckets routed via class-env or default-routing are visible. Each per-bucket row carries chunk_count + bytes_used + the top 2 suggested_policies labels. If stuck>0 an inline Bulk fix N stuck buckets CTA above the stuck sections opens the same <BulkPlacementFixDialog> the modal uses; the drawer refetches on close so post-Apply counts reflect the bulk-fix immediately (US-002 cache invalidation — see Cache invalidation contract below).
3. Run the impact analysis. Inside <ConfirmDrainModal> pick the Full evacuate (decommission) radio — the modal fetches GET /admin/v1/clusters/<id>/drain-impact and renders three categorized counters:
   • Migratable — bucket has a Placement policy AND PickClusterExcluding finds a peer. Will move on the next rebalance tick.
   • Stuck (single policy) — bucket has a Placement policy but every cluster named in it is draining. Will NOT migrate; new PUTs get 503 DrainRefused.
   • Stuck (no policy) — bucket has no Placement (relies on class-env routing) and has chunks routed to the draining cluster. Same outcome: won’t migrate.
4. Fix the stuck buckets. If stuck>0 the modal renders <BulkPlacementFixDialog> behind a “Fix N buckets” CTA. Multi- select rows, pick a suggested policy per bucket (or one uniform policy across all selected via the dialog-level toggle), Apply → the dialog issues PUT /admin/v1/buckets/<name>/placement per bucket and refetches /drain-impact on close.
5. Submit. When stuck=0 the typed-confirm input arms the destructive submit; click Drain. Stuck>0 keeps the submit disabled with the explainer “Drain blocked — fix N stuck buckets”.

For readonly drain the modal skips the impact analysis (no migration will happen, so the categorization is moot). Submit is armed by the typed-confirm input alone.

Drain procedure#

The console flow per cluster, mode-specific:

Stop-writes (readonly):

1. Click Drain → ConfirmDrainModal opens, readonly radio selected.
2. Type the cluster id to arm submit; click Drain (stop-writes).
3. Cluster card flips to draining_readonly state. <DrainProgressBar> renders a single orange stop-writes chip plus an “Upgrade to evacuate” button and a small “Undrain” button.
4. Run your maintenance; click Undrain when done. State flips back to live with no migration cost.

Full evacuate:

1. Click Drain → ConfirmDrainModal opens; flip to the evacuate radio.
2. Run the impact analysis + fix stuck buckets via the BulkPlacementFixDialog (above).
3. Type the cluster id to arm submit; click Drain (X chunks will migrate).
4. Cluster card flips to evacuating state. <DrainProgressBar> renders a red “Evacuating” label + progress bar + three categorized counters. ETA is derived from rate(strata_rebalance_chunks_moved_total{from=<id>}[5m]) and appears only when migratable>0.
5. When chunks_on_cluster transitions >0 → 0 the bar evaluates the deregister-ready preconditions (US-006 drain-cleanup) — deregister_ready flips true ONLY when ALL THREE clear:
   1. total_chunks == 0 (manifest scan reports no chunks tagged to the cluster),
   2. gc_queue_pending_for_cluster == 0 (no rows in gc_entries_v2 whose chunk.Cluster matches the drained id),
   3. no_open_multipart_on_cluster == true (no in-flight multipart uploads whose BackendUploadID carries the drained cluster id as the leading segment). Any unmet condition surfaces in the response’s not_ready_reasons array using the fixed token vocabulary chunks_remaining → gc_queue_pending → open_multipart. The <DrainProgressBar> reads the array and renders an amber Not ready — <reasons> chip instead of the emerald ✓ Ready to deregister chip until every reason clears. The gating prevents the “manifest=0 but GC still has rows / open multipart still on cluster” leak hazard that a naive total_chunks==0 check missed pre-cycle. Once the chip is emerald the rebalance worker logs INFO drain complete, writes a drain.complete audit row, bumps strata_drain_complete_total, and best-effort fans an s3:Drain:Complete event through every sink in STRATA_NOTIFY_TARGETS.
6. Env edit + rolling restart. Drop the cluster id from STRATA_RADOS_CLUSTERS / STRATA_S3_CLUSTERS and rolling-restart the gateway replicas. There is no admin endpoint that performs the deregister — the env shape is the source of truth.

State-aware action buttons (US-007 drain-cleanup)#

The cluster card’s bottom-right action slot renders the right button for the current (state, chunks_on_cluster, deregister_ready, not_ready_reasons) combination — no more accidental Undrain after a full evacuation. The truth table:

The dereg-ready cell button was renamed from Cancel deregister prep → Restore to live (cancel evacuation) in the ralph/drain-followup cycle (US-003) — the prior label contradicted the green “Ready to deregister” chip rendered immediately above it (chip said “act now via env edit”, button said “cancel the preparation”). The new label clarifies that the button is an escape hatch, not the primary action. The button switched from the destructive to the outline variant so the chip dominates as the status indicator; the typed-confirm modal body now reads “Moved chunks remain on target clusters; no rollback. Cluster will accept writes again.” so the operator knows undrain is not a migration reversal. The chip itself gained a title= tooltip with the deregister recipe (“Edit STRATA_RADOS_CLUSTERS env to remove this cluster, then rolling restart. See operator runbook for deregister procedure.”) so the operator who finds the chip on a card has the primary action one hover away.

Cancel deregister prep is the safe-default for the dereg-ready state — the operator types the cluster id to arm the modal (mirroring ConfirmDrainModal), submit issues a plain POST /undrain, and the toast reads “Cluster restored to live. No chunks restored — migrated chunks stay on their target clusters.” Undrain is intentionally hidden in this state so an operator who hovered the cluster after the manifest scan zeroed cannot click through to revert hours of migration.

Drain progress states — physical vs manifest (US-001..US-003 drain-progress-physical)#

<DrainProgressBar> resolves three explicit operator-facing states during evacuating. Pre-cycle the bar surfaced only the manifest chunk count (chunks_on_cluster) as primary — which drops to 0 the instant the rebalance worker CASes the manifest, even though the physical RADOS chunks linger until STRATA_GC_GRACE elapses + the next gc tick deletes them. Operators read “0 Migrating” and assumed drain stalled.

The post-cycle response shape adds three additive fields on GET /admin/v1/clusters/<id>/drain-progress:

Existing fields (chunks_on_cluster, bytes_on_cluster, not_ready_reasons, deregister_ready) are preserved verbatim.

The <DrainProgressBar> state machine reads primary = physical_chunks_on_cluster ?? chunks_on_cluster and renders one of three chips:

A collapsible detail row (<details data-testid=dp-detail>) sits below the headline and unfolds to Manifest chunks: X, GC queue: Y, Physical bytes: Z B. Hidden by default.

Back-compat on backends without ClusterObjectCountProbe (memory, S3 pass-through): physical_chunks_on_cluster and physical_bytes_on_cluster are JSON-null. The bar falls back to chunks_on_cluster as primary, prints a small italic (physical count unavailable on this backend) tooltip next to the headline, and the detail row’s Physical bytes reads unavailable. The 3-state machine collapses to the pre-cycle behaviour on these backends — Migrating until manifest hits 0, then the existing not_ready_reasons / deregister_ready flow.

ETA precision is intentionally deferred. The Awaiting GC chip ships with a static-copy tooltip describing the wait reason (grace

• next tick), not a per-deploy ETA. Precise ETA requires exposing the gateway’s GC tunables (STRATA_GC_GRACE, STRATA_GC_INTERVAL, STRATA_GC_BATCH_SIZE, STRATA_GC_CONCURRENCY, STRATA_GC_SHARDS) via a new /admin/v1/gc-config endpoint plus an ETA formula eta_min = grace_min + ceil(gc_queue / (batch_size × shards) × interval_min). Parked as a P3 follow-up (see ROADMAP “Precise drain-progress ETA from gateway GC tunables”).

Per-poll cost is bounded by the 10 s ClusterStatsCache TTL in the gateway process. Every drain-progress request goes through Get → cache hit → no RADOS call. A miss issues one GetPoolStats per (cluster, pool, ns) tuple plus one ClusterStats call, caches the merged result, returns. Per-replica cache (no shared cache between gateways) — UI polling at 5 s amortises to ~1 RADOS probe per cluster per 10 s regardless of operator-count. Probe errors increment strata_drain_progress_probe_errors_total{cluster, probe} with probe ∈ {stats, object_count} and surface as null fields on the response — the UI degrades to the back-compat fallback.

The smoke harness scripts/smoke-drain-progress-ui.sh (make smoke-drain-progress-ui) drives the 3-state machine end-to-end against a real RADOS lab. It recreates the strata container with smoke-only env overrides (STRATA_REBALANCE_RATE_MB_S=1 to widen the Migrating phase beyond the 3 s poll cadence; STRATA_GC_GRACE=60s to shorten the GC wait into the 5-min script budget), plants 300 ~1 MB objects on a {default:1, cephb:1} bucket, drains cephb evacuate, and asserts each state observed at least once. Both env values are smoke-only; prod defaults are 100 MB/s and 5m — operating clusters at 1 MB/s + 60s would stall migration and risk premature deletion of in-flight references.

Cache invalidation contract (US-002 drain-cleanup)#

drainImpactCache holds the categorized scan for 5 minutes by default. Bucket placement mutations invalidate the cache synchronously before returning 200, so the bulk-fix workflow completes end-to-end within one HTTP round-trip. Triggers:

Invalidate-all is intentional: placement keys may add or remove clusters, so tracking the affected cluster set adds complexity for a minor speedup. The next /drain-impact call rebuilds the categorized scan from scratch and reflects the new policy immediately.

Multipart-blocks-deregister probe (US-004 + US-005 drain-followup)#

The no_open_multipart_on_cluster precondition (gate #3 of the deregister_ready AND) reads meta.Store.ListMultipartUploadsByCluster. Each backend wires it differently:

Init handlers extract the leading cluster id from the BackendUploadID shape cluster\x00bucket\x00key\x00uploadID and persist it; chunk-based RADOS uploads with empty BackendUploadID persist NULL and never match the probe (consistent with the chunk-based router having no init-time cluster binding).

Probe-only fix — runtime routing is unchanged. Open multipart sessions on an evacuating cluster keep finishing gracefully (the handle is the source of truth for UploadPart / Complete / Abort routing); the new column is read by the deregister-readiness gate so the green chip does not flip while the cluster still holds in-flight uploads.

Denormalized lookup tables (US-005 drain-followup)#

GC + multipart per-cluster probes used to issue WHERE cluster=? ALLOW FILTERING against gc_entries_v2 + multipart_uploads — antipattern at scale per the Cassandra gotchas in CLAUDE.md. The ralph/drain-followup cycle (US-005) added two denormalized lookup tables partitioned on (cluster):

Dual-write maintenance: every EnqueueChunkDeletion / AckGCEntry / CreateMultipartUpload / CompleteMultipartUpload / AbortMultipartUpload keeps the primary table and the lookup row in lockstep. Chunk-based RADOS uploads (empty BackendUploadID) + legacy GC entries (empty chunk.Cluster) skip the lookup write so the intermediate state is tolerated.

Upgrade note for operators. Existing deploys carry pre-cycle rows in gc_entries_v2 + multipart_uploads that have no corresponding lookup row. The gateway runs a one-shot Store.ReconcileLookupTables(ctx, logger) at boot (idempotent on re-runs — skips already-present lookup rows; logs per-1000-row progress + a final cluster reconcile: gc_entries=<N>, multipart_uploads=<M>, written_missing=<K> summary). Large tables (>1M rows) may take minutes; the gateway accepts traffic during the reconcile (probe correctness converges as rows backfill). No operator action required — boot once and the lookup tables are populated.

Abort / recovery#

POST /admin/v1/clusters/<id>/undrain works from both draining_readonly and evacuating. It deletes the cluster_state row (absence == live) and invalidates the in-process drain cache. Migrated chunks stay on their new target; only future moves halt. Undrain is idempotent and refused (409 InvalidTransition) from live or removed.

Drain endpoints#

drain-progress numeric fields are nullable: when state=live every numeric field is null. When state=evacuating but no scan has committed yet, the response carries warnings: ["progress scan pending; rebalance worker has not yet committed a tick"] and the UI renders “scan pending” instead of zero counts.

Drain refusal semantics (always strict)#

Drain is unconditionally strict in both modes (US-007 drain-transparency — the former opt-in STRATA_DRAIN_STRICT env was retired). RADOS + S3 PutChunks always refuse to fall back to a draining cluster — they return data.ErrDrainRefused, the gateway maps it to 503 ServiceUnavailable with <Code>DrainRefused</Code> body and Retry-After: 300 header. PUT chunks only — GET / HEAD / DELETE / multipart UploadPart / Complete / Abort / List against draining clusters keep working (drain semantic is stop-write, not stop-read). In-flight multipart sessions persist their initial cluster id in the upload handle (cluster\x00bucket\x00key\x00uploadID) and never re-consult the picker, so UploadPart / Complete / Abort on an already-open multipart finish gracefully on the drained cluster.

Breaking change for Prometheus dashboards. Counter label flipped from reason="drain_strict" to reason="drain_refused" — strata_putchunks_refused_total{reason="drain_refused",cluster} increments per refusal. Wire to alerting if you expect zero refusals in steady state. Legacy STRATA_DRAIN_STRICT env in the environment is ignored at boot with a single WARN log line (remove from your deploy descriptors).

Three-scenario walkthrough#

The smoke harness scripts/smoke-drain-transparency.sh drives every step below against the running compose stack (multi-cluster is the default shape after the compose-collapse cycle); exit-non-zero on any assertion miss. Run via make smoke-drain-transparency once the lab is up (docker compose up -d). The legacy scripts/smoke-drain-lifecycle.sh still validates the basic flip-to- draining contract; the new harness covers mode-picker + impact analysis + multipart graceful contract end-to-end. A third harness scripts/smoke-drain-cleanup.sh (make smoke-drain-cleanup, shipped with US-005 drain-cleanup) walks the 13-step bundle that closes the seven ROADMAP follow-ups: drawer 3-category render, /drain-impact cache invalidation on placement PUT, Pools chunk_count rename, admin force-empty GC enqueue, deregister_ready 3-condition hard-safety, state-aware action buttons, and the Trace Browser recent-traces panel.

Scenario A — Stop-writes drain (maintenance):

Scenario B — Full evacuate (decommission):

Scenario C — Upgrade readonly → evacuate:

Negative paths covered:

• Modal blocks submit when stuck>0 — clicking Drain on the evacuate radio with stuck>0 keeps the submit disabled even when the typed-confirm matches; the explainer text reads “Drain blocked — fix N stuck buckets”. Validated in web/e2e/drain-transparency.spec.ts Scenario B.
• In-flight multipart finishes gracefully — Init+UploadPart+ Complete on a draining cluster all return 200; bytes land on the drained cluster’s pool and are immediately readable. Validated in scripts/smoke-drain-transparency.sh Scenario A step 6.
• Undrain from live or removed → 409 InvalidTransition — the 4- state machine refuses no-op transitions. Validated by the admin API clusters-drain integration test.

Web UI#

The operator console ships UI surfaces for every endpoint described above so day-to-day placement / drain operations no longer need curl. All four surfaces share the clusters TanStack Query key (10 s poll from <ClustersSubsection>, 15 s from <PlacementDrainBanner>) so the gateway is hit once per cache window regardless of how many components read the topology.

Bundle delta of the four surfaces combined: ≤ ~10 KiB gzipped on the Storage / BucketDetail chunks. No new chart libraries — the sparkline reuses the same Recharts wrapper as the Metrics page + Cluster Overview.

E2E coverage lives in web/e2e/placement.spec.ts; see Web UI — End-to-end tests.

Multi-leader scaling#

ralph/rebalance-scale-phase-2 (US-001..US-005) closes the P2 ROADMAP entry “Rebalance worker not sharded — single goroutine bottleneck on large deploys” by mirroring the gc + lifecycle Phase 2 pattern: the rebalance worker is now a sharded fan-out rather than a single goroutine per process.

The bare-default compose lab is now the TiKV-default 2-replica shape (strata-a / strata-b behind nginx LB on :9999) — bare docker compose up -d brings it up directly, no profile flag needed. Use it to exercise multi-leader worker-lease distribution on the TiKV metadata backend. The retired 3-replica labs (lab-tikv-3, lab-cassandra-3) are tracked as a parked P3 follow-up in ROADMAP.md; see the migration note for the full reasoning. The Cassandra-backed equivalent is available via make up-cassandra for a single Cassandra-backed replica (strata-cassandra :9998) side-by-side with the bare default.

STRATA_REBALANCE_SHARDS env#

The supervisor registers the worker with SkipLease: true; the worker owns its own leader election via the rebalance.ShardedFanOut struct, which mirrors the GC worker’s fan-out shape exactly. Each shard goroutine acquires rebalance-leader-<i> independently; the leader chip (leader_for=rebalance in /admin/v1/cluster/nodes) is folded — a replica holding N shards still flips the chip ON exactly once (when it acquires its first shard) and OFF exactly once (when it releases its last). Operators tracking lease distribution should use /admin/v1/cluster/nodes for chip presence + the meta backend’s worker-lock table (worker_locks on Cassandra, s/wl/ keys on TiKV) for individual shard ownership.

Ownership distribution#

Lease distribution across replicas is opportunistic: each replica races to acquire rebalance-leader-0..N-1, the meta-backend compare-and-set (Cassandra) or pessimistic txn (TiKV) decides the winner per shard. On a steady-state lab with R replicas and N shards:

Set SHARDS == replica count for the cleanest 1-shard-per-replica distribution; the chip-folding contract means going larger (SHARDS = 2×replicas or more) is correct but each replica then holds multiple shards and the bench multiplier flattens.

Benchmark + speedup expectations#

The bench harness lives under scripts/bench-rebalance-multi.sh and is wrapped by make bench-rebalance-multi. It seeds N buckets × M chunks each with a {default:1, cephb:1} placement policy, drains default evacuate, and times wall-clock-until-chunks_on_cluster=0 for SHARDS=1 baseline and SHARDS=2 multi-leader on the bare-default 2-replica lab. The historical SHARDS=3 scenario is SKIP’d with an explicit stdout message — the 3-replica lab is retired (see ROADMAP P3 entry “Restore 3-replica TiKV bench (SHARDS=3 rebalance-multi)”).

Expected at SHARDS=2 against the bare-default 2-replica lab: ratio ≤ 60 % of baseline (≥ 1.6× speedup). The bench prints one of three verdicts:

• SPEEDUP_OK — ratio ≤ 40 % of baseline. Multi-leader works.
• SPEEDUP_PARTIAL — ratio 40-70 % of baseline. Investigate per-target token-bucket starvation or shared RADOS write bandwidth.
• SPEEDUP_FAILED — ratio > 70 % of baseline. Regression — the fan-out is not delivering. Script EXITs 1.

The speedup is bounded below 3× because the STRATA_REBALANCE_RATE_MB_S token bucket sits per-replica and both read + write debit it; three replicas concurrently writing the same target RADOS pool also stack on shared OSD bandwidth. See the rebalance bench page for the full cap-shape discussion + comparison with gc / lifecycle Phase 2. Per-replica rate sizing for the host NIC is covered in Bandwidth tuning — the network share calculator there maps GbE link size → safe RATE_MB_S value so the fan-out doesn’t saturate the user-traffic pipe when several replicas hold shards concurrently.

Single-replica fallback#

SHARDS=1 on a single replica reproduces Phase 1 behaviour byte-for-byte: one rebalance-leader-0 lease, one goroutine scanning every bucket, one ProgressTracker contribution per cluster. Operators who do not want the fan-out simply leave STRATA_REBALANCE_SHARDS unset (default 1). The new env is opt-in for scaling, not a breaking change.

Smoke#

The four-scenario smoke harness scripts/smoke-rebalance-scale.sh (make smoke-rebalance-scale) drives:

• A — single-replica fan-out at SHARDS=3, asserts the folded chip emits exactly once and drain converges.
• B — SHARDS=3 multi-leader against a 3-replica lab (SKIP’d in the current bare-default 2-replica shape; tracked as P3 ROADMAP follow-up “Restore 3-replica TiKV bench”).
• C — back-compat SHARDS=1 on the same lab, asserts EXACTLY one replica holds the chip and the iteration log matches the legacy single-leader shape.
• D — replica failover (kill one of three at SHARDS=3), asserts surviving replicas reacquire the freed shard within STRATA_GC_LEASE_TTL.

Scenarios B + D auto-skip on single-replica labs. Skip-77 when $BASE/readyz is unreachable; REQUIRE_LAB=1 converts the skip into a hard fail for CI gating.

See also#

• Rebalance bench — bench harness, expected speedup, cap shape, comparison with gc / lifecycle Phase 2.
• S3 multi-cluster routing — env shape for STRATA_S3_CLUSTERS / STRATA_S3_CLASSES, credentials envelope, rolling-restart workflow.
• Workers — registration shape, supervisor lifecycle, leader-election semantics.
• Observability — full metric family + span shapes shipped by Strata.