Architecture on Strata Documentation

Auth

Mon, 01 Jan 0001 00:00:00 +0000

Auth#

The auth layer lives under internal/auth/. Every request enters through auth.Middleware, which verifies an AWS SigV4 signature, derives a stable identity, and stamps the result onto the request context. The router and handlers downstream never re-derive identity — they read it from auth.FromContext(ctx).Owner.

SigV4 verification#

internal/auth/sigv4.go implements the standard four-step canonicalisation:

Parse Authorization (or X-Amz-* query parameters for presigned URLs) to extract AccessKey, Scope, SignedHeaders, and Signature.
Look up the secret for the access key via the configured static credentials store (internal/auth/static.go) — no IdP federation in this cycle.
Build the canonical request string from Method, URL.Path, the sorted SignedHeaders, and the body hash (x-amz-content-sha256, which may be the literal sentinel STREAMING-AWS4-HMAC-SHA256-PAYLOAD for chunked uploads — see below).
Recompute the signature with the derived signing key and constant-time compare. Mismatch returns ErrSignatureInvalid (HTTP 403, AWS code SignatureDoesNotMatch).

The middleware MUST run before any URL rewriting. The signed canonical string includes the original Host header and the original URL.Path — if the router rewrites either before verification, the signature breaks. See the Router page for the order.

Router

Mon, 01 Jan 0001 00:00:00 +0000

Router#

The router lives in internal/s3api/server.go. It is a single Server.ServeHTTP method that classifies every request along three axes:

Special prefix — /admin/... is the embedded operator console JSON API and bypasses the S3 dispatch entirely.
Bucket vs object scope — the URL path is split at the first slash into (bucket, key). Empty bucket → service-level (ListBuckets, IAM actions). Empty key → bucket-scoped. Both populated → object-scoped.
Query-string sub-resource — within bucket and object scope, the sub-operation is dispatched by query parameter (?cors, ?policy, ?lifecycle, ?uploads, ?uploadId=, ?tagging, …) plus the HTTP method. This is the AWS S3 wire shape; we mirror it.

Dispatch order#

ServeHTTP(w, r):
 1. extractAccessPointAlias(r.Host) # alias.<host> -> rewrite to /<bucket>/...
 2. extractVHostBucket(r.Host, ...) # *.s3.local -> rewrite to /<bucket>/...
 3. if path starts with /admin/ -> handleAdmin
 4. splitPath(r.URL.Path) -> (bucket, key)
 5. bucket == "" -> IAM action / ListBuckets / DLQ-audit listings
 key == "" -> handleBucket(...)
 default -> handleObject(...)

Auth middleware runs before the access-point and vhost rewrites because those rewrites only mutate r.URL.Path after SigV4 has already validated the original. See Auth for the ordering rationale.

PUT flow

Mon, 01 Jan 0001 00:00:00 +0000

PUT flow#

A single S3 PutObject traverses the gateway, the metadata store, and the data backend. The picture below names the components on the hot path so the prose that follows can stay focused on the interesting choices — streaming chunk decode, the manifest compare-and-set, and the failure modes when one tier wins a race against another.

Sequence diagram#

sequenceDiagram
 autonumber
 participant C as S3 client
 participant GW as s3api.Server
 participant Auth as auth.Middleware
 participant Meta as meta.Store
 participant Data as data.Backend
 C->>GW: PUT /bucket/key (+ SigV4 headers, streaming body)
 GW->>Auth: verify signature (canonical request, chain HMAC if streaming)
 Auth-->>GW: identity{Owner, AccessKeyID}
 GW->>Meta: LookupBucket(bucket) → policy + placement
 GW->>Data: PutChunks(stream, placement, classSpec)
 Data->>Data: split body into 4 MiB chunks, write to selected cluster
 Data-->>GW: manifest{chunks[], etag, size, BackendRef}
 GW->>Meta: SetObjectManifest(bucket, key, prevVersion, manifest) — compare-and-set
 alt CAS applied
 Meta-->>GW: applied=true, version=v
 GW-->>C: 200 OK, ETag, x-amz-version-id
 else CAS rejected (concurrent writer landed first)
 Meta-->>GW: applied=false
 GW->>Data: EnqueueChunkDeletion(losingChunks) — GC absorbs the orphan
 GW-->>C: 200 OK with the winning manifest (retry-safe)
 end

Step-by-step#

SigV4 verify. auth.Middleware canonicalises the request, computes the expected signature against the secret in auth.StaticStore, and rejects on mismatch. Streaming chunk uploads (aws-chunked) carry a chain HMAC validated incrementally by auth.streamingReader so a torn body fails before the manifest is touched. See Auth.

Multi-cluster routing

Mon, 01 Jan 0001 00:00:00 +0000

Multi-cluster routing#

Strata can front multiple data clusters (multiple RADOS pools, multiple upstream S3 endpoints, or a mix). Every PUT picks exactly one cluster before any chunks land. The picker is a thin layer over three inputs: the bucket’s placement policy, the per-cluster weights from the cluster_state table, and the storage-class spec inferred from the request headers.

Flowchart#

flowchart TD
 PUT["s3api.Server.putObject"] --> Spec["Resolve storage-class spec<br/>(default / standard-ia / glacier / @cluster)"]
 Spec --> Pin{"spec.Cluster != ''<br/>(@cluster suffix?)"}
 Pin -- yes --> PickPinned["clusterForPlacement(pinnedID)"]
 Pin -- no --> BPolicy{"bucket.Placement<br/>!= nil?"}
 BPolicy -- yes --> Effective["placement.EffectivePolicy(<br/>bucketPolicy, mode, weights, states)"]
 Effective --> EffectiveNonEmpty{"non-empty?"}
 EffectiveNonEmpty -- yes --> Wheel["placement.PickClusterExcluding(<br/>policy, drainSet)"]
 EffectiveNonEmpty -- no --> StrictCheck{"mode == strict?"}
 StrictCheck -- yes --> Refuse[/"503 DrainRefused<br/>(compliance pin)"/]
 StrictCheck -- no --> Default["Synthesised default policy<br/>from cluster weights"]
 Default --> Wheel
 BPolicy -- no --> Default
 Wheel --> Drain{"selected cluster<br/>draining?"}
 Drain -- yes --> Refuse
 Drain -- no --> Put["data.Backend.PutChunks(...)"]
 PickPinned --> DrainPin{"pinned cluster<br/>draining?"}
 DrainPin -- yes --> Refuse
 DrainPin -- no --> Put

The picker contract#

placement.PickCluster (and its drain-aware sibling PickClusterExcluding) lives in internal/data/placement/. The inputs are deterministic:

Meta store

Mon, 01 Jan 0001 00:00:00 +0000

Meta store#

internal/meta/store.go defines meta.Store, the interface every metadata backend implements. The contract is intentionally narrow: only the operations the S3 surface needs, and only with the consistency primitives the backends can all support without bolting a coordinator in front.

Three production-eligible backends satisfy meta.Store:

Backend	When to pick	Notes
`memory`	Tests, smoke pass, single-process demos	In-process tree-map. Naturally ordered. No durability.
`cassandra`	Multi-replica, scale tested against the s3-tests suite	ScyllaDB drops in unchanged (CQL-compatible). Sharded objects table, fan-out + heap-merge listing.
`tikv`	Multi-replica, prefer ordered scans	Native KV via `tikv/client-go`. Implements `RangeScanStore` so `ListObjects` is a single scan.

A new backend MUST satisfy meta.Store and pass the contract suite at internal/meta/storetest/contract.go. The suite is shared across all backends and is the parity oracle.

Drain pipeline

Mon, 01 Jan 0001 00:00:00 +0000

Drain pipeline#

A draining cluster is one that the operator has flagged stop-write so it can be taken out of rotation safely. Strata models the lifecycle as a state machine on the cluster_state row; the rebalance worker migrates chunks off in the evacuating state; the deregister-ready gate keeps the operator honest before they drop the cluster from STRATA_RADOS_CLUSTERS.

State diagram#

stateDiagram-v2
 [*] --> live: cluster_state row created<br/>(weight default 100)
 [*] --> pending: env-only cluster<br/>(no chunks yet)
 pending --> live: POST /activate {weight: N}
 live --> live: PUT /weight {weight: N}
 live --> draining_readonly: POST /drain {mode: "readonly"}
 live --> evacuating: POST /drain {mode: "evacuate"}
 draining_readonly --> evacuating: POST /drain {mode: "evacuate"}<br/>(upgrade — no readonly→evacuate skip)
 draining_readonly --> live: POST /undrain
 evacuating --> live: POST /undrain
 evacuating --> removed: chunks_on_cluster == 0<br/>operator drops env entry
 removed --> [*]: row remains as tombstone

States#

State	Picker	Reads	Scan-on-tick	Operator entry
`pending`	excluded from default wheel; explicit policy still routes	works	no	`POST /admin/v1/clusters/{id}/activate {weight: N}` flips to `live`
`live` (weight > 0)	included proportional to weight	works	no	`PUT /admin/v1/clusters/{id}/weight` adjusts in place
`live` (weight == 0)	excluded from default wheel; explicit policy still routes	works	no	legal “drained but not draining” state — useful for staged decommission
`draining_readonly`	excluded	works	no — readonly is stop-write only	`POST /admin/v1/clusters/{id}/drain {mode: "readonly"}`
`evacuating`	excluded	works	yes — rebalance worker migrates chunks off	`POST /admin/v1/clusters/{id}/drain {mode: "evacuate"}` or upgrade from readonly
`removed`	excluded everywhere	n/a	n/a	operator-flow tombstone — set after deregister

Stop-write semantics: a draining cluster accepts reads, deletes, HEAD, multipart UploadPart / Complete / Abort, and listings. Only fresh PUTs are refused with 503 DrainRefused + Retry-After: 300.

Data backend

Mon, 01 Jan 0001 00:00:00 +0000

Data backend#

internal/data/backend.go defines the data.Backend interface every chunk store implements. The metadata layer keeps the per-object manifest (chunk list, sizes, content hash); the data backend is responsible for opaque fixed-size chunks only. The split is what lets us drop in different backing stores (RADOS, S3, in-memory) without touching the gateway.

Backends#

Backend	Build tag	Notes
`memory`	none	In-process map. Used by tests and the smoke pass.
`rados`	`ceph`	RADOS pools via `goceph` (cgo, librados). Requires `make build` with `-tags ceph` or the docker-built image. Multi-cluster routing supported.
`s3`	none	S3-over-S3 — Strata as a transparent gateway in front of an upstream S3 endpoint. Useful for migrating from MinIO / SeaweedFS / AWS without lifting and shifting data.

Selection is via STRATA_DATA_BACKEND (memory / rados / s3). RADOS requires the configured pool ([rados] classes) to exist; the Storage status page covers the operator-facing health surface.

Workers

Mon, 01 Jan 0001 00:00:00 +0000

Workers#

Background loops run inside the same cmd/strata server binary as the gateway. STRATA_WORKERS= (or --workers=) selects which to run; an empty list runs the gateway only. Each worker is leader-elected, panic- recovered, and supervised — one worker’s panic or lease loss never affects the gateway or sibling workers.

Registry shape#

Each worker has a per-worker file under cmd/strata/workers/<name>.go that calls workers.Register from init():

func init() {
 workers.Register(workers.Worker{
 Name: "gc",
 Build: func(deps Dependencies) (Runner, error) { ... },
 SkipLease: true, // gc fan-out manages its own per-shard leases
 })
}

Build constructs the per-worker runner from the shared workers.Dependencies struct (Logger, Meta, Data, Tracer, Locker, Region, EmitLeader). Per-worker tunables (STRATA_GC_INTERVAL, STRATA_LIFECYCLE_*, …) are read inside Build directly so the dependency surface stays small.

Worker + leader election

Mon, 01 Jan 0001 00:00:00 +0000

Worker + leader election#

Every background worker shares a single supervisor shape: one goroutine per worker, leader-elected on a per-name lease, panic recovered with exponential backoff, heartbeat chip wired to the operator console. This page draws the lifecycle end-to-end and points at the per-worker carve-outs.

Lifecycle flowchart#

flowchart TD
 Boot["cmd/strata server"] --> Resolve["workers.Resolve(STRATA_WORKERS)"]
 Resolve -->|valid| Build["workers.Supervisor.Run(ctx, workers)"]
 Resolve -->|unknown name| Exit[/"exit 2"/]
 Build --> Spawn["spawn 1 goroutine per worker"]
 Spawn --> LeaseGate{"worker.SkipLease?"}
 LeaseGate -- no --> Acquire["leader.Session.Acquire(<name>-leader)"]
 LeaseGate -- yes --> RunnerBuild["Build(deps) → Runner"]
 Acquire -- acquired --> Emit["deps.EmitLeader(name, true)"]
 Emit --> RunnerBuild
 Acquire -- lost / partition --> Acquire
 RunnerBuild --> Run["runner.Run(ctx)"]
 Run -- clean return --> Release["release lease<br/>deps.EmitLeader(name, false)"]
 Run -- panic --> PanicMetric["strata_worker_panic_total{worker, shard}<br/>++"]
 PanicMetric --> Release
 Release --> Backoff{"healthy ≥ 5m?"}
 Backoff -- yes --> Spawn
 Backoff -- no --> Wait["sleep 1s → 5s → 30s → 2m"]
 Wait --> Spawn

Supervisor responsibilities#

workers.Supervisor.Run(ctx, []workers.Worker) is the entrypoint:

Storage status

Mon, 01 Jan 0001 00:00:00 +0000

Storage status — operator guide#

The Storage page (/console/storage) and the Cluster Overview Storage hero card surface the live health of the meta + data backends and the per-storage-class object distribution. This page covers the env vars the UI reads through, what each warning means, and how to interpret the RADOS / TiKV / Cassandra-specific signals.

The page is fed by three small admin endpoints:

Endpoint	Returns	Polled by
`GET /admin/v1/storage/meta`	`MetaHealthReport` (Cassandra peers / TiKV PD stores / memory)	`/storage` Meta tab @ 30 s
`GET /admin/v1/storage/data`	`DataHealthReport` (RADOS pool stats / S3 reachability / memory)	`/storage` Data tab @ 30 s
`GET /admin/v1/storage/classes`	`{classes:[…], pools_by_class:{…}}`	`/storage` Data tab @ 30 s, Cluster Overview hero @ 60 s
`GET /admin/v1/storage/health`	aggregate `{ok, warnings, source}`	`<StorageDegradedBanner>` @ 30 s on every authed page

Schema details live in internal/adminapi/openapi.yaml.

Sharding

Mon, 01 Jan 0001 00:00:00 +0000

Sharding#

Sharding is the single biggest divergence from Ceph RGW: every bucket’s metadata is split across N partitions instead of living in a single bucket-index object. The split avoids RGW’s bucket-index ceiling at large object counts and lets ListObjects scale linearly with N.

Objects table partition key#

The cassandra objects table is partitioned by (bucket_id, shard):

PRIMARY KEY ((bucket_id, shard), key, version_id)
WITH CLUSTERING ORDER BY (key ASC, version_id DESC)

Where shard = fnv32a(key) % N and N is per-bucket (STRATA_BUCKET_SHARDS at bucket creation, default 64). N must be a power of two — meta.IsValidShardCount(n) enforces it. Power-of-two constraint matters for the reshard worker below: when N doubles, every old shard either stays under the new modulo or splits cleanly into two new ones, never three.

Observability

Mon, 01 Jan 0001 00:00:00 +0000

Observability#

Strata ships three correlated observability surfaces: structured logs, an audit log table for state-changing requests, and OTel tracing with a ring-buffer trace browser embedded in the operator console. All three key off the same request_id so an operator can pivot from one to the next without rebuilding context.

Structured logs (slog)#

internal/logging is the canonical setup. The cmd/strata binary (both strata server and strata admin) calls logging.Setup() first thing to install a JSON-handler *slog.Logger driven by STRATA_LOG_LEVEL (DEBUG/INFO/WARN/ERROR; default INFO).