S3 Data Backend#

The S3-over-S3 data backend stores object bytes in any S3-compatible bucket (AWS S3, MinIO, Ceph RGW, Garage). One Strata object = one backend S3 object under the key <bucket-uuid>/<object-uuid>. RADOS remains the default and recommended data backend; this page covers the S3 alternative.

When to choose S3 over RADOS#

The two backends are equal-tier alternatives — pick by what your operations team already runs. Mixing within a single Strata deployment is not supported (STRATA_DATA_BACKEND is a single-value flag).

Required environment variables#

The S3 backend is configured via two JSON envs — STRATA_S3_CLUSTERS (array of bucket-less cluster specs) + STRATA_S3_CLASSES (per-class {cluster, bucket} map). Both REQUIRED when STRATA_DATA_BACKEND=s3. See S3 multi-cluster routing for the full env shape, credentials_ref discriminator, validation rules, and rolling-restart workflow.

The 13 legacy STRATA_S3_BACKEND_* envs retired in ralph/s3-multi-cluster — set the two JSON envs above instead.

docker-compose snippet#

The repo’s deploy/docker/docker-compose.yml ships a ready-made s3-backend profile (MinIO + bucket bootstrap + gateway):

make up-s3-backend     # cassandra + minio + init-minio + strata-s3
make smoke             # exercise the gateway through the new backend

For a custom AWS / Ceph RGW deployment, drop in something like:

strata-gateway:
  image: strata:ceph
  environment:
    STRATA_DATA_BACKEND: s3
    STRATA_META_BACKEND: cassandra
    STRATA_CASSANDRA_HOSTS: cassandra
    STRATA_S3_CLUSTERS: |
      [{"id":"primary","endpoint":"https://s3.us-east-1.amazonaws.com","region":"us-east-1","force_path_style":false,"credentials":{"type":"chain"}}]
    STRATA_S3_CLASSES: |
      {"STANDARD":{"cluster":"primary","bucket":"my-strata-data"}}
    # creds via IRSA on EKS — {"type":"chain"} picks up the web-identity token
  ports:
    - "9999:9000"

Kubernetes / IRSA#

On EKS, attach an IAM role with s3:GetObject, s3:PutObject, s3:DeleteObject, s3:AbortMultipartUpload, s3:ListBucket, s3:GetBucketLifecycle, s3:PutBucketLifecycle, s3:GetBucketCors, and s3:PutBucketCors to the Strata pod’s service account. Set the cluster’s credentials envelope to {"type":"chain"} — the SDK’s default chain picks up the IRSA web-identity token.

Operator-required backend bucket configuration#

These items are load-bearing, not optional notes.

(a) AbortIncompleteMultipartUpload lifecycle rule#

Every gateway crash mid-multipart leaks orphan parts on the backend bucket indefinitely. The backend lifecycle below cleans them up after seven days without any Strata-side effort:

{
  "Rules": [
    {
      "ID": "strata-abort-incomplete-mpu",
      "Status": "Enabled",
      "Filter": {"Prefix": ""},
      "AbortIncompleteMultipartUpload": {"DaysAfterInitiation": 7}
    }
  ]
}

Apply once at bucket creation:

aws s3api put-bucket-lifecycle-configuration \
  --bucket my-strata-data \
  --lifecycle-configuration file://abort-mpu.json

Strata’s lifecycle worker also emits AbortIncompleteMultipartUpload rules when buckets carry one — this safety rule is for the case where no Strata bucket has an explicit rule yet.

(b) Versioning anti-pattern#

Do not enable versioning on a Strata-managed backend bucket that has already been written to with empty VersionID rows. Once versioning is enabled, plain DeleteObject against legacy rows creates delete-markers instead of freeing the bytes — those rows leak forever.

Strata handles versioning-enabled buckets correctly only when versioning was enabled before any writes (per US-008’s defensive design — BackendRef.VersionID captures the version on every PUT, every Delete passes it back). If you must enable versioning on an existing bucket, plan a separate rewrite pass that re-PUTs every object so each row gains a recorded version-id.

(c) Cross-region is anti-pattern#

Co-locate the Strata gateway with the backend bucket in the same region. Cross-region setups (Strata in us-east-1, backend bucket in eu-west-1) inflate every PUT / GET round-trip by a transatlantic hop, double-egress the data, and inflate cost. Strata does not enforce region pairing — it’s an operator concern.

Tested-against backends#

Capability matrix#

Which behaviour ships native vs. routed through Strata’s workers per backend:

Cells marked — mean Strata’s own worker / gateway code path covers the gap — no behaviour is lost, just less work the backend offloads.

Performance characteristics#

• Latency floor: every Strata GET / PUT carries one HTTPS round-trip to the backend. Same-region AWS S3: typically 20–50 ms p99 for small GETs; same-host MinIO: 1–3 ms p99. RADOS in comparison clears < 5 ms p99 on warm objects. Pick S3 when the latency floor is acceptable.
• Throughput: bound by part_size × upload_concurrency (default 16 MiB × 4 = 64 MiB peak buffer per PUT). Single Strata gateway against AWS S3 saturates at roughly 1 GB/s for large multipart uploads on a 10 Gbit NIC; MinIO same-host can reach 4–6 GB/s.
• Cost model: AWS S3 charges per request (PUT, GET, DELETE) and per byte stored / transferred. Strata’s “one object = one backend object” invariant keeps PUT request count proportional to client request count; the 4-MiB-chunks shape (rejected in this PRD) would have inflated PUT count by ~250×. Range GETs do not create extra PUT requests — the backend object stays whole.
• Hot-prefix throttling: AWS partitions by key prefix; the <bucket-uuid>/<object-uuid> UUID prefix gives high-entropy distribution from the first byte. No manual prefix-randomization needed.

Common operational pitfalls#

• force-path-style=false against MinIO / Ceph RGW. Default for AWS; required true for non-AWS endpoints. Symptom: gateway can list the bucket but every PUT / GET returns 403 with a message about virtual-hosted-style addressing.
• Forgot the IRSA pod-identity binding on EKS. Symptom: gateway crashes at boot with “operation error S3: HeadBucket … 403 Forbidden” during the writability probe. Fix: bind the IAM role; do not fall back to baking long-lived access keys into a {"type":"env",...} envelope unless rotation is owned out-of-band.
• Bucket pre-creation skipped. Strata refuses to start if the bucket referenced by a class in STRATA_S3_CLASSES is missing or unwritable. The boot-time PUT/DELETE probe on key .strata-readyz-canary catches read-only mounts, missing IAM permissions, expired creds. Pre-create the bucket with aws s3 mb / mc mb before bringing up the gateway.
• Bucket versioning enabled mid-flight. See note (b) above — leaks bytes on legacy rows. Plan versioning at bucket creation time.
• Cross-region setup. See note (c) above — co-locate gateway and bucket.
• Backend lifecycle pushed by Strata vs. operator-applied rule. Strata re-pushes the derived backend lifecycle on every PutBucketLifecycle. Operator-applied rules (like the abort-MPU rule above) are preserved only when their ID does not collide with rules Strata derives. Use distinct rule IDs (strata-* for Strata, ops-* for operator-managed) to avoid surprises.
• Per-bucket presign passthrough off by default. The ?backendPresign admin endpoint flips it on per-Strata-bucket. Without that flip, presigned GET URLs continue to point at Strata even when the data could be served straight from the backend.

Lifecycle bidirectional mapping (US-014)#

When the data backend is s3, Strata translates a subset of each bucket’s lifecycle rules into a backend bucket lifecycle so the backend handles native transitions and expirations directly. Strata’s stored configuration is the source of truth; the backend config is derived state — it is re-pushed on every PutBucketLifecycle and cleared on DeleteBucketLifecycle.

Translation is best-effort. Rules whose only action is a non-native transition stay with Strata’s lifecycle worker; rules with a native transition AND non-translatable extras get the native parts pushed and the worker keeps owning the rest.

Translation table#

Filter scoping#

Each emitted backend rule’s Filter.Prefix is <bucket-uuid>/<userPrefix> so multiple Strata buckets sharing the same backend bucket do not collide. The <bucket-uuid> matches the prefix Strata uses in BackendRef.Key at PUT time.

Source of truth#

GetBucketLifecycle returns the Strata-stored XML blob, never the backend’s view. Use the backend’s own client (aws s3api get-bucket-lifecycle-configuration --bucket <backend-bucket>) to inspect the derived backend rules.

Failure modes#

A backend translation failure is logged at WARN and does NOT fail the user request. The user-facing PUT succeeds and Strata’s stored config is the authoritative copy; the worker continues to own everything Strata’s stored config describes. Re-PUT the same lifecycle config after the backend recovers to retry translation.