Storage Backend Compatibility

KafScale uses S3-compatible object storage as its source of truth. While designed for AWS S3, it works with any storage backend that implements the S3 API.

Compatibility Matrix

Legend:

• ✅ Native: Standard S3 SDK works with endpoint change
• ⚠️ Interop: Works via compatibility layer, minor config differences
• ❌ Proxy: Requires additional infrastructure

AWS S3

Native support. No endpoint configuration needed.

KafScaleCluster

apiVersion: kafscale.io/v1alpha1
kind: KafScaleCluster
metadata:
  name: production
  namespace: kafscale
spec:
  brokers:
    replicas: 3
  s3:
    bucket: kafscale-production
    region: us-east-1
    credentialsSecretRef: kafscale-s3
  etcd:
    endpoints: []

Credentials Secret

kubectl -n kafscale create secret generic kafscale-s3 \
  --from-literal=AWS_ACCESS_KEY_ID=AKIA... \
  --from-literal=AWS_SECRET_ACCESS_KEY=...

IAM Role (EKS with IRSA)

For production on EKS, use IAM Roles for Service Accounts instead of static credentials:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: kafscale-broker
  namespace: kafscale
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/kafscale-s3-role

Required IAM permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::kafscale-production",
        "arn:aws:s3:::kafscale-production/*"
      ]
    }
  ]
}

DigitalOcean Spaces

Drop-in S3 replacement. Change endpoint and region.

KafScaleCluster

apiVersion: kafscale.io/v1alpha1
kind: KafScaleCluster
metadata:
  name: production
  namespace: kafscale
spec:
  brokers:
    replicas: 3
  s3:
    bucket: kafscale-production
    region: nyc3
    endpoint: https://nyc3.digitaloceanspaces.com
    credentialsSecretRef: kafscale-s3
  etcd:
    endpoints: []

Credentials Secret

Generate Spaces access keys in the DigitalOcean console under API → Spaces Keys.

kubectl -n kafscale create secret generic kafscale-s3 \
  --from-literal=AWS_ACCESS_KEY_ID=DO00... \
  --from-literal=AWS_SECRET_ACCESS_KEY=...

Available regions: nyc3, sfo3, ams3, sgp1, fra1

Cloudflare R2

S3-compatible with zero egress fees. Ideal for high-read workloads.

KafScaleCluster

apiVersion: kafscale.io/v1alpha1
kind: KafScaleCluster
metadata:
  name: production
  namespace: kafscale
spec:
  brokers:
    replicas: 3
  s3:
    bucket: kafscale-production
    region: auto
    endpoint: https://<ACCOUNT_ID>.r2.cloudflarestorage.com
    credentialsSecretRef: kafscale-s3
  etcd:
    endpoints: []

Credentials Secret

Generate R2 API tokens in Cloudflare dashboard under R2 → Manage R2 API Tokens.

kubectl -n kafscale create secret generic kafscale-s3 \
  --from-literal=AWS_ACCESS_KEY_ID=... \
  --from-literal=AWS_SECRET_ACCESS_KEY=...

Note: Replace <ACCOUNT_ID> with your Cloudflare account ID.

Backblaze B2

Cost-effective S3-compatible storage.

KafScaleCluster

apiVersion: kafscale.io/v1alpha1
kind: KafScaleCluster
metadata:
  name: production
  namespace: kafscale
spec:
  brokers:
    replicas: 3
  s3:
    bucket: kafscale-production
    region: us-west-004
    endpoint: https://s3.us-west-004.backblazeb2.com
    credentialsSecretRef: kafscale-s3
  etcd:
    endpoints: []

Credentials Secret

Create application keys in B2 console with read/write access to your bucket.

kubectl -n kafscale create secret generic kafscale-s3 \
  --from-literal=AWS_ACCESS_KEY_ID=<keyID> \
  --from-literal=AWS_SECRET_ACCESS_KEY=<applicationKey>

Endpoint format: https://s3.<region>.backblazeb2.com

Wasabi

Hot cloud storage with flat pricing and no egress fees.

KafScaleCluster

apiVersion: kafscale.io/v1alpha1
kind: KafScaleCluster
metadata:
  name: production
  namespace: kafscale
spec:
  brokers:
    replicas: 3
  s3:
    bucket: kafscale-production
    region: us-east-1
    endpoint: https://s3.us-east-1.wasabisys.com
    credentialsSecretRef: kafscale-s3
  etcd:
    endpoints: []

Credentials Secret

kubectl -n kafscale create secret generic kafscale-s3 \
  --from-literal=AWS_ACCESS_KEY_ID=... \
  --from-literal=AWS_SECRET_ACCESS_KEY=...

Available regions: us-east-1, us-east-2, us-west-1, eu-central-1, eu-west-1, ap-northeast-1, ap-northeast-2

Google Cloud Storage

GCS provides S3 interoperability via its XML API. Requires HMAC keys.

Setup

1. Enable interoperability in GCS console: Storage → Settings → Interoperability
2. Create HMAC keys for a service account

KafScaleCluster

apiVersion: kafscale.io/v1alpha1
kind: KafScaleCluster
metadata:
  name: production
  namespace: kafscale
spec:
  brokers:
    replicas: 3
  s3:
    bucket: kafscale-production
    region: auto
    endpoint: https://storage.googleapis.com
    credentialsSecretRef: kafscale-s3
  etcd:
    endpoints: []

Credentials Secret

Use HMAC keys (not JSON service account keys):

kubectl -n kafscale create secret generic kafscale-s3 \
  --from-literal=AWS_ACCESS_KEY_ID=GOOG... \
  --from-literal=AWS_SECRET_ACCESS_KEY=...

Limitations

• Some S3 features may behave differently (e.g., versioning, lifecycle policies)
• Path-style URLs required (GCS doesn’t support virtual-hosted style for interop)
• Multipart upload semantics may vary slightly

Azure Blob Storage

Azure Blob Storage is not S3-compatible. KafScale needs an S3-compatible proxy in front of Azure Blob. MinIO gateway mode is no longer available, so do not deploy MinIO with gateway azure.

Point the cluster at your proxy the same way you point it at any other S3-compatible endpoint. Set s3.endpoint to the proxy URL, and store the proxy credentials in credentialsSecretRef.

Trade-offs

• Added latency: an extra network hop through the proxy
• Operational overhead: another component to run and monitor
• Availability: the proxy needs a high-availability setup, or it becomes a single point of failure
• Cost: compute for the proxy

Consider a native S3-compatible provider if Azure is not a hard requirement.

MinIO (Self-Hosted)

Run your own S3-compatible storage on any infrastructure: on-prem, edge, or any cloud.

Docker Compose (Development)

The default docker-compose.yml includes MinIO:

docker-compose up -d

MinIO runs on port 9000 with default credentials minioadmin:minioadmin.

KafScaleCluster (Kubernetes)

apiVersion: kafscale.io/v1alpha1
kind: KafScaleCluster
metadata:
  name: demo
  namespace: kafscale
spec:
  brokers:
    replicas: 3
  s3:
    bucket: kafscale-data
    endpoint: http://minio.minio-system.svc:9000
    credentialsSecretRef: kafscale-s3
  etcd:
    endpoints: []

Production MinIO

For production, deploy MinIO in distributed mode with erasure coding:

helm repo add minio https://charts.min.io/
helm install minio minio/minio \
  --namespace minio-system --create-namespace \
  --set mode=distributed \
  --set replicas=4 \
  --set persistence.size=100Gi

Common Considerations

Path Style vs Virtual Hosted Style

Some S3-compatible backends only support path-style URLs. If you encounter bucket resolution issues, ensure your SDK is configured for path-style access.

TLS/SSL

For production, always use HTTPS endpoints. Self-signed certificates may require additional CA configuration in the broker pods.

Regional Latency

Place your storage in the same region as your Kubernetes cluster. Every acknowledged produce waits for an S3 write, so storage write latency is most of your produce latency. AWS recommends the same for Amazon S3: “To optimize performance, we recommend that you access the bucket from Amazon EC2 instances in the same AWS Region when possible. This helps reduce network latency and data transfer costs.” (source)

For Amazon S3, AWS documents small-object latency of roughly 100–200 milliseconds (source). The other backends on this page publish their own figures or none, so measure your backend with kafscale_s3_latency_ms_avg. See Latency and batching.

Bucket Lifecycle

KafScale manages segment files in .kfs format. Configure bucket lifecycle policies carefully. Avoid auto-deletion rules that could remove active segments.

Testing Your Backend

Before deploying KafScale, verify S3 compatibility:

# Using AWS CLI with custom endpoint
aws s3 ls --endpoint-url https://your-endpoint.com

# Create test bucket
aws s3 mb s3://kafscale-test --endpoint-url https://your-endpoint.com

# Upload test object
echo "test" | aws s3 cp - s3://kafscale-test/test.txt --endpoint-url https://your-endpoint.com

# Verify
aws s3 ls s3://kafscale-test/ --endpoint-url https://your-endpoint.com

Tested Configurations

Other backends listed should work but may not be continuously tested. Community reports welcome.