Runtime Settings

All configuration is done via environment variables. Set these in your Helm values, KafScaleCluster CRD, or container spec.

Broker Configuration

Core Settings

Segment and Flush Settings

Tuning guidance:

• Lower latency: Reduce FLUSH_INTERVAL_MS (increases S3 requests)
• Higher throughput: Increase SEGMENT_BYTES (larger batches, fewer uploads)
• Cost optimization: Increase both (fewer S3 PUT requests)

Cache Settings

Cache sizing guidance:

• Size CACHE_BYTES to hold your hot working set (recent segments consumers are reading)
• Monitor kafscale_cache_hit_ratio; target > 90% for typical workloads
• READAHEAD_SEGMENTS helps sequential consumers; increase for high-throughput topics

Topic Settings

Durability Settings

Durability vs Cost tradeoff:

Metrics Settings

Debug and Testing

S3 Configuration

Bucket Settings

Read Replica Settings (Multi-Region)

For reduced read latency in multi-region deployments, configure a read replica bucket using S3 Cross-Region Replication (CRR) or Multi-Region Access Points (MRAP):

Example:

# Primary (writes)
export KAFSCALE_S3_BUCKET=prod-segments
export KAFSCALE_S3_REGION=us-east-1

# Replica (reads in EU)
export KAFSCALE_S3_READ_BUCKET=prod-segments-replica
export KAFSCALE_S3_READ_REGION=eu-west-1

Encryption Settings

For SSE-S3 (AES-256), configure at the bucket level. For SSE-KMS, set the key ARN here.

Credentials

If not using IAM roles (EKS IRSA, EC2 instance profiles):

Health Thresholds

Health states:

S3-Compatible Storage Backends

KafScale works with any S3-compatible storage. Set KAFSCALE_S3_ENDPOINT and credentials for your provider.

See Storage Compatibility for detailed configuration examples.

MinIO example:

spec:
  s3:
    bucket: kafscale-data
    endpoint: http://minio.minio-system.svc:9000
    pathStyle: true
    credentialsSecretRef: kafscale-minio

DigitalOcean Spaces example:

spec:
  s3:
    bucket: kafscale-production
    region: nyc3
    endpoint: https://nyc3.digitaloceanspaces.com
    credentialsSecretRef: kafscale-do

etcd Configuration

etcd stores topic metadata, consumer offsets, and cluster coordination state. It’s the only stateful component you need to back up.

Endpoint Resolution Order

The operator resolves etcd endpoints in this order:

1. KafScaleCluster.spec.etcd.endpoints (CRD)
2. KAFSCALE_OPERATOR_ETCD_ENDPOINTS (env var)
3. Managed etcd (operator creates a 3-node StatefulSet)

Availability Signals

When etcd is unavailable, brokers reject producer/admin/consumer-group operations with REQUEST_TIMED_OUT. Producers see per-partition errors in the Produce response; admin and group APIs return the same code in their response payloads.

Fetch requests for cached segments continue to work—brokers serve reads from their local cache even during etcd outages.

Connection Settings

Examples:

# Single node (development)
KAFSCALE_ETCD_ENDPOINTS=http://etcd.kafscale.svc:2379

# Cluster (production)
KAFSCALE_ETCD_ENDPOINTS=http://etcd-0.etcd:2379,http://etcd-1.etcd:2379,http://etcd-2.etcd:2379

# With TLS
KAFSCALE_ETCD_ENDPOINTS=https://etcd-0.etcd:2379,https://etcd-1.etcd:2379,https://etcd-2.etcd:2379
KAFSCALE_ETCD_CERT_FILE=/etc/kafscale/etcd/tls.crt
KAFSCALE_ETCD_KEY_FILE=/etc/kafscale/etcd/tls.key
KAFSCALE_ETCD_CA_FILE=/etc/kafscale/etcd/ca.crt

Proxy Configuration

The proxy provides a single entry point that routes Kafka requests to the appropriate broker.

The proxy discovers brokers via etcd by default. Use KAFSCALE_PROXY_BACKENDS for static configuration:

KAFSCALE_PROXY_BACKENDS=broker-0:9092,broker-1:9092,broker-2:9092

Helm Values

Console Configuration

The console provides a web UI for cluster monitoring and topic management.

Example:

console:
  env:
    - name: KAFSCALE_CONSOLE_HTTP_ADDR
      value: ":8080"
    - name: KAFSCALE_UI_USERNAME
      valueFrom:
        secretKeyRef:
          name: kafscale-console
          key: username
    - name: KAFSCALE_UI_PASSWORD
      valueFrom:
        secretKeyRef:
          name: kafscale-console
          key: password

Operator Configuration

Core Settings

Managed etcd Settings

When etcd.endpoints is empty in the CRD, the operator deploys a managed etcd cluster:

etcd Snapshot Settings

Snapshots are stored in a separate bucket from broker segments by default.

Broker Defaults

The operator passes these to brokers if not overridden in the CRD:

TLS Configuration

CRD example:

spec:
  tls:
    enabled: true
    secretRef: kafscale-tls

The secret should contain tls.crt and tls.key. For mTLS, include ca.crt.

See Security for complete TLS setup instructions.

External Broker Access

By default, brokers advertise the in-cluster service DNS name. External clients need a reachable address.

CRD Settings (spec.brokers)

LoadBalancer Example (AWS NLB)

spec:
  brokers:
    replicas: 3
    advertisedHost: kafka.example.com
    advertisedPort: 9092
    service:
      type: LoadBalancer
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: nlb
        service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing

NodePort Example

spec:
  brokers:
    replicas: 3
    advertisedHost: node1.example.com
    advertisedPort: 30092
    service:
      type: NodePort
      kafkaNodePort: 30092
      metricsNodePort: 30093

Consumer Group Settings

Session timeout and heartbeat intervals are negotiated by Kafka clients following protocol defaults. KafScale respects the values sent by clients:

These are client-side settings configured in your Kafka producer/consumer, not broker configuration.

Example Configurations

Minimal Production

apiVersion: kafscale.io/v1alpha1
kind: KafScaleCluster
metadata:
  name: prod
  namespace: kafscale
spec:
  brokers:
    replicas: 3
    env:
      - name: KAFSCALE_SEGMENT_BYTES
        value: "16777216"
      - name: KAFSCALE_FLUSH_INTERVAL_MS
        value: "1000"
      - name: KAFSCALE_CACHE_BYTES
        value: "4294967296"
      - name: KAFSCALE_LOG_LEVEL
        value: "warn"
  s3:
    bucket: kafscale-prod
    region: us-east-1
    credentialsSecretRef: kafscale-s3
  etcd:
    endpoints:
      - http://etcd-0.etcd.kafscale.svc:2379
      - http://etcd-1.etcd.kafscale.svc:2379
      - http://etcd-2.etcd.kafscale.svc:2379

DigitalOcean Spaces

apiVersion: kafscale.io/v1alpha1
kind: KafScaleCluster
metadata:
  name: prod
  namespace: kafscale
spec:
  brokers:
    replicas: 3
  s3:
    bucket: kafscale-prod
    region: nyc3
    endpoint: https://nyc3.digitaloceanspaces.com
    credentialsSecretRef: kafscale-do
  etcd:
    endpoints: []

MinIO (On-Prem / Dev)

apiVersion: kafscale.io/v1alpha1
kind: KafScaleCluster
metadata:
  name: dev
  namespace: kafscale
spec:
  brokers:
    replicas: 1
    env:
      - name: KAFSCALE_LOG_LEVEL
        value: "debug"
      - name: KAFSCALE_TRACE_KAFKA
        value: "true"
  s3:
    bucket: kafscale-dev
    endpoint: http://minio.minio-system.svc:9000
    pathStyle: true
    credentialsSecretRef: kafscale-minio
  etcd:
    endpoints: []

Multi-Region with Read Replicas

apiVersion: kafscale.io/v1alpha1
kind: KafScaleCluster
metadata:
  name: global
  namespace: kafscale
spec:
  brokers:
    replicas: 3
    env:
      - name: KAFSCALE_S3_READ_BUCKET
        value: "kafscale-replica"
      - name: KAFSCALE_S3_READ_REGION
        value: "eu-west-1"
  s3:
    bucket: kafscale-primary
    region: us-east-1
    credentialsSecretRef: kafscale-s3
  etcd:
    endpoints:
      - http://etcd-0.etcd.kafscale.svc:2379
      - http://etcd-1.etcd.kafscale.svc:2379
      - http://etcd-2.etcd.kafscale.svc:2379

Next Steps

• Installation for Helm and CRD setup
• Storage Compatibility for non-AWS backends
• Operations for monitoring and scaling
• Security for TLS and authentication