Upgrade and compatibility guide
This is the canonical guide for the unreleased changes after published v1.6.3
(092291b78b8abf1e5bde6cafd92550e9726b46e0). No release version is assigned here.
The release recommendation remains withdrawn pending the gates in
issue #298.
Compatibility has separate boundaries
| Surface | Scope and limits |
|---|---|
| Go source | Compare exported root-module APIs against published v1.6.3, including interface implementations and comparable types. Unkeyed struct literals are not covered; see Use keyed struct literals. This does not compare nested processor modules, binary ABI or all runtime semantics. |
| Kafka wire | Brokers and proxies advertise the API versions they implement. Focused classic consumer-group, Produce/Fetch and native client tests cover those paths; they do not establish every Kafka API or every client/version. Legacy MessageSets are rejected; magic-2 batches are required. |
| Schema Registry REST | The built-in service implements the subset documented in Schema Registry, with Avro, Protobuf and JSON Schema. Its compatibility modes describe schema evolution, separately from Go or Kafka compatibility. The reported matrix used Confluent 8.4.0 Java serializers, confluent-kafka-python 2.16.1 and franz-go pkg/sr 1.8.0. It is not a claim of full Confluent product parity. |
| Kubernetes CRDs | Apply the updated CRD before using new fields. Schema validation and Helm rendering checks do not prove a rolling upgrade on every Kubernetes/operator version. |
| Persisted data | Existing offset/group metadata and LFS envelopes remain readable in the covered tests. Topic recreation introduces generation-specific S3 prefixes; registry hash/descriptor aliases preserve covered schema IDs. Back up etcd and S3 together. Forward readability does not imply that an older binary can interpret new data on rollback. |
Go API migration
Published source remains supported. CoordinatorConfig retains its exact single
field and comparability, so existing code continues to compile:
cfg := broker.CoordinatorConfig{CleanupInterval: time.Second}
configs := map[broker.CoordinatorConfig]string{cfg: "legacy"}
_ = configs
coordinator := broker.NewGroupCoordinator(store, brokerInfo, &cfg)
defer coordinator.Stop()
For retention settings and coordination callbacks, use the new API:
options := broker.CoordinatorOptions{
CleanupInterval: time.Second,
OffsetsRetention: 7 * 24 * time.Hour,
OwnsGroup: leases.Owns,
AcquireGroup: leases.Acquire,
ReleaseGroup: leases.Release,
GroupDeleted: leases.Release,
}
coordinator := broker.NewGroupCoordinatorWithOptions(store, brokerInfo, &options)
defer coordinator.Stop()
Code built from intermediate main revisions that put callbacks or retention
settings on CoordinatorConfig must move them to CoordinatorOptions and use
NewGroupCoordinatorWithOptions. CoordinatorOptions contains functions and is
not comparable. Keep comparable configuration keys in the legacy type or a
separate application-owned value type.
External implementations of the v1.6.3 metadata.Store do not need new methods.
For example, an existing var _ metadata.Store = (*CustomStore)(nil) remains
valid. All-group listing is still available:
allOffsets, err := store.ListConsumerOffsets(ctx)
For a group-specific read, replace application-side filtering with:
offsets, err := metadata.ListGroupConsumerOffsets(ctx, store, groupID)
The helper selects the optional metadata.GroupConsumerOffsetLister interface
when implemented. Otherwise it calls the original ListConsumerOffsets, filters
by exact group ID, preserves offset metadata and propagates errors/cancellation.
The fallback scans all groups, so unrelated backend read failures can affect it.
Implement the optional method to get efficient group-scoped reads; the built-in
in-memory and etcd stores implement it. Existing concrete-store methods remain
available. Other atomic group/offset extensions remain optional too; custom
stores must implement those capabilities to gain the corresponding fencing
semantics. Source compatibility alone does not supply atomicity.
Use keyed struct literals
These exported structs gained fields since v1.6.3. Unkeyed (positional) literals of them written against v1.6.3 no longer compile. Keyed literals compile against both versions.
| Struct | Added fields |
|---|---|
metadata.ConsumerOffset |
Metadata |
metadata.EtcdStoreConfig |
StorageNamespace |
v1alpha1.ClusterConfigSpec |
OffsetsRetentionMinutes (first field), ResponseWriteTimeoutMs |
v1alpha1.KafscaleClusterSpec |
Security (first field) |
offset := metadata.ConsumerOffset{Group: "orders", Topic: "events", Partition: 0, Offset: 42}
broker.Server gained WriteTimeout, and the generated pkg/gen/metadata
types ConsumerGroup and GroupMember gained fields. These structs have
unexported fields, so code outside their packages could not use unkeyed
literals of them against v1.6.3 either. The v1.6.3 unkeyed CoordinatorConfig
literal still compiles.
Deprecation policy
CoordinatorConfig and NewGroupCoordinator are deprecated in favor of the
options API, but remain supported in this release. Deprecated exported APIs
must have a supported replacement and migration guidance alongside them.
Removal requires an announced future breaking release. No removal version is
scheduled. Store.ListConsumerOffsets retains its all-group purpose and is not
deprecated.
LFS checksum behavior
The Node, browser, Python and Java SDK resolvers validate the SHA-256 of the
downloaded object by default. The envelope checksum_alg field selects the
expected value:
checksum_alg |
Expected SHA-256 |
|---|---|
missing or sha256 |
checksum, or sha256 when checksum is empty |
md5, crc32 or none |
sha256 |
| any other value | Resolution fails with an unsupported checksum algorithm error. |
The proxy writes the SHA-256 of the stored object to sha256 for every
algorithm, and every SDK envelope decoder rejects an envelope whose sha256 is
missing or empty before it reads the object. A mismatch
rejects resolution before a record is returned. Objects whose SHA-256 matches
the expected value return their exact bytes. With none, the SDKs still
validate SHA-256. The Go resolver in pkg/lfs validates checksum with the
named algorithm and skips validation for none.
Disabling validation returns the downloaded bytes, including corrupt bytes:
validateChecksum: false in the Node and browser SDKs, validate_checksum=False
in Python, and validateChecksum set to false in the Java LfsResolver
constructor. Plain records pass through, and the envelope payload size limit
still applies. See issue #297.
Deployment and rollback considerations
-
Building from source requires Go 1.26 or newer, as required by the maintained Protobuf reflection library.
-
Empty groups written by older brokers receive a full retention period from first discovery. Apply the updated KafscaleCluster CRD before setting
spec.config.offsetsRetentionMinutes; omit it for the seven-day default or set it to0to disable automatic expiry. -
During a rolling upgrade, deleting a group through an older broker removes its metadata but leaves its offsets. Upgraded brokers discover those offsets and list the group again as
Empty. After all brokers are upgraded, delete the inactive group again to remove its retained offsets. -
Members recorded by older brokers without heartbeat timestamps receive one local session timeout to reconnect after coordinator takeover. They are not expired immediately during a rolling upgrade.
-
On coordinator takeover, recovered heartbeat times receive a five-second expiry margin for clock skew. Locally received heartbeats return to the configured session timeout.
-
Topics that already exist with names outside the allowed characters, such as names with
@or/, keep loading and accept Produce and Fetch as before. Only new creates are rejected. A topic named<name>@<N>shares an S3 prefix with generation N of<name>, so do not delete and re-create<name>while it exists. Move its data to a topic with a valid name and delete it first. - Schema registry rollback: rolling back to a main build from between #277 and #282 registers duplicate versions for 5 of the 54 edge schemas in our upgrade test.
- These are the Avro schemas with the
5e-324and1e-323double defaults, registered with and withoutnormalize=true, and the schema with an empty nested namespace registered withnormalize=true. - Those builds do not match the hashes that #282 stores, so re-registering one of these schemas creates a new version and id.
- The #282 review build
035b209passed that 54-schema regression. This does not establish general rollback safety.
- These are the Avro schemas with the
- The proxy no longer keeps serving through an etcd outage without a time limit. Earlier builds kept using the last broker list with no limit.
- Within
KAFSCALE_PROXY_BACKEND_CACHE_TTL_SEC(default60) of the last successful etcd read, the Kafka listener keeps using its last metadata. Produces still wait, because brokers do not ack while etcd is down. - Past the TTL the proxy reports NotReady, answers Kafka requests with
REQUEST_TIMED_OUT(an all-topics Metadata request gets an empty reply), closes the client connection, and still answers ApiVersions. - The LFS HTTP API returns 503
metadata_unavailableas soon as etcd cannot be read, even within the TTL. - The proxy serves again within a few seconds after etcd answers. etcd clients use a one-second maximum gRPC reconnect backoff, and each connection attempt is given at least five seconds, so recovery is no longer delayed by the default two-minute backoff. The limit does not apply when
KAFSCALE_PROXY_BACKENDSlists static brokers. - Full details: Proxy behavior when etcd is down.
- Within
- Brokers now upload segments with
PutObjectandIf-None-Match: *(#324, #327).- A broker that lost its partition lease can no longer overwrite segments written by the new owner.
- The S3 backend must honor the condition. AWS S3 and MinIO do. Backblaze B2 and the Google Cloud Storage XML API do not.
- Brokers check this at startup and refuse to start if a second write to the same probe key does not return
412 Precondition Failed(#336). A backend that rejects the first conditional write with501or the error codeNotImplementedfails startup at once withbackend rejects If-None-Match: *. A first write that fails with the error codeInvalidArgumentorInvalidRequestfails startup at once withbackend rejected the conditional PutObject (If-None-Match: *). Allows3:DeleteObjecton__health/for probe cleanup. See Conditional writes. - Code that implements
storage.S3Clientitself must makeUploadSegmentfail withstorage.ErrPreconditionFailedwhen the key already exists.
- With brokers found in etcd, the LFS HTTP API now sends
/lfs/produceand multipart completion to the broker that owns the partition.- A broker that does not lead the partition used to make these requests fail with HTTP 502 and leave the uploaded object in S3 as an orphan.
- The proxy now tries the current owner, and when no broker accepts the produce the request gets 503
backend_unavailableand the object is deleted. - This also applies to static backends (
spec.lfsProxy.backends): the produce goes to the owner from the etcd routes when it is known, and the listed brokers are used when it is not known or cannot be dialed. A non-owner answer gives 503 with the object deleted instead of 502 with an orphan, after the owner or another listed broker was tried. - With only one address, for example a broker Service, and no reachable owner, there is no other broker to try.
- Clients that treat 502 from these endpoints as final should also retry on 503 (#318).
LFS proxy S3 credential keys
Before this release, an operator managed LFS proxy (spec.lfsProxy) read only the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY keys of the spec.s3.credentialsSecretRef secret.
- A secret with only the
KAFSCALE_S3_ACCESS_KEYandKAFSCALE_S3_SECRET_KEYkeys, the format the brokers and snapshot jobs use, kept the LFS proxy pods inCreateContainerConfigError. The proxy now accepts either pair. - No migration is needed: secrets with the
AWS_keys work as before and keep priority when both pairs are present. - A secret that has neither pair, or does not exist, no longer blocks the pod; the proxy then uses the default AWS credential chain if one is available.
- The operator flags such a secret, and one it cannot read, with the
S3Credentialscondition set toFalseand a warning in its log. - When the secret holds only part of a pair, the condition message names the keys found, because the etcd snapshot jobs can fail on such a secret instead of falling back.
- The proxy logs which credential source it uses at startup, so check both the condition and that log if LFS uploads fail with S3 access errors after the upgrade.
The LFS proxy forwards AWS_SESSION_TOKEN alongside the AWS key pair. When both
pairs are present, it uses the AWS pair and its token; the KafScale fallback
retains its own token. A permanent AWS pair does not inherit a KafScale token.
The startup credential log includes session_token=true/false for static keys,
without exposing the token. The flag is omitted for default-chain credentials,
which may resolve later through a provider.
When spec.s3.credentialsSecretRef is set, the broker and LFS proxy pods roll once after the upgrade, because their pod templates gain the kafscale.io/s3-credentials-checksum annotation, and roll again whenever a credential key in the secret changes.
- This also happens while the managed etcd snapshot check fails, for example with a 403 on the snapshot bucket:
Readythen staysFalsewithSnapshotAccessFailed, but the broker and LFS proxy workloads are still applied, with every spec change (for example image, replicas and resources), not only credential changes. - The cluster metadata in etcd is still published to match, so a scale-down does not leave removed brokers as partition leaders.
- Earlier releases held back all broker and LFS proxy workload updates while the check failed.
- If you relied on a failed snapshot check to freeze broker updates, that no longer happens.
- Only a new cluster waits for that check before its brokers and LFS proxy are created.
S3 key values in the secret that are empty or only whitespace now count as unset everywhere, and other values are trimmed before use.
- Before, the operator condition, the operator snapshot check and the LFS proxy static keys trimmed them, but the brokers, the LFS proxy fallback keys and the AWS CLI in the etcd snapshot containers used the raw values, so a blank
KAFSCALE_S3_ACCESS_KEYorKAFSCALE_S3_SECRET_KEYnext to a validAWS_*pair gave the snapshot containers an empty or blank key, and a padded value failed to sign. - The snapshot containers now select the pair in their script instead of through
secretKeyRefenvironment variables: theKAFSCALE_S3_ACCESS_KEYandKAFSCALE_S3_SECRET_KEYpair when both keys are set, otherwise theAWS_*pair withAWS_SESSION_TOKEN. - Secrets with valid keys keep working with the same pair and token as before.
- A secret with the
AWS_*pair and blankKAFSCALE_S3_ACCESS_KEYandKAFSCALE_S3_SECRET_KEYkeys now reportsS3CredentialsasTruewithSecretKeysFound. - This changes the managed etcd pod template, which the earlier change in this release already does, so an upgrade from the last release still restarts the managed etcd pods once.
Whitespace here means space, tab, newline, carriage return, vertical tab and
form feed, in the Go code and in the snapshot container scripts alike. Unicode
spaces such as U+00A0 are kept as part of the value. In the last release the
operator snapshot check, the operator checks of the spec.s3 endpoint fields
and spec.s3.credentialsSecretRef, and the LFS proxy S3 key, region and
endpoint settings also trimmed Unicode spaces. They now keep them, so a value
padded with U+00A0 is used as written, as the etcd snapshot containers already
did. Remove such characters from the secret or the settings before upgrading.
Validation scope and rerun commands
The QA reported on bd1813ad2644c56736296f1e5dfff99901707ca8 in #298 passed
required release gates, a 30-minute soak (35,981 exact-byte-validated records),
three five-minute fuzz campaigns (28,794,235 executions), native Go/Python/Java/JS
LFS round trips, and Java Protobuf old-record decoding after registry restart.
Those are historical results on that revision, not a fresh certification of
the candidate head. The referenced /workspace/qa files are external evidence,
not repository files.
For this change, the module API comparison reproduced exactly the two reported
breaks on the original QA revision and found no incompatible exported changes
against v1.6.3 after the repair. The resolver unit tests and the real MinIO S3
overwrite probe failed against the original resolver and passed after the repair,
including valid-object and validation-disabled checks. The md5 and crc32
envelope cases in the Node, browser, Python and Java resolver tests failed
before the checksum_alg repair and pass after it. Focused broker, metadata
and proxy suites passed with the Go race detector. These results do not replace
the historical soak or establish general upgrade/rollback safety.
The focused regressions for this change are:
bash scripts/check_go_api_compatibility.sh
npm --prefix lfs-client-sdk/js test
npm --prefix lfs-client-sdk/js-browser test
(cd lfs-client-sdk/python && python3 -m pytest)
mvn -f lfs-client-sdk/java/pom.xml -Dtest=LfsResolverTest test
go test -race ./pkg/broker ./pkg/metadata ./cmd/broker ./cmd/proxy
The API script uses pinned golang.org/x/exp/cmd/apidiff, checks all exported
root-module packages, and compiles an external fixture with the exact published
Store method set plus comparable/map-key/unkeyed CoordinatorConfig usage.
Runtime tests exercise the legacy constructor, group-offset fallback, optional
capability dispatch, error propagation and broker OffsetFetch through an old
Store implementation. Lease and retention tests exercise the new constructor.
Run the real S3 overwrite probe against a disposable test S3 service with standard AWS credentials configured:
LFS_S3_ENDPOINT=http://127.0.0.1:9000 npm --prefix lfs-client-sdk/js run test:s3
It creates a unique bucket and runs one pass each for sha256, md5, crc32 and none envelopes.
- Each pass uploads 262,144 bytes, verifies exact valid bytes, overwrites the same key with a same-length first-byte mutation, requires default and explicit validation to reject it, and verifies disabled validation returns the altered bytes. It deletes its object and bucket afterward.
- This is a real S3 API test; it does not rerun the historical native Kafka round trip.
- Unit tests must fail against the old resolver and pass against the candidate.
- Before a merge recommendation, inspect every PR review/comment against final head and repeat focused corruption and compatibility regressions.
- Publishing and release version selection are separate decisions.
CodeQL port-conversion triage
The two retained go/incorrect-integer-conversion warnings follow
cmd/proxy/main.go:portFromAddr (strconv.Atoi) into the two envPort fallback
casts. Local SARIF has matching source/sink flows; GitHub alerts #62 and #66
are dismissed, which alone does not establish safety.
The fallback previously accepted values outside both the TCP range and, on a
64-bit machine, the int32 range. A value such as 4294967297 could narrow to
1 if it reached the cast. This is a real configuration-validation defect;
the evidence does not establish an exploitable vulnerability. Parsing address
ports as unsigned 16-bit values and bounding both configured and fallback ports
to 1..65535 makes the narrowing safe. Invalid fallback values use 9092.
Boundary tests cover 1, 65535, zero, negative, 65536, signed-32-bit edges,
wraparound candidates, malformed input and overflow. No alert was dismissed or
reclassified by this change.
Security dependency updates
Builds use Go 1.26.11. The operator’s default managed etcd image is
gcr.io/etcd-development/etcd:v3.6.14, pinned by digest, and snapshot tools
include etcdctl/etcdutl 3.6.14. Allow pulls from gcr.io when using the default.
An explicit KAFSCALE_OPERATOR_ETCD_IMAGE selects the supplied image instead;
update custom images to a patched etcd release as well. Docker Compose uses
etcd 3.5.33 to preserve its existing 3.5 data-format release line.