[QUESTION] Supporting Apache Kafka 4.0

[nhaq-confluent]

Hello!

Apache Kafka 4.0 is introducing KIP-896, a KIP which deprecates client protocol API versions. For the exact list of deprecated versions, please refer to the KIP: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838251#KIP896:RemoveoldclientprotocolAPIversionsinKafka4.0-PublicInterfaces

Since this library uses aiokafka according to the readme, I wanted to raise attention to an issue that was found in aiokafka's support of the metadata request key as suggested in this issue: aio-libs/aiokafka#1085.

If indeed there is a problem, that would mean Faust would not work with AK 4.0 brokers. It seems otherwise Faust is configured to support the latest API version common across both client and broker https://faust-streaming.github.io/faust/userguide/settings.html#broker-api-version

[wbarnha]

Thanks for the detailed write-up. I dug into this against a real broker, and the short answer is: faust works with Apache Kafka 4.0 when using a current aiokafka.

What I tested

I ran faust's live-broker integration suite against a real Apache Kafka 4.0.0 broker (KRaft, metadata.version 4.0-IV3), with aiokafka 0.14.0 and confluent-kafka 2.x:

• raw aiokafka produce → consume round-trip — ✅
• raw confluent-kafka produce → consume round-trip — ✅
• a faust App with an agent: consumer-group join, fetch, and offset commit (through the real Consumer.commit() / _new_offset path) — ✅

All green. That App path exercises the metadata / FindCoordinator / OffsetFetch / OffsetCommit APIs that KIP-896 tightens, so with the default broker_api_version="auto" negotiation aiokafka is selecting request versions a 4.0 broker accepts — i.e. the Metadata-version concern from aio-libs/aiokafka#1085 does not surface with a current aiokafka. (If it did, producer.start() / consumer.start() would fail outright with an UnsupportedVersion error rather than round-tripping.)

Making it a continuous guarantee

#714 adds Kafka 4.0.0 to faust's CI integration matrix (alongside 3.8.1), so both transport drivers are verified against a 4.x broker on every change instead of a one-off check. faust also already picked up broader aiokafka-4.0-readiness work in v0.12.0 (e.g. guarding metadata/admin calls by the negotiated protocol-API version, #682).

Recommendation

If you're on Kafka 4.0, use a recent aiokafka. faust requires aiokafka>=0.10.0 and does not cap the upper bound, so pip install -U aiokafka gets you a 4.0-compatible client; I verified 0.14.0. Very old aiokafka releases that predate its Metadata-version handling could still hit KIP-896 against a 4.0 broker.

Happy to help chase anything down if you can share a specific failure (broker version, aiokafka version, and the traceback), but on current versions this checks out. This can probably be closed once #714 lands — leaving that to the maintainers.

---

Generated by Claude Code