Kafka output: Attempt to produce message larger than configured Producer.MaxMessageBytes

[jrametta]

Hi! We have some network devices with a large number of interfaces and collecting certain metrics produce rather large messages to the kafka output. Some fail to output due to...

gnmic-1 gnmic 2026/04/24 17:47:03.543846 [kafka_output:edge-kafka] worker-0 failed to send a kafka msg to topic '<topic>': kafka: invalid configuration (Attempt to produce message larger than configured Producer.MaxMessageBytes: 1952591 > 1048576)

This is presumably due to Sarama 1MB MaxMessageBytes default. I did not see this parameter exposed through gnmic. Might be something I can quickly update if interested in a PR, but wanted to check if there was another existing solution to this problem that would limit message size (kafka output split-events might fix this by turning one message into 1000+, merge processor or similar may help to reduce (but not limit) size but may require some changes to upstream services).

Any other ideas? Thanks!

[karimra]

A PR is fine, but you will also have to update the max msg size in your broker as well as on the consume side.
Note that higher payload size means lower throughput.
I believe the better solution is to split your payload using split events or by changing the way your subscribe to the targets.