Unexpected Offset Increment of 2 Per Message in Single-Partition Kafka Topic

[vcanuel]

Description:

Hi,

I’ve encountered an issue where, after writing messages to a Kafka topic with a single partition, the offsets increment by 2 instead of the expected increment of 1. For example, the sequence of offsets is 0, 2, 4, etc.
Is this normal ?

Basic information

• Axon Framework version: 4.10.2

• JDK version: 21

• Kafka Extension version: 4.10.0

• My application properties :

axon.kafka.publisher.confirmation-mode=transactional
axon.kafka.bootstrap-servers=localhost:9092
axon.kafka.message-converter-mode=default
axon.kafka.client-id=anonymous
axon.kafka.default-topic=axon-events
axon.kafka.publisher.confirmation-mode=wait_for_ack
axon.kafka.producer.transaction-id-prefix=axontx
axon.kafka.producer.retries=1
axon.kafka.producer.event-processor-mode=subscribing
axon.kafka.producer.acks=all

Steps to Reproduce:

1. Produce messages to a Kafka topic configured with a single partition.
2. Observe the offsets assigned to each message.

Expected Behavior:

Offsets should increment by 1 for each new message (e.g., 0, 1, 2, 3, …).

Actual Behavior:

Offsets increment by 2 for each new message (e.g., 0, 2, 4, …).

[smcvb]

Thanks for filing this concern with us, @vcanuel.
Although this might be asking a lot, I am going to regardless: would you mind trying to reproduce this offset-issue with the example present in this repository?
The example resides in the "kafka-axon-example" folder, by the way.
Sharing a reproducible will help us quite a bit to ensure nothing gets lost in translation.

[vcanuel]

Hi @smcvb,

I have run the Axon example without any issues.

I have pinpointed the problem: it occurs when the confirmation mode is set to transactional:

  kafka:
    publisher:
      confirmation-mode: transactional
    clientid: kafka-axon-example
    producer:
      retries: 1
      bootstrap-servers: localhost:29092
      transaction-id-prefix: axontx

Is this normal ?

[smcvb]

@vcanuel, first and foremost, my sincere apology for the late reply here. The last two weeks where more hectic then I expected.

Concerning your finding on the transaction mode, that's a good detail to have. Note that Axon Framework doesn't do any offset or transactional logic.
The offset is purely calculated by Kafka, and the transaction mode setting is passed along to the Producer that this extension constructs for you.

Nonetheless, I did a little searching and uncovered this issue on the confluent-kafka-python repository.
This issue has the title "Transactional Produce Increments Offset by 2," thus aligning with the behavior you've found.
Furthermore, this issue got closed a day later by the one that created it, with the following comment:

Closing because it seems this is an inherently necessary artifact of transactional commits, and won't be fixed, as per other similar issues.

This led me to do some further research, finding this Stack Overflow post.
The following comment is rather important in there:

That's the way it's designed.
Kafka logs are immutable so an extra "slot" is used at the end of the transaction to indicate whether the transaction was committed or rolled back.
This allows consumers with read_committed isolation level to skip over rolled-back transactions.

In hindsight I should've triggered on the transaction mode you selected. However, I haven't had a lot of recent play time with Kafka, so this nugget of info got lost to me.
Similarly as with the late reply, my apologies for that.

Nonetheless, I am confident we can close this issue as resolved through the above mentioned information.
If questions remain, be sure to add another comment, @vcanuel!

[vcanuel]

Hi @smcvb ,

Thanks for the detailed and thorough explanation!

Regards,

Vikncent.