Skip to content

Commit

Permalink
Add documentation
Browse files Browse the repository at this point in the history
  • Loading branch information
erikvanoosten committed Jan 12, 2025
1 parent 442c970 commit ef2d73d
Showing 1 changed file with 35 additions and 2 deletions.
37 changes: 35 additions & 2 deletions docs/preventing-duplicates.md
Original file line number Diff line number Diff line change
@@ -26,6 +26,39 @@ a partition.
For this to work correctly, your program must process a chunk of records within max-rebalance-duration. The clock
starts the moment the chunk is pushed into the stream and ends when the commits for these records complete.

For more information see the scaladocs in `ConsumerSettings`, read the description of
[pull request #1098](https://github.com/zio/zio-kafka/pull/1098) that introduced this feature, or watch the presentation
In addition, your program must commit the offsets of consumed records. The most straightforward way is to commit to the
Kafka brokers. This is done by calling `.commit` on the offset of consumed records (see the consumer documentation).
However, there are more options: external commits and transactional producing.

### Commit to an external system

When you commit to an external system (e.g. by writing to a relational database) the zio-kafka consumer needs to know
about those commits before it can work in rebalance-safe-commits mode. Inform zio-kafka about external commits by
invoking method `Consumer.registerExternalCommits(offsetBatch: OffsetBatch)` (available since zio-kafka 2.9.2).

Here is what this could look like:

```scala
import zio.kafka.consumer._

consumer.plainStream(Subscription.topics("topic2000"), Serde.string, Serde.string)
.mapZIO { record =>
database.store(record.offset) *> // <-- the external commit
consumer.registerExternalCommits(OffsetBatch(record.offset))
}
.runDrain
```

### Commit with a transactional producer

Although transactional producing is possible with zio-kafka, it is not easy and the code is very messy (see
`ConsumerSpec` for an example). Transactional producing can not be used in combination with rebalance-safe-commits mode.

Zio-kafka v3.0.0 will make transactional producing much easier.

## More information

There is more information in the scaladocs of `ConsumerSettings` and the description of
[pull request #1098](https://github.com/zio/zio-kafka/pull/1098) that introduced this feature.
You can also watch the presentation
[Making ZIO-Kafka Safer And Faster](https://www.youtube.com/watch?v=MJoRwEyyVxM). The relevant part starts at 10:24.

0 comments on commit ef2d73d

Please sign in to comment.