Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve kafka persistence #229

Merged
merged 17 commits into from
Sep 23, 2021
Merged

Improve kafka persistence #229

merged 17 commits into from
Sep 23, 2021

Conversation

terjokhin
Copy link
Contributor

No description provided.

@Z1kkurat
Copy link
Contributor

Didn't even notice this already existed, thanks @Daron666!
Short list of the changes in this PR:

  • using new RebalanceConsumer1 from skafka
  • some changes to core functionality by @nikitapecasa, I will take a look at them once more to ensure everything is in order and nothing more need to be done
  • my rework of the current implementation of kafka-based snapshot persistence - the old one was reading the snapshots in the overriden PartitionFlowOf, now it reads snapshots with caching only once when KeysOf.all is invoked
  • the large integration test mostly by @nikitapecasa ensuring that snapshot recovery works properly

@nikitapecasa nikitapecasa changed the title WIP: Simplify kafka persistence Improve kafka persistence Sep 23, 2021
@nikitapecasa nikitapecasa merged commit 082f0c3 into master Sep 23, 2021
@nikitapecasa
Copy link
Contributor

Also this PR brings

  /** Safeguards a TopicFlow's resource to address a race between processing of messages and release of the resource.
    *
    * It provides following safeties in case of a fiber cancellation and resource's release:
    *
    *  - processing of current kafka messages won't be cancelled (TopicFlow.apply)
    *  - pending offsets would be committed
    *  - accumulated state would be persisted
    *  - resource would be released only after previous steps are completed
    *  - if resource is released before start of messages' processing, then it does nothing (no message processing, no commits, no persists)
    */

@nikitapecasa
Copy link
Contributor

Also during working on this PR it was discovered that current graceful shutdown sequence can be improved #256

@Z1kkurat Z1kkurat deleted the simplify-kafka-persistence branch September 23, 2021 11:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants