Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Batching with preserved offset ordering #695

Merged
merged 30 commits into from
Nov 7, 2024
Merged

Batching with preserved offset ordering #695

merged 30 commits into from
Nov 7, 2024

Conversation

dfakhritdinov
Copy link
Contributor

Based on #687, so kudos going to @mr-git

Main difference, compared with original PR, is in Kafka offset ordering that is preserved in resulted batches.

@coveralls
Copy link

coveralls commented Nov 4, 2024 •
edited
Loading

Pull Request Test Coverage Report for Build 11704780715

Details

  • 46 of 46 (100.0%) changed or added relevant lines in 1 file are covered.
  • 12 unchanged lines in 7 files lost coverage.
  • Overall coverage decreased (-0.08%) to 83.093%
Files with Coverage Reduction New Missed Lines %
core/src/main/scala/com/evolutiongaming/kafka/journal/PayloadType.scala 1 88.24%
core/src/main/scala/com/evolutiongaming/kafka/journal/LogFromAkka.scala 1 28.57%
core/src/main/scala/com/evolutiongaming/kafka/journal/util/TemporalHelper.scala 1 61.11%
replicator/src/main/scala/com/evolutiongaming/kafka/journal/replicator/ReplicateRecords.scala 1 80.0%
journal/src/main/scala/com/evolutiongaming/kafka/journal/Journals.scala 1 91.82%
replicator/src/main/scala/com/evolutiongaming/kafka/journal/replicator/DistributeJob.scala 2 79.35%
journal/src/main/scala/com/evolutiongaming/kafka/journal/HeadCache.scala 5 82.93%
Totals Coverage Status
Change from base Build 11689488796: -0.08%
Covered Lines: 3170
Relevant Lines: 3815
💛 - Coveralls

@dfakhritdinov dfakhritdinov requested a review from mr-git November 5, 2024 15:35
mr-git and others added 27 commits November 6, 2024 15:19
@mr-git
recover new test-cases
1993fa3
@mr-git
add some docs
ebcbf4e
@mr-git
change batching logic and adjust unit-test (to use offset of last a…
1eb2a38 
…ction per kind, not `offset` of all records, including `Mark`'s)
@mr-git
adjust other TopicReplicatorSpec to not expect Mark in batch to a…
b1e08e9 
…dvance offset
@mr-git
cleanup
a291815
@mr-git
improve readability in BatchSpec by using to for deletes batch …
f20b0f0 
…instead of `seqNr`
@mr-git
drop first append, if there is at least one more and delete will …
d64de55 
…discard it
@mr-git
cleanup
45a209b
@mr-git
cleanup
f17911f
@mr-git
improve docs for Batch
df6da92
@mr-git
switch to Vector in Batch
b5c3fd3
@mr-git
add benchmark :(
7bb4164
@mr-git
add benchmark for "reversed" approach - looks much better :)
facff4b
@mr-git
cleanup
0c9b7df
@mr-git
use .flatMap instead of explicit matching with case None => None
4f6130b
@mr-git
add comment on why we have to keep at least one append action when …
ab9ba61 
…`delete` follows
@mr-git @rtar
Update replicator/src/main/scala/com/evolutiongaming/kafka/journal/re…
1920880 
…plicator/Batch.scala

Co-authored-by: Ruslans Tarasovs <[email protected]>
@mr-git
attempt to improve the style - move ActionRecord handling inside `S…
1e48ec4 
…tate`
@mr-git
provide documentation-comment for handle function
503adf8
@mr-git
fully encapsulate the batch-building in State
45266ab
@mr-git
add more details with assumptions
6e58d1e
@mr-git
cleanup benchmarks
8ef3d98
@mr-git
do not allow accidental delete(max)
cb9a32a
@mr-git
adjust documentation in benchmark
e9c2230
@mr-git
fix typos
942cfd9
@mr-git
revert binary incompatible
27c81c4
@mr-git
provide more details in the documentation
a61adde
@mr-git mr-git force-pushed the df/batch-preserve-offsets branch from 2726d4f to cb184a9 Compare November 6, 2024 13:42
mr-git
mr-git approved these changes Nov 7, 2024
View reviewed changes
Copy link
Contributor

@mr-git mr-git left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not see, how origin and version from "second" delete action could affect us negatively

@mr-git mr-git mentioned this pull request Nov 7, 2024
Closed
@dfakhritdinov dfakhritdinov merged commit 75ac715 into master Nov 7, 2024
2 of 3 checks passed
@dfakhritdinov dfakhritdinov deleted the df/batch-preserve-offsets branch November 7, 2024 12:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mr-git mr-git mr-git approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@dfakhritdinov @coveralls @mr-git