Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf: consider atomic compaction units in elision-only compaction heuristic #1083

Closed
jbowens opened this issue Mar 5, 2021 · 1 comment
Closed

Comments

@jbowens
Copy link
Collaborator

jbowens commented Mar 5, 2021

If a file that is eligible for an elision-only compaction contains a single user key, we may need to pull adjacent files into the compaction to ensure we always compact atomic compaction units together. This may dramatically change the space/write IO tradeoff, resulting in rewriting substantially more data than we drop.

This abbreviated log line was taken from a node affected by #1070.

vendor/github.com/cockroachdb/pebble/compaction.go:1410 ⋮ [nXXX] [JOB XXXXX] compacted L6 [046035] (168 M) + L6 [] (0 B) -> L6 [046647 046648 046649 046650 046651 046652 046653 046654 046655 046656 046657 046658 046659 046660 046661 046662 046663 046664 046667 046669 046671 046672 046673 046674 046675 046676 046677 046678 046679 046680 046681 046682 046683 046684 046685 046686 046687 046688 046689 046690 046691 046692] (166 M), in 3.5s, output rate 48 M/s
@jbowens
Copy link
Collaborator Author

jbowens commented May 17, 2021

This was actually #1097.

@jbowens jbowens closed this as completed May 17, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant