Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

release-21.1: kvserver: use ClearRawRange to truncate very large Raft logs #75980

Merged

Conversation

erikgrinaker
Copy link
Contributor

Backport 1/1 commits from #75793.

/cc @cockroachdb/release

Release justification: possible node crash loop.


Raft log truncation was done using point deletes in a single Pebble
batch. If the number of entries to truncate was very large, this could
result in overflowing the Pebble batch, causing the node to panic. This
has been seen to happen e.g. when the snapshot rate was set very low,
effectively stalling snapshot transfers which in turn held up log
truncations for extended periods of time.

This patch uses a Pebble range tombstone if the number of entries to
truncate is very large (>100k). In most common cases, point deletes are
still used, to avoid writing too many range tombstones to Pebble.

Release note (bug fix): Fixed a bug which could cause nodes to crash
when truncating abnormally large Raft logs.

Raft log truncation was done using point deletes in a single Pebble
batch. If the number of entries to truncate was very large, this could
result in overflowing the Pebble batch, causing the node to panic. This
has been seen to happen e.g. when the snapshot rate was set very low,
effectively stalling snapshot transfers which in turn held up log
truncations for extended periods of time.

This patch uses a Pebble range tombstone if the number of entries to
truncate is very large (>100k). In most common cases, point deletes are
still used, to avoid writing too many range tombstones to Pebble.

Release note (bug fix): Fixed a bug which could cause nodes to crash
when truncating abnormally large Raft logs.
@erikgrinaker erikgrinaker requested a review from a team as a code owner February 3, 2022 22:28
@erikgrinaker erikgrinaker self-assigned this Feb 3, 2022
@cockroach-teamcity
Copy link
Member

This change is Reviewable

@erikgrinaker erikgrinaker changed the title kvserver: use ClearRawRange to truncate very large Raft logs release-21.1: kvserver: use ClearRawRange to truncate very large Raft logs Feb 3, 2022
Copy link
Member

@tbg tbg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know the pebble folks feel pretty good about rangedels in 21.2+, but is there something on 21.1 that makes them pause? I assume not, and also the tombstones are only used when you'd likely have a really bad time without them anyway.

@erikgrinaker
Copy link
Contributor Author

Yeah, I thought 21.1 was probably recent enough that it's fine, but worth checking.

@cockroachdb/storage Any significant issues with range tombstones in 21.1?

@itsbilal
Copy link
Member

itsbilal commented Feb 9, 2022

@cockroachdb/storage Any significant issues with range tombstones in 21.1?

None, relative to 21.2 at least. This change should be good on 21.1 as well. There's cockroachdb/pebble#1070 but that covers both versions and is a more general issue.

@erikgrinaker
Copy link
Contributor Author

Thanks @itsbilal!

@erikgrinaker erikgrinaker merged commit 0490498 into cockroachdb:release-21.1 Feb 9, 2022
@erikgrinaker erikgrinaker deleted the backport21.1-75793 branch February 14, 2022 12:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants