Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

What is recommended configuration to use pebble as queue? #4158

Open
kanekv opened this issue Nov 12, 2024 · 2 comments
Open

What is recommended configuration to use pebble as queue? #4158

kanekv opened this issue Nov 12, 2024 · 2 comments

Comments

@kanekv
Copy link

kanekv commented Nov 12, 2024

E.g. what would be an equivalent of this: https://github.com/facebook/rocksdb/wiki/Implement-Queue-Service-Using-RocksDB

Values in my case should be mostly short-lived, and this strictly applies:

In these services, from each queue, new items are added with a higher sequence ID
 and removed from the smallest sequence ID. 
 Users usually read from a queue in sequence ID increasing order.

Also it will delete read items immediately.

Jira issue: PEBBLE-300

@jbowens
Copy link
Collaborator

jbowens commented Nov 12, 2024

Much of the advice in the RocksDB document applies to Pebble as well. The 'Old Item Deletion Problem' is good advice.

On the subject of 'Reclaiming space of deleted items faster,' you may want to use DeleteRange to batch delete swaths of the tail of the queue, if you can delay the deletion. Wide range deletion tombstones are cheaper to skip over than many individual point tombstones.

@kanekv
Copy link
Author

kanekv commented Nov 13, 2024

@jbowens what would be a good configuration for the base level and memtables? I assume it will benefit to make them as big as possible if all items are relatively short-lived?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Community
Development

No branches or pull requests

3 participants