Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

LRU caching for S3 fragments #5061

Open
malhotrashivam opened this issue Jan 23, 2024 · 0 comments
Open

LRU caching for S3 fragments #5061

malhotrashivam opened this issue Jan 23, 2024 · 0 comments
Assignees
@malhotrashivam
Labels
feature request New feature or request parquet Related to the Parquet integration s3
Milestone
Backlog

Comments

@malhotrashivam
Copy link
Contributor

malhotrashivam commented Jan 23, 2024 •
edited
Loading

Our current code for reading files stored in S3 uses a simple modulo based caching where fragment i is stored in slot i%cache_size and will stay there until it is evicted by a more recent fragment. We want to experiment with LRU caching and understand the impact on performance.

Found during #4972
@malhotrashivam malhotrashivam added feature request New feature or request parquet Related to the Parquet integration s3 labels Jan 23, 2024
@malhotrashivam malhotrashivam added this to the 5. Backlog milestone Jan 23, 2024
@malhotrashivam malhotrashivam self-assigned this Jan 23, 2024
@malhotrashivam malhotrashivam mentioned this issue Jan 23, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@malhotrashivam malhotrashivam

Labels
feature request New feature or request parquet Related to the Parquet integration s3
Projects
None yet
Milestone
Backlog
Development

No branches or pull requests
1 participant
@malhotrashivam