Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[r174] Ingester memory improvements by adjusting prealloc #4357

Merged
merged 1 commit into from
Nov 20, 2024
Merged

[r174] Ingester memory improvements by adjusting prealloc #4357

merged 1 commit into from
Nov 20, 2024

Conversation

joe-elliott
Copy link
Member

Backport f71c4c6 from #4344

What this PR does:

Adjusts the use of prealloc for some nice memory improvements. Two adjustments were made:

  • Remove the use of prealloc for trace ids. We were alloc'ing 500 bytes for every trace id and never recouping them in the sync pool.
  • Change the traces pool to use linear buckets to reduce wasted allocs. In larger clusters millions of these slices are alloc'ed per second. We will have no problem with bucket collisions. Bucket size was tuned using an internal cluster and a metric was added to help diagnose future issues.

Seeing ~30% working set reduction in ingesters.

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

@joe-elliott
Ingester memory improvements by adjusting prealloc (#4344)
ac49a31 
* remove trace ids

Signed-off-by: Joe Elliott <[email protected]>

* linear buckets

Signed-off-by: Joe Elliott <[email protected]>

* changelog

Signed-off-by: Joe Elliott <[email protected]>

* tuney tune

Signed-off-by: Joe Elliott <[email protected]>

* metric misses and increase pool size

Signed-off-by: Joe Elliott <[email protected]>

* lint

Signed-off-by: Joe Elliott <[email protected]>

---------

Signed-off-by: Joe Elliott <[email protected]>
(cherry picked from commit f71c4c6)
@joe-elliott joe-elliott added type/bug Something isn't working backport labels Nov 20, 2024
@joe-elliott joe-elliott merged commit 074973b into r174 Nov 20, 2024
21 checks passed
@joe-elliott joe-elliott deleted the backport-4344-to-r174 branch November 20, 2024 15:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mdisibio mdisibio mdisibio approved these changes

@mapno mapno Awaiting requested review from mapno mapno is a code owner

@yvrhdn yvrhdn Awaiting requested review from yvrhdn yvrhdn is a code owner

@zalegrala zalegrala Awaiting requested review from zalegrala zalegrala is a code owner

@electron0zero electron0zero Awaiting requested review from electron0zero electron0zero is a code owner

@ie-pham ie-pham Awaiting requested review from ie-pham ie-pham is a code owner

@stoewer stoewer Awaiting requested review from stoewer stoewer is a code owner

@javiermolinar javiermolinar Awaiting requested review from javiermolinar javiermolinar is a code owner

Assignees
No one assigned
Labels
backport type/bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@joe-elliott @mdisibio