Ingester memory improvements by adjusting prealloc

[joe-elliott]

What this PR does:

Adjusts the use of prealloc for some nice memory improvements. Two adjustments were made:

• Remove the use of prealloc for trace ids. We were alloc'ing 500 bytes for every trace id and never recouping them in the sync pool.
• Change the traces pool to use linear buckets to reduce wasted allocs. In larger clusters millions of these slices are alloc'ed per second. We will have no problem with bucket collisions. Bucket size was tuned using an internal cluster and a metric was added to help diagnose future issues.

Seeing ~30% working set reduction in ingesters.

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[zalegrala]

Looks good to me. The new metric ought to be interesting.

[zalegrala]

I suppose its a trad-off here to alloc or do the math twice.

[joe-elliott]

this doesn't really "alloc" in any way we are concerned about. we're always conscious of allocating memory that escapes to the heap. everything else is cheap. whether this is stored on the stack of just stays in a register it would likely be undetectable from a tempo perf standpoint.

[zalegrala]

Does this alloc, or is the compiler smart enough to see its only used once?

[joe-elliott]

you're really testing my knowledge of compilers :). my guess is this value would never leave a register

[mapno]

You can do escape analysis in Go with go build -gcflags "-m" <file>. It does not escape :P

[github-actions]

This PR must be merged before a backport PR will be created.

[github-actions]

This PR must be merged before a backport PR will be created.

[github-actions]

The backport to r174 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new branch
git switch --create backport-4344-to-r174 origin/r174
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x f71c4c6ccbd81b75826b434d6bf2b82174a757c5

When the conflicts are resolved, stage and commit the changes:

git add . && git cherry-pick --continue

If you have the GitHub CLI installed:

# Push the branch to GitHub:
git push --set-upstream origin backport-4344-to-r174
# Create the PR body template
PR_BODY=$(gh pr view 4344 --json body --template 'Backport f71c4c6ccbd81b75826b434d6bf2b82174a757c5 from #4344{{ "\n\n---\n\n" }}{{ index . "body" }}')
# Create the PR on GitHub
echo "${PR_BODY}" | gh pr create --title '[r174] Ingester memory improvements by adjusting prealloc' --body-file - --label 'type/bug' --label 'backport' --base r174 --milestone r174 --web

Or, if you don't have the GitHub CLI installed (we recommend you install it!):

# Push the branch to GitHub:
git push --set-upstream origin backport-4344-to-r174

# Create a pull request where the `base` branch is `r174` and the `compare`/`head` branch is `backport-4344-to-r174`.

# Remove the local backport branch
git switch main
git branch -D backport-4344-to-r174