Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

release-24.2: stats: use no-op DatumAlloc when decoding EncDatums #137061

Merged
merged 2 commits into from
Dec 11, 2024

Conversation

blathers-crl[bot]
Copy link

@blathers-crl blathers-crl bot commented Dec 9, 2024

Backport 2/2 commits from #136780 on behalf of @yuzefovich.

/cc @cockroachdb/release


tree: make nil DatumAlloc act as a no-op

This commit makes so that nil DatumAlloc value makes the alloc struct a no-op, i.e. each datum type passed by value results in a separate allocation. This can be useful when the callers will keep only a subset of all allocated datums.

Release note: None

stats: use no-op DatumAlloc when decoding EncDatums

This commit fixes a bounded memory leak that could previously occur due to usage of the DatumAlloc when decoding EncDatums into tree.Datums during table stats collection. We have two use cases for decoded tree.Datums:

  • we need it when adding all rows into the sketch (we compute the datum's fingerprint ("hash") to use in the distinct count estimation). This usage is very brief and we don't need the decoded datum afterwards.
  • we also need it when sampling some rows when we decide to keep a particular row. In this case, the datum is needed throughout the whole stats collection job (or until it's replaced by another sample) for the histogram bucket computation.

The main observation is that the DatumAlloc allocates datums in batches of 16 objects, and even if at least one of the objects is kept by the sample, then all others from the same batch are as well. We only perform memory accounting for the ones we explicitly keep, yet others would be considered live by the Go runtime, resulting in a bounded memory leak. This behavior has been present since forever, but it became more problematic in 24.2 release with the introduction of dynamically computing the sample size. To go around this problematic behavior this commit uses nil DatumAlloc which makes it so that every decoded datum incurs a separate allocation. This will have a minor performance hit and increase in total number of allocations, but at least most of them should be short-lived. Alternatively, we could use an "operating" DatumAlloc during fingerprinting and a no-op during the sampling, but we'd need to explicitly nil out the decoded datum after fingerprinting which would result in decoding some datums twice.

Fixes: #136394.

Release note: None


Release justification: bug fix.

@blathers-crl blathers-crl bot requested review from a team as code owners December 9, 2024 23:28
@blathers-crl blathers-crl bot force-pushed the blathers/backport-release-24.2-136780 branch from db8cd89 to e919c08 Compare December 9, 2024 23:28
@blathers-crl blathers-crl bot removed the request for review from a team December 9, 2024 23:28
@blathers-crl blathers-crl bot added the blathers-backport This is a backport that Blathers created automatically. label Dec 9, 2024
@blathers-crl blathers-crl bot requested a review from DrewKimball December 9, 2024 23:28
@blathers-crl blathers-crl bot added the O-robot Originated from a bot. label Dec 9, 2024
Copy link
Author

blathers-crl bot commented Dec 9, 2024

Thanks for opening a backport.

Please check the backport criteria before merging:

  • Backports should only be created for serious
    issues
    or test-only changes.
  • Backports should not break backwards-compatibility.
  • Backports should change as little code as possible.
  • Backports should not change on-disk formats or node communication protocols.
  • Backports should not add new functionality (except as defined
    here).
  • Backports must not add, edit, or otherwise modify cluster versions; or add version gates.
  • All backports must be reviewed by the owning areas TL. For more information as to how that review should be conducted, please consult the backport
    policy
    .
If your backport adds new functionality, please ensure that the following additional criteria are satisfied:
  • There is a high priority need for the functionality that cannot wait until the next release and is difficult to address in another way.
  • The new functionality is additive-only and only runs for clusters which have specifically “opted in” to it (e.g. by a cluster setting).
  • New code is protected by a conditional check that is trivial to verify and ensures that it only runs for opt-in clusters. State changes must be further protected such that nodes running old binaries will not be negatively impacted by the new state (with a mixed version test added).
  • The PM and TL on the team that owns the changed code have signed off that the change obeys the above rules.
  • Your backport must be accompanied by a post to the appropriate Slack
    channel (#db-backports-point-releases or #db-backports-XX-X-release) for awareness and discussion.

Also, please add a brief release justification to the body of your PR to justify this
backport.

@blathers-crl blathers-crl bot added the backport Label PR's that are backports to older release branches label Dec 9, 2024
@cockroach-teamcity
Copy link
Member

This change is Reviewable

@yuzefovich yuzefovich removed the request for review from DrewKimball December 9, 2024 23:43
This commit makes so that `nil` `DatumAlloc` value makes the alloc
struct a no-op, i.e. each datum type passed by value results in
a separate allocation. This can be useful when the callers will keep
only a subset of all allocated datums.

Release note: None
This commit fixes a bounded memory leak that could previously occur due
to usage of the DatumAlloc when decoding EncDatums into `tree.Datum`s
during table stats collection. We have two use cases for decoded
`tree.Datum`s:
- we need it when adding _all_ rows into the sketch (we compute the
datum's fingerprint ("hash") to use in the distinct count estimation).
This usage is very brief and we don't need the decoded datum afterwards.
- we also need it when sampling _some_ rows when we decide to keep
a particular row. In this case, the datum is needed throughout the whole
stats collection job (or until it's replaced by another sample) for the
histogram bucket computation.

The main observation is that the DatumAlloc allocates datums in batches
of 16 objects, and even if at least one of the objects is kept by the
sample, then all others from the same batch are as well. We only perform
memory accounting for the ones we explicitly keep, yet others would be
considered live by the Go runtime, resulting in a bounded memory leak.
This behavior has been present since forever, but it became more
problematic in 24.2 release with the introduction of dynamically
computing the sample size. To go around this problematic behavior this
commit uses `nil` DatumAlloc which makes it so that every decoded datum
incurs a separate allocation. This will have a minor performance hit and
increase in total number of allocations, but at least most of them
should be short-lived. Alternatively, we could use an "operating"
DatumAlloc during fingerprinting and a no-op during the sampling, but
we'd need to explicitly nil out the decoded datum after fingerprinting
which would result in decoding some datums twice.

Release note: None
@yuzefovich yuzefovich force-pushed the blathers/backport-release-24.2-136780 branch from e919c08 to 940ac56 Compare December 10, 2024 21:04
@mgartner mgartner merged commit a3168fb into release-24.2 Dec 11, 2024
19 of 20 checks passed
@mgartner mgartner deleted the blathers/backport-release-24.2-136780 branch December 11, 2024 14:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport Label PR's that are backports to older release branches blathers-backport This is a backport that Blathers created automatically. O-robot Originated from a bot.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants