Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[connector/spanmetrics] Fix memory leak #28847

Merged

Conversation

bripkens
Copy link
Contributor

@bripkens bripkens commented Nov 1, 2023

Why

The spanmetrics connector has a memory leak that occurs when the cumulative temporality is used. The connectorImp#resourceMetrics map is only ever cleaned up in delta temporality.

What

Turn connectorImp#resourceMetrics into a LRU cache with a maximum size. To correctly handle metric resets we also introduce a start timestamp per resourceMetric instance.

References

Fixes #27654

@bripkens bripkens requested review from a team and TylerHelmuth November 1, 2023 12:13
@bripkens bripkens force-pushed the fix-spanmetrics-memory-leak branch from 8137649 to cd81af5 Compare November 1, 2023 12:13
@bripkens bripkens force-pushed the fix-spanmetrics-memory-leak branch from cd81af5 to 8dc2eac Compare November 1, 2023 12:14
@bripkens bripkens changed the title fix: spanmetrics memory leak [connector/spanmetrics] Fix memory leak Nov 1, 2023
@bripkens bripkens force-pushed the fix-spanmetrics-memory-leak branch from 8dc2eac to 2c2baa6 Compare November 1, 2023 12:17
@bripkens bripkens force-pushed the fix-spanmetrics-memory-leak branch from 2c2baa6 to c64fb77 Compare November 1, 2023 15:07
Copy link
Contributor

@albertteoh albertteoh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks very much!

@@ -74,3 +74,17 @@ func (c *Cache[K, V]) Purge() {
c.lru.Purge()
c.RemoveEvictedItems()
}

// ForEach iterates over all the items within the cache, as well as the evicted items (if any).
func (c *Cache[K, V]) ForEach(fn func(k K, v V)) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice

@bripkens bripkens force-pushed the fix-spanmetrics-memory-leak branch from c64fb77 to e929f8b Compare November 13, 2023 13:50
Copy link

@mfilipe mfilipe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't have knowledge to read the code, but this PR seems promises to solve a problem that I have.

@JaredTan95
Copy link
Member

need to fix https://github.com/open-telemetry/opentelemetry-collector-contrib/actions/runs/7083411340/job/19275795975#step:8:148:

Error: ./connector_test.go:901:146: not enough arguments in call to newConnectorImp
	have (*testing.T, consumertest.Consumer, *string, func() HistogramConfig, func() ExemplarsConfig, string, *zap.Logger, nil)
	want (*testing.T, consumer.Metrics, *string, func() HistogramConfig, func() ExemplarsConfig, func() EventsConfig, string, *zap.Logger, *clock.Ticker, ...string) (typecheck)

# Why

The `spanmetrics` connector has a memory leak that occurs when the
cumulative temporality is used. The `connectorImp#resourceMetrics` map
is only ever cleaned up in delta temporality.

# What

Turn `connectorImp#resourceMetrics` into a LRU cache with a maximum
size. To correctly handle metric resets we also introduce a start
timestamp per `resourceMetric` instance.

# References

Fixes open-telemetry#27654

Co-authored-by: Jared Tan <[email protected]>
@bripkens bripkens force-pushed the fix-spanmetrics-memory-leak branch from 3eaa2aa to 7ff7b54 Compare December 5, 2023 08:28
@bripkens
Copy link
Contributor Author

bripkens commented Dec 5, 2023

done @JaredTan95

@TylerHelmuth TylerHelmuth merged commit e254878 into open-telemetry:main Dec 5, 2023
83 checks passed
@github-actions github-actions bot added this to the next release milestone Dec 5, 2023
@bripkens bripkens deleted the fix-spanmetrics-memory-leak branch December 5, 2023 18:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

spanmetrics connector has memory leak
8 participants