Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve performance of corr function #13549

Open
Tracked by #13548
alamb opened this issue Nov 24, 2024 · 1 comment · May be fixed by #13581
Open
Tracked by #13548

Improve performance of corr function #13549

alamb opened this issue Nov 24, 2024 · 1 comment · May be fixed by #13581
Assignees
Labels
enhancement New feature or request

Comments

@alamb
Copy link
Contributor

alamb commented Nov 24, 2024

Is your feature request related to a problem or challenge?

The corr function is used in the h20 benchmark and is quite slow.

https://github.com/apache/datafusion/blob/main/datafusion/functions-aggregate/src/correlation.rs

It would be great to make it faster so our performance in the H2o benchmark improves

Describe the solution you'd like

See details on #13548

  1. Add a benchmark for corr function
  2. Improve performance of corr function (likely by implementing GroupsAccumulator)

Describe alternatives you've considered

No response

Additional context

Used here in @MrPower's benchmark:

https://github.com/MrPowers/mrpowers-benchmarks/blob/0b586a0657d7f6cfd55d89508e15b95e79bd4010/benchmarks/datafusion_h2o_groupby_queries.py#L37

@alamb alamb added the enhancement New feature or request label Nov 24, 2024
@2010YOUY01
Copy link
Contributor

take

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants