Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] Support parquet row group skipping in Polars physical engine #16257

Open
beckernick opened this issue Jul 11, 2024 · 2 comments
Open

[FEA] Support parquet row group skipping in Polars physical engine #16257

beckernick opened this issue Jul 11, 2024 · 2 comments
Assignees
Labels
cudf.polars Issues specific to cudf.polars feature request New feature or request pylibcudf Issues specific to the pylibcudf package Python Affects Python cuDF API.

Comments

@beckernick
Copy link
Member

Polars provides the ability to skip row groups to avoid reading unnecessary data. I believe we aren't currently taking advantage of this the backend, but we could be (discussed offline).

@beckernick beckernick added feature request New feature or request cudf.polars Issues specific to cudf.polars labels Jul 11, 2024
@lithomas1 lithomas1 self-assigned this Jul 11, 2024
@wence-
Copy link
Contributor

wence- commented Sep 3, 2024

This will be done once we hook up the ast expressions in the parquet reader.

@vyasr
Copy link
Contributor

vyasr commented Nov 14, 2024

This currently works, but once we merge #16944 it will be disabled again since we don't support pushdown in the chunked Parquet reader. We will need to enable that again.

@vyasr vyasr added the Python Affects Python cuDF API. label Nov 14, 2024
@wence- wence- added the pylibcudf Issues specific to the pylibcudf package label Nov 25, 2024
@wence- wence- assigned wence- and unassigned lithomas1 Nov 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cudf.polars Issues specific to cudf.polars feature request New feature or request pylibcudf Issues specific to the pylibcudf package Python Affects Python cuDF API.
Projects
Status: Todo
Development

No branches or pull requests

4 participants