Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[GLUTEN-3378][VL] Iceberg support set partition columns in split info #3987

Merged
merged 4 commits into from
Dec 19, 2023

Conversation

liujiayi771
Copy link
Contributor

What changes were proposed in this pull request?

Velox support specifying partitionColumns in the split info, which improves efficiency when reading partition columns as there's no need to read from the parquet file.

Copy link

github-actions bot commented Dec 9, 2023

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/oap-project/gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

Copy link

github-actions bot commented Dec 9, 2023

Run Gluten Clickhouse CI

@liujiayi771
Copy link
Contributor Author

@YannByron @yma11 Could you help review?

Copy link

Run Gluten Clickhouse CI

@liujiayi771 liujiayi771 force-pushed the iceberg-partition-column branch from ff79ef8 to 435e773 Compare December 11, 2023 14:00
Copy link

Run Gluten Clickhouse CI

@liujiayi771 liujiayi771 changed the title [VL] Iceberg support set partition columns in split info [GLUTEN-3378][VL] Iceberg support set partition columns in split info Dec 11, 2023
Copy link

#3378

Copy link

Run Gluten Clickhouse CI

@liujiayi771 liujiayi771 force-pushed the iceberg-partition-column branch from 8bf55a7 to 86348a9 Compare December 14, 2023 11:35
Copy link

Run Gluten Clickhouse CI

@liujiayi771 liujiayi771 force-pushed the iceberg-partition-column branch from 86348a9 to 19db4e3 Compare December 14, 2023 11:37
Copy link

Run Gluten Clickhouse CI

@liujiayi771 liujiayi771 force-pushed the iceberg-partition-column branch from 19db4e3 to 9d5e9b8 Compare December 15, 2023 02:13
Copy link

Run Gluten Clickhouse CI

@yma11
Copy link
Contributor

yma11 commented Dec 18, 2023

@liujiayi771 please rebase this PR to resolve conflicts.

@liujiayi771 liujiayi771 force-pushed the iceberg-partition-column branch from 9d5e9b8 to 6006880 Compare December 18, 2023 07:10
Copy link

Run Gluten Clickhouse CI

@liujiayi771
Copy link
Contributor Author

@yma11 Rebased and CI passed.

@yma11 yma11 merged commit 203e715 into apache:main Dec 19, 2023
17 checks passed
@liujiayi771 liujiayi771 deleted the iceberg-partition-column branch December 19, 2023 02:21
@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query log/native_3987_time.csv log/native_master_12_18_2023_ecf765da9_time.csv difference percentage
q1 34.48 33.40 -1.082 96.86%
q2 25.03 24.92 -0.116 99.53%
q3 37.72 39.07 1.345 103.57%
q4 40.12 39.68 -0.447 98.89%
q5 73.38 72.52 -0.859 98.83%
q6 7.17 7.08 -0.086 98.80%
q7 83.88 85.20 1.320 101.57%
q8 87.14 87.23 0.087 100.10%
q9 125.42 127.21 1.793 101.43%
q10 43.27 45.48 2.211 105.11%
q11 20.75 19.88 -0.868 95.82%
q12 25.34 25.92 0.582 102.30%
q13 46.24 46.99 0.758 101.64%
q14 16.52 15.90 -0.617 96.27%
q15 27.12 30.75 3.631 113.39%
q16 15.44 15.88 0.446 102.89%
q17 102.02 103.20 1.175 101.15%
q18 151.21 151.92 0.707 100.47%
q19 14.27 12.96 -1.307 90.84%
q20 27.96 28.35 0.389 101.39%
q21 227.80 229.29 1.491 100.65%
q22 13.76 13.82 0.055 100.40%
total 1246.04 1256.65 10.608 100.85%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants