[GLUTEN-5901][CH] Support CH backend parquet + delta #5902
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Run Gluten Clickhouse CI |
|
Run Gluten Clickhouse CI |
|
Run Gluten Clickhouse CI |
|
Run Gluten Clickhouse CI |
|
Run Gluten Clickhouse CI |
Support CH backend to read/write parquet with the delta: 1. native read parquet from the delta catalog; 2. fallback write the parquet to the delta catalog ( don't support the DeltaInvariantCheckerExec operator and DeltaTaskStatisticsTracker) ; 3. Use the ClickHouseSparkCatalog as the uniform catalog. Close apache#5901.
|
Run Gluten Clickhouse CI |
|
===== Performance report for TPCH SF2000 with Velox backend, for reference only ====
|
loneylee
pushed a commit
to loneylee/kylin
that referenced
this pull request
Jul 31, 2024
1. Disable gluten in analyze table 2. Gluten support delta 2.3.0 - apache/incubator-gluten#5902 - apache/incubator-gluten#5945 3. Support sum0 4. Fix build - apache/incubator-gluten#5796 - apache/incubator-gluten#5767 5. Fix SlowQueryDetectorTest.testSparderTimeoutCancelJob due to context dirty 6. Cleanup threadlocal contexts 7. Fix GMT+8 not support 8. Fix case class test coverage 9. Add 1 gluten disabled case in analyze table 10.Remove unsupported pushdown filter: 10.1. when we create KylinFileSourceScanExec, we didn't remove subquery filter. 10.2. KylinFileSourceScanExec doesn't inherit from FileSourceScanExec, we miss chance to correct push down filter. 11. native support floor_datetime and ceil_datetime 12. native support kap_add_months and kap_months_between 13. native support _ymdint_between 14. native support truncate 15. native support kylin_split_part 16. native support kylin instr
pfzhan
pushed a commit
to Kyligence/kylin
that referenced
this pull request
Jul 31, 2024
1. Disable gluten in analyze table 2. Gluten support delta 2.3.0 - apache/incubator-gluten#5902 - apache/incubator-gluten#5945 3. Support sum0 4. Fix build - apache/incubator-gluten#5796 - apache/incubator-gluten#5767 5. Fix SlowQueryDetectorTest.testSparderTimeoutCancelJob due to context dirty 6. Cleanup threadlocal contexts 7. Fix GMT+8 not support 8. Fix case class test coverage 9. Add 1 gluten disabled case in analyze table 10.Remove unsupported pushdown filter: 10.1. when we create KylinFileSourceScanExec, we didn't remove subquery filter. 10.2. KylinFileSourceScanExec doesn't inherit from FileSourceScanExec, we miss chance to correct push down filter. 11. native support floor_datetime and ceil_datetime 12. native support kap_add_months and kap_months_between 13. native support _ymdint_between 14. native support truncate 15. native support kylin_split_part 16. native support kylin instr
pfzhan
pushed a commit
to pfzhan/kylin
that referenced
this pull request
Jul 31, 2024
1. Disable gluten in analyze table 2. Gluten support delta 2.3.0 - apache/incubator-gluten#5902 - apache/incubator-gluten#5945 3. Support sum0 4. Fix build - apache/incubator-gluten#5796 - apache/incubator-gluten#5767 5. Fix SlowQueryDetectorTest.testSparderTimeoutCancelJob due to context dirty 6. Cleanup threadlocal contexts 7. Fix GMT+8 not support 8. Fix case class test coverage 9. Add 1 gluten disabled case in analyze table 10.Remove unsupported pushdown filter: 10.1. when we create KylinFileSourceScanExec, we didn't remove subquery filter. 10.2. KylinFileSourceScanExec doesn't inherit from FileSourceScanExec, we miss chance to correct push down filter. 11. native support floor_datetime and ceil_datetime 12. native support kap_add_months and kap_months_between 13. native support _ymdint_between 14. native support truncate 15. native support kylin_split_part 16. native support kylin instr
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Support CH backend to read/write parquet with the delta:
Close #5901.
(Fixes: #5901)
How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)