Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[VL] Followup for native write files #4246

Merged
merged 1 commit into from
Jan 3, 2024

Conversation

ulysses-you
Copy link
Contributor

What changes were proposed in this pull request?

This pr is a followup for #3690 to solve some issues:

  • Remove HiveFileFormat for Spark3.4 shim
  • Add an object mapper VeloxWriteFilesMetrics for velox table write result json string
  • Add task output metrics
  • Add number of dynamic part metric
  • Use thread local to propagate and clear the writePath
  • Make VeloxColumnarWriteFilesExec inherit GlutenPlan to avoid wrong fallback report

How was this patch tested?

add test

Copy link

github-actions bot commented Jan 2, 2024

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/oap-project/gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

Copy link

github-actions bot commented Jan 2, 2024

Run Gluten Clickhouse CI

@FelixYBW FelixYBW requested a review from JkSelf January 2, 2024 08:32
@ulysses-you
Copy link
Contributor Author

@JkSelf any more comments ?

@JkSelf
Copy link
Contributor

JkSelf commented Jan 3, 2024

@JkSelf any more comments ?

LGTM. The ck backend failed. Can you help to fix?

@ulysses-you
Copy link
Contributor Author

it seems it's irrelevant with this pr, all tests paased but finally jenkin failed to post status..

Copy link
Contributor

@JkSelf JkSelf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@ulysses-you ulysses-you merged commit 8797350 into apache:main Jan 3, 2024
17 of 18 checks passed
@ulysses-you ulysses-you deleted the write-files branch January 3, 2024 02:19
@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query log/native_4246_time.csv log/native_master_01_02_2024_3660e312c_time.csv difference percentage
q1 33.33 35.26 1.929 105.79%
q2 25.03 25.20 0.168 100.67%
q3 38.63 38.24 -0.392 98.99%
q4 39.27 39.14 -0.131 99.67%
q5 71.91 71.45 -0.457 99.36%
q6 7.00 7.09 0.092 101.31%
q7 85.29 86.43 1.143 101.34%
q8 86.31 85.68 -0.628 99.27%
q9 121.50 123.79 2.292 101.89%
q10 46.26 43.72 -2.543 94.50%
q11 20.39 20.90 0.517 102.54%
q12 24.91 26.34 1.427 105.73%
q13 47.76 45.95 -1.813 96.20%
q14 17.64 14.28 -3.363 80.94%
q15 27.03 28.62 1.585 105.86%
q16 16.83 15.25 -1.575 90.64%
q17 102.64 102.80 0.163 100.16%
q18 152.09 149.26 -2.831 98.14%
q19 12.72 12.84 0.123 100.97%
q20 28.39 27.65 -0.737 97.41%
q21 230.15 227.40 -2.753 98.80%
q22 13.99 14.05 0.068 100.49%
total 1249.07 1241.36 -7.714 99.38%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants