Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CORE] Execution runtime / native memory manager refactor #6243

Merged
merged 38 commits into from
Jul 1, 2024
Merged

[CORE] Execution runtime / native memory manager refactor #6243

merged 38 commits into from
Jul 1, 2024

Conversation

zhztheplayer
Copy link
Member

@zhztheplayer zhztheplayer commented Jun 27, 2024 •
edited
Loading

The patch introduces a new design for (Velox backend) execution runtime and native memory manager. The new design will make the relevant components much clearer and simpler than before both in concept and code.

The following figure will demonstrate the new design in detail:

New Runtime Arch

In the new design, native memory manager instances will be 1:1 bound to execution runtime instances. Objects stores are still in runtime instances to make sure all the allocated contextual objects don't escape their resident runtimes' lifecycle scope.

It's no longer needed to create memory manager instances from JVM side as they will be managed internally in runtime instances. Spillers will be accounted in a list in runtime and will no longer need to be created during instantiating memory managers.

JVM side MemoryManager and MemoryAllocator APIs are removed as no longer used.

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

1 similar comment
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@zhztheplayer zhztheplayer changed the title [VL] Runtime / native memory manager refactor [CORE][VL] Runtime / native memory manager refactor Jun 27, 2024
@zhztheplayer zhztheplayer changed the title [CORE][VL] Runtime / native memory manager refactor [CORE] Runtime / native memory manager refactor Jun 27, 2024
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

2 similar comments
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@zhztheplayer zhztheplayer changed the title [CORE] Runtime / native memory manager refactor [CORE] Execution runtime / native memory manager refactor Jun 27, 2024
@apache apache deleted a comment from github-actions bot Jun 27, 2024
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@zhztheplayer
Copy link
Member Author

/Benchmark Velox

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@zhztheplayer
Copy link
Member Author

/Benchmark Velox

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

2 similar comments
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 1, 2024

Run Gluten Clickhouse CI

@zhztheplayer zhztheplayer marked this pull request as ready for review July 1, 2024 01:21
@zhztheplayer
Copy link
Member Author

zhztheplayer commented Jul 1, 2024 •
edited
Loading

Issues should be followed up:

  1. OOM CI steps should be untangled, some of them were marked to be fixed but have been passing for sometime
  2. Should use a global instance pool used by ColumnarBatches#create to avoid account ownerships on different runtime
  3. Q23A/Q23B OOM test not passed in CI but not repeatable in local

@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCDS SF2000 with Velox backend, for reference only ====

query log/native_6243_time.csv log/native_master_06_30_2024_86449d0c8_time.csv difference percentage
q1 14.35 14.83 0.481 103.35%
q2 16.63 14.85 -1.772 89.34%
q3 3.94 4.21 0.270 106.84%
q4 64.10 63.51 -0.588 99.08%
q5 7.78 7.38 -0.406 94.78%
q6 2.60 2.32 -0.282 89.16%
q7 3.90 6.03 2.136 154.82%
q8 5.11 5.14 0.033 100.64%
q9 18.90 21.68 2.781 114.71%
q10 10.56 11.58 1.024 109.70%
q11 40.32 36.42 -3.899 90.33%
q12 1.56 2.59 1.031 165.97%
q13 6.80 9.04 2.247 133.07%
q14a 42.41 41.82 -0.591 98.61%
q14b 42.21 38.81 -3.402 91.94%
q15 3.63 3.62 -0.007 99.81%
q16 39.03 39.65 0.627 101.61%
q17 4.91 5.07 0.157 103.20%
q18 7.78 6.20 -1.577 79.72%
q19 2.21 2.34 0.135 106.12%
q20 1.47 1.40 -0.064 95.64%
q21 3.64 1.02 -2.618 28.09%
q22 7.87 7.79 -0.075 99.05%
q23a 85.74 84.02 -1.723 97.99%
q23b 103.44 101.13 -2.308 97.77%
q24a 75.91 69.39 -6.523 91.41%
q24b 76.38 72.20 -4.181 94.53%
q25 4.41 4.35 -0.057 98.71%
q26 2.85 2.85 0.007 100.25%
q27 3.15 3.15 -0.001 99.96%
q28 25.46 28.74 3.276 112.87%
q29 7.40 7.11 -0.294 96.03%
q30 6.05 4.19 -1.858 69.29%
q31 6.56 6.70 0.143 102.18%
q32 1.16 2.13 0.967 183.33%
q33 5.06 4.74 -0.321 93.66%
q34 3.83 3.65 -0.180 95.29%
q35 7.26 6.57 -0.695 90.44%
q36 3.44 3.35 -0.082 97.61%
q37 4.38 4.05 -0.332 92.43%
q38 16.73 11.85 -4.882 70.83%
q39a 3.42 5.69 2.263 166.08%
q39b 2.79 3.27 0.480 117.18%
q40 3.96 3.73 -0.228 94.25%
q41 0.61 0.62 0.007 101.18%
q42 0.92 0.94 0.025 102.70%
q43 3.66 3.89 0.225 106.13%
q44 8.53 8.62 0.090 101.06%
q45 3.63 3.40 -0.235 93.53%
q46 3.14 3.22 0.077 102.44%
q47 14.37 14.40 0.025 100.18%
q48 4.51 4.58 0.068 101.50%
q49 9.40 9.80 0.395 104.20%
q50 25.00 24.23 -0.772 96.91%
q51 8.51 8.51 0.008 100.09%
q52 1.02 1.04 0.018 101.74%
q53 2.01 2.01 0.005 100.27%
q54 3.43 3.23 -0.201 94.14%
q55 1.01 0.99 -0.015 98.47%
q56 4.55 4.53 -0.013 99.71%
q57 8.74 9.28 0.541 106.19%
q58 2.75 2.59 -0.158 94.24%
q59 17.66 16.43 -1.223 93.07%
q60 5.32 4.83 -0.494 90.71%
q61 5.83 5.67 -0.161 97.24%
q62 4.24 4.76 0.519 112.24%
q63 2.30 2.32 0.017 100.72%
q64 49.21 49.99 0.787 101.60%
q65 13.80 13.70 -0.096 99.31%
q66 3.54 4.25 0.707 119.96%
q67 351.00 355.08 4.081 101.16%
q68 4.13 3.65 -0.478 88.43%
q69 9.82 6.25 -3.570 63.66%
q70 7.73 8.60 0.878 111.36%
q71 2.21 2.45 0.247 111.21%
q72 188.02 188.05 0.033 100.02%
q73 4.58 2.26 -2.323 49.33%
q74 21.33 21.26 -0.066 99.69%
q75 23.41 23.20 -0.213 99.09%
q76 9.42 9.42 0.004 100.04%
q77 2.21 2.21 -0.000 99.98%
q78 41.38 41.97 0.583 101.41%
q79 3.59 3.51 -0.085 97.63%
q80 11.19 11.16 -0.027 99.76%
q81 5.13 5.19 0.064 101.25%
q82 10.37 10.14 -0.226 97.82%
q83 1.60 1.55 -0.043 97.32%
q84 2.92 2.68 -0.239 91.79%
q85 6.79 6.54 -0.257 96.21%
q86 3.39 3.21 -0.180 94.71%
q87 13.97 12.09 -1.873 86.59%
q88 25.24 24.76 -0.478 98.10%
q89 3.17 3.25 0.081 102.56%
q90 7.75 9.55 1.797 123.19%
q91 2.67 5.39 2.718 201.76%
q92 1.27 1.40 0.128 110.09%
q93 28.83 28.51 -0.319 98.89%
q94 21.30 21.60 0.301 101.41%
q9 84.88 83.91 -0.975 98.85%
q5 3.51 3.78 0.278 107.93%
q96 12.15 12.34 0.193 101.59%
q97 1.89 1.96 0.062 103.28%
q98 9.20 9.18 -0.024 99.74%
q99 9.20 9.18 -0.024 99.74%
total 1934.80 1914.13 -20.670 98.93%

@zhztheplayer zhztheplayer merged commit ae2d125 into apache:main Jul 1, 2024
41 of 42 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@JkSelf JkSelf JkSelf approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@zhztheplayer @GlutenPerfBot @JkSelf