Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[VL] Minor follow-ups for PRs #6693

Merged
merged 11 commits into from
Aug 6, 2024
Merged

Conversation

zhztheplayer
Copy link
Member

@zhztheplayer zhztheplayer commented Aug 2, 2024

  1. Following PR [VL] Allow specifying maximum batch size for batch resizing #6670, change min batch size to 0.25 * default size
    • Remove option spark.gluten.sql.columnar.backend.velox.resizeBatches.shuffleInput.range
    • Add option spark.gluten.sql.columnar.backend.velox.resizeBatches.shuffleInput.minSize
  2. Following PR [CORE] Propagate SQLConf to code block of TaskResources.runUnsafe #6658, change gluten-it's JVM options to match up with vanilla Spark, since options in JavaModuleOptions are passed to executors in local cluster mode, we'd align executors' and driver's options in that mode.

@github-actions github-actions bot added CORE works for Gluten Core TOOLS labels Aug 2, 2024
Copy link

github-actions bot commented Aug 2, 2024

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/apache/incubator-gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

Copy link

github-actions bot commented Aug 2, 2024

Run Gluten Clickhouse CI

@FelixYBW
Copy link
Contributor

FelixYBW commented Aug 2, 2024

As I documented, since the slice has little overhead, let's just honor the max batch size as 1x.

@zhztheplayer
Copy link
Member Author

zhztheplayer commented Aug 3, 2024

As I documented, since the slice has little overhead, let's just honor the max batch size as 1x.

If I remember correctly in our code there were some places producing columnar batches that use default batch size 4096 as a minimal threshold. Which means we have chance to frequently get batches with size say, 4100 or something. Let's at least relax a little bit to allow that kind of batches to pass? Otherwise very small batches could be produced. I know it's not a big issue but the way we are addressing this doesn't bring much new code to maintain.

Copy link

github-actions bot commented Aug 5, 2024

Run Gluten Clickhouse CI

2 similar comments
Copy link

github-actions bot commented Aug 5, 2024

Run Gluten Clickhouse CI

Copy link

github-actions bot commented Aug 5, 2024

Run Gluten Clickhouse CI

Copy link

github-actions bot commented Aug 5, 2024

Run Gluten Clickhouse CI

Copy link

github-actions bot commented Aug 5, 2024

Run Gluten Clickhouse CI

Copy link

github-actions bot commented Aug 5, 2024

Run Gluten Clickhouse CI

Copy link

github-actions bot commented Aug 5, 2024

Run Gluten Clickhouse CI

Copy link

github-actions bot commented Aug 5, 2024

Run Gluten Clickhouse CI

Copy link

github-actions bot commented Aug 5, 2024

Run Gluten Clickhouse CI

Copy link

github-actions bot commented Aug 6, 2024

Run Gluten Clickhouse CI

@zhztheplayer zhztheplayer merged commit 0625a75 into apache:main Aug 6, 2024
44 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CORE works for Gluten Core TOOLS VELOX
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants