Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[GLUTEN-6483] Support Uniffle 0.9.0 #6484

Merged
merged 1 commit into from
Aug 1, 2024
Merged

[GLUTEN-6483] Support Uniffle 0.9.0 #6484

merged 1 commit into from
Aug 1, 2024

Conversation

SteNicholas
Copy link
Member

@SteNicholas SteNicholas commented Jul 17, 2024 •
edited
Loading

What changes were proposed in this pull request?

Apache Uniffle has released v0.9.0, of which release note could refer to v0.9.0 for more details. This pull request is aimed to support v0.9.0 of Uniffle.

Fixes: #6483.

How was this patch tested?

CI.

@github-actions GitHub Actions
Copy link

#6483

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

1 similar comment
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@SteNicholas SteNicholas marked this pull request as draft July 17, 2024 10:52
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@SteNicholas SteNicholas changed the title [GLUTEN-6483][UNIFFLE] Support Uniffle 0.9.0 [GLUTEN-6483] Support Uniffle 0.9.0 Jul 17, 2024
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@zuston
Copy link
Member

zuston commented Jul 18, 2024

Try to attach spark.rss.client.assignment.shuffle.nodes.max=1 into the uniffle client config.

@SteNicholas SteNicholas force-pushed the GLUTEN-6483 branch 2 times, most recently from d43d712 to 1e7b33c Compare July 18, 2024 09:20
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

1 similar comment
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@SteNicholas
Copy link
Member Author

SteNicholas commented Jul 18, 2024 •
edited
Loading

@zuston, @summaryzb, @xianjingfeng, I have attached spark.rss.client.assignment.shuffle.nodes.max=1 in integration test. But the CI failed with There isn't enough shuffle servers. Should the version upgrade introduce extra configuration of uniffle via apache/incubator-uniffle#1753?

@xianjingfeng
Copy link
Member

@zuston, @summaryzb, @xianjingfeng, I have attached spark.rss.client.assignment.shuffle.nodes.max=1 in integration test. But the CI failed with There isn't enough shuffle servers. Should the version upgrade introduce extra configuration of uniffle via apache/incubator-uniffle#1753?

You can try to set rss.rpc.server.type  to GRPC_NETTY

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@SteNicholas
Copy link
Member Author

@xianjingfeng, I have set rss.rpc.server.type to GRPC_NETTY in server.conf. Unfortunately there is the same exception There isn't enough shuffle servers.

@xianjingfeng
Copy link
Member

xianjingfeng commented Jul 19, 2024 •
edited
Loading

@SteNicholas

incubator-gluten/.github/workflows/velox_docker.yml

Line 507 in 0c519c4

git clone -b branch-0.8 https://github.com/apache/incubator-uniffle.git && \

incubator-gluten/.github/workflows/velox_docker.yml

Line 519 in 0c519c4

cp /opt/incubator-uniffle/server/target/shuffle-server-0.8.1-SNAPSHOT.jar /opt/uniffle/jars/server/

https://github.com/apache/incubator-gluten/actions/runs/9995353296/job/27630015982?pr=6484#step:9:1022

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 1, 2024

Run Gluten Clickhouse CI

@SteNicholas SteNicholas marked this pull request as ready for review August 1, 2024 06:40
@SteNicholas
Copy link
Member Author

SteNicholas commented Aug 1, 2024 •
edited
Loading

Ping @PHILO-HE, @summaryzb, @xianjingfeng. PTAL.

xianjingfeng
xianjingfeng approved these changes Aug 1, 2024
View reviewed changes
Copy link
Member

@xianjingfeng xianjingfeng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@SteNicholas SteNicholas mentioned this pull request Aug 1, 2024
Closed
@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 1, 2024

Run Gluten Clickhouse CI

@SteNicholas SteNicholas requested a review from jackylee-ch August 1, 2024 07:11
summaryzb
summaryzb reviewed Aug 1, 2024
View reviewed changes
.github/workflows/velox_docker.yml Outdated Show resolved Hide resolved
.github/workflows/velox_docker.yml Outdated Show resolved Hide resolved
@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 1, 2024

Run Gluten Clickhouse CI

@SteNicholas SteNicholas requested a review from summaryzb August 1, 2024 07:42
jackylee-ch
jackylee-ch previously approved these changes Aug 1, 2024
View reviewed changes
Copy link
Contributor

@jackylee-ch jackylee-ch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for your work.

@SteNicholas SteNicholas requested a review from PHILO-HE August 1, 2024 08:16
@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 1, 2024

Run Gluten Clickhouse CI

@github-actions github-actions bot added CORE works for Gluten Core INFRA RSS TOOLS labels Aug 1, 2024
@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 1, 2024

Run Gluten Clickhouse CI

@SteNicholas
Copy link
Member Author

@PHILO-HE, the CI of uniffle has already been successful. Please help to merge.

@jackylee-ch
Copy link
Contributor

Just wait for the CI finished. After that, I would merge this pr if no more comments.

@jackylee-ch jackylee-ch merged commit bedb2ee into apache:main Aug 1, 2024
42 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@PHILO-HE PHILO-HE PHILO-HE approved these changes

@xianjingfeng xianjingfeng xianjingfeng approved these changes

@jackylee-ch jackylee-ch jackylee-ch approved these changes

@summaryzb summaryzb Awaiting requested review from summaryzb

Assignees
No one assigned
Labels
CORE works for Gluten Core INFRA RSS TOOLS
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Support Uniffile 0.9.0
6 participants
@SteNicholas @zuston @xianjingfeng @summaryzb @jackylee-ch @PHILO-HE