Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[GLUTEN-5103][VL] Use jvm libhdfs replace c++ libhdfs3 #6172

Merged
merged 3 commits into from
Oct 25, 2024
Merged

[GLUTEN-5103][VL] Use jvm libhdfs replace c++ libhdfs3 #6172

merged 3 commits into from
Oct 25, 2024

Conversation

JkSelf
Copy link
Contributor

@JkSelf JkSelf commented Jun 21, 2024 •
edited
Loading

What changes were proposed in this pull request?

fixes #5103

How was this patch tested?

Local verification.

@github-actions GitHub Actions
Copy link

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/apache/incubator-gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@JkSelf JkSelf force-pushed the libhdfs-replace branch from a289a01 to 85b1266 Compare June 21, 2024 01:30
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@JkSelf JkSelf force-pushed the libhdfs-replace branch from 85b1266 to d2fdd20 Compare June 21, 2024 02:58
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@JkSelf JkSelf force-pushed the libhdfs-replace branch from d2fdd20 to 0beeeb1 Compare June 21, 2024 03:04
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@JkSelf JkSelf force-pushed the libhdfs-replace branch from 0beeeb1 to 591dfa0 Compare June 21, 2024 03:18
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@JkSelf JkSelf force-pushed the libhdfs-replace branch from 591dfa0 to 08ddf67 Compare June 21, 2024 08:19
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

2 similar comments
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@JkSelf JkSelf force-pushed the libhdfs-replace branch from 84336f3 to 7bc8093 Compare June 24, 2024 01:11
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@JkSelf JkSelf force-pushed the libhdfs-replace branch from 7bc8093 to 0e53135 Compare July 11, 2024 08:19
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@JkSelf JkSelf force-pushed the libhdfs-replace branch from 0e53135 to 82439cc Compare July 22, 2024 23:54
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@JkSelf JkSelf force-pushed the libhdfs-replace branch from 82439cc to 1911c45 Compare July 26, 2024 02:53
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@JkSelf JkSelf force-pushed the libhdfs-replace branch from 1911c45 to 22113ea Compare July 30, 2024 08:33
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@PHILO-HE
Copy link
Contributor

Seems velox-hdfs part needs to be removed from vcpkg.json. And ports/libhdfs3 also needs to be removed.

@JkSelf JkSelf force-pushed the libhdfs-replace branch from 22113ea to 66a476a Compare July 31, 2024 02:07
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@FelixYBW
Copy link
Contributor

FelixYBW commented Aug 7, 2024

@JkSelf the PR is depending on the Velox PR, right?

@FelixYBW
Copy link
Contributor

@JkSelf can you rebase the PR?

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

1 similar comment
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

2 similar comments
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@JkSelf JkSelf requested a review from PHILO-HE October 25, 2024 06:02
PHILO-HE
PHILO-HE approved these changes Oct 25, 2024
View reviewed changes
Copy link
Contributor

@PHILO-HE PHILO-HE left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Should velox branch be changed from your own branch?

@JkSelf
Copy link
Contributor Author

JkSelf commented Oct 25, 2024

Looks good! Should velox branch be changed from your own branch?

Yes. Will change to 2024_10_25 branch later. Thanks.

@marin-ma marin-ma merged commit bf3d8d7 into apache:main Oct 25, 2024
47 checks passed
marin-ma added a commit that referenced this pull request Oct 25, 2024
@marin-ma
Revert "[GLUTEN-5103][VL] Use jvm libhdfs replace c++ libhdfs3 (#6172)"
f808b50 
This reverts commit bf3d8d7.
@marin-ma marin-ma mentioned this pull request Oct 25, 2024
Merged
JkSelf pushed a commit that referenced this pull request Oct 25, 2024
@marin-ma
Revert "[GLUTEN-5103][VL] Use jvm libhdfs replace c++ libhdfs3 (#6172)…
3b81ffd 
…" (#7683)

This reverts commit bf3d8d7.
@FelixYBW
Copy link
Contributor

With this PR merged, is our default hdfs is libhdfs, instead of libhdfs3, right? Do we still support libhdfs3? or should wait until libhdfs3 support is merged in Velox? Need we modify jenkins tests?

if we support both how to config? Let's have a new PR to document all the changes on necessary.

@JkSelf JkSelf mentioned this pull request Oct 25, 2024
Merged
yikf pushed a commit to yikf/incubator-gluten that referenced this pull request Oct 28, 2024
@JkSelf @yikf
[GLUTEN-5103][VL] Use jvm libhdfs replace c++ libhdfs3 (apache#6172)
7d8ecec
yikf pushed a commit to yikf/incubator-gluten that referenced this pull request Oct 28, 2024
@marin-ma @yikf
Revert "[GLUTEN-5103][VL] Use jvm libhdfs replace c++ libhdfs3 (apach…
b537341 
…e#6172)" (apache#7683)

This reverts commit bf3d8d7.
@JkSelf JkSelf mentioned this pull request Oct 31, 2024
Closed
JkSelf added a commit to JkSelf/gluten that referenced this pull request Nov 14, 2024
@JkSelf
Revert "[GLUTEN-5103][VL] Use jvm libhdfs replace c++ libhdfs3 (apach…
ae2e6ef 
…e#6172)"

This reverts commit bf3d8d7.
JkSelf added a commit to JkSelf/gluten that referenced this pull request Nov 18, 2024
@JkSelf
Revert "[GLUTEN-5103][VL] Use jvm libhdfs replace c++ libhdfs3 (apach…
01eb682 
…e#6172)"

This reverts commit bf3d8d7.
@liujiayi771 liujiayi771 mentioned this pull request Nov 29, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@PHILO-HE PHILO-HE PHILO-HE approved these changes

Assignees
No one assigned
Labels
BUILD CLICKHOUSE CORE works for Gluten Core DATA_LAKE VELOX
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[VL] Supporting JVM based libhdfs connector
6 participants
@JkSelf @PHILO-HE @FelixYBW @ArnavBalyan @zhouyuan @marin-ma