Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[VL] Set Arrow_SOURCE to AUTO to allow using system arrow libs #6325

Merged
merged 1 commit into from
Jul 15, 2024

Conversation

PHILO-HE
Copy link
Contributor

@PHILO-HE PHILO-HE commented Jul 3, 2024

What changes were proposed in this pull request?

Depends on the fix in facebookincubator/velox#10355.

How was this patch tested?

CI.

Copy link

github-actions bot commented Jul 3, 2024

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/apache/incubator-gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

@Yohahaha
Copy link
Contributor

Yohahaha commented Jul 3, 2024

      set_target_properties(
        Arrow::${LIB_NAME}
        PROPERTIES IMPORTED_LOCATION ${ARROW_LIB_${LIB_NAME}}
                   INTERFACE_INCLUDE_DIRECTORIES ${ARROW_HOME}/install/include)

if use system arrow, seems gluten still use ${ARROW_HOME}/install/include as include dir.

@PHILO-HE
Copy link
Contributor Author

PHILO-HE commented Jul 4, 2024

      set_target_properties(
        Arrow::${LIB_NAME}
        PROPERTIES IMPORTED_LOCATION ${ARROW_LIB_${LIB_NAME}}
                   INTERFACE_INCLUDE_DIRECTORIES ${ARROW_HOME}/install/include)

if use system arrow, seems gluten still use ${ARROW_HOME}/install/include as include dir.

@Yohahaha, thanks for your comment! I just removed the setting for arrow include directories, as they are actually not needed in Gluten.

@Yohahaha
Copy link
Contributor

Yohahaha commented Jul 4, 2024

set(ARROW_INCLUDE_DIR "${ARROW_INSTALL_DIR}/include")

this line should also be removed.

@PHILO-HE
Copy link
Contributor Author

PHILO-HE commented Jul 4, 2024

set(ARROW_INCLUDE_DIR "${ARROW_INSTALL_DIR}/include")

this line should also be removed.

@Yohahaha, yes, just removed. Thanks!

@PHILO-HE PHILO-HE marked this pull request as ready for review July 13, 2024 04:12
@PHILO-HE
Copy link
Contributor Author

@Yohahaha, the corresponding velox patch has been merged. Do you have any other comment on this gluten PR?

@Yohahaha
Copy link
Contributor

@Yohahaha, the corresponding velox patch has been merged. Do you have any other comment on this gluten PR?

If arrow has installed in other places, seems we still set ARROW_HOME under velox path, could you clarify?

if(${CMAKE_BUILD_TYPE} STREQUAL "Debug")
  set(ARROW_HOME
      ${VELOX_HOME}/_build/debug/CMake/resolve_dependency_modules/arrow/arrow_ep/
  )
else()
  set(ARROW_HOME
      ${VELOX_HOME}/_build/release/CMake/resolve_dependency_modules/arrow/arrow_ep
  )
endif()

@Yohahaha
Copy link
Contributor

@Yohahaha, the corresponding velox patch has been merged. Do you have any other comment on this gluten PR?

If arrow has installed in other places, seems we still set ARROW_HOME under velox path, could you clarify?

if(${CMAKE_BUILD_TYPE} STREQUAL "Debug")
  set(ARROW_HOME
      ${VELOX_HOME}/_build/debug/CMake/resolve_dependency_modules/arrow/arrow_ep/
  )
else()
  set(ARROW_HOME
      ${VELOX_HOME}/_build/release/CMake/resolve_dependency_modules/arrow/arrow_ep
  )
endif()

mayby it's not an issue, find_library should find correct arrow lib if ARROW_HOME is wrong.

@PHILO-HE PHILO-HE changed the title Fix thrift lib not linked issue when system arrow libs are used [VL] Set Arrow_SOURCE to AUTO to allow using system arrow libs Jul 15, 2024
@PHILO-HE
Copy link
Contributor Author

@Yohahaha, thanks for your comment! The arrow home is set to let cmake firstly find arrow libs from that path. If it does not exist (because velox finds the lib from system, then it doesn't build arrow from source), cmake will find the libs from system. Though we always expect cmake in Gluten/Velox finds the libs from system now, it may be good to still keep the arrow home path for debugging (e.g., by export Arrow_SOURCE=BUNDELD, we can mandatorily use bundled arrow to exclude any issues due to system arrow).

@zhouyuan zhouyuan merged commit b93f254 into apache:main Jul 15, 2024
42 checks passed
yma11 added a commit to yma11/gluten that referenced this pull request Jul 15, 2024
yma11 added a commit to yma11/gluten that referenced this pull request Jul 15, 2024
yma11 added a commit to yma11/gluten that referenced this pull request Jul 15, 2024
yma11 added a commit to yma11/gluten that referenced this pull request Jul 15, 2024
yma11 added a commit to yma11/gluten that referenced this pull request Jul 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants