[GLUTEN-6253] Use internal udf config to avoid modify the original one

[marin-ma]

The current implementation sets spark.gluten.sql.columnar.backend.velox.udfLibraryPaths on driver side after resolving the library paths. This approach can overwrite the original settings with a local file path on the driver node before sending the SparkConf to all executors, and the executors on different nodes will fail while accessing that path.

This PR sets the resolved library paths to an internal config to avoid the conflicts.

Manually verified on a multi-node cluster.

[github-actions]

#6253

[marin-ma]

@kecookier Could you help to review? Thanks!

[kecookier]

Hi @marin-ma, I have some concerns. In yarn-client mode, we may set an extra configuration --conf spark.gluten.sql.columnar.backend.velox.driver.udfLibraryPaths=file:///path/to/libmyudf.so to let the driver read the local UDF library. The complete configuration is as follows:

--files /path/to/gluten/cpp/build/velox/udf/examples/libmyudf.so
--conf spark.gluten.sql.columnar.backend.velox.udfLibraryPaths=libmyudf.so
# Needed for Yarn client mode
--conf spark.gluten.sql.columnar.backend.velox.driver.udfLibraryPaths=file:///path/to/libmyudf.so

It seems like spark.gluten.sql.columnar.backend.velox.internal.udfLibraryPaths has the same meaning. Could you please clarify this?

[marin-ma]

@kecookier That's the exact configuration I used in #6253

spark.gluten.sql.columnar.backend.velox.udfLibraryPaths and spark.gluten.sql.columnar.backend.velox.driver.udfLibraryPaths are the configurations exposed to users. spark.gluten.sql.columnar.backend.velox.internal.udfLibraryPaths is used as an internal configuration to save the resolved udf library path and is used on native side for loading the libraries. https://github.com/apache/incubator-gluten/pull/6255/files#diff-a633836e086157189ba6a590a79c4823e73937f896ecab711a324586e13aa73aL102-R102

[kecookier]

Thanks for your clarification, it looks good to me.