HiveFileFormat has incompatible class error when running TPC-H q18

[kelvin-qin]

Backend

VL (Velox)

Bug description

[Expected behavior] ：
Run all TPC-H queries normally
[actual behavior]：
Errors are reported when running q18: local class incompatible

Spark version

3.3.2

Spark configurations

No special configurations according to the documentation

System information

centos 8
gcc 8.5.0
openjdk 1.8.0_345
spark 3.3.2
hive 3.1.1
Gluten 1.0.0

Relevant logs

like:
java.io.InvalidClassException: xxx.HiveFileFormat$$anon$1; local class incompatible: stream classdesc serialVersionUID = -2229241455005882481, local class serialVersionUID = -3552051822827161402

[kelvin-qin]

The content points to an anonymous internal class error, I checked the source code and the content of the class Hive is indeed different from the original spark.

[PHILO-HE]

We made some changes for HiveFileFormat and make it overwrite the original spark class. There may have some incompatible issue between spark 331 and 332. Could you try to test with spark 331 to see whether this issue still exists on your side?

[kelvin-qin]

I've pretty much exhausted testing with a combination of CentOS-7, CentOS-8, Spark331, Spark332, and gluten-c7-jar or gluten-c8 -jar before bringing up the issue, and have also eliminated some speculation about jdk version differences（）, and there's still no way to get around the problem.
In addition, the other 21 SQLs run without much problem, but you can see that q18 is performing a CTAS operation, which requires the use of HiveFileFormat.
And this doesn't report an error right off the bat, the exception is thrown after about 8 minutes of executing the sql18.

[kelvin-qin]

As a side note, at gluten 0.5.0, I'm still using spark332, but it runs fine.

[JkSelf]

@kelvin-qin We introduced native write support for Parquet, ORC, and Hive in version 1.0. To achieve this, we needed to overwrite Spark's default HiveFileFormat. Therefore, when utilizing this functionality, it is important to ensure that HiveFileFormat takes precedence over vanilla Spark loading.

To prioritize loading gluten jars over vanilla Spark jars in your environment, you can follow the steps outlined here. Could you please verify whether gluten jars are given higher priority than vanilla Spark jars in your environment?

[kelvin-qin]

It works, thanks so much. @JkSelf
This problem has been bothering me for half a month.
Close this issue.

[kelvin-qin]

fine，close

[lgbo-ustc]

How is this solved? @JkSelf @kelvin-qin 。 We also meet this problem

NATIVE_WRITER_CONF="--conf spark.gluten.sql.native.writer.enabled=true
--conf spark.files=$SPARK_HOME/jars/gluten.jar
--conf spark.driver.extraClassPath=$SPARK_HOME/gluten.jar
--conf spark.executor.extraClassPath=./gluten.jar
--conf spark.driver.userClassPathFirst=true
--conf spark.executor.userClassPathFirst=true
"

[lgbo-ustc]

How is this solved? @JkSelf @kelvin-qin 。 We also meet this problem

NATIVE_WRITER_CONF="--conf spark.gluten.sql.native.writer.enabled=true
--conf spark.files=$SPARK_HOME/jars/gluten.jar
--conf spark.driver.extraClassPath=$SPARK_HOME/gluten.jar
--conf spark.executor.extraClassPath=./gluten.jar
--conf spark.driver.userClassPathFirst=true
--conf spark.executor.userClassPathFirst=true
"

Found the problem. The path configuration spark.driver.extraClassPath is wrong

[kelvin-qin]

How is this solved? @JkSelf @kelvin-qin 。 We also meet this problem

NATIVE_WRITER_CONF="--conf spark.gluten.sql.native.writer.enabled=true
--conf spark.files=$SPARK_HOME/jars/gluten.jar
--conf spark.driver.extraClassPath=$SPARK_HOME/gluten.jar
--conf spark.executor.extraClassPath=./gluten.jar
--conf spark.driver.userClassPathFirst=true
--conf spark.executor.userClassPathFirst=true
"

Found the problem. The path configuration spark.driver.extraClassPath is wrong

@lgbo-ustc Hi, can you post the direct solution under this question? Thank you very much. As I recall it is indeed the relative or absolute path that is causing the problem.

[JkSelf]

@kelvin-qin Maybe you can refer this link . The root cause is that we overwrite some spark classes and need to make spark firstly load gluten jar not vanilla spark jars.

[lgbo-ustc]

How is this solved? @JkSelf @kelvin-qin 。 We also meet this problem

NATIVE_WRITER_CONF="--conf spark.gluten.sql.native.writer.enabled=true
--conf spark.files=$SPARK_HOME/jars/gluten.jar
--conf spark.driver.extraClassPath=$SPARK_HOME/gluten.jar
--conf spark.executor.extraClassPath=./gluten.jar
--conf spark.driver.userClassPathFirst=true
--conf spark.executor.userClassPathFirst=true
"

Found the problem. The path configuration spark.driver.extraClassPath is wrong

@lgbo-ustc Hi, can you post the direct solution under this question? Thank you very much. As I recall it is indeed the relative or absolute path that is causing the problem.

add following configure

--conf spark.gluten.sql.native.writer.enabled=true
--conf spark.files=<your local gluten jar path>
--conf spark.driver.extraClassPath=<your local gluten jar path>
--conf spark.executor.extraClassPath=./gluten.jar
--conf spark.driver.userClassPathFirst=true
--conf spark.executor.userClassPathFirst=true