-
-
Notifications
You must be signed in to change notification settings - Fork 79
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
NPE while executing plugin on unchanged proto files #104
Comments
Is there any progress on this issue? Yes, |
Not sure it'll move forward without some support for the project |
hi, im facing the same error.... some solutions? or where do you put optimizeCodegen=true option? |
The error manifests because of an interplay between
Fortunately Alternatively, it may be that |
### What changes were proposed in this pull request? This PR aims to fix `protobuf` module `Maven` compilation failure due to the following `protocol-jar-maven-plugin` bug. - os72/protoc-jar-maven-plugin#104 Note that this happens only at `Maven` compilation. ### Why are the changes needed? **BEFORE** ``` $ build/mvn -DskipTests clean install -pl connector/protobuf -e -am ... [INFO] Spark Protobuf ..................................... FAILURE [ 7.295 s] ... [ERROR] Failed to execute goal com.github.os72:protoc-jar-maven-plugin:3.11.4:run (default) on project spark-protobuf_2.13: Execution default of goal com.github.os72:protoc-jar-maven-plugin:3.11.4:run failed: Cannot read the array length because "<local6>" is null ``` **AFTER** ``` $ build/mvn -DskipTests clean install -pl connector/protobuf -e -am ... [INFO] Reactor Summary for Spark Project Parent POM 4.0.0-SNAPSHOT: [INFO] [INFO] Spark Project Parent POM ........................... SUCCESS [ 2.601 s] [INFO] Spark Project Tags ................................. SUCCESS [ 5.234 s] [INFO] Spark Project Sketch ............................... SUCCESS [ 5.278 s] [INFO] Spark Project Common Utils ......................... SUCCESS [ 14.695 s] [INFO] Spark Project Local DB ............................. SUCCESS [ 6.258 s] [INFO] Spark Project Networking ........................... SUCCESS [ 9.384 s] [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 7.761 s] [INFO] Spark Project Variant .............................. SUCCESS [ 3.304 s] [INFO] Spark Project Unsafe ............................... SUCCESS [ 8.677 s] [INFO] Spark Project Connect Shims ........................ SUCCESS [ 2.690 s] [INFO] Spark Project Launcher ............................. SUCCESS [ 5.618 s] [INFO] Spark Project Core ................................. SUCCESS [01:11 min] [INFO] Spark Project SQL API .............................. SUCCESS [ 21.759 s] [INFO] Spark Project Catalyst ............................. SUCCESS [01:30 min] [INFO] Spark Project SQL .................................. SUCCESS [01:53 min] [INFO] Spark Protobuf ..................................... SUCCESS [ 18.285 s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 06:26 min [INFO] Finished at: 2024-10-27T21:01:12-07:00 [INFO] ------------------------------------------------------------------------ ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manually run the following command like I did in the above. ``` $ build/mvn -DskipTests clean install -pl connector/protobuf -e -am ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes #48673 from dongjoon-hyun/SPARK-50143. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
…-defined protoc ### What changes were proposed in this pull request? People may use user-defined protoc(as below) for various reasons, for example, if they're on macOS 11, they have to use it since the default version of protoc and protoc-gen-grpc-java plugin do not work on macOS 11. ``` $ export SPARK_PROTOC_EXEC_PATH=/Users/foobar/dev/protoc-4.29.1/protoc $ export CONNECT_PLUGIN_EXEC_PATH=/Users/foobar/dev/protoc-gen-grpc-java-1.59.1/protoc-gen-grpc-java $ ./build/mvn -DskipTests clean package -Puser-defined-protoc -DskipDefaultProtoc ``` But this results in compilation failure: ``` [ERROR] Failed to execute goal com.github.os72:protoc-jar-maven-plugin:3.11.4:run (default) on project spark-protobuf_2.13: Execution default of goal com.github.os72:protoc-jar-maven-plugin:3.11.4:run failed: Cannot read the array length because "<local6>" is null ``` This PR aims to fix this compilation failure, the error is caused by `protocol-jar-maven-plugin` bug: - os72/protoc-jar-maven-plugin#104 ### Why are the changes needed? Before: ``` $ export SPARK_PROTOC_EXEC_PATH=/Users/foobar/dev/protoc-4.29.1/protoc $ export CONNECT_PLUGIN_EXEC_PATH=/Users/foobar/dev/protoc-gen-grpc-java-1.59.1/protoc-gen-grpc-java $ ./build/mvn -DskipTests clean package -Puser-defined-protoc -DskipDefaultProtoc ... [ERROR] Failed to execute goal com.github.os72:protoc-jar-maven-plugin:3.11.4:run (default) on project spark-protobuf_2.13: Execution default of goal com.github.os72:protoc-jar-maven-plugin:3.11.4:run failed: Cannot read the array length because "<local6>" is null ``` After: ``` $ export SPARK_PROTOC_EXEC_PATH=/Users/foobar/dev/protoc-4.29.1/protoc $ export CONNECT_PLUGIN_EXEC_PATH=/Users/foobar/dev/protoc-gen-grpc-java-1.59.1/protoc-gen-grpc-java $ ./build/mvn -DskipTests clean package -Puser-defined-protoc -DskipDefaultProtoc ... [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary for Spark Project Parent POM 4.0.0-SNAPSHOT: [INFO] [INFO] Spark Project Parent POM ........................... SUCCESS [ 7.539 s] [INFO] Spark Project Tags ................................. SUCCESS [ 7.527 s] [INFO] Spark Project Sketch ............................... SUCCESS [ 8.413 s] [INFO] Spark Project Common Utils ......................... SUCCESS [ 29.923 s] [INFO] Spark Project Local DB ............................. SUCCESS [ 10.732 s] [INFO] Spark Project Networking ........................... SUCCESS [ 16.691 s] [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 12.653 s] [INFO] Spark Project Variant .............................. SUCCESS [ 5.262 s] [INFO] Spark Project Unsafe ............................... SUCCESS [ 16.246 s] [INFO] Spark Project Connect Shims ........................ SUCCESS [ 4.356 s] [INFO] Spark Project Launcher ............................. SUCCESS [ 8.666 s] [INFO] Spark Project Core ................................. SUCCESS [02:56 min] [INFO] Spark Project ML Local Library ..................... SUCCESS [ 30.063 s] [INFO] Spark Project GraphX ............................... SUCCESS [ 33.989 s] [INFO] Spark Project Streaming ............................ SUCCESS [ 51.897 s] [INFO] Spark Project SQL API .............................. SUCCESS [ 39.508 s] [INFO] Spark Project Catalyst ............................. SUCCESS [03:15 min] [INFO] Spark Project SQL .................................. SUCCESS [04:56 min] [INFO] Spark Project ML Library ........................... SUCCESS [02:30 min] [INFO] Spark Project Tools ................................ SUCCESS [ 4.463 s] [INFO] Spark Project Hive ................................. SUCCESS [01:33 min] [INFO] Spark Project Connect Common ....................... SUCCESS [ 55.308 s] [INFO] Spark Avro ......................................... SUCCESS [ 33.350 s] [INFO] Spark Protobuf ..................................... SUCCESS [ 38.981 s] [INFO] Spark Project REPL ................................. SUCCESS [ 21.165 s] [INFO] Spark Project Connect Server ....................... SUCCESS [ 58.033 s] [INFO] Spark Project Connect Client ....................... SUCCESS [ 59.078 s] [INFO] Spark Project Assembly ............................. SUCCESS [ 6.703 s] [INFO] Kafka 0.10+ Token Provider for Streaming ........... SUCCESS [ 20.047 s] [INFO] Spark Integration for Kafka 0.10 ................... SUCCESS [ 26.882 s] [INFO] Kafka 0.10+ Source for Structured Streaming ........ SUCCESS [ 41.397 s] [INFO] Spark Project Examples ............................. SUCCESS [ 41.308 s] [INFO] Spark Integration for Kafka 0.10 Assembly .......... SUCCESS [ 11.300 s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Run command: ``` $ export SPARK_PROTOC_EXEC_PATH=/Users/foobar/dev/protoc-4.29.1/protoc $ export CONNECT_PLUGIN_EXEC_PATH=/Users/foobar/dev/protoc-gen-grpc-java-1.59.1/protoc-gen-grpc-java $ ./build/mvn -DskipTests clean package -Puser-defined-protoc -DskipDefaultProtoc ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes #49296 from morvenhuang/SPARK-50671. Authored-by: morvenhuang <[email protected]> Signed-off-by: yangjie01 <[email protected]>
mvn clean
mvn generate-sources
-> generates initiallymvn generate-sources
-> fails withConfig to reproduce:
I assume that happens only when
optimizeCodegen=true
, setting it to false works (at least as workaround)The text was updated successfully, but these errors were encountered: