Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NPE while executing plugin on unchanged proto files #104

Open
vicmosin opened this issue May 12, 2021 · 4 comments
Open

NPE while executing plugin on unchanged proto files #104

vicmosin opened this issue May 12, 2021 · 4 comments

Comments

@vicmosin
Copy link

  1. mvn clean
  2. mvn generate-sources -> generates initially
  3. mvn generate-sources -> fails with
Caused by: java.lang.NullPointerException: Cannot read the array length because "<local6>" is null
    at com.github.os72.protocjar.Protoc.extractStdTypes (Protoc.java:372)
    at com.github.os72.protocjar.maven.ProtocJarMojo.performProtoCompilation (ProtocJarMojo.java:397)
    at com.github.os72.protocjar.maven.ProtocJarMojo.execute (ProtocJarMojo.java:370)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)

Config to reproduce:

         <plugin>
                <groupId>com.github.os72</groupId>
                <artifactId>protoc-jar-maven-plugin</artifactId>
                <version>3.11.4</version>
                <executions>
                    <execution>
                        <phase>generate-sources</phase>
                        <goals>
                            <goal>run</goal>
                        </goals>
                        <configuration>
                            <protocArtifact>com.google.protobuf:protoc:3.12.0</protocArtifact>
                            <includeStdTypes>true</includeStdTypes>
                            <inputDirectories>
                                <include>../proto</include>
                            </inputDirectories>
                            <outputTargets>
                                <outputTarget>
                                    <type>java</type>
                                </outputTarget>
                                <outputTarget>
                                    <type>grpc-java</type>
                                    <pluginArtifact>io.grpc:protoc-gen-grpc-java:${grpc.version}</pluginArtifact>
                                </outputTarget>
                            </outputTargets>
                        </configuration>
                    </execution>
                </executions>
            </plugin>

I assume that happens only when optimizeCodegen=true, setting it to false works (at least as workaround)

@serdaryesilmurat
Copy link

Is there any progress on this issue?

Yes, optimizeCodegen=true seems to be a workaround, but I don't think it should be accepted as a solution.

@os72
Copy link
Owner

os72 commented Dec 5, 2021

Not sure it'll move forward without some support for the project

@Qleoz12
Copy link

Qleoz12 commented Jan 20, 2023

hi, im facing the same error.... some solutions? or where do you put optimizeCodegen=true option?

@commonquail
Copy link

This is #121, #60, #117.

The error manifests because of an interplay between <includeStdTypes>, <protocArtifact>, and <protocVersion>.

  1. <includeStdTypes> triggers a protocol major version lookup
  2. but that lookup only succeeds if <protocVersion> is set and otherwise sets local6=null
  3. but <protocArtifact> usually takes the place of <protocVersion> because of Downloading protoc doesn't respect settings chosen via mvn -s #139 (Fails behind proxy / download mechanism #61).

Fortunately <protocArtifact> supersedes <protocVersion> for protoc resolution so all three settings can be safely specified.

Alternatively, it may be that <includeMavenTypes> can safely and adequately substitute <includeStdTypes> and thereby obviate <protocVersion>. I have not tried this.

dongjoon-hyun added a commit to apache/spark that referenced this issue Oct 28, 2024
### What changes were proposed in this pull request?

This PR aims to fix `protobuf` module `Maven` compilation failure due to the following `protocol-jar-maven-plugin` bug.
- os72/protoc-jar-maven-plugin#104

Note that this happens only at `Maven` compilation.

### Why are the changes needed?

**BEFORE**
```
$ build/mvn -DskipTests clean install -pl connector/protobuf -e -am
...
[INFO] Spark Protobuf ..................................... FAILURE [  7.295 s]
...
[ERROR] Failed to execute goal com.github.os72:protoc-jar-maven-plugin:3.11.4:run (default) on project spark-protobuf_2.13: Execution default of goal com.github.os72:protoc-jar-maven-plugin:3.11.4:run failed:
Cannot read the array length because "<local6>" is null
```

**AFTER**
```
$ build/mvn -DskipTests clean install -pl connector/protobuf -e -am
...
[INFO] Reactor Summary for Spark Project Parent POM 4.0.0-SNAPSHOT:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [  2.601 s]
[INFO] Spark Project Tags ................................. SUCCESS [  5.234 s]
[INFO] Spark Project Sketch ............................... SUCCESS [  5.278 s]
[INFO] Spark Project Common Utils ......................... SUCCESS [ 14.695 s]
[INFO] Spark Project Local DB ............................. SUCCESS [  6.258 s]
[INFO] Spark Project Networking ........................... SUCCESS [  9.384 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [  7.761 s]
[INFO] Spark Project Variant .............................. SUCCESS [  3.304 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [  8.677 s]
[INFO] Spark Project Connect Shims ........................ SUCCESS [  2.690 s]
[INFO] Spark Project Launcher ............................. SUCCESS [  5.618 s]
[INFO] Spark Project Core ................................. SUCCESS [01:11 min]
[INFO] Spark Project SQL API .............................. SUCCESS [ 21.759 s]
[INFO] Spark Project Catalyst ............................. SUCCESS [01:30 min]
[INFO] Spark Project SQL .................................. SUCCESS [01:53 min]
[INFO] Spark Protobuf ..................................... SUCCESS [ 18.285 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  06:26 min
[INFO] Finished at: 2024-10-27T21:01:12-07:00
[INFO] ------------------------------------------------------------------------
```

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Manually run the following command like I did in the above.
```
$ build/mvn -DskipTests clean install -pl connector/protobuf -e -am
```

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #48673 from dongjoon-hyun/SPARK-50143.

Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
LuciferYang pushed a commit to apache/spark that referenced this issue Jan 6, 2025
…-defined protoc

### What changes were proposed in this pull request?
People may use user-defined protoc(as below) for various reasons, for example, if they're on macOS 11, they have to use it since the default version of protoc and protoc-gen-grpc-java plugin do not work on macOS 11.

```
$ export SPARK_PROTOC_EXEC_PATH=/Users/foobar/dev/protoc-4.29.1/protoc
$ export CONNECT_PLUGIN_EXEC_PATH=/Users/foobar/dev/protoc-gen-grpc-java-1.59.1/protoc-gen-grpc-java
$ ./build/mvn -DskipTests clean package -Puser-defined-protoc -DskipDefaultProtoc
```

But this results in compilation failure:

```
[ERROR] Failed to execute goal com.github.os72:protoc-jar-maven-plugin:3.11.4:run (default) on project spark-protobuf_2.13: Execution default of goal com.github.os72:protoc-jar-maven-plugin:3.11.4:run failed: Cannot read the array length because "<local6>" is null
```

This PR aims to fix this compilation failure, the error is caused by `protocol-jar-maven-plugin` bug:

- os72/protoc-jar-maven-plugin#104

### Why are the changes needed?
Before:
```
$ export SPARK_PROTOC_EXEC_PATH=/Users/foobar/dev/protoc-4.29.1/protoc
$ export CONNECT_PLUGIN_EXEC_PATH=/Users/foobar/dev/protoc-gen-grpc-java-1.59.1/protoc-gen-grpc-java
$ ./build/mvn -DskipTests clean package -Puser-defined-protoc -DskipDefaultProtoc
...
[ERROR] Failed to execute goal com.github.os72:protoc-jar-maven-plugin:3.11.4:run (default) on project spark-protobuf_2.13: Execution default of goal com.github.os72:protoc-jar-maven-plugin:3.11.4:run failed: Cannot read the array length because "<local6>" is null
```

After:
```
$ export SPARK_PROTOC_EXEC_PATH=/Users/foobar/dev/protoc-4.29.1/protoc
$ export CONNECT_PLUGIN_EXEC_PATH=/Users/foobar/dev/protoc-gen-grpc-java-1.59.1/protoc-gen-grpc-java
$ ./build/mvn -DskipTests clean package -Puser-defined-protoc -DskipDefaultProtoc
...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Spark Project Parent POM 4.0.0-SNAPSHOT:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [  7.539 s]
[INFO] Spark Project Tags ................................. SUCCESS [  7.527 s]
[INFO] Spark Project Sketch ............................... SUCCESS [  8.413 s]
[INFO] Spark Project Common Utils ......................... SUCCESS [ 29.923 s]
[INFO] Spark Project Local DB ............................. SUCCESS [ 10.732 s]
[INFO] Spark Project Networking ........................... SUCCESS [ 16.691 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 12.653 s]
[INFO] Spark Project Variant .............................. SUCCESS [  5.262 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [ 16.246 s]
[INFO] Spark Project Connect Shims ........................ SUCCESS [  4.356 s]
[INFO] Spark Project Launcher ............................. SUCCESS [  8.666 s]
[INFO] Spark Project Core ................................. SUCCESS [02:56 min]
[INFO] Spark Project ML Local Library ..................... SUCCESS [ 30.063 s]
[INFO] Spark Project GraphX ............................... SUCCESS [ 33.989 s]
[INFO] Spark Project Streaming ............................ SUCCESS [ 51.897 s]
[INFO] Spark Project SQL API .............................. SUCCESS [ 39.508 s]
[INFO] Spark Project Catalyst ............................. SUCCESS [03:15 min]
[INFO] Spark Project SQL .................................. SUCCESS [04:56 min]
[INFO] Spark Project ML Library ........................... SUCCESS [02:30 min]
[INFO] Spark Project Tools ................................ SUCCESS [  4.463 s]
[INFO] Spark Project Hive ................................. SUCCESS [01:33 min]
[INFO] Spark Project Connect Common ....................... SUCCESS [ 55.308 s]
[INFO] Spark Avro ......................................... SUCCESS [ 33.350 s]
[INFO] Spark Protobuf ..................................... SUCCESS [ 38.981 s]
[INFO] Spark Project REPL ................................. SUCCESS [ 21.165 s]
[INFO] Spark Project Connect Server ....................... SUCCESS [ 58.033 s]
[INFO] Spark Project Connect Client ....................... SUCCESS [ 59.078 s]
[INFO] Spark Project Assembly ............................. SUCCESS [  6.703 s]
[INFO] Kafka 0.10+ Token Provider for Streaming ........... SUCCESS [ 20.047 s]
[INFO] Spark Integration for Kafka 0.10 ................... SUCCESS [ 26.882 s]
[INFO] Kafka 0.10+ Source for Structured Streaming ........ SUCCESS [ 41.397 s]
[INFO] Spark Project Examples ............................. SUCCESS [ 41.308 s]
[INFO] Spark Integration for Kafka 0.10 Assembly .......... SUCCESS [ 11.300 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
```

### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
Run command:
```
$ export SPARK_PROTOC_EXEC_PATH=/Users/foobar/dev/protoc-4.29.1/protoc
$ export CONNECT_PLUGIN_EXEC_PATH=/Users/foobar/dev/protoc-gen-grpc-java-1.59.1/protoc-gen-grpc-java
$ ./build/mvn -DskipTests clean package -Puser-defined-protoc -DskipDefaultProtoc
```

### Was this patch authored or co-authored using generative AI tooling?
No.

Closes #49296 from morvenhuang/SPARK-50671.

Authored-by: morvenhuang <[email protected]>
Signed-off-by: yangjie01 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants