Android Gradle conflict with Google firebase-firestore: protobuf-lite nested dependency

[junkomatic]

Describe the bug
I am attempting to test Capillary in a simple Android app (on device), but during the gradle build process I am getting "Error: Program type already present: com.google.protobuf.AnyProto" (or other classes).

I am using Google Firebase-Firestore DB, which implements grpc and uses nested dependency: com.google.protobuf:protobuf-lite:3.0.1.

Also, the google/Capillary library uses nested dependency: com.google.protobuf:protobuf-java:3.4.0.

If I exclude com.google.protobuf from the Capillary lib dependency only, the app will run, then crash when RSAKeyManager is instantiated, with error: "java.lang.NoClassDefFoundError: Failed resolution of: Lcom/google/protobuf/GeneratedMessageV3;" (I will not be implementing any direct client -> capillary service grpc, as both will have access to Firestore DB, and the capillary service will not be exposed to ingress)

To Reproduce
I assume that adding google/Firestore dependencies to the demo/Android app will also cause this build error. It will sync/build in Android Studio, but build error occurs only when the app is run.

Expected behavior
Through some form of gradle dependency exclusions, will I be able to get these packages to run in the same Android app?

Screenshots

I have tried using several combinations of package exclusions, including fully excluding com.google.protobuf from both Firestore and Capillary and implementing either separately:

    implementation('com.google.protobuf:protobuf-java:3.4.0')
    //{OR}
    //implementation('com.google.protobuf:protobuf-lite:3.0.1')

    implementation('com.google.firebase:firebase-firestore:18.2.0', {
        exclude group: 'com.google.protobuf', module: 'protobuf-lite'
    })


    implementation('com.google.capillary:lib-android:1.0.0', {
        exclude group: 'com.google.protobuf', module: 'protobuf-java'
    })

[ejona86]

It seems Capillary should be using protobuf-lite for the generated code in com.google.capillary:lib-android. External-Google has the same restriction as internal-google: android must use lite, non-android (should/generally) use full. And you can't "cross the streams".

If you run into trouble using lite, I may be able to help.

[junkomatic]

I switched the dependencies in both modules 'lib' and 'lib-android' to protobuf-lite, and built those modules. The 'lib' module built fine with no errors. The 'lib-android' module contains a few errors:

These stem from loading the serialized (byte[]) PublicKey in as a proto message. I am not sure yet whether this is absolutely necessary, or could maybe be substituted by a serializable class? Im not really a java pro.

[junkomatic]

I simply added the line:

option optimize_for = LITE_RUNTIME;

to the capillary_internal.proto, and both modules now build without error! I guess now I will try to build the jar and use it in my test projects.

[ejona86]

optimize_for = LITE_RUNTIME is not the appropriate way to use lite; there's no guarantee it will work. You should instead use the lite protoc plugin, com.google.protobuf:protoc-gen-javalite.

[junkomatic]

I lack the knowledge and documentation to apply the plugin: com.google.protobuf:protoc-gen-javalite to the Gradle files myself in any meaningful way. Its always a little unnerving when trying to add things to a project that haven't been maintained in almost 3 years and are not mentioned in any documentation that I can find.

Though, according to the official Protobuf documentation:

"...if the .proto file contains:

option optimize_for = LITE_RUNTIME;

then [message] Foo will include fast implementations of all methods, but will implement the MessageLite interface, which only contains a subset of the methods of Message. In particular, it does not support descriptors or reflection. However, in this mode, the generated code only needs to link against libprotobuf-lite.jar instead of libprotobuf.jar. "
https://developers.google.com/protocol-buffers/docs/reference/java-generated

[ejona86]

I'd file a bug, but protocolbuffers/protobuf#5110 already exists.

Basically, LITE is being rewritten internally and when ready will reemerge as a beautiful butterfly that will dazzle all. But as part of that, LITE is now a completely separate runtime from full proto (they share APIs, but not implementations).

In any case, the optimize_for option has been found to be "the wrong thing," because it's really the consumer of the .proto that should determine what it is optimized for, not the producer (proto author). For example, in Android you want LITE and for server you want full :-D

Using com.google.protobuf:protoc-gen-javalite should be basically the same as using the grpc-java plugin (and replacing "grpc" with "javalite" in the various references). You might find https://github.com/grpc/grpc-java#generated-code useful.

[junkomatic]

ok, I see now that it is a plugin for protoc itself, therefore I should remove the gradle protobuf plugins that automatically generate code, and instead generate the code manually using protoc and then put it in the into the appropriate source directories...

protoc-gen-javalite
readme: https://github.com/protocolbuffers/protobuf/blob/master/java/lite.md
download repo: https://repo1.maven.org/maven2/com/google/protobuf/protoc-gen-javalite/3.0.0/

[ejona86]

You'll want a builtins { remove java }. We do some of this is grpc, but don't look at it too much because we do a lot of complex things :). But the generateProtoTasks seen there is about what yours should look like.

[junkomatic]

Inside the gradle.build for the 'lib' module (this is the only module generating protobuf code) the protobuf{} definition now looks like this:

protobuf {
    protoc {
        artifact = 'com.google.protobuf:protoc:3.0.0'
    }
    plugins {
        javalite {
            artifact = 'com.google.protobuf:protoc-gen-javalite:3.0.0'
        }
    }
    generateProtoTasks {
        all().each { task ->
            task.builtins {
                remove java
            }
            task.plugins {
                javalite { }
            }
        }
    }
}

I am no longer using option optimize _for = LITE_RUNTIME;
I cleaned the 'build' folder in lib and rebuilt, to make sure there was only 'javalite' folder and no 'java' folder in there.
Using dependency: com.google.protobuf:protobuf-lite:3.0.1
Both modules compile, but not tested yet.

[junkomatic]

also the sourceDirs inside the idea{} block needs to be changed to .../javalite instead of .../java.
This seems to be working as far as I can tell.

Ideally, the lib-android module of this library should be independent of the server-side 'lib' module. This way the lib can use the full :protobuf-java dependency, and the 'lib-android' can use :protobuf-lite, with both modules generating the protobuf code independently.

[junkomatic]

Also: In order to run the Capillary Server (lib) alongside the 'com.google.firebase:firebase-admin:6.8.0' dependency required for Firestore, you must also include dependency:
implementation 'io.grpc:grpc-auth:1.19.0'
or else there is an IncompatibleClassChangeError stemming from transitive dependency versions.

[jakett]

With this library, please use "com.google.firebase:firebase-firestore:15.0.0"

If you want to use firestore with higher version, you can use command: "gradlew app: androidDependencies" to identify child components in firestore, and only implement necessary components.

[aquadize]

@jakett I tried your method and it didn't work, since the incompatibility is far beyond include missing classes (or remove duplicates). AFAIK it is not possible to use protobuf-java and firestore newer than 15. I tried 15 and my first Firestore code line throw an exception (the one I use for token auth). Use version 15 is not an option.

Google should provide a compatibility version of Firestore 21.4.3 with protobuf-java 3.11.4.

Unfortunately I am not able to use protobuf-javalite since I need Any message and reflection.

Is there another way to accomplish that? I have to decide to move away from Firestore or to implement the same functionality of the Any message myself.

Regards,
Ariel

[ramonpaolo]

@jakett oh man, thank you so much. I love you bro ❤️