Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reflection registration when building with native-sources #22590

Closed
celsomarques opened this issue Jan 3, 2022 · 23 comments
Closed

Reflection registration when building with native-sources #22590

celsomarques opened this issue Jan 3, 2022 · 23 comments
Labels
area/kafka area/mandrel kind/bug Something isn't working triage/out-of-date This issue/PR is no longer valid or relevant

Comments

@celsomarques
Copy link

Describe the bug

I'm building quarkus native application in two different ways and one works properly and another one don't.

Ok

FROM quay.io/quarkus/ubi-quarkus-mandrel:21.3-java11 AS build
...
RUN ./gradlew build -Dquarkus.package.type=native -Dquarkus.native.native-image-xmx=$BUILD_MEMORY

NOK

## Stage 1: build native sources
FROM gradle:7.3-jdk11 AS gradle-build
...
RUN gradle clean build -Dquarkus.package.type=native-sources


## Stage 2: build quarkus-native
FROM quay.io/quarkus/ubi-quarkus-mandrel:21.3-java11 AS native-build
...
RUN native-image $(cat native-image.args) -J-Xmx$BUILD_MEMORY

@cescoffier investigated this and it seems there are missing reflection needed by Kafka - link

Expected behavior

No response

Actual behavior

No response

How to Reproduce?

Clone this

docker-compose up -d
docker logs -f app-ok
docker logs -f app-nok

Output of uname -a or ver

No response

Output of java -version

No response

GraalVM version (if different from Java)

No response

Quarkus version or git rev

No response

Build tool (ie. output of mvnw --version or gradlew --version)

No response

Additional information

No response

@celsomarques celsomarques added the kind/bug Something isn't working label Jan 3, 2022
@quarkus-bot
Copy link

quarkus-bot bot commented Jan 3, 2022

/cc @Karm, @cescoffier, @galderz, @ozangunalp, @zakkak

@celsomarques celsomarques changed the title Reflection for Kafka building with native-sources Reflection for Kafka - building with native-sources Jan 3, 2022
@cescoffier cescoffier changed the title Reflection for Kafka - building with native-sources Reflection registration when building with native-sources Jan 3, 2022
@cescoffier
Copy link
Member

Renamed the issue, it's not related to Kafka. I don't see where we pass the reflection registration (including the ones for Kafka, but also all the other ones (JSON...)

@cescoffier
Copy link
Member

@geoand @stuartwdouglas any idea?

@geoand
Copy link
Contributor

geoand commented Jan 3, 2022

The reflection registration information (along with a bunch of other stuff) is performed programmatically in the generated feature class, see io.quarkus.runner.AutoFeature in the generated jar.

@cescoffier
Copy link
Member

Hum, does that work when you pass a reflection Config file too?

@geoand
Copy link
Contributor

geoand commented Jan 3, 2022

You mean if the user configures a reflection config file?

@cescoffier
Copy link
Member

@geoand like this: quarkus.native.additional-build-args=-H:ResourceConfigurationFiles=resources-config.json, -H:ReflectionConfigurationFiles=reflection-config.json

The provided reflection-config is very minimal:
https://github.com/celsomarques/quarkus-native-app/blob/master/src/main/resources/reflection-config.json

@geoand
Copy link
Contributor

geoand commented Jan 3, 2022

And the problem is that this additional command line argument is not being passed to GraalVM?

@cescoffier
Copy link
Member

Ok, it's not that. I've removed the file and I still have the issue:

2022-01-03 18:39:46,553 WARN  [org.apa.kaf.cli.NetworkClient] (smallrye-kafka-consumer-thread-0) [Consumer clientId=kafka-consumer-process-status-in, groupId=app-nok] Error connecting to node kafka:9093 (id: -1 rack: null): java.io.IOException: Channel could not be created for socket java.nio.channels.SocketChannel[closed]
	at org.apache.kafka.common.network.Selector.buildAndAttachKafkaChannel(Selector.java:348)
	at org.apache.kafka.common.network.Selector.registerChannel(Selector.java:329)
	at org.apache.kafka.common.network.Selector.connect(Selector.java:256)
	at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:987)
	at org.apache.kafka.clients.NetworkClient.access$600(NetworkClient.java:73)
	at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1158)
	at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1046)
	at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:559)
	at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:265)
	at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236)
	at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:227)
	at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.awaitMetadataUpdate(ConsumerNetworkClient.java:164)
	at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:257)
	at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:480)
	at org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1261)
	at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1230)
	at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1210)
	at io.smallrye.reactive.messaging.kafka.impl.ReactiveKafkaConsumer.lambda$poll$4(ReactiveKafkaConsumer.java:138)
	at io.smallrye.reactive.messaging.kafka.impl.ReactiveKafkaConsumer.lambda$runOnPollingThread$0(ReactiveKafkaConsumer.java:106)
	at io.smallrye.context.impl.wrappers.SlowContextualSupplier.get(SlowContextualSupplier.java:21)
	at io.smallrye.mutiny.operators.uni.builders.UniCreateFromItemSupplier.subscribe(UniCreateFromItemSupplier.java:28)
	at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
	at io.smallrye.mutiny.operators.uni.UniRunSubscribeOn.lambda$subscribe$0(UniRunSubscribeOn.java:27)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.lang.Thread.run(Thread.java:829)
	at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:596)
	at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192)
Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.errors.SaslAuthenticationException: Failed to configure SaslClientAuthenticator
	at org.apache.kafka.common.network.SaslChannelBuilder.buildChannel(SaslChannelBuilder.java:240)
	at org.apache.kafka.common.network.Selector.buildAndAttachKafkaChannel(Selector.java:338)
	... 30 more
Caused by: org.apache.kafka.common.errors.SaslAuthenticationException: Failed to configure SaslClientAuthenticator
Caused by: org.apache.kafka.common.errors.SaslAuthenticationException: Failed to create SaslClient with mechanism PLAIN

So, there is something not passed into native-sources.

@geoand
Copy link
Contributor

geoand commented Jan 3, 2022

I'll have a look tomorrow

@cescoffier
Copy link
Member

The KafkaProcessor does register a type hierarchy:

@BuildStep
    public void withSasl(BuildProducer<ReflectiveClassBuildItem> reflectiveClass,
            BuildProducer<ReflectiveHierarchyBuildItem> reflectiveHierarchy,
            BuildProducer<ExtensionSslNativeSupportBuildItem> sslNativeSupport) {

        reflectiveClass
                .produce(new ReflectiveClassBuildItem(false, false, AbstractLogin.DefaultLoginCallbackHandler.class));
        reflectiveClass.produce(new ReflectiveClassBuildItem(false, false, SaslClientCallbackHandler.class));
        reflectiveClass.produce(new ReflectiveClassBuildItem(false, false, DefaultLogin.class));
        reflectiveClass
                .produce(new ReflectiveClassBuildItem(true, false, false, ScramSaslClient.ScramSaslClientFactory.class));

        // Enable SSL support if kafka.security.protocol is set to something other than PLAINTEXT, which is the default
        String securityProtocol = ConfigProvider.getConfig().getConfigValue("kafka.security.protocol").getValue();
        if (securityProtocol != null && SecurityProtocol.forName(securityProtocol) != SecurityProtocol.PLAINTEXT) {
            sslNativeSupport.produce(new ExtensionSslNativeSupportBuildItem(Feature.KAFKA_CLIENT));
        }

        final Type loginModuleType = Type
                .create(DotName.createSimple(LoginModule.class.getName()), Kind.CLASS);

        reflectiveHierarchy.produce(new ReflectiveHierarchyBuildItem.Builder()
                .type(loginModuleType)
                .source(getClass().getSimpleName() + " > " + loginModuleType.name().toString())
                .build());
    }

If we could find a way to compare both "native" and "native-source" it would help us understand the difference.

@geoand
Copy link
Contributor

geoand commented Jan 4, 2022

And the problem is that this additional command line argument is not being passed to GraalVM?

FWIW, the configuration files are passed to GraalVM.

I'll have a look at what is going on in the generated feature

@geoand
Copy link
Contributor

geoand commented Jan 4, 2022

If we could find a way to compare both "native" and "native-source" it would help us understand the difference.

The only way I can think of is to look at the generated io.quarkus.runner.AutoFeature. When I did so, I saw the classes that the withSasl method registers. For example:

private static void registerClass131(BeforeAnalysisAccess var0) {
        try {
            ClassLoader var1 = Thread.currentThread().getContextClassLoader();
            Class var2 = Class.forName("javax.security.auth.spi.LoginModule", (boolean)0, var1);
            Constructor[] var4 = var2.getDeclaredConstructors();
            Method[] var5 = var2.getDeclaredMethods();
            Field[] var8 = var2.getDeclaredFields();
            Class[] var3 = new Class[]{var2};
            RuntimeReflection.register(var3);
            RuntimeReflection.register((Executable[])var4);
            RuntimeReflection.register((Executable[])var5);
            Version var6 = Version.getCurrent();
            int[] var7 = new int[]{21};
            if (var6.compareTo(var7) < 0) {
                RuntimeReflection.register((boolean)0, var8);
            } else {
                RuntimeReflection.register((boolean)0, (boolean)0, var8);
            }
        } catch (Throwable var9) {
        }

    }

    private static void registerClass132(BeforeAnalysisAccess var0) {
        try {
            ClassLoader var1 = Thread.currentThread().getContextClassLoader();
            Class var2 = Class.forName("org.apache.kafka.common.security.plain.PlainLoginModule", (boolean)0, var1);
            Constructor[] var4 = var2.getDeclaredConstructors();
            Method[] var5 = var2.getDeclaredMethods();
            Field[] var8 = var2.getDeclaredFields();
            Class[] var3 = new Class[]{var2};
            RuntimeReflection.register(var3);
            RuntimeReflection.register((Executable[])var4);
            RuntimeReflection.register((Executable[])var5);
            Version var6 = Version.getCurrent();
            int[] var7 = new int[]{21};
            if (var6.compareTo(var7) < 0) {
                RuntimeReflection.register((boolean)0, var8);
            } else {
                RuntimeReflection.register((boolean)0, (boolean)0, var8);
            }
        } catch (Throwable var9) {
        }

    }

    private static void registerClass133(BeforeAnalysisAccess var0) {
        try {
            ClassLoader var1 = Thread.currentThread().getContextClassLoader();
            Class var2 = Class.forName("org.apache.kafka.common.security.scram.ScramLoginModule", (boolean)0, var1);
            Constructor[] var4 = var2.getDeclaredConstructors();
            Method[] var5 = var2.getDeclaredMethods();
            Field[] var8 = var2.getDeclaredFields();
            Class[] var3 = new Class[]{var2};
            RuntimeReflection.register(var3);
            RuntimeReflection.register((Executable[])var4);
            RuntimeReflection.register((Executable[])var5);
            Version var6 = Version.getCurrent();
            int[] var7 = new int[]{21};
            if (var6.compareTo(var7) < 0) {
                RuntimeReflection.register((boolean)0, var8);
            } else {
                RuntimeReflection.register((boolean)0, (boolean)0, var8);
            }
        } catch (Throwable var9) {
        }

    }

    private static void registerClass134(BeforeAnalysisAccess var0) {
        try {
            ClassLoader var1 = Thread.currentThread().getContextClassLoader();
            Class var2 = Class.forName("org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule", (boolean)0, var1);
            Constructor[] var4 = var2.getDeclaredConstructors();
            Method[] var5 = var2.getDeclaredMethods();
            Field[] var8 = var2.getDeclaredFields();
            Class[] var3 = new Class[]{var2};
            RuntimeReflection.register(var3);
            RuntimeReflection.register((Executable[])var4);
            RuntimeReflection.register((Executable[])var5);
            Version var6 = Version.getCurrent();
            int[] var7 = new int[]{21};
            if (var6.compareTo(var7) < 0) {
                RuntimeReflection.register((boolean)0, var8);
            } else {
                RuntimeReflection.register((boolean)0, (boolean)0, var8);
            }
        } catch (Throwable var9) {
        }

    }

@cescoffier
Copy link
Member

What about the NativeImageSecurityProviderBuildItem ?

@BuildStep
    void addSaslProvidersToNativeImage(BuildProducer<NativeImageSecurityProviderBuildItem> additionalProviders) {
        for (String provider : SASL_PROVIDERS) {
            additionalProviders.produce(new NativeImageSecurityProviderBuildItem(provider));
        }
    }

With:

private static final Set<String> SASL_PROVIDERS = Arrays.stream(new String[] {
            "com.sun.security.sasl.Provider",
            "org.apache.kafka.common.security.scram.internals.ScramSaslClientProvider",
            "org.apache.kafka.common.security.oauthbearer.internals.OAuthBearerSaslClientProvider"
    }).collect(Collectors.toSet());

Maybe also the Elyton stuff, but the reproducer is a Gradle project, so it's hard for me to understand what's pulled.

@geoand
Copy link
Contributor

geoand commented Jan 4, 2022

What about the NativeImageSecurityProviderBuildItem ?

Very good question! Those are simply mapped to -H:AdditionalSecurityProviders on the command line.

It seems like the reproducer is using an old Quarkus version that did not include #21809. Can you try with 2.6.1.Final? It should fix the problem by also including the proper -H:AdditionalSecurityProviders on the generated command.

@geoand
Copy link
Contributor

geoand commented Jan 4, 2022

@cescoffier should we close this as out of date?

@cescoffier
Copy link
Member

@geoand I think so.

@geoand geoand closed this as completed Jan 4, 2022
@geoand geoand added the triage/out-of-date This issue/PR is no longer valid or relevant label Jan 4, 2022
@celsomarques
Copy link
Author

@geoand / @cescoffier, bump version solves the issue!
Sorry about that and thanks a lot!

@geoand
Copy link
Contributor

geoand commented Jan 4, 2022

No problem!

@cmtoan
Copy link

cmtoan commented Jan 24, 2024

Hello,
I have the same probleme with "-H:AdditionalSecurityProviders=software.amazon.msk.auth.iam.internals.IAMSaslClientProvider"

And I have following errors in AWS see here :
Caused by: org.apache.kafka.common.errors.SaslAuthenticationException: Failed to configure SaslClientAuthenticator
Caused by: org.apache.kafka.common.errors.SaslAuthenticationException: Failed to create SaslClient with mechanism AWS_MSK_IAM

Do you have a solution to build native aws-msk-iam-auth with quarkus?

@jvdadda
Copy link

jvdadda commented Mar 18, 2024

@cmtoan Have you succeed to make it work ?

@cmtoan
Copy link

cmtoan commented Mar 19, 2024

@jvdadda I didn't succeed to make it work. I used the mechanism SCRAM-SHA-512 instead.

@jvdadda
Copy link

jvdadda commented Mar 19, 2024

@jvdadda I didn't succeed to make it work. I used the mechanism SCRAM-SHA-512 instead.

@cmtoan Thanks for the reply !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/kafka area/mandrel kind/bug Something isn't working triage/out-of-date This issue/PR is no longer valid or relevant
Projects
None yet
Development

No branches or pull requests

5 participants