Support for AWS MSK IAM authorization

[geonwood]

Hi! I'm trying to connect to MSK using the Confluent Kafka Connect image. I would like to use MSK's IAM authentication feature during this process, but I'm not sure if this is supported by Confluent Connect.

I'm planning to use the following properties:

security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler

The reason I'm asking is that the connection works fine when I set MSK's encryption method to PLAINTEXT. However, I encountered errors when using the properties mentioned above with MSK. Below is the MSK log I found on CloudWatch:

Failed authentication with /INTERNAL_IP (channelId=INTERNAL_IP-INTERNAL_IP-10293) (SSL handshake failed)

I'm wondering if the Confluent Connect image does not support the AWS_MSK_IAM protocol. Any insights?

[patrickherrera]

You'll probably need to do this yourself by building a new image from cp-kafka-connect-base and adding the jar file from one of the releases: https://github.com/aws/aws-msk-iam-auth/releases

Then inside your Dockerfile, copy it to the right place. I have this code:

COPY --from=downloader /tmp/extracted/config-providers/*.jar /usr/share/java/cp-base-new/

# Looks like AWS IAM Auth *also* needs to be placed directly in the classpath in order for the startup checks to work
# This was gleaned from /etc/confluent/docker/launch within the base image and via https://github.com/confluentinc/kafka-images/issues/186
COPY --from=downloader /tmp/extracted/config-providers/aws-msk-iam-auth*.jar /etc/kafka-connect/jars/

Everything else worked fine after that and I was able to use Kafka Connect against MSK with IAM auth purely based on config properties as you describe