Update KRaft SCRAM user creation to support secrets protection (master)

[justinrlee]

Description

Support KRaft SCRAM user creation when confluent secrets are enabled

Fixes # (issue)

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

How Has This Been Tested?

Manually tested

Checklist:

• Any variable/code changes have been validated to be backwards compatible (doesn't break upgrade)
• I have added tests that prove my fix is effective or that my feature works
• If required, I have ensured the changes can be discovered by cp-ansible discovery codebase
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• Any dependent changes have been merged and published in downstream modules

[confluent-cla-assistant]

🎉 All Contributor License Agreements have been signed. Ready to merge.
_{Please push an empty commit if you would like to re-run the checks to verify CLA status for all contributors.}

[justinrlee]

Error without this change:

TASK [confluent.platform.kafka_broker : Create SCRAM Users with KRaft] *******************************************************************************************************************************************************************************************************
Tuesday 10 December 2024  23:50:08 +0000 (0:00:00.120)       0:06:03.464 ******
Tuesday 10 December 2024  23:50:08 +0000 (0:00:00.120)       0:06:03.464 ******
failed: [ip-10-38-1-10.ap-southeast-1.compute.internal] (item={'key': 'admin', 'value': {'principal': 'kafka', 'password': 'kafka1'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "/opt/confluent/confluent-7.8.0/bin/kafka-configs  --bootstrap-server ip-10-38-1-10.ap-southeast-1.compute.internal:9091  --command-config /opt/confluent/etc/kafka/client.properties  --alter --add-config 'SCRAM-SHA-512=[password=kafka1]'  --entity-type users --entity-name kafka\n", "delta": "0:00:02.581914", "end": "2024-12-10 23:50:13.263382", "item": {"key": "admin", "value": {"password": "kafka1", "principal": "kafka"}}, "msg": "non-zero return code", "rc": 1, "start": "2024-12-10 23:50:10.681468", "stderr": "[2024-12-10 23:50:12,923] ERROR Failed to load master key from environment variable. (io.confluent.kafka.security.config.provider.DecryptionEngine)\n[2024-12-10 23:50:12,924] ERROR Failed to initialize the decryption engine (io.confluent.kafka.security.config.provider.DecryptionEngine)\norg.apache.kafka.common.config.ConfigException: Failed to load master key from environment variable.\n\tat io.confluent.kafka.security.config.provider.DecryptionEngine.loadMasterKey(DecryptionEngine.java:61)\n\tat io.confluent.kafka.security.config.provider.DecryptionEngine.<init>(DecryptionEngine.java:38)\n\tat io.confluent.kafka.security.config.provider.SecurePassConfigProvider.initializeDecryptionEngine(SecurePassConfigProvider.java:61)\n\tat io.confluent.kafka.security.config.provider.SecurePassConfigProvider.get(SecurePassConfigProvider.java:132)\n\tat org.apache.kafka.common.config.ConfigTransformer.transform(ConfigTransformer.java:103)\n\tat org.apache.kafka.common.config.AbstractConfig.resolveConfigVariables(AbstractConfig.java:552)\n\tat org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:115)\n\tat org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:149)\n\tat org.apache.kafka.clients.admin.AdminClientConfig.<init>(AdminClientConfig.java:357)\n\tat org.apache.kafka.clients.admin.Admin.create(Admin.java:137)\n\tat kafka.admin.ConfigCommand$.processCommand(ConfigCommand.scala:367)\n\tat kafka.admin.ConfigCommand$.main(ConfigCommand.scala:105)\n\tat kafka.admin.ConfigCommand.main(ConfigCommand.scala)\n[2024-12-10 23:50:12,926] WARN Failed to initialize the decryption engine. (io.confluent.kafka.security.config.provider.SecurePassConfigProvider)\nError while executing config command with args '--bootstrap-server ip-10-38-1-10.ap-southeast-1.compute.internal:9091 --command-config /opt/confluent/etc/kafka/client.properties --alter --add-config SCRAM-SHA-512=[password=kafka1] --entity-type users --entity-name kafka'\norg.apache.kafka.common.config.ConfigException: Invalid value org.apache.kafka.common.config.ConfigException: Invalid value org.apache.kafka.common.config.ConfigException: Failed to load master key from environment variable. for configuration Failed to initialize the decryption engine for configuration Failed to initialize the decryption engine.\n\tat io.confluent.kafka.security.config.provider.SecurePassConfigProvider.initializeDecryptionEngine(SecurePassConfigProvider.java:64)\n\tat io.confluent.kafka.security.config.provider.SecurePassConfigProvider.get(SecurePassConfigProvider.java:132)\n\tat org.apache.kafka.common.config.ConfigTransformer.transform(ConfigTransformer.java:103)\n\tat org.apache.kafka.common.config.AbstractConfig.resolveConfigVariables(AbstractConfig.java:552)\n\tat org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:115)\n\tat org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:149)\n\tat org.apache.kafka.clients.admin.AdminClientConfig.<init>(AdminClientConfig.java:357)\n\tat org.apache.kafka.clients.admin.Admin.create(Admin.java:137)\n\tat kafka.admin.ConfigCommand$.processCommand(ConfigCommand.scala:367)\n\tat kafka.admin.ConfigCommand$.main(ConfigCommand.scala:105)\n\tat kafka.admin.ConfigCommand.main(ConfigCommand.scala)", "stderr_lines": ["[2024-12-10 23:50:12,923] ERROR Failed to load master key from environment variable. (io.confluent.kafka.security.config.provider.DecryptionEngine)", "[2024-12-10 23:50:12,924] ERROR Failed to initialize the decryption engine (io.confluent.kafka.security.config.provider.DecryptionEngine)", "org.apache.kafka.common.config.ConfigException: Failed to load master key from environment variable.", "\tat io.confluent.kafka.security.config.provider.DecryptionEngine.loadMasterKey(DecryptionEngine.java:61)", "\tat io.confluent.kafka.security.config.provider.DecryptionEngine.<init>(DecryptionEngine.java:38)", "\tat io.confluent.kafka.security.config.provider.SecurePassConfigProvider.initializeDecryptionEngine(SecurePassConfigProvider.java:61)", "\tat io.confluent.kafka.security.config.provider.SecurePassConfigProvider.get(SecurePassConfigProvider.java:132)", "\tat org.apache.kafka.common.config.ConfigTransformer.transform(ConfigTransformer.java:103)", "\tat org.apache.kafka.common.config.AbstractConfig.resolveConfigVariables(AbstractConfig.java:552)", "\tat org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:115)", "\tat org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:149)", "\tat org.apache.kafka.clients.admin.AdminClientConfig.<init>(AdminClientConfig.java:357)", "\tat org.apache.kafka.clients.admin.Admin.create(Admin.java:137)", "\tat kafka.admin.ConfigCommand$.processCommand(ConfigCommand.scala:367)", "\tat kafka.admin.ConfigCommand$.main(ConfigCommand.scala:105)", "\tat kafka.admin.ConfigCommand.main(ConfigCommand.scala)", "[2024-12-10 23:50:12,926] WARN Failed to initialize the decryption engine. (io.confluent.kafka.security.config.provider.SecurePassConfigProvider)", "Error while executing config command with args '--bootstrap-server ip-10-38-1-10.ap-southeast-1.compute.internal:9091 --command-config /opt/confluent/etc/kafka/client.properties --alter --add-config SCRAM-SHA-512=[password=kafka1] --entity-type users --entity-name kafka'", "org.apache.kafka.common.config.ConfigException: Invalid value org.apache.kafka.common.config.ConfigException: Invalid value org.apache.kafka.common.config.ConfigException: Failed to load master key from environment variable. for configuration Failed to initialize the decryption engine for configuration Failed to initialize the decryption engine.", "\tat io.confluent.kafka.security.config.provider.SecurePassConfigProvider.initializeDecryptionEngine(SecurePassConfigProvider.java:64)", "\tat io.confluent.kafka.security.config.provider.SecurePassConfigProvider.get(SecurePassConfigProvider.java:132)", "\tat org.apache.kafka.common.config.ConfigTransformer.transform(ConfigTransformer.java:103)", "\tat org.apache.kafka.common.config.AbstractConfig.resolveConfigVariables(AbstractConfig.java:552)", "\tat org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:115)", "\tat org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:149)", "\tat org.apache.kafka.clients.admin.AdminClientConfig.<init>(AdminClientConfig.java:357)", "\tat org.apache.kafka.clients.admin.Admin.create(Admin.java:137)", "\tat kafka.admin.ConfigCommand$.processCommand(ConfigCommand.scala:367)", "\tat kafka.admin.ConfigCommand$.main(ConfigCommand.scala:105)", "\tat kafka.admin.ConfigCommand.main(ConfigCommand.scala)"], "stdout": "", "stdout_lines": []}
failed: [ip-10-38-1-10.ap-southeast-1.compute.internal] (item={'key': 'client', 'value': {'principal': 'client', 'password': 'client-secret'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "/opt/confluent/confluent-7.8.0/bin/kafka-configs  --bootstrap-server ip-10-38-1-10.ap-southeast-1.compute.internal:9091  --command-config /opt/confluent/etc/kafka/client.properties  --alter --add-config 'SCRAM-SHA-512=[password=client-secret]'  --entity-type users --entity-name client\n", "delta": "0:00:02.564595", "end": "2024-12-10 23:50:18.034530", "item": {"key": "client", "value": {"password": "client-secret", "principal": "client"}}, "msg": "non-zero return code", "rc": 1, "start": "2024-12-10 23:50:15.469935", "stderr": "[2024-12-10 23:50:17,689] ERROR Failed to load master key from environment variable. (io.confluent.kafka.security.config.provider.DecryptionEngine)\n[2024-12-10 23:50:17,690] ERROR Failed to initialize the decryption engine (io.confluent.kafka.security.config.provider.DecryptionEngine)\norg.apache.kafka.common.config.ConfigException: Failed to load master key from environment variable.\n\tat io.confluent.kafka.security.config.provider.DecryptionEngine.loadMasterKey(DecryptionEngine.java:61)\n\tat io.confluent.kafka.security.config.provider.DecryptionEngine.<init>(DecryptionEngine.java:38)\n\tat io.confluent.kafka.security.config.provider.SecurePassConfigProvider.initializeDecryptionEngine(SecurePassConfigProvider.java:61)\n\tat io.confluent.kafka.security.config.provider.SecurePassConfigProvider.get(SecurePassConfigProvider.java:132)\n\tat org.apache.kafka.common.config.ConfigTransformer.transform(ConfigTransformer.java:103)\n\tat org.apache.kafka.common.config.AbstractConfig.resolveConfigVariables(AbstractConfig.java:552)\n\tat org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:115)\n\tat org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:149)\n\tat org.apache.kafka.clients.admin.AdminClientConfig.<init>(AdminClientConfig.java:357)\n\tat org.apache.kafka.clients.admin.Admin.create(Admin.java:137)\n\tat kafka.admin.ConfigCommand$.processCommand(ConfigCommand.scala:367)\n\tat kafka.admin.ConfigCommand$.main(ConfigCommand.scala:105)\n\tat kafka.admin.ConfigCommand.main(ConfigCommand.scala)\n[2024-12-10 23:50:17,693] WARN Failed to initialize the decryption engine. (io.confluent.kafka.security.config.provider.SecurePassConfigProvider)\nError while executing config command with args '--bootstrap-server ip-10-38-1-10.ap-southeast-1.compute.internal:9091 --command-config /opt/confluent/etc/kafka/client.properties --alter --add-config SCRAM-SHA-512=[password=client-secret] --entity-type users --entity-name client'\norg.apache.kafka.common.config.ConfigException: Invalid value org.apache.kafka.common.config.ConfigException: Invalid value org.apache.kafka.common.config.ConfigException: Failed to load master key from environment variable. for configuration Failed to initialize the decryption engine for configuration Failed to initialize the decryption engine.\n\tat io.confluent.kafka.security.config.provider.SecurePassConfigProvider.initializeDecryptionEngine(SecurePassConfigProvider.java:64)\n\tat io.confluent.kafka.security.config.provider.SecurePassConfigProvider.get(SecurePassConfigProvider.java:132)\n\tat org.apache.kafka.common.config.ConfigTransformer.transform(ConfigTransformer.java:103)\n\tat org.apache.kafka.common.config.AbstractConfig.resolveConfigVariables(AbstractConfig.java:552)\n\tat org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:115)\n\tat org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:149)\n\tat org.apache.kafka.clients.admin.AdminClientConfig.<init>(AdminClientConfig.java:357)\n\tat org.apache.kafka.clients.admin.Admin.create(Admin.java:137)\n\tat kafka.admin.ConfigCommand$.processCommand(ConfigCommand.scala:367)\n\tat kafka.admin.ConfigCommand$.main(ConfigCommand.scala:105)\n\tat kafka.admin.ConfigCommand.main(ConfigCommand.scala)", "stderr_lines": ["[2024-12-10 23:50:17,689] ERROR Failed to load master key from environment variable. (io.confluent.kafka.security.config.provider.DecryptionEngine)", "[2024-12-10 23:50:17,690] ERROR Failed to initialize the decryption engine (io.confluent.kafka.security.config.provider.DecryptionEngine)", "org.apache.kafka.common.config.ConfigException: Failed to load master key from environment variable.", "\tat io.confluent.kafka.security.config.provider.DecryptionEngine.loadMasterKey(DecryptionEngine.java:61)", "\tat io.confluent.kafka.security.config.provider.DecryptionEngine.<init>(DecryptionEngine.java:38)", "\tat io.confluent.kafka.security.config.provider.SecurePassConfigProvider.initializeDecryptionEngine(SecurePassConfigProvider.java:61)", "\tat io.confluent.kafka.security.config.provider.SecurePassConfigProvider.get(SecurePassConfigProvider.java:132)", "\tat org.apache.kafka.common.config.ConfigTransformer.transform(ConfigTransformer.java:103)", "\tat org.apache.kafka.common.config.AbstractConfig.resolveConfigVariables(AbstractConfig.java:552)", "\tat org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:115)", "\tat org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:149)", "\tat org.apache.kafka.clients.admin.AdminClientConfig.<init>(AdminClientConfig.java:357)", "\tat org.apache.kafka.clients.admin.Admin.create(Admin.java:137)", "\tat kafka.admin.ConfigCommand$.processCommand(ConfigCommand.scala:367)", "\tat kafka.admin.ConfigCommand$.main(ConfigCommand.scala:105)", "\tat kafka.admin.ConfigCommand.main(ConfigCommand.scala)", "[2024-12-10 23:50:17,693] WARN Failed to initialize the decryption engine. (io.confluent.kafka.security.config.provider.SecurePassConfigProvider)", "Error while executing config command with args '--bootstrap-server ip-10-38-1-10.ap-southeast-1.compute.internal:9091 --command-config /opt/confluent/etc/kafka/client.properties --alter --add-config SCRAM-SHA-512=[password=client-secret] --entity-type users --entity-name client'", "org.apache.kafka.common.config.ConfigException: Invalid value org.apache.kafka.common.config.ConfigException: Invalid value org.apache.kafka.common.config.ConfigException: Failed to load master key from environment variable. for configuration Failed to initialize the decryption engine for configuration Failed to initialize the decryption engine.", "\tat io.confluent.kafka.security.config.provider.SecurePassConfigProvider.initializeDecryptionEngine(SecurePassConfigProvider.java:64)", "\tat io.confluent.kafka.security.config.provider.SecurePassConfigProvider.get(SecurePassConfigProvider.java:132)", "\tat org.apache.kafka.common.config.ConfigTransformer.transform(ConfigTransformer.java:103)", "\tat org.apache.kafka.common.config.AbstractConfig.resolveConfigVariables(AbstractConfig.java:552)", "\tat org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:115)", "\tat org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:149)", "\tat org.apache.kafka.clients.admin.AdminClientConfig.<init>(AdminClientConfig.java:357)", "\tat org.apache.kafka.clients.admin.Admin.create(Admin.java:137)", "\tat kafka.admin.ConfigCommand$.processCommand(ConfigCommand.scala:367)", "\tat kafka.admin.ConfigCommand$.main(ConfigCommand.scala:105)", "\tat kafka.admin.ConfigCommand.main(ConfigCommand.scala)"], "stdout": "", "stdout_lines": []}
failed: [ip-10-38-1-10.ap-southeast-1.compute.internal] (item={'key': 'control_center', 'value': {'principal': 'control-center', 'password': 'c3p'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "/opt/confluent/confluent-7.8.0/bin/kafka-configs  --bootstrap-server ip-10-38-1-10.ap-southeast-1.compute.internal:9091  --command-config /opt/confluent/etc/kafka/client.properties  --alter --add-config 'SCRAM-SHA-512=[password=c3p]'  --entity-type users --entity-name control-center\n", "delta": "0:00:02.425001", "end": "2024-12-10 23:50:22.517765", "item": {"key": "control_center", "value": {"password": "c3p", "principal": "control-center"}}, "msg": "non-zero return code", "rc": 1, "start": "2024-12-10 23:50:20.092764", "stderr": "[2024-12-10 23:50:22,179] ERROR Failed to load master key from environment variable. (io.confluent.kafka.security.config.provider.DecryptionEngine)\n[2024-12-10 23:50:22,180] ERROR Failed to initialize the decryption engine (io.confluent.kafka.security.config.provider.DecryptionEngine)\norg.apache.kafka.common.config.ConfigException: Failed to load master key from environment variable.\n\tat io.confluent.kafka.security.config.provider.DecryptionEngine.loadMasterKey(DecryptionEngine.java:61)\n\tat io.confluent.kafka.security.config.provider.DecryptionEngine.<init>(DecryptionEngine.java:38)\n\tat io.confluent.kafka.security.config.provider.SecurePassConfigProvider.initializeDecryptionEngine(SecurePassConfigProvider.java:61)\n\tat io.confluent.kafka.security.config.provider.SecurePassConfigProvider.get(SecurePassConfigProvider.java:132)\n\tat org.apache.kafka.common.config.ConfigTransformer.transform(ConfigTransformer.java:103)\n\tat org.apache.kafka.common.config.AbstractConfig.resolveConfigVariables(AbstractConfig.java:552)\n\tat org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:115)\n\tat org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:149)\n\tat org.apache.kafka.clients.admin.AdminClientConfig.<init>(AdminClientConfig.java:357)\n\tat org.apache.kafka.clients.admin.Admin.create(Admin.java:137)\n\tat kafka.admin.ConfigCommand$.processCommand(ConfigCommand.scala:367)\n\tat kafka.admin.ConfigCommand$.main(ConfigCommand.scala:105)\n\tat kafka.admin.ConfigCommand.main(ConfigCommand.scala)\n[2024-12-10 23:50:22,182] WARN Failed to initialize the decryption engine. (io.confluent.kafka.security.config.provider.SecurePassConfigProvider)\nError while executing config command with args '--bootstrap-server ip-10-38-1-10.ap-southeast-1.compute.internal:9091 --command-config /opt/confluent/etc/kafka/client.properties --alter --add-config SCRAM-SHA-512=[password=c3p] --entity-type users --entity-name control-center'\norg.apache.kafka.common.config.ConfigException: Invalid value org.apache.kafka.common.config.ConfigException: Invalid value org.apache.kafka.common.config.ConfigException: Failed to load master key from environment variable. for configuration Failed to initialize the decryption engine for configuration Failed to initialize the decryption engine.\n\tat io.confluent.kafka.security.config.provider.SecurePassConfigProvider.initializeDecryptionEngine(SecurePassConfigProvider.java:64)\n\tat io.confluent.kafka.security.config.provider.SecurePassConfigProvider.get(SecurePassConfigProvider.java:132)\n\tat org.apache.kafka.common.config.ConfigTransformer.transform(ConfigTransformer.java:103)\n\tat org.apache.kafka.common.config.AbstractConfig.resolveConfigVariables(AbstractConfig.java:552)\n\tat org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:115)\n\tat org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:149)\n\tat org.apache.kafka.clients.admin.AdminClientConfig.<init>(AdminClientConfig.java:357)\n\tat org.apache.kafka.clients.admin.Admin.create(Admin.java:137)\n\tat kafka.admin.ConfigCommand$.processCommand(ConfigCommand.scala:367)\n\tat kafka.admin.ConfigCommand$.main(ConfigCommand.scala:105)\n\tat kafka.admin.ConfigCommand.main(ConfigCommand.scala)", "stderr_lines": ["[2024-12-10 23:50:22,179] ERROR Failed to load master key from environment variable. (io.confluent.kafka.security.config.provider.DecryptionEngine)", "[2024-12-10 23:50:22,180] ERROR Failed to initialize the decryption engine (io.confluent.kafka.security.config.provider.DecryptionEngine)", "org.apache.kafka.common.config.ConfigException: Failed to load master key from environment variable.", "\tat io.confluent.kafka.security.config.provider.DecryptionEngine.loadMasterKey(DecryptionEngine.java:61)", "\tat io.confluent.kafka.security.config.provider.DecryptionEngine.<init>(DecryptionEngine.java:38)", "\tat io.confluent.kafka.security.config.provider.SecurePassConfigProvider.initializeDecryptionEngine(SecurePassConfigProvider.java:61)", "\tat io.confluent.kafka.security.config.provider.SecurePassConfigProvider.get(SecurePassConfigProvider.java:132)", "\tat org.apache.kafka.common.config.ConfigTransformer.transform(ConfigTransformer.java:103)", "\tat org.apache.kafka.common.config.AbstractConfig.resolveConfigVariables(AbstractConfig.java:552)", "\tat org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:115)", "\tat org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:149)", "\tat org.apache.kafka.clients.admin.AdminClientConfig.<init>(AdminClientConfig.java:357)", "\tat org.apache.kafka.clients.admin.Admin.create(Admin.java:137)", "\tat kafka.admin.ConfigCommand$.processCommand(ConfigCommand.scala:367)", "\tat kafka.admin.ConfigCommand$.main(ConfigCommand.scala:105)", "\tat kafka.admin.ConfigCommand.main(ConfigCommand.scala)", "[2024-12-10 23:50:22,182] WARN Failed to initialize the decryption engine. (io.confluent.kafka.security.config.provider.SecurePassConfigProvider)", "Error while executing config command with args '--bootstrap-server ip-10-38-1-10.ap-southeast-1.compute.internal:9091 --command-config /opt/confluent/etc/kafka/client.properties --alter --add-config SCRAM-SHA-512=[password=c3p] --entity-type users --entity-name control-center'", "org.apache.kafka.common.config.ConfigException: Invalid value org.apache.kafka.common.config.ConfigException: Invalid value org.apache.kafka.common.config.ConfigException: Failed to load master key from environment variable. for configuration Failed to initialize the decryption engine for configuration Failed to initialize the decryption engine.", "\tat io.confluent.kafka.security.config.provider.SecurePassConfigProvider.initializeDecryptionEngine(SecurePassConfigProvider.java:64)", "\tat io.confluent.kafka.security.config.provider.SecurePassConfigProvider.get(SecurePassConfigProvider.java:132)", "\tat org.apache.kafka.common.config.ConfigTransformer.transform(ConfigTransformer.java:103)", "\tat org.apache.kafka.common.config.AbstractConfig.resolveConfigVariables(AbstractConfig.java:552)", "\tat org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:115)", "\tat org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:149)", "\tat org.apache.kafka.clients.admin.AdminClientConfig.<init>(AdminClientConfig.java:357)", "\tat org.apache.kafka.clients.admin.Admin.create(Admin.java:137)", "\tat kafka.admin.ConfigCommand$.processCommand(ConfigCommand.scala:367)", "\tat kafka.admin.ConfigCommand$.main(ConfigCommand.scala:105)", "\tat kafka.admin.ConfigCommand.main(ConfigCommand.scala)"], "stdout": "", "stdout_lines": []}
failed: [ip-10-38-1-10.ap-southeast-1.compute.internal] (item={'key': 'user1', 'value': {'principal': 'user1', 'password': 'u1p'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "/opt/confluent/confluent-7.8.0/bin/kafka-configs  --bootstrap-server ip-10-38-1-10.ap-southeast-1.compute.internal:9091  --command-config /opt/confluent/etc/kafka/client.properties  --alter --add-config 'SCRAM-SHA-512=[password=u1p]'  --entity-type users --entity-name user1\n", "delta": "0:00:02.677105", "end": "2024-12-10 23:50:27.242204", "item": {"key": "user1", "value": {"password": "u1p", "principal": "user1"}}, "msg": "non-zero return code", "rc": 1, "start": "2024-12-10 23:50:24.565099", "stderr": "[2024-12-10 23:50:26,879] ERROR Failed to load master key from environment variable. (io.confluent.kafka.security.config.provider.DecryptionEngine)\n[2024-12-10 23:50:26,880] ERROR Failed to initialize the decryption engine (io.confluent.kafka.security.config.provider.DecryptionEngine)\norg.apache.kafka.common.config.ConfigException: Failed to load master key from environment variable.\n\tat io.confluent.kafka.security.config.provider.DecryptionEngine.loadMasterKey(DecryptionEngine.java:61)\n\tat io.confluent.kafka.security.config.provider.DecryptionEngine.<init>(DecryptionEngine.java:38)\n\tat io.confluent.kafka.security.config.provider.SecurePassConfigProvider.initializeDecryptionEngine(SecurePassConfigProvider.java:61)\n\tat io.confluent.kafka.security.config.provider.SecurePassConfigProvider.get(SecurePassConfigProvider.java:132)\n\tat org.apache.kafka.common.config.ConfigTransformer.transform(ConfigTransformer.java:103)\n\tat org.apache.kafka.common.config.AbstractConfig.resolveConfigVariables(AbstractConfig.java:552)\n\tat org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:115)\n\tat org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:149)\n\tat org.apache.kafka.clients.admin.AdminClientConfig.<init>(AdminClientConfig.java:357)\n\tat org.apache.kafka.clients.admin.Admin.create(Admin.java:137)\n\tat kafka.admin.ConfigCommand$.processCommand(ConfigCommand.scala:367)\n\tat kafka.admin.ConfigCommand$.main(ConfigCommand.scala:105)\n\tat kafka.admin.ConfigCommand.main(ConfigCommand.scala)\n[2024-12-10 23:50:26,890] WARN Failed to initialize the decryption engine. (io.confluent.kafka.security.config.provider.SecurePassConfigProvider)\nError while executing config command with args '--bootstrap-server ip-10-38-1-10.ap-southeast-1.compute.internal:9091 --command-config /opt/confluent/etc/kafka/client.properties --alter --add-config SCRAM-SHA-512=[password=u1p] --entity-type users --entity-name user1'\norg.apache.kafka.common.config.ConfigException: Invalid value org.apache.kafka.common.config.ConfigException: Invalid value org.apache.kafka.common.config.ConfigException: Failed to load master key from environment variable. for configuration Failed to initialize the decryption engine for configuration Failed to initialize the decryption engine.\n\tat io.confluent.kafka.security.config.provider.SecurePassConfigProvider.initializeDecryptionEngine(SecurePassConfigProvider.java:64)\n\tat io.confluent.kafka.security.config.provider.SecurePassConfigProvider.get(SecurePassConfigProvider.java:132)\n\tat org.apache.kafka.common.config.ConfigTransformer.transform(ConfigTransformer.java:103)\n\tat org.apache.kafka.common.config.AbstractConfig.resolveConfigVariables(AbstractConfig.java:552)\n\tat org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:115)\n\tat org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:149)\n\tat org.apache.kafka.clients.admin.AdminClientConfig.<init>(AdminClientConfig.java:357)\n\tat org.apache.kafka.clients.admin.Admin.create(Admin.java:137)\n\tat kafka.admin.ConfigCommand$.processCommand(ConfigCommand.scala:367)\n\tat kafka.admin.ConfigCommand$.main(ConfigCommand.scala:105)\n\tat kafka.admin.ConfigCommand.main(ConfigCommand.scala)", "stderr_lines": ["[2024-12-10 23:50:26,879] ERROR Failed to load master key from environment variable. (io.confluent.kafka.security.config.provider.DecryptionEngine)", "[2024-12-10 23:50:26,880] ERROR Failed to initialize the decryption engine (io.confluent.kafka.security.config.provider.DecryptionEngine)", "org.apache.kafka.common.config.ConfigException: Failed to load master key from environment variable.", "\tat io.confluent.kafka.security.config.provider.DecryptionEngine.loadMasterKey(DecryptionEngine.java:61)", "\tat io.confluent.kafka.security.config.provider.DecryptionEngine.<init>(DecryptionEngine.java:38)", "\tat io.confluent.kafka.security.config.provider.SecurePassConfigProvider.initializeDecryptionEngine(SecurePassConfigProvider.java:61)", "\tat io.confluent.kafka.security.config.provider.SecurePassConfigProvider.get(SecurePassConfigProvider.java:132)", "\tat org.apache.kafka.common.config.ConfigTransformer.transform(ConfigTransformer.java:103)", "\tat org.apache.kafka.common.config.AbstractConfig.resolveConfigVariables(AbstractConfig.java:552)", "\tat org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:115)", "\tat org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:149)", "\tat org.apache.kafka.clients.admin.AdminClientConfig.<init>(AdminClientConfig.java:357)", "\tat org.apache.kafka.clients.admin.Admin.create(Admin.java:137)", "\tat kafka.admin.ConfigCommand$.processCommand(ConfigCommand.scala:367)", "\tat kafka.admin.ConfigCommand$.main(ConfigCommand.scala:105)", "\tat kafka.admin.ConfigCommand.main(ConfigCommand.scala)", "[2024-12-10 23:50:26,890] WARN Failed to initialize the decryption engine. (io.confluent.kafka.security.config.provider.SecurePassConfigProvider)", "Error while executing config command with args '--bootstrap-server ip-10-38-1-10.ap-southeast-1.compute.internal:9091 --command-config /opt/confluent/etc/kafka/client.properties --alter --add-config SCRAM-SHA-512=[password=u1p] --entity-type users --entity-name user1'", "org.apache.kafka.common.config.ConfigException: Invalid value org.apache.kafka.common.config.ConfigException: Invalid value org.apache.kafka.common.config.ConfigException: Failed to load master key from environment variable. for configuration Failed to initialize the decryption engine for configuration Failed to initialize the decryption engine.", "\tat io.confluent.kafka.security.config.provider.SecurePassConfigProvider.initializeDecryptionEngine(SecurePassConfigProvider.java:64)", "\tat io.confluent.kafka.security.config.provider.SecurePassConfigProvider.get(SecurePassConfigProvider.java:132)", "\tat org.apache.kafka.common.config.ConfigTransformer.transform(ConfigTransformer.java:103)", "\tat org.apache.kafka.common.config.AbstractConfig.resolveConfigVariables(AbstractConfig.java:552)", "\tat org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:115)", "\tat org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:149)", "\tat org.apache.kafka.clients.admin.AdminClientConfig.<init>(AdminClientConfig.java:357)", "\tat org.apache.kafka.clients.admin.Admin.create(Admin.java:137)", "\tat kafka.admin.ConfigCommand$.processCommand(ConfigCommand.scala:367)", "\tat kafka.admin.ConfigCommand$.main(ConfigCommand.scala:105)", "\tat kafka.admin.ConfigCommand.main(ConfigCommand.scala)"], "stdout": "", "stdout_lines": []}

Task run after change:

TASK [confluent.platform.kafka_broker : Create SCRAM Users with KRaft with Secrets Protection enabled] ***********************************************************************************************************************************************************************
Wednesday 11 December 2024  00:02:28 +0000 (0:00:00.107)       0:06:06.186 ****
Wednesday 11 December 2024  00:02:28 +0000 (0:00:00.107)       0:06:06.185 ****
changed: [ip-10-38-1-10.ap-southeast-1.compute.internal] => (item={'key': 'admin', 'value': {'principal': 'kafka', 'password': 'kafka1'}})
changed: [ip-10-38-1-10.ap-southeast-1.compute.internal] => (item={'key': 'client', 'value': {'principal': 'client', 'password': 'client-secret'}})
changed: [ip-10-38-1-10.ap-southeast-1.compute.internal] => (item={'key': 'control_center', 'value': {'principal': 'control-center', 'password': 'c3p'}})
changed: [ip-10-38-1-10.ap-southeast-1.compute.internal] => (item={'key': 'user1', 'value': {'principal': 'user1', 'password': 'u1p'}})