Topology Operator with publicly exposed TLS RabbitMQ Cluster

[romainfd]

Hello,

We have a publicly exposed RabbitMQ using TLS and certificate as authentification.
To configure the Topology Operator, we added the RabbitMQ CA in the Topology Operator (following this) but we are getting "401 Unauthorized" errors

Logs (reconciliation & RabbitMQ)

{
   "level":"error",
   "ts":"2024-01-11T19:16:05Z",
   "msg":"Reconciler error",
   "controller":"binding",
   "controllerGroup":"rabbitmq.com",
   "controllerKind":"Binding",
   "Binding":{
      "name":"samu570.ack-queue-binding",
      "namespace":"default"
   },
   "namespace":"default",
   "name":"samu570.ack-queue-binding",
   "reconcileID":"6670896f-ea4b-4fb3-88bd-6e976aa4c6f6",
   "error":"Error: API responded with a 401 Unauthorized",
   "stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:226"
}

And in the RabbitMQ logs : [warning] <0.30900.2> HTTP access denied: user 'default_user_bNlYv-A_1rESCl5iBWB' - invalid credentials

Trying to fix the issuer, we found this and this in the README saying "It uses the generated default user secret from RabbitmqCluster (set in rabbitmqcluster.status.binding) to authenticate with RabbitMQ server."
Running kubectl describe rmq, we can see that we still have the binding on rabbitmq-default-user. Running kubectl get secret rabbitmq-default-user -o go-template='{{range $k,$v := .data}}{{printf "%s: " $k}}{{if not $v}}{{$v}}{{else}}{{$v | base64decode}}{{end}}{{"\n"}}{{end}}' indeed gives us username: default_user_bNlYv-A_1rESCl5iBWB.
So the Topology Operator is correctly retrieving the user but it somehow is unauthorized...

Besides, our Cluster is publicly exposed and we use X509 certificates to authenticate users. Could the problem be related to this?

If needed, we have the following in our RabbitmqCluster's spec.rabbitmq.additionalConfig

      ssl_options.verify = verify_peer
      ssl_options.fail_if_no_peer_cert = true
     
      loopback_users.guest = false
      default_user = guest
      default_pass = guest

      load_definitions = /tmp/rabbitmq/config/definitions.json      

      # For Dashboard UI to work | # Ref.: https://cjshelton.github.io/blog/2019/12/18/rabbitmq-client-certificate-authentication.html#server-side-changes
      auth_mechanisms.1 = PLAIN
      # x509 cert Auth | Ref.: https://github.com/rabbitmq/rabbitmq-server/tree/main/deps/rabbitmq_auth_mechanism_ssl#usage
      auth_mechanisms.1 = EXTERNAL
      # Use CN | Ref.: https://github.com/rabbitmq/rabbitmq-server/tree/main/deps/rabbitmq_auth_mechanism_ssl#common-name
      ssl_cert_login_from = common_name

Looking at the code where the Topology Operator sets its user, I was wondering if we should create a custom user for us? Or we should create a user on RabbitMQ based on the Topology Operator certificate? But then, how do we specify to the Topology Operator to use the EXTERNAL auth mechanism (using the certificate instead of username/password credentials)?

NB : so far, we are using our own CA so we could set the SAN to rabbitmq.default.svc. However, based on these #233 #644 and #1185, we understand that the correct way to go is to rely on a single certificate on the URL and set our RabbitmqCluster's metadata.annotations.rabbitmq.com/operator-connection-uri: "https://ourUrl.com:5671". Is that correct?

[romainfd]

Following this discussion, we tried commenting out these lines and reapply le RabbitmqCluster

loopback_users.guest = false
default_user = guest
default_pass = guest

But we still get the same errors.
Why is it talking about HTTP access when we want TLS calls?
Besides, kubectl exec -it rabbitmq-server-0 -- rabbitmqctl list_users doesn't show the default user. Is it normal that it is not visible?

[romainfd]

We actually solved it by removing the load_definitions = /tmp/rabbitmq/config/definitions.json which caused the default_user creation to be ignored!

However, we still have a few questions regarding this topic:

• When using metadata.annotations.rabbitmq.com/operator-connection-uri does the request go outside of the cluster through the Internet back into the RabbitMQ or does it just tell the TLS verification to accept the URI?
• What do loopback_users actually do? I can't find a clear doc online about this
• Finally, the default_user has very high rights on the cluster (administrator tag & .* configure/read/write rights) to be able to setup everything. Is it secure to have such a user on a publicly available RabbitMQ or is it advised to do something different?

[Zerpet]

Apologies for the late reply 🙏 Indeed, by importing the definitions, the default user is not created, but the Secret is. It is confusing. Perhaps we should just parse the additional configuration, and skip the Secret creation if certain keys, like load_definitions, are present.

When using metadata.annotations.rabbitmq.com/operator-connection-uri does the request go outside of the cluster through the Internet back into the RabbitMQ or does it just tell the TLS verification to accept the URI?

It depends how Kubernetes network is wired. If you are exposing the cluster using a cloud load balancer, like Google Cloud Load Balancer, the request will likely go into Google's cloud infra, and then back to the cluster. If you are exposing the cluster using something like NGINX ingress, it may determine that the IP is inside the cluster, and route the traffic without going to the internet.

What do loopback_users actually do? I can't find a clear doc online about this

You have the loopback_users briefly documented here. You have to search through the page, but a ctrl-f or cmd-f will find it quickly. I'm pasting here the description: List of users which are only permitted to connect to the broker via a loopback interface (i.e. localhost).

It's a security mechanism to limit some users to connect using only localhost. By default, it's the well known user guest.

Finally, the default_user has very high rights on the cluster (administrator tag & .* configure/read/write rights) to be able to setup everything. Is it secure to have such a user on a publicly available RabbitMQ or is it advised to do something different?

It depends who you ask 🙂 This user is necessary for the Topology Operator to perform its operations. You can rotate the password every certain amount of time. You will have to update the user password in RabbitMQ and update the Secret in Kubernetes.

[romainfd]

Hello @Zerpet,
Thank you so much for your very detailed answer and sorry for the delay!

• Regarding the default user creation, I indeed think that the Secret presence is what got us confused as we thought it was actually created! I think that reading the logs more carefully could have helped us as I remember seeing some logs about default user being ignored... but the Secret presence was also saying something else...
• Regarding the connection-uri: thank you for the details. In both ways, from what I understand, it doesn't really on some Kube DNS updates. We will see what happens with our Cloud provider then!
• Regarding the loopback_users, thank you for the details and pointer!
• Regarding the default_user rights, thank you for the answer as well. We will probably go towards some kind of password rotation indeed!