Getting certificates working with openshift subscription

[Andtalath]

Hello!

We have been running the operator by gitops with argo with manual download of the manifests and manual fixes to the files, but would like to try and use the subscription to simplify things.
However, we keep running into issues.

Changing certs seemed possible to do with editing the subscription, but I never got the validatingWebhookConfiguration to accept the certificates.

I tried with some basic cert-manager, but since we aren't using it, and it didn't work initially, it kinda ebbed out.

So, has anyone managed to make the messaging-topology-operator work as a subscription in openshift without cert-manager?

Is there in general a guide or something to make it work?

The error I got was:
http: TLS handshake error from 10.130.1.56:46140: remote error: tls: bad certificate
Each time a webhook was called (IP is from that type of source as well).

I did add clientConfig with a cert-chain as necessary, and the same cert chain validated curls towards what I believe to be one of the addresses.
(I'd love to find a list of expected names btw, it was a bit of a guesswork to know which names where needed in the cert).

[DanielePalaia]

Hi @Andtalath,

Not sure if this can help you on what you are doing, but this operator is also deployed with OLM both in operator-hub (https://operatorhub.io/operator/rabbitmq-messaging-topology-operator) and openshift-marketplace. In this way you can deploy and manage the operator using an openshift subscription. The certificates are managed by OLM itself in this way without cert-manager

[Andtalath]

@DanielePalaia Thanks for answer =)
However, that is the one I am using.

Installing it without editing the subscription to add my own cert gives me two types of errors:

1. The TLS errors from above.
2. An error message a bit hard to replicate now since we have switched back to running it through argo pipeline atm, but the issue was related to:
   https://rabbitmq.com/kubernetes/operator/install-topology-operator.html
   "Generate Key/Certificate Pair
   First, generate one or more key/certificate pairs for webhook admission. These certificates must be valid for webhook-service.rabbitmq-system.svc." And that the certificate that was launched didn't include that SAN.

I've only so far been troubleshooting it, not the main OLM rabbitmq-operator (since it works fine when we install the manifests directly through argo).
So, your experience is that you don't need to generate certs beforehand and it's just going to work?

[DanielePalaia]

@Andtalath Yes OLM should provision and rotate them for you: https://olm.operatorframework.io/docs/advanced-tasks/adding-admission-and-conversion-webhooks/
It should do what cert-manager normally does for non OLM deployments

This is how we test the operator as well in our pipeline.
You should be able also to provision your own certificates but I've never tested it in OLM through an OLM subscription.

[shanehughes]

G'day,

I too have struck this issue (now that we've got 1.12.1). I suspect it comes down to how the operator Deployment is mounting the webhook certificate (at least, this is what it looks like to me).

Essentially, we end up with:

- name: webhook-cert
  secret:
    secretName: messaging-topology-operator-service-cert
    items:
      - key: tls.crt
        path: tls.crt
      - key: tls.key
        path: tls.key
    defaultMode: 420

in the Deployment object. Now, the ValidatingWebhookConfiguration objects seem to reference the correct service, but the invoked URL looks like: https://webhook-service.[namespace].svc:443/... - the cert that is generated and supplied in messaging-topology-operator-service-cert is therefore not valid - it doesn't contain a SAN for webhook-service.

I think it's a problem with the way that the certificate for the webhook service is mounted, but, at the same time, I don't actually see one generated automatically for that (webhook-service) endpoint.

[shanehughes]

Actually, a more complete snippet would help here. The volume related information in the Deployment looks like this (from a straight-up OLM deployment):

    volumeMounts:
      - name: webhook-cert
        mountPath: /tmp/k8s-webhook-server/serving-certs
      - name: apiservice-cert
        mountPath: /apiserver.local.config/certificates
    ...
volumes:
  - name: cert
    secret:
      secretName: webhook-server-cert
      defaultMode: 420
  - name: apiservice-cert
    secret:
      secretName: messaging-topology-operator-service-cert
      items:
        - key: tls.crt
          path: apiserver.crt
        - key: tls.key
          path: apiserver.key
      defaultMode: 420
  - name: webhook-cert
    secret:
      secretName: messaging-topology-operator-service-cert
      items:
        - key: tls.crt
          path: tls.crt
        - key: tls.key
          path: tls.key
      defaultMode: 420

Note that the cert volume is never used at all (nor is it mounted). At the same time, it doesn't actually exist either...

No cert-manager in this environment.

[DanielePalaia]

I think this was replied here #701