Add https certificate for http://prombench.prometheus.io/

[bboreham]

Aside from being generally good practice, I hit this when I wanted to point https://demo.promlens.com/ at http://prombench.prometheus.io/prometheus-meta/; it refuses to talk to non-https endpoints.

[Vandit1604]

Hi, I’d like to work on this issue. Should I use cert-manager for automated TLS, or would you prefer creating a Secret with the key and certificate?

[bboreham]

We’d have to rotate it, right? So it should be automated.

[Vandit1604]

Yes, exactly. Cert-Manager automates certificate rotation, so once it's set up, it will handle renewals for us. Going with that.

[Vandit1604]

The Cert Manager setup is failing due to some NGINX configuration right now, (I Suppose, Still investigating).
The Cert Manager is creating a challenge object, but when the Solver pod attempts to handle it, a 404 error is returned. The Cert Manager pod logs confirm this with the following message:

E1102 18:43:02.885272       1 sync.go:208] "propagation check failed" err="wrong status code '404', expected '200'" logger="cert-manager.controller" resource_name="prometheus-meta-1-3643360222-881729279" resource_namespace="default" resource_kind="Challenge" resource_version="v1" dnsName="prombench.prometheus.io" type="HTTP-01"

The Solver pod is unable to reach the challenge at the http://prombench.prometheus.io/.well-known/acme-challenge/<token>
Still Investigating.

[Vandit1604]

I raised a PR to add an HTTPS certificate to prombench.prometheus.io using Cert-Manager and Let's Encrypt. I'm facing some DNS issues during local testing. With some tweaks, I managed to solve the ACME challenge and issue the certificate locally, but if there's a more streamlined way to test this setup, any guidance would be appreciated.