[Bug]: when I enable mtls, zot pod failed to start with error: Startup probe failed

[sonyafenge]

zot version

v2.1.2

Describe the bug

when I enable mtls, zot pod failed to start with error: Startup probe failed: Get "https://10.0.0.20:5000/v2/": remote error: tls: certificate required.

Could you please confirm:

1. if Helm value yaml support bypass mtls for Startup probe.
2. if there are any example documents or md for mtls config.

To reproduce

1. install helm with mtls enabled value yaml:

helm install zot project-zot/zot -n zot -f ./zot-value-lb-mtls.yaml

cat ./zot-value-lb-mtls.yaml

# Default values for zot.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
service:
  type: LoadBalancer
  port: 443
  targetPort: 5000
  # Annotations to add to the service
httpGet:
  scheme: HTTPS
mountConfig: true
configFiles:
  config.json: |-
    {
      "storage": { "rootDirectory": "/var/lib/registry" },
      "http": {
        "address": "0.0.0.0",
        "port": "5000",
        "realm":"zot",
        "tls": {
          "cert": "/secret/tls/tls.crt",
          "key": "/secret/tls/tls.key",
          "cacert": "/secret/tls/ca.crt"
        }
      },
      "log": { "level": "debug" },
      "extensions": {
        "search": {
          "cve": {
            "updateInterval": "2h"
          }
        },
        "ui": {
          "enable": true
        }
      }
    }
mountSecret: true
externalSecrets:
  - secretName: "zot-server-secret"
    mountPath: "/secret/tls"

2. check pod status and get error:```
   Warning Unhealthy 46s (x4 over 96s) kubelet Startup probe failed: Get "https://10.0.0.20:5000/v2/": remote error: tls: certificate required

### Expected behavior

1. pods start successfully when enable mtls;
2. have config in value.yaml to bypass sercurity for health check.

### Screenshots

<img width="1301" alt="Image" src="https://github.com/user-attachments/assets/26e66914-5e63-4e29-bdfc-c171e3865963" />

### Additional context

_No response_

[rchincha]

"cacert": "/secret/tls/ca.crt"

^ specifying this, causes zot to expect a client-side cert. Try removing that line in the config.

Thanks for trying out zot. Cheers!

[sonyafenge]

"cacert": "/secret/tls/ca.crt"

^ specifying this, causes zot to expect a client-side cert. Try removing that line in the config.

Thanks for trying out zot. Cheers!

if removing that line in the config, it will be disable mtls, and change to tls. do you have any example for mtls certification?

[rchincha]

The better option then maybe to enable basic auth via htpasswd under http config, and liveness probe can use that.

livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
    httpHeaders:
      - name: Authorization
        value: Basic aGE6aGE=

[andaaron]

Do we have a healthz endpoint? Maybe we should. There was an issue about it somewhere.

L.E. here it was: #2487
There are several other links there.

[rchincha]

No, but use /v2/ as the endpoint.

[rchincha]

We should really mark this a DUPLICATE of #2487

[sonyafenge]

The better option then maybe to enable basic auth via htpasswd under http config, and liveness probe can use that.

livenessProbe:
httpGet:
path: /healthz
port: 8080
httpHeaders:
- name: Authorization
value: Basic aGE6aGE=

after I added auth to http config, I can got zot pod start successfully

[rchincha]

Closing as resolved.