Readinessprobe fails with 500 (Fails due to plugins)

[gviswalingam]

Just applying this manifest from OPA tutorial and I seem to get readiness probe failed : 500

Grant OPA/kube-mgmt read-only access to resources. This lets kube-mgmt replicate resources into OPA so they can be used in policies.

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: opa-viewer
roleRef:
  kind: ClusterRole
  name: view
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: Group
  name: system:serviceaccounts:opa
  apiGroup: rbac.authorization.k8s.io
---

Define role for OPA/kube-mgmt to update configmaps with policy status.

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: opa
  name: configmap-modifier
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["update", "patch"]
---

Grant OPA/kube-mgmt role defined above.

kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: opa
  name: opa-configmap-modifier
roleRef:
  kind: Role
  name: configmap-modifier
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: Group
  name: system:serviceaccounts:opa
  apiGroup: rbac.authorization.k8s.io
---
kind: Service
apiVersion: v1
metadata:
  name: opa
  namespace: opa
spec:
  selector:
    app: opa
  ports:
  - name: https
    protocol: TCP
    port: 443
    targetPort: 8443
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: opa
  namespace: opa
  name: opa
spec:
  replicas: 1
  selector:
    matchLabels:
      app: opa
  template:
    metadata:
      labels:
        app: opa
      name: opa
    spec:
      containers:
        # WARNING: OPA is NOT running with an authorization policy configured. This
        # means that clients can read and write policies in OPA. If you are
        # deploying OPA in an insecure environment, be sure to configure
        # authentication and authorization on the daemon. See the Security page for
        # details: https://www.openpolicyagent.org/docs/security.html.
        - name: opa
          image: openpolicyagent/opa:0.52.0
          args:
            - "run"
            - "--server"
            - "--tls-cert-file=/certs/tls.crt"
            - "--tls-private-key-file=/certs/tls.key"
            - "--addr=0.0.0.0:8443"
            - "--addr=http://127.0.0.1:8181"
            - "--set=services.default.url=http://host.minikube.internal:8888"
            - "--set=bundles.default.resource=bundle.tar.gz"
            - "--log-format=json-pretty"
            - "--set=status.console=true"
            - "--set=decision_logs.console=true"
          volumeMounts:
            - readOnly: true
              mountPath: /certs
              name: opa-server
          readinessProbe:
            httpGet:
              path: /health?plugins&bundle
              scheme: HTTPS
              port: 8443
            initialDelaySeconds: 3
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /health
              scheme: HTTPS
              port: 8443
            initialDelaySeconds: 3
            periodSeconds: 5
        - name: kube-mgmt
          image: openpolicyagent/kube-mgmt:2.0.1
          args:
            - "--replicate-cluster=v1/namespaces"
            - "--replicate=networking.k8s.io/v1/ingresses"
      volumes:
        - name: opa-server
          secret:
            secretName: opa-server

kubectl apply -f admission-controller.yaml

Added extra debian container and exec into it then curl I got below response

Connected to localhost (127.0.0.1) port 8443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=opa.opa.svc
*  start date: May 17 08:52:58 2023 GMT
*  expire date: Mar  1 08:52:58 2297 GMT
*  issuer: CN=admission_ca
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55cf9ceb52e0)
> GET /health?plugins HTTP/2
> Host: localhost:8443
> user-agent: curl/7.74.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 500
< content-type: application/json
< content-length: 43
< date: Thu, 18 May 2023 11:54:09 GMT
<
{"error":"one or more plugins are not up"}
* Connection #0 to host localhost left intact

Any idea on troubleshooting would be helpful guys ?

Thanks

[charlieegan3]

Hi @gviswalingam, would you be able to share the logs from the OPA container?

[gviswalingam]

Thanks will give it a try

[gviswalingam]

That Resolved readiness probe but I am facing authorization problem

ukgvis@GBR-L-PF284EHS:/policies$ kubectl get po -A
NAMESPACE NAME READY STATUS RESTARTS AGE
default debian-77d4b9c77d-rc9cl 0/1 CrashLoopBackOff 59 (2m55s ago) 4h41m
ingress-nginx ingress-nginx-admission-create-xwsnx 0/1 Completed 0 32h
ingress-nginx ingress-nginx-admission-patch-jkcpk 0/1 Completed 1 32h
ingress-nginx ingress-nginx-controller-6cc5ccb977-zc7wn 1/1 Running 0 32h
kube-system coredns-787d4945fb-8kjc9 1/1 Running 0 32h
kube-system etcd-minikube 1/1 Running 0 32h
kube-system kube-apiserver-minikube 1/1 Running 0 32h
kube-system kube-controller-manager-minikube 1/1 Running 0 32h
kube-system kube-proxy-wjzz8 1/1 Running 0 32h
kube-system kube-scheduler-minikube 1/1 Running 0 32h
kube-system storage-provisioner 1/1 Running 0 32h
opa opa-56cf8b7d6-tltrf 2/2 Running 0 12m
ukgvis@GBR-L-PF284EHS:/policies$ kubectl get deploy -A
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
default debian 0/1 1 0 7h32m
ingress-nginx ingress-nginx-controller 1/1 1 1 32h
kube-system coredns 1/1 1 1 32h
opa opa 1/1 1 1 12m
ukgvis@GBR-L-PF284EHS:/policies$ kubectl exec -it opa-56cf8b7d6-tltrf -n opa -c opa -- /bin/bash
error: unable to upgrade connection: Authorization error (user=kube-apiserver-kubelet-client, verb=create, resource=nodes, subresource=proxy)
ukgvis@GBR-L-PF284EHS:/policies$ kubectl logs -l app=opa -c opa -f
Error from server (InternalError): Internal error occurred: Authorization error (user=kube-apiserver-kubelet-client, verb=get, resource=nodes, subresource=proxy)

[gviswalingam]

Also I described the pod, found the events
seen node ready but listed node and looks fine
Events:
Type Reason Age From Message

---

Normal Scheduled 17m default-scheduler Successfully assigned opa/opa-56cf8b7d6-tltrf to minikube
Normal Pulled 17m kubelet Container image "openpolicyagent/opa:0.52.0" already present on machine
Normal Created 17m kubelet Created container opa
Normal Started 17m kubelet Started container opa
Normal Pulled 17m kubelet Container image "openpolicyagent/kube-mgmt:2.0.1" already present on machine
Normal Created 17m kubelet Created container kube-mgmt
Normal Started 17m kubelet Started container kube-mgmt
Warning NodeNotReady 16s (x6 over 4m32s) node-controller Node is not ready
ukgvis@GBR-L-PF284EHS:~/policies$ kubectl get node -A
NAME STATUS ROLES AGE VERSION
minikube Ready control-plane 32h v1.26.3

[charlieegan3]

Hi again, the error:

Error from server (InternalError): Internal error occurred: Authorization error (user=kube-apiserver-kubelet-client, verb=get, resource=nodes, subresource=proxy)

Isn't from OPA, at least it looks like it's come from kubectl.