x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" #428

gviswalingam · 2023-05-19T11:52:19Z

gviswalingam
May 19, 2023

I was following the OPA Tutorial documentation and seem to be getting problem with self signed certification

https://www.openpolicyagent.org/docs/latest/kubernetes-tutorial/

ukgvis@GBR-L-PF284EHS:~/policies$ kubectl create -f qa-namespace.yaml
Error from server (InternalError): error when creating "qa-namespace.yaml": Internal error occurred: failed calling webhook "validating-webhook.openpolicyagent.org": failed to call webhook: Post "https://opa.opa.svc:443/?timeout=10s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "admission_ca")

Any idea guys ?

Answered by charlieegan3

May 23, 2023

I can't see any POSTs in those logs, can you check they cover the period where the ingress is created?

We can have a look at the policies in OPA like this:

kubectl port-forward svc/opa -n opa 8181:8181

then, in another tab:

curl --silent localhost:8181/v1/policies/opa/policy/ingress-allowlist.rego | jq -r .result.raw

And the main.rego:

curl --silent localhost:8181/v1/policies/opa/policy/main.rego | jq -r .result.raw

Please can you format the output so it's easy to read on GitHub by using '```' blocks 🙂. This makes it easier to read your messages.

View full answer

anderseknert · 2023-05-19T12:05:36Z

anderseknert
May 19, 2023
Maintainer

Hi @gviswalingam 👋

Could you check the Kubernetes API server logs? There are usually some hints as to what would cause verification to fail there. We've had some issues with that in the past, but I thought the last fixes would have taken care of that. Could be that there are some new requirements for the latest Kube version... not sure we've tried running with that yet.

19 replies

gviswalingam May 23, 2023
Author

ukgvis@GBR-L-PF284EHS:~$ kubectl logs --tail=200 opa-5c7d44b75b-8wmqf -n opa
Defaulted container "opa" out of: opa, kube-mgmt
"time": "2023-05-23T07:08:51Z"
}
{
"decision_id": "19a6dca9-f820-48c4-84f2-06bbc4026b3a",
"input": {
"apiVersion": "admission.k8s.io/v1",
"kind": "AdmissionReview",
"request": {
"dryRun": false,
"kind": {
"group": "coordination.k8s.io",
"kind": "Lease",
"version": "v1"
},
"name": "ingress-nginx-leader",
"namespace": "ingress-nginx",
"object": {
"apiVersion": "coordination.k8s.io/v1",
"kind": "Lease",
"metadata": {
"creationTimestamp": "2023-05-22T10:56:36Z",
"managedFields": [
{
"apiVersion": "coordination.k8s.io/v1",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:spec": {
"f:acquireTime": {},
"f:holderIdentity": {},
"f:leaseDurationSeconds": {},
"f:leaseTransitions": {},
"f:renewTime": {}
}
},
"manager": "nginx-ingress-controller",
"operation": "Update",
"time": "2023-05-23T07:08:51Z"
}
],
"name": "ingress-nginx-leader",
"namespace": "ingress-nginx",
"resourceVersion": "129518",
"uid": "fd151452-db40-4862-ad8c-d6a8fb442d61"
},
"spec": {
"acquireTime": "2023-05-22T10:56:36.405843Z",
"holderIdentity": "ingress-nginx-controller-6cc5ccb977-hvr8l",
"leaseDurationSeconds": 30,
"leaseTransitions": 0,
"renewTime": "2023-05-23T07:08:51.091742Z"
}
},
"oldObject": {
"apiVersion": "coordination.k8s.io/v1",
"kind": "Lease",
"metadata": {
"creationTimestamp": "2023-05-22T10:56:36Z",
"managedFields": [
{
"apiVersion": "coordination.k8s.io/v1",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:spec": {
"f:acquireTime": {},
"f:holderIdentity": {},
"f:leaseDurationSeconds": {},
"f:leaseTransitions": {},
"f:renewTime": {}
}
},
"manager": "nginx-ingress-controller",
"operation": "Update",
"time": "2023-05-23T07:08:43Z"
}
],
"name": "ingress-nginx-leader",
"namespace": "ingress-nginx",
"resourceVersion": "129518",
"uid": "fd151452-db40-4862-ad8c-d6a8fb442d61"
},
"spec": {
"acquireTime": "2023-05-22T10:56:36.405843Z",
"holderIdentity": "ingress-nginx-controller-6cc5ccb977-hvr8l",
"leaseDurationSeconds": 30,
"leaseTransitions": 0,
"renewTime": "2023-05-23T07:08:43.471794Z"
}
},
"operation": "UPDATE",
"options": {
"apiVersion": "meta.k8s.io/v1",
"kind": "UpdateOptions"
},
"requestKind": {
"group": "coordination.k8s.io",
"kind": "Lease",
"version": "v1"
},
"requestResource": {
"group": "coordination.k8s.io",
"resource": "leases",
"version": "v1"
},
"resource": {
"group": "coordination.k8s.io",
"resource": "leases",
"version": "v1"
},
"uid": "0e1fec60-8db6-4dd7-8bc7-2717f3fdcfcf",
"userInfo": {
"extra": {
"authentication.kubernetes.io/pod-name": [
"ingress-nginx-controller-6cc5ccb977-hvr8l"
],
"authentication.kubernetes.io/pod-uid": [
"ab4ccc57-916f-4c72-9842-787f181a0149"
]
},
"groups": [
"system:serviceaccounts",
"system:serviceaccounts:ingress-nginx",
"system:authenticated"
],
"uid": "3f9ff135-0479-4ff5-8968-b9b3f4daa14f",
"username": "system:serviceaccount:ingress-nginx:ingress-nginx"
}
}
},
"labels": {
"id": "26dbbab1-d60e-493a-a576-9ff94706f977",
"version": "0.52.0"
},
"level": "info",
"metrics": {
"counter_server_query_cache_hit": 1,
"timer_rego_external_resolve_ns": 2200,
"timer_rego_query_eval_ns": 567996,
"timer_server_handler_ns": 2031686
},
"msg": "Decision Log",
"path": "system/main",
"req_id": 3114,
"requested_by": "10.244.0.1:16882",
"result": {
"apiVersion": "admission.k8s.io/v1",
"kind": "AdmissionReview",
"response": {
"allowed": true,
"uid": "0e1fec60-8db6-4dd7-8bc7-2717f3fdcfcf"
}
},
"time": "2023-05-23T07:08:51Z",
"timestamp": "2023-05-23T07:08:51.117608749Z",
"type": "openpolicyagent.org/decision_logs"
}
{
"client_addr": "10.244.0.1:16882",
"level": "info",
"msg": "Sent response.",
"req_id": 3114,
"req_method": "POST",
"req_path": "/",
"resp_bytes": 135,
"resp_duration": 4.053973,
"resp_status": 200,
"time": "2023-05-23T07:08:51Z"
}
{
"client_addr": "127.0.0.1:40106",
"level": "info",
"msg": "Sent response.",
"req_id": 3115,
"req_method": "PUT",
"req_path": "/v1/policies/opa/policy/main.rego",
"resp_bytes": 3,
"resp_duration": 1.175992,
"resp_status": 200,
"time": "2023-05-23T07:08:51Z"
}
{
"client_addr": "127.0.0.1:40116",
"level": "info",
"msg": "Received request.",
"req_id": 3116,
"req_method": "PUT",
"req_path": "/v1/policies/opa/policy/ingress-allowlist.rego",
"time": "2023-05-23T07:08:51Z"
}
{
"client_addr": "127.0.0.1:40116",
"level": "info",
"msg": "Sent response.",
"req_id": 3116,
"req_method": "PUT",
"req_path": "/v1/policies/opa/policy/ingress-allowlist.rego",
"resp_bytes": 1377,
"resp_duration": 9.564335,
"resp_status": 400,
"time": "2023-05-23T07:08:51Z"
}

charlieegan3 May 23, 2023
Maintainer

I can't see any post requests for an ingress resource in those logs.

If you update the web hook config to only run admission checks for ingress resources, there might be less going on.

cat > webhook-configuration.yaml <<EOF
kind: ValidatingWebhookConfiguration
apiVersion: admissionregistration.k8s.io/v1
metadata:
  name: opa-validating-webhook
webhooks:
  - name: validating-webhook.openpolicyagent.org
    namespaceSelector:
      matchExpressions:
      - key: openpolicyagent.org/webhook
        operator: NotIn
        values:
        - ignore
    rules:
      - operations: ["CREATE", "UPDATE"]
        apiGroups: ["networking.k8s.io"]
        apiVersions: ["v1"]
        resources: ["Ingress"]
    clientConfig:
      caBundle: $(cat ca.crt | base64 | tr -d '\n')
      service:
        namespace: opa
        name: opa
    admissionReviewVersions: ["v1"]
    sideEffects: None
EOF

kubectl apply -f webhook-configuration.yaml

gviswalingam May 23, 2023
Author

I am still getting lots of logs

ukgvis@GBR-L-PF284EHS:$ kubectl delete -f webhook-configuration.yaml
validatingwebhookconfiguration.admissionregistration.k8s.io "opa-validating-webhook" deleted
ukgvis@GBR-L-PF284EHS:$ cat > webhook-configuration.yaml <<EOF

kind: ValidatingWebhookConfiguration
apiVersion: admissionregistration.k8s.io/v1
metadata:
name: opa-validating-webhook
webhooks:

name: validating-webhook.openpolicyagent.org
namespaceSelector:
matchExpressions:

key: openpolicyagent.org/webhook
operator: NotIn
values:

ignore
rules:

operations: ["CREATE", "UPDATE"]
apiGroups: ["networking.k8s.io"]
apiVersions: ["v1"]
resources: ["Ingress"]
clientConfig:
caBundle: $(cat ca.crt | base64 | tr -d '\n')
service:
namespace: opa
name: opa
admissionReviewVersions: ["v1"]
sideEffects: None
EOF
ukgvis@GBR-L-PF284EHS:$ kubectl apply -f webhook-configuration.yaml
validatingwebhookconfiguration.admissionregistration.k8s.io/opa-validating-webhook created
ukgvis@GBR-L-PF284EHS:$ kubectl logs --tail=300 opa-5c7d44b75b-8wmqf -n opa
Defaulted container "opa" out of: opa, kube-mgmt
"req_path": "/health",
"resp_bytes": 3,
"resp_duration": 0.375903,
"resp_status": 200,
"time": "2023-05-23T10:15:00Z"
}
{
"client_addr": "127.0.0.1:41832",
"level": "info",
"msg": "Received request.",
"req_id": 65857,
"req_method": "PUT",
"req_path": "/v1/policies/opa/policy/ingress-allowlist.rego",
"time": "2023-05-23T10:15:00Z"
}
{
"client_addr": "127.0.0.1:41832",
"level": "info",
"msg": "Sent response.",
"req_id": 65857,
"req_method": "PUT",
"req_path": "/v1/policies/opa/policy/ingress-allowlist.rego",
"resp_bytes": 1377,
"resp_duration": 4.311831,
"resp_status": 400,
"time": "2023-05-23T10:15:00Z"
}
{
"client_addr": "127.0.0.1:41832",
"level": "info",
"msg": "Received request.",
"req_id": 65858,
"req_method": "PUT",
"req_path": "/v1/policies/opa/policy/ingress-allowlist.rego",
"time": "2023-05-23T10:15:01Z"
}
{
"client_addr": "127.0.0.1:41832",
"level": "info",
"msg": "Sent response.",
"req_id": 65858,
"req_method": "PUT",
"req_path": "/v1/policies/opa/policy/ingress-allowlist.rego",
"resp_bytes": 1377,
"resp_duration": 4.380132,
"resp_status": 400,
"time": "2023-05-23T10:15:01Z"
}
{
"client_addr": "127.0.0.1:41832",
"level": "info",
"msg": "Received request.",
"req_id": 65859,
"req_method": "PUT",
"req_path": "/v1/policies/opa/policy/main.rego",
"time": "2023-05-23T10:15:01Z"
}
{
"client_addr": "127.0.0.1:41832",
"level": "info",
"msg": "Sent response.",
"req_id": 65859,
"req_method": "PUT",
"req_path": "/v1/policies/opa/policy/main.rego",
"resp_bytes": 3,
"resp_duration": 0.598404,
"resp_status": 200,
"time": "2023-05-23T10:15:01Z"
}
{
"client_addr": "127.0.0.1:41844",
"level": "info",
"msg": "Received request.",
"req_id": 65860,
"req_method": "PUT",
"req_path": "/v1/policies/opa/policy/main.rego",
"time": "2023-05-23T10:15:01Z"
}
{
"client_addr": "127.0.0.1:41844",
"level": "info",
"msg": "Sent response.",
"req_id": 65860,
"req_method": "PUT",
"req_path": "/v1/policies/opa/policy/main.rego",
"resp_bytes": 3,
"resp_duration": 0.797605,
"resp_status": 200,
"time": "2023-05-23T10:15:01Z"
}
{
"client_addr": "127.0.0.1:41860",
"level": "info",
"msg": "Received request.",
"req_id": 65861,
"req_method": "PUT",
"req_path": "/v1/policies/opa/policy/ingress-allowlist.rego",
"time": "2023-05-23T10:15:01Z"
}
{
"client_addr": "127.0.0.1:41860",
"level": "info",
"msg": "Sent response.",
"req_id": 65861,
"req_method": "PUT",
"req_path": "/v1/policies/opa/policy/ingress-allowlist.rego",
"resp_bytes": 1377,
"resp_duration": 4.010829,
"resp_status": 400,
"time": "2023-05-23T10:15:01Z"
}
{
"client_addr": "127.0.0.1:41860",
"level": "info",
"msg": "Received request.",
"req_id": 65862,
"req_method": "PUT",
"req_path": "/v1/policies/opa/policy/ingress-allowlist.rego",
"time": "2023-05-23T10:15:01Z"
}
{
"client_addr": "127.0.0.1:41860",
"level": "info",
"msg": "Sent response.",
"req_id": 65862,
"req_method": "PUT",
"req_path": "/v1/policies/opa/policy/ingress-allowlist.rego",
"resp_bytes": 1377,
"resp_duration": 7.002651,
"resp_status": 400,
"time": "2023-05-23T10:15:01Z"
}
{
"client_addr": "127.0.0.1:41860",
"level": "info",
"msg": "Received request.",
"req_id": 65863,
"req_method": "PUT",
"req_path": "/v1/policies/opa/policy/main.rego",
"time": "2023-05-23T10:15:02Z"
}
{
"client_addr": "127.0.0.1:41860",
"level": "info",
"msg": "Sent response.",
"req_id": 65863,
"req_method": "PUT",
"req_path": "/v1/policies/opa/policy/main.rego",
"resp_bytes": 3,
"resp_duration": 1.193309,
"resp_status": 200,
"time": "2023-05-23T10:15:02Z"
}
{
"client_addr": "127.0.0.1:41862",
"level": "info",
"msg": "Received request.",
"req_id": 65864,
"req_method": "PUT",
"req_path": "/v1/policies/opa/policy/ingress-allowlist.rego",
"time": "2023-05-23T10:15:02Z"
}
{
"client_addr": "127.0.0.1:41862",
"level": "info",
"msg": "Sent response.",
"req_id": 65864,
"req_method": "PUT",
"req_path": "/v1/policies/opa/policy/ingress-allowlist.rego",
"resp_bytes": 1377,
"resp_duration": 4.718334,
"resp_status": 400,
"time": "2023-05-23T10:15:02Z"
}
{
"client_addr": "127.0.0.1:41862",
"level": "info",
"msg": "Received request.",
"req_id": 65865,
"req_method": "PUT",
"req_path": "/v1/policies/opa/policy/main.rego",
"time": "2023-05-23T10:15:02Z"
}
{
"client_addr": "127.0.0.1:41862",
"level": "info",
"msg": "Sent response.",
"req_id": 65865,
"req_method": "PUT",
"req_path": "/v1/policies/opa/policy/main.rego",
"resp_bytes": 3,
"resp_duration": 0.552204,
"resp_status": 200,
"time": "2023-05-23T10:15:02Z"
}
{
"client_addr": "127.0.0.1:41868",
"level": "info",
"msg": "Received request.",
"req_id": 65866,
"req_method": "PUT",
"req_path": "/v1/policies/opa/policy/main.rego",
"time": "2023-05-23T10:15:02Z"
}
{
"client_addr": "127.0.0.1:41868",
"level": "info",
"msg": "Sent response.",
"req_id": 65866,
"req_method": "PUT",
"req_path": "/v1/policies/opa/policy/main.rego",
"resp_bytes": 3,
"resp_duration": 0.486404,
"resp_status": 200,
"time": "2023-05-23T10:15:02Z"
}
{
"client_addr": "127.0.0.1:41870",
"level": "info",
"msg": "Received request.",
"req_id": 65867,
"req_method": "PUT",
"req_path": "/v1/policies/opa/policy/ingress-allowlist.rego",
"time": "2023-05-23T10:15:02Z"
}
{
"client_addr": "127.0.0.1:41870",
"level": "info",
"msg": "Sent response.",
"req_id": 65867,
"req_method": "PUT",
"req_path": "/v1/policies/opa/policy/ingress-allowlist.rego",
"resp_bytes": 1377,
"resp_duration": 1.761113,
"resp_status": 400,
"time": "2023-05-23T10:15:02Z"
}
{
"client_addr": "127.0.0.1:41870",
"level": "info",
"msg": "Received request.",
"req_id": 65868,
"req_method": "PUT",
"req_path": "/v1/policies/opa/policy/ingress-allowlist.rego",
"time": "2023-05-23T10:15:03Z"
}
{
"client_addr": "127.0.0.1:41870",
"level": "info",
"msg": "Sent response.",
"req_id": 65868,
"req_method": "PUT",
"req_path": "/v1/policies/opa/policy/ingress-allowlist.rego",
"resp_bytes": 1377,
"resp_duration": 2.84302,
"resp_status": 400,
"time": "2023-05-23T10:15:03Z"
}
{
"client_addr": "127.0.0.1:41870",
"level": "info",
"msg": "Received request.",
"req_id": 65869,
"req_method": "PUT",
"req_path": "/v1/policies/opa/policy/main.rego",
"time": "2023-05-23T10:15:03Z"
}
{
"client_addr": "127.0.0.1:41870",
"level": "info",
"msg": "Sent response.",
"req_id": 65869,
"req_method": "PUT",
"req_path": "/v1/policies/opa/policy/main.rego",
"resp_bytes": 3,
"resp_duration": 0.324203,
"resp_status": 200,
"time": "2023-05-23T10:15:03Z"
}
{
"client_addr": "127.0.0.1:41874",
"level": "info",
"msg": "Received request.",
"req_id": 65870,
"req_method": "PUT",
"req_path": "/v1/policies/opa/policy/ingress-allowlist.rego",
"time": "2023-05-23T10:15:03Z"
}
{
"client_addr": "127.0.0.1:41874",
"level": "info",
"msg": "Sent response.",
"req_id": 65870,
"req_method": "PUT",
"req_path": "/v1/policies/opa/policy/ingress-allowlist.rego",
"resp_bytes": 1377,
"resp_duration": 4.673334,
"resp_status": 400,
"time": "2023-05-23T10:15:03Z"
}

charlieegan3 May 23, 2023
Maintainer

I can't see any POSTs in those logs, can you check they cover the period where the ingress is created?

We can have a look at the policies in OPA like this:

kubectl port-forward svc/opa -n opa 8181:8181

then, in another tab:

curl --silent localhost:8181/v1/policies/opa/policy/ingress-allowlist.rego | jq -r .result.raw

And the main.rego:

curl --silent localhost:8181/v1/policies/opa/policy/main.rego | jq -r .result.raw

Please can you format the output so it's easy to read on GitHub by using '```' blocks 🙂. This makes it easier to read your messages.

Answer selected by gviswalingam

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Open Policy Agent

x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" #428

{{title}}

Replies: 1 comment 19 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Open Policy Agent

x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" #428

gviswalingam May 19, 2023

Replies: 1 comment · 19 replies

anderseknert May 19, 2023 Maintainer

gviswalingam May 23, 2023 Author

charlieegan3 May 23, 2023 Maintainer

gviswalingam May 23, 2023 Author

charlieegan3 May 23, 2023 Maintainer

gviswalingam
May 19, 2023

Replies: 1 comment 19 replies

anderseknert
May 19, 2023
Maintainer

gviswalingam May 23, 2023
Author

charlieegan3 May 23, 2023
Maintainer

gviswalingam May 23, 2023
Author

charlieegan3 May 23, 2023
Maintainer