Issue recovering from expired control plane certificates after prolonged cluster shutdown #1630

darcystan · 2023-06-05T15:45:01Z

darcystan
Jun 5, 2023

Our OKD 4.10 cluster was shutdown for a few months while testing a trial version of openshift, however after starting it back up we couldn't log in through the oc client. We noticed that oc get csr shows several certs in a pending state:

[root@master0 ~]# oc get csr
NAME        AGE     SIGNERNAME                                    REQUESTOR                                                                   REQUESTEDDURATION   CONDITION
csr-22cn9   5h51m   kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   <none>              Pending
csr-22k4m   18h     kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   <none>              Pending

...

We tried following the procedure documented here: https://docs.okd.io/4.10/backup_and_restore/control_plane_backup_and_restore/disaster_recovery/scenario-3-expired-certs.html

However when trying to approve the certificate we get a Forbidden error:

[root@master0 ~]# oc adm certificate approve csr-zxspl
Error from server (Forbidden): certificatesigningrequests.certificates.k8s.io "csr-zxspl" is forbidden: User "system:serviceaccount:openshift-machine-config-operator:node-bootstrapper" cannot update resource "certificatesigningrequests/approval" in API group "certificates.k8s.io" at the cluster scope

The only kubeconfig that will even work with oc get csr is /etc/kubernetes/kubeconfig, using /etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/node-kubeconfigs/lb-int.kubeconfig (mentioned here - https://access.redhat.com/solutions/5953441) doesn't work, we get "error: You must be logged in to the server (Unauthorized)". We tried the other kubeconfig files as well without any luck.

OS Version:

# cat /etc/os-release
NAME="Fedora Linux"
VERSION="35 (CoreOS)"
ID=fedora
VERSION_ID=35
VERSION_CODENAME=""
PLATFORM_ID="platform:f35"
PRETTY_NAME="Fedora CoreOS 35"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:35"
HOME_URL="https://getfedora.org/coreos/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora-coreos/"
SUPPORT_URL="https://github.com/coreos/fedora-coreos-tracker/"
BUG_REPORT_URL="https://github.com/coreos/fedora-coreos-tracker/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=35
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=35
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="CoreOS"
VARIANT_ID=coreos
OSTREE_VERSION='410.35.202206241934-0'
DEFAULT_HOSTNAME=localhost

Any ideas on getting these certificates approved?

vrutkovs · 2023-06-05T15:53:31Z

vrutkovs
Jun 5, 2023
Maintainer

The only kubeconfig that will even work with oc get csr is /etc/kubernetes/kubeconfig, using /etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/node-kubeconfigs/lb-int.kubeconfig (mentioned here - access.redhat.com/solutions/5953441) doesn't work, we get "error: You must be logged in to the server (Unauthorized)". We tried the other kubeconfig files as well without any luck.

Does it also fail using other kubeconfigs in this dir? /etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/node-kubeconfigs/localhost.kubeconfig should work imo

0 replies

darcystan · 2023-06-05T17:08:11Z

darcystan
Jun 5, 2023
Author

Yes, it fails with the other kubeconfigs as well. For the locahost.kubeconfig I get an Unauthorized message:

[root@master0 ~]# export KUBECONFIG=/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/node-kubeconfigs/localhost.kubeconfig
[root@master0 ~]# oc get csr
error: You must be logged in to the server (Unauthorized)

Here's with loglevel set to 10:

[root@master0 ~]# oc get csr --loglevel 10
I0605 16:53:12.279664 1144388 loader.go:372] Config loaded from file:  /etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/node-kubeconfigs/localhost.kubeconfig
I0605 16:53:12.280499 1144388 round_trippers.go:466] curl -v -XGET  -H "User-Agent: oc/4.7.0 (linux/amd64) kubernetes/unknown" -H "Accept: application/json, */*" 'https://localhost:6443/api?timeout=32s'
I0605 16:53:12.281117 1144388 round_trippers.go:495] HTTP Trace: DNS Lookup for localhost resolved to [{::1 } {127.0.0.1 }]
I0605 16:53:12.281341 1144388 round_trippers.go:510] HTTP Trace: Dial to tcp:[::1]:6443 succeed
I0605 16:53:12.286685 1144388 round_trippers.go:570] HTTP Statistics: DNSLookup 0 ms Dial 0 ms TLSHandshake 4 ms ServerProcessing 0 ms Duration 6 ms
I0605 16:53:12.286701 1144388 round_trippers.go:577] Response Headers:
I0605 16:53:12.286708 1144388 round_trippers.go:580]     Date: Mon, 05 Jun 2023 16:53:12 GMT
I0605 16:53:12.286719 1144388 round_trippers.go:580]     Audit-Id: 423c0066-d35a-497a-a87c-aa1eb6d8b8cf
I0605 16:53:12.286729 1144388 round_trippers.go:580]     Cache-Control: no-cache, private
I0605 16:53:12.286738 1144388 round_trippers.go:580]     Content-Type: application/json
I0605 16:53:12.286747 1144388 round_trippers.go:580]     Content-Length: 129
I0605 16:53:12.301301 1144388 request.go:1181] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
I0605 16:53:12.303135 1144388 cached_discovery.go:121] skipped caching discovery info due to Unauthorized
I0605 16:53:12.303228 1144388 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: oc/4.7.0 (linux/amd64) kubernetes/unknown" 'https://localhost:6443/api?timeout=32s'
I0605 16:53:12.303926 1144388 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 0 ms Duration 0 ms
I0605 16:53:12.303938 1144388 round_trippers.go:577] Response Headers:
I0605 16:53:12.303952 1144388 round_trippers.go:580]     Audit-Id: 86e592e2-4e9d-4a93-b3d8-35488343c346
I0605 16:53:12.303959 1144388 round_trippers.go:580]     Cache-Control: no-cache, private
I0605 16:53:12.303967 1144388 round_trippers.go:580]     Content-Type: application/json
I0605 16:53:12.303976 1144388 round_trippers.go:580]     Content-Length: 129
I0605 16:53:12.303982 1144388 round_trippers.go:580]     Date: Mon, 05 Jun 2023 16:53:12 GMT
I0605 16:53:12.305133 1144388 request.go:1181] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
I0605 16:53:12.306370 1144388 cached_discovery.go:121] skipped caching discovery info due to Unauthorized
I0605 16:53:12.306392 1144388 shortcut.go:89] Error loading discovery information: Unauthorized
I0605 16:53:12.306454 1144388 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: oc/4.7.0 (linux/amd64) kubernetes/unknown" 'https://localhost:6443/api?timeout=32s'
I0605 16:53:12.307097 1144388 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 0 ms Duration 0 ms
I0605 16:53:12.307127 1144388 round_trippers.go:577] Response Headers:
I0605 16:53:12.307135 1144388 round_trippers.go:580]     Content-Type: application/json
I0605 16:53:12.307141 1144388 round_trippers.go:580]     Content-Length: 129
I0605 16:53:12.307148 1144388 round_trippers.go:580]     Date: Mon, 05 Jun 2023 16:53:12 GMT
I0605 16:53:12.307157 1144388 round_trippers.go:580]     Audit-Id: 358615be-dc73-4339-8e6a-f23a3ba8033f
I0605 16:53:12.307165 1144388 round_trippers.go:580]     Cache-Control: no-cache, private
I0605 16:53:12.308176 1144388 request.go:1181] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
I0605 16:53:12.309215 1144388 cached_discovery.go:121] skipped caching discovery info due to Unauthorized
I0605 16:53:12.309338 1144388 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: oc/4.7.0 (linux/amd64) kubernetes/unknown" 'https://localhost:6443/api?timeout=32s'
I0605 16:53:12.309946 1144388 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 0 ms Duration 0 ms
I0605 16:53:12.309958 1144388 round_trippers.go:577] Response Headers:
I0605 16:53:12.309963 1144388 round_trippers.go:580]     Date: Mon, 05 Jun 2023 16:53:12 GMT
I0605 16:53:12.309967 1144388 round_trippers.go:580]     Audit-Id: df1ae698-23e2-4430-a782-8fd464263e91
I0605 16:53:12.309970 1144388 round_trippers.go:580]     Cache-Control: no-cache, private
I0605 16:53:12.309974 1144388 round_trippers.go:580]     Content-Type: application/json
I0605 16:53:12.309978 1144388 round_trippers.go:580]     Content-Length: 129
I0605 16:53:12.310667 1144388 request.go:1181] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
I0605 16:53:12.311780 1144388 cached_discovery.go:121] skipped caching discovery info due to Unauthorized
I0605 16:53:12.311856 1144388 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: oc/4.7.0 (linux/amd64) kubernetes/unknown" 'https://localhost:6443/api?timeout=32s'
I0605 16:53:12.312462 1144388 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 0 ms Duration 0 ms
I0605 16:53:12.312472 1144388 round_trippers.go:577] Response Headers:
I0605 16:53:12.312476 1144388 round_trippers.go:580]     Content-Length: 129
I0605 16:53:12.312480 1144388 round_trippers.go:580]     Date: Mon, 05 Jun 2023 16:53:12 GMT
I0605 16:53:12.312484 1144388 round_trippers.go:580]     Audit-Id: d18602af-9086-47f8-a30a-04201559614b
I0605 16:53:12.312487 1144388 round_trippers.go:580]     Cache-Control: no-cache, private
I0605 16:53:12.312491 1144388 round_trippers.go:580]     Content-Type: application/json
I0605 16:53:12.313737 1144388 request.go:1181] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
I0605 16:53:12.314872 1144388 cached_discovery.go:121] skipped caching discovery info due to Unauthorized
I0605 16:53:12.315003 1144388 helpers.go:219] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "Unauthorized",
  "reason": "Unauthorized",
  "code": 401
}]
F0605 16:53:12.315075 1144388 helpers.go:118] error: You must be logged in to the server (Unauthorized)
goroutine 1 [running]:
k8s.io/klog/v2.stacks(0x1)
        /go/src/github.com/openshift/oc/_rpmbuild/BUILD/openshift-clients-0.0.1/__gopath/src/github.com/openshift/oc/vendor/k8s.io/klog/v2/klog.go:1140 +0x8a
k8s.io/klog/v2.(*loggingT).output(0x5b23f60, 0x3, 0x0, 0xc000233c70, 0x2, {0x484c265, 0x10}, 0xc000180000, 0x0)
        /go/src/github.com/openshift/oc/_rpmbuild/BUILD/openshift-clients-0.0.1/__gopath/src/github.com/openshift/oc/vendor/k8s.io/klog/v2/klog.go:1088 +0x66f
k8s.io/klog/v2.(*loggingT).printDepth(0xc000907740, 0x39, 0x0, {0x0, 0x0}, 0xc000ad2500, {0xc000152220, 0x1, 0x1})
        /go/src/github.com/openshift/oc/_rpmbuild/BUILD/openshift-clients-0.0.1/__gopath/src/github.com/openshift/oc/vendor/k8s.io/klog/v2/klog.go:735 +0x1ae
k8s.io/klog/v2.FatalDepth(...)
        /go/src/github.com/openshift/oc/_rpmbuild/BUILD/openshift-clients-0.0.1/__gopath/src/github.com/openshift/oc/vendor/k8s.io/klog/v2/klog.go:1628
k8s.io/kubectl/pkg/cmd/util.fatal({0xc000907740, 0x39}, 0xc000953540)
        /go/src/github.com/openshift/oc/_rpmbuild/BUILD/openshift-clients-0.0.1/__gopath/src/github.com/openshift/oc/vendor/k8s.io/kubectl/pkg/cmd/util/helpers.go:96 +0xc5
k8s.io/kubectl/pkg/cmd/util.checkErr({0x401f240, 0xc000953540}, 0x3d1bb20)
        /go/src/github.com/openshift/oc/_rpmbuild/BUILD/openshift-clients-0.0.1/__gopath/src/github.com/openshift/oc/vendor/k8s.io/kubectl/pkg/cmd/util/helpers.go:191 +0x7d7
k8s.io/kubectl/pkg/cmd/util.CheckErr(...)
        /go/src/github.com/openshift/oc/_rpmbuild/BUILD/openshift-clients-0.0.1/__gopath/src/github.com/openshift/oc/vendor/k8s.io/kubectl/pkg/cmd/util/helpers.go:118
k8s.io/kubectl/pkg/cmd/get.NewCmdGet.func2(0xc00070e000, {0xc000925ef0, 0x1, 0x3})
        /go/src/github.com/openshift/oc/_rpmbuild/BUILD/openshift-clients-0.0.1/__gopath/src/github.com/openshift/oc/vendor/k8s.io/kubectl/pkg/cmd/get/get.go:181 +0xc8
github.com/spf13/cobra.(*Command).execute(0xc00070e000, {0xc000925ec0, 0x3, 0x3})
        /go/src/github.com/openshift/oc/_rpmbuild/BUILD/openshift-clients-0.0.1/__gopath/src/github.com/openshift/oc/vendor/github.com/spf13/cobra/command.go:860 +0x5f8
github.com/spf13/cobra.(*Command).ExecuteC(0xc00098db80)
        /go/src/github.com/openshift/oc/_rpmbuild/BUILD/openshift-clients-0.0.1/__gopath/src/github.com/openshift/oc/vendor/github.com/spf13/cobra/command.go:974 +0x3bc
github.com/spf13/cobra.(*Command).Execute(...)
        /go/src/github.com/openshift/oc/_rpmbuild/BUILD/openshift-clients-0.0.1/__gopath/src/github.com/openshift/oc/vendor/github.com/spf13/cobra/command.go:902
k8s.io/component-base/cli.run(0xc00098db80)
        /go/src/github.com/openshift/oc/_rpmbuild/BUILD/openshift-clients-0.0.1/__gopath/src/github.com/openshift/oc/vendor/k8s.io/component-base/cli/run.go:146 +0x325
k8s.io/component-base/cli.RunNoErrOutput(...)
        /go/src/github.com/openshift/oc/_rpmbuild/BUILD/openshift-clients-0.0.1/__gopath/src/github.com/openshift/oc/vendor/k8s.io/component-base/cli/run.go:84
main.main()
        /go/src/github.com/openshift/oc/_rpmbuild/BUILD/openshift-clients-0.0.1/__gopath/src/github.com/openshift/oc/cmd/oc/oc.go:79 +0x385

goroutine 4 [chan receive]:
k8s.io/klog/v2.(*loggingT).flushDaemon(0x0)
        /go/src/github.com/openshift/oc/_rpmbuild/BUILD/openshift-clients-0.0.1/__gopath/src/github.com/openshift/oc/vendor/k8s.io/klog/v2/klog.go:1283 +0x6a
created by k8s.io/klog/v2.init.0
        /go/src/github.com/openshift/oc/_rpmbuild/BUILD/openshift-clients-0.0.1/__gopath/src/github.com/openshift/oc/vendor/k8s.io/klog/v2/klog.go:420 +0xfb

goroutine 14 [select]:
io.(*pipe).Read(0xc00094a9c0, {0xc0001ea000, 0x1000, 0xc00006d601})
        /usr/lib/golang/src/io/pipe.go:57 +0xb7
io.(*PipeReader).Read(0x4, {0xc0001ea000, 0x1200c89, 0x3563780})
        /usr/lib/golang/src/io/pipe.go:134 +0x25
bufio.(*Scanner).Scan(0xc0001a2f00)
        /usr/lib/golang/src/bufio/scan.go:215 +0x865
github.com/openshift/oc/pkg/cli/admin/mustgather.newPrefixWriter.func1()
        /go/src/github.com/openshift/oc/_rpmbuild/BUILD/openshift-clients-0.0.1/__gopath/src/github.com/openshift/oc/pkg/cli/admin/mustgather/mustgather.go:527 +0x105
created by github.com/openshift/oc/pkg/cli/admin/mustgather.newPrefixWriter
        /go/src/github.com/openshift/oc/_rpmbuild/BUILD/openshift-clients-0.0.1/__gopath/src/github.com/openshift/oc/pkg/cli/admin/mustgather/mustgather.go:526 +0x21a

goroutine 15 [select]:
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0, {0x401f540, 0xc000a24000}, 0x1, 0xc0000a0360)
        /go/src/github.com/openshift/oc/_rpmbuild/BUILD/openshift-clients-0.0.1/__gopath/src/github.com/openshift/oc/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:167 +0x13b
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0, 0x12a05f200, 0x0, 0x0, 0x0)
        /go/src/github.com/openshift/oc/_rpmbuild/BUILD/openshift-clients-0.0.1/__gopath/src/github.com/openshift/oc/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /go/src/github.com/openshift/oc/_rpmbuild/BUILD/openshift-clients-0.0.1/__gopath/src/github.com/openshift/oc/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90
k8s.io/apimachinery/pkg/util/wait.Forever(0x0, 0x0)
        /go/src/github.com/openshift/oc/_rpmbuild/BUILD/openshift-clients-0.0.1/__gopath/src/github.com/openshift/oc/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:81 +0x28
created by k8s.io/component-base/logs.InitLogs
        /go/src/github.com/openshift/oc/_rpmbuild/BUILD/openshift-clients-0.0.1/__gopath/src/github.com/openshift/oc/vendor/k8s.io/component-base/logs/logs.go:179 +0x85

goroutine 25 [IO wait]:
internal/poll.runtime_pollWait(0x7f6e28677f98, 0x72)
        /usr/lib/golang/src/runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0xc000453200, 0xc000211500, 0x0)
        /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
        /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000453200, {0xc000211500, 0x1427, 0x1427})
        /usr/lib/golang/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc000453200, {0xc000211500, 0xc000211505, 0x9b})
        /usr/lib/golang/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc0003e2038, {0xc000211500, 0x400, 0xc000ab37f0})
        /usr/lib/golang/src/net/net.go:183 +0x45
crypto/tls.(*atLeastReader).Read(0xc0003cf398, {0xc000211500, 0x0, 0x12052ed})
        /usr/lib/golang/src/crypto/tls/conn.go:777 +0x3d
bytes.(*Buffer).ReadFrom(0xc000a2e278, {0x401b000, 0xc0003cf398})
        /usr/lib/golang/src/bytes/buffer.go:204 +0x98
crypto/tls.(*Conn).readFromUntil(0xc000a2e000, {0x40208e0, 0xc0003e2038}, 0x1427)
        /usr/lib/golang/src/crypto/tls/conn.go:799 +0xe5
crypto/tls.(*Conn).readRecordOrCCS(0xc000a2e000, 0x0)
        /usr/lib/golang/src/crypto/tls/conn.go:606 +0x112
crypto/tls.(*Conn).readRecord(...)
        /usr/lib/golang/src/crypto/tls/conn.go:574
crypto/tls.(*Conn).Read(0xc000a2e000, {0xc000a5e000, 0x1000, 0x1733ec0})
        /usr/lib/golang/src/crypto/tls/conn.go:1277 +0x16f
bufio.(*Reader).Read(0xc0004277a0, {0xc000a462e0, 0x9, 0x1741e82})
        /usr/lib/golang/src/bufio/bufio.go:227 +0x1b4
io.ReadAtLeast({0x401ad80, 0xc0004277a0}, {0xc000a462e0, 0x9, 0x9}, 0x9)
        /usr/lib/golang/src/io/io.go:328 +0x9a
io.ReadFull(...)
        /usr/lib/golang/src/io/io.go:347
golang.org/x/net/http2.readFrameHeader({0xc000a462e0, 0x9, 0xc001a76480}, {0x401ad80, 0xc0004277a0})
        /go/src/github.com/openshift/oc/_rpmbuild/BUILD/openshift-clients-0.0.1/__gopath/src/github.com/openshift/oc/vendor/golang.org/x/net/http2/frame.go:237 +0x6e
golang.org/x/net/http2.(*Framer).ReadFrame(0xc000a462a0)
        /go/src/github.com/openshift/oc/_rpmbuild/BUILD/openshift-clients-0.0.1/__gopath/src/github.com/openshift/oc/vendor/golang.org/x/net/http2/frame.go:498 +0x95
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc000ab3f98)
        /go/src/github.com/openshift/oc/_rpmbuild/BUILD/openshift-clients-0.0.1/__gopath/src/github.com/openshift/oc/vendor/golang.org/x/net/http2/transport.go:2101 +0x130
golang.org/x/net/http2.(*ClientConn).readLoop(0xc00011a300)
        /go/src/github.com/openshift/oc/_rpmbuild/BUILD/openshift-clients-0.0.1/__gopath/src/github.com/openshift/oc/vendor/golang.org/x/net/http2/transport.go:1997 +0x6f
created by golang.org/x/net/http2.(*Transport).newClientConn
        /go/src/github.com/openshift/oc/_rpmbuild/BUILD/openshift-clients-0.0.1/__gopath/src/github.com/openshift/oc/vendor/golang.org/x/net/http2/transport.go:725 +0xac5

I also noticed the oc client on the nodes is old, I would have thought it get's updated with cluster updates but I guess not. Is it fine to replace the oc binary on them?

2 replies

vrutkovs Aug 14, 2023
Maintainer

If "few month" is longer than 120 days, then localhost-recovery and other node certs have expired. We're working on automatic master renewal CSR approve - see https://issues.redhat.com/browse/API-1604

For now the workaround is:

find control-plane-... certs and kubeconfig in /etc/kubertnetes
contruct a new kubeconfig using tls.crt, tls.key and ca bundle
use this kubeconfig to approve CSRs
wait for the operator to regenerate localhost.kubeconfig
wait for it to stabilize

In general we don't support shutting down the cluster longer than 30 days, but in extreme cases you can shutdown the cluster for ~90 days (until kubeconfigs in /etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/node-kubeconfigs/ expire)

darcystan Aug 17, 2023
Author

This suggestion worked, thank you!

darcystan · 2023-06-14T19:21:41Z

darcystan
Jun 14, 2023
Author

I tried with an updated oc binary as well but same result:

[root@master0 ~]# export KUBECONFIG=/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/node-kubeconfigs/localhost.kubeconfig
[root@master0 ~]# /tmp/oc get csr
error: You must be logged in to the server (Unauthorized)
[root@master0 ~]# /tmp/oc get csr --loglevel 10
I0614 19:20:36.146821  999690 loader.go:374] Config loaded from file:  /etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/node-kubeconfigs/localhost.kubeconfig
I0614 19:20:36.147524  999690 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: oc/4.12.0 (linux/amd64) kubernetes/854f807" 'https://localhost:6443/api?timeout=32s'
I0614 19:20:36.148203  999690 round_trippers.go:495] HTTP Trace: DNS Lookup for localhost resolved to [{::1 } {127.0.0.1 }]
I0614 19:20:36.148397  999690 round_trippers.go:510] HTTP Trace: Dial to tcp:[::1]:6443 succeed
I0614 19:20:36.153320  999690 round_trippers.go:553] GET https://localhost:6443/api?timeout=32s 401 Unauthorized in 5 milliseconds
I0614 19:20:36.153338  999690 round_trippers.go:570] HTTP Statistics: DNSLookup 0 ms Dial 0 ms TLSHandshake 3 ms ServerProcessing 0 ms Duration 5 ms
I0614 19:20:36.153344  999690 round_trippers.go:577] Response Headers:
I0614 19:20:36.153353  999690 round_trippers.go:580]     Audit-Id: 4f4f8fee-d377-4792-9d9e-3d35c1d5340c
I0614 19:20:36.153361  999690 round_trippers.go:580]     Cache-Control: no-cache, private
I0614 19:20:36.153368  999690 round_trippers.go:580]     Content-Type: application/json
I0614 19:20:36.153376  999690 round_trippers.go:580]     Content-Length: 129
I0614 19:20:36.153383  999690 round_trippers.go:580]     Date: Wed, 14 Jun 2023 19:20:36 GMT
I0614 19:20:36.153574  999690 request.go:1154] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
I0614 19:20:36.153819  999690 cached_discovery.go:119] skipped caching discovery info due to Unauthorized
I0614 19:20:36.153909  999690 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: oc/4.12.0 (linux/amd64) kubernetes/854f807" 'https://localhost:6443/api?timeout=32s'
I0614 19:20:36.154576  999690 round_trippers.go:553] GET https://localhost:6443/api?timeout=32s 401 Unauthorized in 0 milliseconds
I0614 19:20:36.154590  999690 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 0 ms Duration 0 ms
I0614 19:20:36.154598  999690 round_trippers.go:577] Response Headers:
I0614 19:20:36.154606  999690 round_trippers.go:580]     Audit-Id: e3299a57-cbbc-429d-aad8-daea9d173173
I0614 19:20:36.154614  999690 round_trippers.go:580]     Cache-Control: no-cache, private
I0614 19:20:36.154618  999690 round_trippers.go:580]     Content-Type: application/json
I0614 19:20:36.154626  999690 round_trippers.go:580]     Content-Length: 129
I0614 19:20:36.154633  999690 round_trippers.go:580]     Date: Wed, 14 Jun 2023 19:20:36 GMT
I0614 19:20:36.154749  999690 request.go:1154] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
I0614 19:20:36.154862  999690 cached_discovery.go:119] skipped caching discovery info due to Unauthorized
I0614 19:20:36.154908  999690 shortcut.go:100] Error loading discovery information: Unauthorized
I0614 19:20:36.154970  999690 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: oc/4.12.0 (linux/amd64) kubernetes/854f807" 'https://localhost:6443/api?timeout=32s'
I0614 19:20:36.155482  999690 round_trippers.go:553] GET https://localhost:6443/api?timeout=32s 401 Unauthorized in 0 milliseconds
I0614 19:20:36.155543  999690 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 0 ms Duration 0 ms
I0614 19:20:36.155570  999690 round_trippers.go:577] Response Headers:
I0614 19:20:36.155595  999690 round_trippers.go:580]     Audit-Id: b25a0ac3-5d66-4655-9edc-782461b97a94
I0614 19:20:36.155617  999690 round_trippers.go:580]     Cache-Control: no-cache, private
I0614 19:20:36.155640  999690 round_trippers.go:580]     Content-Type: application/json
I0614 19:20:36.155685  999690 round_trippers.go:580]     Content-Length: 129
I0614 19:20:36.155711  999690 round_trippers.go:580]     Date: Wed, 14 Jun 2023 19:20:36 GMT
I0614 19:20:36.155849  999690 request.go:1154] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
I0614 19:20:36.156060  999690 cached_discovery.go:119] skipped caching discovery info due to Unauthorized
I0614 19:20:36.156171  999690 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: oc/4.12.0 (linux/amd64) kubernetes/854f807" 'https://localhost:6443/api?timeout=32s'
I0614 19:20:36.156766  999690 round_trippers.go:553] GET https://localhost:6443/api?timeout=32s 401 Unauthorized in 0 milliseconds
I0614 19:20:36.156780  999690 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 0 ms Duration 0 ms
I0614 19:20:36.156784  999690 round_trippers.go:577] Response Headers:
I0614 19:20:36.156794  999690 round_trippers.go:580]     Date: Wed, 14 Jun 2023 19:20:36 GMT
I0614 19:20:36.156802  999690 round_trippers.go:580]     Audit-Id: 320e6c06-0946-45c2-8a2d-acc6606d8c59
I0614 19:20:36.156809  999690 round_trippers.go:580]     Cache-Control: no-cache, private
I0614 19:20:36.156818  999690 round_trippers.go:580]     Content-Type: application/json
I0614 19:20:36.156826  999690 round_trippers.go:580]     Content-Length: 129
I0614 19:20:36.156936  999690 request.go:1154] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
I0614 19:20:36.157053  999690 cached_discovery.go:119] skipped caching discovery info due to Unauthorized
I0614 19:20:36.157101  999690 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: oc/4.12.0 (linux/amd64) kubernetes/854f807" 'https://localhost:6443/api?timeout=32s'
I0614 19:20:36.157689  999690 round_trippers.go:553] GET https://localhost:6443/api?timeout=32s 401 Unauthorized in 0 milliseconds
I0614 19:20:36.157770  999690 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 0 ms Duration 0 ms
I0614 19:20:36.157811  999690 round_trippers.go:577] Response Headers:
I0614 19:20:36.157863  999690 round_trippers.go:580]     Content-Type: application/json
I0614 19:20:36.157947  999690 round_trippers.go:580]     Content-Length: 129
I0614 19:20:36.158001  999690 round_trippers.go:580]     Date: Wed, 14 Jun 2023 19:20:36 GMT
I0614 19:20:36.158075  999690 round_trippers.go:580]     Audit-Id: 6d382ab1-9f23-48d4-88cf-a38302701db9
I0614 19:20:36.158130  999690 round_trippers.go:580]     Cache-Control: no-cache, private
I0614 19:20:36.158334  999690 request.go:1154] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
I0614 19:20:36.158591  999690 cached_discovery.go:119] skipped caching discovery info due to Unauthorized
I0614 19:20:36.158806  999690 helpers.go:246] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "Unauthorized",
  "reason": "Unauthorized",
  "code": 401
}]
error: You must be logged in to the server (Unauthorized)

0 replies

asaf-refaeli · 2023-08-14T10:09:25Z

asaf-refaeli
Aug 14, 2023

Hi, I am having the exact same issue, did you manage to solve it?
@darcystan

4 replies

darcystan Aug 17, 2023
Author

Yes, after trying the latest suggestion from @vrutkovs it worked! A quick recap:

Copied the control plane kubeconfig:
cp /etc/kubernetes/static-pod-resources/kube-apiserver-certs/configmaps/control-plane-node-kubeconfig/kubeconfig /tmp/kubeconfig

Updated the certificate values in the copied kubeconfig:

certificate-authority: /etc/kubernetes/static-pod-resources/kube-apiserver-certs/configmaps/trusted-ca-bundle/ca-bundle.crt
...
client-certificate: /etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/control-plane-node-admin-client-cert-key/tls.crt
client-key: /etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/control-plane-node-admin-client-cert-key/tls.key

Set KUBECONFIG env var to the new temp kubeconfig file:
export KUBECONFIG=/tmp/kubeconfig

Approved the certs:
oc get csr -o name --insecure-skip-tls-verify | xargs oc adm certificate approve --insecure-skip-tls-verify

vrutkovs Aug 17, 2023
Maintainer

Perfect. We're working on making this happen seamlessly, stay tuned

ahmedalazazy Oct 7, 2024

oc get csr -o name --insecure-skip-tls-verify
The connection to the server localhost:6443 was refused - did you specify the right host or port?

DizzyThermal Oct 15, 2024

@ahmedalazazy

After following @darcystan 's steps, I still couldn't get my API Server up and healthy - same error as you (localhost:6443 was refused)

My issue was that my etcd certificates were expired (3 year anniversary of deploying the cluster).
If you're on OKD 4.8 or below, this might be your issue too.

If crictl ps -a shows etcd container Exited or similar, you may want to try the following steps:
https://gist.github.com/ikurni/236845cbbc04c1115e5ab91df4ae3f65

Cheers!

snirgu · 2023-11-04T21:08:56Z

snirgu
Nov 4, 2023

Hi regard power on openshift cluster v4.10 after more then 90 days it was off.
Is there a verified process to a bring the cluster to life again?

1 reply

vrutkovs Nov 5, 2023
Maintainer

No, not yet

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Issue recovering from expired control plane certificates after prolonged cluster shutdown #1630

{{title}}

Replies: 5 comments 7 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Issue recovering from expired control plane certificates after prolonged cluster shutdown #1630

darcystan Jun 5, 2023

Replies: 5 comments · 7 replies

vrutkovs Jun 5, 2023 Maintainer

darcystan Jun 5, 2023 Author

vrutkovs Aug 14, 2023 Maintainer

darcystan Aug 17, 2023 Author

darcystan Jun 14, 2023 Author

asaf-refaeli Aug 14, 2023

darcystan Aug 17, 2023 Author

vrutkovs Aug 17, 2023 Maintainer

ahmedalazazy Oct 7, 2024

DizzyThermal Oct 15, 2024

snirgu Nov 4, 2023

vrutkovs Nov 5, 2023 Maintainer

darcystan
Jun 5, 2023

Replies: 5 comments 7 replies

vrutkovs
Jun 5, 2023
Maintainer

darcystan
Jun 5, 2023
Author

vrutkovs Aug 14, 2023
Maintainer

darcystan Aug 17, 2023
Author

darcystan
Jun 14, 2023
Author

asaf-refaeli
Aug 14, 2023

darcystan Aug 17, 2023
Author

vrutkovs Aug 17, 2023
Maintainer

snirgu
Nov 4, 2023

vrutkovs Nov 5, 2023
Maintainer