v1.28.3+rke2r2，rke2-server start failed #5052
Comments
liyang516
changed the title
|
Check the pod logs under /var/log/pods and containerd log at /var/lib/rancher/rke2/agent/containerd/containerd.log to see why etcd isn't starting. If this is airgapped, confirm that the etcd image is available on your mirror. |
|
I'm getting the same set of errors when I run the install script according to the RKE2 quickstart guide (even with the firewall turned off to see if that's the issue). I've been trying to figure out what's going on for days, but I'm stuck because this is my first time trying to run anything related to docker/kubernetes/rke2/rancher. I'll be following this thread to see what the solution is.
containerd log |
|
Solution:
This works as expected, and things work fine. Took me 5 days straight to figure that out. |
|
@brandond Hi，thanks for your reply. # tail -f 1.log
2023-11-20T10:54:57.308352068+08:00 stderr F {"level":"info","ts":"2023-11-20T02:54:57.308285Z","caller":"fileutil/purge.go:44","msg":"started to purge file","dir":"/var/lib/rancher/rke2/server/db/etcd/member/snap","suffix":"snap","max":5,"interval":"30s"}
2023-11-20T10:54:57.308399627+08:00 stderr F {"level":"info","ts":"2023-11-20T02:54:57.308308Z","caller":"fileutil/purge.go:44","msg":"started to purge file","dir":"/var/lib/rancher/rke2/server/db/etcd/member/wal","suffix":"wal","max":5,"interval":"30s"}
2023-11-20T10:54:57.311396105+08:00 stderr F {"level":"info","ts":"2023-11-20T02:54:57.311245Z","caller":"embed/etcd.go:726","msg":"starting with client TLS","tls-info":"cert = /var/lib/rancher/rke2/server/tls/etcd/server-client.crt, key = /var/lib/rancher/rke2/server/tls/etcd/server-client.key, client-cert=, client-key=, trusted-ca = /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt, client-cert-auth = true, crl-file = ","cipher-suites":[]}
2023-11-20T10:54:57.311418482+08:00 stderr F {"level":"info","ts":"2023-11-20T02:54:57.311319Z","caller":"embed/etcd.go:597","msg":"serving peer traffic","address":"127.0.0.1:2380"}
2023-11-20T10:54:57.311423555+08:00 stderr F {"level":"info","ts":"2023-11-20T02:54:57.311373Z","caller":"embed/etcd.go:569","msg":"cmux::serve","address":"127.0.0.1:2380"}
2023-11-20T10:54:57.311427353+08:00 stderr F {"level":"info","ts":"2023-11-20T02:54:57.311314Z","caller":"embed/etcd.go:597","msg":"serving peer traffic","address":"10.57.131.130:2380"}
2023-11-20T10:54:57.311434672+08:00 stderr F {"level":"info","ts":"2023-11-20T02:54:57.31139Z","caller":"embed/etcd.go:569","msg":"cmux::serve","address":"10.57.131.130:2380"}
2023-11-20T10:54:57.311619926+08:00 stderr F {"level":"info","ts":"2023-11-20T02:54:57.311501Z","caller":"embed/etcd.go:278","msg":"now serving peer/client/metrics","local-member-id":"1652804ea3b95f3f","initial-advertise-peer-urls":["https://10.57.131.130:2380"],"listen-peer-urls":["https://10.57.131.130:2380","https://127.0.0.1:2380"],"advertise-client-urls":["https://10.57.131.130:2379"],"listen-client-urls":["https://10.57.131.130:2379","https://127.0.0.1:2379"],"listen-metrics-urls":["http://127.0.0.1:2381"]}
2023-11-20T10:54:57.311691718+08:00 stderr F {"level":"info","ts":"2023-11-20T02:54:57.311538Z","caller":"embed/etcd.go:855","msg":"serving metrics","address":"http://127.0.0.1:2381"}
2023-11-20T10:55:12.309345393+08:00 stderr F {"level":"warn","ts":"2023-11-20T02:55:12.308926Z","caller":"etcdserver/server.go:2085","msg":"failed to publish local member to cluster through raft","local-member-id":"1652804ea3b95f3f","local-member-attributes":"{Name:vm1-470fdc6d ClientURLs:[https://10.57.131.130:2379]}","request-path":"/0/members/1652804ea3b95f3f/attributes","publish-timeout":"15s","error":"etcdserver: request timed out"}
# tail 2.log
2023-11-20T10:10:56.4017914+08:00 stderr F {"level":"info","ts":"2023-11-20T02:10:56.401601Z","caller":"embed/serve.go:250","msg":"serving client traffic securely","traffic":"grpc","address":"127.0.0.1:2379"}
2023-11-20T10:10:56.401810415+08:00 stderr F {"level":"info","ts":"2023-11-20T02:10:56.401602Z","caller":"embed/serve.go:250","msg":"serving client traffic securely","traffic":"grpc","address":"10.57.131.130:2379"}
2023-11-20T10:10:58.974717526+08:00 stderr F {"level":"info","ts":"2023-11-20T02:10:58.974378Z","caller":"osutil/interrupt_unix.go:64","msg":"received signal; shutting down","signal":"terminated"}
2023-11-20T10:10:58.974770941+08:00 stderr F {"level":"info","ts":"2023-11-20T02:10:58.974526Z","caller":"embed/etcd.go:376","msg":"closing etcd server","name":"vm1-acc0de71","data-dir":"/var/lib/rancher/rke2/server/db/etcd","advertise-peer-urls":["https://10.57.131.130:2380"],"advertise-client-urls":["https://10.57.131.130:2379"]}
2023-11-20T10:10:59.779775767+08:00 stderr F {"level":"info","ts":"2023-11-20T02:10:59.77964Z","caller":"etcdserver/server.go:1465","msg":"skipped leadership transfer for single voting member cluster","local-member-id":"1652804ea3b95f3f","current-leader-member-id":"1652804ea3b95f3f"}
2023-11-20T10:10:59.786814959+08:00 stderr F {"level":"info","ts":"2023-11-20T02:10:59.786622Z","caller":"embed/etcd.go:579","msg":"stopping serving peer traffic","address":"10.57.131.130:2380"}
2023-11-20T10:10:59.78694604+08:00 stderr F {"level":"info","ts":"2023-11-20T02:10:59.786807Z","caller":"embed/etcd.go:584","msg":"stopped serving peer traffic","address":"10.57.131.130:2380"}
2023-11-20T10:10:59.786963102+08:00 stderr F {"level":"info","ts":"2023-11-20T02:10:59.786849Z","caller":"embed/etcd.go:579","msg":"stopping serving peer traffic","address":"127.0.0.1:2380"}
2023-11-20T10:10:59.787017621+08:00 stderr F {"level":"info","ts":"2023-11-20T02:10:59.786928Z","caller":"embed/etcd.go:584","msg":"stopped serving peer traffic","address":"127.0.0.1:2380"}
2023-11-20T10:10:59.787038907+08:00 stderr F {"level":"info","ts":"2023-11-20T02:10:59.786947Z","caller":"embed/etcd.go:378","msg":"closed etcd server","name":"vm1-acc0de71","data-dir":"/var/lib/rancher/rke2/server/db/etcd","advertise-peer-urls":["https://10.57.131.130:2380"],"advertise-client-urls":["https://10.57.131.130:2379"]}
|
Please confirm that you have disabled firewalld, ufw, or any other iptables-based host firewalls. It seems that both rke2 and etcd itself cannot communicate with the etcd ports. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Environmental Info:
RKE2 Version:
v1.28.3+rke2r2
Node(s) CPU architecture, OS, and Version:
OS： Ubuntu-22.04.1
Arch：x86-64
Cluster Configuration:
Describe the bug:
rke2-server can not start, log always output "Waiting for etcd server to become available"
Steps To Reproduce:
Expected behavior:
rke2-server service start succeed
Actual behavior:
rke2-server service start failed
Other:
It seems that etcd Auto sync endpoints failed
The text was updated successfully, but these errors were encountered: