/var/lib/rancher/rke2/server/cred/passwd newer than datastore and could cause a cluster outage. Remove the file(s ) from disk and restart to be recreated from datastore. #4035

kongzi68 · 2023-03-20T07:17:58Z

kongzi68
Mar 20, 2023

Environmental Info:
RKE2 Version:
rke2 version v1.23.6+rke2r2 (40d712e)
go version go1.17.5b7

Node(s) CPU architecture, OS, and Version:
Linux k8s-master02 5.4.0-144-generic #161-Ubuntu SMP Fri Feb 3 14:49:04 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:
3 master node，9 worker node

Describe the bug:

rke2-server always restart，check rke2-server logs：
Mar 20 12:46:18 k8s-master02 rke2[2939538]: time="2023-03-20T12:46:18+08:00" level=fatal msg="/var/lib/rancher/rke2/server/cred/passwd newer than datastore and could cause a cluster outage. Remove the file(s ) from disk and restart to be recreated from datastore."
delete the file /var/lib/rancher/rke2/server/cred/passwd，rke2-server restart succeeded。
but，Why is this fatal log generated?

Steps To Reproduce:

Installed RKE2:

Expected behavior:

Actual behavior:

Additional context / logs:

Mar 20 12:46:09 k8s-master02 rke2[2939538]: time="2023-03-20T12:46:09+08:00" level=warning msg="not running in CIS mode"
Mar 20 12:46:09 k8s-master02 rke2[2939538]: time="2023-03-20T12:46:09+08:00" level=info msg="Starting rke2 v1.23.6+rke2r2 (40d712e5081ac87e30e8f328f738130acf2c31f8)"
Mar 20 12:46:09 k8s-master02 rke2[2939538]: time="2023-03-20T12:46:09+08:00" level=info msg="Managed etcd cluster bootstrap already complete and initialized"
Mar 20 12:46:09 k8s-master02 rke2[2939538]: time="2023-03-20T12:46:09+08:00" level=info msg="Starting temporary etcd to reconcile with datastore"
Mar 20 12:46:10 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:10.602+0800","caller":"embed/etcd.go:131","msg":"configuring peer listeners","listen-peer-urls":["http://127.0.0.1:2400"]}
Mar 20 12:46:10 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:10.602+0800","caller":"embed/etcd.go:139","msg":"configuring client listeners","listen-client-urls":["http://127.0.0.1:2399"
]}
Mar 20 12:46:10 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:10.602+0800","caller":"embed/etcd.go:308","msg":"starting an etcd server","etcd-version":"3.5.3","git-sha":"Not provided (us
e ./build instead of go build)","go-version":"go1.17.5b7","go-os":"linux","go-arch":"amd64","max-cpu-set":8,"max-cpu-available":8,"member-initialized":true,"name":"k8s-master02-ca3014b1","data-dir":"/var/lib
/rancher/rke2/server/db/etcd-tmp","wal-dir":"","wal-dir-dedicated":"","member-dir":"/var/lib/rancher/rke2/server/db/etcd-tmp/member","force-new-cluster":true,"heartbeat-interval":"500ms","election-timeout":"
5s","initial-election-tick-advance":true,"snapshot-count":100000,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["http://127.0.0.1:2400"],"listen-peer-urls":["http://127.0.0.1:2400"],"advertis
e-client-urls":["http://127.0.0.1:2399"],"listen-client-urls":["http://127.0.0.1:2399"],"listen-metrics-urls":[],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"","initial-cluster-state":"new","initia
l-cluster-token":"","quota-size-bytes":2147483648,"pre-vote":true,"initial-corrupt-check":false,"corrupt-check-time-interval":"0s","auto-compaction-mode":"","auto-compaction-retention":"0s","auto-compaction-
interval":"0s","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"}
Mar 20 12:46:10 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:10.679+0800","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/var/lib/rancher/rke2/server/db/etcd-tmp/
member/snap/db","took":"76.873602ms"}
Mar 20 12:46:11 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:11.158+0800","caller":"etcdserver/server.go:508","msg":"recovered v2 store from snapshot","snapshot-index":101283861,"snapsh
ot-size":"18 kB"}
Mar 20 12:46:11 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:11.648+0800","caller":"etcdserver/server.go:521","msg":"recovered v3 backend from snapshot","backend-size-bytes":535216128,"
backend-size":"535 MB","backend-size-in-use-bytes":32702464,"backend-size-in-use":"33 MB"}
Mar 20 12:46:11 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:11.897+0800","caller":"etcdserver/raft.go:526","msg":"discarding uncommitted WAL entries","entry-index":101328371,"commit-in
dex-from-wal":101328370,"number-of-discarded-entries":1}
Mar 20 12:46:11 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:11.983+0800","caller":"etcdserver/raft.go:556","msg":"forcing restart member","cluster-id":"5e35a9446cd2dbd9","local-member-
id":"a98197a544df4572","commit-index":101328372}
Mar 20 12:46:11 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:11.986+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"a98197a544df4572 switched to configuration voters=(
2103080054411749682 11891573723487800989 12214210400486376818)"}
Mar 20 12:46:11 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:11.986+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"a98197a544df4572 became follower at term 57"}
Mar 20 12:46:11 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:11.987+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"newRaft a98197a544df4572 [peers: [1d2fa42ab96be932,
a5075b4c18dc0e9d,a98197a544df4572], term: 57, commit: 101328372, applied: 101283861, lastindex: 101328372, lastterm: 57]"}
Mar 20 12:46:11 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:11.987+0800","caller":"api/capability.go:75","msg":"enabled capabilities for version","cluster-version":"3.5"}
Mar 20 12:46:11 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:11.987+0800","caller":"membership/cluster.go:278","msg":"recovered/added member from store","cluster-id":"5e35a9446cd2dbd9",
"local-member-id":"a98197a544df4572","recovered-remote-peer-id":"1d2fa42ab96be932","recovered-remote-peer-urls":["https://192.168.0.192:2380"]}
Mar 20 12:46:11 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:11.987+0800","caller":"membership/cluster.go:278","msg":"recovered/added member from store","cluster-id":"5e35a9446cd2dbd9",
"local-member-id":"a98197a544df4572","recovered-remote-peer-id":"a5075b4c18dc0e9d","recovered-remote-peer-urls":["https://192.168.0.182:2380"]}
Mar 20 12:46:11 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:11.987+0800","caller":"membership/cluster.go:278","msg":"recovered/added member from store","cluster-id":"5e35a9446cd2dbd9",
"local-member-id":"a98197a544df4572","recovered-remote-peer-id":"a98197a544df4572","recovered-remote-peer-urls":["https://192.168.0.59:2380"]}
Mar 20 12:46:11 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:11.987+0800","caller":"membership/cluster.go:287","msg":"set cluster version from store","cluster-version":"3.5"}
Mar 20 12:46:11 k8s-master02 rke2[2939538]: {"level":"warn","ts":"2023-03-20T12:46:11.988+0800","caller":"auth/store.go:1220","msg":"simple token is not cryptographically signed"}
Mar 20 12:46:11 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:11.988+0800","caller":"mvcc/kvstore.go:345","msg":"restored last compact revision","meta-bucket-name":"meta","meta-bucket-na
me-key":"finishedCompactRev","restored-compact-revision":155866720}
Mar 20 12:46:12 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:12.017+0800","caller":"mvcc/kvstore.go:415","msg":"kvstore restored","current-rev":155869524}
Mar 20 12:46:12 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:12.017+0800","caller":"etcdserver/quota.go:94","msg":"enabled backend quota with default value","quota-name":"v3-applier","q
uota-size-bytes":2147483648,"quota-size":"2.1 GB"}
Mar 20 12:46:12 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:12.018+0800","caller":"rafthttp/peer.go:133","msg":"starting remote peer","remote-peer-id":"1d2fa42ab96be932"}
Mar 20 12:46:12 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:12.018+0800","caller":"rafthttp/pipeline.go:72","msg":"started HTTP pipelining with remote peer","local-member-id":"a98197a5
44df4572","remote-peer-id":"1d2fa42ab96be932"}
Mar 20 12:46:12 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:12.018+0800","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"a98197a544d
f4572","remote-peer-id":"1d2fa42ab96be932"}
Mar 20 12:46:12 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:12.018+0800","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"a98197a544d
f4572","remote-peer-id":"1d2fa42ab96be932"}
Mar 20 12:46:12 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:12.019+0800","caller":"rafthttp/peer.go:137","msg":"started remote peer","remote-peer-id":"1d2fa42ab96be932"}
Mar 20 12:46:12 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:12.019+0800","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream M
essage","local-member-id":"a98197a544df4572","remote-peer-id":"1d2fa42ab96be932"}
Mar 20 12:46:12 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:12.019+0800","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream M
sgApp v2","local-member-id":"a98197a544df4572","remote-peer-id":"1d2fa42ab96be932"}
Mar 20 12:46:12 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:12.019+0800","caller":"rafthttp/transport.go:317","msg":"added remote peer","local-member-id":"a98197a544df4572","remote-pee
r-id":"1d2fa42ab96be932","remote-peer-urls":["https://192.168.0.192:2380"]}
Mar 20 12:46:12 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:12.019+0800","caller":"rafthttp/peer.go:133","msg":"starting remote peer","remote-peer-id":"a5075b4c18dc0e9d"}
Mar 20 12:46:12 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:12.019+0800","caller":"rafthttp/pipeline.go:72","msg":"started HTTP pipelining with remote peer","local-member-id":"a98197a5
44df4572","remote-peer-id":"a5075b4c18dc0e9d"}
Mar 20 12:46:12 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:12.020+0800","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"a98197a544d
f4572","remote-peer-id":"a5075b4c18dc0e9d"}
Mar 20 12:46:12 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:12.020+0800","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"a98197a544d
f4572","remote-peer-id":"a5075b4c18dc0e9d"}
Mar 20 12:46:12 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:12.020+0800","caller":"rafthttp/peer.go:137","msg":"started remote peer","remote-peer-id":"a5075b4c18dc0e9d"}
Mar 20 12:46:12 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:12.020+0800","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream M
essage","local-member-id":"a98197a544df4572","remote-peer-id":"a5075b4c18dc0e9d"}
Mar 20 12:46:12 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:12.020+0800","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream M
sgApp v2","local-member-id":"a98197a544df4572","remote-peer-id":"a5075b4c18dc0e9d"}
Mar 20 12:46:12 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:12.020+0800","caller":"rafthttp/transport.go:317","msg":"added remote peer","local-member-id":"a98197a544df4572","remote-pee
r-id":"a5075b4c18dc0e9d","remote-peer-urls":["https://192.168.0.182:2380"]}
Mar 20 12:46:12 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:12.021+0800","caller":"etcdserver/server.go:842","msg":"starting etcd server","local-member-id":"a98197a544df4572","local-se
rver-version":"3.5.3","cluster-id":"5e35a9446cd2dbd9","cluster-version":"3.5"}
Mar 20 12:46:12 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:12.021+0800","caller":"etcdserver/server.go:752","msg":"starting initial election tick advance","election-ticks":10}
Mar 20 12:46:12 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:12.023+0800","caller":"embed/etcd.go:581","msg":"serving peer traffic","address":"127.0.0.1:2400"}
Mar 20 12:46:12 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:12.023+0800","caller":"embed/etcd.go:553","msg":"cmux::serve","address":"127.0.0.1:2400"}
Mar 20 12:46:12 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:12.023+0800","caller":"embed/etcd.go:277","msg":"now serving peer/client/metrics","local-member-id":"a98197a544df4572","init
ial-advertise-peer-urls":["http://127.0.0.1:2400"],"listen-peer-urls":["http://127.0.0.1:2400"],"advertise-client-urls":["http://127.0.0.1:2399"],"listen-client-urls":["http://127.0.0.1:2399"],"listen-metric
s-urls":[]}
Mar 20 12:46:12 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:12.186+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"a98197a544df4572 switched to configuration voters=(
11891573723487800989 12214210400486376818)"}
Mar 20 12:46:12 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:12.186+0800","caller":"membership/cluster.go:472","msg":"removed member","cluster-id":"5e35a9446cd2dbd9","local-member-id":"
a98197a544df4572","removed-remote-peer-id":"1d2fa42ab96be932","removed-remote-peer-urls":["https://192.168.0.192:2380"]}
Mar 20 12:46:12 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:12.186+0800","caller":"rafthttp/peer.go:330","msg":"stopping remote peer","remote-peer-id":"1d2fa42ab96be932"}
Mar 20 12:46:12 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:12.186+0800","caller":"rafthttp/stream.go:294","msg":"stopped TCP streaming connection with remote peer","stream-writer-type
":"unknown stream","remote-peer-id":"1d2fa42ab96be932"}
Mar 20 12:46:12 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:12.186+0800","caller":"rafthttp/stream.go:294","msg":"stopped TCP streaming connection with remote peer","stream-writer-type
":"unknown stream","remote-peer-id":"1d2fa42ab96be932"}
Mar 20 12:46:12 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:12.186+0800","caller":"rafthttp/pipeline.go:85","msg":"stopped HTTP pipelining with remote peer","local-member-id":"a98197a5
44df4572","remote-peer-id":"1d2fa42ab96be932"}
Mar 20 12:46:12 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:12.186+0800","caller":"rafthttp/stream.go:442","msg":"stopped stream reader with remote peer","stream-reader-type":"stream M
sgApp v2","local-member-id":"a98197a544df4572","remote-peer-id":"1d2fa42ab96be932"}
Mar 20 12:46:12 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:12.186+0800","caller":"rafthttp/stream.go:442","msg":"stopped stream reader with remote peer","stream-reader-type":"stream M
essage","local-member-id":"a98197a544df4572","remote-peer-id":"1d2fa42ab96be932"}
Mar 20 12:46:12 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:12.186+0800","caller":"rafthttp/peer.go:335","msg":"stopped remote peer","remote-peer-id":"1d2fa42ab96be932"}
Mar 20 12:46:12 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:12.186+0800","caller":"rafthttp/transport.go:355","msg":"removed remote peer","local-member-id":"a98197a544df4572","removed-
remote-peer-id":"1d2fa42ab96be932"}
Mar 20 12:46:12 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:12.186+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"a98197a544df4572 switched to configuration voters=(
12214210400486376818)"}
Mar 20 12:46:12 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:12.186+0800","caller":"membership/cluster.go:472","msg":"removed member","cluster-id":"5e35a9446cd2dbd9","local-member-id":"
a98197a544df4572","removed-remote-peer-id":"a5075b4c18dc0e9d","removed-remote-peer-urls":["https://192.168.0.182:2380"]}
Mar 20 12:46:12 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:12.186+0800","caller":"rafthttp/peer.go:330","msg":"stopping remote peer","remote-peer-id":"a5075b4c18dc0e9d"}
Mar 20 12:46:12 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:12.186+0800","caller":"rafthttp/stream.go:294","msg":"stopped TCP streaming connection with remote peer","stream-writer-type
":"unknown stream","remote-peer-id":"a5075b4c18dc0e9d"}
Mar 20 12:46:12 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:12.186+0800","caller":"rafthttp/stream.go:294","msg":"stopped TCP streaming connection with remote peer","stream-writer-type
":"unknown stream","remote-peer-id":"a5075b4c18dc0e9d"}
Mar 20 12:46:12 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:12.186+0800","caller":"rafthttp/pipeline.go:85","msg":"stopped HTTP pipelining with remote peer","local-member-id":"a98197a5
44df4572","remote-peer-id":"a5075b4c18dc0e9d"}
Mar 20 12:46:12 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:12.186+0800","caller":"rafthttp/stream.go:442","msg":"stopped stream reader with remote peer","stream-reader-type":"stream M
sgApp v2","local-member-id":"a98197a544df4572","remote-peer-id":"a5075b4c18dc0e9d"}
Mar 20 12:46:12 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:12.186+0800","caller":"rafthttp/stream.go:442","msg":"stopped stream reader with remote peer","stream-reader-type":"stream M
essage","local-member-id":"a98197a544df4572","remote-peer-id":"a5075b4c18dc0e9d"}
Mar 20 12:46:12 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:12.186+0800","caller":"rafthttp/peer.go:335","msg":"stopped remote peer","remote-peer-id":"a5075b4c18dc0e9d"}
Mar 20 12:46:12 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:12.186+0800","caller":"rafthttp/transport.go:355","msg":"removed remote peer","local-member-id":"a98197a544df4572","removed-
remote-peer-id":"a5075b4c18dc0e9d"}
Mar 20 12:46:17 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:17.989+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"a98197a544df4572 is starting a new election at term
 57"}
Mar 20 12:46:17 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:17.989+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"a98197a544df4572 became pre-candidate at term 57"}
Mar 20 12:46:17 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:17.989+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"a98197a544df4572 received MsgPreVoteResp from a9819
7a544df4572 at term 57"}
Mar 20 12:46:17 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:17.989+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"a98197a544df4572 became candidate at term 58"}
Mar 20 12:46:17 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:17.989+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"a98197a544df4572 received MsgVoteResp from a98197a5
44df4572 at term 58"}
Mar 20 12:46:17 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:17.989+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"a98197a544df4572 became leader at term 58"}
Mar 20 12:46:17 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:17.989+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"raft.node: a98197a544df4572 elected leader a98197a5
44df4572 at term 58"}
Mar 20 12:46:18 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:17.989+0800","caller":"etcdserver/server.go:2044","msg":"published local member to cluster through raft","local-member-id":"
a98197a544df4572","local-member-attributes":"{Name:k8s-master02-ca3014b1 ClientURLs:[http://127.0.0.1:2399]}","request-path":"/0/members/a98197a544df4572/attributes","cluster-id":"5e35a9446cd2dbd9","publish-
timeout":"15s"}
Mar 20 12:46:18 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:17.989+0800","caller":"embed/serve.go:98","msg":"ready to serve client requests"}
Mar 20 12:46:18 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:17.990+0800","caller":"embed/serve.go:140","msg":"serving client traffic insecurely; this is strongly discouraged!","address
":"127.0.0.1:2399"}
Mar 20 12:46:18 k8s-master02 rke2[2939538]: time="2023-03-20T12:46:17+08:00" level=info msg="Defragmenting etcd database"
Mar 20 12:46:18 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:17.992+0800","caller":"v3rpc/maintenance.go:89","msg":"starting defragment"}
Mar 20 12:46:18 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:18.002+0800","caller":"backend/backend.go:497","msg":"defragmenting","path":"/var/lib/rancher/rke2/server/db/etcd-tmp/member
/snap/db","current-db-size-bytes":535216128,"current-db-size":"535 MB","current-db-size-in-use-bytes":32706560,"current-db-size-in-use":"33 MB"}
Mar 20 12:46:18 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:18.375+0800","caller":"backend/backend.go:549","msg":"finished defragmenting directory","path":"/var/lib/rancher/rke2/server
/db/etcd-tmp/member/snap/db","current-db-size-bytes-diff":-502640640,"current-db-size-bytes":32575488,"current-db-size":"33 MB","current-db-size-in-use-bytes-diff":-139264,"current-db-size-in-use-bytes":3256
7296,"current-db-size-in-use":"33 MB","took":"382.552939ms"}
Mar 20 12:46:18 k8s-master02 rke2[2939538]: {"level":"info","ts":"2023-03-20T12:46:18.375+0800","caller":"v3rpc/maintenance.go:95","msg":"finished defragment"}
Mar 20 12:46:18 k8s-master02 rke2[2939538]: {"level":"warn","ts":"2023-03-20T12:46:18.375+0800","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2023-03-20T12:46:17.992+0800","time spe
nt":"382.713565ms","remote":"127.0.0.1:39212","response type":"/etcdserverpb.Maintenance/Defragment","request count":-1,"request size":-1,"response count":-1,"response size":-1,"request content":""}
Mar 20 12:46:18 k8s-master02 rke2[2939538]: time="2023-03-20T12:46:18+08:00" level=info msg="etcd temporary data store connection OK"
Mar 20 12:46:18 k8s-master02 rke2[2939538]: time="2023-03-20T12:46:18+08:00" level=info msg="Reconciling bootstrap data between datastore and disk"
Mar 20 12:46:18 k8s-master02 rke2[2939538]: time="2023-03-20T12:46:18+08:00" level=fatal msg="/var/lib/rancher/rke2/server/cred/passwd newer than datastore and could cause a cluster outage. Remove the file(s
) from disk and restart to be recreated from datastore."
Mar 20 12:46:18 k8s-master02 systemd[1]: rke2-server.service: Main process exited, code=exited, status=1/FAILURE
Mar 20 12:46:18 k8s-master02 systemd[1]: rke2-server.service: Failed with result 'exit-code'.
Mar 20 12:46:18 k8s-master02 systemd[1]: Failed to start Rancher Kubernetes Engine v2 (server).
Mar 20 12:46:23 k8s-master02 systemd[1]: rke2-server.service: Scheduled restart job, restart counter is at 100.
Mar 20 12:46:23 k8s-master02 systemd[1]: Stopped Rancher Kubernetes Engine v2 (server).

Answered by foliet

Nov 13, 2023

Sorry, I miss your reply. It seems that I did change these two files when restoring using the snapshot.

View full answer

brandond · 2023-03-20T09:39:05Z

brandond
Mar 20, 2023
Maintainer

delete the file /var/lib/rancher/rke2/server/cred/passwd，rke2-server restart succeeded。
but，Why is this fatal log generated?

Something other than rke2 is modifying that file on disk so that its modification time is newer than the copy in the datastore.

3 replies

foliet Oct 12, 2023

I currently have an rke2 cluster that contains a master node and a worker node. I have also encountered this problem since I restored the data using a snapshot. This error will be reported every time the master node is restarted. I have to delete the passwd file before I can start it successfully. What are the possible reasons?

brandond Oct 12, 2023
Maintainer

Is your token or agent-token set to a different value than you were using when the snapshot was taken?

foliet Nov 13, 2023

Sorry, I miss your reply. It seems that I did change these two files when restoring using the snapshot.

Answer selected by brandond

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

/var/lib/rancher/rke2/server/cred/passwd newer than datastore and could cause a cluster outage. Remove the file(s ) from disk and restart to be recreated from datastore. #4035

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment 3 replies

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

/var/lib/rancher/rke2/server/cred/passwd newer than datastore and could cause a cluster outage. Remove the file(s ) from disk and restart to be recreated from datastore. #4035

kongzi68 Mar 20, 2023

Replies: 1 comment · 3 replies

brandond Mar 20, 2023 Maintainer

foliet Oct 12, 2023

brandond Oct 12, 2023 Maintainer

foliet Nov 13, 2023

kongzi68
Mar 20, 2023

Replies: 1 comment 3 replies

brandond
Mar 20, 2023
Maintainer

brandond Oct 12, 2023
Maintainer