v1-rke2/server-bootstrap: 403 Forbidden

[bflick21]

Environmental Info:
RKE2 Version: v1.27.11+rke2r1

Node(s) CPU architecture, OS, and Version:

Linux k8smaster 4.15.0-142-generic #146~16.04.1-Ubuntu SMP Tue Apr 13 09:27:15 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Cluster Configuration:
I had an ubuntu 16.04 master node, a proxmox VM with nested virtualization enabled by using the "Host" cpu type. This seemed to work adding 2 windows agents. I needed to install longhorn block storage provider though, so I need 3 control plane nodes. So, I went through uninstall process and have started back up the k8smaster node again. When I try to add a second proxmox ubuntu 16.04 VM to the cluster the service won't start

Describe the bug:
fatal msg="starting kubernetes: preparing server: https://X.X.X.X:9345/v1-rke2/bootstrap: 403 Forbidden"

Steps To Reproduce:

• Installed RKE2: Installed RKE2 v1.27.11+rke2r1 with the install script method of this from https://docs.rke2.io/install/ha on a proxmox VM using the Host cpu option, and 8G ram (non-balooning).

Expected behavior:

I would expect to be able to add a second node
Actual behavior:

Adding a second ubuntu 16.04 rke2-server fails with /v1-rke2/server-bootstrap: 403 Forbidden
Additional context / logs:

Edited for clarity

[brandond]

Are you sure you're passing the correct token to the second node when adding it to the cluster?

[bflick21]

I'm fairly certain. I had to recreate the token and copy it into /etc/rancher/rke2/config.yaml

server: https://X.X.X.X:9345
token: <MOST_RECENTLY_CREATED_TOKEN>
cni:
- calico
tls-san:
- cluster.mydomain.ltd
.
.
.

[bflick21]

I know for certain the right token is in place, at this point and it is still giving me this error. But, I had a followon question, when creating the HA server, how come it says that all the server control plane nodes should be added before the agents?

[brandond]

Theres not really enough information provided here for me to really tell what's going on. Make sure that the uninstall removed all traces of the original install from the nodes, and confirm that you have the correct token from the new cluster when joining nodes. There's really not much else that will cause this.

how come it says that all the server control plane nodes should be added before the agents

It is a best practice to have servers with the etcd and control-plane nodes before adding agents. The agents should retry until everything is up, but you will probably get lots of warnings and errors on the nodes until things settle out. Is this causing problems in your environment?

[bflick21]

No problems, good to know that the agents should just keep retrying until the server is available though.

[bflick21]

So, now I have upgraded to 18.04 for both the first server and the second. I still can't add a second.

Log produced on the first node with journalctl --boot --lines=all -u rke2-server
Image is from the second node, and the token is correct. I've tried with the server as an IP and a hostname.

log.txt

[brandond]

Do you have a proxy or something else that is blocking that connection? What do you get if you do curl -vks https://192.168.103.152:9345/ping ?

[bflick21]

• Trying 192.168.103.152...
• TCP_NODELAY set
• Connected to 192.168.103.152 (192.168.103.152) port 9345 (#0)
• ALPN, offering h2
• ALPN, offering http/1.1
• successfully set certificate verify locations:
• CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
• TLSv1.3 (OUT), TLS handshake, Client hello (1):
• TLSv1.3 (IN), TLS handshake, Server hello (2):
• TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
• TLSv1.3 (IN), TLS handshake, Unknown (8):
• TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
• TLSv1.3 (IN), TLS handshake, Request CERT (13):
• TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
• TLSv1.3 (IN), TLS handshake, Certificate (11):
• TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
• TLSv1.3 (IN), TLS handshake, CERT verify (15):
• TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
• TLSv1.3 (IN), TLS handshake, Finished (20):
• TLSv1.3 (OUT), TLS change cipher, Client hello (1):
• TLSv1.3 (OUT), TLS Unknown, Certificate Status (22):
• TLSv1.3 (OUT), TLS handshake, Certificate (11):
• TLSv1.3 (OUT), TLS Unknown, Certificate Status (22):
• TLSv1.3 (OUT), TLS handshake, Finished (20):
• SSL connection using TLSv1.3 / TLS_CHACHA20_POLY1305_SHA256
• ALPN, server accepted to use h2
• Server certificate:
• subject: O=rke2; CN=rke2
• start date: Mar 8 18:27:41 2024 GMT
• expire date: Mar 8 18:27:41 2025 GMT
• issuer: CN=rke2-server-ca@1709922461
• SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
• Using HTTP2, server supports multi-use
• Connection state changed (HTTP/2 confirmed)
• Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
• TLSv1.3 (OUT), TLS Unknown, Unknown (23):
• TLSv1.3 (OUT), TLS Unknown, Unknown (23):
• TLSv1.3 (OUT), TLS Unknown, Unknown (23):
• Using Stream ID: 1 (easy handle 0x562096ce6540)
• TLSv1.3 (OUT), TLS Unknown, Unknown (23):

GET /ping HTTP/2
Host: 192.168.103.152:9345
User-Agent: curl/7.58.0
Accept: /

• TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
• TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
• TLSv1.3 (IN), TLS Unknown, Unknown (23):
• Connection state changed (MAX_CONCURRENT_STREAMS updated)!
• TLSv1.3 (OUT), TLS Unknown, Unknown (23):
• TLSv1.3 (IN), TLS Unknown, Unknown (23):
• TLSv1.3 (IN), TLS Unknown, Unknown (23):
  < HTTP/2 200
  < content-type: text/plain
  < content-length: 4
  < date: Fri, 08 Mar 2024 20:27:33 GMT
  <
• TLSv1.3 (IN), TLS Unknown, Unknown (23):
• Connection #0 to host 192.168.103.152 left intact
  pong

[bflick21]

no proxy, as far as I'm aware of

[github-actions]

This repository uses a bot to automatically label issues which have not had any activity (commit/comment/label) for 45 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the bot can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the bot will automatically close the issue in 14 days. Thank you for your contributions.