Error 'failed to get CA certs: https://127.0.0.1:6444/cacerts: 401 Unauthorized'

[crimewaffle]

Environmental Info:
K3s Version:
k3s version v1.27.5+k3s1 (8d074ec)
go version go1.20.7

Node(s) CPU architecture, OS, and Version:
Linux k3s-h01-n02 6.1.0-12-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.52-1 (2023-09-07) x86_64 GNU/Linux

Cluster Configuration:
1 Manager, 1 Worker (Agent)

Describe the bug:
k3s-agent.service unable to start up due to (authentication?) issues 401 error (See logs)

Steps To Reproduce:
Install on Debian 12 using curl -sfL https://get.k3s.io | K3S_URL=https://[HIDDEN]:6443 K3S_TOKEN=[HIDDEN] sh -
Start it (systemctl enable --now k3s-agent.service)

• Installed K3s:

Expected behavior:
Should be able to connect and authenticate; Service should start properly

Actual behavior:
Service stays at 'activating (start)'

Additional context / logs:
Output of systemctl status k3s-agent.service:

● k3s-agent.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s-agent.service; enabled; preset: enabled)
     Active: activating (start) since Tue 2023-09-19 09:26:09 CEST; 3h 23min ago
       Docs: https://k3s.io
   Main PID: 473 (k3s-agent)
      Tasks: 10
     Memory: 162.8M
        CPU: 30.741s
     CGroup: /system.slice/k3s-agent.service
             └─473 "/usr/local/bin/k3s agent"

Sep 19 12:49:13 k3s-h01-n02 k3s[473]: time="2023-09-19T12:49:13+02:00" level=error msg="failed to get CA certs: https://127.0.0.1:6444/cacerts: 401 Unauthorized"

Service file (/etc/systemd/system/k3s-agent.service):

[Unit]
Description=Lightweight Kubernetes
Documentation=https://k3s.io
Wants=network-online.target
After=network-online.target

[Install]
WantedBy=multi-user.target

[Service]
Type=notify
EnvironmentFile=-/etc/default/%N
EnvironmentFile=-/etc/sysconfig/%N
EnvironmentFile=-/etc/systemd/system/k3s-agent.service.env
KillMode=process
Delegate=yes
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=1048576
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
TimeoutStartSec=0
Restart=always
RestartSec=5s
ExecStartPre=/bin/sh -xc '! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service'
ExecStartPre=-/sbin/modprobe br_netfilter
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/k3s \
    agent \

/etc/systemd/system/k3s-agent.service.env:

K3S_TOKEN='[HIDDEN]'
K3S_URL='https://[HIDDEN]:6443'

Output of ss -tlpn:

State               Recv-Q              Send-Q                             Local Address:Port                              Peer Address:Port              Process
LISTEN              0                   4096                                   127.0.0.1:6444                                   0.0.0.0:*                  users:(("k3s-agent",pid=473,fd=7))

Output of wget https://127.0.0.1:6444/cacerts --no-check-certificate

--2023-09-19 13:00:30--  https://127.0.0.1:6444/cacerts
Connecting to 127.0.0.1:6444... connected.
WARNING: The certificate of ‘127.0.0.1’ is not trusted.
WARNING: The certificate of ‘127.0.0.1’ doesn't have a known issuer.
HTTP request sent, awaiting response... 401 Unauthorized

Username/Password Authentication Failed.

Please let me know if you need anything else
Thanks in advance

[brandond]

Sounds like you have a typo in the token. Check that the value of K3S_TOKEN on the agent matches the value from /var/lib/rancher/k3s/server/token or /var/lib/rancher/k3s/server/agent-token on the server.