Unable to connect to proxy when setting up agent nodes #9861

pgregorio-signalytic · 2024-04-03T16:31:37Z

pgregorio-signalytic
Apr 3, 2024

Environmental Info:
K3s Version:

root@master:/home/vagrant# k3s -v
k3s version v1.28.8+k3s1 (653dd61a)
go version go1.21.8

Node(s) CPU architecture, OS, and Version:

root@master:/home/vagrant# uname -a
Linux master 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:

1 server, 2 agents:

Vagrant.configure(2) do |config|
  config.vm.box = "generic/ubuntu2204"

  config.vm.provider "virtualbox" do |vb|
    vb.memory = "1024"
  end

  config.vm.define "master" do |master|
    master.vm.hostname = "master"
    master.vm.network "private_network", ip: "192.168.56.2"
  end

  config.vm.define "node1" do |node1|
    node1.vm.hostname = "node1"
    node1.vm.network "private_network", ip: "192.168.56.11"
  end

  config.vm.define "node2" do |node2|
    node2.vm.hostname = "node2"
    node2.vm.network "private_network", ip: "192.168.56.22"
  end
end

Describe the bug:

When setting up the agent nodes, I get the following errors:

Failed to connect to proxy. Empty dialer response
Remotedialer proxy error" error="dial tcp 10.0.2.15:6443: connect: connection refused

Steps To Reproduce:

Installed K3s with the following instructions:

K3S

Use Vagrant to easily spin multiple VMs (requires VirtualBox). First we need to install both packages:

# Install virtualbox and vagrant
sudo apt install virtualbox vagrant -y

We should now be able to create the VMs we need to test k3s:

# Run the VMs
vagrant up

This should spin up 3 Ubuntu 22.04 VMs with their respective hostnames and IP addresses as defined on the Vagrantfile.

Update firewall

For Ubuntu VMs, a few extra steps are needed:

# It is recommended to turn off ufw (uncomplicated firewall)
ufw disable

# If you wish to keep ufw enabled, by default, the following rules are required
ufw allow 6443/tcp                   #apiserver
ufw allow from 10.42.0.0/16 to any   #pods
ufw allow from 10.43.0.0/16 to any   #services

MASTER

On the master VM, install k3s:

# Install k3s as the master
curl -sfL https://get.k3s.io | sh -

# Check that the service is running
systemctl status k3s

# Check the nodes
k3s kubectl get nodes

Get the token we need to use when installing k3s on the other nodes:

cat /var/lib/rancher/k3s/server/node-token

NODE1

On the node1 VM, install k3s as an agent:

# Install k3s as an agent node
curl -sfL https://get.k3s.io | K3S_URL=https://<MASTER_IP>:6443 K3S_TOKEN=<TOKEN> sh -

NODE2

On the node2 VM, install k3s as an agent:

# Install k3s as an agent node
curl -sfL https://get.k3s.io | K3S_URL=https://<MASTER_IP>:6443 K3S_TOKEN=<TOKEN> sh -

Expected behavior:

The agent nodes should be able to connect to the master node.

Actual behavior:

The installation on both agent nodes get stuck at and eventually times out:

[INFO]  systemd: Starting k3s-agent

Additional context / logs:

● k3s-agent.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s-agent.service; enabled; vendor preset: enabled)
     Active: activating (start) since Wed 2024-04-03 16:18:27 UTC; 1min 49s ago
       Docs: https://k3s.io
    Process: 3911 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service 2>/dev/null (code=exited, status=0/SUCCESS)
    Process: 3913 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
    Process: 3914 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
   Main PID: 3915 (k3s-agent)
      Tasks: 16
     Memory: 251.7M
        CPU: 3.144s
     CGroup: /system.slice/k3s-agent.service
             ├─3915 "/usr/local/bin/k3s agent"
             └─3938 "containerd " "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""

Apr 03 16:20:08 node1 k3s[3915]: time="2024-04-03T16:20:08Z" level=error msg="Remotedialer proxy error" error="dial tcp 10.0.2.15:6443: connect: connection refused"
Apr 03 16:20:10 node1 k3s[3915]: W0403 16:20:10.470622    3915 reflector.go:535] k8s.io/[email protected]/tools/cache/reflector.go:229: failed to list *v1.Endpoints: Get "https://127.0.0.1:6444/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&limit=500&resourceVersion=0": read tcp 127.0.0.1:39464->127.0.0.1:6444: read: connection reset by peer - error from a previous attempt: read tcp 127.0.0.1:42520->127.0.0.1:6444: read: connection reset by peer
Apr 03 16:20:10 node1 k3s[3915]: I0403 16:20:10.470982    3915 trace.go:236] Trace[1903520160]: "Reflector ListAndWatch" name:k8s.io/[email protected]/tools/cache/reflector.go:229 (03-Apr-2024 16:20:00.430) (total time: 10040ms):
Apr 03 16:20:10 node1 k3s[3915]: Trace[1903520160]: ---"Objects listed" error:Get "https://127.0.0.1:6444/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&limit=500&resourceVersion=0": read tcp 127.0.0.1:39464->127.0.0.1:6444: read: connection reset by peer - error from a previous attempt: read tcp 127.0.0.1:42520->127.0.0.1:6444: read: connection reset by peer 10039ms (16:20:10.470)
Apr 03 16:20:10 node1 k3s[3915]: Trace[1903520160]: [10.040077312s] [10.040077312s] END
Apr 03 16:20:10 node1 k3s[3915]: E0403 16:20:10.471095    3915 reflector.go:147] k8s.io/[email protected]/tools/cache/reflector.go:229: Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: Get "https://127.0.0.1:6444/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&limit=500&resourceVersion=0": read tcp 127.0.0.1:39464->127.0.0.1:6444: read: connection reset by peer - error from a previous attempt: read tcp 127.0.0.1:42520->127.0.0.1:6444: read: connection reset by peer
Apr 03 16:20:12 node1 k3s[3915]: time="2024-04-03T16:20:12Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": EOF"
Apr 03 16:20:13 node1 k3s[3915]: time="2024-04-03T16:20:13Z" level=info msg="Connecting to proxy" url="wss://10.0.2.15:6443/v1-k3s/connect"
Apr 03 16:20:13 node1 k3s[3915]: time="2024-04-03T16:20:13Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 10.0.2.15:6443: connect: connection refused"
Apr 03 16:20:13 node1 k3s[3915]: time="2024-04-03T16:20:13Z" level=error msg="Remotedialer proxy error" error="dial tcp 10.0.2.15:6443: connect: connection refused"

Apr 03 16:18:27 node1 systemd[1]: Starting Lightweight Kubernetes...
Apr 03 16:18:27 node1 sh[3911]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Apr 03 16:18:27 node1 k3s[3915]: time="2024-04-03T16:18:27Z" level=info msg="Acquiring lock file /var/lib/rancher/k3s/data/.lock"
Apr 03 16:18:27 node1 k3s[3915]: time="2024-04-03T16:18:27Z" level=info msg="Preparing data dir /var/lib/rancher/k3s/data/ef92c15b8f1c59f266ea4a9589be87b1b799d1682673f29721e8a1f7b0a4a97b"
Apr 03 16:18:30 node1 k3s[3915]: time="2024-04-03T16:18:30Z" level=info msg="Starting k3s agent v1.28.8+k3s1 (653dd61a)"
Apr 03 16:18:30 node1 k3s[3915]: time="2024-04-03T16:18:30Z" level=info msg="Adding server to load balancer k3s-agent-load-balancer: 192.168.56.2:6443"
Apr 03 16:18:30 node1 k3s[3915]: time="2024-04-03T16:18:30Z" level=info msg="Running load balancer k3s-agent-load-balancer 127.0.0.1:6444 -> [192.168.56.2:6443] [default: 192.168.56.2:6443]"
Apr 03 16:18:31 node1 k3s[3915]: time="2024-04-03T16:18:31Z" level=info msg="Module overlay was already loaded"
Apr 03 16:18:31 node1 k3s[3915]: time="2024-04-03T16:18:31Z" level=info msg="Module br_netfilter was already loaded"
Apr 03 16:18:31 node1 k3s[3915]: time="2024-04-03T16:18:31Z" level=info msg="Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400"
Apr 03 16:18:31 node1 k3s[3915]: time="2024-04-03T16:18:31Z" level=info msg="Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600"
Apr 03 16:18:31 node1 k3s[3915]: time="2024-04-03T16:18:31Z" level=info msg="Set sysctl 'net/ipv4/conf/all/forwarding' to 1"
Apr 03 16:18:31 node1 k3s[3915]: time="2024-04-03T16:18:31Z" level=info msg="Set sysctl 'net/netfilter/nf_conntrack_max' to 131072"
Apr 03 16:18:31 node1 k3s[3915]: time="2024-04-03T16:18:31Z" level=info msg="Logging containerd to /var/lib/rancher/k3s/agent/containerd/containerd.log"
Apr 03 16:18:31 node1 k3s[3915]: time="2024-04-03T16:18:31Z" level=info msg="Running containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd"
Apr 03 16:18:32 node1 k3s[3915]: time="2024-04-03T16:18:32Z" level=info msg="containerd is now running"
Apr 03 16:18:32 node1 k3s[3915]: time="2024-04-03T16:18:32Z" level=info msg="Getting list of apiserver endpoints from server"
Apr 03 16:18:32 node1 k3s[3915]: time="2024-04-03T16:18:32Z" level=info msg="Updated load balancer k3s-agent-load-balancer default server address -> 10.0.2.15:6443"
Apr 03 16:18:32 node1 k3s[3915]: time="2024-04-03T16:18:32Z" level=info msg="Adding server to load balancer k3s-agent-load-balancer: 10.0.2.15:6443"
Apr 03 16:18:32 node1 k3s[3915]: time="2024-04-03T16:18:32Z" level=info msg="Removing server from load balancer k3s-agent-load-balancer: 192.168.56.2:6443"
Apr 03 16:18:32 node1 k3s[3915]: time="2024-04-03T16:18:32Z" level=info msg="Updated load balancer k3s-agent-load-balancer server addresses -> [10.0.2.15:6443] [default: 10.0.2.15:6443]"
Apr 03 16:18:32 node1 k3s[3915]: time="2024-04-03T16:18:32Z" level=info msg="Running kubelet --address=0.0.0.0 --allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --feature-gates=CloudDualStackNodeIPs=true --healthz-bind-address=127.0.0.1 --hostname-override=node1 --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --node-ip=10.0.2.15 --node-labels= --pod-infra-container-image=rancher/mirrored-pause:3.6 --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key"
Apr 03 16:18:32 node1 k3s[3915]: time="2024-04-03T16:18:32Z" level=info msg="Connecting to proxy" url="wss://10.0.2.15:6443/v1-k3s/connect"
Apr 03 16:18:32 node1 k3s[3915]: time="2024-04-03T16:18:32Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 10.0.2.15:6443: connect: connection refused"
Apr 03 16:18:32 node1 k3s[3915]: time="2024-04-03T16:18:32Z" level=error msg="Remotedialer proxy error" error="dial tcp 10.0.2.15:6443: connect: connection refused"
Apr 03 16:18:32 node1 k3s[3915]: time="2024-04-03T16:18:32Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": read tcp 127.0.0.1:38152->127.0.0.1:6444: read: connection reset by peer"
Apr 03 16:18:37 node1 k3s[3915]: time="2024-04-03T16:18:37Z" level=info msg="Connecting to proxy" url="wss://10.0.2.15:6443/v1-k3s/connect"
Apr 03 16:18:37 node1 k3s[3915]: time="2024-04-03T16:18:37Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": read tcp 127.0.0.1:38314->127.0.0.1:6444: read: connection reset by peer"
Apr 03 16:18:37 node1 k3s[3915]: time="2024-04-03T16:18:37Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 10.0.2.15:6443: connect: connection refused"
Apr 03 16:18:37 node1 k3s[3915]: time="2024-04-03T16:18:37Z" level=error msg="Remotedialer proxy error" error="dial tcp 10.0.2.15:6443: connect: connection refused"
Apr 03 16:18:42 node1 k3s[3915]: time="2024-04-03T16:18:42Z" level=info msg="Connecting to proxy" url="wss://10.0.2.15:6443/v1-k3s/connect"
Apr 03 16:18:42 node1 k3s[3915]: time="2024-04-03T16:18:42Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 10.0.2.15:6443: connect: connection refused"
Apr 03 16:18:42 node1 k3s[3915]: time="2024-04-03T16:18:42Z" level=error msg="Remotedialer proxy error" error="dial tcp 10.0.2.15:6443: connect: connection refused"
Apr 03 16:18:42 node1 k3s[3915]: time="2024-04-03T16:18:42Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": read tcp 127.0.0.1:57764->127.0.0.1:6444: read: connection reset by peer"
Apr 03 16:18:43 node1 k3s[3915]: W0403 16:18:43.148052    3915 reflector.go:535] k8s.io/[email protected]/tools/cache/reflector.go:229: failed to list *v1.Endpoints: Get "https://127.0.0.1:6444/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&limit=500&resourceVersion=0": read tcp 127.0.0.1:57806->127.0.0.1:6444: read: connection reset by peer - error from a previous attempt: read tcp 127.0.0.1:57744->127.0.0.1:6444: read: connection reset by peer
Apr 03 16:18:43 node1 k3s[3915]: I0403 16:18:43.148308    3915 trace.go:236] Trace[1871427865]: "Reflector ListAndWatch" name:k8s.io/[email protected]/tools/cache/reflector.go:229 (03-Apr-2024 16:18:32.941) (total time: 10207ms):
Apr 03 16:18:43 node1 k3s[3915]: Trace[1871427865]: ---"Objects listed" error:Get "https://127.0.0.1:6444/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&limit=500&resourceVersion=0": read tcp 127.0.0.1:57806->127.0.0.1:6444: read: connection reset by peer - error from a previous attempt: read tcp 127.0.0.1:57744->127.0.0.1:6444: read: connection reset by peer 10207ms (16:18:43.148)
Apr 03 16:18:43 node1 k3s[3915]: Trace[1871427865]: [10.207250423s] [10.207250423s] END
Apr 03 16:18:43 node1 k3s[3915]: E0403 16:18:43.148448    3915 reflector.go:147] k8s.io/[email protected]/tools/cache/reflector.go:229: Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: Get "https://127.0.0.1:6444/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&limit=500&resourceVersion=0": read tcp 127.0.0.1:57806->127.0.0.1:6444: read: connection reset by peer - error from a previous attempt: read tcp 127.0.0.1:57744->127.0.0.1:6444: read: connection reset by peer
Apr 03 16:18:47 node1 k3s[3915]: time="2024-04-03T16:18:47Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": read tcp 127.0.0.1:58004->127.0.0.1:6444: read: connection reset by peer"
Apr 03 16:18:47 node1 k3s[3915]: time="2024-04-03T16:18:47Z" level=info msg="Connecting to proxy" url="wss://10.0.2.15:6443/v1-k3s/connect"
Apr 03 16:18:47 node1 k3s[3915]: time="2024-04-03T16:18:47Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 10.0.2.15:6443: connect: connection refused"
Apr 03 16:18:47 node1 k3s[3915]: time="2024-04-03T16:18:47Z" level=error msg="Remotedialer proxy error" error="dial tcp 10.0.2.15:6443: connect: connection refused"
Apr 03 16:18:52 node1 k3s[3915]: time="2024-04-03T16:18:52Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": read tcp 127.0.0.1:60786->127.0.0.1:6444: read: connection reset by peer"
Apr 03 16:18:52 node1 k3s[3915]: time="2024-04-03T16:18:52Z" level=info msg="Connecting to proxy" url="wss://10.0.2.15:6443/v1-k3s/connect"
Apr 03 16:18:52 node1 k3s[3915]: time="2024-04-03T16:18:52Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 10.0.2.15:6443: connect: connection refused"
Apr 03 16:18:52 node1 k3s[3915]: time="2024-04-03T16:18:52Z" level=error msg="Remotedialer proxy error" error="dial tcp 10.0.2.15:6443: connect: connection refused"
Apr 03 16:18:54 node1 k3s[3915]: W0403 16:18:54.105835    3915 reflector.go:535] k8s.io/[email protected]/tools/cache/reflector.go:229: failed to list *v1.Endpoints: Get "https://127.0.0.1:6444/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&limit=500&resourceVersion=0": EOF - error from a previous attempt: read tcp 127.0.0.1:60788->127.0.0.1:6444: read: connection reset by peer
Apr 03 16:18:54 node1 k3s[3915]: I0403 16:18:54.105899    3915 trace.go:236] Trace[232591820]: "Reflector ListAndWatch" name:k8s.io/[email protected]/tools/cache/reflector.go:229 (03-Apr-2024 16:18:43.982) (total time: 10123ms):
Apr 03 16:18:54 node1 k3s[3915]: Trace[232591820]: ---"Objects listed" error:Get "https://127.0.0.1:6444/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&limit=500&resourceVersion=0": EOF - error from a previous attempt: read tcp 127.0.0.1:60788->127.0.0.1:6444: read: connection reset by peer 10122ms (16:18:54.105)
Apr 03 16:18:54 node1 k3s[3915]: Trace[232591820]: [10.123021188s] [10.123021188s] END
Apr 03 16:18:54 node1 k3s[3915]: E0403 16:18:54.105911    3915 reflector.go:147] k8s.io/[email protected]/tools/cache/reflector.go:229: Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: Get "https://127.0.0.1:6444/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&limit=500&resourceVersion=0": EOF - error from a previous attempt: read tcp 127.0.0.1:60788->127.0.0.1:6444: read: connection reset by peer
Apr 03 16:18:57 node1 k3s[3915]: time="2024-04-03T16:18:57Z" level=info msg="Connecting to proxy" url="wss://10.0.2.15:6443/v1-k3s/connect"
Apr 03 16:18:57 node1 k3s[3915]: time="2024-04-03T16:18:57Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 10.0.2.15:6443: connect: connection refused"
Apr 03 16:18:57 node1 k3s[3915]: time="2024-04-03T16:18:57Z" level=error msg="Remotedialer proxy error" error="dial tcp 10.0.2.15:6443: connect: connection refused"
Apr 03 16:18:57 node1 k3s[3915]: time="2024-04-03T16:18:57Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": read tcp 127.0.0.1:60948->127.0.0.1:6444: read: connection reset by peer"
Apr 03 16:19:02 node1 k3s[3915]: time="2024-04-03T16:19:02Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": read tcp 127.0.0.1:38052->127.0.0.1:6444: read: connection reset by peer"
Apr 03 16:19:02 node1 k3s[3915]: time="2024-04-03T16:19:02Z" level=info msg="Connecting to proxy" url="wss://10.0.2.15:6443/v1-k3s/connect"
Apr 03 16:19:02 node1 k3s[3915]: time="2024-04-03T16:19:02Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 10.0.2.15:6443: connect: connection refused"
Apr 03 16:19:02 node1 k3s[3915]: time="2024-04-03T16:19:02Z" level=error msg="Remotedialer proxy error" error="dial tcp 10.0.2.15:6443: connect: connection refused"
Apr 03 16:19:06 node1 k3s[3915]: W0403 16:19:06.982008    3915 reflector.go:535] k8s.io/[email protected]/tools/cache/reflector.go:229: failed to list *v1.Endpoints: Get "https://127.0.0.1:6444/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&limit=500&resourceVersion=0": EOF - error from a previous attempt: read tcp 127.0.0.1:38196->127.0.0.1:6444: read: connection reset by peer
Apr 03 16:19:06 node1 k3s[3915]: I0403 16:19:06.982401    3915 trace.go:236] Trace[2059427684]: "Reflector ListAndWatch" name:k8s.io/[email protected]/tools/cache/reflector.go:229 (03-Apr-2024 16:18:56.861) (total time: 10120ms):
Apr 03 16:19:06 node1 k3s[3915]: Trace[2059427684]: ---"Objects listed" error:Get "https://127.0.0.1:6444/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&limit=500&resourceVersion=0": EOF - error from a previous attempt: read tcp 127.0.0.1:38196->127.0.0.1:6444: read: connection reset by peer 10120ms (16:19:06.981)
Apr 03 16:19:06 node1 k3s[3915]: Trace[2059427684]: [10.120741384s] [10.120741384s] END
Apr 03 16:19:06 node1 k3s[3915]: E0403 16:19:06.982546    3915 reflector.go:147] k8s.io/[email protected]/tools/cache/reflector.go:229: Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: Get "https://127.0.0.1:6444/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&limit=500&resourceVersion=0": EOF - error from a previous attempt: read tcp 127.0.0.1:38196->127.0.0.1:6444: read: connection reset by peer
Apr 03 16:19:07 node1 k3s[3915]: time="2024-04-03T16:19:07Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": read tcp 127.0.0.1:38292->127.0.0.1:6444: read: connection reset by peer"
Apr 03 16:19:07 node1 k3s[3915]: time="2024-04-03T16:19:07Z" level=info msg="Connecting to proxy" url="wss://10.0.2.15:6443/v1-k3s/connect"
Apr 03 16:19:07 node1 k3s[3915]: time="2024-04-03T16:19:07Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 10.0.2.15:6443: connect: connection refused"
Apr 03 16:19:07 node1 k3s[3915]: time="2024-04-03T16:19:07Z" level=error msg="Remotedialer proxy error" error="dial tcp 10.0.2.15:6443: connect: connection refused"
Apr 03 16:19:12 node1 k3s[3915]: time="2024-04-03T16:19:12Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": EOF"
Apr 03 16:19:13 node1 k3s[3915]: time="2024-04-03T16:19:13Z" level=info msg="Connecting to proxy" url="wss://10.0.2.15:6443/v1-k3s/connect"
Apr 03 16:19:13 node1 k3s[3915]: time="2024-04-03T16:19:13Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 10.0.2.15:6443: connect: connection refused"
Apr 03 16:19:13 node1 k3s[3915]: time="2024-04-03T16:19:13Z" level=error msg="Remotedialer proxy error" error="dial tcp 10.0.2.15:6443: connect: connection refused"
Apr 03 16:19:17 node1 k3s[3915]: time="2024-04-03T16:19:17Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": read tcp 127.0.0.1:41024->127.0.0.1:6444: read: connection reset by peer"
Apr 03 16:19:18 node1 k3s[3915]: time="2024-04-03T16:19:18Z" level=info msg="Connecting to proxy" url="wss://10.0.2.15:6443/v1-k3s/connect"
Apr 03 16:19:18 node1 k3s[3915]: time="2024-04-03T16:19:18Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 10.0.2.15:6443: connect: connection refused"
Apr 03 16:19:18 node1 k3s[3915]: time="2024-04-03T16:19:18Z" level=error msg="Remotedialer proxy error" error="dial tcp 10.0.2.15:6443: connect: connection refused"
Apr 03 16:19:20 node1 k3s[3915]: W0403 16:19:20.710333    3915 reflector.go:535] k8s.io/[email protected]/tools/cache/reflector.go:229: failed to list *v1.Endpoints: Get "https://127.0.0.1:6444/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&limit=500&resourceVersion=0": read tcp 127.0.0.1:35684->127.0.0.1:6444: read: connection reset by peer - error from a previous attempt: read tcp 127.0.0.1:41088->127.0.0.1:6444: read: connection reset by peer
Apr 03 16:19:20 node1 k3s[3915]: I0403 16:19:20.710377    3915 trace.go:236] Trace[1014071780]: "Reflector ListAndWatch" name:k8s.io/[email protected]/tools/cache/reflector.go:229 (03-Apr-2024 16:19:10.543) (total time: 10166ms):
Apr 03 16:19:20 node1 k3s[3915]: Trace[1014071780]: ---"Objects listed" error:Get "https://127.0.0.1:6444/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&limit=500&resourceVersion=0": read tcp 127.0.0.1:35684->127.0.0.1:6444: read: connection reset by peer - error from a previous attempt: read tcp 127.0.0.1:41088->127.0.0.1:6444: read: connection reset by peer 10166ms (16:19:20.710)
Apr 03 16:19:20 node1 k3s[3915]: Trace[1014071780]: [10.166928099s] [10.166928099s] END
Apr 03 16:19:20 node1 k3s[3915]: E0403 16:19:20.710386    3915 reflector.go:147] k8s.io/[email protected]/tools/cache/reflector.go:229: Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: Get "https://127.0.0.1:6444/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&limit=500&resourceVersion=0": read tcp 127.0.0.1:35684->127.0.0.1:6444: read: connection reset by peer - error from a previous attempt: read tcp 127.0.0.1:41088->127.0.0.1:6444: read: connection reset by peer
Apr 03 16:19:22 node1 k3s[3915]: time="2024-04-03T16:19:22Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": read tcp 127.0.0.1:35732->127.0.0.1:6444: read: connection reset by peer"
Apr 03 16:19:23 node1 k3s[3915]: time="2024-04-03T16:19:23Z" level=info msg="Connecting to proxy" url="wss://10.0.2.15:6443/v1-k3s/connect"
Apr 03 16:19:23 node1 k3s[3915]: time="2024-04-03T16:19:23Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 10.0.2.15:6443: connect: connection refused"
Apr 03 16:19:23 node1 k3s[3915]: time="2024-04-03T16:19:23Z" level=error msg="Remotedialer proxy error" error="dial tcp 10.0.2.15:6443: connect: connection refused"
Apr 03 16:19:27 node1 k3s[3915]: time="2024-04-03T16:19:27Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": read tcp 127.0.0.1:35860->127.0.0.1:6444: read: connection reset by peer"
Apr 03 16:19:28 node1 k3s[3915]: time="2024-04-03T16:19:28Z" level=info msg="Connecting to proxy" url="wss://10.0.2.15:6443/v1-k3s/connect"
Apr 03 16:19:28 node1 k3s[3915]: time="2024-04-03T16:19:28Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 10.0.2.15:6443: connect: connection refused"
Apr 03 16:19:28 node1 k3s[3915]: time="2024-04-03T16:19:28Z" level=error msg="Remotedialer proxy error" error="dial tcp 10.0.2.15:6443: connect: connection refused"
Apr 03 16:19:32 node1 k3s[3915]: time="2024-04-03T16:19:32Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": EOF"
Apr 03 16:19:33 node1 k3s[3915]: time="2024-04-03T16:19:33Z" level=info msg="Connecting to proxy" url="wss://10.0.2.15:6443/v1-k3s/connect"
Apr 03 16:19:33 node1 k3s[3915]: time="2024-04-03T16:19:33Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 10.0.2.15:6443: connect: connection refused"
Apr 03 16:19:33 node1 k3s[3915]: time="2024-04-03T16:19:33Z" level=error msg="Remotedialer proxy error" error="dial tcp 10.0.2.15:6443: connect: connection refused"
Apr 03 16:19:37 node1 k3s[3915]: time="2024-04-03T16:19:37Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": EOF"
Apr 03 16:19:38 node1 k3s[3915]: time="2024-04-03T16:19:38Z" level=info msg="Connecting to proxy" url="wss://10.0.2.15:6443/v1-k3s/connect"
Apr 03 16:19:38 node1 k3s[3915]: time="2024-04-03T16:19:38Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 10.0.2.15:6443: connect: connection refused"
Apr 03 16:19:38 node1 k3s[3915]: time="2024-04-03T16:19:38Z" level=error msg="Remotedialer proxy error" error="dial tcp 10.0.2.15:6443: connect: connection refused"
Apr 03 16:19:41 node1 k3s[3915]: W0403 16:19:41.276237    3915 reflector.go:535] k8s.io/[email protected]/tools/cache/reflector.go:229: failed to list *v1.Endpoints: Get "https://127.0.0.1:6444/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&limit=500&resourceVersion=0": read tcp 127.0.0.1:51740->127.0.0.1:6444: read: connection reset by peer - error from a previous attempt: read tcp 127.0.0.1:51712->127.0.0.1:6444: read: connection reset by peer
Apr 03 16:19:41 node1 k3s[3915]: I0403 16:19:41.276282    3915 trace.go:236] Trace[788827548]: "Reflector ListAndWatch" name:k8s.io/[email protected]/tools/cache/reflector.go:229 (03-Apr-2024 16:19:31.151) (total time: 10125ms):
Apr 03 16:19:41 node1 k3s[3915]: Trace[788827548]: ---"Objects listed" error:Get "https://127.0.0.1:6444/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&limit=500&resourceVersion=0": read tcp 127.0.0.1:51740->127.0.0.1:6444: read: connection reset by peer - error from a previous attempt: read tcp 127.0.0.1:51712->127.0.0.1:6444: read: connection reset by peer 10125ms (16:19:41.276)
Apr 03 16:19:41 node1 k3s[3915]: Trace[788827548]: [10.125206812s] [10.125206812s] END
Apr 03 16:19:41 node1 k3s[3915]: E0403 16:19:41.276292    3915 reflector.go:147] k8s.io/[email protected]/tools/cache/reflector.go:229: Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: Get "https://127.0.0.1:6444/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&limit=500&resourceVersion=0": read tcp 127.0.0.1:51740->127.0.0.1:6444: read: connection reset by peer - error from a previous attempt: read tcp 127.0.0.1:51712->127.0.0.1:6444: read: connection reset by peer
Apr 03 16:19:42 node1 k3s[3915]: time="2024-04-03T16:19:42Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": read tcp 127.0.0.1:51798->127.0.0.1:6444: read: connection reset by peer"
Apr 03 16:19:43 node1 k3s[3915]: time="2024-04-03T16:19:43Z" level=info msg="Connecting to proxy" url="wss://10.0.2.15:6443/v1-k3s/connect"
Apr 03 16:19:43 node1 k3s[3915]: time="2024-04-03T16:19:43Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 10.0.2.15:6443: connect: connection refused"
Apr 03 16:19:43 node1 k3s[3915]: time="2024-04-03T16:19:43Z" level=error msg="Remotedialer proxy error" error="dial tcp 10.0.2.15:6443: connect: connection refused"
Apr 03 16:19:47 node1 k3s[3915]: time="2024-04-03T16:19:47Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": EOF"
Apr 03 16:19:48 node1 k3s[3915]: time="2024-04-03T16:19:48Z" level=info msg="Connecting to proxy" url="wss://10.0.2.15:6443/v1-k3s/connect"
Apr 03 16:19:48 node1 k3s[3915]: time="2024-04-03T16:19:48Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 10.0.2.15:6443: connect: connection refused"
Apr 03 16:19:48 node1 k3s[3915]: time="2024-04-03T16:19:48Z" level=error msg="Remotedialer proxy error" error="dial tcp 10.0.2.15:6443: connect: connection refused"
Apr 03 16:19:52 node1 k3s[3915]: time="2024-04-03T16:19:52Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": EOF"
Apr 03 16:19:53 node1 k3s[3915]: time="2024-04-03T16:19:53Z" level=info msg="Connecting to proxy" url="wss://10.0.2.15:6443/v1-k3s/connect"
Apr 03 16:19:53 node1 k3s[3915]: time="2024-04-03T16:19:53Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 10.0.2.15:6443: connect: connection refused"
Apr 03 16:19:53 node1 k3s[3915]: time="2024-04-03T16:19:53Z" level=error msg="Remotedialer proxy error" error="dial tcp 10.0.2.15:6443: connect: connection refused"
Apr 03 16:19:57 node1 k3s[3915]: time="2024-04-03T16:19:57Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": read tcp 127.0.0.1:37044->127.0.0.1:6444: read: connection reset by peer"
Apr 03 16:19:58 node1 k3s[3915]: time="2024-04-03T16:19:58Z" level=info msg="Connecting to proxy" url="wss://10.0.2.15:6443/v1-k3s/connect"
Apr 03 16:19:58 node1 k3s[3915]: time="2024-04-03T16:19:58Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 10.0.2.15:6443: connect: connection refused"
Apr 03 16:19:58 node1 k3s[3915]: time="2024-04-03T16:19:58Z" level=error msg="Remotedialer proxy error" error="dial tcp 10.0.2.15:6443: connect: connection refused"
Apr 03 16:20:02 node1 k3s[3915]: time="2024-04-03T16:20:02Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": read tcp 127.0.0.1:42192->127.0.0.1:6444: read: connection reset by peer"
Apr 03 16:20:03 node1 k3s[3915]: time="2024-04-03T16:20:03Z" level=info msg="Connecting to proxy" url="wss://10.0.2.15:6443/v1-k3s/connect"
Apr 03 16:20:03 node1 k3s[3915]: time="2024-04-03T16:20:03Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 10.0.2.15:6443: connect: connection refused"
Apr 03 16:20:03 node1 k3s[3915]: time="2024-04-03T16:20:03Z" level=error msg="Remotedialer proxy error" error="dial tcp 10.0.2.15:6443: connect: connection refused"
Apr 03 16:20:07 node1 k3s[3915]: time="2024-04-03T16:20:07Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": read tcp 127.0.0.1:42448->127.0.0.1:6444: read: connection reset by peer"
Apr 03 16:20:08 node1 k3s[3915]: time="2024-04-03T16:20:08Z" level=info msg="Connecting to proxy" url="wss://10.0.2.15:6443/v1-k3s/connect"
Apr 03 16:20:08 node1 k3s[3915]: time="2024-04-03T16:20:08Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 10.0.2.15:6443: connect: connection refused"
Apr 03 16:20:08 node1 k3s[3915]: time="2024-04-03T16:20:08Z" level=error msg="Remotedialer proxy error" error="dial tcp 10.0.2.15:6443: connect: connection refused"
Apr 03 16:20:10 node1 k3s[3915]: W0403 16:20:10.470622    3915 reflector.go:535] k8s.io/[email protected]/tools/cache/reflector.go:229: failed to list *v1.Endpoints: Get "https://127.0.0.1:6444/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&limit=500&resourceVersion=0": read tcp 127.0.0.1:39464->127.0.0.1:6444: read: connection reset by peer - error from a previous attempt: read tcp 127.0.0.1:42520->127.0.0.1:6444: read: connection reset by peer
Apr 03 16:20:10 node1 k3s[3915]: I0403 16:20:10.470982    3915 trace.go:236] Trace[1903520160]: "Reflector ListAndWatch" name:k8s.io/[email protected]/tools/cache/reflector.go:229 (03-Apr-2024 16:20:00.430) (total time: 10040ms):
Apr 03 16:20:10 node1 k3s[3915]: Trace[1903520160]: ---"Objects listed" error:Get "https://127.0.0.1:6444/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&limit=500&resourceVersion=0": read tcp 127.0.0.1:39464->127.0.0.1:6444: read: connection reset by peer - error from a previous attempt: read tcp 127.0.0.1:42520->127.0.0.1:6444: read: connection reset by peer 10039ms (16:20:10.470)
Apr 03 16:20:10 node1 k3s[3915]: Trace[1903520160]: [10.040077312s] [10.040077312s] END
Apr 03 16:20:10 node1 k3s[3915]: E0403 16:20:10.471095    3915 reflector.go:147] k8s.io/[email protected]/tools/cache/reflector.go:229: Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: Get "https://127.0.0.1:6444/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&limit=500&resourceVersion=0": read tcp 127.0.0.1:39464->127.0.0.1:6444: read: connection reset by peer - error from a previous attempt: read tcp 127.0.0.1:42520->127.0.0.1:6444: read: connection reset by peer
Apr 03 16:20:12 node1 k3s[3915]: time="2024-04-03T16:20:12Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": EOF"
Apr 03 16:20:13 node1 k3s[3915]: time="2024-04-03T16:20:13Z" level=info msg="Connecting to proxy" url="wss://10.0.2.15:6443/v1-k3s/connect"
Apr 03 16:20:13 node1 k3s[3915]: time="2024-04-03T16:20:13Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 10.0.2.15:6443: connect: connection refused"
Apr 03 16:20:13 node1 k3s[3915]: time="2024-04-03T16:20:13Z" level=error msg="Remotedialer proxy error" error="dial tcp 10.0.2.15:6443: connect: connection refused"
Apr 03 16:20:17 node1 k3s[3915]: time="2024-04-03T16:20:17Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": read tcp 127.0.0.1:39680->127.0.0.1:6444: read: connection reset by peer"
Apr 03 16:20:18 node1 k3s[3915]: time="2024-04-03T16:20:18Z" level=info msg="Connecting to proxy" url="wss://10.0.2.15:6443/v1-k3s/connect"
Apr 03 16:20:18 node1 k3s[3915]: time="2024-04-03T16:20:18Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 10.0.2.15:6443: connect: connection refused"
Apr 03 16:20:18 node1 k3s[3915]: time="2024-04-03T16:20:18Z" level=error msg="Remotedialer proxy error" error="dial tcp 10.0.2.15:6443: connect: connection refused"
Apr 03 16:20:22 node1 k3s[3915]: time="2024-04-03T16:20:22Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": read tcp 127.0.0.1:57740->127.0.0.1:6444: read: connection reset by peer"
Apr 03 16:20:23 node1 k3s[3915]: time="2024-04-03T16:20:23Z" level=info msg="Connecting to proxy" url="wss://10.0.2.15:6443/v1-k3s/connect"
Apr 03 16:20:23 node1 k3s[3915]: time="2024-04-03T16:20:23Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 10.0.2.15:6443: connect: connection refused"
Apr 03 16:20:23 node1 k3s[3915]: time="2024-04-03T16:20:23Z" level=error msg="Remotedialer proxy error" error="dial tcp 10.0.2.15:6443: connect: connection refused"
Apr 03 16:20:27 node1 k3s[3915]: time="2024-04-03T16:20:27Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": read tcp 127.0.0.1:57920->127.0.0.1:6444: read: connection reset by peer"
Apr 03 16:20:28 node1 k3s[3915]: time="2024-04-03T16:20:28Z" level=info msg="Connecting to proxy" url="wss://10.0.2.15:6443/v1-k3s/connect"
Apr 03 16:20:28 node1 k3s[3915]: time="2024-04-03T16:20:28Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 10.0.2.15:6443: connect: connection refused"
Apr 03 16:20:28 node1 k3s[3915]: time="2024-04-03T16:20:28Z" level=error msg="Remotedialer proxy error" error="dial tcp 10.0.2.15:6443: connect: connection refused"
Apr 03 16:20:32 node1 k3s[3915]: time="2024-04-03T16:20:32Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": read tcp 127.0.0.1:58574->127.0.0.1:6444: read: connection reset by peer"
Apr 03 16:20:33 node1 k3s[3915]: time="2024-04-03T16:20:33Z" level=info msg="Connecting to proxy" url="wss://10.0.2.15:6443/v1-k3s/connect"
Apr 03 16:20:33 node1 k3s[3915]: time="2024-04-03T16:20:33Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 10.0.2.15:6443: connect: connection refused"
Apr 03 16:20:33 node1 k3s[3915]: time="2024-04-03T16:20:33Z" level=error msg="Remotedialer proxy error" error="dial tcp 10.0.2.15:6443: connect: connection refused"

Answered by brandond

Apr 3, 2024

https://docs.k3s.io/installation/configuration#configuration-file

You can create the config file before installation, or pass the arg to the install script.

View full answer

pgregorio-signalytic · 2024-04-03T16:33:05Z

pgregorio-signalytic
Apr 3, 2024
Author

I believe there were issues with older iptables but when I looked at my version, it seems to be okay:

root@master:/home/vagrant# iptables --version
iptables v1.8.7 (nf_tables)

0 replies

brandond · 2024-04-03T17:22:43Z

brandond
Apr 3, 2024
Collaborator

Apr 03 16:18:57 node1 k3s[3915]: time="2024-04-03T16:18:57Z" level=info msg="Connecting to proxy" url="wss://10.0.2.15:6443/v1-k3s/connect"
Apr 03 16:18:57 node1 k3s[3915]: time="2024-04-03T16:18:57Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 10.0.2.15:6443: connect: connection refused"

The agent can't connect to the server. Did you need to use --node-ip on the server to tell it to use the IP address of a different interface for cluster communication? Or do you need to open up ports on that node so that the agent can connect?

7 replies

pgregorio-signalytic Apr 3, 2024
Author

I tried:

root@master:/home/vagrant# k3s kubectl get nodes -o wide
NAME     STATUS   ROLES                  AGE   VERSION        INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
master   Ready    control-plane,master   79m   v1.28.8+k3s1   10.0.2.15     <none>        Ubuntu 22.04.3 LTS   5.15.0-91-generic   containerd://1.7.11-k3s2
root@master:/home/vagrant# k3s server -i 192.168.56.2
INFO[0000] Starting k3s v1.28.8+k3s1 (653dd61a)         
INFO[0000] Configuring sqlite3 database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s 
INFO[0000] Configuring database table schema and indexes, this may take a moment... 
INFO[0000] Database tables and indexes are up to date   
INFO[0000] Kine available at unix://kine.sock           
INFO[0000] Reconciling bootstrap data between datastore and disk 
INFO[0000] certificate CN=kube-apiserver signed by CN=k3s-server-ca@1712160986: notBefore=2024-04-03 16:16:26 +0000 UTC notAfter=2025-04-03 17:42:06 +0000 UTC 
INFO[0000] certificate CN=etcd-peer signed by CN=etcd-peer-ca@1712160986: notBefore=2024-04-03 16:16:26 +0000 UTC notAfter=2025-04-03 17:42:06 +0000 UTC 
INFO[0000] certificate CN=etcd-server signed by CN=etcd-server-ca@1712160986: notBefore=2024-04-03 16:16:26 +0000 UTC notAfter=2025-04-03 17:42:06 +0000 UTC 
FATA[0000] starting kubernetes: preparing server: init cluster datastore and https: listen tcp :6443: bind: address already in use

brandond Apr 3, 2024
Collaborator

You just tried to run a second instance of the k3s server on the same node, which predictably did not work. You should modify the configuration file or service args, and then restart the service.

pgregorio-signalytic Apr 3, 2024
Author

So I ran:

root@master:/home/vagrant# nano /etc/systemd/system/k3s.service

# ExecStart=/usr/local/bin/k3s \
#     server \
#     --node-ip=192.168.56.2 \

root@master:/home/vagrant# systemctl daemon-reload
root@master:/home/vagrant# systemctl restart k3s

root@master:/home/vagrant# k3s kubectl get nodes -o wide
NAME     STATUS   ROLES                  AGE    VERSION        INTERNAL-IP    EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
master   Ready    control-plane,master   116m   v1.28.8+k3s1   192.168.56.2   <none>        Ubuntu 22.04.3 LTS   5.15.0-91-generic   containerd://1.7.11-k3s2
node1    Ready    <none>                 99s    v1.28.8+k3s1   10.0.2.15      <none>        Ubuntu 22.04.3 LTS   5.15.0-91-generic   containerd://1.7.11-k3s2
node2    Ready    <none>                 9s     v1.28.8+k3s1   10.0.2.15      <none>        Ubuntu 22.04.3 LTS   5.15.0-91-generic   containerd://1.7.11-k3s2

I guess I should do the same thing for the agent nodes, but it seems to work, I'm able to connect the nodes now. In this case, I modified the systemd service args. If I wanted to modify the configuration file, which file is that exactly? Also, is the internal IP address something I can set before/during installation instead of after the fact?

brandond Apr 3, 2024
Collaborator

https://docs.k3s.io/installation/configuration#configuration-file

You can create the config file before installation, or pass the arg to the install script.

Answer selected by pgregorio-signalytic

pgregorio-signalytic Apr 3, 2024
Author

Does this look good or do I need to fix anything else?

root@master:/home/vagrant# k3s kubectl get nodes -o wide
NAME     STATUS   ROLES                  AGE     VERSION        INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
master   Ready    control-plane,master   3m30s   v1.28.8+k3s1   192.168.56.2    <none>        Ubuntu 22.04.3 LTS   5.15.0-91-generic   containerd://1.7.11-k3s2
node1    Ready    <none>                 76s     v1.28.8+k3s1   192.168.56.11   <none>        Ubuntu 22.04.3 LTS   5.15.0-91-generic   containerd://1.7.11-k3s2
node2    Ready    <none>                 6s      v1.28.8+k3s1   192.168.56.22   <none>        Ubuntu 22.04.3 LTS   5.15.0-91-generic   containerd://1.7.11-k3s2

Are the agent nodes supposed to say <none> under ROLES?

brandond Apr 3, 2024
Collaborator

yes, agents don't have any roles by default.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to connect to proxy when setting up agent nodes #9861

{{title}}

Replies: 2 comments 7 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

Unable to connect to proxy when setting up agent nodes #9861

pgregorio-signalytic Apr 3, 2024

K3S

Update firewall

MASTER

NODE1

NODE2

Replies: 2 comments · 7 replies

pgregorio-signalytic Apr 3, 2024 Author

brandond Apr 3, 2024 Collaborator

pgregorio-signalytic Apr 3, 2024 Author

brandond Apr 3, 2024 Collaborator

pgregorio-signalytic Apr 3, 2024 Author

brandond Apr 3, 2024 Collaborator

pgregorio-signalytic Apr 3, 2024 Author

brandond Apr 3, 2024 Collaborator

pgregorio-signalytic
Apr 3, 2024

Replies: 2 comments 7 replies

pgregorio-signalytic
Apr 3, 2024
Author

brandond
Apr 3, 2024
Collaborator

pgregorio-signalytic Apr 3, 2024
Author

brandond Apr 3, 2024
Collaborator

pgregorio-signalytic Apr 3, 2024
Author

brandond Apr 3, 2024
Collaborator

pgregorio-signalytic Apr 3, 2024
Author

brandond Apr 3, 2024
Collaborator