k3s service load balancer is in a create and terminate pod loop

[migs35323]

Environmental Info:
K3s Version: 1.24.16 (this also happened in previous patch versions)

Node(s) CPU architecture, OS, and Version: 5 vps, ubuntu 22 servers

Cluster Configuration: 1 master 4 workers

Describe the bug:

so when i something happens to one a node and the load balancer needs to "rebalance" or there is new service load balancer created, klipper goes into a loop of creating and destroying containers, see picture.
anyone can give me an idea on the cause of this situation, basically i have traefik as a daemonset setted up in a 5 node machine,
nodes are hosted in cloud and are all using an external ip address
this is my traefik load balancer service:

(ip and ports are "censored")

apiVersion: v1
kind: Service
metadata:
  annotations:
    field.cattle.io/publicEndpoints: >-
      [{"addresses":["1.1.1.1","2.2.2.2","3.3.3.3"],"port":80,"protocol":"TCP","serviceName":"kube-system:traefik","allNodes":false},{"addresses":["1.1.1.1","2.2.2.2","3.3.3.3"],"port":443,"protocol":"TCP","serviceName":"kube-system:traefik","allNodes":false}]
    meta.helm.sh/release-name: traefik
    meta.helm.sh/release-namespace: kube-system
  finalizers:
    - service.kubernetes.io/load-balancer-cleanup
  labels:
    app.kubernetes.io/instance: traefik-kube-system
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: traefik
    helm.sh/chart: traefik
  name: traefik
  namespace: kube-system
spec:
  allocateLoadBalancerNodePorts: true
  clusterIP: 10.43.999.999
  clusterIPs:
    - 10.43.333.999
  externalIPs:
    - 1.1.1.1
    - 3.3.3.3
    - 4.4.4.4
    - 2.2.2.2
    - 5.5.5.5
  externalTrafficPolicy: Local
  healthCheckNodePort: 99999
  internalTrafficPolicy: Cluster
  ipFamilies:
    - IPv4
  ipFamilyPolicy: PreferDualStack
  loadBalancerSourceRanges:
    - 1.1.1.1/32
    - 3.3.3.3/32
    - 4.4.4.4/32
    - 2.2.2.2/32
    - 5.5.5.5/32
  ports:
    - name: web
      nodePort: 99998
      port: 80
      protocol: TCP
      targetPort: web
    - name: websecure
      nodePort: 99997
      port: 443
      protocol: TCP
      targetPort: websecure
  selector:
    app.kubernetes.io/instance: traefik-kube-system
    app.kubernetes.io/name: traefik
  sessionAffinity: ClientIP
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 10800
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
      - ip: 1.1.1.1
      - ip: 2.2.2.2
      - ip: 3.3.3.3

klipper daemon set that gets created:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  annotations:
    deprecated.daemonset.template.generation: '9414'
    field.cattle.io/publicEndpoints: >-
      [{"nodeName":":vps-1","addresses":["1.1.1.1"],"port":80,"protocol":"TCP","podName":"kube-system:svclb-traefik-091f1f66-5f6zl","allNodes":false},{"nodeName":":vps-1","addresses":["1.1.1.1"],"port":443,"protocol":"TCP","podName":"kube-system:svclb-traefik-091f1f66-5f6zl","allNodes":false},{"nodeName":":vps-2","addresses":["4.4.4.4"],"port":80,"protocol":"TCP","podName":"kube-system:svclb-traefik-091f1f66-9jzvv","allNodes":false},{"nodeName":":vps-2","addresses":["4.4.4.4"],"port":443,"protocol":"TCP","podName":"kube-system:svclb-traefik-091f1f66-9jzvv","allNodes":false}]
    objectset.rio.cattle.io/id: ''
    objectset.rio.cattle.io/owner-gvk: /v1, Kind=Service
    objectset.rio.cattle.io/owner-name: traefik
    objectset.rio.cattle.io/owner-namespace: kube-system
  generation: 9414
  labels:
    svccontroller.k3s.cattle.io/nodeselector: 'true'
    svccontroller.k3s.cattle.io/svcname: traefik
    svccontroller.k3s.cattle.io/svcnamespace: kube-system
  name: svclb-traefik-091f1f66
  namespace: kube-system
spec:
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: svclb-traefik-091f1f66
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: svclb-traefik-091f1f66
        svccontroller.k3s.cattle.io/svcname: traefik
        svccontroller.k3s.cattle.io/svcnamespace: kube-system
    spec:
      automountServiceAccountToken: false
      containers:
        - env:
            - name: SRC_PORT
              value: '80'
            - name: SRC_RANGES
              value: >-
                2.2.2.2/32 3.3.3.3/32 1.1.1.1/32
                4.4.4.4/32 5.5.5.5/32
            - name: DEST_PROTO
              value: TCP
            - name: DEST_PORT
              value: '31817'
            - name: DEST_IPS
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: status.hostIP
          image: rancher/klipper-lb:v0.4.4
          imagePullPolicy: IfNotPresent
          name: lb-tcp-80
          ports:
            - containerPort: 80
              hostPort: 80
              name: lb-tcp-80
              protocol: TCP
          resources: {}
          securityContext:
            capabilities:
              add:
                - NET_ADMIN
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
        - env:
            - name: SRC_PORT
              value: '443'
            - name: SRC_RANGES
              value: >-
                3.3.3.3/32 1.1.1.1/32 4.4.4.4/32
                5.5.5.5/32 2.2.2.2/32
            - name: DEST_PROTO
              value: TCP
            - name: DEST_PORT
              value: '31763'
            - name: DEST_IPS
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: status.hostIP
          image: rancher/klipper-lb:v0.4.4
          imagePullPolicy: IfNotPresent
          name: lb-tcp-443
          ports:
            - containerPort: 443
              hostPort: 443
              name: lb-tcp-443
              protocol: TCP
          resources: {}
          securityContext:
            capabilities:
              add:
                - NET_ADMIN
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      nodeSelector:
        svccontroller.k3s.cattle.io/enablelb: 'true'
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        sysctls:
          - name: net.ipv4.ip_forward
            value: '1'
      serviceAccount: svclb
      serviceAccountName: svclb
      terminationGracePeriodSeconds: 30
      tolerations:
        - effect: NoSchedule
          key: node-role.kubernetes.io/master
          operator: Exists
        - effect: NoSchedule
          key: node-role.kubernetes.io/control-plane
          operator: Exists
        - key: CriticalAddonsOnly
          operator: Exists
  updateStrategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
    type: RollingUpdate
status:
  currentNumberScheduled: 5
  desiredNumberScheduled: 5
  numberAvailable: 3
  numberMisscheduled: 0
  numberReady: 3
  numberUnavailable: 2
  observedGeneration: 9414
  updatedNumberScheduled: 1

pictures:

extra notes:

all my current nodes have the svccontroller.k3s.cattle.io/enablelb=true
i have all the nodes with svccontroller.k3s.cattle.io/lbpool=pool1
and one single node with svccontroller.k3s.cattle.io/lbpool=pool2

the load balancer config is based of several attempts and threads found in this repository so my services get to see the clients addresses
eventually after a very long time the "loop" stops and it all the klipper pods get ok, but this is taking a very long time...

[brandond]

Can you show more information on what the loop looks like? I see that there are some pods getting recreated, but that is generally expected when the backing service changes - especially since you are using externalTrafficPolicy: Local and servicelb needs to create or terminate pods to track changes to which nodes have healthy pods for the service.

At this point I'm not seeing anything that's not expected behavior, given how you've configured the service.

[migs35323]

i fail to understand how can this be normal
like i mentioned this is a 5 node cluster,
picture taken after 30 minutes:

what i mean by looping is the daemon set was creating and destroying hundred of pods and this happens for more than 2 hours
as you can see in the first picture in the thread, the daemon set was in "generation 8622", from a new daemonset, this happened after 1 node went down (one vps got downtime)
this second picture was taken after about half an hour. as you can see, to schedule 5 pods in 5 nodes in half an hour he created and destroyed 900+ pods
this is happening across the whole cluster, it's not like a problematic node or a few set of nodes, sometimes i get 4 pods running for a brief moment and then it kills them all except 1, and it gets all all over again..

keep in mind this is only happening with the klipper service load balancer, no other daemon sets or deployments are having this behavior.

i would believe this was normal if the pods get killed and restarted one or 2 times or problems if a port was being used...
but i fail to understand this behaviors...

the logs are no great help:
the load balancer pods showed this error before being killed /usr/bin/entry: line 84: can't open /pause: Interrupted system call
master node journalctl:

Sep 07 18:54:05 Master-Node k3s[2523409]: I0907 18:54:05.979395 2523409 event.go:294] "Event occurred" object="kube-system/traefik" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="UpdatedLoadBalancer" message="Updated LoadBalancer with new IPs: [1.1.1.1 2.2.2.2 3.3.3.3 4.4.4.4] -> [1.1.1.1 5.5.5.5 2.2.2.2 3.3.3.3 4.4.4.4]"
Sep 07 18:54:06 Master-Node k3s[2523409]: I0907 18:54:06.082434 2523409 event.go:294] "Event occurred" object="kube-system/traefik" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="EnsuringLoadBalancer" message="Ensuring load balancer"
Sep 07 18:54:06 Master-Node k3s[2523409]: I0907 18:54:06.145331 2523409 event.go:294] "Event occurred" object="kube-system/traefik" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="AppliedDaemonSet" message="Applied LoadBalancer DaemonSet kube-system/svclb-traefik-091f1f66"
Sep 07 18:54:06 Master-Node k3s[2523409]: I0907 18:54:06.751788 2523409 scope.go:110] "RemoveContainer" containerID="9657ecacd2dd7b881334043a646dcbf93feaa202da44a2338ccd606612c198fa"
Sep 07 18:54:06 Master-Node k3s[2523409]: I0907 18:54:06.764808 2523409 scope.go:110] "RemoveContainer" containerID="4a81a424b1dba7d344d930c0826534df04a27ba0d44fc9ae4b30a94c4c55080f"
Sep 07 18:54:06 Master-Node k3s[2523409]: I0907 18:54:06.770307 2523409 scope.go:110] "RemoveContainer" containerID="9657ecacd2dd7b881334043a646dcbf93feaa202da44a2338ccd606612c198fa"
Sep 07 18:54:06 Master-Node k3s[2523409]: E0907 18:54:06.771153 2523409 remote_runtime.go:604] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = an error occurred when try to find container \"9657ecacd2dd7b881334043a646dcbf93feaa202da44a2338ccd606612c198fa\": not found" containerID="9657ecacd2dd7b881334043a646dcbf93feaa202da44a2338ccd606612c198fa"
Sep 07 18:54:06 Master-Node k3s[2523409]: I0907 18:54:06.771314 2523409 pod_container_deletor.go:52] "DeleteContainer returned error" containerID={Type:containerd ID:9657ecacd2dd7b881334043a646dcbf93feaa202da44a2338ccd606612c198fa} err="failed to get container status \"9657ecacd2dd7b881334043a646dcbf93feaa202da44a2338ccd606612c198fa\": rpc error: code = NotFound desc = an error occurred when try to find container \"9657ecacd2dd7b881334043a646dcbf93feaa202da44a2338ccd606612c198fa\": not found"

... the following logs happened over and over and over ....

Sep 07 20:29:54 Master-Node k3s[2523409]: I0907 20:29:54.081575 2523409 event.go:294] "Event occurred" object="kube-system/traefik" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="UpdatedLoadBalancer" message="Updated LoadBalancer with new IPs: [1.1.1.1 2.2.2.2 3.3.3.3] -> [4.4.4.4 1.1.1.1 2.2.2.2 3.3.3.3]"
Sep 07 20:29:54 Master-Node k3s[2523409]: I0907 20:29:54.300770 2523409 event.go:294] "Event occurred" object="kube-system/traefik" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="EnsuringLoadBalancer" message="Ensuring load balancer"
Sep 07 20:29:54 Master-Node k3s[2523409]: I0907 20:29:54.330541 2523409 event.go:294] "Event occurred" object="kube-system/svclb-traefik-091f1f66" fieldPath="" kind="DaemonSet" apiVersion="apps/v1" type="Normal" reason="SuccessfulDelete" message="Deleted pod: svclb-traefik-091f1f66-7szth"
Sep 07 20:29:54 Master-Node k3s[2523409]: I0907 20:29:54.481144 2523409 event.go:294] "Event occurred" object="kube-system/traefik" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="AppliedDaemonSet" message="Applied LoadBalancer DaemonSet kube-system/svclb-traefik-091f1f66"
Sep 07 20:29:54 Master-Node k3s[2523409]: E0907 20:29:54.705929 2523409 daemon_controller.go:319] kube-system/svclb-traefik-091f1f66 failed with : error storing status for daemon set &v1.DaemonSet{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"svclb-traefik-091f1f66", GenerateName:"", Namespace:"kube-system", SelfLink:"", UID:"488aa1d8-65b4-4e93-9027-464630b903b1", ResourceVersion:"108005324", Generation:20432, CreationTimestamp:time.Date(2023, time.September, 4, 18, 31, 0, 0, time.Local), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"objectset.rio.cattle.io/hash":"836fcbce022d5dae5b36694fe1eaf389c93af7dc", "svccontroller.k3s.cattle.io/nodeselector":"true", "svccontroller.k3s.cattle.io/svcname":"traefik", "svccontroller.k3s.cattle.io/svcnamespace":"kube-system"}, Annotations:map[string]string{"deprecated.daemonset.template.generation":"20432", "field.cattle.io/publicEndpoints":"[{\"nodeName\":\":worker-1\",\"addresses\":[\"2.2.2.2\"],\"port\":80,\"protocol\":\"TCP\",\"podName\":\"kube-system:svclb-traefik-091f1f66-h55c7\",\"allNodes\":false},{\"nodeName\":\":worker-1\",\"addresses\":[\"2.2.2.2\"],\"port\":443,\"protocol\":\"TCP\",\"podName\":\"kube-system:svclb-traefik-091f1f66-h55c7\",\"allNodes\":false},{\"nodeName\":\":Master-Node\",\"addresses\":[\"1.1.1.1\"],\"port\":80,\"protocol\":\"TCP\",\"podName\":\"kube-system:svclb-traefik-091f1f66-x9xf7\",\"allNodes\":false},{\"nodeName\":\":Master-Node\",\"addresses\":[\"1.1.1.1\"],\"port\":443,\"protocol\":\"TCP\",\"podName\":\"kube-system:svclb-traefik-091f1f66-x9xf7\",\"allNodes\":false},{\"nodeName\":\":node-2\",\"addresses\":[\"3.3.3.3\"],\"port\":80,\"protocol\":\"TCP\",\"podName\":\"kube-system:svclb-traefik-091f1f66-4kvs2\",\"allNodes\":false},{\"nodeName\":\":node-2\",\"addresses\":[\"3.3.3.3\"],\"port\":443,\"protocol\":\"TCP\",\"podName\":\"kube-system:svclb-traefik-091f1f66-4kvs2\",\"allNodes\":false}]", "objectset.rio.cattle.io/applied":"H4sIAAAAAAAA/9RW3W7jOA99lQGvHTWO0yQ18F0Ubb9FsTtpkGT2ZhAUtEw32siSIdGeBoXffSE37bho+oMFdoG5ckWRR+ThKZkHwEr9Sc4rayAFrCp/0sQQwU6ZHFK4RCqtWRFDBCUx5sgI6QOgMZaRlTU+HG32F0n2xMIpKyQyaxLKnqiAAdGb9/aHITe4a3aQwkkTR19+Vyb/34pcoyR9GGewJEiBHVKhdp9y9xXKELOrMxr4vWcqoY1AOuqKWauSPGNZQWpqrSPQmJF+t8Qt+i2kMEsmhcwkDUej/DRHOs2SyeRsXFBMWCSzM3mWYDHNJUTgGymtYWe1Jid2ie+hGZuTJ02SretKq+mDCN/IVzx8wv8NIg5QvpE6GxwAB8OzuIiLyQQe798I9RXJQNTP9B+gRJbbP545xKp6G7xtI2AqK41MXWxPbp/oz7vY/x2FPSKwZlva2vBBz+dShtPa7shAWqD2FEF4BJUh5yH9/gBkmu57yGe1vLhd3CzXEEGDug6m2RDa6IXD8nz+29Wq5xLPZiIZi3iSiDieniSjL5NTEQ/PRDwKn2CIZ6diNJyK0WksJuNgSWIRz6ZiOhaj/gOXV6v17WJ5s77pPbC+WLz2eZlmEs/i6Sun68Vznv93tgwsFYp0vqTi+e8FcviH8oxce7G1nq8X0LbtJgJV4l3AcmjkltzJTquqIjfQWdoMxViM4eCzqLVeWK3kHlK4LuaWF448GYZniQeZyGowG0IElXX8yP9zOxbWMaSzYQQhg5+nY9HOspVWPxGzicCRt7WTFJQZFEGydor3F9Yw3XOnaKwwU1qxokf55jmk32F+tb49v/x6PYdN2wb2PhbEeJz8IoqYTpJfQBGBz3ckMR4nfU10x6MA/5oqNhGENbHqzdn3phUZzDTp7GmdHH/Z771k7ftaM8RCVc1YqOq2sO4HurwvKWg3HVR/ts176wMiYKvJPf1KCNOtKEgypDC3K7mlvNZhu+0o9CSUNHBWkwjz1Bli8iH9Ej2TC8u9CljdWry6V559p6V/AnmgalBpNPQm8iPGhVOsJOrzPLfG3xi9Px6wCZO/rnJkWrFDprt9oDV0RJm7b93F40a8/2awQaVDVyCNw9bbV4G15QvfbpN0iu+EUTtHhud1mZF7KjSHdBhBTl45yo9dmc72VXl/xLwkzPeQDtv27wAAAP//Lntx8QUKAAA", "objectset.rio.cattle.io/id":"", "objectset.rio.cattle.io/owner-gvk":"/v1, Kind=Service", "objectset.rio.cattle.io/owner-name":"traefik", "objectset.rio.cattle.io/owner-namespace":"kube-system"}, OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ZZZ_DeprecatedClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"k3s", Operation:"Update", APIVersion:"apps/v1", Time:time.Date(2023, time.September, 4, 18, 31, 0, 0, time.Local), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc037982270), Subresource:""}, v1.ManagedFieldsEntry{Manager:"rancher", Operation:"Update", APIVersion:"apps/v1", Time:time.Date(2023, time.September, 4, 18, 31, 2, 0, time.Local), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc0379822a0), Subresource:""}, v1.ManagedFieldsEntry{Manager:"k3s", Operation:"Update", APIVersion:"apps/v1", Time:time.Date(2023, time.September, 7, 20, 29, 52, 0, time.Local), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc0379822e8), Subresource:"status"}}}, Spec:v1.DaemonSetSpec{Selector:(*v1.LabelSelector)(0xc0665d3c60), Template:v1.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"app":"svclb-traefik-091f1f66", "svccontroller.k3s.cattle.io/svcname":"traefik", "svccontroller.k3s.cattle.io/svcnamespace":"kube-system"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ZZZ_DeprecatedClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.PodSpec{Volumes:[]v1.Volume(nil), InitContainers:[]v1.Container(nil), Containers:[]v1.Container{v1.Container{Name:"lb-tcp-80", Image:"rancher/klipper-lb:v0.4.4", Command:[]string(nil), Args:[]string(nil), WorkingDir:"", Ports:[]v1.ContainerPort{v1.ContainerPort{Name:"lb-tcp-80", HostPort:80, ContainerPort:80, Protocol:"TCP", HostIP:""}}, EnvFrom:[]v1.EnvFromSource(nil), Env:[]v1.EnvVar{v1.EnvVar{Name:"SRC_PORT", Value:"80", ValueFrom:(*v1.EnvVarSource)(nil)}, v1.EnvVar{Name:"SRC_RANGES", Value:"2.2.2.2/32 3.3.3.3/32 1.1.1.1/32 5.5.5.5/32 4.4.4.4/32", ValueFrom:(*v1.EnvVarSource)(nil)}, v1.EnvVar{Name:"DEST_PROTO", Value:"TCP", ValueFrom:(*v1.EnvVarSource)(nil)}, v1.EnvVar{Name:"DEST_PORT", Value:"31817", ValueFrom:(*v1.EnvVarSource)(nil)}, v1.EnvVar{Name:"DEST_IPS", Value:"", ValueFrom:(*v1.EnvVarSource)(0xc0665d3c80)}}, Resources:v1.ResourceRequirements{Limits:v1.ResourceList(nil), Requests:v1.ResourceList(nil)}, VolumeMounts:[]v1.VolumeMount(nil), VolumeDevices:[]v1.VolumeDevice(nil), LivenessProbe:(*v1.Probe)(nil), ReadinessProbe:(*v1.Probe)(nil), StartupProbe:(*v1.Probe)(nil), Lifecycle:(*v1.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"IfNotPresent", SecurityContext:(*v1.SecurityContext)(0xc055619620), Stdin:false, StdinOnce:false, TTY:false}, v1.Container{Name:"lb-tcp-443", Image:"rancher/klipper-lb:v0.4.4", Command:[]string(nil), Args:[]string(nil), WorkingDir:"", Ports:[]v1.ContainerPort{v1.ContainerPort{Name:"lb-tcp-443", HostPort:443, ContainerPort:443, Protocol:"TCP", HostIP:""}}, EnvFrom:[]v1.EnvFromSource(nil), Env:[]v1.EnvVar{v1.EnvVar{Name:"SRC_PORT", Value:"443", ValueFrom:(*v1.EnvVarSource)(nil)}, v1.EnvVar{Name:"SRC_RANGES", Value:"2.2.2.2/32 3.3.3.3/32 1.1.1.1/32 5.5.5.5/32 4.4.4.4/32", ValueFrom:(*v1.EnvVarSource)(nil)}, v1.EnvVar{Name:"DEST_PROTO", Value:"TCP", ValueFrom:(*v1.EnvVarSource)(nil)}, v1.EnvVar{Name:"DEST_PORT", Value:"31763", ValueFrom:(*v1.EnvVarSource)(nil)}, v1.EnvVar{Name:"DEST_IPS", Value:"", ValueFrom:(*v1.EnvVarSource)(0xc0665d3cc0)}}, Resources:v1.ResourceRequirements{Limits:v1.ResourceList(nil), Requests:v1.ResourceList(nil)}, VolumeMounts:[]v1.VolumeMount(nil), VolumeDevices:[]v1.VolumeDevice(nil), LivenessProbe:(*v1.Probe)(nil), ReadinessProbe:(*v1.Probe)(nil), StartupProbe:(*v1.Probe)(nil), Lifecycle:(*v1.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"IfNotPresent", SecurityContext:(*v1.SecurityContext)(0xc055619860), Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]v1.EphemeralContainer(nil), RestartPolicy:"Always", TerminationGracePeriodSeconds:(*int64)(0xc00e6fd148), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string{"svccontroller.k3s.cattle.io/enablelb":"true"}, ServiceAccountName:"svclb", DeprecatedServiceAccount:"svclb", AutomountServiceAccountToken:(*bool)(0xc00e6fd16a), NodeName:"", HostNetwork:false, HostPID:false, HostIPC:false, ShareProcessNamespace:(*bool)(nil), SecurityContext:(*v1.PodSecurityContext)(0xc07e640a80), ImagePullSecrets:[]v1.LocalObjectReference(nil), Hostname:"", Subdomain:"", Affinity:(*v1.Affinity)(nil), SchedulerName:"default-scheduler", Tolerations:[]v1.Toleration{v1.Toleration{Key:"node-role.kubernetes.io/master", Operator:"Exists", Value:"", Effect:"NoSchedule", TolerationSeconds:(*int64)(nil)}, v1.Toleration{Key:"node-role.kubernetes.io/control-plane", Operator:"Exists", Value:"", Effect:"NoSchedule", TolerationSeconds:(*int64)(nil)}, v1.Toleration{Key:"CriticalAddonsOnly", Operator:"Exists", Value:"", Effect:"", TolerationSeconds:(*int64)(nil)}}, HostAliases:[]v1.HostAlias(nil), PriorityClassName:"", Priority:(*int32)(nil), DNSConfig:(*v1.PodDNSConfig)(nil), ReadinessGates:[]v1.PodReadinessGate(nil), RuntimeClassName:(*string)(nil), EnableServiceLinks:(*bool)(nil), PreemptionPolicy:(*v1.PreemptionPolicy)(nil), Overhead:v1.ResourceList(nil), TopologySpreadConstraints:[]v1.TopologySpreadConstraint(nil), SetHostnameAsFQDN:(*bool)(nil), OS:(*v1.PodOS)(nil)}}, UpdateStrategy:v1.DaemonSetUpdateStrategy{Type:"RollingUpdate", RollingUpdate:(*v1.RollingUpdateDaemonSet)(0xc05f60d5f0)}, MinReadySeconds:0, RevisionHistoryLimit:(*int32)(0xc00e6fd198)}, Status:v1.DaemonSetStatus{CurrentNumberScheduled:5, NumberMisscheduled:0, DesiredNumberScheduled:5, NumberReady:3, ObservedGeneration:20431, UpdatedNumberScheduled:0, NumberAvailable:3, NumberUnavailable:2, CollisionCount:(*int32)(nil), Conditions:[]v1.DaemonSetCondition(nil)}}: Operation cannot be fulfilled on daemonsets.apps "svclb-traefik-091f1f66": the object has been modified; please apply your changes to the latest version and try again
Sep 07 20:29:55 Master-Node k3s[2523409]: I0907 20:29:55.032784 2523409 event.go:294] "Event occurred" object="kube-system/traefik" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="UpdatedLoadBalancer" message="Updated LoadBalancer with new IPs: [4.4.4.4 1.1.1.1 2.2.2.2 3.3.3.3] -> [1.1.1.1 2.2.2.2 3.3.3.3]"
Sep 07 20:29:55 Master-Node k3s[2523409]: I0907 20:29:55.207667 2523409 trace.go:219] Trace[494449422]: "Delete" url:/api/v1/namespaces/kube-system/pods/svclb-traefik-091f1f66-btsbb,user-agent:k3s/v1.24.16+k3s1 (linux/amd64) kubernetes/ccae12e,audit-id:e4bae4c9-0e5b-4847-90bf-ee47bf242c56,client:4.4.4.4,accept:application/vnd.kubernetes.protobuf,application/json,protocol:HTTP/2.0 (07-Sep-2023 20:29:54.675) (total time: 532ms):
Sep 07 20:29:55 Master-Node k3s[2523409]: Trace[494449422]: ---"Object deleted from database" 529ms (20:29:55.204)
Sep 07 20:29:55 Master-Node k3s[2523409]: Trace[494449422]: [532.1288ms] [532.1288ms] END
Sep 07 20:29:55 Master-Node k3s[2523409]: I0907 20:29:55.222186 2523409 event.go:294] "Event occurred" object="kube-system/traefik" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="EnsuringLoadBalancer" message="Ensuring load balancer"
Sep 07 20:29:55 Master-Node k3s[2523409]: I0907 20:29:55.467925 2523409 event.go:294] "Event occurred" object="kube-system/svclb-traefik-091f1f66" fieldPath="" kind="DaemonSet" apiVersion="apps/v1" type="Normal" reason="SuccessfulCreate" message="Created pod: svclb-traefik-091f1f66-kwf72"
Sep 07 20:29:55 Master-Node k3s[2523409]: I0907 20:29:55.670261 2523409 event.go:294] "Event occurred" object="kube-system/traefik" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="AppliedDaemonSet" message="Applied LoadBalancer DaemonSet kube-system/svclb-traefik-091f1f66"
Sep 07 20:29:55 Master-Node k3s[2523409]: E0907 20:29:55.797964 2523409 daemon_controller.go:319] kube-system/svclb-traefik-091f1f66 failed with : error storing status for daemon set &v1.DaemonSet{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"svclb-traefik-091f1f66", GenerateName:"", Namespace:"kube-system", SelfLink:"", UID:"488aa1d8-65b4-4e93-9027-464630b903b1", ResourceVersion:"108005345", Generation:20433, CreationTimestamp:time.Date(2023, time.September, 4, 18, 31, 0, 0, time.Local), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"objectset.rio.cattle.io/hash":"836fcbce022d5dae5b36694fe1eaf389c93af7dc", "svccontroller.k3s.cattle.io/nodeselector":"true", "svccontroller.k3s.cattle.io/svcname":"traefik", "svccontroller.k3s.cattle.io/svcnamespace":"kube-system"}, Annotations:map[string]string{"deprecated.daemonset.template.generation":"20433", "field.cattle.io/publicEndpoints":"[{\"nodeName\":\":worker-1\",\"addresses\":[\"2.2.2.2\"],\"port\":80,\"protocol\":\"TCP\",\"podName\":\"kube-system:svclb-traefik-091f1f66-h55c7\",\"allNodes\":false},{\"nodeName\":\":worker-1\",\"addresses\":[\"2.2.2.2\"],\"port\":443,\"protocol\":\"TCP\",\"podName\":\"kube-system:svclb-traefik-091f1f66-h55c7\",\"allNodes\":false},{\"nodeName\":\":Master-Node\",\"addresses\":[\"1.1.1.1\"],\"port\":80,\"protocol\":\"TCP\",\"podName\":\"kube-system:svclb-traefik-091f1f66-x9xf7\",\"allNodes\":false},{\"nodeName\":\":Master-Node\",\"addresses\":[\"1.1.1.1\"],\"port\":443,\"protocol\":\"TCP\",\"podName\":\"kube-system:svclb-traefik-091f1f66-x9xf7\",\"allNodes\":false},{\"nodeName\":\":node-2\",\"addresses\":[\"3.3.3.3\"],\"port\":80,\"protocol\":\"TCP\",\"podName\":\"kube-system:svclb-traefik-091f1f66-4kvs2\",\"allNodes\":false},{\"nodeName\":\":node-2\",\"addresses\":[\"3.3.3.3\"],\"port\":443,\"protocol\":\"TCP\",\"podName\":\"kube-system:svclb-traefik-091f1f66-4kvs2\",\"allNodes\":false}]", "objectset.rio.cattle.io/applied":"H4sIAAAAAAAA/9RWUU/jRhD+K6d5dpY4Nomx1AcEtELtBSvJ9eUUofV6TLZZ71q7Yx8R8n+v1gQwIgFatZX6ZHZ25tuZbz5m8gC8lr+jddJoSIHXtTtpQwhgK3UBKVxyrIxeIkEAFRIvOHFIH4BrbYiTNNr5o8n/QEEOiVlpmOBECpk0J9JjQHD03vzQaEd37RZSOGnD4MuvUhc/LdG2UuCHcZpXCCmQ5VjK7afcXc2Fj9k2OY7czhFW0AUgLPbFrGSFjnhVQ6obpQJQPEf1bokb7jaQQhJNS5ELHE8mxWnB8TSPptOzuMQQeRklZ+Is4uWsEBCAa4UwmqxRCi3bRm6Apk2BDhUKMrYvrcEPIlwr3vDwCf8jROyhXCtUPtoDjsZnYRmW0yk83h8JdTUKT9RL+g9QcRKb35455HV9HLzrAiCsasUJ+9iB3D7Rn3ex/zsKB0TwhkxlGk17PZ8L4U8rs0UNacmVwwD8I1xqtA7S7w+Auu2/+3yWi4vb7GaxggBarhpvSsbQBa8cFufzX66WA5cwSVgUs3AasTCcnUSTL9NTFo7PWDjxH28Ik1M2Gc/Y5DRk09hbopCFyYzNYjYZPnB5tVzdZoub1c3ggdVF9tbndZpRmISzN07X2XOeP1tTeZZKiapYYPn8d8bJ/0M54tQ4tjGOrjPoum4dgKz4nceyXIsN2pOtknWNdqTytB2zmMWw98kapTKjpNhBCtfl3FBm0aEmeJa4l4moR8kYAqiNpUf+n9uRGUuQJuMAfAYvp0PR1pARRj0Rsw7AojONFeiV6RWBorGSdhdGE95Tr2he81wqSRIf5VsUkH6H+dXq9vzy6/Uc1l3n2ftYEHEcfaiIv9b/KHlxmbBJNGX/jCJm0+h/oAjP5zuSiONoqIn+eBDgX1PFOgC/JpaDOfvetELNc4Uqf1onh192OydIuaHWNBKTdRszWd+Wxv7gthgOGejWPdRwts0H6wMCIKPQPv1K8NOtLFEQpDA3S7HBolF+u23R98SXNLJGIfPz1GokdD79ijtC65d77bH6tXh1Lx25Xkt/B3JP1ahWXONR5EeMCytJCq7Oi8Jod6PV7nDA2k/+pi444ZIsJ7zbeVp9R6S++9ZfPG7E+2+at1wq3xVIQ7/1drVnbfHKt98kveJ7YTTWoqZ5U+VonwotIB0HUKCTFotDV7q3fZXOHTAvkBc7SMdd92cAAAD//wqDQxoFCgAA", "objectset.rio.cattle.io/id":"", "objectset.rio.cattle.io/owner-gvk":"/v1, Kind=Service", "objectset.rio.cattle.io/owner-name":"traefik", "objectset.rio.cattle.io/owner-namespace":"kube-system"}, OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ZZZ_DeprecatedClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"k3s", Operation:"Update", APIVersion:"apps/v1", Time:time.Date(2023, time.September, 4, 18, 31, 0, 0, time.Local), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc0621d9998), Subresource:""}, v1.ManagedFieldsEntry{Manager:"rancher", Operation:"Update", APIVersion:"apps/v1", Time:time.Date(2023, time.September, 4, 18, 31, 2, 0, time.Local), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc0621d99c8), Subresource:""}, v1.ManagedFieldsEntry{Manager:"k3s", Operation:"Update", APIVersion:"apps/v1", Time:time.Date(2023, time.September, 7, 20, 29, 53, 0, time.Local), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc0621d9a70), Subresource:"status"}}}, Spec:v1.DaemonSetSpec{Selector:(*v1.LabelSelector)(0xc01cfe3e00), Template:v1.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"app":"svclb-traefik-091f1f66", "svccontroller.k3s.cattle.io/svcname":"traefik", "svccontroller.k3s.cattle.io/svcnamespace":"kube-system"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ZZZ_DeprecatedClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.PodSpec{Volumes:[]v1.Volume(nil), InitContainers:[]v1.Container(nil), Containers:[]v1.Container{v1.Container{Name:"lb-tcp-80", Image:"rancher/klipper-lb:v0.4.4", Command:[]string(nil), Args:[]string(nil), WorkingDir:"", Ports:[]v1.ContainerPort{v1.ContainerPort{Name:"lb-tcp-80", HostPort:80, ContainerPort:80, Protocol:"TCP", HostIP:""}}, EnvFrom:[]v1.EnvFromSource(nil), Env:[]v1.EnvVar{v1.EnvVar{Name:"SRC_PORT", Value:"80", ValueFrom:(*v1.EnvVarSource)(nil)}, v1.EnvVar{Name:"SRC_RANGES", Value:"2.2.2.2/32 3.3.3.3/32 1.1.1.1/32 5.5.5.5/32 4.4.4.4/32", ValueFrom:(*v1.EnvVarSource)(nil)}, v1.EnvVar{Name:"DEST_PROTO", Value:"TCP", ValueFrom:(*v1.EnvVarSource)(nil)}, v1.EnvVar{Name:"DEST_PORT", Value:"31817", ValueFrom:(*v1.EnvVarSource)(nil)}, v1.EnvVar{Name:"DEST_IPS", Value:"", ValueFrom:(*v1.EnvVarSource)(0xc01cfe3e20)}}, Resources:v1.ResourceRequirements{Limits:v1.ResourceList(nil), Requests:v1.ResourceList(nil)}, VolumeMounts:[]v1.VolumeMount(nil), VolumeDevices:[]v1.VolumeDevice(nil), LivenessProbe:(*v1.Probe)(nil), ReadinessProbe:(*v1.Probe)(nil), StartupProbe:(*v1.Probe)(nil), Lifecycle:(*v1.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"IfNotPresent", SecurityContext:(*v1.SecurityContext)(0xc06e1d8060), Stdin:false, StdinOnce:false, TTY:false}, v1.Container{Name:"lb-tcp-443", Image:"rancher/klipper-lb:v0.4.4", Command:[]string(nil), Args:[]string(nil), WorkingDir:"", Ports:[]v1.ContainerPort{v1.ContainerPort{Name:"lb-tcp-443", HostPort:443, ContainerPort:443, Protocol:"TCP", HostIP:""}}, EnvFrom:[]v1.EnvFromSource(nil), Env:[]v1.EnvVar{v1.EnvVar{Name:"SRC_PORT", Value:"443", ValueFrom:(*v1.EnvVarSource)(nil)}, v1.EnvVar{Name:"SRC_RANGES", Value:"3.3.3.3/32 1.1.1.1/32 5.5.5.5/32 4.4.4.4/32 2.2.2.2/32", ValueFrom:(*v1.EnvVarSource)(nil)}, v1.EnvVar{Name:"DEST_PROTO", Value:"TCP", ValueFrom:(*v1.EnvVarSource)(nil)}, v1.EnvVar{Name:"DEST_PORT", Value:"31763", ValueFrom:(*v1.EnvVarSource)(nil)}, v1.EnvVar{Name:"DEST_IPS", Value:"", ValueFrom:(*v1.EnvVarSource)(0xc01cfe3e60)}}, Resources:v1.ResourceRequirements{Limits:v1.ResourceList(nil), Requests:v1.ResourceList(nil)}, VolumeMounts:[]v1.VolumeMount(nil), VolumeDevices:[]v1.VolumeDevice(nil), LivenessProbe:(*v1.Probe)(nil), ReadinessProbe:(*v1.Probe)(nil), StartupProbe:(*v1.Probe)(nil), Lifecycle:(*v1.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"IfNotPresent", SecurityContext:(*v1.SecurityContext)(0xc06e1d80c0), Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]v1.EphemeralContainer(nil), RestartPolicy:"Always", TerminationGracePeriodSeconds:(*int64)(0xc083f5eed8), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string{"svccontroller.k3s.cattle.io/enablelb":"true"}, ServiceAccountName:"svclb", DeprecatedServiceAccount:"svclb", AutomountServiceAccountToken:(*bool)(0xc083f5eefa), NodeName:"", HostNetwork:false, HostPID:false, HostIPC:false, ShareProcessNamespace:(*bool)(nil), SecurityContext:(*v1.PodSecurityContext)(0xc0424f3b90), ImagePullSecrets:[]v1.LocalObjectReference(nil), Hostname:"", Subdomain:"", Affinity:(*v1.Affinity)(nil), SchedulerName:"default-scheduler", Tolerations:[]v1.Toleration{v1.Toleration{Key:"node-role.kubernetes.io/master", Operator:"Exists", Value:"", Effect:"NoSchedule", TolerationSeconds:(*int64)(nil)}, v1.Toleration{Key:"node-role.kubernetes.io/control-plane", Operator:"Exists", Value:"", Effect:"NoSchedule", TolerationSeconds:(*int64)(nil)}, v1.Toleration{Key:"CriticalAddonsOnly", Operator:"Exists", Value:"", Effect:"", TolerationSeconds:(*int64)(nil)}}, HostAliases:[]v1.HostAlias(nil), PriorityClassName:"", Priority:(*int32)(nil), DNSConfig:(*v1.PodDNSConfig)(nil), ReadinessGates:[]v1.PodReadinessGate(nil), RuntimeClassName:(*string)(nil), EnableServiceLinks:(*bool)(nil), PreemptionPolicy:(*v1.PreemptionPolicy)(nil), Overhead:v1.ResourceList(nil), TopologySpreadConstraints:[]v1.TopologySpreadConstraint(nil), SetHostnameAsFQDN:(*bool)(nil), OS:(*v1.PodOS)(nil)}}, UpdateStrategy:v1.DaemonSetUpdateStrategy{Type:"RollingUpdate", RollingUpdate:(*v1.RollingUpdateDaemonSet)(0xc0723899e0)}, MinReadySeconds:0, RevisionHistoryLimit:(*int32)(0xc083f5efa8)}, Status:v1.DaemonSetStatus{CurrentNumberScheduled:5, NumberMisscheduled:0, DesiredNumberScheduled:5, NumberReady:3, ObservedGeneration:20431, UpdatedNumberScheduled:1, NumberAvailable:3, NumberUnavailable:2, CollisionCount:(*int32)(nil), Conditions:[]v1.DaemonSetCondition(nil)}}: Operation cannot be fulfilled on daemonsets.apps "svclb-traefik-091f1f66": the object has been modified; please apply your changes to the latest version and try again
Sep 07 20:29:55 Master-Node k3s[2523409]: I0907 20:29:55.994522 2523409 event.go:294] "Event occurred" object="kube-system/svclb-traefik-091f1f66" fieldPath="" kind="DaemonSet" apiVersion="apps/v1" type="Normal" reason="SuccessfulDelete" message="Deleted pod: svclb-traefik-091f1f66-kwf72"
Sep 07 20:29:56 Master-Node k3s[2523409]: I0907 20:29:56.701289 2523409 event.go:294] "Event occurred" object="kube-system/traefik" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="UpdatedLoadBalancer" message="Updated LoadBalancer with new IPs: [1.1.1.1 2.2.2.2 3.3.3.3] -> [4.4.4.4 1.1.1.1 2.2.2.2 3.3.3.3]"
Sep 07 20:29:56 Master-Node k3s[2523409]: I0907 20:29:56.787188 2523409 event.go:294] "Event occurred" object="kube-system/traefik" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="EnsuringLoadBalancer" message="Ensuring load balancer"
Sep 07 20:29:56 Master-Node k3s[2523409]: I0907 20:29:56.854828 2523409 event.go:294] "Event occurred" object="kube-system/traefik" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="AppliedDaemonSet" message="Applied LoadBalancer DaemonSet kube-system/svclb-traefik-091f1f66"
Sep 07 20:29:57 Master-Node k3s[2523409]: I0907 20:29:57.676241 2523409 event.go:294] "Event occurred" object="kube-system/traefik" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="UpdatedLoadBalancer" message="Updated LoadBalancer with new IPs: [4.4.4.4 1.1.1.1 2.2.2.2 3.3.3.3] -> [1.1.1.1 2.2.2.2 3.3.3.3]"
Sep 07 20:29:57 Master-Node k3s[2523409]: I0907 20:29:57.817290 2523409 event.go:294] "Event occurred" object="kube-system/traefik" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="EnsuringLoadBalancer" message="Ensuring load balancer"
Sep 07 20:29:57 Master-Node k3s[2523409]: I0907 20:29:57.883677 2523409 event.go:294] "Event occurred" object="kube-system/svclb-traefik-091f1f66" fieldPath="" kind="DaemonSet" apiVersion="apps/v1" type="Normal" reason="SuccessfulCreate" message="Created pod: svclb-traefik-091f1f66-qcrbs"
Sep 07 20:29:57 Master-Node k3s[2523409]: I0907 20:29:57.911705 2523409 event.go:294] "Event occurred" object="kube-system/traefik" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="AppliedDaemonSet" message="Applied LoadBalancer DaemonSet kube-system/svclb-traefik-091f1f66"
Sep 07 20:29:58 Master-Node k3s[2523409]: I0907 20:29:58.301654 2523409 event.go:294] "Event occurred" object="kube-system/svclb-traefik-091f1f66" fieldPath="" kind="DaemonSet" apiVersion="apps/v1" type="Normal" reason="SuccessfulDelete" message="Deleted pod: svclb-traefik-091f1f66-qcrbs"
Sep 07 20:29:59 Master-Node k3s[2523409]: I0907 20:29:59.750817 2523409 event.go:294] "Event occurred" object="kube-system/traefik" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="UpdatedLoadBalancer" message="Updated LoadBalancer with new IPs: [1.1.1.1 2.2.2.2 3.3.3.3] -> [4.4.4.4 1.1.1.1 2.2.2.2 3.3.3.3]"
Sep 07 20:29:59 Master-Node k3s[2523409]: I0907 20:29:59.850517 2523409 event.go:294] "Event occurred" object="kube-system/traefik" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="EnsuringLoadBalancer" message="Ensuring load balancer"

[migs35323]

heres a video demo:
Screencast from 08-09-2023 12:50:15.webm

heres a watch for kubectl get nodes in a load balancer service with 4 ports and 5 nodes:

$ kubectl get pods -n kube-system -w
NAME                                      READY   STATUS    RESTARTS      AGE
local-path-provisioner-687d6d7765-hpjnq   1/1     Running   0             37d
coredns-74448699cf-h6b8v                  1/1     Running   0             17d
metrics-server-6b7fbb7969-kf7tm           1/1     Running   0             3d19h
metrics-server-6b7fbb7969-876db           1/1     Running   0             3d19h
traefik-j7fx5                             1/1     Running   0             22h
traefik-6ddmg                             1/1     Running   0             22h
traefik-sn25w                             1/1     Running   0             22h
traefik-4pprw                             1/1     Running   0             22h
svclb-mailu-front-ext-3f2caa0f-mq6rv      5/5     Running   5 (21h ago)   3d1h
traefik-p4p4d                             1/1     Running   0             22h
metrics-server-6b7fbb7969-qrkx9           1/1     Running   1 (21h ago)   3d19h
svclb-traefik-091f1f66-gt96l              2/2     Running   0             17h
svclb-traefik-091f1f66-2gt5m              2/2     Running   0             17h
svclb-traefik-091f1f66-fpkqj              2/2     Running   0             17h
svclb-traefik-091f1f66-z4bfp              2/2     Running   0             17h
svclb-traefik-091f1f66-js5js              2/2     Running   0             17h
svclb-some-service-eb51b554-fl2bv         0/4     Pending   0             0s
svclb-some-service-eb51b554-fl2bv         0/4     Pending   0             0s
svclb-some-service-eb51b554-7dcnw         0/4     Pending   0             0s
svclb-some-service-eb51b554-b9vjh         0/4     Pending   0             0s
svclb-some-service-eb51b554-7dcnw         0/4     Pending   0             0s
svclb-some-service-eb51b554-gxcnc         0/4     Pending   0             0s
svclb-some-service-eb51b554-b9vjh         0/4     Pending   0             0s
svclb-some-service-eb51b554-fl2bv         0/4     ContainerCreating   0             0s
svclb-some-service-eb51b554-bv7jg         0/4     Pending             0             0s
svclb-some-service-eb51b554-gxcnc         0/4     Pending             0             0s
svclb-some-service-eb51b554-bv7jg         0/4     Pending             0             0s
svclb-some-service-eb51b554-7dcnw         0/4     ContainerCreating   0             0s
svclb-some-service-eb51b554-bv7jg         0/4     ContainerCreating   0             1s
svclb-some-service-eb51b554-gxcnc         0/4     ContainerCreating   0             1s
svclb-some-service-eb51b554-b9vjh         0/4     ContainerCreating   0             1s
svclb-some-service-eb51b554-b9vjh         4/4     Running             0             2s
svclb-some-service-eb51b554-7dcnw         0/4     Terminating         0             2s
svclb-some-service-eb51b554-fl2bv         0/4     Terminating         0             2s
svclb-some-service-eb51b554-gxcnc         0/4     Terminating         0             2s
svclb-some-service-eb51b554-bv7jg         0/4     Terminating         0             2s
svclb-some-service-eb51b554-b9vjh         4/4     Terminating         0             2s
svclb-some-service-eb51b554-fl2bv         4/4     Terminating         0             2s
svclb-some-service-eb51b554-bv7jg         4/4     Terminating         0             3s
svclb-some-service-eb51b554-gxcnc         4/4     Terminating         0             4s
svclb-some-service-eb51b554-gxcnc         0/4     Terminating         0             4s
svclb-some-service-eb51b554-gxcnc         0/4     Terminating         0             4s
svclb-some-service-eb51b554-gxcnc         0/4     Terminating         0             4s
svclb-some-service-eb51b554-ghtc4         0/4     Pending             0             1s
svclb-some-service-eb51b554-ghtc4         0/4     Pending             0             1s
svclb-some-service-eb51b554-bv7jg         0/4     Terminating         0             5s
svclb-some-service-eb51b554-fl2bv         0/4     Terminating         0             5s
svclb-some-service-eb51b554-bv7jg         0/4     Terminating         0             5s
svclb-some-service-eb51b554-fl2bv         0/4     Terminating         0             5s
svclb-some-service-eb51b554-ghtc4         0/4     ContainerCreating   0             1s
svclb-some-service-eb51b554-b9vjh         0/4     Terminating         0             5s
svclb-some-service-eb51b554-bv7jg         0/4     Terminating         0             5s
svclb-some-service-eb51b554-ghtc4         0/4     Terminating         0             1s
svclb-some-service-eb51b554-fl2bv         0/4     Terminating         0             5s
svclb-some-service-eb51b554-b9vjh         0/4     Terminating         0             5s
svclb-some-service-eb51b554-b9vjh         0/4     Terminating         0             5s
svclb-some-service-eb51b554-79b28         0/4     Pending             0             0s
svclb-some-service-eb51b554-4x2sj         0/4     Pending             0             0s
svclb-some-service-eb51b554-79b28         0/4     Pending             0             0s
svclb-some-service-eb51b554-l5qlv         0/4     Pending             0             0s
svclb-some-service-eb51b554-4x2sj         0/4     Pending             0             0s
svclb-some-service-eb51b554-l5qlv         0/4     Pending             0             0s
svclb-some-service-eb51b554-79b28         0/4     ContainerCreating   0             0s
svclb-some-service-eb51b554-4x2sj         0/4     ContainerCreating   0             0s
svclb-some-service-eb51b554-79b28         0/4     Terminating         0             0s
svclb-some-service-eb51b554-l5qlv         0/4     ContainerCreating   0             0s
svclb-some-service-eb51b554-4x2sj         0/4     Terminating         0             0s
svclb-some-service-eb51b554-l5qlv         0/4     Terminating         0             1s
svclb-some-service-eb51b554-ghtc4         4/4     Terminating         0             2s
svclb-some-service-eb51b554-79b28         4/4     Terminating         0             2s
svclb-some-service-eb51b554-ghtc4         0/4     Terminating         0             3s
svclb-some-service-eb51b554-ghtc4         0/4     Terminating         0             4s
svclb-some-service-eb51b554-ghtc4         0/4     Terminating         0             4s
svclb-some-service-eb51b554-pljt8         0/4     Pending             0             0s
svclb-some-service-eb51b554-l5qlv         4/4     Terminating         0             3s
svclb-some-service-eb51b554-pljt8         0/4     Pending             0             0s
svclb-some-service-eb51b554-79b28         0/4     Terminating         0             3s
svclb-some-service-eb51b554-79b28         0/4     Terminating         0             3s
svclb-some-service-eb51b554-pljt8         0/4     ContainerCreating   0             0s
svclb-some-service-eb51b554-79b28         0/4     Terminating         0             3s
svclb-some-service-eb51b554-l7xf9         0/4     Pending             0             0s
svclb-some-service-eb51b554-l7xf9         0/4     Pending             0             0s
svclb-some-service-eb51b554-pljt8         0/4     Terminating         0             0s
svclb-some-service-eb51b554-l7xf9         0/4     ContainerCreating   0             0s
svclb-some-service-eb51b554-7dcnw         4/4     Terminating         0             8s
svclb-some-service-eb51b554-l7xf9         0/4     Terminating         0             0s
svclb-some-service-eb51b554-l5qlv         0/4     Terminating         0             4s
svclb-some-service-eb51b554-l5qlv         0/4     Terminating         0             4s
svclb-some-service-eb51b554-4x2sj         4/4     Terminating         0             4s
svclb-some-service-eb51b554-l5qlv         0/4     Terminating         0             4s
svclb-some-service-eb51b554-t4s2s         0/4     Pending             0             0s
svclb-some-service-eb51b554-t4s2s         0/4     Pending             0             0s
svclb-some-service-eb51b554-t4s2s         0/4     ContainerCreating   0             0s
svclb-some-service-eb51b554-t4s2s         0/4     Terminating         0             1s
svclb-some-service-eb51b554-l7xf9         4/4     Terminating         0             2s
svclb-some-service-eb51b554-pljt8         4/4     Terminating         0             3s
svclb-some-service-eb51b554-t4s2s         4/4     Terminating         0             2s
svclb-some-service-eb51b554-l7xf9         0/4     Terminating         0             3s
svclb-some-service-eb51b554-l7xf9         0/4     Terminating         0             3s
svclb-some-service-eb51b554-l7xf9         0/4     Terminating         0             3s
svclb-some-service-eb51b554-4x2sj         0/4     Terminating         0             6s
svclb-some-service-eb51b554-4x2sj         0/4     Terminating         0             6s
svclb-some-service-eb51b554-lfdqn         0/4     Pending             0             0s
svclb-some-service-eb51b554-4x2sj         0/4     Terminating         0             6s
svclb-some-service-eb51b554-lfdqn         0/4     Pending             0             0s
svclb-some-service-eb51b554-72zvn         0/4     Pending             0             0s
svclb-some-service-eb51b554-lfdqn         0/4     ContainerCreating   0             0s
svclb-some-service-eb51b554-72zvn         0/4     Pending             0             0s
svclb-some-service-eb51b554-72zvn         0/4     ContainerCreating   0             0s
svclb-some-service-eb51b554-lfdqn         0/4     Terminating         0             0s
svclb-some-service-eb51b554-72zvn         0/4     Terminating         0             0s
svclb-some-service-eb51b554-t4s2s         0/4     Terminating         0             3s
svclb-some-service-eb51b554-t4s2s         0/4     Terminating         0             3s
svclb-some-service-eb51b554-t4s2s         0/4     Terminating         0             3s
svclb-some-service-eb51b554-csmrq         0/4     Pending             0             0s
svclb-some-service-eb51b554-csmrq         0/4     Pending             0             0s
svclb-some-service-eb51b554-csmrq         0/4     ContainerCreating   0             0s
svclb-some-service-eb51b554-csmrq         0/4     Terminating         0             0s
svclb-some-service-eb51b554-pljt8         0/4     Terminating         0             5s
svclb-some-service-eb51b554-pljt8         0/4     Terminating         0             5s
svclb-some-service-eb51b554-pljt8         0/4     Terminating         0             5s
svclb-some-service-eb51b554-wmcp4         0/4     Pending             0             0s
svclb-some-service-eb51b554-lfdqn         4/4     Terminating         0             2s
svclb-some-service-eb51b554-wmcp4         0/4     Pending             0             0s
svclb-some-service-eb51b554-wmcp4         0/4     ContainerCreating   0             0s
svclb-some-service-eb51b554-wmcp4         0/4     Terminating         0             0s
svclb-some-service-eb51b554-7dcnw         0/4     Terminating         0             14s
svclb-some-service-eb51b554-csmrq         4/4     Terminating         0             2s
svclb-some-service-eb51b554-7dcnw         0/4     Terminating         0             14s
svclb-some-service-eb51b554-7dcnw         0/4     Terminating         0             14s
svclb-some-service-eb51b554-wjpn2         0/4     Pending             0             1s
svclb-some-service-eb51b554-wjpn2         0/4     Pending             0             1s
svclb-some-service-eb51b554-wjpn2         0/4     Terminating         0             1s
svclb-some-service-eb51b554-csmrq         0/4     Terminating         0             3s
svclb-some-service-eb51b554-wjpn2         0/4     Terminating         0             1s
svclb-some-service-eb51b554-csmrq         0/4     Terminating         0             3s
svclb-some-service-eb51b554-csmrq         0/4     Terminating         0             3s
svclb-some-service-eb51b554-gnjqh         0/4     Pending             0             0s
svclb-some-service-eb51b554-gnjqh         0/4     Pending             0             0s
svclb-some-service-eb51b554-lfdqn         0/4     Terminating         0             4s
svclb-some-service-eb51b554-lfdqn         0/4     Terminating         0             4s
svclb-some-service-eb51b554-lfdqn         0/4     Terminating         0             4s
svclb-some-service-eb51b554-gnjqh         0/4     Terminating         0             0s
svclb-some-service-eb51b554-gnjqh         0/4     Terminating         0             0s
svclb-some-service-eb51b554-tpgkn         0/4     Pending             0             0s
svclb-some-service-eb51b554-tpgkn         0/4     Pending             0             0s
svclb-some-service-eb51b554-tpgkn         0/4     ContainerCreating   0             0s
svclb-some-service-eb51b554-wmcp4         4/4     Terminating         0             3s
svclb-some-service-eb51b554-tpgkn         0/4     Terminating         0             1s
svclb-some-service-eb51b554-72zvn         4/4     Terminating         0             5s
svclb-some-service-eb51b554-wmcp4         0/4     Terminating         0             4s
svclb-some-service-eb51b554-wmcp4         0/4     Terminating         0             4s
svclb-some-service-eb51b554-wmcp4         0/4     Terminating         0             4s
svclb-some-service-eb51b554-hqzz2         0/4     Pending             0             0s
svclb-some-service-eb51b554-hqzz2         0/4     Pending             0             0s
svclb-some-service-eb51b554-hqzz2         0/4     ContainerCreating   0             0s
svclb-some-service-eb51b554-hqzz2         0/4     Terminating         0             0s
svclb-some-service-eb51b554-tpgkn         4/4     Terminating         0             2s
svclb-some-service-eb51b554-gnjqh         4/4     Terminating         0             3s
svclb-some-service-eb51b554-72zvn         0/4     Terminating         0             7s
svclb-some-service-eb51b554-72zvn         0/4     Terminating         0             7s
svclb-some-service-eb51b554-72zvn         0/4     Terminating         0             7s
svclb-some-service-eb51b554-tpgkn         0/4     Terminating         0             3s
svclb-some-service-eb51b554-f5gdf         0/4     Pending             0             0s
svclb-some-service-eb51b554-tpgkn         0/4     Terminating         0             3s
svclb-some-service-eb51b554-tpgkn         0/4     Terminating         0             3s
svclb-some-service-eb51b554-f5gdf         0/4     Pending             0             0s
svclb-some-service-eb51b554-xpf62         0/4     Pending             0             0s
svclb-some-service-eb51b554-f5gdf         0/4     ContainerCreating   0             0s
svclb-some-service-eb51b554-xpf62         0/4     Pending             0             0s
svclb-some-service-eb51b554-xpf62         0/4     ContainerCreating   0             0s
svclb-some-service-eb51b554-gnjqh         0/4     Terminating         0             4s
svclb-some-service-eb51b554-gnjqh         0/4     Terminating         0             4s
svclb-some-service-eb51b554-gnjqh         0/4     Terminating         0             4s
svclb-some-service-eb51b554-ff9nd         0/4     Pending             0             0s
svclb-some-service-eb51b554-ff9nd         0/4     Pending             0             0s
svclb-some-service-eb51b554-f5gdf         0/4     Terminating         0             1s
svclb-some-service-eb51b554-xpf62         0/4     Terminating         0             1s
svclb-some-service-eb51b554-ff9nd         0/4     ContainerCreating   0             0s
svclb-some-service-eb51b554-hqzz2         4/4     Terminating         0             3s
svclb-some-service-eb51b554-ff9nd         0/4     Terminating         0             1s
svclb-some-service-eb51b554-xpf62         4/4     Terminating         0             2s
svclb-some-service-eb51b554-ff9nd         4/4     Terminating         0             2s
svclb-some-service-eb51b554-xpf62         0/4     Terminating         0             3s
svclb-some-service-eb51b554-xpf62         0/4     Terminating         0             3s
svclb-some-service-eb51b554-xpf62         0/4     Terminating         0             3s
svclb-some-service-eb51b554-h68nn         0/4     Pending             0             0s
svclb-some-service-eb51b554-h68nn         0/4     Pending             0             0s
svclb-some-service-eb51b554-h68nn         0/4     ContainerCreating   0             0s
svclb-some-service-eb51b554-hqzz2         0/4     Terminating         0             5s
svclb-some-service-eb51b554-hqzz2         0/4     Terminating         0             5s
svclb-some-service-eb51b554-hqzz2         0/4     Terminating         0             5s
svclb-some-service-eb51b554-ff9nd         0/4     Terminating         0             3s
svclb-some-service-eb51b554-pw84t         0/4     Pending             0             0s
svclb-some-service-eb51b554-ff9nd         0/4     Terminating         0             3s
svclb-some-service-eb51b554-ff9nd         0/4     Terminating         0             3s
svclb-some-service-eb51b554-pw84t         0/4     Pending             0             0s
svclb-some-service-eb51b554-pw84t         0/4     ContainerCreating   0             0s
svclb-some-service-eb51b554-vwz5c         0/4     Pending             0             0s
svclb-some-service-eb51b554-vwz5c         0/4     Pending             0             0s
svclb-some-service-eb51b554-pw84t         0/4     Terminating         0             0s
svclb-some-service-eb51b554-h68nn         0/4     Terminating         0             1s
svclb-some-service-eb51b554-vwz5c         0/4     ContainerCreating   0             0s
svclb-some-service-eb51b554-vwz5c         0/4     Terminating         0             1s
svclb-some-service-eb51b554-h68nn         4/4     Terminating         0             2s
svclb-some-service-eb51b554-pw84t         4/4     Terminating         0             2s
svclb-some-service-eb51b554-vwz5c         4/4     Terminating         0             2s
svclb-some-service-eb51b554-h68nn         0/4     Terminating         0             3s
svclb-some-service-eb51b554-h68nn         0/4     Terminating         0             3s
svclb-some-service-eb51b554-h68nn         0/4     Terminating         0             3s
svclb-some-service-eb51b554-m2w2g         0/4     Pending             0             0s
svclb-some-service-eb51b554-m2w2g         0/4     Pending             0             0s
svclb-some-service-eb51b554-m2w2g         0/4     ContainerCreating   0             0s
svclb-some-service-eb51b554-vwz5c         0/4     Terminating         0             3s
svclb-some-service-eb51b554-vwz5c         0/4     Terminating         0             3s
svclb-some-service-eb51b554-m2w2g         0/4     Terminating         0             1s
svclb-some-service-eb51b554-vwz5c         0/4     Terminating         0             3s
svclb-some-service-eb51b554-kqm6f         0/4     Pending             0             0s
svclb-some-service-eb51b554-kqm6f         0/4     Pending             0             1s
svclb-some-service-eb51b554-kqm6f         0/4     ContainerCreating   0             1s
svclb-some-service-eb51b554-pw84t         0/4     Terminating         0             4s
svclb-some-service-eb51b554-pw84t         0/4     Terminating         0             4s
svclb-some-service-eb51b554-pw84t         0/4     Terminating         0             4s
svclb-some-service-eb51b554-kfhfh         0/4     Pending             0             0s
svclb-some-service-eb51b554-kfhfh         0/4     Pending             0             0s
svclb-some-service-eb51b554-kqm6f         0/4     Terminating         0             1s
svclb-some-service-eb51b554-kfhfh         0/4     ContainerCreating   0             1s
svclb-some-service-eb51b554-f5gdf         4/4     Terminating         0             9s
svclb-some-service-eb51b554-kfhfh         0/4     Terminating         0             1s
svclb-some-service-eb51b554-m2w2g         4/4     Terminating         0             3s
svclb-some-service-eb51b554-kqm6f         4/4     Terminating         0             3s
svclb-some-service-eb51b554-wjpn2         4/4     Terminating         0             14s
svclb-some-service-eb51b554-m2w2g         0/4     Terminating         0             4s
svclb-some-service-eb51b554-m2w2g         0/4     Terminating         0             4s
svclb-some-service-eb51b554-m2w2g         0/4     Terminating         0             4s
svclb-some-service-eb51b554-mpc6b         0/4     Pending             0             0s
svclb-some-service-eb51b554-mpc6b         0/4     Pending             0             0s
svclb-some-service-eb51b554-mpc6b         0/4     ContainerCreating   0             1s
svclb-some-service-eb51b554-kfhfh         4/4     Terminating         0             3s
svclb-some-service-eb51b554-kqm6f         0/4     Terminating         0             4s
svclb-some-service-eb51b554-mpc6b         0/4     Terminating         0             1s
svclb-some-service-eb51b554-kqm6f         0/4     Terminating         0             4s
svclb-some-service-eb51b554-kqm6f         0/4     Terminating         0             4s
svclb-some-service-eb51b554-54cj9         0/4     Pending             0             0s
svclb-some-service-eb51b554-54cj9         0/4     Pending             0             0s
svclb-some-service-eb51b554-54cj9         0/4     ContainerCreating   0             0s
svclb-some-service-eb51b554-kfhfh         0/4     Terminating         0             4s
svclb-some-service-eb51b554-kfhfh         0/4     Terminating         0             4s
svclb-some-service-eb51b554-kfhfh         0/4     Terminating         0             4s
svclb-some-service-eb51b554-5g9d8         0/4     Pending             0             0s
svclb-some-service-eb51b554-5g9d8         0/4     Pending             0             0s

as i said earlier this happens for a few hours, every time there is some change, either a node down or a change in the load balancer service

[brandond]

You're not really providing enough information to determine why this is occurring. Is the daemonset getting updated, or is something else terminating the pods and causing the daemonset controller to recreate them? Are the node labels flapping? Are the service endpoints changing? I would probably look at the events for one of the terminated pods to see why it was killed, as well as monitor the endpoints for your service with kubectl get endpoints -n some-namespace some-service -w. Monitor the servicelb DaemonSet to see what's changing. You might also look at the k3s service logs to see if events or log messages from the controller provide any clues as to what might be changing.