ghost pod

[gfrankliu]

We've been running k3s on this Dell amd64 server for the last couple of years.
We have this one ghost pod and not sure how to delete it. We have upgraded k3s several times but that has no impact on this ghost pod. I'm not sure how that pod got into that state. Any idea how we can delete it?

$ kubectl get pod | grep feeding-service-0 
feeding-service-0                      1/1     Terminating   0             580d
feeding-service-0                      1/1     Running       0             92m
$ kubectl get sts | grep feeding-service
feeding-service                   1/1     294d
$

[brandond]

You could try force-deleting it using kubectl, or checking to see if there is still a container running using ctr and delete it if so.

[brandond]

I assumed you'd redacted something and introduced a typo; it happens a lot. It shouldn't event be possible to have two pods with the same name. Resources are stored by type, namespace, and name in the datastore, so they'd end up in the same key - and you can't have more than one entry per key. Can you show the output of kubectl get pod -o yaml for these pods?

Its possible there is some corruption in your datastore but again, not even sure how you've got multiple things in the same key.

[gfrankliu]

That's literal, not redacted :)

I ran kubectl get pod -o yaml and can see both pods in the output:

For the ghost pod:

apiVersion: v1
items:
- apiVersion: v1
  kind: Pod
  metadata:
    annotations:
      kubectl.kubernetes.io/restartedAt: "2023-04-21T18:27:48Z"
    creationTimestamp: "2023-04-21T18:27:49Z"
    deletionGracePeriodSeconds: 300
    deletionTimestamp: "2023-04-21T18:34:47Z"
    generateName: feeding-service-
    labels:
      app: feeding-service
      controller-revision-hash: feeding-service-cb6469d7d
      statefulset.kubernetes.io/pod-name: feeding-service-0
    name: feeding-service-0
    namespace: default
    ownerReferences:
    - apiVersion: apps/v1
      blockOwnerDeletion: true
      controller: true
      kind: StatefulSet
      name: feeding-service
      uid: e1ba77ad-f997-4afb-92e7-1099222d2064
    resourceVersion: "8946981"
    uid: 68c71f74-eb73-439b-a81f-bbce1fada68d
  spec: 
    containers:
...
...
  status:
    conditions:
    - lastProbeTime: null
      lastTransitionTime: "2023-04-21T18:27:49Z"
      status: "True"
      type: Initialized
    - lastProbeTime: null
      lastTransitionTime: "2023-04-21T18:27:56Z"
      status: "True"
      type: Ready
    - lastProbeTime: null
      lastTransitionTime: "2023-04-21T18:27:56Z"
      status: "True"
      type: ContainersReady
    - lastProbeTime: null
      lastTransitionTime: "2023-04-21T18:27:49Z"
      status: "True"
      type: PodScheduled
    containerStatuses:
    - containerID: containerd://1f6dbed1d5472deada0c264b555be9ab22fc92e05d6d2b42b42e62ec136d0ddc
      image: <redacted>
      imageID: <redacted>
      lastState: {}
      name: feeding-service
      ready: true
      restartCount: 0
      started: true
      state:
        running:
          startedAt: "2023-04-21T18:27:55Z"
    hostIP: 192.168.6.115
    phase: Running
    podIP: 7.200.0.4
    podIPs:
    - ip: 7.200.0.4
    qosClass: Burstable
    startTime: "2023-04-21T18:27:49Z"

The containerID 1f6dbed1d5472deada0c264b555be9ab22fc92e05d6d2b42b42e62ec136d0ddc doesn't seem to exist when I run sudo k3s ctr container ls | grep 1f6dbed1d5472deada0c264b555be9ab22fc92e05d6d2b42b42e62ec136d0ddc

Now, for the good pod:

- apiVersion: v1
  kind: Pod
  metadata:
    annotations:
      kubectl.kubernetes.io/restartedAt: "2024-11-21T17:08:08Z"
    creationTimestamp: "2024-11-21T17:08:13Z"
    generateName: feeding-service-
    labels:
      app: feeding-service
      apps.kubernetes.io/pod-index: "0"
      controller-revision-hash: feeding-service-967775967
      statefulset.kubernetes.io/pod-name: feeding-service-0
    name: feeding-service-0
    namespace: default
    ownerReferences:
    - apiVersion: apps/v1
      blockOwnerDeletion: true
      controller: true
      kind: StatefulSet
      name: feeding-service
      uid: 9c2c6892-d444-49a0-936f-91dc9161e8dc
    resourceVersion: "32130347"
    uid: 34955958-8c32-41a1-817f-2faae9fc3dcc
  spec:
    containers:
...
...
  status:
    conditions:
    - lastProbeTime: null
      lastTransitionTime: "2024-11-21T17:08:14Z"
      status: "True"
      type: PodReadyToStartContainers
    - lastProbeTime: null
      lastTransitionTime: "2024-11-21T17:08:13Z"
      status: "True"
      type: Initialized
    - lastProbeTime: null
      lastTransitionTime: "2024-11-21T17:08:14Z"
      status: "True"
      type: Ready
    - lastProbeTime: null
      lastTransitionTime: "2024-11-21T17:08:14Z"
      status: "True"
      type: ContainersReady
    - lastProbeTime: null
      lastTransitionTime: "2024-11-21T17:08:13Z"
      status: "True"
      type: PodScheduled
    containerStatuses:
    - containerID: containerd://48a299da82dffd5fa2ce1b48425b7019a934afd6e44e8777cbe5306bdbb437c9
      image: <redacted>
      imageID: <redacted>
      lastState: {}
      name: feeding-service
      ready: true
      restartCount: 0
      started: true
      state:
        running:
          startedAt: "2024-11-21T17:08:14Z"
    hostIP: 192.168.6.56
    hostIPs:
    - ip: 192.168.6.56
    phase: Running
    podIP: 7.200.0.37
    podIPs:
    - ip: 7.200.0.37
    qosClass: Burstable
    startTime: "2024-11-21T17:08:13Z"

This containerID 48a299da82dffd5fa2ce1b48425b7019a934afd6e44e8777cbe5306bdbb437c9 indeed exists:

$ sudo k3s ctr container ls | grep 48a299da82dffd5fa2ce1b48425b7019a934afd6e44e8777cbe5306bdbb437c9
48a299da82dffd5fa2ce1b48425b7019a934afd6e44e8777cbe5306bdbb437c9    <image-url-redacted>         io.containerd.runc.v2   

[brandond]

What version of K3s are you running? What are you using for your datastore? I expect you're using Kine? Are there any errors in your logs regarding compact failing? That pod is pretty old, whatever happened it probably happened quite a while ago, and you may need to go into the DB and manually delete the row with resourceVersion: "8946981"

[gfrankliu]

No, I don't see any compact failing errors in the syslog.

The cluster was created and the ghost pod have been there for a long time. k3s version has been upgraded a few time since then, so I guess the version doesn't really matter. Today, the version running on this server is v1.30.1+k3s1

The single node cluster was initially created with command:

curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644 --disable-network-policy --flannel-backend=host-gw --cluster-cidr=7.200.0.0/16 --service-cidr=7.201.0.0/16 --disable traefik

How do I manually manipulate the db?

[brandond]

You'd need to grab the sqlite binaries, either from the project website or from your distro packages. You can then stop K3s, and run sqlite /var/lib/rancher/k3s/server/db/state.db. At that point you can use standard sql commands to query or delete rows.

You might take a backup of the state.db file before modifying it. It shouldn't be terribly large.

[gfrankliu]

I ran delete from kine where id = 8946981; and now kubectl get pod no longer shows the ghost pod. Thanks!