k8s 1.17 kubelet doesn't start on coreos

[cann0nf0dder]

What happened:
After upgrade to kubernetes 1.17.5 kubelet does not start
I couldn't find documentation about change to kubelet startup in the release notes.
Found few people running into the same issue on slack. No solution found.

What you expected to happen:
Kubelet starts fine same as on 1.16 branch.

How to reproduce it (as minimally and precisely as possible):
Upgrade to 1.17.5 on CoreOS

Anything else we need to know?:
Workaround provided below to share my finding with the community

Standard CoreOS kubelet.service

[Unit]
Description=kubelet
Wants=rpc-statd.service

[Service]
User=root
EnvironmentFile=/etc/kubernetes/kubelet.env
Environment="RKT_RUN_ARGS=--uuid-file-save=/var/cache/kubelet-pod.uuid \
  --mount volume=resolv,target=/etc/resolv.conf \
  --mount volume=etc-cni-net,target=/etc/cni/net.d \
  --mount volume=var-lib-cni,target=/var/lib/cni \
  --mount volume=opt-cni-bin,target=/opt/cni/bin \
  --mount volume=var-log,target=/var/log \
  --mount volume=root-docker,target=/root/.docker \
  --mount volume=etc-k8s-cfg,target=/etc/kubernetes/config \
  --mount volume=var-lib-calico,target=/var/lib/calico \
  --volume var-lib-calico,kind=host,source=/var/lib/calico \
  --volume resolv,kind=host,source=/etc/resolv.conf \
  --volume etc-cni-net,kind=host,source=/etc/cni/net.d \
  --volume var-lib-cni,kind=host,source=/var/lib/cni \
  --volume opt-cni-bin,kind=host,source=/opt/cni/bin \
  --volume var-log,kind=host,source=/var/log \
  --volume root-docker,kind=host,source=/root/.docker \
  --volume etc-k8s-cfg,kind=host,source=/etc/kubernetes/config \
  --insecure-options=image"

ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
ExecStartPre=/bin/mkdir -p /etc/kubernetes/pki
ExecStartPre=/bin/mkdir -p /opt/cni/bin
ExecStartPre=/bin/mkdir -p /var/lib/cni
ExecStartPre=/bin/mkdir -p /etc/cni/net.d
ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/pki/ca.crt"
ExecStartPre=-/usr/bin/rkt rm --uuid-file=/var/cache/kubelet-pod.uuid
ExecStart=/usr/lib/coreos/kubelet-wrapper \
--config=/etc/kubernetes/config/kubelet.yaml \
--cni-conf-dir=/etc/cni/net.d \
--exit-on-lock-contention \
--kubeconfig=/etc/kubernetes/kubeconfig \
--lock-file=/var/run/lock/kubelet.lock \
--network-plugin=cni

ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid

After upgrading to 1.17.5 kubelet doesn't start with the following logs:

Apr 21 08:21:38 nodename kubelet-wrapper[1841]: + exec /usr/bin/rkt run --uuid-file-save=/var/cache/kubelet-pod.uuid --mount volume=resolv,target=/etc/resolv.conf --mount volume=etc-cni-net,target=/etc/cni/net.d --mount>
Apr 21 08:21:40 nodename kubelet-wrapper[1841]: --config=/etc/kubernetes/config/kubelet.yaml: command not supported
Apr 21 09:03:13 nodename kubelet-wrapper[971]: Usage:
Apr 21 09:03:13 nodename kubelet-wrapper[971]:   kubelet [command]
Apr 21 09:03:13 nodename kubelet-wrapper[971]: Available Commands:
Apr 21 09:03:13 nodename kubelet-wrapper[971]:   help                     Help about any command
Apr 21 09:03:13 nodename kubelet-wrapper[971]:   kube-apiserver
Apr 21 09:03:13 nodename kubelet-wrapper[971]:   kube-controller-manager
Apr 21 09:03:13 nodename kubelet-wrapper[971]:   kube-proxy
Apr 21 09:03:13 nodename kubelet-wrapper[971]:   kube-scheduler
Apr 21 09:03:13 nodename kubelet-wrapper[971]:   kubectl                  kubectl controls the Kubernetes cluster manager
Apr 21 09:03:13 nodename kubelet-wrapper[971]:   kubelet

I've noticed that the coreos specific kubelet-wrapper expects one of the following commands before the parameters:
kubelet, kube-apiserver, kube-controller-manager, kube-proxy, kubelet
I've gone ahead and added kubelet to first line under the kubelet-wrapper and I was able to start the kubelet and make first api-server upgrade successful.

Workaround kubelet.service config:

ExecStartPre=-/usr/bin/rkt rm --uuid-file=/var/cache/kubelet-pod.uuid
ExecStart=/usr/lib/coreos/kubelet-wrapper \
kubelet --config=/etc/kubernetes/config/kubelet.yaml \
--cni-conf-dir=/etc/cni/net.d \
--exit-on-lock-contention \
--kubeconfig=/etc/kubernetes/kubeconfig \
--lock-file=/var/run/lock/kubelet.lock \
--network-plugin=cni

Don't know if this is a lack of documentation on the recent kubelet change in k8s or coreos kubelet-wrapper specific issue, I thought I'll share it here for comments/thoughts.

Environment:

• Kubernetes version (use kubectl version): 1.16.6->1.17.5 upgrade
• Cloud provider or hardware configuration: CoreOS
• OS (e.g: cat /etc/os-release):
  cat /etc/os-release
  NAME=Fedora
  VERSION="28 (Twenty Eight)"
  ID=fedora
  VERSION_ID=28
  PLATFORM_ID="platform:f28"
  PRETTY_NAME="Fedora 28 (Twenty Eight)"
  ANSI_COLOR="0;34"
  CPE_NAME="cpe:/o:fedoraproject:fedora:28"
  HOME_URL="https://fedoraproject.org/"
  SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
  BUG_REPORT_URL="https://bugzilla.redhat.com/"
  REDHAT_BUGZILLA_PRODUCT="Fedora"
  REDHAT_BUGZILLA_PRODUCT_VERSION=28
  REDHAT_SUPPORT_PRODUCT="Fedora"
  REDHAT_SUPPORT_PRODUCT_VERSION=28
  PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
• Kernel (e.g. uname -a):
  Linux 4.19.86-coreos Unit test coverage in Kubelet is lousy. (~30%) #1 SMP Mon Dec 2 20:13:38 -00 2019 x86_64 x86_64 x86_64 GNU/Linux

[athenabot]

/sig node
/sig network

These SIGs are my best guesses for this issue. Please comment /remove-sig <name> if I am incorrect about one.

🤖 I am a bot run by vllry. 👩‍🔬

[cann0nf0dder]

/remove-sig network

[k8s-ci-robot]

@cann0nf0dder: Those labels are not set on the issue: sig/sig/network

In response to this:

/remove-sig sig/network

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[liggitt]

It appears the kubelet-wrapper script is using hyperkube, which was deprecated in 1.17 (xref #84662).

Additionally, it appears the kubelet-wrapper script is explicitly unsupported:

This repo is not in alignment with current versions of Kubernetes, and will not be active in the future.

[liggitt]

I think the coreos/coreos-kubernetes#930 issue is probably the right place to track this