kubeadm init should not set --allow-privileges=true in the default generated kube-apiserver.yaml manifest

[brianpursley]

Is this a BUG REPORT or FEATURE REQUEST?

FEATURE REQUEST

Versions

kubeadm version (use kubeadm version):

kubeadm version: &version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.0-beta.2-78-g1112abd470a5ba.dev-1592428154-dirty", GitCommit:"1112abd470a5ba787d463931c14902caee992c4a", GitTreeState:"dirty", BuildDate:"2020-06-17T21:10:24Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.0-beta.2.318+656a01fb044d6d-dirty", GitCommit:"656a01fb044d6d45a9b5fc7ceb06fc0b77018407", GitTreeState:"dirty", BuildDate:"2020-06-22T13:35:42Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.0-beta.2-78-g1112abd470a5ba.dev-1592428154-dirty", GitCommit:"1112abd470a5ba787d463931c14902caee992c4a", GitTreeState:"dirty", BuildDate:"2020-06-17T21:09:46Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration: on-prem cluster
• OS (e.g. from /etc/os-release):

NAME="Ubuntu"
VERSION="20.04 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

• Kernel (e.g. uname -a):

Linux k8s-master 5.4.0-33-generic #37-Ubuntu SMP Thu May 21 12:53:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

• Others:

What happened?

I created a cluster using kubeadm init and it set --allow-privileged=true when based on the documentation, I thought that my cluster would not allow privileged by default.

What you expected to happen?

According to the documentation, the default value for --allow-privileged flag of apiserver is false.

But when you use kubeadm to initialize your cluster, it explicitly sets --allow-privileged=true in the generated kube-apiserver.yaml.

People should check and be aware of the settings in the configs of their own cluster, but this is a little surprising given that the documented default is false.

I am proposing that kubeadm should choose the more secure default (false) and require people to explicitly enable --allow-privileged on their cluster if they want it.

How to reproduce it (as minimally and precisely as possible)?

kubeadm init
cat /etc/kubernetes/manifests/kube-apiserver.yaml

Anything else we need to know?

I made an initial (simple) attempt at this with kubernetes/kubernetes#92381, but soon discovered that kind depends on kubeadm init setting --allow-privileged=true, which is needed for kind to work. Please be aware that changing this might require some coordination with the kind repo to make sure it is still able to init a cluster with allow-privileges=true.

Maybe a flag could be added to kubeadm init that would like kind (or others) request this setting if they really want it, but it should be an opt-in thing.

[neolit123]

/priority awaiting-more-evidence
/kind feature

@brianpursley as mentioned on slack, i'd like to understand all the side-effects of --allow-privileged=false

as can be seen here:
https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/phases/addons/proxy/manifests.go#L82-L83

kubeadm deploys the kube-proxy containers as priviliged. does this mean that kube-proxy will no longer work with kubeadm?

also how about the static pods for the rest of the control-plane components?

[brianpursley]

OK. From what I know, this controls whether or not you are allowed to use securityContext.privileged: true in pod manifests.

I don't know if it has any effects internally. If changing this setting breaks the system, then I think maybe I will end up creating a different issue to remove the setting from apiserver 😁

[neolit123]

OK. From what I know, this controls whether or not you are allowed to use securityContext.privileged: true in pod manifests.

yes, my understanding as well. which means that toggling this to false, breaks the kubeadm control-plane and kube-proxy deployment.

I don't know if it has any effects internally. If changing this setting breaks the system, then I think maybe I will end up creating a different issue to remove the setting from apiserver

if someone can manage to get their cluster to run with this option set to =false that is fine, but it seems to me kubeadm components still need it.

here we explored the complexity of not running the control-plane as priviliged and we faced a number of blockers:
#1367

also not sure how we can solve this for kube-proxy anytime soon, given the demands it has.

[brianpursley]

OK I see what you're saying. when I look at one of my kube proxies, it has this in it:

    securityContext:
      privileged: true

That does seem to make it more complicated to turn off.

[neolit123]

@brianpursley i'm going to close this because we don't have a clear proposal in place.
today users might be able to hack around it, by e.g. skipping kube-proxy, but this is is out of scope for kubeadm for the time being (i.e. skipping kube-proxy by default)

thanks.

[brianpursley]

@neolit123 no problem, I understand. Thanks for explaining to me about kube-proxy