Add useful information for tokens #357
Conversation
Signed-off-by: manuelbuil <[email protected]>
| ::: | ||
|
|
||
| :::warning | ||
| The new server token is not written in `/var/lib/rancher/k3s/server/token` |
There was a problem hiding this comment.
Why is this the case? Couldn't the new token be appended to the /var/lib/rancher/k3s/server/token file? If this was for security reasons we could state that here likely.
There was a problem hiding this comment.
Yeah this is not the intended result. The server should be writing to that file https://github.com/k3s-io/k3s/blob/master/pkg/server/token.go#L89-L92
There was a problem hiding this comment.
Sorry, I thought I had tested it but now I realize this is wrong :(. Thanks for catching that up
| @@ -161,3 +161,10 @@ If you do not specify a new token, one will be generated for you. | |||
| `--token` value | Existing token used to join a server or agent to a cluster [$K3S_TOKEN] | |||
| `--new-token` value | New token that replaces existing token | |||
|
|
|||
| :::warning | |||
| Data encrypted before the rotation will need the old server token to decrypt | |||
There was a problem hiding this comment.
We attempt to migrate all data that depends on the token with https://github.com/k3s-io/k3s/blob/master/pkg/cluster/storage.go#L304. What data is not being reencrypted?
There was a problem hiding this comment.
Yeah I don't think this is accurate. Changing the server token does NOT change the encryption config, so the same key is used to encrypt/decrypt the secrets.
There was a problem hiding this comment.
Ok, I got confused by a comment. The only encrypted object which will need the old server token are old snapshots then, right? I wanted to add this warning because when executing this command, you get a warning with something like keep the old token but perhaps it is too late once you read that warning? The token gets replaced a few seconds after that
There was a problem hiding this comment.
The snapshots themselves are not encrypted either. Any snapshots taken with the old token value, will still have the bootstrap data IN the snapshot encrypted with the old token, so you'd need to use the old token when restoring it.
There was a problem hiding this comment.
Thanks Brad, for some reason I was misunderstanding this. I thought the snapshot was also encrypted 🤦
There was a problem hiding this comment.
One extra related thing. The current documentation says:
The server token is also used as the PBKDF2 passphrase for the key used to encrypt confidential information that is persisted to the datastore, such as the secrets-encryption configuration, wireguard keys, and private keys for cluster CA certificates and service-account tokens.
- secrets-encryption configuration is correct
- wireguard keys is probably outdated and refers to the old wireguard backend, right? The current wireguard key is generated by flannel and stored in plain text in "/run/flannel/wgkey" (which is probably something we should change)
- private keys for cluster CA certificates When I cat the private keys in
/var/lib/rancher/k3s/server/tls/, I don't see any being encrypted (e.g. client-ca.key or server-ca.key), so I suspect this statement is wrong. Or is there a config to enable this? I can't find it - service-account tokens IIUC, to have a token listed under the service account, you need to create a secret manually and associate it with the service account. In that case, this will fall into the "secrets-encryption" category, or?
There was a problem hiding this comment.
Its kinda complicated. This struct holds paths to files on disk:
https://github.com/k3s-io/k3s/blob/cd4ddedbc9782cbe9b5dcc411df2addae7b2f3b4/pkg/daemons/config/types.go#L291-L307
All the paths listed in that struct are read from disk and stuffed into a map, where the key is the field name and the value is the content of file at that path on disk along with a modification timestamp. This is then serialized to JSON, encrypted with the key, and stuffed into the bootstrap key in the datastore.
You can find the code for translating from ControlRuntimeBootstrap to map[string]File here:
https://github.com/k3s-io/k3s/blob/cd4ddedbc9782cbe9b5dcc411df2addae7b2f3b4/pkg/bootstrap/bootstrap.go#L23-L54
| @@ -51,7 +51,7 @@ Bootstrap | `n/a` | `n/a` | |||
|
|
|||
| If no token is provided when starting the first server in the cluster, one is created with a random password. The server token is always written to `/var/lib/rancher/k3s/server/token`, in secure format. | |||
|
|
|||
| The server token can be used to join both server and agent nodes to the cluster. It cannot be changed once the cluster has been created, and anyone with access to the server token essentially has full administrator access to the cluster. This token should be guarded carefully. | |||
| The server token can be used to join both server and agent nodes to the cluster. Unless manually rotated with [`k3s token rotate`](#k3s-token-rotate), the server token does not change and anyone with access to the server token essentially has full administrator access to the cluster. This token should be guarded carefully. | |||
There was a problem hiding this comment.
I don't think this belongs here. The fact that it can be rotated does not alter the fact that anyone who has access to it has full admin access. If you want to call out that it can/should be rotated if compromised, do so in a separate paragraph or sentence.
There was a problem hiding this comment.
My intention was to change the sentence: It cannot be changed once the cluster has been created. This is not accurate because it surely can be changed with k3s token rotate or?. I can try to find some better wording
There was a problem hiding this comment.
Lets just take that out, and start the second sentence with "Anyone with access to the server token...". We can get into rotation later.
| @@ -74,7 +74,7 @@ The agent token is written to `/var/lib/rancher/k3s/server/agent-token`, in secu | |||
| Support for the `k3s token` command and the ability to join nodes with bootstrap tokens is available starting with the 2023-02 releases (v1.26.2+k3s1, v1.25.7+k3s1, v1.24.11+k3s1, v1.23.17+k3s1). | |||
| ::: | |||
|
|
|||
| K3s supports dynamically generated, automatically expiring agent bootstrap tokens. Bootstrap tokens can only be used to join agents. | |||
| K3s supports dynamically generated, automatically expiring agent [bootstrap tokens](https://kubernetes.io/docs/reference/access-authn-authz/bootstrap-tokens/). Bootstrap tokens can only be used to join agents and are consumed like the agent tokens with the config `--agent-token` or `K3S_AGENT_TOKEN`. | |||
There was a problem hiding this comment.
| K3s supports dynamically generated, automatically expiring agent [bootstrap tokens](https://kubernetes.io/docs/reference/access-authn-authz/bootstrap-tokens/). Bootstrap tokens can only be used to join agents and are consumed like the agent tokens with the config `--agent-token` or `K3S_AGENT_TOKEN`. | |
| K3s supports dynamically generated, automatically expiring agent [bootstrap tokens](https://kubernetes.io/docs/reference/access-authn-authz/bootstrap-tokens/). Bootstrap tokens can only be used to join agents and are consumed like the agent tokens with the config `--agent-token` or `K3S_AGENT_TOKEN`. |
This is not correct. You do not need to set --agent-token to use bootstrap tokens. They are created using the CLI or kubectl.
There was a problem hiding this comment.
I wanted to express that, to consume the already created bootstrap token, you must use --agent-token or K3S_AGENT_TOKEN, or is there any other way? We were a bit confused by the matrix https://docs.k3s.io/cli/token#token-types as it was not clear how to consume the bootstrap tokens.
There was a problem hiding this comment.
On agents, you just pass the token via --token, corresponding config key, or the K3S_TOKEN env var.
--agent-token is set on servers to specify a static token that can only be used to join agents. Before bootstrap token support, this was the only way to get an unprivileged token that could not be used to join servers.
| @@ -149,7 +149,7 @@ Flag | Description | |||
| Available as of the October 2023 releases (v1.28.2+k3s1, v1.27.7+k3s1, v1.26.10+k3s1, v1.25.15+k3s1). | |||
| ::: | |||
|
|
|||
| Rotate original server token with a new bootstrap token. After running this command, all servers and any agents that originally joined with the old token must be restarted with the new token. | |||
| Rotate original server token with a new bootstrap token. After running this command, all servers and any agents that originally joined with the old token must be restarted with the new token. This bootstrap token never expires. | |||
There was a problem hiding this comment.
Ugh our reuse of the term "bootstrap" sucks. This is the server token used to encrypt the bootstrap data, it is NOT an "agent bootstrap token" and I think we should not use the term "bootstrap token" to refer to anything other than an agent bootstrap token. This should just say "server token".
There was a problem hiding this comment.
Ok, let me try to rewrite it a bit better
| @@ -161,3 +161,10 @@ If you do not specify a new token, one will be generated for you. | |||
| `--token` value | Existing token used to join a server or agent to a cluster [$K3S_TOKEN] | |||
| `--new-token` value | New token that replaces existing token | |||
|
|
|||
| :::warning | |||
| Data encrypted before the rotation will need the old server token to decrypt | |||
There was a problem hiding this comment.
Yeah I don't think this is accurate. Changing the server token does NOT change the encryption config, so the same key is used to encrypt/decrypt the secrets.
| ::: | ||
|
|
||
| :::warning | ||
| The new server token is not written in `/var/lib/rancher/k3s/server/token` |
Added some extra information which could be useful for other users