How to access k3s ETCD mgmt endpoint?

[mogliang]

Hi all,

We are working on cluster api controlplane/bootstrap provider for k3s. We are facing issue that how to access target cluster etcd mgmt api from mgmt cluster.

For the default kubeadm cp provider, etcd are hosted as pod, so mgmt cluster use apiserver proxy to access etcd mgmt endpoint. However, for k3s, etcd are hosted as a process, so there is no direct way to access it from mgmt cluster.

As discussed in #9818, we found out there is way to interact with etcd mgmt feature through node annotation, it can meet part of our need, but i'm wondering there are similar way that we can leverage to implment all features below???

1. get etcd health state, it shall consider as healthy if:
1. all controlplane Machine CRs in mgmt cluster have coresponding etcd member.
2. get etcd members list from each etcd member, the list shall be the same.
3. get alarms list of each etcd member, it should be No alarm.

2. reconcile etcd in controlplane reconcile loop
1. loop all etcd members, remove etcd member which does not in controlplane machine CRs.

3. remove etcd before removing controlplane node

Some background: currently, our solution is to create a pod as etcd proxy, just want to check if there is better way to do so.
k3s-io/cluster-api-k3s#75

[brandond]

Where do these requirements come from? While you can interact directly with the embedded etcd, we recommend against it.

The embedded etcd cluster should be managed through node resources. Deleting nodes will delete the corresponding cluster member. Recent releases of k3s include a status condition for etcd that you can use to check whether the node is a voting member of the cluster, as well as whether or not it has any alarms. I would recommend using this status condition, rather than re-implementing the same checks in your capi provider.

The only thing that the embedded etcd controller does not currently handle is automatically removing members from the cluster that do not have a corresponding Kubernetes node. We didn't feel like this was something that could be done safely.

[mogliang]

Thanks, would you please provide more details about the new etcd conditions you mentioned? It will be nice to avoid any direct etcd endpoint call.

Besides, if user use sqlite or other db as etcd source, will conditions and remove annotation be different?

#Where do these requirements come from?
These are from kubeadm controlplane design. https://github.com/kubernetes-sigs/cluster-api/blob/main/docs/proposals/20191017-kubeadm-based-control-plane.md#scenario-3-three-replicas-one-unresponsive-etcd-member-one-different-unhealthy-machine
If embedded etcd controller already implemented the these features, then we can just interact with etcd controller in k3s provider.

[brandond]

The status condition and status types are at:

k3s/pkg/etcd/etcd.go

Lines 75 to 80 in 7f65975

	etcdStatusType = v1.NodeConditionType("EtcdIsVoter")
	StatusUnjoined MemberStatus = "unjoined"
	StatusUnhealthy MemberStatus = "unhealthy"
	StatusLearner MemberStatus = "learner"
	StatusVoter MemberStatus = "voter"

if user use sqlite or other db as etcd source, will conditions and remove annotation be different?

There are no etcd status conditions when using kine. With kine, the datastore is always considered healthy. If the database is not available, k3s will fail to start, or exit if previously running.

[mogliang]

Thanks @brandond !

I drafted an proposal for implementing ETCD mgmt in k3s provider, would you mind take some time to review and comment?

k3s-io/cluster-api-k3s#97