Make Kine work in RKE2

[caroline-suse-rancher]

This issue is to track the work of integrating Kine into RKE2, as per the internal Jira ticket here.

Describe the solution you'd like
Add Kine into RKE2 to allow for additional datastore options.

Additional context
At this time RKE2 can only be configured with etcd as a datastore, what we would instead like to see is that RKE2 can be configured with sqlite, postgres, nats, or etcd as a datastore.

[VestigeJ]

// new feature previous behavior is that the flag is simply unsupported and will fail to start or ignore.

##Environment Details
Validated using COMMIT=eb2d438a2fe6b426ecd00cb8e829ddc728a246b7

Infrastructure

• Cloud

Node(s) CPU architecture, OS, and version:

Linux 5.14.21-150500.53-default x86_64 GNU/Linux
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP5"

Cluster Configuration:

NAME               STATUS   ROLES                  AGE    VERSION
ip-1-1-2-2         Ready    control-plane,master   107m   v1.29.3+rke2r1
ip-1-1-2-9         Ready    <none>                 110m   v1.29.3+rke2r1
ip-1-1-3-22        Ready    control-plane,master   115m   v1.29.3+rke2r1
ip-1-1-2-77        Ready    control-plane,master   109m   v1.29.3+rke2r1

Config.yaml:

node-external-ip: 1.1.2.2
token: YOUR_TOKEN_HERE
write-kubeconfig-mode: 644
debug: true
kine-tls: true
embedded-registry: true
datastore-endpoint: postgres://k3s:[email protected]:5432/k3s

Reproduction

$ curl https://get.rke2.io --output install-"rke2".sh
$ sudo chmod +x install-"rke2".sh
$ sudo groupadd --system etcd && sudo useradd -s /sbin/nologin --system -g etcd etcd
$ sudo modprobe ip_vs_rr
$ sudo modprobe ip_vs_wrr
$ sudo modprobe ip_vs_sh
$ sudo printf "on_oovm.panic_on_oom=0 \nvm.overcommit_memory=1 \nkernel.panic=10 \nkernel.panic_ps=1 \nkernel.panic_on_oops=1 \n" > ~/60-rke2-cis.conf
$ sudo cp 60-rke2-cis.conf /etc/sysctl.d/
$ sudo systemctl restart systemd-sysctl
$ COMMIT=eb2d438a2fe6b426ecd00cb8e829ddc728a246b7
$ sudo INSTALL_RKE2_COMMIT=$COMMIT INSTALL_RKE2_EXEC=server ./install-rke2.sh
$ go_rke2
$ set_kubefig rke2
$ w2 kg no,po,svc -A
$ sudo etcdctl endpoint status --cacert=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --endpoints=unixs:///var/lib/rancher/rke2/server/kine.sock -w table

Results:

$ sudo etcdctl endpoint status --cacert=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --endpoints=unixs:///var/lib/rancher/rke2/server/kine.sock -w table

+------------------------------------------------+----+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|                    ENDPOINT                    | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+------------------------------------------------+----+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| unixs:///var/lib/rancher/rke2/server/kine.sock |  0 |         |   13 MB |      true |      false |         0 |          0 |                  0 |        |
+------------------------------------------------+----+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

$ kg po,svc -A

NAMESPACE     NAME                                                       READY   STATUS      RESTARTS   AGE
kube-system   pod/cloud-controller-manager-ip-23                         1/1     Running     0          12m
kube-system   pod/kube-apiserver-ip-23                                   1/1     Running     0          12m
kube-system   pod/kube-scheduler-ip-23                                   1/1     Running     0          12m
kube-system   pod/kube-controller-manager-ip-23                          1/1     Running     0          12m
kube-system   pod/kube-proxy-ip-23                                       1/1     Running     0          12m
kube-system   pod/helm-install-rke2-coredns-q2qkc                        0/1     Completed   0          11m
kube-system   pod/helm-install-rke2-canal-cm9km                          0/1     Completed   0          11m
kube-system   pod/rke2-canal-cgp44                                       2/2     Running     0          11m
kube-system   pod/rke2-coredns-rke2-coredns-autoscaler-b49765765-m82ws   1/1     Running     0          11m
kube-system   pod/helm-install-rke2-snapshot-validation-webhook-fq6cc    0/1     Completed   0          11m
kube-system   pod/helm-install-rke2-snapshot-controller-crd-rlxfx        0/1     Completed   0          11m
kube-system   pod/rke2-snapshot-validation-webhook-54c5989b65-495mm      1/1     Running     0          11m
kube-system   pod/helm-install-rke2-metrics-server-7vmjh                 0/1     Completed   0          11m
kube-system   pod/helm-install-rke2-snapshot-controller-jxgtf            0/1     Completed   2          11m
kube-system   pod/rke2-snapshot-controller-59cc9cd8f4-6mpfr              1/1     Running     0          11m
kube-system   pod/rke2-metrics-server-544c8c66fc-dsvjl                   1/1     Running     0          11m
kube-system   pod/rke2-coredns-rke2-coredns-5b7d84d764-dxz7c             1/1     Running     0          11m
kube-system   pod/helm-install-rke2-ingress-nginx-kpffv                  0/1     Completed   0          11m
kube-system   pod/rke2-ingress-nginx-controller-4rdn5                    1/1     Running     0          11m
kube-system   pod/rke2-canal-c96xt                                       2/2     Running     0          7m10s
kube-system   pod/kube-proxy-ip-23                                       1/1     Running     0          7m9s
kube-system   pod/rke2-coredns-rke2-coredns-5b7d84d764-w7nk6             1/1     Running     0          7m9s
kube-system   pod/rke2-ingress-nginx-controller-kvpsc                    1/1     Running     0          6m44s
kube-system   pod/kube-apiserver-ip-23                                   1/1     Running     0          5m41s
kube-system   pod/kube-controller-manager-ip-23                          1/1     Running     0          5m34s
kube-system   pod/cloud-controller-manager-ip-23                         1/1     Running     0          5m36s
kube-system   pod/kube-scheduler-ip-23                                   1/1     Running     0          5m34s
kube-system   pod/kube-proxy-ip-23                                       1/1     Running     0          5m39s
kube-system   pod/rke2-canal-cg86n                                       2/2     Running     0          5m46s
kube-system   pod/rke2-ingress-nginx-controller-zfndp                    1/1     Running     0          5m22s
kube-system   pod/kube-apiserver-ip-23                                   1/1     Running     0          3m28s
kube-system   pod/kube-scheduler-ip-23                                   1/1     Running     0          3m32s
kube-system   pod/kube-controller-manager-ip-23                          1/1     Running     0          3m32s
kube-system   pod/kube-proxy-ip-23                                       1/1     Running     0          3m28s
kube-system   pod/cloud-controller-manager-ip-23                         1/1     Running     0          3m21s
kube-system   pod/rke2-canal-g4ggg                                       2/2     Running     0          3m35s
kube-system   pod/rke2-ingress-nginx-controller-x95wx                    1/1     Running     0          3m13s

NAMESPACE     NAME                                              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)         AGE
default       service/kubernetes                                ClusterIP   10.43.0.1       <none>        443/TCP         12m
kube-system   service/rke2-coredns-rke2-coredns                 ClusterIP   10.43.0.10      <none>        53/UDP,53/TCP   11m
kube-system   service/rke2-snapshot-validation-webhook          ClusterIP   10.43.172.254   <none>        443/TCP         11m
kube-system   service/rke2-metrics-server                       ClusterIP   10.43.111.103   <none>        443/TCP         11m
kube-system   service/rke2-ingress-nginx-controller-admission   ClusterIP   10.43.210.176   <none>        443/TCP         11m

$ sudo etcdctl endpoint status --cacert=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --endpoints=unixs:///var/lib/rancher/rke2/server/kine.sock -w table

+------------------------------------------------+----+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|                    ENDPOINT                    | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+------------------------------------------------+----+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| unixs:///var/lib/rancher/rke2/server/kine.sock |  0 |         |   14 MB |      true |      false |         0 |          0 |                  0 |        |
+------------------------------------------------+----+---------+---------+-----------+------------+-----------+------------+--------------------+--------+