Practical upper limit on number of machines in a k3s cluster?

[matthewadams]

What are the practical upper limits on the number of hardware nodes that can participate in an HA k3s cluster? Like, order of magnitude, is it in the 10s, 100s, 1000s, etc? What kind of OS resource (memory, CPU) increases are expected as the number of hardware nodes increases? Is there a limit on the number of server nodes versus agent nodes. Like, what it they were all server nodes?

[dereknola]

K3s falls inline with what upstream K8s supports.. There isn't an artificially limit to server vs agent nodes, but realistically at some point you don't want all your nodes to be server. The amount of control-plane traffic likely will not scale in line with application traffic and resource usage. So you are allocating resources on nodes that could otherwise be used for applications if you go "all servers".

[matthewadams]

Ok, as I expected. We'll just have to experiment to see what the numbers look like. Thanks. 🙏🏼

[brandond]

what it they were all server nodes?

etcd doesn't recommend having more than 7 nodes.
ref: https://etcd.io/docs/v3.5/faq/#what-is-maximum-cluster-size

Similarly, there is little value to having more than 2 control-plane nodes, as most of the controllers are lease-locked and only run on one node at a time anyway.

I would probably do more research into how HA works for the various Kubernetes components so you understand exactly how everything functions when scaled up or out, prior to making any decisions about how to grow your cluster.

[matthewadams]

Thanks for the input & pointers.