Feature request: Add replicasPerNode for reliable deployments of H/A node-local services

[mumoshu]

Hey!

Thanks a lot for sharing the great work and providing us the world's best observability service.

I'm filing this feature request believing this fits nicely within the scope of this project, while filling a big missing-piece in the current K8s ecosystem.

Is your feature request related to a problem? Please describe.

A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

I've had a hard time designing and implementing robust and efficient deployments of node-local dns and all kinds of node-based ingress controllers(Nginx, envoy-based ones, etc.).

Basically the design issues distill into the below two points:

1. You should either use SO_REUSEPORT or K8s "Service Topology" (node-local dns) or externalTrafficPolicy: Local(ingresses) in order to expose a set of pods per node to the world with a minimum network hop.
2. Use of SO_REUSEPORT or externalTrafficPolicy: Local implies that you must have two or more pods "per node" in order to provide high-availability

Describe the solution you'd like

A clear and concise description of what you want to happen.

Add replicasPerNode to ExtendedDaemonSet.

For example, giving it a value of 2 results in an extended daemonset to produce at least 2 pods per node. I say "at least" as it can create one more pod per node in a canary deployment.

Implementation-wise, this would require:

• ReplicasPerNode *int added to the ExtendedDaemonSetSpec struct and the CRD updated accordingly
• The extended daemonset controller to be modified so that it can handle two or more non-canary extended-daemonset-replicaset
  • I believe it is currently "designed" to work with only one non-canary replicaset per extended daemonset. Although some functions like cleanupReplicaSet does seem like being designed to support arbitrary number of replicasets.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

A straightforward way to achieve point 2 is to deploy two or more daemonsets(I've been calling them "a set of daemonsets"). However, updating such set of daemonsets reliably isn't straightforward.

For example, kubectl apply -f set-of-daemonsets.yaml isn't usually what you want, because it can cause all the concurrent daemonset rolling update can start by bringing down all the daemonsets' pods in a specific node, which bring down the whole service provided by the pods on the node.

A possible solution would be to coordinate rolling-updates of the set of daemonsets. It's basically running kubectl apply -f first.daemonset.yaml and then kubectl apply -f second.daemonset.yaml and keep doing this until the last daemonset. With shell scripting, this can be a for loop. In a GitOps way, it can be a separate commit per daemonset.

The solution above works, but not in an ideal way. The ideal way would be a more declarative one. More concretely, a single kubectl apply or alike should be enough for a reliable update of a set of daemonsets.

Additional context

Add any other context or screenshots about the feature request here.

I'm eager to submit a pull request for this if this makes sense to be within this project.