docs: Add design document for Control Plane Ingress HA

Sees: #2381

docs/developer/architecture/control-plane-ingress.rst

@@ -0,0 +1,163 @@
+Control Plane Ingress
+=====================
+
+Context
+-------
+
+Initially, Control Plane Ingress Controller was deployed using a DaemonSet with
+a ClusterIP service using Bootstrap Control Plane IP as External IP and then
+configuring all Control Plane components using this “External IP” (like OIDC
+and various UIs).
+
+So it means, we can only reach those components using Bootstrap Control
+Plane IP, and if, for whatever reason, Bootstrap is down you can no longer
+access any UIs and you need to reconfigure all components manually, change the
+IP used everywhere to another Control Plane node IP, in order to restore
+access to Control Plane components.
+
+Here, we want to solve this issue and make Control Plane components Highly
+Available, so if you lose one node, including Bootstrap Node, you can still
+access various UIs.
+NOTE: In this document, we do not talk about the High Availability of the
+components itself but really only to the access through the Ingress
+(e.g.: We do not want to solve salt-master HA here).
+
+User Stories
+------------
+
+MetalK8s and Grafana UIs HA
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+I have a multi-node MetalK8s cluster, with at least 3 Control Plane nodes
+if I lose one of the Control Plane nodes (including the Bootstrap one)
+I can still access and authenticate on the MetalK8s and Grafana UIs.
+
+Design Choices
+--------------
+
+To have a proper HA Control Plane Ingress we want to use a
+`Virtual IP <https://en.wikipedia.org/wiki/Virtual_IP_address>`_ using
+`MetalLB <https://metallb.universe.tf/>`_ so that we can rely on layer2
+ARP requests when possible or let MetalLB advertise IPs using BGP when
+the user router support it.
+
+NOTE: MetalLB will be used only for Control Plane Ingress and not
+for Workload Plane.
+
+But in some network, it may be not possible to use one of those methods
+so we also let the possibility to not use MetalLB but instead just
+assign an External IP, provided by the user, that we expect to be a
+Virtual IP, and we do not manage on our side but it’s managed by the user
+using whatever mechanism to switch this IP between Control Plane nodes.
+
+To summarize 3 different deployments possible depending on the user
+environment:
+
+- Using VIP managed by Layer2 ARP with MetalLB (builtin MetalK8s)
+- Using VIP managed by BGP announce with MetalLB (builtin MetalK8s)
+- Using a user-provided IP that should be able to switch between Control
+  Plane nodes (managed by the user)
+
+NOTE: In all those 3 approaches we want the user to provide the Control
+Plane Ingress IP he wants to use.
+
+Rejected Design Choices
+-----------------------
+
+TODO
+
+Implementation Details
+----------------------
+
+Control Plane Ingress IP
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+In order to configure all Control Plane components we need a single IP as
+Control Plane Ingress, so we expect the user to provide this Control Plane
+Ingress IP in the Bootstrap configuration file.
+To have some backward compatibility this Ingress IP is only mandatory when
+you use MetalLB and will default to Bootstrap Control Plane IP if not
+(so that we have the same behavior as before).
+
+This Control Plane Ingress IP can be changed at any time just by editing
+the Bootstrap configuration file and follow a simple documented procedure
+with some Salt states to reconfigure every component that needs to re-configured.
+
+NOTE: Changing this Control Plane Ingress IP means we need to reconfigure
+all Kubernetes APIServer since we use this Ingress IP as an OIDC provider.
+
+MetalLB Configuration
+~~~~~~~~~~~~~~~~~~~~~
+
+MetalLB is not deployed in every environment so it needs to be configurable
+from the Bootstrap configuration file, that’s why we have a new field about
+MetalLB in the Control Plane network section.
+
+MetalLB can use 2 different protocols either Layer2 either BGP, by default
+we will use a simple Layer2 configuration for MetalLB so that if the user
+environment allows using Layer2 protocol of MetalLB you just have to enable
+MetalLB, provide an Ingress IP for the Control Plane and that it.
+
+Default MetalLB configuration:
+
+.. code-block:: yaml
+
+    address-pools:
+    - name: ingress-ip
+      protocol: layer2
+      addresses:
+      - <control-plane ingress ip>/32
+      auto-assign: false
+
+But, since BGP router configuration can be really different between various
+environments, we need to make sure this
+`MetalLB configuration <https://metallb.universe.tf/configuration/#bgp-configuration>`_
+is exposed, that’s why we also have the ability to override default MetalLB
+configuration from the Bootstrap configuration file in order to provide BGP
+router information.
+
+Same as the Control Plane Ingress IP, we can switch from non-MetalLB to MetalLB
+(and the opposite) at any time just by following the same procedure.
+
+Deployment
+~~~~~~~~~~
+
+As for every other addon in MetalK8s, we will use the MetalLB helm chart and
+render this one using a specific “option” file. But this one will not be always
+deployed as we only want to deploy it when a specific key is set in the
+Bootstrap configuration file, so in the Salt pillar at the end.
+
+When we use MetalLB we do not want to use the same NGINX Ingress Controller
+deployments, since MetalLB will be the entry point in the Kubernetes cluster
+we do not need to use a DaemonSet running on every Control Plane nodes,
+instead, we will use a Deployment with 2 replicas.
+
+We also need to configure the Service for Ingress Controller differently
+depending on if we use MetalLB or not when we use it we want to use a
+LoadBalancer service and set the LoadBalancerIP to IngressIP provided by
+the user. If we do not use MetalLB then we use ClusterIP Service with IngressIP
+provided by the user as External IPs.
+
+It means the deployment of NGINX Ingress Controller depends on some Salt
+pillar values, also since we want to be able to switch between MetalLB and
+non-MetalLB we need to make sure the Salt states that deploy NGINX Ingress
+Controller remove no-longer-needed objects (e.g.: if you switch from
+non-MetalLB to MetalLB you want to remove the DaemonSet for NGINX Ingress
+Controller).
+
+Documentation
+-------------
+
+- Describe all new Bootstrap configuration fields (with some links
+  to MetalLB documentation to configure MetalLB to use BGP protocol)
+- Add a simple procedure to change the Control Plane Ingress IP and
+  reconfigure all Control Plane conponents that need to.
+
+Test Plan
+---------
+
+Add some End-to-End tests in the CI:
+
+- Use MetalLB and a VIP as Control Plane Ingress IP
+- Change Control Plane Ingress IP using documented procedure
+- Switch from non-MetalLB to MetalLB using documented procedure

docs/developer/architecture/index.rst

@@ -10,6 +10,7 @@ Architecture Documents
    centralized-cli
    ci
    configurations
+   control-plane-ingress
    deployment
    logs
    metalk8s-ui