Skip to content

Commit

Permalink
Add steps to allow cluster bootstrap to complete with Cilium in kube-…
Browse files Browse the repository at this point in the history
…proxy-free mode
  • Loading branch information
simu committed Aug 5, 2024
1 parent 6ccd46e commit c282d87
Showing 1 changed file with 54 additions and 5 deletions.
59 changes: 54 additions & 5 deletions docs/modules/ROOT/partials/install/bootstrap-nodes.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,46 @@ while ! curl --connect-timeout 1 "${API_URL}/healthz" -k &>/dev/null; do
done && echo -e "\nAPI is up"
----

. Patch Cilium config to allow control plane bootstrap to succeed
+
[NOTE]
====
We need to temporarily adjust the Cilium config to not use full kube-proxy replacement, since we currently don't have a way to disable the initial OpenShift-managed kube-proxy deployment.
Additionally, Because the cloudscale Cloud Controller Manager accesses the K8s API via service IP, we need to configure Cilium to provide partial kube-proxy replacement so that the CCM can start and untaint the control plane nodes so that other pods can be scheduled.
====
+
[source,bash]
----
export KUBECONFIG="${INSTALLER_DIR}/auth/kubeconfig"
while ! kubectl get ciliumconfig -A &>/dev/null; do
echo -n "."
sleep 2
done && echo -e "\nCiliumConfig CR is present"
kubectl patch -n cilium ciliumconfig cilium-enterprise --type=merge \
-p '{
"spec": {
"cilium": {
"kubeProxyReplacement": "false",
"nodePort": {
"enabled": true
},
"socketLB": {
"enabled": true
},
"sessionAffinity": true,
"externalIPs": {
"enabled": true
},
"hostPort": {
"enabled": true
}
}
}
}'
----

. Deploy control plane nodes
+
[source,bash,subs="attributes+"]
Expand Down Expand Up @@ -92,6 +132,20 @@ terraform apply
popd
----

. Disable OpenShift kube-proxy deployment and revert Cilium patch
+
[source,bash]
----
kubectl patch network.operator cluster --type=merge \
-p '{"spec":{"deployKubeProxy":false}}'
kubectl -n cilium replace -f catalog/manifests/cilium/olm/cluster-network-07-cilium-ciliumconfig.yaml
while ! kubectl -n cilium get cm cilium-config -oyaml | grep 'kube-proxy-replacement: "true"' &>/dev/null; do
echo -n "."
sleep 2
done && echo -e "\nCilium config updated"
kubectl -n cilium rollout restart ds/cilium
----

. Review and merge the LB hieradata MR (listed in Terraform output `hieradata_mr`) and run Puppet on the LBs after the deploy job has completed
+
[source,bash]
Expand All @@ -103,11 +157,6 @@ done

. Approve node certs
+
[source,bash]
----
export KUBECONFIG="${INSTALLER_DIR}/auth/kubeconfig"
----
+
include::partial$install/approve-node-csrs.adoc[]

. Label infra nodes
Expand Down

0 comments on commit c282d87

Please sign in to comment.