From d4540cf39257d58b7ce4c286442d6daff2865ee2 Mon Sep 17 00:00:00 2001 From: Simon Gerber Date: Thu, 2 May 2024 09:40:45 +0200 Subject: [PATCH 1/2] Adjust install instructions to work with pre-configured Cilium `k8sServiceHost` Helm value The new install instructions assume that the following configuration is present somewhere in the Project Syn hierarchy: ``` parameters: cilium: cilium_helm_values: k8sServiceHost: api-int.${openshift:baseDomain} ``` --- .../install-config-cloudscale-exoscale.adoc | 6 ++- .../install/install-config-openstack.adoc | 20 +++++---- .../install/install-config-vsphere.adoc | 10 +++-- .../partials/install/prepare-commodore.adoc | 41 ++++++++++++++++++- .../partials/install/prepare-syn-config.adoc | 3 -- .../ROOT/partials/install/run-installer.adoc | 9 +++- 6 files changed, 68 insertions(+), 21 deletions(-) diff --git a/docs/modules/ROOT/partials/install/install-config-cloudscale-exoscale.adoc b/docs/modules/ROOT/partials/install/install-config-cloudscale-exoscale.adoc index 43a3e865..0c529b64 100644 --- a/docs/modules/ROOT/partials/install/install-config-cloudscale-exoscale.adoc +++ b/docs/modules/ROOT/partials/install/install-config-cloudscale-exoscale.adoc @@ -14,8 +14,8 @@ mkdir -p "${INSTALLER_DIR}" cat > "${INSTALLER_DIR}/install-config.yaml" < +baseDomain: ${BASE_DOMAIN} <1> platform: none: {} networking: @@ -25,3 +25,5 @@ pullSecret: | sshKey: "$(cat $SSH_PUBLIC_KEY)" EOF ---- +<1> Make sure that the values here match the value of `$CLUSTER_DOMAIN` when combined as `.`. +Otherwise, the installation will most likely fail. diff --git a/docs/modules/ROOT/partials/install/install-config-openstack.adoc b/docs/modules/ROOT/partials/install/install-config-openstack.adoc index 2d94f40e..6ee68103 100644 --- a/docs/modules/ROOT/partials/install/install-config-openstack.adoc +++ b/docs/modules/ROOT/partials/install/install-config-openstack.adoc @@ -24,9 +24,9 @@ EOF cat > "${INSTALLER_DIR}/install-config.yaml" < + name: ${CLUSTER_ID} <1> +baseDomain: ${BASE_DOMAIN} <1> +compute: <2> - architecture: amd64 hyperthreading: Enabled name: worker @@ -37,7 +37,7 @@ compute: <1> rootVolume: size: 100 type: __DEFAULT__ # TODO: is this generally applicable? - additionalSecurityGroupIDs: <2> + additionalSecurityGroupIDs: <3> - ${CILIUM_SECURITY_GROUP_ID} controlPlane: architecture: amd64 @@ -50,11 +50,11 @@ controlPlane: rootVolume: size: 100 type: __DEFAULT__ # TODO: is this generally applicable? - additionalSecurityGroupIDs: <2> + additionalSecurityGroupIDs: <3> - ${CILIUM_SECURITY_GROUP_ID} platform: openstack: - cloud: shiftstack <3> + cloud: shiftstack <4> externalNetwork: ${EXTERNAL_NETWORK_NAME} apiFloatingIP: ${API_VIP} ingressFloatingIP: ${INGRESS_VIP} @@ -67,9 +67,11 @@ pullSecret: | sshKey: "$(cat $SSH_PUBLIC_KEY)" EOF ---- -<1> We only provision a single compute machine set. +<1> Make sure that the values here match the value of `$CLUSTER_DOMAIN` when combined as `.`. +Otherwise, the installation will most likely fail. +<2> We only provision a single compute machine set. The final machine sets will be configured through Project Syn. -<2> We attach the Cilium security group to both the control plane and the worker nodes. +<3> We attach the Cilium security group to both the control plane and the worker nodes. This ensures that there's no issues with Cilium traffic during bootstrapping. -<3> This field must match the entry in `clouds` in the `clouds.yaml` file. +<4> This field must match the entry in `clouds` in the `clouds.yaml` file. If you're following this guide, you shouldn't need to adjust this. diff --git a/docs/modules/ROOT/partials/install/install-config-vsphere.adoc b/docs/modules/ROOT/partials/install/install-config-vsphere.adoc index f39dfc13..7250ee12 100644 --- a/docs/modules/ROOT/partials/install/install-config-vsphere.adoc +++ b/docs/modules/ROOT/partials/install/install-config-vsphere.adoc @@ -12,9 +12,9 @@ mkdir -p "${INSTALLER_DIR}" cat > "${INSTALLER_DIR}/install-config.yaml" < + name: ${CLUSTER_ID} <1> +baseDomain: ${BASE_DOMAIN} <1> +compute: <2> - architecture: amd64 hyperthreading: Enabled name: worker @@ -60,5 +60,7 @@ pullSecret: | sshKey: "$(cat $SSH_PUBLIC_KEY)" EOF ---- -<1> We only provision a single compute machine set. +<1> Make sure that the values here match the value of `$CLUSTER_DOMAIN` when combined as `.`. +Otherwise, the installation will most likely fail. +<2> We only provision a single compute machine set. The final machine sets will be configured through Project Syn. diff --git a/docs/modules/ROOT/partials/install/prepare-commodore.adoc b/docs/modules/ROOT/partials/install/prepare-commodore.adoc index a168c68e..239ed2eb 100644 --- a/docs/modules/ROOT/partials/install/prepare-commodore.adoc +++ b/docs/modules/ROOT/partials/install/prepare-commodore.adoc @@ -18,12 +18,51 @@ mkdir -p inventory/classes/ git clone $(curl -sH"Authorization: Bearer $(commodore fetch-token)" "${COMMODORE_API_URL}/tenants/${TENANT_ID}" | jq -r '.gitRepo.url') inventory/classes/${TENANT_ID} ---- -. Add Cilium to cluster configuration +. Configure the cluster's domain in Project Syn ++ +[source,bash] +---- +export CLUSTER_DOMAIN="${CLUSTER_ID}.${BASE_DOMAIN}" <1> +---- +<1> Adjust this as necessary if you're using a non-standard cluster domain. ++ +[IMPORTANT] +==== +The cluster domain configured here must be correct. +The value is used to configure how Cilium connects to the cluster's K8s API. +==== + [source,bash] ---- pushd "inventory/classes/${TENANT_ID}/" +yq eval -i ".parameters.openshift.baseDomain = \"${CLUSTER_DOMAIN}\"" \ + ${CLUSTER_ID}.yml + +git commit -a -m "Configure cluster domain for ${CLUSTER_ID}" +---- + +. Add Cilium to cluster configuration ++ +[NOTE] +==== +These instructions assume that Cilium is configured to use `api-int.${CLUSTER_DOMAIN}:6443` to connect to the cluster's K8s API. +To ensure that that's the case, add the configuration shown below somewhere in the Project Syn config hierarchy. + +[source,yaml] +---- +parameters: + cilium: + cilium_helm_values: + k8sServiceHost: api-int.${openshift:baseDomain} + k8sServicePort: "6443" +---- + +For VSHN, this configuration is set in the https://git.vshn.net/syn/commodore-defaults/-/merge_requests/1789[Commodore global defaults (internal)]. +==== ++ +[source,bash] +---- yq eval -i '.applications += ["cilium"]' ${CLUSTER_ID}.yml yq eval -i '.parameters.networkpolicy.networkPlugin = "cilium"' ${CLUSTER_ID}.yml diff --git a/docs/modules/ROOT/partials/install/prepare-syn-config.adoc b/docs/modules/ROOT/partials/install/prepare-syn-config.adoc index c4d2d493..cd09a547 100644 --- a/docs/modules/ROOT/partials/install/prepare-syn-config.adoc +++ b/docs/modules/ROOT/partials/install/prepare-syn-config.adoc @@ -18,9 +18,6 @@ fi + [source,bash,subs="attributes+"] ---- -yq eval -i ".parameters.openshift.baseDomain = \"${CLUSTER_DOMAIN}\"" \ - ${CLUSTER_ID}.yml - yq eval -i ".parameters.openshift.infraID = \"$(jq -r .infraID "${INSTALLER_DIR}/metadata.json")\"" \ ${CLUSTER_ID}.yml diff --git a/docs/modules/ROOT/partials/install/run-installer.adoc b/docs/modules/ROOT/partials/install/run-installer.adoc index 5ed48607..7e420607 100644 --- a/docs/modules/ROOT/partials/install/run-installer.adoc +++ b/docs/modules/ROOT/partials/install/run-installer.adoc @@ -25,12 +25,17 @@ yq w -i "${INSTALLER_DIR}/manifests/cluster-ingress-02-config.yml" \ cp catalog/manifests/cilium/olm/* target/manifests/ ---- -. Extract the cluster domain from the generated manifests +. Verify that the generated cluster domain matches the desired cluster domain + [source,bash] ---- -export CLUSTER_DOMAIN=$(yq e '.spec.baseDomain' \ +GEN_CLUSTER_DOMAIN=$(yq e '.spec.baseDomain' \ "${INSTALLER_DIR}/manifests/cluster-dns-02-config.yml") +if [ "$GEN_CLUSTER_DOMAIN" != "$CLUSTER_DOMAIN" ]; then + echo -e "\033[0;31mGenerated cluster domain doesn't match expected cluster domain: Got '$GEN_CLUSTER_DOMAIN', want '$CLUSTER_DOMAIN'\033[0;0m" +else + echo -e "\033[0;32mGenerated cluster domain matches expected cluster domain.\033[0;0m" +fi ---- . Prepare install manifests and ignition config From c16baf6f4757a635c3e0f3d93295430db45d95d5 Mon Sep 17 00:00:00 2001 From: Simon Gerber Date: Thu, 2 May 2024 10:12:15 +0200 Subject: [PATCH 2/2] Adjust Cilium migration how-to to work with the existing k8sServiceHost config --- .../how-tos/network/migrate-to-cilium.adoc | 22 ++++++++++++------- 1 file changed, 14 insertions(+), 8 deletions(-) diff --git a/docs/modules/ROOT/pages/how-tos/network/migrate-to-cilium.adoc b/docs/modules/ROOT/pages/how-tos/network/migrate-to-cilium.adoc index 18f58dca..5c313032 100644 --- a/docs/modules/ROOT/pages/how-tos/network/migrate-to-cilium.adoc +++ b/docs/modules/ROOT/pages/how-tos/network/migrate-to-cilium.adoc @@ -143,20 +143,26 @@ yq eval -i '.parameters.networkpolicy.ignoredNamespaces = ["openshift-oauth-apis "${CLUSTER_ID}.yml" ---- -. Configure component `cilium`. -We explicitly configure the K8s API endpoint to ensure that the Cilium operator doesn't access the API through the cluster network during the migration. +. Verify that the cluster's `api-int` DNS record exists + -TIP: When running Cilium with `kubeProxyReplacement=partial`, the API endpoint configuration can be removed after the migration is completed. +[source,bash] +---- +export CLUSTER_DOMAIN=$(kubectl get dns cluster -ojsonpath='{.spec.baseDomain}') +kubectl --as=cluster-admin -n openshift-dns exec ds/node-resolver -- nslookup api-int.${CLUSTER_DOMAIN} +---- + -.Explicitly configure the K8s API endpoint +[TIP] +==== +If the command doesn't return a valid record for `api-int`, the Cilium config needs to be adjusted to use the `api` record to ensure that the Kubernetes API can be accessed. + [source,bash] ---- -yq -i '.parameters.cilium.cilium_helm_values.k8sServiceHost="api-int.${openshift:baseDomain}"' \ - "${CLUSTER_ID}.yml" <1> -yq -i '.parameters.cilium.cilium_helm_values.k8sServicePort="6443"' \ +yq -i '.parameters.cilium.cilium_helm_values.k8sServiceHost="api.${openshift:baseDomain}"' \ "${CLUSTER_ID}.yml" ---- -<1> On vSphere clusters, you may need to use `api.${openshift:baseDomain}`. +==== + +. Configure component `cilium`. + .Configure the cluster Pod and Service CIDRs [source,bash]