Skip to content

Commit

Permalink
Add how-to for Cilium egress gateway (static egress IPs)
Browse files Browse the repository at this point in the history
  • Loading branch information
simu committed Oct 25, 2023
1 parent a120a19 commit bf62336
Show file tree
Hide file tree
Showing 2 changed files with 157 additions and 0 deletions.
156 changes: 156 additions & 0 deletions docs/modules/ROOT/pages/how-tos/network/cilium-egress-ip.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,156 @@
= Configure static Egress IPs with Cilium

== Prerequisites

* `cluster-admin` privileges
* Cluster is running Cilium.
See the xref:how-tos/network/migrate-to-cilium.adoc[migrating to Cilium] how-to for a guide to migrate a cluster to Cilium.
* `kubectl`
* `jq`
* `curl`
* Working `commodore` command

== Prepare for configuration

. Select cluster
+
[source,bash]
----
export CLUSTER_ID=c-cluster-id-1234 <1>
export COMMODORE_API_URL=https://api.syn.vshn.net <2>
export TENANT_ID=$(curl -sH "Authorization: Bearer $(commodore fetch-token)" \
"${COMMODORE_API_URL}/clusters/${CLUSTER_ID}" | jq -r '.tenant')
----
<1> Replace with the Project Syn cluster ID of the cluster to migrate
<2> Replace with the Lieutenant API on which the cluster is registered

== Configure Cilium to support static egress IPs

. Compile cluster catalog
+
[source,bash]
----
commodore catalog compile "$CLUSTER_ID" <1>
----
<1> We recommend switching to an empty directory to run this command.
Alternatively, switch to your existing directory for the cluster.

. Configure Cilium
+
[source,bash]
----
pushd "inventory/classes/$TENANT_ID"
yq -i '.parameters.cilium.cilium_helm_values.egressGateway.enabled=true' \
"${CLUSTER_ID}.yml"
yq -i '.parameters.cilium.cilium_helm_values.bpf.masquerade=true' \
"${CLUSTER_ID}.yml"
yq -i '.parameters.cilium.cilium_helm_values.l7Proxy=false' \
"${CLUSTER_ID}.yml" <1>
----
<1> Currently (as of Cilium 1.14), the egress gateway feature isn't fully compatible with L7 policies.
To avoid issues, Cilium recommends disabling the L7 proxy when using egress gateway.
See also https://github.com/cilium/cilium/issues/19642[GitHub issue cilium/cilium#19642].

. Commit and push changes
+
[source,bash]
----
git commit -am "Configure Cilium egress gateway on $CLUSTER_ID"
git push origin master <1>
----
<1> Optionally, you can push to a branch and merge once you're satisfied with the change.

. Compile catalog
+
[source,bash]
----
popd
commodore catalog compile "$CLUSTER_ID" --push -i
----

== Configure egress IPs

.Exoscale
[%collapsible]
====
On Exoscale, you can allocate https://community.exoscale.com/documentation/compute/eip/[elastic IPs (EIPs)] to use as static egress IPs.
To use an EIP as a static egress IP, you need to attach it to a cluster node.
Once you've done that, you can configure the following `CiliumEgresGatewayPolicy`:
[source,yaml]
----
apiVersion: cilium.io/v2
kind: CiliumEgressGatewayPolicy
metadata:
name: egress-ip <1>
spec:
destinationCIDRs:
- 0.0.0.0/0 <2>
egressGroups:
- egressIP: 203.0.113.100 <3>
nodeSelector:
matchLabels:
kubernetes.io/hostname: infra-abcd <4>
selectors: <5>
- podSelector:
matchLabels:
io.kubernetes.pod.namespace: example-namespace <6>
----
<1> Name of the policy, select a descriptive name if possible.
<2> Destination CIDRs which should be routed through the static egress IP.
In-cluster CIDRs are automatically excluded by Cilium.
<3> Exoscale EIP which you've allocated and attached to a cluster node.
<4> Cluster node to which you've attached the Exoscale EIP.
<5> This section allows you to select pods whose traffic should be routed through the static egress IP
Generally, entries are standard pod selectors.
<6> Cilium supports a special label key `io.kubernetes.pod.namespace` which allows selecting all pods in a specific namespace.
IMPORTANT: If the `egressIP` and `nodeSelector` of the entry in `egressGroups` don't match, egress traffic will be dropped.
====

.Cluster in private network
[%collapsible]
====
For clusters deployed in a private network, you can use the following configuration to use an IP outside the DHCP range as a static egress IP
[source,yaml]
----
apiVersion: cilium.io/v2
kind: CiliumEgressGatewayPolicy
metadata:
name: egress-ip <1>
spec:
destinationCIDRs:
- 0.0.0.0/0 <2>
excludedCIDRs:
- 172.18.200.0/24 <3>
egressGroups:
- egressIP: 172.18.200.30 <4>
nodeSelector:
matchLabels:
node-role.kubernetes.io/infra: "" <5>
selectors: <6>
- podSelector:
matchLabels:
io.kubernetes.pod.namespace: example-namespace <7>
----
<1> Name of the policy, select a descriptive name if possible.
<2> Destination CIDRs which should be routed through the static egress IP.
In-cluster CIDRs are automatically excluded by Cilium.
<3> We exclude the cluster's private network CIDR from the set of destinations that should be routed through the static egress IP.
While Cilium auto-excludes the cluster node IPs, we most likely want the whole private network CIDR to be excluded from the egress gateway policy.
Omit this configuration if you want traffic to other systems in the cluster's private network CIDR to be routed through the static egress IP.
<4> Select an IP in the cluster's private network CIDR (outside the DHCP range) as the static egress IP.
<5> Select infra nodes as egress gateway nodes.
Cilium will select one of the nodes to route egress traffic through the defined static egress IP.
<6> This section allows you to select pods whose traffic should be routed through the static egress IP
Generally, entries are standard pod selectors.
<7> Cilium supports a special label key `io.kubernetes.pod.namespace` which allows selecting all pods in a specific namespace.
NOTE: This configuration is untested, please update the documentation once you've tested this.
====

== Upstream documentation

* https://docs.cilium.io/en/stable/network/egress-gateway/[Cilium egress gateway documentation]
1 change: 1 addition & 0 deletions docs/modules/ROOT/partials/nav.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,7 @@
* Networking
** xref:oc4:ROOT:how-tos/network/migrate-to-cilium.adoc[]
** xref:oc4:ROOT:how-tos/network/cilium-egress-ip.adoc[]
* Ingress
** xref:oc4:ROOT:how-tos/ingress/self-signed-ingress-cert.adoc[]
Expand Down

0 comments on commit bf62336

Please sign in to comment.