Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add how-to for Cilium egress gateway (static egress IPs) #285

Merged
merged 1 commit into from
Oct 31, 2023
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
163 changes: 163 additions & 0 deletions docs/modules/ROOT/pages/how-tos/network/cilium-egress-ip.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,163 @@
= Configure static Egress IPs with Cilium

== Prerequisites

* `cluster-admin` privileges
* Cluster is running Cilium.
See the xref:how-tos/network/migrate-to-cilium.adoc[migrating to Cilium] how-to for a guide to migrate a cluster to Cilium.
* `kubectl`
* `jq`
* `curl`
* Working `commodore` command

== Prepare for configuration

. Select cluster
+
[source,bash]
----
export CLUSTER_ID=c-cluster-id-1234 <1>
export COMMODORE_API_URL=https://api.syn.vshn.net <2>
export TENANT_ID=$(curl -sH "Authorization: Bearer $(commodore fetch-token)" \
"${COMMODORE_API_URL}/clusters/${CLUSTER_ID}" | jq -r '.tenant')
----
<1> Replace with the Project Syn cluster ID of the cluster to migrate
<2> Replace with the Lieutenant API on which the cluster is registered

== Configure Cilium to support static egress IPs

. Compile cluster catalog
+
[source,bash]
----
commodore catalog compile "$CLUSTER_ID" <1>
----
<1> We recommend switching to an empty directory to run this command.
Alternatively, switch to your existing directory for the cluster.

. Configure Cilium
+
[source,bash]
----
pushd "inventory/classes/$TENANT_ID"
yq -i '.parameters.cilium.egress_gateway.enabled=true' \
"${CLUSTER_ID}.yml" <1>
----
<1> Currently (as of Cilium 1.14), the egress gateway feature isn't fully compatible with L7 policies.
To avoid issues, Cilium recommends disabling the L7 proxy when using egress gateway.
Due to that, component cilium sets `l7Proxy=false` by default when `egress_gateway.enabled=true` is set.
See also https://github.com/cilium/cilium/issues/19642[GitHub issue cilium/cilium#19642].

. Commit and push changes
+
[source,bash]
----
git commit -am "Configure Cilium egress gateway on $CLUSTER_ID"
git push origin master <1>
----
<1> Optionally, you can push to a branch and merge once you're satisfied with the change.

. Compile catalog
+
[source,bash]
----
popd
commodore catalog compile "$CLUSTER_ID" --push -i
----

. After the change has been rolled out by ArgoCD, restart the Cilium daemonset and operator
+
[source,bash]
----
kubectl --as=cluster-admin -n cilium rollout restart ds/cilium
kubectl --as=cluster-admin -n cilium rollout restart deploy/cilium-operator
----

== Configure egress IPs

.Exoscale
[%collapsible]
====
On Exoscale, you can allocate https://community.exoscale.com/documentation/compute/eip/[elastic IPs (EIPs)] to use as static egress IPs.
To use an EIP as a static egress IP, you need to attach it to a cluster node.

Once you've done that, you can configure `CiliumEgresGatewayPolicy` through component `cilium`:

[source,yaml]
----
parameters:
cilium:
egress_gateway:
egress-ip: <1>
spec:
destinationCIDRs:
- 0.0.0.0/0 <2>
egressGroups: <3>
- egressIP: 203.0.113.100 <4>
nodeSelector:
matchLabels:
kubernetes.io/hostname: infra-abcd <5>
selectors: <6>
- podSelector:
matchLabels:
io.kubernetes.pod.namespace: example-namespace <7>
----
<1> Name of the policy, select a descriptive name if possible.
<2> Destination CIDRs which should be routed through the static egress IP.
In-cluster CIDRs are automatically excluded by Cilium.
<3> `egressGroups` is only supported by Cilium Enterprise Edition.
See this https://isovalent.com/blog/post/2022-05-static-egress-gateway[blog post] for more details on the enterprise-only egress gateway features.
<4> Exoscale EIP which you've allocated and attached to a cluster node.
<5> Cluster node to which you've attached the Exoscale EIP.
<6> This section allows you to select pods whose traffic should be routed through the static egress IP
Generally, entries are standard pod selectors.
<7> Cilium supports a special label key `io.kubernetes.pod.namespace` which allows selecting all pods in a specific namespace.

IMPORTANT: If the `egressIP` and `nodeSelector` of the entry in `egressGroups` don't match, egress traffic will be dropped.
====

.Cluster in private network
[%collapsible]
====
For clusters deployed in a private network, you can use the following configuration to use an IP outside the DHCP range as a static egress IP

[source,yaml]
----
parameters:
cilium:
egress_gateway:
egress-ip: <1>
spec:
destinationCIDRs:
- 0.0.0.0/0 <2>
excludedCIDRs:
- 172.18.200.0/24 <3>
egressGroups:
- egressIP: 172.18.200.30 <4>
nodeSelector:
matchLabels:
node-role.kubernetes.io/infra: "" <5>
selectors: <6>
- podSelector:
matchLabels:
io.kubernetes.pod.namespace: example-namespace <7>
----
<1> Name of the policy, select a descriptive name if possible.
<2> Destination CIDRs which should be routed through the static egress IP.
In-cluster CIDRs are automatically excluded by Cilium.
<3> We exclude the cluster's private network CIDR from the set of destinations that should be routed through the static egress IP.
While Cilium automatically excludes the cluster node IPs, we most likely want the whole private network CIDR to be excluded from the egress gateway policy.
Omit this configuration if you want traffic to other systems in the cluster's private network CIDR to be routed through the static egress IP.
<4> Select an IP in the cluster's private network CIDR (outside the DHCP range) as the static egress IP.
<5> Select infra nodes as egress gateway nodes.
Cilium will select one of the nodes to route egress traffic through the defined static egress IP.
<6> This section allows you to select pods whose traffic should be routed through the static egress IP
Generally, entries are standard pod selectors.
<7> Cilium supports a special label key `io.kubernetes.pod.namespace` which allows selecting all pods in a specific namespace.

NOTE: This configuration is untested, please update the documentation once you've tested this.
====

== Upstream documentation

* https://docs.cilium.io/en/stable/network/egress-gateway/[Cilium egress gateway documentation]
1 change: 1 addition & 0 deletions docs/modules/ROOT/partials/nav.adoc
Original file line number Diff line number Diff line change
@@ -104,6 +104,7 @@

* Networking
** xref:oc4:ROOT:how-tos/network/migrate-to-cilium.adoc[]
** xref:oc4:ROOT:how-tos/network/cilium-egress-ip.adoc[]

* Ingress
** xref:oc4:ROOT:how-tos/ingress/self-signed-ingress-cert.adoc[]