-
Notifications
You must be signed in to change notification settings - Fork 2
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add how-to for Cilium egress gateway (static egress IPs)
- Loading branch information
Showing
2 changed files
with
157 additions
and
0 deletions.
There are no files selected for viewing
156 changes: 156 additions & 0 deletions
156
docs/modules/ROOT/pages/how-tos/network/cilium-egress-ip.adoc
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,156 @@ | ||
= Configure static Egress IPs with Cilium | ||
|
||
== Prerequisites | ||
|
||
* `cluster-admin` privileges | ||
* Cluster is running Cilium. | ||
See the xref:how-tos/network/migrate-to-cilium.adoc[migrating to Cilium] how-to for a guide to migrate a cluster to Cilium. | ||
* `kubectl` | ||
* `jq` | ||
* `curl` | ||
* Working `commodore` command | ||
|
||
== Prepare for configuration | ||
|
||
. Select cluster | ||
+ | ||
[source,bash] | ||
---- | ||
export CLUSTER_ID=c-cluster-id-1234 <1> | ||
export COMMODORE_API_URL=https://api.syn.vshn.net <2> | ||
export TENANT_ID=$(curl -sH "Authorization: Bearer $(commodore fetch-token)" \ | ||
"${COMMODORE_API_URL}/clusters/${CLUSTER_ID}" | jq -r '.tenant') | ||
---- | ||
<1> Replace with the Project Syn cluster ID of the cluster to migrate | ||
<2> Replace with the Lieutenant API on which the cluster is registered | ||
|
||
== Configure Cilium to support static egress IPs | ||
|
||
. Compile cluster catalog | ||
+ | ||
[source,bash] | ||
---- | ||
commodore catalog compile "$CLUSTER_ID" <1> | ||
---- | ||
<1> We recommend switching to an empty directory to run this command. | ||
Alternatively, switch to your existing directory for the cluster. | ||
|
||
. Configure Cilium | ||
+ | ||
[source,bash] | ||
---- | ||
pushd "inventory/classes/$TENANT_ID" | ||
yq -i '.parameters.cilium.cilium_helm_values.egressGateway.enabled=true' \ | ||
"${CLUSTER_ID}.yml" | ||
yq -i '.parameters.cilium.cilium_helm_values.bpf.masquerade=true' \ | ||
"${CLUSTER_ID}.yml" | ||
yq -i '.parameters.cilium.cilium_helm_values.l7Proxy=false' \ | ||
"${CLUSTER_ID}.yml" <1> | ||
---- | ||
<1> Currently (as of Cilium 1.14), the egress gateway feature isn't fully compatible with L7 policies. | ||
To avoid issues, Cilium recommends disabling the L7 proxy when using egress gateway. | ||
See also https://github.com/cilium/cilium/issues/19642[GitHub issue cilium/cilium#19642]. | ||
|
||
. Commit and push changes | ||
+ | ||
[source,bash] | ||
---- | ||
git commit -am "Configure Cilium egress gateway on $CLUSTER_ID" | ||
git push origin master <1> | ||
---- | ||
<1> Optionally, you can push to a branch and merge once you're satisfied with the change. | ||
|
||
. Compile catalog | ||
+ | ||
[source,bash] | ||
---- | ||
popd | ||
commodore catalog compile "$CLUSTER_ID" --push -i | ||
---- | ||
|
||
== Configure egress IPs | ||
|
||
.Exoscale | ||
[%collapsible] | ||
==== | ||
On Exoscale, you can allocate https://community.exoscale.com/documentation/compute/eip/[elastic IPs (EIPs)] to use as static egress IPs. | ||
To use an EIP as a static egress IP, you need to attach it to a cluster node. | ||
Once you've done that, you can configure the following `CiliumEgresGatewayPolicy`: | ||
[source,yaml] | ||
---- | ||
apiVersion: cilium.io/v2 | ||
kind: CiliumEgressGatewayPolicy | ||
metadata: | ||
name: egress-ip <1> | ||
spec: | ||
destinationCIDRs: | ||
- 0.0.0.0/0 <2> | ||
egressGroups: | ||
- egressIP: 203.0.113.100 <3> | ||
nodeSelector: | ||
matchLabels: | ||
kubernetes.io/hostname: infra-abcd <4> | ||
selectors: <5> | ||
- podSelector: | ||
matchLabels: | ||
io.kubernetes.pod.namespace: example-namespace <6> | ||
---- | ||
<1> Name of the policy, select a descriptive name if possible. | ||
<2> Destination CIDRs which should be routed through the static egress IP. | ||
In-cluster CIDRs are automatically excluded by Cilium. | ||
<3> Exoscale EIP which you've allocated and attached to a cluster node. | ||
<4> Cluster node to which you've attached the Exoscale EIP. | ||
<5> This section allows you to select pods whose traffic should be routed through the static egress IP | ||
Generally, entries are standard pod selectors. | ||
<6> Cilium supports a special label key `io.kubernetes.pod.namespace` which allows selecting all pods in a specific namespace. | ||
IMPORTANT: If the `egressIP` and `nodeSelector` of the entry in `egressGroups` don't match, egress traffic will be dropped. | ||
==== | ||
|
||
.Cluster in private network | ||
[%collapsible] | ||
==== | ||
For clusters deployed in a private network, you can use the following configuration to use an IP outside the DHCP range as a static egress IP | ||
[source,yaml] | ||
---- | ||
apiVersion: cilium.io/v2 | ||
kind: CiliumEgressGatewayPolicy | ||
metadata: | ||
name: egress-ip <1> | ||
spec: | ||
destinationCIDRs: | ||
- 0.0.0.0/0 <2> | ||
excludedCIDRs: | ||
- 172.18.200.0/24 <3> | ||
egressGroups: | ||
- egressIP: 172.18.200.30 <4> | ||
nodeSelector: | ||
matchLabels: | ||
node-role.kubernetes.io/infra: "" <5> | ||
selectors: <6> | ||
- podSelector: | ||
matchLabels: | ||
io.kubernetes.pod.namespace: example-namespace <7> | ||
---- | ||
<1> Name of the policy, select a descriptive name if possible. | ||
<2> Destination CIDRs which should be routed through the static egress IP. | ||
In-cluster CIDRs are automatically excluded by Cilium. | ||
<3> We exclude the cluster's private network CIDR from the set of destinations that should be routed through the static egress IP. | ||
While Cilium auto-excludes the cluster node IPs, we most likely want the whole private network CIDR to be excluded from the egress gateway policy. | ||
Omit this configuration if you want traffic to other systems in the cluster's private network CIDR to be routed through the static egress IP. | ||
<4> Select an IP in the cluster's private network CIDR (outside the DHCP range) as the static egress IP. | ||
<5> Select infra nodes as egress gateway nodes. | ||
Cilium will select one of the nodes to route egress traffic through the defined static egress IP. | ||
<6> This section allows you to select pods whose traffic should be routed through the static egress IP | ||
Generally, entries are standard pod selectors. | ||
<7> Cilium supports a special label key `io.kubernetes.pod.namespace` which allows selecting all pods in a specific namespace. | ||
NOTE: This configuration is untested, please update the documentation once you've tested this. | ||
==== | ||
|
||
== Upstream documentation | ||
|
||
* https://docs.cilium.io/en/stable/network/egress-gateway/[Cilium egress gateway documentation] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters