Skip to content

Commit

Permalink
Update networking docs
Browse files Browse the repository at this point in the history 
Signed-off-by: Brad Davidson <[email protected]>
  • Loading branch information
@brandond
brandond committed Feb 3, 2025
1 parent 8fe21bb commit 8e91692
Showing 1 changed file with 29 additions and 20 deletions.
49 changes: 29 additions & 20 deletions docs/networking/basic_network_options.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,20 +2,28 @@
title: Network Options
---

RKE2 requires a CNI plugin to connect pods and services. The Canal CNI plugin is the default but all CNI plugins are supported. All CNI
plugins get installed via a helm chart after the main components are up and running and can be customized by modifying the helm chart options.
Kubernetes requires installation of one or more CNI plugins to provide Pod networking. RKE2 includes support for five CNIs: Canal, Cilium, Calico, Flannel, and Multus. Only Calico and Flannel support Microsoft Windows. Multus is a secondary CNI Plugin, and must be enabled alongside a primary CNI Plugin. For more information, see the [Multus and SR-IOV](multus_sriov.md) documentation.

Canal is the default CNI Plugin, but all bundled plugins are supported. Bundled CNI Plugins are installed via Helm chart, and can be customized by depolying a HelmChartConfig with customized values. For more information on using HelmChartConfig resources, see the [Helm Integration](../helm.md) documentation, and the CNI-specific examples provided below.

## Install a CNI plugin
## Select a CNI plugin

RKE2 integrates with four different CNI plugins: Canal, Cilium, Calico and Flannel. Note that only Calico and Flannel are options for RKE2 deployments with Windows nodes.
Use the `cni` [configuration file key](../install/configuration.md) to select the CNI Plugin you wish to use. If you do not want to use any of the bundled CNI Plugins, you can set `cni` to `none`. Note that nodes will remain NotReady and be tainted unschedulable until a CNI Plugin is installed.

The next tabs inform how to deploy each CNI plugin and override the default options:
```yaml
cni: canal
```
Bundled CNI Plugins are provided as AddOns that deploy a HelmChart resource, as described in the [Helm Integration](../helm.md) documentation. CNI Plugin charts are named `rke2-<CNI-PLUGIN-NAME>` and can be found in the `kube-system` namespace.

To customize the Helm chart values for a bundled CNI Plugin chart, you must create a HelmChartConfig resource that matches the name and namespace of its corresponding HelmChart. See the tabs below for examples of customizing the chart values for each of the bundled CNI Plugins.

Default chart values can be found by browsing the [RKE2 charts repository](https://github.com/rancher/rke2-charts/tree/main/charts), and referencing `values.yaml` for the version of the chart bundled with your RKE2 version.

<Tabs groupId = "CNIplugin" queryString>
<TabItem value="Canal CNI plugin" default>

Canal means using Flannel for inter-node traffic and Calico for intra-node traffic and network policies. By default, it will use vxlan encapsulation to create an overlay network among nodes. Canal is deployed by default in RKE2 and thus nothing must be configured to activate it. To override the default Canal options you should create a HelmChartConfig resource. The HelmChartConfig resource must match the name and namespace of its corresponding HelmChart. For example to override the flannel interface, you can apply the following config:
Canal uses Flannel for inter-node traffic and Calico for intra-node traffic and network policies. By default, it will use vxlan encapsulation to create an overlay network among nodes. For example, to override the flannel interface, you can apply the following chart values:

```yaml
# /var/lib/rancher/rke2/server/manifests/rke2-canal-config.yaml
Expand All @@ -31,7 +39,7 @@ spec:
iface: "eth1"
```

Starting with RKE2 v1.23 it is possible to use flannel's [wireguard backend](https://github.com/flannel-io/flannel/blob/master/Documentation/backends.md#wireguard) for in-kernel WireGuard encapsulation and encryption ([Users of kernels < 5.6 need to install a module](https://www.wireguard.com/install/)). This can be achieved using the following config:
Starting with RKE2 v1.23 it is possible to use flannel's [wireguard backend](https://github.com/flannel-io/flannel/blob/master/Documentation/backends.md#wireguard) for in-kernel WireGuard encapsulation and encryption ([Users of kernels < 5.6 need to install a module](https://www.wireguard.com/install/)). This can be achieved using the following chart values:

```yaml
# /var/lib/rancher/rke2/server/manifests/rke2-canal-config.yaml
Expand Down Expand Up @@ -64,7 +72,7 @@ Please check [Known issues and Limitations](../known_issues.md) if you experienc
</TabItem>
<TabItem value="Cilium CNI plugin" default>

To deploy Cilium, pass `cilium` as the value of the `--cni` flag. Ensure that the nodes have the right required kernel version (>= 4.9.17) and they meet the [requirements](https://docs.cilium.io/en/stable/operations/system_requirements/). To override the default options, please use a HelmChartConfig resource. The HelmChartConfig resource must match the name and namespace of its corresponding HelmChart. For example, to enable eni:
When using Cilium, you must wnsure that nodes have a supported kernel version (>= 4.9.17) and they meet the [requirements](https://docs.cilium.io/en/stable/operations/system_requirements/). To override the default options, please use a HelmChartConfig resource. The HelmChartConfig resource must match the name and namespace of its corresponding HelmChart. For example, to enable eni:

```yaml
# /var/lib/rancher/rke2/server/manifests/rke2-cilium-config.yaml
Expand All @@ -82,7 +90,7 @@ spec:

For more information about values available in the Cilium chart, please refer to the [rke2-charts repository](https://github.com/rancher/rke2-charts/blob/main/charts/rke2-cilium/rke2-cilium/1.14.400/values.yaml)

Cilium includes advanced features to fully replace kube-proxy and implement the routing of services using eBPF instead of iptables. It is not recommended to replace kube-proxy by Cilium if your kernel is not v5.8 or newer, as important bug fixes and features will be missing. To activate this mode, deploy rke2 with the flag `--disable-kube-proxy` and the following cilium configuration:
Cilium includes advanced features to fully replace kube-proxy and implement the routing of services using eBPF instead of iptables. It is not recommended to replace kube-proxy by Cilium if your kernel is not v5.8 or newer, as important bug fixes and features will be missing. To activate this mode, deploy rke2 with `disable-kube-proxy: true` in the configuration file, and the following chart values:

```yaml
# /var/lib/rancher/rke2/server/manifests/rke2-cilium-config.yaml
Expand All @@ -95,14 +103,14 @@ metadata:
spec:
valuesContent: |-
kubeProxyReplacement: true
k8sServiceHost: <KUBE_API_SERVER_IP>
k8sServicePort: <KUBE_API_SERVER_PORT>
k8sServiceHost: "localhost"
k8sServicePort: "6443"
```

For more information, please check the [upstream docs](https://docs.cilium.io/en/stable/network/kubernetes/kubeproxy-free/)

Cilium includes also an observability platform called [Hubble](https://docs.cilium.io/en/stable/overview/intro/#what-is-hubble)
To enable Hubble the following configuration is required:
To enable Hubble, use the following chart values:

```yaml
# /var/lib/rancher/rke2/server/manifests/rke2-cilium-config.yaml
Expand All @@ -123,12 +131,12 @@ spec:
```

:::warning
Cilium is currently not supported in the Windows installation of RKE2
Cilium is currently not supported on Windows.
:::

</TabItem>
<TabItem value="Calico CNI plugin" default>
To deploy Calico as the CNI plugin for RKE2 pass `calico` as the value of the `--cni` flag. To override the default options, please use a HelmChartConfig resource. The HelmChartConfig resource must match the name and namespace of its corresponding HelmChart. For example, to change the mtu:
For example, to change the interface MTU, you can use the following chart values:

```yaml
# /var/lib/rancher/rke2/server/manifests/rke2-calico-config.yaml
Expand All @@ -145,7 +153,7 @@ spec:
mtu: 9000
```

Because of a kernel bug in versions previous to 5.7, by default we are disabling the checksum offload done by the kernel. That config caps TCP performance to ~2.5Gbps. If you require higher throughput and have a kernel version greater than 5.7, you can enable the checksum offloading by using the following HelmChartConfig:
Because of a kernel bug in versions previous to 5.7, Calico disables hardware checksum offload. That config caps TCP performance to ~2.5Gbps. If you require higher throughput and have a kernel version greater than 5.7, you can enable the checksum offloading by using the following HelmChartConfig:

```yaml
# /var/lib/rancher/rke2/server/manifests/rke2-calico-config.yaml
Expand All @@ -169,25 +177,26 @@ Calico requires the iptables or xtables-nft package to be installed on the node

</TabItem>
<TabItem value="Flannel CNI plugin" default>
Starting with RKE2 2024 Feb release (v1.29.2, v1.28.7, v1.27.11, v1.26.14), Flannel can be deployed as the CNI plugin. To do so, pass `flannel` as the value of the `--cni` flag.
Starting with RKE2 2024 Feb release (v1.29.2, v1.28.7, v1.27.11, v1.26.14), Flannel can be deployed as the CNI plugin.

:::note
Only vxlan backend is supported at this point
Only the `vxlan` backend is supported.
:::

:::warning
Flannel does not support network policies. Therefore, it is not recommended for hardened installations
Flannel does not support network policies. Therefore, it is not recommended for hardened installations.
:::

:::warning
Flannel support in RKE2 is currently experimental. Do not run it on production systems before extensive testing
Flannel support in RKE2 is currently experimental. Do not run it on production systems without extensive testing.
:::

</TabItem>
</Tabs>

## Dual-stack configuration

IPv4/IPv6 dual-stack networking enables the allocation of both IPv4 and IPv6 addresses to Pods and Services. To configure RKE2 in dual-stack mode, in the control-plane nodes, you must set a valid IPv4/IPv6 dual-stack cidr for pods and services. To do so, use the flags `--cluster-cidr` and `--service-cidr` for example:
IPv4/IPv6 dual-stack networking enables the allocation of both IPv4 and IPv6 addresses to Pods and Services. To configure RKE2 in dual-stack mode, in the control-plane nodes, you must set a valid IPv4/IPv6 dual-stack cidr for pods and services. To do so, use the `cluster-cidr` and `service-cidr` configuration file keys:

```yaml
#/etc/rancher/rke2/config.yaml
Expand Down

0 comments on commit 8e91692

Please sign in to comment.