Enable CNI

We assume the host is already served by barrel.

CNI Introduction

CNI is a competition for CNM, and each offers an approach to integrate SDN into container cluster. The reasons why we are pivoting from CNM to CNI are:

1. Docker has been sticking with CNM, but CNM is fading away as CNI is sweeping the community.
2. Eru cluster is expected to communicate with K8s cluster, and using CNI in Eru cluster simplifies everything.

Version Requirements

Version must be newer than v21.11.30-rc.

Deployment Guide

1. Configure Docker

This is an one-time step, only needed for the first-time deployment.

We need to add additional runtimes in dockerd config in /etc/docker/daemon.json:

{
    #"hosts": ..
    #"cluster-store": ... # keep other settings intact

    "runtimes": {
        "barrel-cni": {
            "path": "/usr/bin/eru-barrel-cni",
            "runtimeArgs": [ "oci", "--config", "/etc/docker/cni.yaml", "--" ]
        }
    }
}

/usr/bin/eru-barrel-cni and /etc/docker/cni.yaml don't have to exist by the time we re-configure dockerd, so stay calm.

2. Restart Dockerd

This is an one-time step, only needed for the first-time deployment.

Since we updated Dockerd configuration, we need to restart dockerd by

systemctl restart docker

If dockerd has been set "live-restore": true, running containers won't be affected by restarting immediately, but we still have to restart them by hand one by one.

If dockerd doesn't enable live-restore, restarting dockerd will immediately force all running containers to restart, and this must be some kind of risk to take into account.

3. Setup CNI

This is an one-time step, only needed for the first-time deployment.

This step counts on the CNI provider, please refer the specific CNI documentation.

Generally speaking, we have to ensure following configuration:

• /etc/cni/net.d/ must have the specific CNI configuration, such as /etc/cni/net.d/10-calico.conf
• /opt/cni/bin/ must have the specific CNI binary, such as /opt/cni/bin/calico and /opt/cni/bin/calico-ipam.
• /etc/docker/cni.yaml must be all set.

Let me give you a real world example for Calico CNI.

3.1. /etc/cni/net.d/10-calico.conf

{
    "name": "calico",
    "cniVersion": "0.3.1",
    "type": "calico",
    "log_level": "INFO",
    "etcd_endpoints": "http://127.0.0.1:2379",
    "log_file_path": "/var/log/calico/cni/cni.log",
    "ipam": {
        "type": "calico-ipam",
        "ipv4_pools": ["calico-pool-2"]
    }
}

Do's and don'ts:

• check etcd_endpoints, because it could be some other address.
• check ipv4_pools, you can use calicoctl get ipp to fetch all pools and for i in $(docker ps | awk '/redis/ {print $1}'); do docker inspect $i -f '{{.HostConfig.NetworkMode}}'; done to check what ippool running containers are using.
• don't change cniVersion, it doesn't exactly follow the result of calicoctl version.

3.2. /opt/cni/bin/

mkdir -p /opt/cni/bin/
curl -L -o /opt/cni/bin/calico https://github.com/projectcalico/cni-plugin/releases/download/v3.4.0/calico-amd64
chmod 755 /opt/cni/bin/calico
curl -L -o /opt/cni/bin/calico-ipam https://github.com/projectcalico/cni-plugin/releases/download/v3.4.0/calico-ipam-amd64
chmod 755 /opt/cni/bin/calico-ipam

Do's and don'ts

• check calico version by calicoctl version, revise version in download url if necessary.

3.3. /etc/docker/cni.yaml

oci_bin: /usr/sbin/runc

cni_conf_dir: /etc/cni/net.d/
cni_bin_dir: /opt/cni/bin/
cni_ifname: eth0
cni_log: /var/log/cni.log

log_driver: file:///var/log/docker-cni.log
log_level: debug

Do's and don'ts:

• check oci_bin, make sure the runc binary is there. For some extremely old server on which runc doesn't exist, contact zc for solutions.

4. Check CNI

Create a container using CNI:

docker run -it --rm --runtime barrel-cni --net none bash bash

Then run ip a inside the container, check if the ip address is allocated from specified ippool.

Then press ctrl-d to exit container, and run the following command to make sure ip address has been released:

calicoctl ipam show --ip=$ipv4

5. Enable CNI in Barrel

5.1. Configure

Typically barrel configuration is located at /etc/eru/barrel.conf, then add additional line inside the file:

# /etc/eru/barrel.conf
BARREL_ENABLE_CNI=1

5.2. Restart

Then restart barrel service by systemctl restart eru-barrel. This is supposed to be pain free.

5.3 Eru-agent

Then check if eru-agent is using barrel to inspect containers:

# /etc/eru/agent.yaml

docker:
  #endpoint: unix:///var/run/docker.sock # bad setting
  endpoint: unix:///var/run/barrel.sock  # good one

Restarting eru-agent is also required if eru-agent configuration changed.

That's all.