Skip to content

Commit

Permalink
Merge pull request #166 from dceara/simplify-deployment
Browse files Browse the repository at this point in the history 
Simplify deployment and some minor cleanups.
  • Loading branch information
@dceara
dceara authored Jun 2, 2023
2 parents 978f284 + a8d51b4 commit 2b179a0
Show file tree
Hide file tree
Showing 20 changed files with 136 additions and 319 deletions.
25 changes: 21 additions & 4 deletions .cirrus.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,17 +8,35 @@ low_scale_task:
image: family/ubuntu-2210-amd64
platform: linux
memory: 8G
disk: 20
disk: 40

env:
DEPENDENCIES: git ansible podman podman-docker
DEPENDENCIES: git ansible podman
FEDORA_DEP: ansible-collection-ansible-posix
ansible-collection-ansible-utils
CIRRUS_WORKING_DIR: /root/ovn-heater
PHYS_DEPLOYMENT: ${CIRRUS_WORKING_DIR}/physical-deployments/ci.yml

runtime_cache:
folder: runtime-cache

# Make sure we use all the disk available to us. Similar to the steps
# described at:
# https://cloud.google.com/compute/docs/disks/resize-persistent-disk
#
# Use "findmnt -n -o SOURCE /" and parse outputs of the form:
# /dev/<device><partition-number>
# OR
# /dev/<device><partition-number>[text]
resize_disk_script:
- d=$(findmnt -n -o SOURCE / | cut -f 1 -d '[' | cut -f 3 -d '/' | grep -oE '[a-zA-Z]+')
- p=$(findmnt -n -o SOURCE / | cut -f 1 -d '[' | cut -f 3 -d '/' | grep -oE '[0-9]+')
- t=$(df --output=fstype /root | grep -v Type)
- growpart /dev/$d $p || true
- '[ "$t" = "ext4" ] && resize2fs /dev/$d$p || true'
- '[ "$t" = "xfs" ] && xfs_growfs -d /root || true'
- '[ "$t" = "btrfs" ] && btrfs filesystem resize max /root || true'

configure_ssh_script:
- mkdir -p /root/.ssh/
- ssh-keygen -t rsa -N '' -q -f /root/.ssh/id_rsa
Expand All @@ -37,16 +55,15 @@ low_scale_task:

unpack_caches_script:
- mkdir -p runtime runtime-cache
- docker load -i runtime-cache/ovn-multi-node.tar || true
- tar -xzf runtime-cache/runtime.tar.gz || true
- podman load -i runtime/ovn-fake-multinode/ovn-multi-node-image.tar || true

install_script:
- 'sed -i "s/<host>/$(hostname)/g" ${PHYS_DEPLOYMENT}'
- ./do.sh install

pack_caches_script:
- rm -rf runtime-cache/*
- docker save -o runtime-cache/ovn-multi-node.tar ovn/ovn-multi-node:latest
- tar -czf runtime-cache/runtime.tar.gz runtime

upload_caches:
Expand Down
56 changes: 5 additions & 51 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@ a simulated OVN cluster deployed with
[ovn-fake-multinode](https://github.com/ovn-org/ovn-fake-multinode).

**NOTE**: This script is designed to be used on test machines only. It
performs disruptive changes to the machines it is run on (e.g., create
insecure docker registries, cleanup existing docker containers).
performs disruptive changes to the machines it is run on (e.g.,cleanup
existing containers).

# Prerequisites

Expand All @@ -21,9 +21,6 @@ insecure docker registries, cleanup existing docker containers).
- provisions all other `OVN` nodes with the required software packages
and with the correct version of `ovn-fake-multinode` to run simulated/fake
`OVN` chassis.
- runs a docker registry where the `ovn-fake-multinode` (i.e.,
`ovn/ovn-multi-node`) and `ovn-tester` images are pushed and from which all
other `OVN` nodes will pull the image.

* TESTER: One machine to run the `ovn-tester` container which runs the python
ovn-tester code. Like the ORCHESTRATOR, the TESTER also needs to be able to
Expand Down Expand Up @@ -63,7 +60,7 @@ Therefore, the values need to be resolvable by both of these entities and
need to resolve to the same host. `localhost` will not work since this does
not resolve to a unique host.

## Minimal requirements on the ORCHESTRATOR node (tested on Fedora 32)
## Minimal requirements on the ORCHESTRATOR node (tested on Fedora 38 and Ubuntu 22.10)

### Install required packages:

Expand All @@ -78,46 +75,6 @@ dnf install -y git ansible \
sudo apt -y install ansible
```

## Minimal requirements on the TESTER node (tested on Fedora 36)

### Make docker work with nested containers (disable cgroup hierarchy):

#### RPM-based Fedora 32+
```
dnf install -y grubby
grubby --update-kernel=ALL --args="systemd.unified_cgroup_hierarchy=0"
reboot
```

#### DEB-based
Edit /etc/default/grub and add `systemd.unified_cgroup_hierarchy=0` at the
end of the `GRUB_CMDLINE_LINUX_DEFAULT` variable.

```
sudo update-grub
sudo reboot
````
## Minimal requirements on the OVN-CENTRAL and OVN-WORKER-NODEs
### Make docker work with nested containers (disable cgroup hierarchy):
#### RPM-based Fedora 32+
```
dnf install -y grubby
grubby --update-kernel=ALL --args="systemd.unified_cgroup_hierarchy=0"
reboot
```
#### DEB-based
Edit /etc/default/grub and add `systemd.unified_cgroup_hierarchy=0` at the
end of the `GRUB_CMDLINE_LINUX_DEFAULT` variable.
```
sudo update-grub
sudo reboot
````
# Installation

## Ensure all nodes can be accessed passwordless via SSH
Expand All @@ -144,9 +101,6 @@ A sample file written for the deployment described above is available at
`physical-deployments/physical-deployment.yml`.

The file should contain the following mandatory sections and fields:
- `registry-node`: the hostname (or IP) of the node that will store the
docker private registry. In usual cases this is should be the ORCHESTRATOR
machine.
- `internal-iface`: the name of the Ethernet interface used by the underlay
(DB and tunnel traffic). This can be overridden per node if needed.
- `tester-node`:
Expand Down Expand Up @@ -284,7 +238,7 @@ cd ~/ovn-heater

This executes `<scenario>` on the physical deployment (specifically on the
`ovn-tester` container on the TESTER). Current scenarios also cleanup the
environment, i.e., remove all docker containers from all physical nodes.
environment, i.e., remove all containers from all physical nodes.
**NOTE**: If the environment needs to be explictly cleaned up, we can also
execute before running the scenario:

Expand All @@ -301,7 +255,7 @@ consist of:
stored.
- html reports
- a copy of the `hosts` ansible inventory used for the test.
- OVN docker container logs (i.e., ovn-northd, ovn-controller, ovs-vswitchd,
- OVN container logs (i.e., ovn-northd, ovn-controller, ovs-vswitchd,
ovsdb-server logs).
- physical nodes journal files.
- perf sampling results if enabled
Expand Down
85 changes: 26 additions & 59 deletions do.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,15 +16,11 @@ clustered_db=${CLUSTERED_DB:-True}
ovn_fmn_utils=${topdir}/ovn-fake-multinode-utils
ovn_fmn_playbooks=${ovn_fmn_utils}/playbooks
ovn_fmn_generate=${ovn_fmn_utils}/generate-hosts.py
ovn_fmn_docker=${ovn_fmn_utils}/generate-docker-cfg.py
ovn_fmn_podman=${ovn_fmn_utils}/generate-podman-cfg.py
ovn_fmn_get=${ovn_fmn_utils}/get-config-value.py
ovn_fmn_ip=${rundir}/ovn-fake-multinode/ip_gen.py
ovn_fmn_translate=${ovn_fmn_utils}/translate_yaml.py
hosts_file=${rundir}/hosts
installer_log_file=${rundir}/installer-log
docker_daemon_file=${rundir}/docker-daemon.json
podman_registry_file=${rundir}/registries.conf
log_collector_file=${rundir}/log-collector.sh
log_perf_file=${rundir}/perf.sh
process_monitor_file=${rundir}/process-monitor.py
Expand Down Expand Up @@ -104,8 +100,6 @@ function generate() {
mkdir -p ${rundir}

PYTHONPATH=${topdir}/utils ${ovn_fmn_generate} ${phys_deployment} ${rundir} ${ovn_fmn_repo} ${ovn_fmn_branch} > ${hosts_file}
PYTHONPATH=${topdir}/utils ${ovn_fmn_docker} ${phys_deployment} > ${docker_daemon_file}
PYTHONPATH=${topdir}/utils ${ovn_fmn_podman} ${phys_deployment} > ${podman_registry_file}
cp ${ovn_fmn_utils}/process-monitor.py ${process_monitor_file}
cp ${ovn_fmn_utils}/scripts/log-collector.sh ${log_collector_file}
cp ${ovn_fmn_utils}/scripts/perf.sh ${log_perf_file}
Expand All @@ -115,16 +109,16 @@ function install_deps_local_rpm() {
echo "-- Installing local dependencies"
yum install redhat-lsb-core datamash \
python3-pip python3-netaddr python3 python3-devel \
podman podman-docker \
podman \
--skip-broken -y
[ -e /usr/bin/pip ] || ln -sf /usr/bin/pip3 /usr/bin/pip

}

function install_deps_local_deb() {
echo "-- Installing local dependencies"
apt -y install datamash podman podman-docker python3-pip \
python3-netaddr python3 python3-all-dev
apt -y install datamash podman python3-pip \
python3-netaddr python3 python3-all-dev python3-venv
}

function install_deps_remote() {
Expand All @@ -133,28 +127,6 @@ function install_deps_remote() {
-i ${hosts_file}
}

function run_registry() {
containers=$(docker ps --all --filter='name=(ovn|registry)' \
| grep -v "CONTAINER ID" | awk '{print $1}' || true)
for container_name in $containers
do
docker stop $container_name
docker rm $container_name
done
[ -d /var/lib/registry ] || mkdir /var/lib/registry -p
docker run --privileged -d --name registry -p 5000:5000 \
-v /var/lib/registry:/var/lib/registry --restart=always docker.io/library/registry:2

# This is requried on the orchestrator for local image build/push to work
cp /etc/containers/registries.conf /etc/containers/registries.conf.bak
cat > /etc/containers/registries.conf << EOF
[registries.insecure]
registries = ['localhost:5000']
[registries.block]
registries = []
EOF
}

function install_venv() {
pushd ${rundir}
if [ ! -f ${ovn_heater_venv}/bin/activate ]; then
Expand All @@ -170,19 +142,6 @@ function install_venv() {
popd
}

function configure_docker() {
echo "-- Configuring local registry on tester nodes"
if which podman
then
echo "-- Configuring podman local registry on all nodes"
ansible-playbook ${ovn_fmn_playbooks}/configure-podman-registry.yml -i ${hosts_file}
else
echo "-- Configuring docker local registry on all nodes"
ansible-playbook ${ovn_fmn_playbooks}/configure-docker-registry.yml -i ${hosts_file}
fi

}

function clone_component() {
local comp_name=$1
local comp_repo=$2
Expand Down Expand Up @@ -277,7 +236,7 @@ function install_ovn_fake_multinode() {
[ -n "$RPM_OVN_HOST" ] && wget $RPM_OVN_HOST
fi

docker images | grep -q 'ovn/ovn-multi-node' || rebuild_needed=1
podman images | grep -q 'ovn/ovn-multi-node' || rebuild_needed=1

if [ ${rebuild_needed} -eq 1 ]; then
if [ -z "${OS_IMAGE_OVERRIDE}" ]; then
Expand All @@ -295,23 +254,22 @@ function install_ovn_fake_multinode() {
fi

# Build images locally.
OS_IMAGE=$os_image OVS_SRC_PATH=${rundir}/ovs OVN_SRC_PATH=${rundir}/ovn EXTRA_OPTIMIZE=${EXTRA_OPTIMIZE} USE_OVSDB_ETCD=${USE_OVSDB_ETCD} ./ovn_cluster.sh build
OS_IMAGE=$os_image OVS_SRC_PATH=${rundir}/ovs OVN_SRC_PATH=${rundir}/ovn \
EXTRA_OPTIMIZE=${EXTRA_OPTIMIZE} USE_OVSDB_ETCD=${USE_OVSDB_ETCD} \
RUNC_CMD=podman ./ovn_cluster.sh build
fi
# Tag and push image
docker tag ovn/ovn-multi-node localhost:5000/ovn/ovn-multi-node
docker push localhost:5000/ovn/ovn-multi-node

popd
}

function install_ovn_tester() {
ssh_key=$(${ovn_fmn_get} ${phys_deployment} tester-node ssh_key)
# We need to copy the files into a known directory within the Docker
# context directory. Otherwise, Docker can't find the files we reference.
# We need to copy the files into a known directory within the container
# runtime context directory. Otherwise, podman can't find the files we
# reference.
cp ${ssh_key} .
ssh_key_file=${rundir_name}/$(basename ${ssh_key})
docker build -t ovn/ovn-tester --build-arg SSH_KEY=${ssh_key_file} -f ${topdir}/Dockerfile ${topdir}
docker tag ovn/ovn-tester localhost:5000/ovn/ovn-tester
docker push localhost:5000/ovn/ovn-tester
podman build -t ovn/ovn-tester --build-arg SSH_KEY=${ssh_key_file} -f ${topdir}/Dockerfile ${topdir}
}

# Prepare OVS bridges and cleanup containers.
Expand All @@ -320,13 +278,24 @@ function init_ovn_fake_multinode() {
ansible-playbook ${ovn_fmn_playbooks}/deploy-minimal.yml -i ${hosts_file}
}

# Pull image on all nodes
# Pull image on all nodes.
function pull_ovn_fake_multinode() {
# Pull image on all nodes
echo "-- Saving the ovn/ovn-multi-node image and pulling it on all nodes."

pushd ${rundir}/ovn-fake-multinode
rm -f ovn-multi-node-image.tar
podman save --format oci-archive -o ovn-multi-node-image.tar \
ovn/ovn-multi-node:latest
ansible-playbook ${ovn_fmn_playbooks}/pull-fake-multinode.yml -i ${hosts_file}
popd
}

function pull_ovn_tester() {
echo "-- Saving the ovn/ovn-tester image and pulling it on the tester."

rm -f ovn-tester-image.tar
podman save --format oci-archive -o ovn-tester-image.tar \
ovn/ovn-tester:latest
ansible-playbook ${ovn_fmn_playbooks}/pull-ovn-tester.yml -i ${hosts_file}
}

Expand All @@ -342,9 +311,7 @@ function install() {
die_distro
fi
install_deps_remote
run_registry
install_venv
configure_docker
install_ovn_fake_multinode
init_ovn_fake_multinode
pull_ovn_fake_multinode
Expand Down Expand Up @@ -498,7 +465,7 @@ function run_test() {
fi

tester_host=$(${ovn_fmn_get} ${phys_deployment} tester-node name)
if ! ssh root@${tester_host} docker exec \
if ! ssh root@${tester_host} podman exec \
ovn-tester python3 -u /ovn-tester/ovn_tester.py \
/physical-deployment.yml /test-scenario.yml ;
then
Expand Down
45 changes: 0 additions & 45 deletions ovn-fake-multinode-utils/generate-docker-cfg.py
View file

This file was deleted.

Loading

0 comments on commit 2b179a0

Please sign in to comment.