Ceph: workloads adoption with cinder volume

Enable back the cinder volume commands on the source cloud, and resume testing of the ceph-backed volume attached to the test VM. Extend volume/backup/snapshot/attachment commands to wait for the previous step results. Follow the EDPM Post Ceph steps of HCI VA to prepare adopted workloads for using Ceph backend on EDPM. Add Nova discover host command (step 5 of the HCI VA). Add Nova Ceph custom configs to properly configure ceph vms pool for libvirt. Combine nova-ceph related configurations and nova FFU related ones into a single nova-compute-extraconfig service (by design, having two dataplane services for Nova in the same node set is not supported). Note about available choises for libvirt storage backends for Nova Add nova_libvirt_backend to control either to deploy with the local or ceph storage EDPM backends Signed-off-by: Bohdan Dobrelia <[email protected]>
openstack-k8s-operators · Apr 19, 2024 · a0ca760 · a0ca760
1 parent 7938110
commit a0ca760
Show file tree

Hide file tree

Showing 12 changed files with 296 additions and 33 deletions.
diff --git a/docs_dev/assemblies/development_environment.adoc b/docs_dev/assemblies/development_environment.adoc
@@ -108,13 +108,16 @@ export EDPM_COMPUTE_ADDITIONAL_NETWORKS=$(jq -c . /tmp/addtional_nets.json)
 export STANDALONE_COMPUTE_DRIVER=ironic
 export NTP_SERVER=pool.ntp.org  # Only necessary if not on the RedHat network ...
 export EDPM_COMPUTE_CEPH_ENABLED=false  # Optional
+export EDPM_COMPUTE_CEPH_NOVA=false # Optional
 export EDPM_COMPUTE_SRIOV_ENABLED=false # Without this the standalone deploy fails when compute driver is ironic.
 ----
 
 [Note]
 ===
 If `EDPM_COMPUTE_CEPH_ENABLED=false` is set, TripleO configures `Glance` with
 `Swift` as a backend.
+If `EDPM_COMPUTE_CEPH_NOVA=false` is set, TripleO configures `Nova/Libvirt` with
+a local storage backend.
 ===
 '''
 
@@ -293,19 +296,20 @@ ping -c 4 $(openstack server show baremetal-test -f json -c addresses | jq -r .a
 
 ==== Virtual Machine Steps
 
+Create a test VM instance with a test volume attachement:
+
 [,bash]
 ----
 cd ~/data-plane-adoption
 bash tests/roles/development_environment/files/pre_launch.bash
 ----
 
+This also creates a test Cinder volume, a backup from it, and a snapshot of it.
+
 '''
 
 ==== Ceph Storage Steps
 
-Make sure a cinder-volume backend is properly configured, or skip below steps
-to create a test workload without volume attachments.
-
 Confirm the image UUID can be seen in Ceph's images pool.
 
 [,bash]
@@ -318,22 +322,6 @@ Create a Barbican secret
 openstack secret store --name testSecret --payload 'TestPayload'
 ```
 
-Create a Cinder volume, a backup from it, and snapshot it.
-
-[,bash]
-----
-openstack volume create --image cirros --bootable --size 1 disk
-openstack volume backup create --name backup disk
-openstack volume snapshot create --volume disk snapshot
-----
-
-Add volume to the test VM
-
-[,bash]
-----
-openstack server add volume test disk
-----
-
 == Performing the Data Plane Adoption
 
 The development environment is now set up, you can go to the https://openstack-k8s-operators.github.io/data-plane-adoption/[Adoption

diff --git a/docs_user/modules/proc_adopting-compute-services-to-the-data-plane.adoc b/docs_user/modules/proc_adopting-compute-services-to-the-data-plane.adoc
@@ -7,8 +7,42 @@
 .Prerequisites
 
 * Remaining source cloud xref:stopping-infrastructure-management-and-compute-services_{context}[Stopping infrastructure management and Compute services] on Compute hosts.
+* Ceph backend for Nova/Libvirt is configured xref:configuring-a-ceph-backend_{context}[Configuring a Ceph backend].
 * Make sure the IPAM is configured:
 +
+----
+PODIFIED_DB_ROOT_PASSWORD=$(oc get -o json secret/osp-secret | jq -r .data.DbRootPassword | base64 -d)
+
+alias openstack="oc exec -t openstackclient -- openstack"
+FIP=192.168.122.20
+declare -A computes
+export computes=(
+  ["standalone.localdomain"]="192.168.122.100"
+  # ...
+)
+----
+
+== Pre-checks
+
+* Make sure the IPAM is configured
+
+----
+PODIFIED_DB_ROOT_PASSWORD=$(oc get -o json secret/osp-secret | jq -r .data.DbRootPassword | base64 -d)
+CEPH_FSID=$(oc get secret ceph-conf-files -o json | jq -r '.data."ceph.conf"' | base64 -d | grep fsid | sed -e 's/fsid = //'
+
+alias openstack="oc exec -t openstackclient -- openstack"
+FIP=192.168.122.20
+declare -A computes
+export computes=(
+  ["standalone.localdomain"]="192.168.122.100"
+  # ...
+)
+----
+
+== Pre-checks
+
+* Make sure the IPAM is configured
+
 ----
 oc apply -f - <<EOF
 apiVersion: network.openstack.org/v1beta1
@@ -80,6 +114,7 @@ EOF
 * Define the shell variables necessary to run the script that runs the fast-forward upgrade:
 ----
 PODIFIED_DB_ROOT_PASSWORD=$(oc get -o json secret/osp-secret | jq -r .data.DbRootPassword | base64 -d)
+CEPH_FSID=$(oc get secret ceph-conf-files -o json | jq -r '.data."ceph.conf"' | base64 -d | grep fsid | sed -e 's/fsid = //'
 
 alias openstack="oc exec -t openstackclient -- openstack"
 FIP=192.168.122.20
@@ -156,7 +191,7 @@ rm -f id*
 cd -
 ----
 
-. Create a `nova-compute-extra-config` service:
+. Create a `nova-compute-extra-config` service (no ceph backend in use):
 +
 [source,yaml]
 ----
@@ -190,6 +225,61 @@ EOF
 The secret `nova-cell<X>-compute-config` is auto-generated for each
 `cell<X>`. You must specify `nova-cell<X>-compute-config` and `nova-migration-ssh-key` for each custom `OpenStackDataPlaneService` related to the Compute service.
 
+That service removes pre-FFU workarounds and configures Nova compute
+services for local storage backend.
+
+* Or, create a `nova-compute-extra-config` service service (with ceph backend):
++
+[source,yaml]
+----
+oc apply -f - <<EOF
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: nova-compute-extraconfig
+  namespace: openstack
+data:
+  19-nova-compute-cell1-workarounds.conf: |
+    [workarounds]
+    disable_compute_service_check_for_ffu=true
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: ceph-nova
+  namespace: openstack
+data:
+  03-ceph-nova.conf: |
+    [libvirt]
+    images_type=rbd
+    images_rbd_pool=vms
+    images_rbd_ceph_conf=/etc/ceph/ceph.conf
+    images_rbd_glance_store_name=default_backend
+    images_rbd_glance_copy_poll_interval=15
+    images_rbd_glance_copy_timeout=600
+    rbd_user=openstack
+    rbd_secret_uuid=$CEPH_FSID
+---
+apiVersion: dataplane.openstack.org/v1beta1
+kind: OpenStackDataPlaneService
+metadata:
+  name: nova-compute-extraconfig
+  namespace: openstack
+spec:
+  label: nova.compute.extraconfig
+  configMaps:
+    - nova-compute-extraconfig
+    - ceph-nova
+  secrets:
+    - nova-cell1-compute-config
+    - nova-migration-ssh-key
+  playbook: osp.edpm.nova
+EOF
+----
++
+That service removes pre-FFU workarounds and configures Nova compute
+services for Ceph storage backend.
+
 ifeval::["{build}" == "downstream"]
 . Create a secret for the subscription manager and a secret for the Red Hat registry:
 +
@@ -215,6 +305,11 @@ EOF
 ----
 endif::[]
 
+* Deploy OpenStackDataPlaneNodeSet:
+These should be cell-specific configurations.
+For multi-cell, config maps and OpenStack dataplane services should be named like `nova-custom-ceph-cellX` and `nova-compute-extraconfig-cellX`.
+
+* Deploy OpenStackDataPlaneNodeSet:
 . Deploy the `OpenStackDataPlaneNodeSet` CR:
 +
 [source,yaml]
@@ -393,6 +488,41 @@ endif::[]
 EOF
 ----
 +
+* Prepare adopted EDPM workloads to use Ceph backend for Cinder, if configured so
++
+[source,yaml]
+----
+oc patch osdpns/openstack --type=merge --patch "
+spec:
+  services:
+    - repo-setup
+    - download-cache
+    - bootstrap
+    - configure-network
+    - validate-network
+    - install-os
+    - configure-os
+    - run-os
+    - install-certs
+    - ceph-client
+    - libvirt
+    - nova-compute-extraconfig
+    - ovn
+    - neutron-metadata
+  nodeTemplate:
+    extraMounts:
+    - extraVolType: Ceph
+      volumes:
+      - name: ceph
+        secret:
+          secretName: ceph-conf-files
+      mounts:
+      - name: ceph
+        mountPath: "/etc/ceph"
+        readOnly: true
+"
+----
++
 * Replace `<bridge_mappings>` with the value of the bridge mappings in your configuration, for example, `"datacentre:br-ctlplane"`.
 
 . Ensure that the `ovn-controller` settings that are configured in the `OpenStackDataPlaneNodeSet` CR are the same as were set in the Compute nodes before adoption. This configuration is stored in the `external_ids`` column in the `Open_vSwitch` table in the Open vSwitch database:
@@ -403,6 +533,10 @@ ovs-vsctl list Open .
 external_ids        : {hostname=standalone.localdomain, ovn-bridge=br-int, ovn-bridge-mappings=<bridge_mappings>, ovn-chassis-mac-mappings="datacentre:1e:0a:bb:e6:7c:ad", ovn-encap-ip="172.19.0.100", ovn-encap-tos="0", ovn-encap-type=geneve, ovn-match-northd-version=False, ovn-monitor-all=True, ovn-ofctrl-wait-before-clear="8000", ovn-openflow-probe-interval="60", ovn-remote="tcp:ovsdbserver-sb.openstack.svc:6642", ovn-remote-probe-interval="60000", rundir="/var/run/openvswitch", system-id="2eec68e6-aa21-4c95-a868-31aeafc11736"}
 ...
 ----
+
+Note that you should retain the original `OpenStackDataPlaneNodeSet` services
+composition, except the inserted `ceph-client` service.
+
 +
 * Replace `<bridge_mappings>` with the value of the bridge mappings in your configuration, for example, `"datacentre:br-ctlplane"`.
 

diff --git a/docs_user/modules/proc_performing-a-fast-forward-upgrade-on-compute-services.adoc b/docs_user/modules/proc_performing-a-fast-forward-upgrade-on-compute-services.adoc
@@ -109,6 +109,12 @@ oc exec -it nova-cell0-conductor-0 -- nova-manage db online_data_migrations
 oc exec -it nova-cell1-conductor-0 -- nova-manage db online_data_migrations
 ----
 
+. Discover Nova compute hosts in the cell:
++
+----
+oc rsh nova-cell0-conductor-0 nova-manage cell_v2 discover_hosts --verbose
+----
+
 . Verify if Compute services can stop the existing test VM instance:
 +
 ----

diff --git a/tests/playbooks/test_rollback_with_ceph.yaml b/tests/playbooks/test_rollback_with_ceph.yaml
@@ -9,6 +9,7 @@
     manila_backend: cephfs
     cinder_volume_backend: ceph
     cinder_backup_backend: ceph
+    nova_libvirt_backend: ceph
   module_defaults:
     ansible.builtin.shell:
       executable: /bin/bash

diff --git a/tests/playbooks/test_with_ceph.yaml b/tests/playbooks/test_with_ceph.yaml
@@ -9,6 +9,7 @@
     manila_backend: cephfs
     cinder_volume_backend: ceph
     cinder_backup_backend: ceph
+    nova_libvirt_backend: ceph
   module_defaults:
     ansible.builtin.shell:
       executable: /bin/bash

diff --git a/tests/playbooks/test_with_ironic.yaml b/tests/playbooks/test_with_ironic.yaml
@@ -25,6 +25,7 @@
       executable: /bin/bash
   vars:
     ironic_adoption: true
+    nova_libvirt_backend: local
   roles:
     - role: development_environment
     - role: backend_services

diff --git a/tests/roles/dataplane_adoption/defaults/main.yaml b/tests/roles/dataplane_adoption/defaults/main.yaml
@@ -123,3 +123,5 @@ neutron_public_interface_name: "{{ dataplane_public_iface | default('eth0') }}"
 edpm_sshd_allowed_ranges: "{{ ['192.168.122.0/24'] if dataplane_os_net_config_set_route|default(true)|bool else ['0.0.0.0/0'] }}"
 edpm_neutron_sriov_agent_enabled: true
 edpm_neutron_dhcp_agent_enabled: true
+
+nova_libvirt_backend: local