Merge pull request #186 from RedHat-EMEA-SSA-Team/devel - 2021-12-17

Merge devel into master - 2021-12-17

README.md

@@ -99,16 +99,20 @@ We are now ready to install `libvirt` as our hypervisor, provision VMs and prepa
 
 Here is an example about [_cluster.yml_](cluster-example.yml) file that contains information about the cluster that is going to be installed.
 
-| variable | describtion  |
-|---|---|
-|cluster_name  |Name of the cluster to be installed |
-|public_domain  |Root domain that will be used for your cluster.  |
-|ip_families|Decide whether you want IPv4, IPv6 or dual-stack, detault: "['IPv4']"|
-|public_ip  |Override for public ip entries. defaults to `hostvars['localhost']['ansible_default_ipv4']['address']`. |
-|public_ipv6  |Override for public ip entries. defaults to `hostvars['localhost']['ansible_default_ipv6']['address']`. |
-|dns_provider  |DNS provider, value can be _route53_, _cloudflare_, _gcp_, _azure_,_transip_ or _none_. Check __Setup public DNS records__ for more info. |
-|letsencrypt_account_email  |Email address that is used to create LetsEncrypt certs. If _cloudflare_account_email_ is not present for CloudFlare DNS recods, _letsencrypt_account_email_ is also used with CloudFlare DNS account email |
-|image_pull_secret|Token to be used to authenticate to the Red Hat image registry. You can download your pull secret from https://cloud.redhat.com/openshift/install/metal/user-provisioned |
+| variable | description  |Default|
+|---|---|---|
+|`cluster_name`               |Name of the cluster to be installed | **Required** |
+|`dns_provider`               |DNS provider, value can be _route53_, _cloudflare_, _gcp_, _azure_,_transip_ or _none_. Check __Setup public DNS records__ for more info. | **Required** |
+|`image_pull_secret`          |Token to be used to authenticate to the Red Hat image registry. You can download your pull secret from https://cloud.redhat.com/openshift/install/metal/user-provisioned | **Required** |
+|`letsencrypt_account_email`  |Email address that is used to create LetsEncrypt certs. If _cloudflare_account_email_ is not present for CloudFlare DNS recods, _letsencrypt_account_email_ is also used with CloudFlare DNS account email |  **Required** |
+|`public_domain`              |Root domain that will be used for your cluster.  | **Required** |
+|`ip_families`                |Decide whether you want IPv4, IPv6 or dual-stack. | `['IPv4']` |
+|`listen_address`             |Listen address for the load balancer on your host system.  |`hostvars['localhost']['ansible_default_ipv4']['address']` |
+|`listen_address_ipv6`        |Same as listen_address but for IPv6 |`hostvars['localhost']['ansible_default_ipv6']['address']`|
+|`public_ip`                  |Optional to overwrite public ip, if it is different from `listen_address`. Used for dns records at your dns_provider. | `listen_address` |
+|`public_ipv6`                |Same as `public_ip` but for IPv6 | `listen_address_ipv6` |
+|`masters_schedulable`        |Optional to overwrite masters schedulable| `false` |
+|`sdn_plugin_name`            |Optional to change the SDN plugin between `OVNKubernetes` or `OpenShiftSDN` | `OVNKubernetes` |
 
 ### Cluster design (single node, compact or normal)
 
@@ -152,8 +156,8 @@ masters_schedulable: false
 
 ### Setup public DNS records
 
-Current tools allow use of three DNS providers: _AWS Route53_, _Cloudflare_, _GCP DNS_ or _none_.
-If you want to use _Route53_, _Cloudflare_ or _GCP_ as your DNS provider, you have to add a few variables. Check the instructions below.
+Current tools allow use of three DNS providers: _AWS Route53_, _Cloudflare_, _DigitalOcean_, _GCP DNS_ or _none_.
+If you want to use _Route53_, _Cloudflare_, _DigitalOcean_ or _GCP_ as your DNS provider, you have to add a few variables. Check the instructions below.
 
 DNS records are constructed based on _cluster_name_ and _public_domain_ values. With above values DNS records should be
 - api._cluster_name_._public_domain_
@@ -167,11 +171,12 @@ Please configure in `cluster.yml` all necessary credentials:
 
 | DNS provider | Variables  |
 |---|---|
+|Azure|`azure_client_id: 'client_id'`<br/>`azure_secret: 'key'`<br/>`azure_subscription_id: 'subscription_id'`<br/>`azure_tenant: 'tenant_id'`<br/>`azure_resource_group: 'dns_zone_resource_group'` |
 |CloudFlare|`cloudflare_account_email: [email protected]` <br> Use the global api key here! (API-Token is not supported!) (Details in #86) <br>`cloudflare_account_api_token: 9348234sdsd894.....` <br>  `cloudflare_zone: domain.tld`|
-|Route53 / AWS|`aws_access_key: key` <br/>`aws_secret_key: secret` <br/>`aws_zone: domain.tld` <br/>|
+|DigitalOcean|`digitalocean_token: e7a6f82c3245b65cf4.....` <br>  `digitalocean_zone: domain.tld`|
 |GCP|`gcp_project: project-name `<br/>`gcp_managed_zone_name: 'zone-name'`<br/>`gcp_managed_zone_domain: 'example.com.'`<br/>`gcp_serviceaccount_file: ../gcp_service_account.json` |
-|Azure|`azure_client_id: 'client_id'`<br/>`azure_secret: 'key'`<br/>`azure_subscription_id: 'subscription_id'`<br/>`azure_tenant: 'tenant_id'`<br/>`azure_resource_group: 'dns_zone_resource_group'` |
 |Hetzner|`hetzner_account_api_token: 93543ade82AA$73.....` <br>  `hetzner_zone: domain.tld`|
+|Route53 / AWS|`aws_access_key: key` <br/>`aws_secret_key: secret` <br/>`aws_zone: domain.tld` <br/>|
 |TransIP|`transip_token: eyJ0eXAiOiJKV....` <br> `transip_zone: domain.tld`|
 |none|With `dns_provider: none` the playbooks will not create public dns entries. (It will skip letsencrypt too) Please create public dns entries if you want to access your cluster.|
 
@@ -181,6 +186,8 @@ Please configure in `cluster.yml` all necessary credentials:
 |---|---|---|
 |`storage_nfs`|false|Setup a local NFS server, create a Storage Class (with [nfs-subdir-external-provisioner](https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner) ) pointing to it, and use that StorageClass for the internal Registry Storage|
 |`vm_autostart`|false|Create cluster VMs with `autostart` enabled|
+|`vm_storage_backend`|`qcow2`|You can choose between default `qcow2` and `lvm` as storage backend.|
+|`vm_storage_backend_location`|empty|Important for vm_storage_backend lvm, please add the volume group for example `vg0`|
 |`auth_redhatsso`|empty|Install Red Hat SSO, checkout  [_cluster-example.yml_](cluster-example.yml) for an example |
 |`auth_htpasswd`|empty|Install htpasswd, checkout  [_cluster-example.yml_](cluster-example.yml) for an example |
 |`auth_github`|empty|Install GitHub IDP, checkout  [_cluster-example.yml_](cluster-example.yml) for an example |
@@ -210,10 +217,17 @@ Please configure in `cluster.yml` all necessary credentials:
 * [Disk management (add disk to vm, wipe node)](docs/disk-management.md)
 * [How to passthrough nvme or gpu (pci-passthrough](docs/pci-passthrough.md)
 * [How to install OKD](docs/how-to-install-okd.md)
+* [Virsh commands cheatsheet to manage KVM guest virtual machines](https://computingforgeeks.com/virsh-commands-cheatsheet/)
 
 # Useful commands
 
 | Problem | Command |
 |---|---|
 |Check haproxy connections| ```podman exec -ti openshift-4-loadbalancer-${cluster_name} ./watch-stats.sh```
 |Start cluster after reboot|```./ansible/04-start-cluster.yml```
+
+
+
+# Stargazers over time
+
+[![Stargazers over time](https://starchart.cc/RedHat-EMEA-SSA-Team/hetzner-ocp4.svg)](https://starchart.cc/RedHat-EMEA-SSA-Team/hetzner-ocp4)

ansible/00-provision-hetzner.yml

@@ -10,7 +10,7 @@
   tasks:
   - name: Add hetzner server to inventory
     add_host: 
-      name: "{{ hetzner_hostname }}"
+      name: "{{ hetzner_ip }}"
 
 - name: install hetzner server
   hosts: all
@@ -24,4 +24,3 @@
       name: provision-hetzner
     tags:
       - provision-hetzner
-

ansible/roles/letsencrypt/tasks/create-digitalocean.yml

@@ -0,0 +1,12 @@
+---
+
+- name: Create DNS record at DigitalOcean
+  community.digitalocean.digital_ocean_domain_record:
+    oauth_token: "{{ digitalocean_token }}"
+    state: present
+    domain: "{{ digitalocean_zone }}"
+    type: TXT
+    name: "{{ item.0.key | replace(public_domain, '') | regex_replace('\\.$', '') }}"
+    data: "{{ item.1 }}"
+    force_update: yes
+  loop: "{{ challenge_data_dns | default({}) | dict2items | subelements('value') }}"

ansible/roles/letsencrypt/tasks/destroy-digitalocean.yml

@@ -0,0 +1,12 @@
+---
+
+- name: Destroy DNS record at DigitalOcean
+  community.digitalocean.digital_ocean_domain_record:
+    oauth_token: "{{ digitalocean_token }}"
+    state: absent
+    domain: "{{ digitalocean_zone }}"
+    type: TXT
+    name: "{{ item.0.key | replace(public_domain, '') | regex_replace('\\.$', '') }}"
+    data: "{{ item.1 }}"
+    force_update: yes
+  loop: "{{ challenge_data_dns | default({}) | dict2items | subelements('value') }}"

ansible/roles/letsencrypt/tasks/main.yml

@@ -137,8 +137,6 @@
   loop: "{{ challenge_data_dns | default({}) | dict2items | subelements('value') }}"
   when: le_dns_provider == "transip" and sample_com_challenge is changed
 
-
-
 - name: DNS record info
   debug:
     msg: "{{ item.0.key }} TXT {{ item.1 }}"
@@ -148,7 +146,7 @@
 - name: Include DNS provider
   include: "create-{{ le_dns_provider }}.yml"
   when:
-  - le_dns_provider in ['hetzner']
+  - le_dns_provider in ['hetzner', 'digitalocean']
   - sample_com_challenge is changed
 
 - pause:
@@ -232,7 +230,6 @@
   loop: "{{ challenge_data_dns | default({}) | dict2items | subelements('value') }}"
   when: le_dns_provider == "azure" and sample_com_challenge is changed
 
-
 - name: Delete DNS record at TransIP
   uri:
     url: "https://api.transip.nl/v6/domains/{{ transip_zone }}/dns"
@@ -251,11 +248,10 @@
   loop: "{{ challenge_data_dns | default({}) | dict2items | subelements('value') }}"
   when: le_dns_provider == "transip" and sample_com_challenge is changed
 
-
 - name: Include DNS provider
   include: "destroy-{{ le_dns_provider }}.yml"
   when:
-  - le_dns_provider in ['hetzner']
+  - le_dns_provider in ['hetzner', 'digitalocean']
   - sample_com_challenge is changed
 
 - name: concat root ca and intermediate

ansible/roles/openshift-4-cluster/defaults/main.yml

@@ -15,6 +15,12 @@ ip_families:
 listen_address: "{{ hostvars['localhost']['ansible_default_ipv4']['address'] | default('') }}"
 listen_address_ipv6: "{{ hostvars['localhost']['ansible_default_ipv6']['address'] | default('') }}"
 
+# Default: qcow2
+# For more performance use LVM
+vm_storage_backend: qcow2
+# Only necessary for LVM, ignored by qcow but should be '/var/lib/libvirt/images/'
+vm_storage_backend_location:
+
 master_count: 3
 master_vcpu: 4
 master_memory_size: 16384
@@ -36,7 +42,7 @@ vm_autostart: false
 # Important: OpenShift version must match to RHEL CoreOS version!
 
 # reference to OpenShift version
-openshift_version: 4.8.2
+openshift_version: 4.9.5
 openshift_install_command: "/opt/openshift-install-{{ openshift_version }}/openshift-install"
 # dev-pre:
 # {{ openshift_mirror }}/pub/openshift-v4/clients/ocp-dev-preview
@@ -53,7 +59,7 @@ opm_download_url: "{{ openshift_location }}/opm-linux-{{ opm_version }}.tar.gz"
 opm_dest: "/opt/openshift-client-{{ openshift_client_version }}/"
 
 # reference to coreos qcow file
-coreos_version: 4.8.2
+coreos_version: 4.9.0
 coreos_download_url: "{{ openshift_mirror }}/pub/openshift-v4/dependencies/rhcos/{{ coreos_version.split('.')[:2]|join('.') }}/{{ coreos_version }}/rhcos-{{coreos_version}}-x86_64-qemu.x86_64.qcow2.gz"
 coreos_csum_url: "{{ openshift_mirror }}/pub/openshift-v4/dependencies/rhcos/{{ coreos_version.split('.')[:2]|join('.') }}/{{ coreos_version }}/sha256sum.txt"
 

ansible/roles/openshift-4-cluster/tasks/create-network.yml

@@ -15,19 +15,21 @@
   set_fact:
     vn_subnet_ipv6: "{{ hostvars['localhost']['ansible_default_ipv6']['address'].split(':')[:4] | join(':') | string}}:{{ '%x' % vn_subnet.split('.')[2] | int }}"
     ipv6_listen_public:
-      - "{{ public_ipv6 | default(listen_address_ipv6) }}"
+      - "{{ listen_address_ipv6 }}"
     ipv6_listen_private:
       - "{{ hostvars['localhost']['ansible_default_ipv6']['address'].split(':')[:4] | join(':') | string}}:{{ '%x' % vn_subnet.split('.')[2] | int }}::1"
   when: "'IPv6' in ip_families"
+  tags: always
 
 - name: Build IPv4 subnet
   set_fact:
     vn_subnet_ipv4:  "{{ vn_subnet.split('.')[:3] | join('.') }}"
     ipv4_listen_private:
       - "{{ vn_subnet.split('.')[:3] | join('.')}}.1"
     ipv4_listen_public:
-      - "{{ public_ip | default(listen_address) }}"
+      - "{{ listen_address }}"
   when: "'IPv4' in ip_families"
+  tags: always
 
 - name: Build list of nodes
   set_fact:
@@ -157,6 +159,5 @@
     path: /etc/hosts
     marker: "# {mark} ANSIBLE MANAGED BLOCK {{ cluster_name }}.{{ public_domain }}"
     block: |
-      {{ public_ip | default(listen_address) }} api.{{ cluster_name }}.{{ public_domain }}
+      {{ listen_address }} api.{{ cluster_name }}.{{ public_domain }}
   tags: public_dns
-  when: dns_provider == 'none'

ansible/roles/openshift-4-cluster/tasks/create-vm.yml

@@ -6,17 +6,36 @@
   command: "qemu-img create -f qcow2 -b {{ coreos_image_location }} /var/lib/libvirt/images/{{ vm_instance_name }}.qcow2 {{ vm_root_disk_size }}"
   args:
     creates: "/var/lib/libvirt/images/{{ vm_instance_name }}.qcow2"
+  when: vm_storage_backend == "qcow2"
+
+- name: Convert coreos qcow into raw image
+  command: "qemu-img convert {{ coreos_image_location }} -O raw {{ coreos_image_location }}.raw "
+  args:
+    creates: "{{ coreos_image_location }}.raw"
+  when: vm_storage_backend == "lvm"
+
+- name: Create logical volume
+  lvol:
+    vg: "{{ vm_storage_backend_location }}"
+    lv: "{{ vm_instance_name }}"
+    state: present
+    size: "{{ vm_root_disk_size }}"
+  when: vm_storage_backend == "lvm"
+
+- name: Copy/dd coreos into device
+  command: "dd if={{ coreos_image_location }}.raw of=/dev/{{ vm_storage_backend_location }}/{{ vm_instance_name }} bs=4M"
+  when: vm_storage_backend == "lvm"
 
 - name: Copy ignition {{ vm_instance_name }}
   copy:
     src: "{{ vm_ignition_file }}"
     dest: "/var/lib/libvirt/images/{{ vm_instance_name }}.ign"
     mode: '0644'
-  
-# - name: Debug - create /tmp/{{ vm_instance_name }}.virt.xml
-#   template:
-#     src: "vm.xml.j2"
-#     dest: "/tmp/{{ vm_instance_name }}.virt.xml"
+
+- name: Debug - create /tmp/{{ vm_instance_name }}.virt.xml
+  template:
+    src: "vm.xml.j2"
+    dest: "/tmp/{{ vm_instance_name }}.virt.xml"
 
 - name: Define VirtualMachine {{ vm_instance_name }}
   virt:
@@ -25,7 +44,7 @@
     xml: "{{ lookup('template', 'templates/vm.xml.j2') }}"
 
 - name: Start VirtualMachine {{ vm_instance_name }}
-  virt: 
+  virt:
     name: "{{ vm_instance_name }}"
     state: running
     autostart: "{{ vm_autostart }}"

ansible/roles/openshift-4-cluster/tasks/create.yml

@@ -65,6 +65,9 @@
     le_hetzner_account_api_token: "{{ hetzner_account_api_token }}"
     le_hetzner_zone: "{{ hetzner_zone }}"
 
+    le_digitalocean_token: "{{ digitalocean_token }}"
+    le_digitalocean_zone: "{{ digitalocean_zone }}"
+
   tags: letsencrypt
   when: letsencrypt_disabled == false
 

ansible/roles/openshift-4-cluster/tasks/destroy-network.yml

@@ -5,7 +5,8 @@
     tasks_from: destroy.yml
   vars:
     pd_provider: "{{ dns_provider }}"
-    pd_public_ip: "{{ listen_address }}"
+    pd_public_ip: "{% if 'IPv4' in ip_families %}{{ public_ip | default(listen_address) }}{% endif %}"
+    pd_public_ipv6: "{% if 'IPv6' in ip_families %}{{ public_ipv6 | default(listen_address_ipv6) }}{% endif %}"
     pd_cloudflare_account_api_token: "{{ cloudflare_account_api_token }}"
     pd_cloudflare_zone: "{{ cloudflare_zone }}"
     pd_aws_access_key: "{{ aws_access_key }}"
@@ -23,7 +24,6 @@
     marker: "# {mark} ANSIBLE MANAGED BLOCK {{ cluster_name }}.{{ public_domain }}"
     block: ""
   tags: public_dns
-  when: dns_provider == 'none'
 
 - name: Destroy OpenShift 4 load balancer
   import_role:

ansible/roles/openshift-4-cluster/tasks/destroy-vm.yml

@@ -11,10 +11,19 @@
     name: "{{ vm_instance_name }}"
   ignore_errors: yes
 
+- name: Delete logical volume
+  lvol:
+    vg: "{{ vm_storage_backend_location }}"
+    lv: "{{ vm_instance_name }}"
+    state: absent
+    force: yes
+  when: vm_storage_backend == "lvm"
+
 - name: Delete disk {{ vm_instance_name }}
   file:
     path: "/var/lib/libvirt/images/{{ vm_instance_name }}.qcow2"
     state: absent
+  when: vm_storage_backend == "qcow2"
 
 - name: Delete ignition {{ vm_instance_name }}
   file:

ansible/roles/openshift-4-cluster/tasks/post-install-storage-nfs.yml

@@ -36,8 +36,15 @@
     group: root
     mode: 0644
     block: |
+      {% if 'IPv4' in ip_families -%}
       {{ storage_nfs_path_prefix }}/{{ cluster_name }}-pv-infra-registry {{ vn_subnet }}/24(rw,sync,no_root_squash)
       {{ storage_nfs_path_prefix }}/{{ cluster_name }}-pv-user-pvs {{ vn_subnet }}/24(rw,sync,no_root_squash)
+      {% endif -%}
+      {% if 'IPv6' in ip_families -%}
+      {{ storage_nfs_path_prefix }}/{{ cluster_name }}-pv-infra-registry {{ vn_subnet_ipv6 }}::/80(rw,sync,no_root_squash)
+      {{ storage_nfs_path_prefix }}/{{ cluster_name }}-pv-user-pvs {{ vn_subnet_ipv6 }}::/80(rw,sync,no_root_squash)
+      {% endif -%}
+
   tags: exports
 
 - name: Adjust directory permissions
@@ -74,7 +81,7 @@
           storage: 100Gi
         nfs:
           path: "{{ storage_nfs_path_prefix }}/{{ cluster_name }}-pv-infra-registry"
-          server: "{{ vn_subnet.split('.')[:3] | join('.')}}.1"
+          server: "host.compute.local"
         persistentVolumeReclaimPolicy: Recycle
 
 - name: Create registry-storage pvc
@@ -277,13 +284,13 @@
                   - name: PROVISIONER_NAME
                     value: redhat-emea-ssa-team/hetzner-ocp4
                   - name: NFS_SERVER
-                    value: "{{ vn_subnet.split('.')[:3] | join('.')}}.1"
+                    value: "host.compute.local"
                   - name: NFS_PATH
                     value: "{{ storage_nfs_path_prefix }}/{{ cluster_name }}-pv-user-pvs"
             volumes:
               - name: nfs-client-root
                 nfs:
-                  server: "{{ vn_subnet.split('.')[:3] | join('.')}}.1"
+                  server: "host.compute.local"
                   path: "{{ storage_nfs_path_prefix }}/{{ cluster_name }}-pv-user-pvs"
 
 - name: Storage Class

ansible/roles/openshift-4-cluster/tasks/prepare-host.yml

@@ -38,3 +38,7 @@
     insertafter: '^\[Network\]'
     regexp: '^IPForward='
   when: stat_result.stat.exists == True
+
+# Install ansible collection for digitalocean dns provider
+- name: Ansible collection community.digitalocean
+  command: "ansible-galaxy collection install community.digitalocean"

ansible/roles/openshift-4-cluster/templates/network.xml.j2

@@ -8,19 +8,17 @@
             <hostname>host.{{ vn_internal_domain }}</hostname>
             <hostname>api-int.{{ vn_public_domain }}</hostname>
             <hostname>api.{{ vn_public_domain }}</hostname>
-            {% if dns_provider == 'none' %}
             <hostname>oauth-openshift.apps.{{ vn_public_domain }}</hostname>
-            {% endif %}
+            <hostname>console-openshift-console.apps.{{ vn_public_domain }}</hostname>
         </host>
         {% endif %}
         {% if 'IPv6' in ip_families  %}
         <host ip='{{ vn_subnet_ipv6 }}::1'>
             <hostname>host.{{ vn_internal_domain }}</hostname>
             <hostname>api-int.{{ vn_public_domain }}</hostname>
             <hostname>api.{{ vn_public_domain }}</hostname>
-            {% if dns_provider == 'none' %}
             <hostname>oauth-openshift.apps.{{ vn_public_domain }}</hostname>
-            {% endif %}
+            <hostname>console-openshift-console.apps.{{ vn_public_domain }}</hostname>
         </host>
         {% endif %}
     </dns>