Cloud integration documentation update

content/en/providers/cloud-compute/_index.md

@@ -5,44 +5,46 @@ weight: 30
 type: "docs"
 ---
 
-This documentation covers how to join the EGI Cloud federation as a provider. If
-you are interested in joining please first contact EGI operations team at
-[[email protected]](mailto:[email protected]), expressing interest and
-providing few details about:
-
-- the projects you may be involved in as cloud provider.
-
-- the user communities you want to support (AKA Virtual Organisations or VOs).
-  You can also support the 'long-tail of science' through the vo.access.egi.eu VO.
-
-- the technologies (Cloud Management Framework) you want to provide.
-
+This documentation covers how to join the EGI Cloud federation (also known as
+Federated Cloud or FedCloud) as a provider.  If you are interested in joining
+please first contact EGI at [[email protected]](mailto:[email protected]), expressing
+interest and providing few details about:
+
+- the projects you may be involved in as a cloud provider
+- the user communities you want to support (i.e.
+  [Virtual Organisations](https://confluence.egi.eu/display/EGIG/Virtual+organisation)
+  or VOs). You can also support the ’long-tail of science’ through the
+  vo.access.egi.eu VO
+- the technologies (Cloud Management platforms) you want to provide
 - details on the current status of your deployment (to be installed or already
-  installed, already used or not, how it is used, who uses the services...)
-
-Integration of cloud stacks into EGI FedCloud follows a well-defined path, with
-certain steps which need to be taken, depending on the cloud stack in question.
-By integration here, we refer to the proper interoperation with EGI
-infrastructure services such as accounting, monitoring, authentication and
-authorisation, etc. These configurations make your site discoverable and
-usable by the communities you wish to support, and allow EGI to support you in
-operational and technical matters.
-
-Integration of these services implies specific configuration actions which you
-need to take on your site. These aim to be unintrusive and are mostly to
-facilitate access to your site by the communities you wish to support, without
-interfering with normal operations. This can be summarised essentially as:
-
-1. Network configuration
-1. Permissions configuration
-1. AAI configuration
-1. Accounting configuration
-1. Information system integration
-1. VM and appliance repository configuration
+  installed, already used or not, how it is used, who uses the services, etc).
+
+Integration of cloud stacks into EGI FedCloud follows a well-defined path,
+depending on the particularities of the cloud stack in question. By integration,
+we refer to the proper interoperation with EGI infrastructure services such as
+accounting, monitoring, authentication and authorisation, etc. These
+configurations make your site discoverable and usable by the communities you
+wish to support, and allow EGI to support you in operational and technical
+matters.
+
+The series of actions that needs to be taken to achieve Cloud integration is
+summarised below:
+
+1. **Registration and certification** of the site as a
+   [Federated Resource Centre](../joining/federated-resource-centre/). Briefly a
+   site parses several intermediate stages - Registered, Candidate,
+   Uncertified - until fully Certified.
+1. [**Requirements**](./requirements/) - hardware and software, network
+   configuration
+1. **Registration of endpoints in the Configuration Database**
+   ([GOCDB](../../internal/configuration-database/)), a service provided by EGI
+   as a central registry to record information about the EGI Infrastructure
+   topology and which contains general information about all participating sites
+1. **Authentication and Authorization Integration** (AAI), including the setup
+   of an `ops` local project that will be used for monitoring your site
+1. **Accounting** configuration
+1. **Installation validation**
 
 If at any time you experience technical difficulties or need support, please
-[open a ticket](https://ggus.eu) or discuss the matter with us
-[on the forum](https://community.egi.eu)
-
-You can follow find integration guides for your cloud management in this
-documentation.
+[open a ticket](https://ggus.eu/). Dedicated integration sessions are available
+also on request.

content/en/providers/cloud-compute/openstack/_index.md

@@ -3,17 +3,17 @@ title: "OpenStack"
 weight: 20
 type: "docs"
 description: >
-  Integration of OpenStack providers
+  Integration with OpenStack
 ---
 
-This manual provides information on how to set up a Resource Centre providing
+This section provides information on how to set up a Resource Centre providing
 cloud resources in the EGI infrastructure. Integration with FedCloud requires a
 _working OpenStack installation_ as a pre-requirement. EGI supports any recent
 [OpenStack version](https://releases.openstack.org) (tested from OpenStack
 Mitaka).
 
-EGI expects the following OpenStack services to be available and accessible from
-outside your site:
+The following OpenStack services are expected to be available and accessible
+from outside the site:
 
 - Keystone
 - Nova
@@ -22,112 +22,37 @@ outside your site:
 - Neutron
 - Swift (if providing Object Storage)
 
-FedCloud components are distributed through
-[CMD (Cloud Middleware Distribution)](https://confluence.egi.eu/display/EGIBG/Cloud+Middleware+Distribution)
-These docker containers come pre-packaged and ready to use in the EGI FedCloud
-Appliance so you do not need to install any extra components on your site but
-just run a VM and configure it appropriately to interact with your services.
-
 The integration is performed by a set of EGI components that interact with the
 OpenStack services APIs:
 
-![image](openstacksite.png)
-
 - Authentication of EGI users into your system is performed by configuring the
-  native OpenID Connect support of Keystone. Support for legacy VOs using VOMS
-  requires the installation of the **Keystone-VOMS Authorization plugin** to
-  allow users with a valid VOMS proxy to obtain tokens to access your OpenStack
-  deployment.
+  native OpenID Connect support of Keystone
 - **cASO** collects accounting data from OpenStack and uses **SSM** to send the
   records to the central accounting database on the EGI Accounting service
-  ([APEL](https://apel.github.io/))
-- **cloud-info-provider** registers the RC configuration and description through
-  the [Messaging service](../../../internal/messaging) to facilitate service
-  discovery
-- **cloudkeeper** (and **cloudkeeper-os**) synchronises with
-  [EGI AppDB](https://appdb.egi.eu/browse/cloud) so new or updated images can be
-  provided by the RC to user communities (VO).
+  ([APEL](https://apel.github.io/)). cASO and SSM are operated by the site.
+- EGI runs the cloud-info-provider and cloudkeeper instances to provide
+  information discovery and VM image synchronisation
 
-Not all EGI components need to share the same credentials. They are individually
-configured, you can use different credentials and permissions if desired.
+![image](openstacksite.png)
 
 ## Installation options
 
-EGI distributes the integration components as:
-
-- A Virtual Appliance (VA) that uses Docker containers to bundle all of the
-  components in a single VM and just needs minor configuration to get started
-- RPM and DEB Packages in the
-  [CMD distribution](https://confluence.egi.eu/display/EGIBG/Cloud+Middleware+Distribution)
-
-### FedCloud Virtual Appliance
-
-The EGI FedCloud Appliance is available at
-[AppDB](https://appdb.egi.eu/store/vappliance/fedcloud.integration.appliance.openstack)
-as an OVA file. You can easily extract the VMDK disk by untaring and optionally
-converting it to your preferred format with qemu-img:
-
-```shell
-# get image and extract VMDK
-$ curl $(curl "https://appdb.egi.eu/store/vm/image/fc90d1aa-b0ae-46a0-b457-96f6f7a7d446:7875/json?strict" \
-    | jq -r .url) \
-    | tar x "*.vmdk"
-# convert to qcow2
-$ qemu-img convert -O qcow2 FedCloud-Appliance.Ubuntu.*.vmdk fedcloud-appliance.qcow2
-```
-
-The appliance running at your OpenStack must:
-
-- Have a host certificate to send the accounting information to the accounting
-  repository. DN of the host certificate must be registered in GOCDB with
-  service type `eu.egi.cloud.accounting`. The host certificate and key in PEM
-  format are expected in `/etc/grid-security/hostcert.pem` and
-  `/etc/grid-security/hostkey.pem` respectively.
-- Have enough disk space for handling the VM image replication (\~ 100GB for
-  `fedcloud.egi.eu` VO). By default these are stored at /image_data. You can
-  mount a volume at that location.
-
-### Upgrading the OpenStack Appliance
-
-#### From 2018.05.07 or newer to 2021.03.12
-
-Configuration changes:
-
-- Removes BDII, service is no longer in use
-- A cloud-info-provider cron is added
-- Uses AMS for pushing accounting records. New configuration file for ssmsend is
-  available
-
-#### From 2017.08.09 to 2018.05.07
-
-Configuration changes:
-
-- This upgrade moves the `voms.json` file to the respective `caso` and
-  `cloudkeeper-os` directories under `/etc/`
-- No other changes in configuration are needed
+cASO and SSM releases can be be obtained from GitHub:
 
-#### From 20160403 to 2017.08.09
-
-There are several major changes between these versions, namely:
-
-- atrope has been deprecated and cloudkeeper is used instead. The configuration
-  cannot be reused directly and the new services need to be configured as
-  described above
-- caso is upgraded to version 1.1.1, the configuration file has some
-  incompatible changes.
-- A new bdii.service is available for managing the process is available.
-
-### CMD Packages
-
-The CMD-OS repository provides packages that have gone through a quality
-assurance process for the supported distributions. Packages are available via
-the [EGI repository](https://repository.egi.eu).
+- [cASO](https://github.com/IFCA-Advanced-Computing/caso/releases). cASO
+  containers are available in the
+  [fedcloud-caso](https://github.com/EGI-Federation/fedcloud-catchall-operations/pkgs/container/fedcloud-caso)
+  package from
+  [fedcloud-catchall-operations](https://github.com/EGI-Federation/fedcloud-catchall-operations)
+- [SSM](https://github.com/apel/ssm/releases).
 
 ## Open Ports
 
+### Inbound
+
 The following **services** must be accessible to allow access to an
 OpenStack-based FedCloud site (default ports listed below, can be adjusted to
-your installation)
+your installation).
 
 <!-- markdownlint-disable line-length -->
 
@@ -141,36 +66,91 @@ your installation)
 
 <!-- markdownlint-enable line-length -->
 
-### Outgoing ports
+### Outbound
 
 The EGI Cloud components require the following outgoing connections open:
 
 <!-- markdownlint-disable line-length -->
 
-| Port         | Host                    | Note                                                            |
-| ------------ | ----------------------- | --------------------------------------------------------------- |
-| **443**/TCP  | `msg.argo.grnet.gr`     | ARGO Messaging System (used to send accounting records by SSM). |
-| **8443**/TCP | `msg.argo.grnet.gr`     | AMS authentication (used to send accounting records by SSM).    |
-| **443**/TCP  | `vmcaster.appdb.egi.eu` | AppDB image lists (used by cloudkeeper).                        |
-| **8080**/TCP | `cephrgw01.ifca.es`     | Swift server hosting EGI images (used by cloudkeeper).          |
+| Port         | Host                | Note                                                            |
+| ------------ | ------------------- | --------------------------------------------------------------- |
+| **443**/TCP  | `msg.argo.grnet.gr` | ARGO Messaging System (used to send accounting records by SSM). |
+| **8443**/TCP | `msg.argo.grnet.gr` | AMS authentication (used to send accounting records by SSM).    |
 
 <!-- markdownlint-enable line-length -->
 
-Images listed in AppDB may be hosted in other servers besides
-`cephrgw01.ifca.es`. Check the specific VO-wide image lists for details.
+## Users
 
-## Permissions
+### Local Users
 
-This is an overview of the expected account permissions used in an OpenStack
-site, these accounts can be merged as needed for your deployment:
+In order to get accounting information from your OpenStack, cASO needs to be run
+with a user that is a member of the projects to extract accounting information
+from and it's allowed to access `identity:list_users` and
+`identity:list_projects` in Keystone. Check
+[cASO documentation](https://caso.readthedocs.io/en/stable/configuration.html#user-credentials-required)
+for further information.
 
-<!-- markdownlint-disable line-length -->
+### Federated Users
 
-| Component    | Permission                                                                                   |
-| ------------ | -------------------------------------------------------------------------------------------- |
-| cloud-info   | Member of all projects supporting EGI VOs                                                    |
-| accounting   | Member of all projects and able to list users (allowed to `identity:list_users` in keystone) |
-| cloud-keeper | Permission to manage the images for all the projects supporting EGI VOs                      |
-| Other users  | Automatically created by Keystone and permission set as configured in the mappings           |
+Regular user accounts will be managed by the
+[Federated Identity](https://docs.openstack.org/keystone/latest/admin/federation/federated_identity.html)
+features of OpenStack. These users are created into a specific OpenStack domain
+for every configured identity provider. All users within the `egi.eu` domain
+will have a unique username. For users whose community identity is managed by
+Check-in, this identifier is of the form `<uniqueID>@egi.eu`. The `<uniqueID>`
+portion is an opaque identifier issued by Check-in, for example:
 
-<!-- markdownlint-enable line-length -->
+```shell
+$ openstack domain list
++----------------------------------+----------------------------------+---------+---------------------------------------------------------------+
+| ID                               | Name                             | Enabled | Description                                                   |
++----------------------------------+----------------------------------+---------+---------------------------------------------------------------+
+| 0125ed0ebc8045a49ed8c34c2a78740d | 0125ed0ebc8045a49ed8c34c2a78740d | True    | Auto generated federated domain for Identity Provider: egi.eu |
+| default                          | Default                          | True    | The default domain                                            |
++----------------------------------+----------------------------------+---------+---------------------------------------------------------------+
+
+$ openstack user list --domain 0125ed0ebc8045a49ed8c34c2a78740d
++------------------------------------------------------------------+-------------------------------------------------------------------------+
+| ID                                                               | Name                                                                    |
++------------------------------------------------------------------+-------------------------------------------------------------------------+
+| 2c096b11a1410d44e3936fa40479ad26eaa649cfd6887f06b3c6669e5d6c03d0 | [email protected]                   |
+| 933c692b53192e4d893e5ed5c026aa444acb4d75f6ee6c304422861207ce1ea5 | e9c37aa0d1XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX2867bc43581b835c@egi.eu |
+| d52112709a37975903576f80f37dde4604d1a227c53cb1fef43c45981673640c | 529a87e5ceXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXe714cb1309cc3907@egi.eu |
++------------------------------------------------------------------+-------------------------------------------------------------------------+
+```
+
+If you have set the email of the user in the mapping, you will be able to also
+get this information:
+
+```shell
+$ openstack user show d52112709a37975903576f80f37dde4604d1a227c53cb1fef43c45981673640c
++---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| Field               | Value                                                                                                                                                      |
++---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| domain_id           | 0125ed0ebc8045a49ed8c34c2a78740d                                                                                                                           |
+| email               | [email protected]                                                                                                                                   |
+| enabled             | True                                                                                                                                                       |
+| federated           | [{'idp_id': 'egi.eu', 'protocols': [{'protocol_id': 'openid', 'unique_id': '529a87e5ceXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXe714cb1309cc3907%40egi.eu'}]}] |
+| id                  | d52112709a37975903576f80f37dde4604d1a227c53cb1fef43c45981673640c                                                                                           |
+| name                | 529a87e5ceXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXe714cb1309cc3907@egi.eu                                                                                    |
+| options             | {}                                                                                                                                                         |
+| password_expires_at | None                                                                                                                                                       |
++---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+
+```
+
+Every VO has a VO identity card available via the
+[Operations Portal](https://operations-portal.egi.eu/vo/a/list), where you can
+also get contact information for the VO managers.
+
+VMs created by
+[EGI's Infrastructure Manager](../../../users/compute/orchestration/im/) have
+additional metadata properties that can help to identify the workload:
+
+```shell
+$ openstack server show 0f3e1420-4480-4bea-95f1-9920a70b324d -c properties -f yaml
+properties:
+  eu.egi.cloud.orchestrator: es.upv.grycap.im
+  eu.egi.cloud.orchestrator.id: 0afdc7ba-bf5d-11ed-9e89-86ce117c3fcf
+  eu.egi.cloud.orchestrator.url: https://appsgrycap.i3m.upv.es:31443/im
+  eu.egi.cloud.orchestrator.user: __OPENID__XXXXXXredacted
+```