From 96e5ebd2a1bddb403acc43cbedf0dd5cc50701f6 Mon Sep 17 00:00:00 2001 From: Baptiste Grenier Date: Thu, 12 Oct 2023 10:42:31 +0200 Subject: [PATCH 1/4] Fix dictionaries (#617) Some more fixes for spelling dictionaries. --- .github/actions/spelling/allow.txt | 2 +- .github/actions/spelling/expect.txt | 1 - .github/actions/spelling/patterns.txt | 2 ++ 3 files changed, 3 insertions(+), 2 deletions(-) diff --git a/.github/actions/spelling/allow.txt b/.github/actions/spelling/allow.txt index 5cb16ba2e5..671645b2a9 100644 --- a/.github/actions/spelling/allow.txt +++ b/.github/actions/spelling/allow.txt @@ -1,4 +1,3 @@ -a2enmod aai aaiproxy aarc @@ -274,6 +273,7 @@ HLR Horovod hostcert hostkey +HPC HPDA htc htcondor diff --git a/.github/actions/spelling/expect.txt b/.github/actions/spelling/expect.txt index 9b9cfd4308..c168b1a43b 100644 --- a/.github/actions/spelling/expect.txt +++ b/.github/actions/spelling/expect.txt @@ -1,6 +1,5 @@ CVMFS diracsgm -microk8s mytoken reponame sgm diff --git a/.github/actions/spelling/patterns.txt b/.github/actions/spelling/patterns.txt index b91c9b908a..72c0750561 100644 --- a/.github/actions/spelling/patterns.txt +++ b/.github/actions/spelling/patterns.txt @@ -147,3 +147,5 @@ mailto:[-a-zA-Z=;:/?%&0-9+@.]{3,} \bSPECcpu2006\b \bWLCG.*\b \bfrontends?\b +\bmicrok8s\b +\ba2enmod\b From ac3da05af7a823115c4c9d45c0f550bc7e8d3da1 Mon Sep 17 00:00:00 2001 From: Andrea Manzi Date: Thu, 12 Oct 2023 11:39:39 +0200 Subject: [PATCH 2/4] Remove data orchestrator and openrdm docs (#611) Remove data orchestrator and openrdm docs (for now). Co-authored-by: Baptiste Grenier --- content/en/users/data/management/_index.md | 26 +- .../users/data/management/open-rdm/_index.md | 33 -- .../en/users/data/management/rucio/_index.md | 82 ---- .../data/management/rucio/admin/_index.md | 424 ------------------ .../data/management/rucio/commands/_index.md | 198 -------- .../data/management/rucio/dteam-vo/_index.md | 108 ----- .../rucio/getting-started/_index.md | 158 ------- 7 files changed, 9 insertions(+), 1020 deletions(-) delete mode 100644 content/en/users/data/management/open-rdm/_index.md delete mode 100644 content/en/users/data/management/rucio/_index.md delete mode 100644 content/en/users/data/management/rucio/admin/_index.md delete mode 100644 content/en/users/data/management/rucio/commands/_index.md delete mode 100644 content/en/users/data/management/rucio/dteam-vo/_index.md delete mode 100644 content/en/users/data/management/rucio/getting-started/_index.md diff --git a/content/en/users/data/management/_index.md b/content/en/users/data/management/_index.md index d28e5f1023..0c06319bba 100644 --- a/content/en/users/data/management/_index.md +++ b/content/en/users/data/management/_index.md @@ -26,37 +26,29 @@ and can be accessed from most [compute services](../../compute). ## Generic data management -These higher-level data management services are available to researchers: +This higher-level data management service is available to researchers: -- [EGI Rucio](rucio) is tailored to medium/big scientific collaborations, allowing - users to organise, manage, and access their data at scale. Data can be distributed - across heterogeneous data centers at widely distributed locations. - [EGI DataHub](datahub) is a high-performance data management solution that offers unified data access across multiple types of underlying storage, allowing users to share, collaborate and easily perform computations on the stored data. ## Specialized data management -The following specialized data management services are also available: +The following specialized data management service is also available: - [EGI Data Transfer](data-transfer) is a low-level service to move data from one [Grid](../storage/grid-storage) or [Object](../storage/object-storage) storage - to another. It is used internally by [Rucio](rucio) to schedule transfers based - on the data policies defined by the users. -- [openRDM](open-rdm) is a combined [FAIR](https://en.wikipedia.org/wiki/FAIR_data) - data management platform, Electronic Laboratory Notebook (ELN) and Inventory Management System - allowing a complete overview of workflows and information, from initial data generation - to data analysis and publication. + to another. -## Use-cases for storing and managing research data +## Use-cases for storing research data Depending on the type of the employed compute services and the use-cases being addressed, users might need to choose different data service to store, access, and manage data. -| User | Data storage | Data management (optional) | -| -------------- | -------------------------------------------------------- | -------------------------- | -| **Cloud user** | Block and Object storage | DataHub | -| **HTC user** | Grid storage | Rucio | -| **HPC user** | High-performance parallel file systems or Object storage | DataHub or Rucio | +| User | Data storage | +| -------------- | -------------------------------------------------------- | +| **Cloud user** | Block and Object storage | +| **HTC user** | Grid storage | +| **HPC user** | High-performance parallel file systems or Object storage | The following sections offer detailed descriptions for each data management service. diff --git a/content/en/users/data/management/open-rdm/_index.md b/content/en/users/data/management/open-rdm/_index.md deleted file mode 100644 index 2f27511082..0000000000 --- a/content/en/users/data/management/open-rdm/_index.md +++ /dev/null @@ -1,33 +0,0 @@ ---- -title: openRDM -linkTitle: Research Data Management -type: docs -weight: 50 -weight: -description: > - Organise data in research projects with openRDM ---- - -While most other [data management services](..) are available in the EGI infrastructure, -there are specialized services in the EGI portfolio that are offered for in-house -installations by research communities, with support for customization and -configuration from EGI. - -## What is it? - -The [openRDM](https://marketplace.eosc-portal.eu/services/openrdm-eu) service, -an Research Data Management (RDM) tool, -offers **advanced organisation of data during ongoing research projects, -as an integrated environment with data management and digital lab notebook**. - -openRDM combines a data management platform with a digital lab notebook and a sample and -protocol management system. It enables scientists to meet the ever-increasing requirements -from funding agencies, journals, and academic institutions to publish data according to the -[FAIR](https://en.wikipedia.org/wiki/FAIR_data) data principles – according to which data -should be _Findable_, _Accessible_, _Interoperable_ and _Reusable_. - -{{% alert title="Note" color="info" %}} The openRDM service is based around the -active research data management (ARDM) platform [openBIS](https://openbis.ch/), see the -[documentation](https://openbis.ch/index.php/docs/user-documentation/) -for more details. -{{% /alert %}} diff --git a/content/en/users/data/management/rucio/_index.md b/content/en/users/data/management/rucio/_index.md deleted file mode 100644 index 29a9430bf3..0000000000 --- a/content/en/users/data/management/rucio/_index.md +++ /dev/null @@ -1,82 +0,0 @@ ---- -title: Rucio -linkTitle: Data Orchestrator -type: docs -weight: 40 -description: >- - Organise and access data at scale with Rucio ---- - -## What is it? - -Built on more than a decade of experience in LHC experiments, Rucio serves the -data needs of any modern scientific experiments. Rucio can manage large amounts -of data, countless numbers of files, heterogeneous storage systems, globally -distributed data centres, with monitoring and analytics. - -Rucio **allows management of data with expressive statements**. You to say what -you want, and Rucio will figure out the details of how to do it. For example, -three copies of my file on different continents with a backup on tape. You can -also automatically remove copies of data after a set period or once its access -popularity drops. - -While Rucio is extremely scalable, the STFC Rucio Data Management Service is -designed for smaller communities, with expected data needs up to tens of -Petabytes. The fact that the underlying Rucio infrastructure is managed by STFC, -allows communities to easily start using and/or test Rucio with little setup -cost. - -## Requirements to consider to use Rucio - -For Rucio to manage your data in this setup, Rucio will need X.509 certificate -access, or soon, through -[EGI Check-in](../../../../providers/check-in/_index.md) to: - -- Your experiments Storage Element that is X.509 capable -- Your experiments Tape Archive that is X.509 capable -- Your experiments VOMS server information to check Users credentials against - -Rucio is a system that sits on top of already established storage elements to -unify users access for data management, and retrieval. Rucio consists of a -database of the storage element's details, users and their access credential -information, access levels, the data, and its location, Rucio is not a direct -data storage solution. - -## Rucio Use Cases - -Rucio is a data management software, that integrates with your experiments -currently provisioned storage. This section will highlight some use cases that -Rucio will fill for your experiment. - -### Management of data for archive when no longer used actively - -A simple use case for Rucio is to manage data between 'hot' storage, made of -HDDs or SSD, and 'cold' storage made up of tape. When your experiment generates -data that data will be accessed much more than older data as your colleagues work -with the data. Within Rucio the data will be registered to be on the 'hot' and -'cold' storage. This ensures the integrity of the data by providing multiple -copies of the data, one on the slower tape archive, and one copy on the more -easy to access HDDs and SSDs. Then as the usage of this data declines, the data -on the 'hot' storage can be removed to make way for newer data that is more -frequently accessed. Should the archived data be requested again Rucio can stage -the data from tape back to disk making it available for users. - -### Management of data between sites for jobs - -Another useful use case for Rucio is to manage the data between different sites -within your experiment. This can provide users with better access to the data -that they want to work on. Another option is to have Rucio integrated with your -workflow management software (Panda and DIRAC both have integrated with Rucio), -so data can be moved to sites as job slots are available, streamlining the -data flow for the user. - -## Official Rucio Pages - -- [Rucio Homepage](https://rucio.cern.ch/) -- [Rucio Documentation](https://rucio.cern.ch/documentation/) - -## Multi-VO specific pages - -- [Multi-VO Rucio at RAL as a service](https://www.scd.stfc.ac.uk/Pages/SCD-STFC-Rucio-Data-Management-Service.aspx) -- [Privacy Policy](https://www.scd.stfc.ac.uk/Pages/STFC-Rucio-Privacy-Notice.aspx) -- [Acceptable Use Policy](https://www.scd.stfc.ac.uk/Pages/STFC-Rucio-Acceptable-Use-Policy.aspx) diff --git a/content/en/users/data/management/rucio/admin/_index.md b/content/en/users/data/management/rucio/admin/_index.md deleted file mode 100644 index 74b7500ef4..0000000000 --- a/content/en/users/data/management/rucio/admin/_index.md +++ /dev/null @@ -1,424 +0,0 @@ ---- -title: Administering Rucio -linkTitle: Administration -type: docs -weight: 30 -description: >- - Help Rucio admins understand and perform actions for their VO ---- - -Within Rucio there are several levels of administrators. There are the **super -admins**, which are the staff that runs Multi-VO Rucio. Then there are virtual -organisation (VO) specific admins that will look after the day-to-day operations -of their VO. Below are some of the tasks that VO admins will need to do to set -up and maintain their VO. - -## Creating Accounts, Identities, and Quotas - -To add new users within your VO, you will need to communicate with Rucio as the -VO admin. Then using the rucio-admin commands, create a new account and add -identities to the account. The account is a username with no permissions, or -authentication methods. The identities bind authentication methods and -permissions to the account. The account you want to create identities for is -input as an argument. Accounts will have different permissions and access (such -as how much data they can store on a particular -[RSE](https://rucio.cern.ch/documentation/started/concepts/rucio_storage_element/)). - -### CLI Example - -```shell -$ rucio-admin account add \ - --type USER \ - --email jdoe@email.com jdoe - -Added new account: jdoe - -$ rucio-admin identity add \ - --account jdoe \ - --type USER \ - --id userjdoe \ - --email jdoe@email.com \ - --password jdoepass - -Added new identity to account: userjdoe-jdoe - -$ rucio-admin account set-limits jdoe storagesite1 100GB - -Set account limit for account jdoe on RSE storagesite1: 100.000 GB -``` - -### Python Client Example - -```python ->>> from rucio.client import Client ->>> CLIENT = Client() ->>> CLIENT.add_account('jdoe', 'USER', 'jdoe@email.com') -True ->>> CLIENT.add_identity('jdoe', 'USER', 'jdoe@email.com') -True ->>> CLIENT.set_account_limit('jdoe', 'storagesite1', 107374182400, 'global') -True -``` - -## Creating RSE(s) - -[Rucio Storage Elements](https://rucio.cern.ch/documentation/started/concepts/rucio_storage_element/) -(RSEs) are how Rucio represents the physical storage available to your VO. As -with many aspects of Rucio there are a lot of optional attributes that can be -set for an RSE, but as a minimum a protocol for transfers need to be added -before it can be used. - -### Creating RSE(s) CLI Example - -```shell -$ rucio-admin rse add NEW_RSE - -Added new deterministic RSE: NEW_RSE - -$ rucio-admin rse add-protocol \ - --hostname test.org \ - --scheme gsiftp \ - --prefix '/filepath/rucio/' \ - --port 8443 NEW_RSE \ - --domain-json '{ - "wan": { - "read": 1, - "write": 1, - "third_party_copy": 0, - "delete": 1 - }, - "lan": { - "read": 1, - "write": 1, - "third_party_copy": 0, - "delete": 1 - } - }' -``` - -### Creating RSE(s) Python Client Example - -```python ->>> from rucio.client import Client ->>> CLIENT = Client() ->>> CLIENT.add_rse('NEW_RSE') -True ->>> CLIENT.add_protocol( - 'NEW_RSE', { - 'hostname': 'test.org', - 'scheme': 'gsiftp', - 'prefix': '/filepath/rucio/', - 'port': 8443, - 'impl': 'rucio.rse.protocols.gfalv2.Default', - 'domain': { - "wan": { - "read": 1, - "write": 1, - "third_party_copy": 0, - "delete": 1 - }, - "lan": { - "read": 1, - "write": 1, - "third_party_copy": 0, - "delete": 1 - } - } - } - ) -True -``` - -## Updating RSE Protocols - -On occasion, it may be necessary to change or update an RSE protocol. Unlike -settings (`rucio-admin rse update`) or attributes -(`rucio-admin rse set-attribute`), there isn't a direct CLI function for -changing a protocol. It would therefore be necessary to remove -(`rucio-admin rse delete-protocol`) and then add -(`rucio-admin rse add-protocol`) it again using different information. -Alternatively, the Python client has additional functionality to directly update -or swap the priority of RSE protocols. For example to update the `impl` without -changing anything else (the `data` argument is used to update the protocol, with -the other settings used to specify the protocol to change): - -```python ->>> from rucio.client import Client ->>> CLIENT = Client() ->>> CLIENT.update_protocols( - rse='NEW_RSE', - scheme='gsiftp', - data={ - 'impl': rucio.rse.protocols.gfal.Default' - }, - hostname='test.org', - port=8433 - ) -True -``` - -To swap the priority of two protocols for the third party copy operation: - -```python ->>> from rucio.client import Client ->>> CLIENT = Client() ->>> CLIENT.swap_protocols( - rse='NEW_RSE', - domain='wan', - operation='third_party_copy', - scheme_a='gsiftp', - scheme_b='root' - ) -True -``` - -It's also worth noting that when an RSE is deleted using -`rucio-admin rse delete`, the entry remains in the database. This "soft" -deletion means that attempting to add a new RSE with the same name as a deleted -RSE will fail. This is due to the RSE not having a unique name/VO combination. -In practice, it is therefore better to update a badly configured RSE rather than -attempting to delete and re-add it. However, if the latter method is preferred, -it is possible manually rename the deleted RSE in the database (as there are no -foreign key constraints on its name, just the ID and VO) so that the old name -can be re-used. - -## Basic Usage - -This section covers some of the basic Rucio functions that can be run once the -VO has accounts and RSEs set up. As with the setup, there are many options that -won't be covered here. For more information refer to either the main -documentation or the help for the function in question. - -### Daemons - -Most operations in Rucio (such as transfers, deletions, rule evaluation) require -one or more of the -[daemons](https://rucio.cern.ch/documentation/started/main_components#daemons) -to be running in order to take effect. For a multi-VO instance, these should be -running for all VOs already. However, on new VO's joining Rucio some updating of -the daemons will be necessary. - -If it seems like it is not quite right please contact the Rucio team through -[GGUS](https://ggus.eu/?mode=ticket_submit). - -## Uploading Data - -In Rucio files and their replicas are represented by Data Identifiers -([DIDs](https://rucio.cern.ch/documentation/started/concepts/file_dataset_container)), -which are composed of a scope and name. Furthermore, multiple files can be -attached to a dataset, which in turn can be attached to a container (which can -be attached to another container and so on). Datasets and containers are also -represented by DIDs. - -Scopes are always associated with a particular Rucio account, and must be added -to Rucio using an admin account. If no scope is provided when uploading, Rucio -will default to `user.`, but this still needs to have been added by an -admin. - -Once a file has been uploaded via the CLI or Python client, it can then be -attached to a dataset. It's worth noting that by default, some Rucio commands -will not list files, only datasets. - -### Uploading Data CLI Example - -Assuming the file `test.txt` exists locally: - -```shell -$ rucio-admin scope add --account root --scope user.root - -Added new scope to account: user.root-root - -$ rucio upload --rse NEW_RSE test.txt - -2020-08-14 15:28:15,059 INFO Preparing upload for file test.txt -2020-08-14 15:28:15,235 INFO Successfully added replica in Rucio catalogue at NEW_RSE -2020-08-14 15:28:15,334 INFO Successfully added replication rule at NEW_RSE -2020-08-14 15:28:15,579 INFO Trying upload with mock to NEW_RSE -2020-08-14 15:28:15,579 295 INFO Trying upload with mock to NEW_RSE -2020-08-14 15:28:15,580 INFO Successful upload of temporary file. mock://test.org:123/filepath/rucio/user/root/46/6b/test.txt.rucio.upload -2020-08-14 15:28:15,580 295 INFO Successful upload of temporary file. mock://test.org:123/filepath/rucio/user/root/46/6b/test.txt.rucio.upload -2020-08-14 15:28:15,580 INFO Successfully uploaded file test.txt -2020-08-14 15:28:15,580 295 INFO Successfully uploaded file test.txt -2020-08-14 15:28:15,583 295 DEBUG Starting new HTTPS connection (1): rucio:443 -2020-08-14 15:28:15,598 295 DEBUG https://rucio:443 "POST /traces/ HTTP/1.1" 201 7 -2020-08-14 15:28:15,662 295 DEBUG https://rucio:443 "PUT /replicas HTTP/1.1" 200 0 - -$ rucio list-dids user.root:test.txt --filter type=ALL -+--------------------+--------------+ -| SCOPE:NAME | [DID TYPE] | -|--------------------+--------------| -| user.root:test.txt | FILE | -+--------------------+--------------+ - -$ rucio add-dataset user.root:test_dataset - -Added user.root:test_dataset - -$ rucio attach user.root:test_dataset user.root:test.txt - -DIDs successfully attached to user.root:test_dataset - -$ rucio list-content user.root:test_dataset -+--------------------+--------------+ -| SCOPE:NAME | [DID TYPE] | -|--------------------+--------------| -| user.root:test.txt | FILE | -+--------------------+--------------+ -``` - -### Uploading Data Python Client Example - -Assuming the file `test.txt` exists locally: - -```python ->>> from rucio.client import Client ->>> CLIENT = Client() ->>> CLIENT.add_scope('root', 'user.root') - -True - ->>> from rucio.client.uploadclient import UploadClient ->>> UPLOAD_CLIENT = UploadClient() ->>> UPLOAD_CLIENT.upload([{'path': 'test.txt', 'rse': 'NEW_RSE'}]) - -2020-08-14 14:47:31,147 8431 DEBUG Starting new HTTPS connection (1): rucio:443 -2020-08-14 14:47:31,166 8431 DEBUG https://rucio:443 "POST /traces/ HTTP/1.1" 201 7 -2020-08-14 14:47:31,224 8431 DEBUG https://rucio:443 "PUT /replicas HTTP/1.1" 200 None -0 - ->>> list(CLIENT.list_dids('user.root', {}, type='all')) - -[u'test.txt'] ->>> CLIENT.add_dataset('user.root', 'test_dataset') - -True - ->>> CLIENT.attach_dids('user.root', 'test_dataset', [{'scope': 'user.root', 'name': 'test.txt'}]) ->>> list(CLIENT.list_content('user.root', 'test_dataset')) - -[{u'adler32': u'00000001', u'name': u'test.txt', u'bytes': 0, u'scope': u'user.root', u'type': u'FILE', u'md5': u'd41d8cd98f00b204e9800998ecf8427e'}] -``` - -## Adding Replication Rules - -Once a DID exists within the Rucio catalogue, replicas of that file, dataset or -collection are created and maintained by -[Replication rules](https://rucio.cern.ch/documentation/started/concepts/replica_management). -By uploading a file to a particular RSE, a replication rule is created for that -file, however rules can also be added for existing DIDs. As a minimum an RSE and -number of copies must be specified, but further options such as lifetime of the -rule and selecting RSEs based on user set attributes are also possible. - -### Adding Replication Rules CLI Example - -```shell - $ rucio list-rules --account root - -ID ACCOUNT SCOPE:NAME STATE[OK/REPL/STUCK] RSE_EXPRESSION COPIES EXPIRES (UTC) CREATED (UTC) --------------------------------- --------- ------------------ ---------------------- ---------------- -------- --------------- ------------------- -991f9ace7ed74cad989efde90b6a23c5 root user.root:test.txt OK[1/0/0] NEW_RSE 1 2020-08-14 15:28:15 -$ rucio add-rule user.root:test_dataset 1 NEW_RSE -bd51b767ef524878bb3cc68db16d2374 - - $ rucio list-rules --account root - -ID ACCOUNT SCOPE:NAME STATE[OK/REPL/STUCK] RSE_EXPRESSION COPIES EXPIRES (UTC) CREATED (UTC) --------------------------------- --------- ---------------------- ---------------------- ---------------- -------- --------------- ------------------- -991f9ace7ed74cad989efde90b6a23c5 root user.root:test.txt OK[1/0/0] NEW_RSE 1 2020-08-14 15:28:15 -bd51b767ef524878bb3cc68db16d2374 root user.root:test_dataset OK[1/0/0] NEW_RSE 1 2020-08-14 15:47:15 -``` - -### Adding Replication Rules Python Client Example - - - -```python ->>> from rucio.client import Client ->>> CLIENT = Client() ->>> list(CLIENT.list_account_rules('root')) - -[{u'locks_ok_cnt': 1, u'source_replica_expression': None, u'locks_stuck_cnt': 0, u'purge_replicas': False, u'rse_expression': u'NEW_RSE', u'updated_at': datetime.datetime(2020, 8, 14, 15, 28, 15), u'meta': None, -u'child_rule_id': None, u'id': u'991f9ace7ed74cad989efde90b6a23c5', u'ignore_account_limit': False, u'error': None, u'weight': None, u'locks_replicating_cnt': 0, u'notification': u'NO', u'copies': 1, u'comments': None, -u'split_container': False, u'priority': 3, u'state': u'OK', u'scope': u'user.root', u'subscription_id': None, u'stuck_at': None, u'ignore_availability': False, u'eol_at': None, u'expires_at': None, u'did_type': u'FILE', -u'account': u'root', u'locked': False, u'name': u'test.txt', u'created_at': datetime.datetime(2020, 8, 14, 15, 28, 15), u'activity': u'User Subscriptions', u'grouping': u'DATASET'}] - ->>> CLIENT.add_replication_rule([{'scope': 'user.root', 'name': 'test_dataset'}], 1, 'NEW_RSE') - -[u'76b262b45dca4e769221224e1ccf5c7a'] - ->>> list(CLIENT.list_account_rules('root')) - -[{u'locks_ok_cnt': 1, u'source_replica_expression': None, u'locks_stuck_cnt': 0, u'purge_replicas': False, u'rse_expression': u'NEW_RSE', u'updated_at': datetime.datetime(2020, 8, 14, 15, 28, 15), u'meta': None, -u'child_rule_id': None, u'id': u'991f9ace7ed74cad989efde90b6a23c5', u'ignore_account_limit': False, u'error': None, u'weight': None, u'locks_replicating_cnt': 0, u'notification': u'NO', u'copies': 1, u'comments': None, -u'split_container': False, u'priority': 3, u'state': u'OK', u'scope': u'user.root', u'subscription_id': None, u'stuck_at': None, u'ignore_availability': False, u'eol_at': None, u'expires_at': None, u'did_type': u'FILE', -u'account': u'root', u'locked': False, u'name': u'test.txt', u'created_at': datetime.datetime(2020, 8, 14, 15, 28, 15), u'activity': u'User Subscriptions', u'grouping': u'DATASET'}, {u'locks_ok_cnt': 1, -u'source_replica_expression': None, u'locks_stuck_cnt': 0, u'purge_replicas': False, u'rse_expression': u'NEW_RSE', u'updated_at': datetime.datetime(2020, 8, 14, 15, 47, 15), u'meta': None, u'child_rule_id': None, u'id': -u'bd51b767ef524878bb3cc68db16d2374', u'ignore_account_limit': False, u'error': None, u'weight': None, u'locks_replicating_cnt': 0, u'notification': u'NO', u'copies': 1, u'comments': None, u'split_container': False, u'priority': -3, u'state': u'OK', u'scope': u'user.root', u'subscription_id': None, u'stuck_at': None, u'ignore_availability': False, u'eol_at': None, u'expires_at': None, u'did_type': u'DATASET', u'account': u'root', u'locked': False, -u'name': u'test_dataset', u'created_at': datetime.datetime(2020, 8, 14, 15, 47, 15), u'activity': u'User Subscriptions', u'grouping': u'DATASET'}] -``` - - - -## Multi-VO Features - -From a users perspective, whether the instance is multi or single VO should not -change any functionality. Furthermore, depending on the client setup, VO does -not need to be provided. There are however, some occasions when an optional -argument for the VO can be given in a multi-VO instance. - -### Swapping VOs - -Just like how an identity can be associated with (and used to authenticate -against) multiple accounts, the same identity can be used for accounts at more -than one VO. Account and identity can be retrieved from the config file if -present, and the VO set there will be used (unless the environment variable -`RUCIO_VO` is also set, in which case the latter takes precedent). Both will be -ignored however if the VO is passed as an optional argument in the CLI or Python -client. Using this optional argument allows a user to quickly run commands on a -different VO they have access to. - -#### Swapping VOs CLI Example - -```shell -$ rucio whoami - -status : ACTIVE -account : jdoe_abc_account -account_type : USER -created_at : 2020-08-07T08:27:29 -updated_at : 2020-08-07T08:27:29 -suspended_at : None -deleted_at : None -email : N/A - -$ rucio --vo xyz whoami - -status : ACTIVE -account : jdoe_xyz_account -account_type : USER -created_at : 2020-08-11T12:13:58 -updated_at : 2020-08-11T12:13:58 -suspended_at : None -deleted_at : None -email : N/A -``` - -#### Swapping VOs Python Client Example - -```python ->>> from rucio.client import Client ->>> CLIENT = Client() ->>> CLIENT.whoami() - -{u'status': u'ACTIVE', u'account': u'jdoe_abc_account', u'account_type': u'USER', u'created_at': u'2020-08-07T08:27:29', u'updated_at': u'2020-08-07T08:27:29', u'suspended_at': None, u'deleted_at': None, u'email': u'N/A'} - ->>> CLIENT_XYZ = Client(vo='xyz') ->>> CLIENT_XYZ.whoami() - -{u'status': u'ACTIVE', u'account': u'jdoe_xyz_account', u'account_type': u'USER', u'created_at': u'2020-08-11T12:13:58', u'updated_at': u'2020-08-11T12:13:58', u'suspended_at': None, u'deleted_at': None, u'email': u'N/A'} -``` diff --git a/content/en/users/data/management/rucio/commands/_index.md b/content/en/users/data/management/rucio/commands/_index.md deleted file mode 100644 index e694c200f6..0000000000 --- a/content/en/users/data/management/rucio/commands/_index.md +++ /dev/null @@ -1,198 +0,0 @@ ---- -title: Rucio Command-Line Interface -linkTitle: Command Line -type: docs -weight: 20 -description: >- - The most common Rucio commands ---- - -## Introduction to Rucio commands - -There are many commands found within Rucio CLI that you may want to become -familiar with. In this guide I will provide a few of the common commands wanted -by new users. - -To find more of the commands that you may want to use type `rucio` into the -containerised client will provide all of the arguments for Rucio. Typing in the -command followed by `-h`, or `--help` will provide you with all the options that -are available as well as some explanation for each. - -### ping - -Is the simplest command that a user can use to ask the Rucio server which -version it is using. - -```shell -$ rucio ping -``` - -This checks that there is a connection between the containerised client and the -server. - -### whoami - -Another simple command, which asks the server for the information Rucio has on -the current user. - -```shell -$ rucio whoami -``` - -This will return output like the following: - -```shell -status : ACTIVE -account : user1 -account_type : USER -created_at : YYYY-MM-DDTHH:MM:SS -updated_at : YYYY-MM-DDTHH:MM:SS -suspended_at : None -deleted_at : None -email : myemail@domail.country -``` - -This ensures that you know which user you are interacting with Rucio as, this is -very important if you get multiple accounts. But also verifies that the client -is set up correctly. - -### upload - -A Rucio command that allows you to upload files from your current environment to -any RSE within your VO. - -```shell -$ rucio upload [-h] --rse RSE [--lifetime LIFETIME] [--scope SCOPE] - [--impl IMPL] [--register-after-upload] [--summary] - [--guid GUID] [--protocol PROTOCOL] [--pfn PFN] - [--name NAME] [--transfer-timeout TRANSFER_TIMEOUT] - [--recursive] - args [args ...] -``` - -Several of the options you will not need to use as they will be set by the Rucio -VO Admins when they set up the RSEs. Below are a list of options that you may -find useful: - -- `RSE` is the Rucio Storage Element or site that you wish to store the data at, - the list of available RSEs can be seen for your VO with the command - `rucio list-rses`. -- `Lifetime` is how long you wish the file to exist, not specifying will make - the file permanent until rucio is told to delete it. -- `Scope` can be used in many ways, but often can be an experiment name, or a - user space, all users have their own scope assigned to them `user.`. -- `Register-after-upload` allows for files to be uploaded to the destination, - and then registered with Rucio, rather than the other way around. This can be - useful if your connection is intermittent. -- `Name` is the name of the file that it will be registered to Rucio with, if - this is not set it will be the name of the file or files provided. -- `Recursive` Allows you to set the argument to a directory, and all files within - that directory and any subdirectories will be uploaded. -- `Args` is the path to the file, or files you wish to upload, this can be a - single file, directory (with recursive set), or a list of files separated with - a space e.g. - -```shell -rucio upload --rse main-rse file1 file2 file3 file4 -``` - -### get - -A Rucio command to download files from any RSE in your VO to your local -environment. - -```shell -$ rucio get [-h] [--dir DIR] [--allow-tape] [--rse RSE] [--rses RSES] - [--impl IMPL] [--protocol PROTOCOL] [--nrandom NRANDOM] - [--ndownloader NDOWNLOADER] [--no-subdir] [--pfn PFN] - [--archive-did ARCHIVE_DID] [--no-resolve-archives] - [--ignore-checksum] [--transfer-timeout TRANSFER_TIMEOUT] - [--transfer-speed-timeout TRANSFER_SPEED_TIMEOUT] [--aria] - [--filter FILTER] [--scope SCOPE] [--metalink METALINK_FILE] - [--deactivate-file-download-exceptions] - [dids [dids ...]] -``` - -- `Dir` is the location within the container you wish for the files to be - downloaded (if you wish to move these files outside of the container, you may - want to mount a volume in the container to allow the files to persist). -- `RSE(s)` specifying which RSE(s) you wish to download the files from, leaving - this blank will allow Rucio to decide which RSE(s) are best. -- `nrandom` allows you to specify a number and if the target is a dataset or - container will download n files from that DID. This allows you to check are - correct before committing to download the entire dataset or container. -- `dids` is the data identifier for the file, dataset or container you wish to - download. - -### add-rule - -Create a rule which Rucio will work to make true. These are often how files are -moved from site to site. Creating a rule that says file x (which is currently at -storagesite1), needs to be at storagesite2. Upon creation of the rule, Rucio -will ensure that the file is moved from where is closest to the new site. - -```shell -$ rucio add-rule [-h] [--weight WEIGHT] [--lifetime LIFETIME] - [--grouping {DATASET,ALL,NONE}] [--locked] - [--source-replica-expression SOURCE_REPLICA_EXPRESSION] - [--notify NOTIFY] [--activity ACTIVITY] - [--comment COMMENT] [--ask-approval] [--asynchronous] - [--delay-injection DELAY_INJECTION] - [--account RULE_ACCOUNT] [--skip-duplicates] - dids [dids ...] copies rse_expression -``` - -- `lifetime` How long you want the file to persist before it can be deleted by - Rucio. -- `locked` sets the dataset or container to a locked state, that prevents other - files from being added or removed. -- `dids` the files within Rucio you wish to be replicates. -- `copies` How many copies of the data you want to make. -- `rse_expression` can either be a specific RSE, or can be a filter. Expression, - such as `tape=True` or `country=UK` and Rucio will place as many copies as was - requested in different sites (when possible), to fulfil the rule. - -### delete-rule - -A command to delete a rule which you have created. Just because you have deleted -a rule does not mean the file will be deleted. But it will adjust your quota -accordingly. Other people within your VO may also have a rule that states the -file needs to be at the same site. - -```shell -$ rucio delete-rule [-h] [--purge-replicas] [--all] - [--rse_expression RSE_EXPRESSION] [--rses RSES] - [--account RULE_ACCOUNT] - rule_id -``` - -- `all` should not be used by users it it will attempt to delete all rules. -- `rse_expression` which RSE expression encapsulates the rules you wish to - delete, either rse_expression, or RSE needs to be specified. -- `RSES` exactly which RSE is the target of the rule deletion. -- `account` which account requires the rule to be deleted, this is generally only - needed by Rucio Admins and does not need to be specified if you are deleting - your own rules. -- `rule_id` is a Rucio specific ID for the file that you wish to be deleted, a - list of the rules, and their rule_ids that are within your account can be - retrieved by running `rucio list-rules --account youraccountname`. - -### list-rules - -A command to list all the rules related to an account, a DID, or a file. - -```shell -$ rucio list-rules [-h] [--id RULE_ID] [--traverse] [--csv] [--file FILE] - [--account RULE_ACCOUNT] - [--subscription ACCOUNT SUBSCRIPTION] - [did] -``` - -Provide a full list of the files IDs, account, scope, state, RSE/expression -copies and expiry. - -- `account` specify which account you wish to see the replication rules. -- `file` If you know the name of a specific file, this allows you to see all the - rules are associated with the file. -- `did` If a dataset or container are listed, all rules associated with the - specific DID will be displayed. diff --git a/content/en/users/data/management/rucio/dteam-vo/_index.md b/content/en/users/data/management/rucio/dteam-vo/_index.md deleted file mode 100644 index 54fa226c8a..0000000000 --- a/content/en/users/data/management/rucio/dteam-vo/_index.md +++ /dev/null @@ -1,108 +0,0 @@ ---- -title: "Dteam Specific Documentation" -type: docs -linkTitle: "Rucio Dteam" -weight: 40 -description: >- - How to get set up with the dteam VO ---- - -## Rucio-client setup - -The setup for the container is the same as that found in the [Getting Started](../getting-started/) -section. But is repeated here for ease. - -To get the Rucio client that is set up for dteam please use this -[Rucio Client](https://hub.docker.com/repository/docker/egifedcloud/rucioclient). -This would be done by running the command: - -```shell -$ docker run \ - -v :/opt/rucio/etc/rucio.cfg \ - -v :/opt/rucio/etc/usercert \ - -v :/opt/rucio/etc/usercert \ - -e RUCIO_CFG_CLIENT_CERT=/opt/rucio/etc/usercert.pem \ - -e RUCIO_CFG_CLIENT_KEY=/opt/rucio/etc/userkey.pem \ - -e RUCIO_CFG_CA_CERT=/opt/rucio/etc/web/ca-first.pem \ - --name=rucio-client \ - -it \ - -d egifedcloud/rucioclient:1.23.17 -``` - -Once the container is running you will need to copy some files, to have them -owned by the container user, rather then root, and then change the permissions -on those files so that they are appropriate for voms-proxy creation. To start -with step into the container by running: - -```shell -$ docker exec -it rucio-client bash -``` - -Once inside the container you can then copy and edit file permissions with the -following: - -```shell -$ cp /opt/rucio/etc/usercert /opt/rucio/etc/usercert.pem -$ cp /opt/rucio/etc/userkey /opt/rucio/etc/userkey.pem -$ chmod 600 /opt/rucio/etc/usercert.pem -$ chmod 400 /opt/rucio/etc/userkey.pem -``` - -You should now be able to generate a VOMS proxy using the credentials loaded -into the container, this is done by running the following command within the -container: - -```shell -$ voms-proxy-init --voms dteam -``` - -## Rucio configuration setup - -Inside your docker container edit the `rucio.cfg` file to include your 3 -character VO name, and account name. This will then be loaded into the Rucio -client. - -```ini -[common] -logdir = /var/log/rucio -multi_vo = True -loglevel = INFO -[client] -rucio_host = https://rucio-server.gridpp.rl.ac.uk:443 -auth_host = https://rucio-server.gridpp.rl.ac.uk:443 -vo = dtm -account = -ca_cert = /opt/rucio/etc/web/ca-first.pem -auth_type = x509_proxy -client_cert = /opt/rucio/etc/usercert.pem -client_key = /opt/rucio/etc/userkey.pem -client_x509_proxy = /tmp/x509up_u1000 -request_retries = 5 -``` - -## Confirmation of Client setup - -Once this is complete you should now have access to Rucio. This can be confirmed -with a ping and a whoami commands to verify one, the connection to the Rucio -host and two, that you are authenticating successfully as your user. - -```shell -$ rucio ping -1.23.17 -$ rucio whoami -status : ACTIVE -account : user -account_type : USER -created_at : YYYY-MM-DDThh:mm:ss -suspended_at : None -updated_at : YYYY-MM-DDThh:mm:ss -deleted_at : None -email : user@email.co.uk -``` - -Once these messages have been displayed with the relevent information, as a user -you should now have access to the Dteam VO, and can create rules, upload and -download files from the various RSEs. - -If you have any issues please do contact the -[Multi-VO admin / dteam VO admins](mailto:rucio-support@stfc365.onmicrosoft.com). diff --git a/content/en/users/data/management/rucio/getting-started/_index.md b/content/en/users/data/management/rucio/getting-started/_index.md deleted file mode 100644 index c06045d13a..0000000000 --- a/content/en/users/data/management/rucio/getting-started/_index.md +++ /dev/null @@ -1,158 +0,0 @@ ---- -title: Getting Started with Rucio -linkTitle: Getting Started -type: docs -weight: 10 -description: >- - How to get started with Rucio ---- - -## Rucio terms - -- [**Rucio Storage Element**](https://rucio.cern.ch/documentation/started/concepts/rucio_storage_element/) - (RSE) is another name for an endpoint, or storage solution. -- [**Rules**](https://rucio.cern.ch/documentation/started/concepts/replica_management) are an - instruction to Rucio to do a certain thing. This can be to ensure file _x_ has - at least 1 copy at _storagesite1_, or ensure file _y_ is on tape, or even on - tape at more than one location, or even file _z_ has 2 copies at any site - within a selection of sites. How you set up the RSE and the attributes you - give them allows for many different strategies to transfer and organise data. - Once a rule is created, Rucio will get to work to ensure that the rule is - satisfied at all times. -- **File** is single file within Rucio. -- **Dataset** is a collection of files, which may be a collection or related - results, or data. -- **Container** is a collection of Datasets which may build a larger subset of a - whole experiment. -- **Scope** is a collection in which files, datasets, and containers are placed. - Users will have their own scope, often user.username. But also experiments, - sub-experiments, or however you wish to organise the data can also have scopes. - Accounts can be given access to scopes by VO admins. -- **Data Identifier** (DID) uniquely identifies data in Rucio. It is made up - from the scope and the filename, separated by a colon (e.g. - _experiment1:file1_). - -## Getting started as a new user - -### Account creation - -1. To get set up with a Rucio account please create a ticket on - [GGUS](https://ggus.eu/?mode=ticket_submit). Please fill in the form with a - subject, description, ticket category - service request, priority - less - urgent, and under routing information please select Assign to support unit - - Rucio). Within the ticket description please include: - - - Desired Username (usually initials and surname e.g. John Doe would have - jdoe) - - Your email - - Name of the experiment / VO you are part of - - The subject of your eScience certificate - -**If you want password access we can organise a video call to explain or take -sensitive information if you prefer** -In Terms of testing you can join the test VO (dteam) to try Rucio as a service -and its capabilities. -**Please note that we are working on allowing Rucio accounts to be created and -accessed with IAM services,** **and -[EGI Check-in](https://docs.egi.eu/users/check-in/), but currently only support -x509 and password access.** - -1. Once our team has this information we will create you a Rucio account. - -### Docker container setup - -1. You will then need to install a containerised client on your computer. - - - Install Docker to run the container - - (for Windows users I would recommend - using WSL2) - - Follow the docker instructions to ensure it is running correctly. - - Using OpenSSL you will need to split your grid certificate bundle into the - certificate and key: - - ```shell - $ openssl pkcs12 -in <*.pfx> -out /sensible/path/usercert.pem -clcerts -nokeys - $ openssl pkcs12 -in <*.pfx> -out /sensible/path/userkey.pem -nocerts -nodes - ``` - -1. Run the Docker container using the following command: - -When running the block of code below, please replace all items within `<>` with -the relevant information. This uses a Rucio container that was set up for the EGI -communities. - -```shell -$ run \ - -e RUCIO_CFG_RUCIO_HOST=https://rucio-server.gridpp.rl.ac.uk:443 \ - -e RUCIO_CFG_AUTH_HOST=https://rucio-server.gridpp.rl.ac.uk:443 \ - -e RUCIO_CFG_AUTH_TYPE=x509_proxy \ - -e RUCIO_CFG_CLIENT_VO=<3 CHAR VO NAME LOWERCASE> \ - -e RUCIO_CFG_CLIENT_CERT=/opt/rucio/etc/usercert.pem \ - -e RUCIO_CFG_CLIENT_KEY=/opt/rucio/etc/userkey.pem \ - -e RUCIO_CFG_ACCOUNT= \ - -e RUCIO_CFG_CA_CERT=/opt/rucio/etc/web/ca-first.pem \ - -v :/opt/rucio/etc/web/ca-first.pem \ - -v :/opt/rucio/etc/usercert \ - -v :/opt/rucio/etc/userkey \ - --name=rucio-client \ - -it \ - -d egifedcloud/rucioclient:1.23.17 -``` - -This block of code may look large but it is configuring Rucio to connect to the -Multi-VO Rucio at RAL, your account and VO details, where you are loading them -into the container, and mounting the authentication details into the container. - -**The UK eScience CA 2B can be [obtained here](https://ca.grid-support.ac.uk/). -The 3 characters VO name will be provided to you when you sign up for a Rucio -account.** - -1. Run the following commands inside the docker container to finalise set up: - -```shell -$ cp /opt/rucio/etc/usercert /opt/rucio/etc/usercert.pem -$ cp /opt/rucio/etc/userkey /opt/rucio/etc/userkey.pem -$ chmod 600 /opt/rucio/etc/usercert.pem -$ chmod 400 /opt/rucio/etc/userkey.pem -``` - -### Rucio configuration setup - -You need to edit the `/opt/rucio/etc/rucio.cfg` file, this then needs to be -lightly edited to add your account name. This will then be loaded into the Rucio -client. - -```ini -[common] -logdir = /var/log/rucio -multi_vo = True -loglevel = INFO -[client] -rucio_host = https://rucio-server.gridpp.rl.ac.uk:443 -auth_host = https://rucio-server.gridpp.rl.ac.uk:443 -vo = <3 character VO name> -account = -ca_cert = /opt/rucio/etc/web/ca-first.pem -auth_type = x509_proxy -client_cert = /opt/rucio/etc/usercert.pem -client_key = /opt/rucio/etc/userkey.pem -client_x509_proxy = /tmp/x509up_u1000 -request_retries = 5 -``` - -**You should now have a fully set up Containerised Client for your Rucio -Account** **and VO which you can start in docker and use whenever you need it.** - -- If not please contact Rucio support - -## Getting started as a new VO - -- To get set up with a new VO on Multi-VO Rucio account please create a ticket - on [GGUS](https://ggus.eu/?mode=ticket_submit). Please fill in the form with a - subject, description, ticket category - service request, priority - less - urgent, and under routing information please select 'assign to support unit' - - Rucio. - -- We will set up a meeting to discuss Rucio, your needs, sites, and current set - up to ensure that Rucio can work for you, and will track progress with the - ticket. From 474078fbcddf94774e1da3048866f380b3ce8947 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Enol=20Fern=C3=A1ndez?= Date: Thu, 12 Oct 2023 10:44:29 +0100 Subject: [PATCH 3/4] Remove the PaaS and Dodas docs. Move ec3 under IM (#612) As EGI-ACE is over we keep the docs for the services in the EGI portfolio. EC3 moves into IM and the portal for EC3 is removed as it is to be deprecated. --- **Related issue :** #611 Co-authored-by: Baptiste Grenier --- .../en/users/compute/orchestration/_index.md | 2 - .../compute/orchestration/dodas/_index.md | 38 ------------------ .../orchestration/{ => im}/ec3/_index.md | 0 .../orchestration/{ => im}/ec3/apps/_index.md | 2 +- .../{ => im}/ec3/apps/ecas/_index.md | 0 .../apps/ecas/ecas-application-dashboard.png | Bin .../ec3/apps/ecas/ecas-container-1.png | Bin .../ec3/apps/ecas/ecas-container-2.png | Bin .../{ => im}/ec3/apps/ecas/ecas-end.png | Bin .../{ => im}/ec3/apps/ecas/ecas-endpoint.png | Bin .../{ => im}/ec3/apps/ecas/ecas-explore.png | Bin .../{ => im}/ec3/apps/ecas/ecas-front.png | Bin .../{ => im}/ec3/apps/ecas/ecas-infra-io.png | Bin .../ec3/apps/ecas/ecas-infra-server.png | Bin .../{ => im}/ec3/apps/ecas/ecas-instance.png | Bin .../{ => im}/ec3/apps/ecas/ecas-jupyter.png | Bin .../ec3/apps/ecas/ecas-jupyterhub.png | Bin .../{ => im}/ec3/apps/ecas/ecas-list.png | Bin .../{ => im}/ec3/apps/ecas/ecas-lrms.png | Bin .../{ => im}/ec3/apps/ecas/ecas-manage.png | Bin .../{ => im}/ec3/apps/ecas/ecas-oph_term.png | Bin .../{ => im}/ec3/apps/ecas/ecas-os.png | Bin .../{ => im}/ec3/apps/ecas/ecas-reduce.png | Bin .../{ => im}/ec3/apps/ecas/ecas-size.png | Bin .../{ => im}/ec3/apps/ecas/ecas-space.png | Bin .../{ => im}/ec3/apps/ecas/ecas-summary.png | Bin .../{ => im}/ec3/apps/ecas/grafana-login.png | Bin .../{ => im}/ec3/apps/htc/_index.md | 0 .../orchestration/{ => im}/ec3/cli/_index.md | 0 .../{ => im}/ec3/developers/_index.md | 2 +- .../{ => im}/ec3/portal/_index.md | 0 .../{ => im}/ec3/portal/ec3-endpoint.png | Bin .../{ => im}/ec3/portal/ec3-lrms.png | Bin .../{ => im}/ec3/portal/ec3-nodes.png | Bin .../{ => im}/ec3/portal/ec3-portal.png | Bin .../{ => im}/ec3/portal/ec3-size.png | Bin .../orchestration/indigo-paas/_index.md | 13 ------ 37 files changed, 2 insertions(+), 55 deletions(-) delete mode 100644 content/en/users/compute/orchestration/dodas/_index.md rename content/en/users/compute/orchestration/{ => im}/ec3/_index.md (100%) rename content/en/users/compute/orchestration/{ => im}/ec3/apps/_index.md (97%) rename content/en/users/compute/orchestration/{ => im}/ec3/apps/ecas/_index.md (100%) rename content/en/users/compute/orchestration/{ => im}/ec3/apps/ecas/ecas-application-dashboard.png (100%) rename content/en/users/compute/orchestration/{ => im}/ec3/apps/ecas/ecas-container-1.png (100%) rename content/en/users/compute/orchestration/{ => im}/ec3/apps/ecas/ecas-container-2.png (100%) rename content/en/users/compute/orchestration/{ => im}/ec3/apps/ecas/ecas-end.png (100%) rename content/en/users/compute/orchestration/{ => im}/ec3/apps/ecas/ecas-endpoint.png (100%) rename content/en/users/compute/orchestration/{ => im}/ec3/apps/ecas/ecas-explore.png (100%) rename content/en/users/compute/orchestration/{ => im}/ec3/apps/ecas/ecas-front.png (100%) rename content/en/users/compute/orchestration/{ => im}/ec3/apps/ecas/ecas-infra-io.png (100%) rename content/en/users/compute/orchestration/{ => im}/ec3/apps/ecas/ecas-infra-server.png (100%) rename content/en/users/compute/orchestration/{ => im}/ec3/apps/ecas/ecas-instance.png (100%) rename content/en/users/compute/orchestration/{ => im}/ec3/apps/ecas/ecas-jupyter.png (100%) rename content/en/users/compute/orchestration/{ => im}/ec3/apps/ecas/ecas-jupyterhub.png (100%) rename content/en/users/compute/orchestration/{ => im}/ec3/apps/ecas/ecas-list.png (100%) rename content/en/users/compute/orchestration/{ => im}/ec3/apps/ecas/ecas-lrms.png (100%) rename content/en/users/compute/orchestration/{ => im}/ec3/apps/ecas/ecas-manage.png (100%) rename content/en/users/compute/orchestration/{ => im}/ec3/apps/ecas/ecas-oph_term.png (100%) rename content/en/users/compute/orchestration/{ => im}/ec3/apps/ecas/ecas-os.png (100%) rename content/en/users/compute/orchestration/{ => im}/ec3/apps/ecas/ecas-reduce.png (100%) rename content/en/users/compute/orchestration/{ => im}/ec3/apps/ecas/ecas-size.png (100%) rename content/en/users/compute/orchestration/{ => im}/ec3/apps/ecas/ecas-space.png (100%) rename content/en/users/compute/orchestration/{ => im}/ec3/apps/ecas/ecas-summary.png (100%) rename content/en/users/compute/orchestration/{ => im}/ec3/apps/ecas/grafana-login.png (100%) rename content/en/users/compute/orchestration/{ => im}/ec3/apps/htc/_index.md (100%) rename content/en/users/compute/orchestration/{ => im}/ec3/cli/_index.md (100%) rename content/en/users/compute/orchestration/{ => im}/ec3/developers/_index.md (95%) rename content/en/users/compute/orchestration/{ => im}/ec3/portal/_index.md (100%) rename content/en/users/compute/orchestration/{ => im}/ec3/portal/ec3-endpoint.png (100%) rename content/en/users/compute/orchestration/{ => im}/ec3/portal/ec3-lrms.png (100%) rename content/en/users/compute/orchestration/{ => im}/ec3/portal/ec3-nodes.png (100%) rename content/en/users/compute/orchestration/{ => im}/ec3/portal/ec3-portal.png (100%) rename content/en/users/compute/orchestration/{ => im}/ec3/portal/ec3-size.png (100%) delete mode 100644 content/en/users/compute/orchestration/indigo-paas/_index.md diff --git a/content/en/users/compute/orchestration/_index.md b/content/en/users/compute/orchestration/_index.md index 4b65659c6f..f007873c5a 100644 --- a/content/en/users/compute/orchestration/_index.md +++ b/content/en/users/compute/orchestration/_index.md @@ -21,8 +21,6 @@ choose (or gets used automatically) in specific scenarios: | Service Name | Workload Type | Use-Case | | ------------------------------------------- | ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | | [Infrastructure Manager](im) | VMs, containers, storage | Used to run workloads on a single IaaS Cloud provider. | -| [Elastic Cloud Compute Cluster](ec3) | VMs, containers, storage | Used when you need to run workloads on clusters that can be elastically scaled and potentially span more than one IaaS Cloud provider. | | [Workload Manager](workload-manager) | Jobs | Used to efficiently distribute, manage, and monitor computing workloads. | -| [Dynamic On-Demand Analysis Service](dodas) | Containers, storage (caches) | Used when you need to process your data either interactively or via a batch system. | The following sections offer more details about each of these orchestrators. diff --git a/content/en/users/compute/orchestration/dodas/_index.md b/content/en/users/compute/orchestration/dodas/_index.md deleted file mode 100644 index 4e9be014dd..0000000000 --- a/content/en/users/compute/orchestration/dodas/_index.md +++ /dev/null @@ -1,38 +0,0 @@ ---- -title: Dynamic On-Demand Analysis Software -linkTitle: Dynamic On-Demand Analysis Software -type: docs -weight: 40 -description: > - Instantiate on-demand customizable infrastructures for data analysis ---- - -## What is it? - -[Dynamic On-Demand Analysis Software (DODAS)](https://dodas-iam.cloud.cnaf.infn.it/login) -enables the execution of user analysis code both in batch mode and -interactively via the Jupyter interface. DODAS is highly customizable and offers -several building blocks that can be combined together in order to create the -best service composition for a given use case. The currently available blocks -allow to combine Jupyter and HTCondor as well as Jupyter and Spark or simply a -jupyter interface. In addition, they allow the management of data via caches to -optimize the processing of remote data. This can be done either via XCache or -MinIO S3 object storage capabilities. DODAS is based on docker containers and -the related orchestration relies on Kubernetes that enables the possibility to -compose the building blocks via a web-based user interface thanks to Kubeapps. - -In order to deploy it services over various backend, DODAS relies on -[Infrastructure Manager](../im) and -[INDIGO PaaS](https://indigo-paas.cloud.ba.infn.it/home/login) - -{{% alert title="Tip" color="info" %}} If you want to give it a try and deploy -your cluster in the EGI Federation: -[DODAS](https://dodas-iam.cloud.cnaf.infn.it/login). {{% /alert %}} - -{{% alert title="Note" color="info" %}} For detailed information about DODAS -please see its [documentation](https://web.infn.it/dodas/index.php/en/). It was -also presented in one of the [EGI Webinars](https://www.egi.eu/trainings-and-webinars/), more -details are available on the [indico page](https://indico.egi.eu/event/5695/) -and the video recording is available on -[YouTube](https://www.youtube.com/watch?v=bcURl4ESRW8&ab_channel=EGI). -{{% /alert %}} diff --git a/content/en/users/compute/orchestration/ec3/_index.md b/content/en/users/compute/orchestration/im/ec3/_index.md similarity index 100% rename from content/en/users/compute/orchestration/ec3/_index.md rename to content/en/users/compute/orchestration/im/ec3/_index.md diff --git a/content/en/users/compute/orchestration/ec3/apps/_index.md b/content/en/users/compute/orchestration/im/ec3/apps/_index.md similarity index 97% rename from content/en/users/compute/orchestration/ec3/apps/_index.md rename to content/en/users/compute/orchestration/im/ec3/apps/_index.md index dc719bbb9f..d5ec6a2842 100644 --- a/content/en/users/compute/orchestration/ec3/apps/_index.md +++ b/content/en/users/compute/orchestration/im/ec3/apps/_index.md @@ -29,7 +29,7 @@ Check the dedicated [ECAS documentation](./ecas/). ### Kubernetes Check the -[Cloud Container Compute documentation](../../../cloud-container-compute). +[Cloud Container Compute documentation](../../../../cloud-container-compute). ### Mesos + Marathon + Chronos diff --git a/content/en/users/compute/orchestration/ec3/apps/ecas/_index.md b/content/en/users/compute/orchestration/im/ec3/apps/ecas/_index.md similarity index 100% rename from content/en/users/compute/orchestration/ec3/apps/ecas/_index.md rename to content/en/users/compute/orchestration/im/ec3/apps/ecas/_index.md diff --git a/content/en/users/compute/orchestration/ec3/apps/ecas/ecas-application-dashboard.png b/content/en/users/compute/orchestration/im/ec3/apps/ecas/ecas-application-dashboard.png similarity index 100% rename from content/en/users/compute/orchestration/ec3/apps/ecas/ecas-application-dashboard.png rename to content/en/users/compute/orchestration/im/ec3/apps/ecas/ecas-application-dashboard.png diff --git a/content/en/users/compute/orchestration/ec3/apps/ecas/ecas-container-1.png b/content/en/users/compute/orchestration/im/ec3/apps/ecas/ecas-container-1.png similarity index 100% rename from content/en/users/compute/orchestration/ec3/apps/ecas/ecas-container-1.png rename to content/en/users/compute/orchestration/im/ec3/apps/ecas/ecas-container-1.png diff --git a/content/en/users/compute/orchestration/ec3/apps/ecas/ecas-container-2.png b/content/en/users/compute/orchestration/im/ec3/apps/ecas/ecas-container-2.png similarity index 100% rename from content/en/users/compute/orchestration/ec3/apps/ecas/ecas-container-2.png rename to content/en/users/compute/orchestration/im/ec3/apps/ecas/ecas-container-2.png diff --git a/content/en/users/compute/orchestration/ec3/apps/ecas/ecas-end.png b/content/en/users/compute/orchestration/im/ec3/apps/ecas/ecas-end.png similarity index 100% rename from content/en/users/compute/orchestration/ec3/apps/ecas/ecas-end.png rename to content/en/users/compute/orchestration/im/ec3/apps/ecas/ecas-end.png diff --git a/content/en/users/compute/orchestration/ec3/apps/ecas/ecas-endpoint.png b/content/en/users/compute/orchestration/im/ec3/apps/ecas/ecas-endpoint.png similarity index 100% rename from content/en/users/compute/orchestration/ec3/apps/ecas/ecas-endpoint.png rename to content/en/users/compute/orchestration/im/ec3/apps/ecas/ecas-endpoint.png diff --git a/content/en/users/compute/orchestration/ec3/apps/ecas/ecas-explore.png b/content/en/users/compute/orchestration/im/ec3/apps/ecas/ecas-explore.png similarity index 100% rename from content/en/users/compute/orchestration/ec3/apps/ecas/ecas-explore.png rename to content/en/users/compute/orchestration/im/ec3/apps/ecas/ecas-explore.png diff --git a/content/en/users/compute/orchestration/ec3/apps/ecas/ecas-front.png b/content/en/users/compute/orchestration/im/ec3/apps/ecas/ecas-front.png similarity index 100% rename from content/en/users/compute/orchestration/ec3/apps/ecas/ecas-front.png rename to content/en/users/compute/orchestration/im/ec3/apps/ecas/ecas-front.png diff --git a/content/en/users/compute/orchestration/ec3/apps/ecas/ecas-infra-io.png b/content/en/users/compute/orchestration/im/ec3/apps/ecas/ecas-infra-io.png similarity index 100% rename from content/en/users/compute/orchestration/ec3/apps/ecas/ecas-infra-io.png rename to content/en/users/compute/orchestration/im/ec3/apps/ecas/ecas-infra-io.png diff --git a/content/en/users/compute/orchestration/ec3/apps/ecas/ecas-infra-server.png b/content/en/users/compute/orchestration/im/ec3/apps/ecas/ecas-infra-server.png similarity index 100% rename from content/en/users/compute/orchestration/ec3/apps/ecas/ecas-infra-server.png rename to content/en/users/compute/orchestration/im/ec3/apps/ecas/ecas-infra-server.png diff --git a/content/en/users/compute/orchestration/ec3/apps/ecas/ecas-instance.png b/content/en/users/compute/orchestration/im/ec3/apps/ecas/ecas-instance.png similarity index 100% rename from content/en/users/compute/orchestration/ec3/apps/ecas/ecas-instance.png rename to content/en/users/compute/orchestration/im/ec3/apps/ecas/ecas-instance.png diff --git a/content/en/users/compute/orchestration/ec3/apps/ecas/ecas-jupyter.png b/content/en/users/compute/orchestration/im/ec3/apps/ecas/ecas-jupyter.png similarity index 100% rename from content/en/users/compute/orchestration/ec3/apps/ecas/ecas-jupyter.png rename to content/en/users/compute/orchestration/im/ec3/apps/ecas/ecas-jupyter.png diff --git a/content/en/users/compute/orchestration/ec3/apps/ecas/ecas-jupyterhub.png b/content/en/users/compute/orchestration/im/ec3/apps/ecas/ecas-jupyterhub.png similarity index 100% rename from content/en/users/compute/orchestration/ec3/apps/ecas/ecas-jupyterhub.png rename to content/en/users/compute/orchestration/im/ec3/apps/ecas/ecas-jupyterhub.png diff --git a/content/en/users/compute/orchestration/ec3/apps/ecas/ecas-list.png b/content/en/users/compute/orchestration/im/ec3/apps/ecas/ecas-list.png similarity index 100% rename from content/en/users/compute/orchestration/ec3/apps/ecas/ecas-list.png rename to content/en/users/compute/orchestration/im/ec3/apps/ecas/ecas-list.png diff --git a/content/en/users/compute/orchestration/ec3/apps/ecas/ecas-lrms.png b/content/en/users/compute/orchestration/im/ec3/apps/ecas/ecas-lrms.png similarity index 100% rename from content/en/users/compute/orchestration/ec3/apps/ecas/ecas-lrms.png rename to content/en/users/compute/orchestration/im/ec3/apps/ecas/ecas-lrms.png diff --git a/content/en/users/compute/orchestration/ec3/apps/ecas/ecas-manage.png b/content/en/users/compute/orchestration/im/ec3/apps/ecas/ecas-manage.png similarity index 100% rename from content/en/users/compute/orchestration/ec3/apps/ecas/ecas-manage.png rename to content/en/users/compute/orchestration/im/ec3/apps/ecas/ecas-manage.png diff --git a/content/en/users/compute/orchestration/ec3/apps/ecas/ecas-oph_term.png b/content/en/users/compute/orchestration/im/ec3/apps/ecas/ecas-oph_term.png similarity index 100% rename from content/en/users/compute/orchestration/ec3/apps/ecas/ecas-oph_term.png rename to content/en/users/compute/orchestration/im/ec3/apps/ecas/ecas-oph_term.png diff --git a/content/en/users/compute/orchestration/ec3/apps/ecas/ecas-os.png b/content/en/users/compute/orchestration/im/ec3/apps/ecas/ecas-os.png similarity index 100% rename from content/en/users/compute/orchestration/ec3/apps/ecas/ecas-os.png rename to content/en/users/compute/orchestration/im/ec3/apps/ecas/ecas-os.png diff --git a/content/en/users/compute/orchestration/ec3/apps/ecas/ecas-reduce.png b/content/en/users/compute/orchestration/im/ec3/apps/ecas/ecas-reduce.png similarity index 100% rename from content/en/users/compute/orchestration/ec3/apps/ecas/ecas-reduce.png rename to content/en/users/compute/orchestration/im/ec3/apps/ecas/ecas-reduce.png diff --git a/content/en/users/compute/orchestration/ec3/apps/ecas/ecas-size.png b/content/en/users/compute/orchestration/im/ec3/apps/ecas/ecas-size.png similarity index 100% rename from content/en/users/compute/orchestration/ec3/apps/ecas/ecas-size.png rename to content/en/users/compute/orchestration/im/ec3/apps/ecas/ecas-size.png diff --git a/content/en/users/compute/orchestration/ec3/apps/ecas/ecas-space.png b/content/en/users/compute/orchestration/im/ec3/apps/ecas/ecas-space.png similarity index 100% rename from content/en/users/compute/orchestration/ec3/apps/ecas/ecas-space.png rename to content/en/users/compute/orchestration/im/ec3/apps/ecas/ecas-space.png diff --git a/content/en/users/compute/orchestration/ec3/apps/ecas/ecas-summary.png b/content/en/users/compute/orchestration/im/ec3/apps/ecas/ecas-summary.png similarity index 100% rename from content/en/users/compute/orchestration/ec3/apps/ecas/ecas-summary.png rename to content/en/users/compute/orchestration/im/ec3/apps/ecas/ecas-summary.png diff --git a/content/en/users/compute/orchestration/ec3/apps/ecas/grafana-login.png b/content/en/users/compute/orchestration/im/ec3/apps/ecas/grafana-login.png similarity index 100% rename from content/en/users/compute/orchestration/ec3/apps/ecas/grafana-login.png rename to content/en/users/compute/orchestration/im/ec3/apps/ecas/grafana-login.png diff --git a/content/en/users/compute/orchestration/ec3/apps/htc/_index.md b/content/en/users/compute/orchestration/im/ec3/apps/htc/_index.md similarity index 100% rename from content/en/users/compute/orchestration/ec3/apps/htc/_index.md rename to content/en/users/compute/orchestration/im/ec3/apps/htc/_index.md diff --git a/content/en/users/compute/orchestration/ec3/cli/_index.md b/content/en/users/compute/orchestration/im/ec3/cli/_index.md similarity index 100% rename from content/en/users/compute/orchestration/ec3/cli/_index.md rename to content/en/users/compute/orchestration/im/ec3/cli/_index.md diff --git a/content/en/users/compute/orchestration/ec3/developers/_index.md b/content/en/users/compute/orchestration/im/ec3/developers/_index.md similarity index 95% rename from content/en/users/compute/orchestration/ec3/developers/_index.md rename to content/en/users/compute/orchestration/im/ec3/developers/_index.md index a6e61e48f5..5195d77aef 100644 --- a/content/en/users/compute/orchestration/ec3/developers/_index.md +++ b/content/en/users/compute/orchestration/im/ec3/developers/_index.md @@ -29,6 +29,6 @@ following process: {{% alert title="Note" color="info" %}} For more information please contact * Miguel Caballer: `micafer1 upv.es` -* Amanda Calatrav: `amcaar i3m.upv.es` +* Amanda Calatrava: `amcaar i3m.upv.es` {{% /alert %}} diff --git a/content/en/users/compute/orchestration/ec3/portal/_index.md b/content/en/users/compute/orchestration/im/ec3/portal/_index.md similarity index 100% rename from content/en/users/compute/orchestration/ec3/portal/_index.md rename to content/en/users/compute/orchestration/im/ec3/portal/_index.md diff --git a/content/en/users/compute/orchestration/ec3/portal/ec3-endpoint.png b/content/en/users/compute/orchestration/im/ec3/portal/ec3-endpoint.png similarity index 100% rename from content/en/users/compute/orchestration/ec3/portal/ec3-endpoint.png rename to content/en/users/compute/orchestration/im/ec3/portal/ec3-endpoint.png diff --git a/content/en/users/compute/orchestration/ec3/portal/ec3-lrms.png b/content/en/users/compute/orchestration/im/ec3/portal/ec3-lrms.png similarity index 100% rename from content/en/users/compute/orchestration/ec3/portal/ec3-lrms.png rename to content/en/users/compute/orchestration/im/ec3/portal/ec3-lrms.png diff --git a/content/en/users/compute/orchestration/ec3/portal/ec3-nodes.png b/content/en/users/compute/orchestration/im/ec3/portal/ec3-nodes.png similarity index 100% rename from content/en/users/compute/orchestration/ec3/portal/ec3-nodes.png rename to content/en/users/compute/orchestration/im/ec3/portal/ec3-nodes.png diff --git a/content/en/users/compute/orchestration/ec3/portal/ec3-portal.png b/content/en/users/compute/orchestration/im/ec3/portal/ec3-portal.png similarity index 100% rename from content/en/users/compute/orchestration/ec3/portal/ec3-portal.png rename to content/en/users/compute/orchestration/im/ec3/portal/ec3-portal.png diff --git a/content/en/users/compute/orchestration/ec3/portal/ec3-size.png b/content/en/users/compute/orchestration/im/ec3/portal/ec3-size.png similarity index 100% rename from content/en/users/compute/orchestration/ec3/portal/ec3-size.png rename to content/en/users/compute/orchestration/im/ec3/portal/ec3-size.png diff --git a/content/en/users/compute/orchestration/indigo-paas/_index.md b/content/en/users/compute/orchestration/indigo-paas/_index.md deleted file mode 100644 index 3e62d3f10e..0000000000 --- a/content/en/users/compute/orchestration/indigo-paas/_index.md +++ /dev/null @@ -1,13 +0,0 @@ ---- -title: INDIGO PaaS Orchestrator -linkTitle: PaaS Orchestrator -draft: true -type: docs -weight: 30 -description: > - Subtitle here ---- - -## What is it? - -TBD From 7d1fe54f6a4add891110286a1282c438e45e015f Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 24 Oct 2023 10:45:19 +0200 Subject: [PATCH 4/4] Bump actions/setup-node from 3 to 4 (#619) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bumps [actions/setup-node](https://github.com/actions/setup-node) from 3 to 4.
Release notes

Sourced from actions/setup-node's releases.

v4.0.0

What's Changed

In scope of this release we changed version of node runtime for action from node16 to node20 and updated dependencies in actions/setup-node#866

Besides, release contains such changes as:

New Contributors

Full Changelog: https://github.com/actions/setup-node/compare/v3...v4.0.0

v3.8.2

What's Changed

Full Changelog: https://github.com/actions/setup-node/compare/v3...v3.8.2

v3.8.1

What's Changed

In scope of this release, the filter was removed within the cache-save step by @​dmitry-shibanov in actions/setup-node#831. It is filtered and checked in the toolkit/cache library.

Full Changelog: https://github.com/actions/setup-node/compare/v3...v3.8.1

v3.8.0

What's Changed

Bug fixes:

Feature implementations:

Documentation changes:

Update dependencies:

... (truncated)

Commits

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=actions/setup-node&package-manager=github_actions&previous-version=3&new-version=4)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/build.yml | 2 +- .github/workflows/build_pr_preview.yml | 2 +- .github/workflows/deploy.yml | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 8fbb7f7501..eabdb4b1c5 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -30,7 +30,7 @@ jobs: extended: true - name: Setup Node - uses: actions/setup-node@v3 + uses: actions/setup-node@v4 with: node-version: "lts/*" diff --git a/.github/workflows/build_pr_preview.yml b/.github/workflows/build_pr_preview.yml index 943d63558c..4c21c3d40b 100644 --- a/.github/workflows/build_pr_preview.yml +++ b/.github/workflows/build_pr_preview.yml @@ -38,7 +38,7 @@ jobs: extended: true - name: Setup Node - uses: actions/setup-node@v3 + uses: actions/setup-node@v4 with: node-version: "lts/*" diff --git a/.github/workflows/deploy.yml b/.github/workflows/deploy.yml index e3c47c54ca..c7cfadf9cc 100644 --- a/.github/workflows/deploy.yml +++ b/.github/workflows/deploy.yml @@ -37,7 +37,7 @@ jobs: extended: true - name: Setup Node - uses: actions/setup-node@v3 + uses: actions/setup-node@v4 with: node-version: "lts/*"