Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

some tables didn't adhere to general md standards and were fixed. #77

Merged
merged 1 commit into from
Nov 29, 2023
Merged

some tables didn't adhere to general md standards and were fixed. #77

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
72 changes: 36 additions & 36 deletions docs/connection.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,18 +29,18 @@ The following environment variables can be used to configure the broker:
> We use [RabbitMQ](https://hub.docker.com/_/rabbitmq) >= `3.8.16` including
> the management plugins.

Variable | Description
:--------------------|:----------------------------------------------
`MQ_VHOST` | Default vhost other than `/`
`MQ_VERIFY` | Set to `verify_none` to disable verification of client certificate
`MQ_USER` | Default user (with admin rights)
`MQ_PASSWORD_HASH` | Password hash for the above user
`CEGA_CONNECTION` | DSN URL for the shovels and federated queues with CentralEGA
`MQ_SERVER_CERT` | Path to the server SSL certificate
`MQ_SERVER_KEY` | Path to the server SSL key
`MQ_CA` | Path to the CA root certificate
`MQ_VERIFY` | Require the clients to have valid TLS certificates (`verify_peer`) or do not require clients to have certificates (`verify_none`)
`NOTLS` | Run the server without TLS enabled (default is to run the server with TLS activated)
| Variable | Description |
|:-------------------|:----------------------------------------------------------------------------------------------------------------------------------|
| `MQ_VHOST` | Default vhost other than `/` |
| `MQ_VERIFY` | Set to `verify_none` to disable verification of client certificate |
| `MQ_USER` | Default user (with admin rights) |
| `MQ_PASSWORD_HASH` | Password hash for the above user |
| `CEGA_CONNECTION` | DSN URL for the shovels and federated queues with CentralEGA |
| `MQ_SERVER_CERT` | Path to the server SSL certificate |
| `MQ_SERVER_KEY` | Path to the server SSL key |
| `MQ_CA` | Path to the CA root certificate |
| `MQ_VERIFY` | Require the clients to have valid TLS certificates (`verify_peer`) or do not require clients to have certificates (`verify_none`) |
| `NOTLS` | Run the server without TLS enabled (default is to run the server with TLS activated) |

> NOTE:
> For SDA stand-alone do not use `CEGA_CONNECTION` and do not set up
Expand Down Expand Up @@ -70,28 +70,28 @@ amqp[s]://<user>:<password>@<cega-host>:<port>/<vhost>
versioning and is internal to CentralEGA. The queues connected to that
exchange are also internal to CentralEGA.

Name | Purpose
:----------------|:------------------------------------------------
files | Triggers for file ingestion
completed | When files are backed up
verified | When files are properly ingested and verified
errors | User-related errors
inbox | Notifications of uploaded files
| Name | Purpose |
|:----------|:----------------------------------------------|
| files | Triggers for file ingestion |
| completed | When files are backed up |
| verified | When files are properly ingested and verified |
| errors | User-related errors |
| inbox | Notifications of uploaded files |

`LocalMQ` contains two exchanges named `sda` and `to_cega`, and the
following queues, in the default `vhost`:

Name | Purpose
:----------------|:---------------------------------------
archived | Archived files.
completed | Files are backed up
error | User-related errors
files | Receive notification for ingestion from `CEGAMQ` or Orchestrator
inbox | Notifications of uploaded files
ingest | Trigger for file ingestion
mappings | Received Dataset to file mapping
accessionIDs | Receive Accession IDs from `CEGAMQ` or Orchestrator
verified | Files ingested and verified
| Name | Purpose |
|:-------------|:-----------------------------------------------------------------|
| archived | Archived files. |
| completed | Files are backed up |
| error | User-related errors |
| files | Receive notification for ingestion from `CEGAMQ` or Orchestrator |
| inbox | Notifications of uploaded files |
| ingest | Trigger for file ingestion |
| mappings | Received Dataset to file mapping |
| accessionIDs | Receive Accession IDs from `CEGAMQ` or Orchestrator |
| verified | Files ingested and verified |

`LocalMQ` registers `CEGAMQ` as an *upstream* and listens to the
incoming messages in `files` using a *federated queue*. Ingestion
Expand All @@ -110,12 +110,12 @@ CentralEGA (using the routing key based on the name
`files.<internal_queue_name>`). We propagate the different status of the
workflow to CentralEGA, using the following routing keys:

Name | Purpose
---------------------|:-------------------------------------------------
files.completed | For back-up files, ready to be distributed
files.error | In case a user-related error is detected
files.inbox | For inbox file operations
files.verified | For files ready to request accessionID
| Name | Purpose |
|-----------------|:-------------------------------------------|
| files.completed | For back-up files, ready to be distributed |
| files.error | In case a user-related error is detected |
| files.inbox | For inbox file operations |
| files.verified | For files ready to request accessionID |

Note that we do not need at the moment a queue to store the completed
message, nor the errors, as we forward them to Central EGA.
Expand Down
74 changes: 37 additions & 37 deletions docs/dataout.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,43 +15,43 @@ SDA-DOA
Configuration
-------------

Variable name | Default value | Description
:-----------------------------------|:----------------------------------------------------------------------|:---------------------
`REST_ENABLED` | true | Enables/disables REST endpoints of DOA
`SSL_ENABLED` | true | Enables/disables TLS for DOA REST endpoints
`KEYSTORE_PATH` | /etc/ega/ssl/server.cert | Path to server keystore file
`KEYSTORE_PASSWORD` | | Password for the keystore
`OUTBOX_ENABLED` | true | Enables/disables the outbox functionality
`OUTBOX_QUEUE` | exportRequests | MQ queue name for files/datasets export requests
`OUTBOX_LOCATION` | /ega/outbox/p11-%s/files/ | Outbox location with placeholder for the username
`BROKER_HOST` | private-mq | Local RabbitMQ broker hostname
`BROKER_PORT` | 5671 | Local RabbitMQ broker port
`BROKER_VHOST` | / | Local RabbitMQ broker virtual host
`BROKER_VALIDATE` | true | Validate server MQ certificate or not
`DB_INSTANCE` | db | Database hostname
`DB_PORT` | 5432 | Database port
`POSTGRES_DB` | lega | Database name
`ROOT_CERT_PATH` | /etc/ega/ssl/CA.cert | Path to the CA file for database connectivity
`CERT_PATH` | /etc/ega/ssl/client.cert | Path to the client cert for database connectivity
`CERT_KEY` | /etc/ega/ssl/client.key | Path to the client key for database connectivity
`POSTGRES_USER` | lega_out | Database username
`POSTGRES_PASSWORD` | | Database password
`S3_ENDPOINT` | vault | S3 server hostname
`S3_PORT` | 443 | S3 server port
`S3_ACCESS_KEY` | minio | S3 access key
`S3_SECRET_KEY` | miniostorage | S3 secret key
`S3_REGION` | us-west-1 | S3 region
`S3_BUCKET` | lega | S3 bucket to use
`S3_SECURE` | true | true if S3 backend should be accessed over HTTPS
`S3_ROOT_CERT_PATH` | /etc/ssl/certs/ca-certificates.crt | Path to the CA certs file for S3 connectivity
`ARCHIVE_PATH` | / | Path to the filesystem-archive
`PASSPORT_PUBLIC_KEY_PATH` | /etc/ega/jwt/passport.pem | Path to the public key for passport JWT validation
`OPENID_CONFIGURATION_URL` | <https://login.elixir-czech.org/oidc/.well-known/openid-configuration>| URL of the OpenID configuration endpoint
`VISA_PUBLIC_KEY_PATH` | /etc/ega/jwt/visa.pem | Path to the public key for visas JWT validation
`CRYPT4GH_PRIVATE_KEY_PATH` | /etc/ega/crypt4gh/key.pem | Path to the Crypt4GH private key
`CRYPT4GH_PRIVATE_KEY_PASSWORD_PATH`| /etc/ega/crypt4gh/key.pass | Path to the Crypt4GH private key passphrase
`LOGSTASH_HOST` | | Hostname of the Logstash instance (if any)
`LOGSTASH_PORT` | | Port of the Logstash instance (if any)
| Variable name | Default value | Description |
|:-------------------------------------|:-----------------------------------------------------------------------|:---------------------------------------------------|
| `REST_ENABLED` | true | Enables/disables REST endpoints of DOA |
| `SSL_ENABLED` | true | Enables/disables TLS for DOA REST endpoints |
| `KEYSTORE_PATH` | /etc/ega/ssl/server.cert | Path to server keystore file |
| `KEYSTORE_PASSWORD` | | Password for the keystore |
| `OUTBOX_ENABLED` | true | Enables/disables the outbox functionality |
| `OUTBOX_QUEUE` | exportRequests | MQ queue name for files/datasets export requests |
| `OUTBOX_LOCATION` | /ega/outbox/p11-%s/files/ | Outbox location with placeholder for the username |
| `BROKER_HOST` | private-mq | Local RabbitMQ broker hostname |
| `BROKER_PORT` | 5671 | Local RabbitMQ broker port |
| `BROKER_VHOST` | / | Local RabbitMQ broker virtual host |
| `BROKER_VALIDATE` | true | Validate server MQ certificate or not |
| `DB_INSTANCE` | db | Database hostname |
| `DB_PORT` | 5432 | Database port |
| `POSTGRES_DB` | lega | Database name |
| `ROOT_CERT_PATH` | /etc/ega/ssl/CA.cert | Path to the CA file for database connectivity |
| `CERT_PATH` | /etc/ega/ssl/client.cert | Path to the client cert for database connectivity |
| `CERT_KEY` | /etc/ega/ssl/client.key | Path to the client key for database connectivity |
| `POSTGRES_USER` | lega_out | Database username |
| `POSTGRES_PASSWORD` | | Database password |
| `S3_ENDPOINT` | vault | S3 server hostname |
| `S3_PORT` | 443 | S3 server port |
| `S3_ACCESS_KEY` | minio | S3 access key |
| `S3_SECRET_KEY` | miniostorage | S3 secret key |
| `S3_REGION` | us-west-1 | S3 region |
| `S3_BUCKET` | lega | S3 bucket to use |
| `S3_SECURE` | true | true if S3 backend should be accessed over HTTPS |
| `S3_ROOT_CERT_PATH` | /etc/ssl/certs/ca-certificates.crt | Path to the CA certs file for S3 connectivity |
| `ARCHIVE_PATH` | / | Path to the filesystem-archive |
| `PASSPORT_PUBLIC_KEY_PATH` | /etc/ega/jwt/passport.pem | Path to the public key for passport JWT validation |
| `OPENID_CONFIGURATION_URL` | <https://login.elixir-czech.org/oidc/.well-known/openid-configuration> | URL of the OpenID configuration endpoint |
| `VISA_PUBLIC_KEY_PATH` | /etc/ega/jwt/visa.pem | Path to the public key for visas JWT validation |
| `CRYPT4GH_PRIVATE_KEY_PATH` | /etc/ega/crypt4gh/key.pem | Path to the Crypt4GH private key |
| `CRYPT4GH_PRIVATE_KEY_PASSWORD_PATH` | /etc/ega/crypt4gh/key.pass | Path to the Crypt4GH private key passphrase |
| `LOGSTASH_HOST` | | Hostname of the Logstash instance (if any) |
| `LOGSTASH_PORT` | | Port of the Logstash instance (if any) |

### Outbox functionality

Expand Down
Loading