CentralEGA
contains a database of users, with IDs and passwords.
A solution has been devised using Apache Mina SSHD to facilitate user authentication through either a password or an RSA key, directly against the CentralEGA database. The user is locked within their home folder, which is done programmatically using RootedFileSystem.
The solution uses CentralEGA
's user IDs but can also be extended to
use LifeScience AAI IDs (from which the @elixir-europe.org
suffix is removed).
The procedure is as follows. The inbox is started without any created
user. When a user wants to log into the inbox (actually, only sftp
uploads are allowed), the code looks up the username in a local
cache, and, if not found, queries the CentralEGA
REST endpoint. Upon the user's return, their credentials are stored in the local cache, and a home directory is established for the user. The user now gets logged in if the password
or public key authentication succeeds. Upon subsequent login attempts,
only the local cache is queried, until the user's credentials
expire. The cache has a default TTL of 5 minutes, and is wiped clean
upon reboot (as a cache should). Default TTL can be configured via CACHE_TTL
env var.
The user's home directory is created when its credentials upon successful login. Moreover, for each user, detection is performed to ascertain when the file upload is completed, and the checksum for the uploaded file is computed.
Default storage back-end for the inbox is local file-system. Additionally, support for the S3 service is provided as a back-end option. It can be enabled using S3-related env-vars (see configuration details below).
If S3 is enabled, then files are still going to be stored locally, but after successful upload, they will going to be uploaded to the specified S3 back-end. With this approach local file-system plays role of so called "staging area", while S3 is the real final destination for the uploaded files.
Environment variables used:
Variable name | Default value | Description |
---|---|---|
BROKER_USERNAME | guest | RabbitMQ broker username |
BROKER_PASSWORD | guest | RabbitMQ broker password |
BROKER_HOST | mq | RabbitMQ broker host |
BROKER_PORT | 5672 | RabbitMQ broker port |
BROKER_VHOST | / | RabbitMQ broker vhost |
BROKER_EXCHANGE | sda | RabbitMQ broker exchange |
BROKER_ROUTING_KEY | files | RabbitMQ broker routing key |
INBOX_PORT | 2222 | Inbox port |
INBOX_LOCATION | /ega/inbox/ | Path to POSIX Inbox backend |
INBOX_FS_PATH | Prefix path when custom filesystem is used on top of POSIX | |
INBOX_KEYPAIR | Path to RSA keypair file | |
KEYSTORE_TYPE | JKS | Keystore type to use, JKS or PKCS12 |
KEYSTORE_PATH | /etc/ega/inbox.jks | Path to Keystore file |
KEYSTORE_PASSWORD | Password to access the Keystore | |
CACHE_TTL | 300.0 | CEGA credentials time-to-live |
CEGA_ENDPOINT | CEGA REST endpoint | |
CEGA_ENDPOINT_CREDS | CEGA REST credentials | |
S3_ENDPOINT | inbox-backend:9000 | Inbox S3 backend URL |
S3_REGION | us-east-1 | Inbox S3 backend region (us-east-1 is default in Minio) |
S3_ACCESS_KEY | Inbox S3 backend access key (S3 disabled if not specified) | |
S3_SECRET_KEY | Inbox S3 backend secret key (S3 disabled if not specified) | |
S3_BUCKET | Inbox S3 backend secret bucket (S3 disabled if not specified) | |
USE_SSL | true | true if S3 Inbox backend should be accessed by HTTPS |
LOGSTASH_HOST | Hostname of the Logstash instance (if any) | |
LOGSTASH_PORT | Port of the Logstash instance (if any) |
If LOGSTASH_HOST
or LOGSTASH_PORT
is empty, Logstash logging will not be enabled.
In addition, environment variables can be used to configure log level for different packages. Package loggers can be configured using corresponding package names, for example, to turn of logs of Spring, one can set environment variable LOGGING_LEVEL_ORG_SPRINGFRAMEWORK=OFF
, or to set Mina's own logs to debug: LOGGING_LEVEL_SE_NBIS_LEGA_INBOX=DEBUG
, etc.
For local development/testing see instructions in dev_utils folder. There is an README file in the dev_utils folder with sections for running the pipeline locally using Docker Compose.