Microservice implementing the Global Alliance for Genomics and Health (GA4GH) Workflow Execution Service (WES) API specification for the execution of workflows written in the Common Workflow Language (CWL).
cwl-WES is a core service of the ELIXIR Cloud & AAI project.
cwl-WES (formerly: WES-ELIXIR) is a Flask/Gunicorn application that makes use of Connexion to implement the GA4GH WES OpenAPI specification. It enables clients/users to execute CWL workflows in the cloud via a GA4GH Task Execution Service (TES)-compatible execution backend (e.g., TESK or Funnel). Workflows can be sent for execution, previous runs can be listed, and the status and run information of individual runs can be queried. The service leverages cwl-tes to interpret CWL workflows, break them down into individual tasks and emit GA4GH TES-compatible HTTP requests to a configured TES instance. Access to endpoints can be configured to require JSON Web Token-based access tokens, such as those issued by ELIXIR AAI. Run information is stored in a MongoDB database.
Note that development is currently in beta stage. Check the website badge at the top of this document for a publicly available test deployment. Further test deployments can be found at the ELIXIR Cloud & AAI's resource listings.
cwl-WES is developed and maintained by the ELIXIR Cloud & AAI project, a multinational effort aimed at establishing and implementing FAIR research in the Life Sciences.
See separate instructions available here.
Ensure you have the following software installed:
- Docker (18.06.1-ce, build e68fc7a)
- docker-compose (1.22.0, build f46880fe)
- Git (1.8.3.1)
These are the versions used for development/testing. Other versions may or may not work. Please let us know if you encounter any issues with newer versions than the listed ones.
Create data directory and required subdiretories:
mkdir -p data/cwl_wes/db data/cwl_wes/output data/cwl_wes/tmpClone repository:
git clone https://github.com/elixir-cloud-aai/cwl-WES.git appTraverse to app directory:
cd app-
Via the app configuration file
vi cwl_wes/config/app_config.yaml
-
Via environment variables
A few configuration settings can be overridden by environment variables:
export <ENV_VAR_NAME>=<VALUE>
List of the available environment variables:
Variable Description MONGO_HOST MongoDB host endpoint MONGO_PORT MongoDB service port MONGO_DBNAME MongoDB database name MONGO_USERNAME MongoDB client username MONGO_PASSWORD MongoDB client password RABBIT_HOST RabbitMQ host endpoint RABBIT_PORT RabbitMQ service port
Build and run services in detached/daemonized mode:
docker-compose up -d --buildCreate a .netrc file with credentials for accessing an FTP server:
cat << EOF > .netrc
machine <HOST>
login <USERNAME>
password <PASSWORD>
EOFDon't forget to replace
<HOST>,<USERNAME>and<PASSWORD>with real values.If you do not know what to put here, creating an empty file
.netrcwith, e.g.,touch .netrcwill be fine for testing purposes.
Copy the file into the running worker container(s):
for cont in $(docker ps --all | grep cwl-wes_wes-worker | cut -f1 -d" "); do
docker cp .netrc "${cont}:/tmp/user"
doneVisit Swagger UI:
firefox http://localhost:7777/ga4gh/wes/v1/uiExample values to start a simple CWL test workflow via the POST /runs
endpoint:
workflow_params: {"input":{"class":"File","path":"ftp://ftp-private.ebi.ac.uk/upload/foivos/test.txt"}}`
workflow_type: CWL
workflow_type_version`: v1.0
workflow_url: https://github.com/uniqueg/cwl-example-workflows/blob/master/hashsplitter-workflow.cwlLeave the rest of the values empty and hit the Try it out! button.
You can also use the service through curl. For example, to send a request to
the GET /runs endpoint:
curl -X GET \
--header 'Accept: application/json' \
'http://localhost:7777/ga4gh/wes/v1/runs' All endpoints except for GET /service-info can be configured to require the
presence of a valid JWT Bearer token in a request's header. This can be enabled
by setting the following paraneter in the app configuration to
True:
# Security settings
security:
authorization_required: TrueTo send authorized requests, you must be in possession of a valid JWT Bearer token. The app was developed against JWTs issues by ELIXIR AAI, although JWTs issued by other identity providers may work.
If you want to utilize actual compute resources in any of the ELIXIR test deployments, you must be a member of a specific ELIXIR user group. You can apply here to be added to that group (ELIXIR AAI only!). When applying, please include a brief description of why you would like to make use of our test deployments and please note that they are for fair use only (please don't try to run your production workflows on them or we will need to re-think our strategy of offering free resources for testing/developing).
To access protected endpoints via curl, you only need to add an Authorization
header to your request, followed by the Bearer prefix and your JWT token, like
so:
curl -X GET \
--header 'Accept: application/json' \
--header 'Authorization: Bearer <YOUR_TOKEN>' \
'http://localhost:7777/ga4gh/wes/v1/runs' If you are making use of the Swagger UI, you will need to click on the
Authorize button (which appears on the right-hand side of the top bar upon
enabling the authorization_required option) of the Swagger UI website and
enter your JWT token in the api_key field, preceded by the Bearer prefix:
This project is a community effort and lives off your contributions, be it in the form of bug reports, feature requests, discussions, or fixes and other code changes. Please refer to our organization's contributing guidelines if you are interested to contribute. Please mind the code of conduct for all interactions with the community.
Development of the app is currently still in beta stage, and current versions are for internal use only. We are aiming to have a fully spec-compliant version of the app available soon. The plan is to then adopt a semantic versioning scheme in which we would shadow WES spec versioning, with added date stamp patches for any patch-level changes to our service.
This project is covered by the Apache License 2.0 also shipped with this repository.
The project is a collaborative effort under the umbrella of ELIXIR Cloud & AAI. Follow the link to get in touch with us via chat or email. Please mention the name of this service for any inquiry, proposal, question etc.