Nextflow Tower (Sequra) Wiki

Nextflow Tower/Sequra: A Comprehensive Open-Source platform for Bioinformatics

Nextflow is an open-source workflow management system designed to facilitate the development, execution, and sharing of data-intensive and scalable scientific workflows. It allows researchers and data scientists to define complex computational pipelines as code, making it easy to automate and reproduce their data analysis tasks across various computing environments. Nextflow Tower is the centralized command post for the management of Nextflow data pipelines with a graphical web application. It brings monitoring, logging, and observability to distributed workflows and simplifies the deployment of pipelines on any cloud, cluster, or laptop.

With Nextflow Tower, users can launch pre-configured pipelines with ease, while the flexible API provides programmatic integration to meet the needs of organizations building on Nextflow Tower. Workflow developers can publish pipelines to shared workspaces and administrators can set up and manage the infrastructure required to run data analysis at scale. To read more about the Tower with Sequra, click here.

*Note: Nextflow Tower is being rebranded to Seqera Platform! All WEHI RCP Bioimaging members should have access to Nextflow Tower, click here to see more details about access to Nextflow Tower with your WEHI account.

Some benefits of the Tower include:

• Running Nextflow pipelines without the need for command-line skills
• Monitoring pipelines run by anyone within a lab group
• Sharing of pipelines and pipeline configuration within WEHI

WEHI Workspaces

Currently, the following workspaces are available:

• WEHI_Genomics_Lab: For members of the Advanced Genomics Facility
• WEHI_Bahlo_Lab: For members of the Bahlo Lab
• WEHI_BAC: For members of the Bioimage Analysis Core (This is the one you should apply access for)
• WEHI: A general-purpose workspace that is visible to anyone in WEHI. Use this if you need to share your Nextflow executions with someone in WEHI, and there is no lab-specific workspace that you all share. Sensitive or private pipelines or data should not be used in this workspace.

Privacy Considerations

When choosing to use Nextflow Tower, please be aware that Nextflow Tower captures and stores the log files from Nextflow, which may contain:

• Input and output filenames
• Command output (standard out)
• Error messages (standard error)
• Sample identifiers
• Custom logging commands (view operators)
• Pipeline code including process names and configuration
• Pipeline parameter values
• Any reports you have configured with Tower: https://help.tower.nf/latest/reports/overview/

Only use Nextflow Tower if it is safe for:

• All members of the workspace (generally, an entire lab) to be able to access these logs
• Research Computing Platform to have access to these logs (RCP administers each workspace)
• BioCommons administrators to have access to these logs (BioCommons runs the Nextflow Tower server)
• Copies of the logs to be stored outside of WEHI, since Tower stores a copy of these logs

To run the Tower on SLURM-shell

This link will give you instructions on how to set up, run, and add new pipelines on the Tower via SLURM-shell. Here is a demo (To be edited).

Here is a GitHub link to a Nextflow template repository.