From 8f128307db47151c233dec7455b25500f6ebc834 Mon Sep 17 00:00:00 2001 From: Emeline Favreau <9661216+EmelineFavreau@users.noreply.github.com> Date: Wed, 18 Oct 2023 12:26:03 +0200 Subject: [PATCH 1/8] Update cambridge.md The instructions include the latest changes () in config file: https://github.com/nf-core/configs/blob/master/conf/cambridge.config --- docs/cambridge.md | 48 ++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 45 insertions(+), 3 deletions(-) diff --git a/docs/cambridge.md b/docs/cambridge.md index 598ab7adc..cd84cbf3c 100644 --- a/docs/cambridge.md +++ b/docs/cambridge.md @@ -5,10 +5,52 @@ To use, run the pipeline with `-profile cambridge`. This will download and launc with a setup suitable for the Cambridge HPC cluster. Using this profile, either a docker image containing all of the required software will be downloaded, and converted to a Singularity image or a Singularity image downloaded directly before execution of the pipeline. -The latest version of Nextflow is not installed by default on the Cambridge HPC cluster. You will need to install it into a directory you have write access to. -Follow these instructions from the Nextflow documentation. +### Install Nextflow +The latest version of Nextflow is not installed by default on the Cambridge HPC cluster CSD3. You will need to install it into a directory you have write access to. +Follow [these instructions](https://www.nextflow.io/docs/latest/getstarted.html#) from the Nextflow documentation. -- Install Nextflow : [here](https://www.nextflow.io/docs/latest/getstarted.html#) +``` +# move to desired directory on HPC +cd /home//path/to/dir +# get the newest version +wget -qO- https://get.nextflow.io | bash +``` + +### Obtain updated java version +The java version on the HPC needs updating. + +``` +cd ~/ +wget https://download.oracle.com/java/20/latest/jdk-20_linux-x64_bin.tar.gz +tar xvfz jdk-20_linux-x64_bin.tar.gz +# add these lines to .bashrc +export JAVA_HOME=/home/ef479/jdk-20.0.1 +export PATH=/home/ef479/jdk-20.0.1/bin:$PATH +# Once above is done “java --version” should return: +java --version +java 20.0.1 2023-04-18 +``` + +### Set up Singularity +Singularity allows the use of containers. You also needs to make a directory to store Singularity cache. +``` +module load singularity +# make a directory for the cache +mkdir -p /home//rds/hpc-work/path/to/cache/dir +``` + +### Run Nextflow +Here is an example with the nf-core pipeline sarek ([read documentation here](https://nf-co.re/sarek/3.3.2)). +The user includes the project name, the node and the cache directory for Singularity. +``` +# create a working directory in rds/hpc-work +mkdir /home//rds/hpc-work/dir/to/test +cd /home//rds/hpc-work/dir/to/test + +# Launch the nf-core pipeline for a test database +# with the Cambridge profile +nextflow run nf-core/sarek -profile test,cambridge.config --partition "cclake" --project "NAME-SL3-CPU" --cacheDir "/home//rds/hpc-work/path/to/cache/dir" --outdir nf-sarek-test +``` All of the intermediate files required to run the pipeline will be stored in the `work/` directory. It is recommended to delete this directory after the pipeline has finished successfully because it can get quite large, and all of the main output files will be saved in the `results/` directory anyway. From faa406ed6fd3bc4630e67677276614d98fc83071 Mon Sep 17 00:00:00 2001 From: nf-core-bot Date: Wed, 18 Oct 2023 12:49:20 +0000 Subject: [PATCH 2/8] [automated] Fix linting with Prettier --- docs/cambridge.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/docs/cambridge.md b/docs/cambridge.md index cd84cbf3c..848eda30a 100644 --- a/docs/cambridge.md +++ b/docs/cambridge.md @@ -6,17 +6,19 @@ with a setup suitable for the Cambridge HPC cluster. Using this profile, either and converted to a Singularity image or a Singularity image downloaded directly before execution of the pipeline. ### Install Nextflow + The latest version of Nextflow is not installed by default on the Cambridge HPC cluster CSD3. You will need to install it into a directory you have write access to. Follow [these instructions](https://www.nextflow.io/docs/latest/getstarted.html#) from the Nextflow documentation. ``` -# move to desired directory on HPC +# move to desired directory on HPC cd /home//path/to/dir # get the newest version wget -qO- https://get.nextflow.io | bash ``` ### Obtain updated java version + The java version on the HPC needs updating. ``` @@ -31,8 +33,10 @@ java --version java 20.0.1 2023-04-18 ``` -### Set up Singularity +### Set up Singularity + Singularity allows the use of containers. You also needs to make a directory to store Singularity cache. + ``` module load singularity # make a directory for the cache @@ -40,8 +44,10 @@ mkdir -p /home//rds/hpc-work/path/to/cache/dir ``` ### Run Nextflow + Here is an example with the nf-core pipeline sarek ([read documentation here](https://nf-co.re/sarek/3.3.2)). The user includes the project name, the node and the cache directory for Singularity. + ``` # create a working directory in rds/hpc-work mkdir /home//rds/hpc-work/dir/to/test From 1f73a79c78029c656c5bdedb5ee066a83e31f766 Mon Sep 17 00:00:00 2001 From: Emeline Favreau <9661216+EmelineFavreau@users.noreply.github.com> Date: Tue, 24 Oct 2023 10:42:09 +0100 Subject: [PATCH 3/8] Update cambridge.md alternative way of getting nextflow using conda --- docs/cambridge.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/docs/cambridge.md b/docs/cambridge.md index 848eda30a..2d9d6ebc6 100644 --- a/docs/cambridge.md +++ b/docs/cambridge.md @@ -16,6 +16,25 @@ cd /home//path/to/dir # get the newest version wget -qO- https://get.nextflow.io | bash ``` +Alternatively, install Nextflow with conda + +``` +module load miniconda/3 + +# set up Bioconda according to the Bioconda documentation, notably setting up channels +conda config --add channels defaults +conda config --add channels bioconda +conda config --add channels conda-forge + +# create the environment env_nf, and install the tool nextflow +conda create --name env_nf nextflow + +# activate the environment containing nextflow +conda activate env_nf + +# once done with the environment, deactivate +conda deactivate +``` ### Obtain updated java version From fe1d2eceb96dd26b1ebdc460f98bbb27f5b49bdb Mon Sep 17 00:00:00 2001 From: Emeline Favreau <9661216+EmelineFavreau@users.noreply.github.com> Date: Wed, 25 Oct 2023 10:48:03 +0100 Subject: [PATCH 4/8] Update cambridge.md Conda install is suggested first as a preferred method. Singularity cache is now a bash variable. --- docs/cambridge.md | 55 +++++++++++++++++++++++------------------------ 1 file changed, 27 insertions(+), 28 deletions(-) diff --git a/docs/cambridge.md b/docs/cambridge.md index 2d9d6ebc6..2586966e8 100644 --- a/docs/cambridge.md +++ b/docs/cambridge.md @@ -7,16 +7,7 @@ and converted to a Singularity image or a Singularity image downloaded directly ### Install Nextflow -The latest version of Nextflow is not installed by default on the Cambridge HPC cluster CSD3. You will need to install it into a directory you have write access to. -Follow [these instructions](https://www.nextflow.io/docs/latest/getstarted.html#) from the Nextflow documentation. - -``` -# move to desired directory on HPC -cd /home//path/to/dir -# get the newest version -wget -qO- https://get.nextflow.io | bash -``` -Alternatively, install Nextflow with conda +The latest version of Nextflow is not installed by default on the Cambridge HPC cluster CSD3. You can install it with conda: ``` module load miniconda/3 @@ -36,45 +27,53 @@ conda activate env_nf conda deactivate ``` -### Obtain updated java version - -The java version on the HPC needs updating. +Alternatively, you can install Nextflow into a directory you have write access to. +Follow [these instructions](https://www.nextflow.io/docs/latest/getstarted.html#) from the Nextflow documentation. This alternative method requires also to update java. ``` -cd ~/ +# move to desired directory on HPC +cd /home//path/to/dir + +# get the newest version +wget -qO- https://get.nextflow.io | bash + +# update java version to the latest wget https://download.oracle.com/java/20/latest/jdk-20_linux-x64_bin.tar.gz tar xvfz jdk-20_linux-x64_bin.tar.gz -# add these lines to .bashrc -export JAVA_HOME=/home/ef479/jdk-20.0.1 -export PATH=/home/ef479/jdk-20.0.1/bin:$PATH -# Once above is done “java --version” should return: + +# if all tools are compatible with the java version you chose, add these lines to .bashrc +export JAVA_HOME=/home//path/to/dir/jdk-20.0.1 +export PATH=/home//path/to/dir/jdk-20.0.1/bin:$PATH + +# Once above is done `java --version` should return `java 20.0.1 2023-04-18` java --version -java 20.0.1 2023-04-18 + ``` ### Set up Singularity -Singularity allows the use of containers. You also needs to make a directory to store Singularity cache. +Singularity allows the use of containers and will use a caching strategy. First, you need to set the `NXF_SINGULARITY_CACHEDIR` bash environment variable, pointing at your hpc-work location. + +``` +# do this once per login, or add these lines to .bashrc +export NXF_SINGULARITY_CACHEDIR=/home//rds/hpc-work/path/to/cache/dir +``` + +Once done, and ready to use Nextflow, load the Singularity module. ``` module load singularity -# make a directory for the cache -mkdir -p /home//rds/hpc-work/path/to/cache/dir ``` ### Run Nextflow Here is an example with the nf-core pipeline sarek ([read documentation here](https://nf-co.re/sarek/3.3.2)). -The user includes the project name, the node and the cache directory for Singularity. +The user includes the project name and the node. ``` -# create a working directory in rds/hpc-work -mkdir /home//rds/hpc-work/dir/to/test -cd /home//rds/hpc-work/dir/to/test - # Launch the nf-core pipeline for a test database # with the Cambridge profile -nextflow run nf-core/sarek -profile test,cambridge.config --partition "cclake" --project "NAME-SL3-CPU" --cacheDir "/home//rds/hpc-work/path/to/cache/dir" --outdir nf-sarek-test +nextflow run nf-core/sarek -profile test,cambridge.config --partition "cclake" --project "NAME-SL3-CPU" --outdir nf-sarek-test ``` All of the intermediate files required to run the pipeline will be stored in the `work/` directory. It is recommended to delete this directory after the pipeline From 5b4c79de359aa63eecc83b5d4d7210f831a25d9d Mon Sep 17 00:00:00 2001 From: Emeline Favreau <9661216+EmelineFavreau@users.noreply.github.com> Date: Wed, 25 Oct 2023 10:50:59 +0100 Subject: [PATCH 5/8] Update cambridge.config This removes the params config that could have broken input validation. --- conf/cambridge.config | 1 - 1 file changed, 1 deletion(-) diff --git a/conf/cambridge.config b/conf/cambridge.config index 28aa063a3..9618e6083 100644 --- a/conf/cambridge.config +++ b/conf/cambridge.config @@ -15,7 +15,6 @@ params { singularity { enabled = true autoMounts = true - cacheDir = params.cacheDir } process { From 79d08b9f72fb2cd8e48d5f0f455bc0cba3033efc Mon Sep 17 00:00:00 2001 From: Emeline Favreau <9661216+EmelineFavreau@users.noreply.github.com> Date: Thu, 26 Oct 2023 10:09:50 +0100 Subject: [PATCH 6/8] Update cambridge.config removing cachedir in params --- conf/cambridge.config | 1 - 1 file changed, 1 deletion(-) diff --git a/conf/cambridge.config b/conf/cambridge.config index 9618e6083..8e5e3805c 100644 --- a/conf/cambridge.config +++ b/conf/cambridge.config @@ -5,7 +5,6 @@ params { config_profile_url = "https://docs.hpc.cam.ac.uk/hpc" partition = null project = null - cacheDir = null max_memory = 192.GB max_cpus = 56 max_time = 12.h From 783a21a601490b7126417b3634a4642d39a832bf Mon Sep 17 00:00:00 2001 From: Emeline Favreau <9661216+EmelineFavreau@users.noreply.github.com> Date: Fri, 27 Oct 2023 11:24:51 +0100 Subject: [PATCH 7/8] Update cambridge.md - singularity is now an option to load as module, because it should be loaded on login by default. - the cache directory contains more explanation as to why it can be created --- docs/cambridge.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/docs/cambridge.md b/docs/cambridge.md index 2586966e8..511913e67 100644 --- a/docs/cambridge.md +++ b/docs/cambridge.md @@ -50,18 +50,21 @@ java --version ``` -### Set up Singularity +### Set up Singularity cache -Singularity allows the use of containers and will use a caching strategy. First, you need to set the `NXF_SINGULARITY_CACHEDIR` bash environment variable, pointing at your hpc-work location. +Singularity allows the use of containers and will use a caching strategy. First, you might want to set the `NXF_SINGULARITY_CACHEDIR` bash environment variable, pointing at your hpc-work location. If not, it will be automatically assigned to the current directory. ``` # do this once per login, or add these lines to .bashrc export NXF_SINGULARITY_CACHEDIR=/home//rds/hpc-work/path/to/cache/dir ``` -Once done, and ready to use Nextflow, load the Singularity module. +Once done, and ready to use Nextflow, you can check that the Singularity module is loaded by default when logging on the cluster. ``` +module list + +# If singularity is not loaded: module load singularity ``` From 130f6e57070fcae44e40f1a5b359629e06eeabef Mon Sep 17 00:00:00 2001 From: nf-core-bot Date: Fri, 27 Oct 2023 11:01:58 +0000 Subject: [PATCH 8/8] [automated] Fix linting with Prettier --- docs/cambridge.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/cambridge.md b/docs/cambridge.md index 511913e67..55935e37b 100644 --- a/docs/cambridge.md +++ b/docs/cambridge.md @@ -27,7 +27,7 @@ conda activate env_nf conda deactivate ``` -Alternatively, you can install Nextflow into a directory you have write access to. +Alternatively, you can install Nextflow into a directory you have write access to. Follow [these instructions](https://www.nextflow.io/docs/latest/getstarted.html#) from the Nextflow documentation. This alternative method requires also to update java. ```