AnnotSV Singularity on an HPC Error `command not found` #184

yr542 · 2023-06-13T14:35:33Z

I am trying to use AnnotSV on my Illumina Manta vcf outputs specifically the diploid vcfs. However, I cannot seem to make it work with a HPO in Singularity.

There are 2 input variables:

singularity_container="/path/to/annotsv-3.3.4--py311hdfd78af_0.sif"
input_dir="path/to/Illumina_Manta_outputs

The code I used:

# Loop through the files in the input directory
for file in "$input_dir"/*.vcf; do
    # Get the base name of the file without extension
    filename=$(basename "$file" .vcf)

    # Create the output subdirectory within the input directory
    output_dir_parent="$(dirname "$input_dir")"
    decompressed_dir="$output_dir_parent/Step_3___AnnotSV_Main"
    mkdir -p "$decompressed_dir"
    echo "Output directory: $decompressed_dir"
    output_dir="$decompressed_dir/AnnotSV_Batch_Copied"
    mkdir -p "$output_dir"

    # Set the output file path with "_AnnotSV" appended
    output_file="$output_dir/${filename}_AnnotSV_"

    # Run AnnotSV via Singularity with the base name as output prefix
    singularity exec "$singularity_container" /usr/local/etc/AnnotSV/configfile -SVinputFile "$file" -genomeBuild GRCh38 -hpo "HP:0000009" -outputFile "$output_file" 
done

Yet I get the error:

/usr/local/etc/AnnotSV/configfile: line 108: P_loss_phen: command not found
/usr/local/etc/AnnotSV/configfile: line 109: P_loss_hpo: command not found
/usr/local/etc/AnnotSV/configfile: line 110: P_loss_source: command not found
/usr/local/etc/AnnotSV/configfile: line 111: P_loss_coord: command not found
/usr/local/etc/AnnotSV/configfile: line 112: P_ins_phen: command not found
/usr/local/etc/AnnotSV/configfile: line 113: P_ins_hpo: command not found
/usr/local/etc/AnnotSV/configfile: line 114: P_ins_source: command not found
/usr/local/etc/AnnotSV/configfile: line 115: P_ins_coord: command not found
/usr/local/etc/AnnotSV/configfile: line 116: P_inv_phen: command not found
/usr/local/etc/AnnotSV/configfile: line 117: P_inv_hpo: command not found
/usr/local/etc/AnnotSV/configfile: line 118: P_inv_source: command not found
/usr/local/etc/AnnotSV/configfile: line 119: P_inv_coord: command not found
/usr/local/etc/AnnotSV/configfile: line 120: po_P_gain_phen: command not found

but when I try to not define a path in the Singularity it tells me that it cannot find a refGene.txt. What do I do for it to get to work for my HPO. My HPO will change based on what is required.

The text was updated successfully, but these errors were encountered:

tsnowlan · 2023-06-13T15:15:05Z

Singularity is affected by your local environment, so it can be a bit finicky. Try using singularity exec --cleanenv ..., it can solve a lot of mysterious / hard to troubleshoot issues.

Beyond that, it seems like it might be trying to run the tcl file with bash. Can you post the Singularity definition file? It's usually at /.singularity.d/Singularity inside the image.

lgmgeo · 2023-06-13T15:33:40Z

Thanks Tor for helping @yr542

yr542 · 2023-06-13T16:28:27Z

How would I do that I can only tell you that I downloaded it?

yr542 · 2023-06-14T00:52:50Z

singularity exec --cleanenv "$singularity_container" /usr/local/etc/AnnotSV/configfile -SVinputFile "$file" -genomeBuild GRCh38 -hpo "HP:0001249" -outputFile "$output_file" -annotationsDir "$phenotype_to_genes"
did not change the error that persists

/usr/local/etc/AnnotSV/configfile: line 11: -benignAF:: command not found
/usr/local/etc/AnnotSV/configfile: line 13: -candidateGenesFiltering:: command not found
/usr/local/etc/AnnotSV/configfile: line 15: -genomeBuild:: command not found
/usr/local/etc/AnnotSV/configfile: line 17: -hpo:: command not found
/usr/local/etc/AnnotSV/configfile: line 19: -includeCI:: command not found

and the error continues.

tsnowlan · 2023-06-14T09:08:02Z

Ah, I see now. You're calling the configfile, which the image tries to interpret using /bin/sh, rather than AnnotSV. Try with:

singularity exec --cleanenv "$singularity_container" AnnotSV \
    -SVinputFile "$file" \
    -genomeBuild GRCh38 \
    -hpo "HP:0001249"

yr542 · 2023-06-14T13:55:21Z

Attempt 1: Running the command you gave above

When I did that I got this error ...checking the annotation data sources (June 14 2023 - 09:54) ############################################################################ "/usr/local/share/AnnotSV/Annotations_Human/Genes/GRCh38/refGene.txt.gz" file doesn't exist Please check your install - Exit with error. ############################################################################

Attempt 2: Adding the annotationsDir + path
I do think it needs to have an HPO reference file such as phenotype_to_genes.txt but it doesn't seem to accept it when I try and use your above command + annotationsDir because it says refGene.txt.gz file doesn't exist

Attempt 3: Adding the annotationsDir + path to zipped phentype_to_genes.txt (using bgzip) renamed to refGene.txt

But I get the error:

refGene.txt.gz" file doesn't exist
Please check your install - Exit with error.

when the path definitely exists.

lgmgeo · 2023-06-15T06:36:55Z

Can you try to bind your annotationsDir in Singularity?

singularity exec --bind /usr/local/share/AnnotSV/Annotations_Human \
    --cleanenv "$singularity_container" AnnotSV \
    -SVinputFile "$file" \
    -genomeBuild GRCh38 \
    -annotationsDir /usr/local/share/AnnotSV/Annotations_Human \
    -hpo "HP:0001249"

yr542 · 2023-06-15T13:07:58Z

When I do this I get the following error repeatedly

FATAL:   container creation failed: mount /usr/local/share/AnnotSV/Annotations_Human->/usr/local/share/AnnotSV/Annotations_Human error: while mounting /usr/local/share/AnnotSV/Annotations_Human: mount source /usr/local/share/AnnotSV/Annotations_Human doesn't exist

For the script there are 2 variables that have been defined:

singularity_container - the path to the singularity container on my HPC
input_dir a path to a folder with my Manta vcfs (only)

the script was modified according to your specifications and the modified version is placed below:

Attempt 1: Following you suggestion

for file in "$input_dir"/*.vcf; do
    # Get the base name of the file without extension
    filename=$(basename "$file" .vcf)

    # Create the output subdirectory within the input directory
    output_dir_parent="$(dirname "$input_dir")"
    decompressed_dir="$output_dir_parent/Step_3___AnnotSV_Main"
    mkdir -p "$decompressed_dir"
    echo "Output directory: $decompressed_dir"
    output_dir="$decompressed_dir/AnnotSV_Batch_Copied"
    mkdir -p "$output_dir"

    # Set the output file path with "_AnnotSV" appended
    output_file="$output_dir/${filename}_AnnotSV_"

    # Run AnnotSV via Singularity with the base name as output prefix
    singularity exec --bind /usr/local/share/AnnotSV/Annotations_Human \
    --cleanenv "$singularity_container" AnnotSV \ 
    -SVinputFile "$file" \
    -genomeBuild GRCh38 \
    -annotationsDir /usr/local/share/AnnotSV/Annotations_Human \
    -hpo "HP:0001249" \
    -outputFile "$output_file"
done

Attempt 2: Modifying the path fed to annotationsDir

Below is only the singularity command modified in the same for loop

# Run AnnotSV via Singularity with the base name as output prefix
    singularity exec --bind /usr/local/share/AnnotSV/Annotations_Human \
    --cleanenv "$singularity_container" AnnotSV -SVinputFile "$file" \
    -genomeBuild GRCh38 \
    -annotationsDir "$singularity_container" /usr/local/share/AnnotSV/Annotations_Human \
    -hpo "HP:0001249" \
    -outputFile "$output_file"

However, it results in the same errors in the .out file.

Attempt 3: Adding back the phenotype_to_genes path where phenotype_to_genes.txt has been zipped and renamed refGene.txt with the --bind

The for loop stayed the same except this section was modified shown below:

    # Run AnnotSV via Singularity with the base name as output prefix
    singularity exec --bind /usr/local/share/AnnotSV/Annotations_Human \
    --cleanenv "$singularity_container" AnnotSV -SVinputFile "$file" \
    -genomeBuild GRCh38 \
    -annotationsDir "$phenotype_to_genes" \
    -hpo "HP:0001249" \
    -outputFile "$output_file"

However, it resulted in yet again the same error.

Attempt 4: Trying cleaning with the phenotype_to_genes.txt has been zipped and renamed refGene.txt again

I only modified the following inside the for loop

    # Run AnnotSV via Singularity with the base name as output prefix
    singularity exec --cleanenv "$singularity_container" AnnotSV -SVinputFile "$file" \
    -genomeBuild GRCh38 \
    -annotationsDir "$phenotype_to_genes" \
    -hpo "HP:0001249" \
    -outputFile "$output_file"

And I got the following error when the file definitely exists:

############################################################################
"path/to/refGene.txt.gz" file doesn't exist
Please check your install - Exit with error.
############################################################################

It took the path to the directory I gave it and then tried to go to Annotations_Human/Genes/GRCh38/refGene.txt.gz the subdirectory Annotations_Human which I did not specify?

I do not believe I can use the singularity container path in the annotationsDir.

kiranpatil222 · 2023-06-15T16:18:10Z

If your SIF file works fine,, Please let me know the working command and Please provide Singularity DEF file for it

yr542 · 2023-06-15T17:05:15Z

I got the singularity file for AnnotSV when I did this singularity pull docker://quay.io/biocontainers/annotsv I do not believe I can attach my singularity file here?

kiranpatil222 · 2023-06-15T17:44:58Z

Oohh I See.. Biggest issue with Docker / Singularity when we pull,, there isnt any instructions on how to use its executables,, i also have had similar issues when i pull docker imager through singularity but then No idea how to use it

yr542 · 2023-06-15T18:02:47Z

Usually it goes with singularity exec but there is something that doesn't work well with the refGene this is why I have this post

nvnieuwk · 2023-06-16T07:05:25Z

Hi can you try binding the directory with your input files and annotations directory to the container? This can be done with --bind (see here for more information about this).

yr542 · 2023-06-21T18:29:30Z

I do not believe this AnnotSV singularity works for HPO does anyone have a working example using this Singularity?

nvnieuwk · 2023-06-22T07:31:16Z

I'm using it in a nextflow pipeline on an HPC cluster and it works perfectly for me...

yr542 · 2023-06-22T15:23:32Z

Could I have the code you used? As it does not seem to work for me?

nvnieuwk · 2023-06-23T07:16:56Z

Nextflow arranges the execution of singularity containers internally, so I'm not sure it'll help you, but here is the code that does the execution of AnnotSV: https://github.com/CenterForMedicalGeneticsGhent/nf-cmgg-structural/blob/add-expansionhunter/modules/nf-core/annotsv/annotsv/main.nf

kiranpatil222 · 2023-06-27T10:09:28Z

Did anyone able to run above annotsv with singularity

yr542 · 2023-06-28T15:25:12Z

I couldn't get it to work. If anyone has a specific example of how AnnotSV works could they please post it? The Nextflow link to code is a bit complicated and I was just looking for code specific to AnnotSV.

lgmgeo · 2023-06-29T06:30:29Z

Sorry, I never evaluated AnnotSV via Singularity ;o(

nvnieuwk · 2023-06-29T06:34:43Z

I'm not sure either why it's not working for everyone else. I didn't create the container myself (it's automatically generated via bioconda). Maybe you could ask more info on the bioconda gitter?

yr542 · 2023-06-29T20:06:03Z

I think this means that the AnnotSV does not work with Singularity?

nvnieuwk · 2023-06-30T07:44:53Z

Have you tried converting the docker container to a singularity image? maybe this could work?

yr542 · 2023-08-06T02:24:22Z

I can only put a Singularity on the cluster. Is there a way to use the Github repository and then a phenotype to genes file to make HPO terms?

lgmgeo added the Docker/Singularity/Bioconda label Jun 13, 2023

ManavalanG mentioned this issue Sep 6, 2023

Conda (and container) version of AnnotSV runs into error when run with -hpo arg #195

Open

lgmgeo mentioned this issue Sep 13, 2024

AnnotSV BioConda Environment: Reference Issues For GRCh38 #255

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

AnnotSV Singularity on an HPC Error `command not found` #184

AnnotSV Singularity on an HPC Error `command not found` #184

yr542 commented Jun 13, 2023 •

edited

Loading

tsnowlan commented Jun 13, 2023

lgmgeo commented Jun 13, 2023

yr542 commented Jun 13, 2023

yr542 commented Jun 14, 2023 •

edited

Loading

tsnowlan commented Jun 14, 2023

yr542 commented Jun 14, 2023 •

edited

Loading

lgmgeo commented Jun 15, 2023

yr542 commented Jun 15, 2023 •

edited

Loading

kiranpatil222 commented Jun 15, 2023

yr542 commented Jun 15, 2023

kiranpatil222 commented Jun 15, 2023

yr542 commented Jun 15, 2023

nvnieuwk commented Jun 16, 2023

yr542 commented Jun 21, 2023 •

edited

Loading

nvnieuwk commented Jun 22, 2023

yr542 commented Jun 22, 2023

nvnieuwk commented Jun 23, 2023

kiranpatil222 commented Jun 27, 2023

yr542 commented Jun 28, 2023

lgmgeo commented Jun 29, 2023

nvnieuwk commented Jun 29, 2023

yr542 commented Jun 29, 2023

nvnieuwk commented Jun 30, 2023

yr542 commented Aug 6, 2023

AnnotSV Singularity on an HPC Error command not found #184

AnnotSV Singularity on an HPC Error command not found #184

Comments

yr542 commented Jun 13, 2023 • edited Loading

tsnowlan commented Jun 13, 2023

lgmgeo commented Jun 13, 2023

yr542 commented Jun 13, 2023

yr542 commented Jun 14, 2023 • edited Loading

tsnowlan commented Jun 14, 2023

yr542 commented Jun 14, 2023 • edited Loading

lgmgeo commented Jun 15, 2023

yr542 commented Jun 15, 2023 • edited Loading

kiranpatil222 commented Jun 15, 2023

yr542 commented Jun 15, 2023

kiranpatil222 commented Jun 15, 2023

yr542 commented Jun 15, 2023

nvnieuwk commented Jun 16, 2023

yr542 commented Jun 21, 2023 • edited Loading

nvnieuwk commented Jun 22, 2023

yr542 commented Jun 22, 2023

nvnieuwk commented Jun 23, 2023

kiranpatil222 commented Jun 27, 2023

yr542 commented Jun 28, 2023

lgmgeo commented Jun 29, 2023

nvnieuwk commented Jun 29, 2023

yr542 commented Jun 29, 2023

nvnieuwk commented Jun 30, 2023

yr542 commented Aug 6, 2023

AnnotSV Singularity on an HPC Error `command not found` #184

AnnotSV Singularity on an HPC Error `command not found` #184

yr542 commented Jun 13, 2023 •

edited

Loading

yr542 commented Jun 14, 2023 •

edited

Loading

yr542 commented Jun 14, 2023 •

edited

Loading

yr542 commented Jun 15, 2023 •

edited

Loading

yr542 commented Jun 21, 2023 •

edited

Loading