Feature request: Add support for Singularity containers in the tool wrapper

[ps-account]

Hi! There is currently no support for Singularity containers in the tool wrapper.

A solution would be to add a box similar to the Docker box, but for Singularity, which can then link to a local or public sregistry.

A workaround is to download the container beforehand and call singularity as a command-line application, but that makes the workflow quite a bit less portable.

[vsoch]

@pimpim I think maybe Singularity needs to be added as an integration to cwlts (I see as a dependency with DockerRequirement) first? --> https://github.com/rabix/cwl-ts/blob/bf9065e0403a4d609e8c7ae4d905bf1562344a11/src/mappings/d2sb/DockerRequirement.ts a maintainer here can probably clarify. If that mapping exists, then this portion seems pretty plug and play.

[vsoch]

And maintainers - if you are able to give a general gist / list of steps to take, I can gladly offer help to do this! I know Singularity pretty well, and webby and data structures pretty well, so with a little guided stumbling I think I could be of help. And it would be fun too :)

[ps-account]

Thanks for the pointer vsoch! Not sure if I shall open an issue at cwlts as well.. will just wait for responses from the rabix community first.

[ps-account]

There seems to be a cross-dependency on OCI compatibility of Singularity to make this a truly portable implementation: https://twitter.com/biocrusoe/status/947420025002086400

So I added an issue in the singularity repository as well: apptainer/singularity#1235

[ps-account]

Also of interest might be this: https://twitter.com/biocrusoe/status/947421085708021760

For now, you can work with your preferred CWL runner to add support for Singularity via CWL's "DockerRequirement". Less portably: a CWL runner could create their own extension: "$vendor_name:SingularityRequirement".

[mayacoda]

From what I gathered, Singularity isn't officially part of the CWL spec like Docker is, so I'm not sure if executors would know what to do with it. As far as Composer is concerned, you can add anything to the CWL that doesn't have its own GUI. It ignores all keys that cwlts doesn't encode, so anything you add to the code will stay there.

[vsoch]

@pimpim this was my intuition as well - you should first address adding Singularity to the CWL spec, and then it folds nicely into the cwltool and we can then come back to this particular integration! :)

[ps-account]

thanks @mayacoda and @vsoch, I also bumped into that. One prerequisite might be OCI compatibility of Singularity, so I made an issue to keep track of that apptainer/singularity#1235

On the CWL side there’ll be some effort to work on singularity-cwl compatibility in next week’s openbio winter codefest https://docs.google.com/document/d/1RlDUWRFqMcy4V2vvkA1_ENsVo6TXge2wIO_Nf73Itk0/mobilebasic

[ghost]

@rdwrt zombie thread alert! Was there an outcome on singularity and CWL? I can't see anything in the specs and the discussions were possibly before I rejoined the meetings. Thanks!

[mr-c]

The use of the Singularity engine to execute Docker format software containers was added the CWL reference runner using --singularity (though this might be optional in the future if we detect the presence of singularity and the absence of docker).

With this approach the CWL documents remain as portable as ever, no vendor extension is needed. Just regular DockerRequirements.

This approach has been adopted by other CWL runners including the toil-cwl-runner and is under evaluation for CWLEXEC (IBM's CWL implementation for LSF).

[ghost]

@mr-c super cool! Thanks for explaining and that is an elegant solution!