Skip to content

TypeScript

Jump to bottom
Alexis Lucattini edited this page Mar 1, 2023 · 16 revisions

Typescript in CWL

CWL supports JavaScript expressions, so why not write them in TypeScript?

Why TypeScript

TypeScript is like JavaScript but with support for types. This means that TypeScript can highlight potential errors in your code before running it, reducing the chance of bugs.

TypeScript has a lot of support with a large community making it an easy language to learn, and find the right answers to your problems via forums such as StackOverflow

TypeScript can "transpile" to JS ES5.1 which greatly increases CWL compatability.

Using an IDE such as VSCode or Webstorm can ensure that code errors are picked up quickly at create and compilation time, rather than at run time.

We can use the auto-generated cwl-ts-auto or rabix-cwlts to generate much of the code for us.

Package Management with Yarn

Death to node_modules, we're on the yarn v3 train!!

cwl-ica cli users can skip this step since this is all handled for you.

First ensure npm and therefore node is installed and is version 18+

$ type node
## node is hashed (/usr/bin/node)
$ node --version
## v18.8.0

Enable the corepack with the following command And get yarn ready for action!

corepack enable
corepack prepare yarn@stable --activate

Now make sure yarn is available, don't stress as to the version of yarn it is versioned in the project

$ type yarn
## yarn is hashed (/home/alexiswl/.yarn/bin/yarn)

$ yarn --version
1.22.19

Exercise 1: Create a CWL expression 'get bam file from directory' using TypeScript

The full output of this exercise can be seen with the existing expression at expressions/get-bam-file-from-directory/1.0.1

Aims

  • Create a CWL expression that can collect a bam file from a directory given a directory and the name of the bam file.
  • The JavaScript logic should be written in TypeScript and then transpiled into ES5 JavaScript
  • The JavaScript logic will then be imported into the CWL Expression using the $import syntax.

Create the CWL Expression template

If you don't have the cwl-ica software simply create a file with the contents shown below

cwl-ica create-expression-from-template \
  --expression-name get-bam-file-from-directory \
  --expression-version 1.0.2 \
  --username "${CWL_ICA_DEFAULT_USER}"

This should yield a file at expressions/get-bam-file-from-directory/1.0.2 with the following contents:

Empty Expression Contents
Click to expand! 
cwlVersion: v1.1
class: ExpressionTool

# Extensions
$namespaces:
    s: https://schema.org/
    ilmn-tes: http://platform.illumina.com/rdf/ica/
$schemas:
  - https://schema.org/version/latest/schemaorg-current-http.rdf

# Metadata
s:author:
    class: s:Person
    s:name: __INSERT_USER_NAME_HERE__
    s:email: __INSERT_EMAIL_ADDRESS_HERE__
    s:identifier: __INSERT_ORCID_IDENTIFIER_HERE__

# ID/Docs
id: get-bam-file-from-directory--1.0.2
label: get-bam-file-from-directory v(1.0.2)
doc: |
    Documentation for get-bam-file-from-directory v1.0.2

inputs: []

outputs: []

expression: "$\n{\n/*\nInsert Expression here\n*/\n}\n"

Setting the inputs

Let's set the inputs of the CWL Expression to the following:

inputs:
  input_dir:
    label: input dir
    doc: |
      Input directory with the bam file present
    type: Directory
  bam_nameroot:
    label: bam nameroot
    doc: |
      nameroot of the bam file
    type: string

Setting the outputs

Let's set the outputs of the CWL Expression to the following:

outputs:
  bam_file:
    label: bam file
    doc: |
      The bam file output with the .bai attribute
    type: File
    secondaryFiles:
      - pattern: ".bai"
        required: true

Set the expression

And let's set the expression component to the following

expression: >-
  ${
    return {"bam_file": get_bam_file_from_directory(inputs.input_dir, inputs.bam_nameroot)};
  }

We will then write the function get_bam_file_from_directory in the next steps in TypeScript.

Initialising the TypeScript directory

Head to the directory containing the CWL expression we've just added in the previous step.

This step aims to:

  • Initialise the tsconfig file (and edit the compiler options)
  • Ensure npm modules are installed

Via cwl-ica cli

We create our directory adjacent to the expression file with the following command

cwl-ica append-typescript-directory-to-cwl-expression-tool \
  --expression-path expressions/get-bam-file-from-directory/1.0.2/get-bam-file-from-directory__1.0.2.cwl

Manual

We can use the initialise_typescript_expression_directory.sh shell script in the cwl-ica cli bin directory.
An example of the invocation may be

initialise_typescript_expression_directory.sh \
  --typescript-expression-dir expressions/get-bam-file-from-directory/1.0.2/typescript-expressions/ \
  --package-name "get-bam-file-from-directory"

Both of these options run the following commands inside expressions/get-bam-file-from-directory/1.0.2/typescript-expressions/

# Initialise directory
$ yarn init -2

# Upgrade yarn to the latest stable version
$ yarn set version stable

# Do a install test
$ yarn install

# Check the version of yarn
$ yarn --version
## 3.2.3

# Add project requirements
yarn add --dev \
  typescript \
  ts-jest \
  jest \
  cwl-ts-auto \
  cwlts \
  @types/jest

The requirement source pages are linked below:

I recommend ignoring the following patterns in your git repo

  • .yarn/
  • .yarnrc.yml
  • .pnp*
  • node_modules/

By providing a yarn.lock and package.json file adjacent to our typescript expression, we ensure that the next developer is able to reproduce our efforts. These should remain committed / version controlled in your repository.

Observing the tsconfig json

Inside the typescript-expressions directory, there should exist a file called tsconfig.json.

This tells TypeScript to transpile to es5.

Observing the jest.config json

We also have a jest configuration file created for us.

This tells jest to look for files under tests/ that end in .test.ts.

We should see that a blank file already exists under tests/ with this naming convention.

Creating the TypeScript file

Note, please don't copy / paste, please type this out in your IDE (like webstorm) and watch the hints / magic unfold To assist with your IDE set up, please consult the IDE section at the bottom of this readme.

Inside the typescript-expressions directory, should exist a blank file called get-bam-file-from-directory__1.0.2.ts.

Typescript imports

Import File and Directory from the cwl-ts-auto package. Since CWL handles files and directories as interfaces at runtime, we use the IFile and IDirectory notation.

We also import the Directory and File class enums.

import {
    Directory_class,
    DirectoryProperties as IDirectory,
    File_class,
    FileProperties as IFile
} from "cwl-ts-auto";

Initialise the function

export function get_bam_file_from_directory(input_dir: IDirectory, bam_nameroot: string): IFile {
    
}

The input_dir is of type IDirectory, meaning it should adhere to the object construct

{
  "class": "Directory",
  "basename": "<string>",
  "location": "<...>"
}

While bam_nameroot is just a 'string'.

We place IFile at the end of the function definition to tell the TypeScript linter we expect a File object to be returned by this function.

Initialise the variables and check the inputs

We want to make sure that the input directory is actually a directory! The directory's listing attribute should be defined and not null

We also want to make sure that the bam nameroot is defined.

We then collect the listing as a variable asserting that it's a list of type File and Directory

export function get_bam_file_from_directory(input_dir: IDirectory, bam_nameroot: string): IFile {
    /*
    Initialise the output file object
    */
    let output_bam_obj: File | null = null
    let output_bam_index_obj: File | null = null
    
    /*
    Check input_dir is a directory and has a listing
    */
    if (input_dir.class_ === undefined || input_dir.class_ !== "Directory"){
        throw new Error("Could not confirm that the first argument was a directory")
    }
    if (input_dir.listing === undefined || input_dir.listing === null){
        throw new Error(`Could not collect listing from directory "${input_dir.basename}"`)
    }
    
    /*
    Check that the bam_nameroot input is defined
    */
    if (bam_nameroot === undefined || bam_nameroot === null){
        throw new Error("Did not receive a name of a bam file")
    }
    
    /*
    Collect listing as a variable
    */
    const input_listing: (IDirectory | IFile)[] = input_dir.listing
  
    // TBC ...
}

Find the bam file in the listing

Let's iterate over the listing and collect the bam file based on name matching

export function get_bam_file_from_directory(input_dir: Directory, bam_nameroot: string): File {
    // Initialise and checks...
  
    /*
    Iterate through the file listing
    */
    for (const listing_item of input_listing) {
        if (listing_item.class_ === File_class.FILE && listing_item.basename === bam_nameroot + ".bam"){
            /*
            Got the bam file
            */
            output_bam_obj = listing_item
            break
        }
    }
}

Finding the secondary file

Let's iterate over the listing item again and aim to find the secondary file for this bam file

export function get_bam_file_from_directory(input_dir: Directory, bam_nameroot: string): File {
    // Initialise, checks, and first listing loops ...
    /*
    Iterate through the listing again and look for the secondary file
    */
    for (const listing_item of input_listing){
        if (listing_item.class_ === File_class.FILE && listing_item.basename === bam_nameroot + ".bam.bai"){
            /*
            Got the bam index file
            */
            output_bam_index_obj = listing_item
            break
        }
    }
    
    // TBC...
}

Check we have found both objects

We throw errors if we couldn't find either the bam file or the bam index

export function get_bam_file_from_directory(input_dir: Directory, bam_nameroot: string): File {
    // Initialise, checks, and both listing loops
    /*
    Ensure we found the bam object
    */
    if (output_bam_obj === null){
        throw new Error(`Could not find bam file in the directory ${input_dir.basename}`)
    }
    
    /*
    Ensure we found the bam index object
    */
    if (output_bam_index_obj === null) {
        throw new Error(`Could not find secondary file in the directory ${input_dir.basename}`)
    }
    
    // TBC...
}

Assign secondary files and return

export function get_bam_file_from_directory(input_dir: Directory, bam_nameroot: string): File {
    // Initialise, checks, and both listing loops and output checks
    /*
    Assign bam index as a secondary file of the output bam object
    */
    output_bam_obj.secondaryFiles = [
        output_bam_index_obj
    ]

    return output_bam_obj
}

Final output

Click to expand! 
import {
    Directory_class,
    DirectoryProperties as IDirectory,
    File_class,
    FileProperties as IFile
} from "cwl-ts-auto";

export function get_bam_file_from_directory(input_dir: IDirectory, bam_nameroot: string): IFile {
    /*
    Initialise the output file object
    */
    let output_bam_obj: IFile | null = null
    let output_bam_index_obj: IFile | null = null

    /*
    Check input_dir is a directory and has a listing
    */
    if (input_dir.class_ === undefined || input_dir.class_ !== Directory_class.DIRECTORY){
        throw new Error("Could not confirm that the first argument was a directory")
    }
    if (input_dir.listing === undefined || input_dir.listing === null){
        throw new Error(`Could not collect listing from directory "${input_dir.basename}"`)
    }

    /*
    Check that the bam_nameroot input is defined
    */
    if (bam_nameroot === undefined || bam_nameroot === null){
        throw new Error("Did not receive a name of a bam file")
    }

    /*
    Collect listing as a variable
    */
    const input_listing: (IDirectory | IFile)[] = input_dir.listing

    /*
    Iterate through the file listing
    */
    for (const listing_item of input_listing) {
        if (listing_item.class_ === File_class.FILE && listing_item.basename === bam_nameroot + ".bam"){
            /*
            Got the bam file
            */
            output_bam_obj = listing_item
            break
        }
    }

    /*
    Iterate through the listing again and look for the secondary file
    */
    for (const listing_item of input_listing){
        if (listing_item.class_ === File_class.FILE && listing_item.basename === bam_nameroot + ".bam.bai"){
            /*
            Got the bam index file
            */
            output_bam_index_obj = listing_item
            break
        }
    }

    /*
    Ensure we found the bam object
    */
    if (output_bam_obj === null){
        throw new Error(`Could not find bam file in the directory ${input_dir.basename}`)
    }

    /*
    Check the secondary file has been defined
    */
    if (output_bam_obj.secondaryFiles !== undefined){
        // Picked up index object in the recursion step
    }
    else if (output_bam_index_obj === null) {
        throw new Error(`Could not find secondary file in the directory ${input_dir.basename}`)
    } else {
        /*
        Assign bam index as a secondary file of the output bam object
        */
        output_bam_obj.secondaryFiles = [
            output_bam_index_obj
        ]
    }

    return output_bam_obj
}

Advanced - Find the bam file recursively

The script we've just created will only look in the top directory of the listing, can we call this script recursively as to find the bam file in a subdirectory instead?

Let's add a third parameter to the function called recursive and set it to false by default

export function get_bam_file_from_directory(input_dir: IDirectory, bam_nameroot: string, recursive: boolean=false): IFile {
    // Logic
}

Then we'll add a clause to the first listing, where if the listing item is a directory, we call the function on that directory as an input.

export function get_bam_file_from_directory(input_dir: IDirectory, bam_nameroot: string, recursive: boolean=false): IFile {
    // Initialise, checks,
  
    /*
    Iterate through the file listing
    */
    for (const listing_item of input_listing) {
        if (listing_item.class_ === File_class.FILE && listing_item.basename === bam_nameroot + ".bam"){
            /*
            Got the bam file
            */
            output_bam_obj = listing_item
            break
        }
        if (listing_item.class_ === Directory_class.DIRECTORY && recursive){
            try {
                // Consider that the bam file might not be in this subdirectory and that is okay
                output_bam_obj = get_bam_file_from_directory(listing_item, bam_nameroot, recursive)
            } catch (error){
                // Dont need to report an error though, just continue
            }
            if (output_bam_obj !== null){
                break
            }
        }
    }
    // Output checks and return statement
}

The final product should look like so

Click to expand! 
import {
    Directory_class,
    DirectoryProperties as IDirectory,
    File_class,
    FileProperties as IFile
} from "cwl-ts-auto";

export function get_bam_file_from_directory(input_dir: IDirectory, bam_nameroot: string, recursive: boolean=false): IFile {
    /*
    Initialise the output file object
    */
    let output_bam_obj: IFile | null = null
    let output_bam_index_obj: IFile | null = null

    /*
    Check input_dir is a directory and has a listing
    */
    if (input_dir.class_ === undefined || input_dir.class_ !== Directory_class.DIRECTORY){
        throw new Error("Could not confirm that the first argument was a directory")
    }
    if (input_dir.listing === undefined || input_dir.listing === null){
        throw new Error(`Could not collect listing from directory "${input_dir.basename}"`)
    }

    /*
    Check that the bam_nameroot input is defined
    */
    if (bam_nameroot === undefined || bam_nameroot === null){
        throw new Error("Did not receive a name of a bam file")
    }

    /*
    Collect listing as a variable
    */
    const input_listing: (IDirectory | IFile)[] = input_dir.listing

    /*
    Iterate through the file listing
    */
    for (const listing_item of input_listing) {
        if (listing_item.class_ === File_class.FILE && listing_item.basename === bam_nameroot + ".bam"){
            /*
            Got the bam file
            */
            output_bam_obj = listing_item
            break
        }
        if (listing_item.class_ === Directory_class.DIRECTORY && recursive){
            try {
                // Consider that the bam file might not be in this subdirectory and that is okay
                output_bam_obj = get_bam_file_from_directory(listing_item, bam_nameroot, recursive)
            } catch (error){
                // Dont need to report an error though, just continue
            }
            if (output_bam_obj !== null){
                break
            }
        }
    }

    /*
    Iterate through the listing again and look for the secondary file
    */
    for (const listing_item of input_listing){
        if (listing_item.class_ === File_class.FILE && listing_item.basename === bam_nameroot + ".bam.bai"){
            /*
            Got the bam index file
            */
            output_bam_index_obj = listing_item
            break
        }
    }

    /*
    Ensure we found the bam object
    */
    if (output_bam_obj === null){
        throw new Error(`Could not find bam file in the directory ${input_dir.basename}`)
    }

    /*
    Check the secondary file has been defined
    */
    if (output_bam_obj.secondaryFiles !== undefined){
        // Picked up index object in the recursion step
    }
    else if (output_bam_index_obj === null) {
        throw new Error(`Could not find secondary file in the directory ${input_dir.basename}`)
    } else {
        /*
        Assign bam index as a secondary file of the output bam object
        */
        output_bam_obj.secondaryFiles = [
            output_bam_index_obj
        ]
    }

    return output_bam_obj
}

Transpile TypeScript to JavaScript

We use the tsc function to compile our code to JavaScript.

Since we have a cleaned directory we use the cwl-ica typescript-expression-validate command to temporarily:

  • install all the requirements inside package.json
  • convert the typescript into javascript,
  • run the test suite
  • convert the javascript into specialised cwljs code.

Non cwl-ica cli users can use the validate_typescript_expressions_directory.sh script from the cwl-ica bin directory to complete this process.

cwl-ica typescript-expression-validate \
  --typescript-expression-dir "expressions/get-bam-file-from-directory/1.0.2/typescript-expressions/"
# OR
validate_typescript_expressions_directory.sh \
  --typescript-expression-dir "expressions/get-bam-file-from-directory/1.0.2/typescript-expressions/" \
  --cwlify-js-code

This should create a file called get_bam_file_from_directory__1.0.2.js.

Note all of our typing is gone, our const and let declarations have all been converted to var.
The TypeScript for loops have also been converted to simpler iterables.
Our backtick strings have also been replaced with concat methods.

We should also have a .cwljs file in our directory, this is the one we will include in our expression.

If you haven't done so already, I recommend overriding the cwljs file type as JavaScript in your IDE.

Build the test suite

Note: Please don't copy and paste, just watch the magic happen as you type!

Inside the tests folder there should be a file called get_bam_file_from_directory__1.0.2.test.ts, that magically passed in the previous step.

The .test.ts suffix is important for jest to know which files to read.

Update the test file with the following contents

import {
    Directory_class,
    DirectoryProperties as IDirectory,
    File_class,
    FileProperties as IFile
} from "cwl-ts-auto";

import {get_bam_file_from_directory} from "../get-bam-file-from-directory__1.0.2";

Setting our inputs for our test file

Add the following variables to our tests file.

We are emulating the input_dir for a CWL directory here.

Click to expand! 
// imports

// Test the get bam file from directory
const INPUT_BAM_FILE_NAMEROOT = "footest"
const INPUT_BAM_FILE_NAMEEXT = ".bam"
const INPUT_BAM_FILE_BASENAME = INPUT_BAM_FILE_NAMEROOT + INPUT_BAM_FILE_NAMEEXT
const INPUT_OUTPUT_DIRECTORY_LOCATION = "outputs/output-directory"

// When in the top directory
const INPUT_SHALLOW_LISTING: IDirectory = {
    class_: Directory_class.DIRECTORY,
    location: `${INPUT_OUTPUT_DIRECTORY_LOCATION}`,
    listing: [
        {
            class_: File_class.FILE,
            basename: "index.html",
            location: `${INPUT_OUTPUT_DIRECTORY_LOCATION}/index.html`
        },
        {
            class_: File_class.FILE,
            basename: `${INPUT_BAM_FILE_BASENAME}`,
            location: `${INPUT_OUTPUT_DIRECTORY_LOCATION}/${INPUT_BAM_FILE_BASENAME}`
        },
        {
            class_: File_class.FILE,
            basename: `${INPUT_BAM_FILE_BASENAME}.bai`,
            location: `${INPUT_OUTPUT_DIRECTORY_LOCATION}/${INPUT_BAM_FILE_BASENAME}.bai`
        },
        {
            class_: File_class.FILE,
            basename: "logs.txt",
            location: `${INPUT_OUTPUT_DIRECTORY_LOCATION}/logs.txt`
        }
    ]
}

// When in a nested directory
const INPUT_NESTED_DIRECTORY_NAME = "nested-directory";
const INPUT_DEEP_LISTING: IDirectory = {
    class_: Directory_class.DIRECTORY,
    location: `${INPUT_OUTPUT_DIRECTORY_LOCATION}`,
    listing: [
        {
            class_: File_class.FILE,
            basename: "index.html",
            location: `${INPUT_OUTPUT_DIRECTORY_LOCATION}/index.html`
        },
        {
            class_: Directory_class.DIRECTORY,
            location: `${INPUT_OUTPUT_DIRECTORY_LOCATION}/${INPUT_NESTED_DIRECTORY_NAME}`,
            listing: [
                {
                    class_: File_class.FILE,
                    basename: `${INPUT_BAM_FILE_BASENAME}`,
                    location: `${INPUT_OUTPUT_DIRECTORY_LOCATION}/${INPUT_NESTED_DIRECTORY_NAME}/${INPUT_BAM_FILE_BASENAME}`
                },
                {
                    class_: File_class.FILE,
                    basename: `${INPUT_BAM_FILE_BASENAME}.bai`,
                    location: `${INPUT_OUTPUT_DIRECTORY_LOCATION}/${INPUT_NESTED_DIRECTORY_NAME}/${INPUT_BAM_FILE_BASENAME}.bai`
                },
                {
                    class_: File_class.FILE,
                    basename: "logs.txt",
                    location: `${INPUT_OUTPUT_DIRECTORY_LOCATION}/${INPUT_NESTED_DIRECTORY_NAME}/logs.txt`
                }
            ]
        }
    ]
}

Add in the expected outputs

We expect a CWL File object as an output.

We define that accordingly

Click to expand! 
const EXPECTED_SHALLOW_LISTING_OUTPUT_BAM_FILE: IFile = {
    class_: File_class.FILE,
    basename: `${INPUT_BAM_FILE_BASENAME}`,
    location: `${INPUT_OUTPUT_DIRECTORY_LOCATION}/${INPUT_BAM_FILE_BASENAME}`,
    secondaryFiles: [
        {
            class_: File_class.FILE,
            basename: `${INPUT_BAM_FILE_BASENAME}.bai`,
            location: `${INPUT_OUTPUT_DIRECTORY_LOCATION}/${INPUT_BAM_FILE_BASENAME}.bai`
        }
    ]
}

const EXPECTED_DEEP_LISTING_OUTPUT_BAM_FILE: File = {
    class_: File_class.FILE,
    basename: `${INPUT_BAM_FILE_BASENAME}`,
    location: `${INPUT_OUTPUT_DIRECTORY_LOCATION}/${INPUT_NESTED_DIRECTORY_NAME}/${INPUT_BAM_FILE_BASENAME}`,
    secondaryFiles: [
        {
            class_: File_class.FILE,
            basename: `${INPUT_BAM_FILE_BASENAME}.bai`,
            location: `${INPUT_OUTPUT_DIRECTORY_LOCATION}/${INPUT_NESTED_DIRECTORY_NAME}/${INPUT_BAM_FILE_BASENAME}.bai`
        }
    ]
}

Add in the tests

We use the describe and test functions here to ensure the expected output objects equal the obtained objects after running the inputs through the get_bam_file_from_directory function.

Head to the ts-jest docs to learn what other comparisons can be done

describe('Test Shallow Directory Listing', function () {
    // Get script path
    test("Test the get bam file from directory function non-recursively", () => {
        expect(
            get_bam_file_from_directory(INPUT_SHALLOW_LISTING, INPUT_BAM_FILE_NAMEROOT)
        ).toMatchObject(EXPECTED_SHALLOW_LISTING_OUTPUT_BAM_FILE)
    })
});

describe('Test Deep Directory Listing', function () {
    // Get script path
    test("Test the deep listing function", () => {
        expect(
            get_bam_file_from_directory(INPUT_DEEP_LISTING, INPUT_BAM_FILE_NAMEROOT, true)
        ).toMatchObject(EXPECTED_DEEP_LISTING_OUTPUT_BAM_FILE)
    })
});

Final test file

The final test file should look something like this:

Click to expand! 
import {
    Directory_class,
    DirectoryProperties as IDirectory,
    File_class,
    FileProperties as IFile
} from "cwl-ts-auto";

import {get_bam_file_from_directory} from "../get-bam-file-from-directory__1.0.2";

// Test the get bam file from directory
const INPUT_BAM_FILE_NAMEROOT = "footest"
const INPUT_BAM_FILE_NAMEEXT = ".bam"
const INPUT_BAM_FILE_BASENAME = INPUT_BAM_FILE_NAMEROOT + INPUT_BAM_FILE_NAMEEXT
const INPUT_OUTPUT_DIRECTORY_LOCATION = "outputs/output-directory"

// When in the top directory
const INPUT_SHALLOW_LISTING: IDirectory = {
    class_: Directory_class.DIRECTORY,
    location: `${INPUT_OUTPUT_DIRECTORY_LOCATION}`,
    listing: [
        {
            class_: File_class.FILE,
            basename: "index.html",
            location: `${INPUT_OUTPUT_DIRECTORY_LOCATION}/index.html`
        },
        {
            class_: File_class.FILE,
            basename: `${INPUT_BAM_FILE_BASENAME}`,
            location: `${INPUT_OUTPUT_DIRECTORY_LOCATION}/${INPUT_BAM_FILE_BASENAME}`
        },
        {
            class_: File_class.FILE,
            basename: `${INPUT_BAM_FILE_BASENAME}.bai`,
            location: `${INPUT_OUTPUT_DIRECTORY_LOCATION}/${INPUT_BAM_FILE_BASENAME}.bai`
        },
        {
            class_: File_class.FILE,
            basename: "logs.txt",
            location: `${INPUT_OUTPUT_DIRECTORY_LOCATION}/logs.txt`
        }
    ]
}

// When in a nested directory directory
const INPUT_NESTED_DIRECTORY_NAME = "nested-directory";
const INPUT_DEEP_LISTING: IDirectory = {
    class_: Directory_class.DIRECTORY,
    location: `${INPUT_OUTPUT_DIRECTORY_LOCATION}`,
    listing: [
        {
            class_: File_class.FILE,
            basename: "index.html",
            location: `${INPUT_OUTPUT_DIRECTORY_LOCATION}/index.html`
        },
        {
            class_: Directory_class.DIRECTORY,
            location: `${INPUT_OUTPUT_DIRECTORY_LOCATION}/${INPUT_NESTED_DIRECTORY_NAME}`,
            listing: [
                {
                    class_: File_class.FILE,
                    basename: `${INPUT_BAM_FILE_BASENAME}`,
                    location: `${INPUT_OUTPUT_DIRECTORY_LOCATION}/${INPUT_NESTED_DIRECTORY_NAME}/${INPUT_BAM_FILE_BASENAME}`
                },
                {
                    class_: File_class.FILE,
                    basename: `${INPUT_BAM_FILE_BASENAME}.bai`,
                    location: `${INPUT_OUTPUT_DIRECTORY_LOCATION}/${INPUT_NESTED_DIRECTORY_NAME}/${INPUT_BAM_FILE_BASENAME}.bai`
                },
                {
                    class_: File_class.FILE,
                    basename: "logs.txt",
                    location: `${INPUT_OUTPUT_DIRECTORY_LOCATION}/${INPUT_NESTED_DIRECTORY_NAME}/logs.txt`
                }
            ]
        }
    ]
}


const EXPECTED_SHALLOW_LISTING_OUTPUT_BAM_FILE: IFile = {
    class_: File_class.FILE,
    basename: `${INPUT_BAM_FILE_BASENAME}`,
    location: `${INPUT_OUTPUT_DIRECTORY_LOCATION}/${INPUT_BAM_FILE_BASENAME}`,
    secondaryFiles: [
        {
            class_: File_class.FILE,
            basename: `${INPUT_BAM_FILE_BASENAME}.bai`,
            location: `${INPUT_OUTPUT_DIRECTORY_LOCATION}/${INPUT_BAM_FILE_BASENAME}.bai`
        }
    ]
}

const EXPECTED_DEEP_LISTING_OUTPUT_BAM_FILE: File = {
    class_: File_class.FILE,
    basename: `${INPUT_BAM_FILE_BASENAME}`,
    location: `${INPUT_OUTPUT_DIRECTORY_LOCATION}/${INPUT_NESTED_DIRECTORY_NAME}/${INPUT_BAM_FILE_BASENAME}`,
    secondaryFiles: [
        {
            class_: File_class.FILE,
            basename: `${INPUT_BAM_FILE_BASENAME}.bai`,
            location: `${INPUT_OUTPUT_DIRECTORY_LOCATION}/${INPUT_NESTED_DIRECTORY_NAME}/${INPUT_BAM_FILE_BASENAME}.bai`
        }
    ]
}


describe('Test Shallow Directory Listing', function () {
    // Get script path
    test("Test the get bam file from directory function non-recursively", () => {
        expect(
            get_bam_file_from_directory(INPUT_SHALLOW_LISTING, INPUT_BAM_FILE_NAMEROOT)
        ).toMatchObject(EXPECTED_SHALLOW_LISTING_OUTPUT_BAM_FILE)
    })
});

describe('Test Deep Directory Listing', function () {
    // Get script path
    test("Test the deep listing function", () => {
        expect(
            get_bam_file_from_directory(INPUT_DEEP_LISTING, INPUT_BAM_FILE_NAMEROOT, true)
        ).toMatchObject(EXPECTED_DEEP_LISTING_OUTPUT_BAM_FILE)
    })
});

Running our tests

We re-run the validation scripts above, this will take a minute or so.

cwl-ica typescript-expression-validate \
  --typescript-expression-dir "expressions/get-bam-file-from-directory/1.0.2/typescript-expressions/"
# OR
validate_typescript_expressions_directory.sh \
  --typescript-expression-dir "expressions/get-bam-file-from-directory/1.0.2/typescript-expressions/" \
  --cwlify-js-code

Viewing the output summary

A file called summary.txt should now exist in our tests directory. This file should be committed and version controlled.

The summary file tells us the following information:

  • Did TypeScript have any warnings or errors transpiling?
  • Which tests passed and which failed?
  • How many lines of code / functions were covered by the tests?

Conversion of CWL-compatible JavaScript (.cwljs)

This is done by the validation commands above so don't need to reproduce this But it's good to know

The JS output from TypeScript cannot be imported into CWL just yet, otherwise CWL will complain.

We do a couple of magical touch ups with sed before proceeding.

Let's go through each of the components separately.

  • 1d removes the line use strict;
  • /^exports\.*/d; removes all exports lines i.e _exports._esModule = true;
  • /*require\("cwl-ts-auto"\)*/d; removes all require lines
  • /^Object\.defineProperty\(exports*/d; removes a Object definition property statement at the header
  • s%//(.*)%/* \1 */%; converts single line comments (that use the // syntax) to /comment/ syntax
  • s%class_%class%g; converts class_ attribute to class, since cwl-ts-auto will use the class_ over class attribute
  • s%:\ %:%g; converts ": " to ":" since CWL will otherwise interpret ": " as a yaml key
  • s%cwl_ts_auto_1.File_class.FILE%"File"%g; Replaces the File_class enum with just "File"
  • s%cwl_ts_auto_1.Directory_class.%"Directory"%g; Replaces the Directory_class enum with just "Directory"

Import our CWLJS into the CWL expression

Under the requirements.InlineJavascriptRequirement we include the JavaScript code

requirements:
  InlineJavascriptRequirement:
    expressionLib:
      - $include: ./typescript-expressions/get_bam_file_from_directory.cwljs

Deep listing updates

We add in the deep listing requirement if we wish to run our expression recursively

requirements:
  # InlineJavaScriptRequirement...
  LoadListingRequirement:
    loadListing: deep_listing

We must also update the expression such that the recursive parameter is set to true

expression: >-
  ${
    return {"bam_file": get_bam_file_from_directory(inputs.input_dir, inputs.bam_nameroot, true)};
  }

Test in cwl

Let's create a test directory

mkdir inputs-test
touch inputs-test/foo.bam
touch inputs-test/foo.bam.bai

Run the cwl expression

cwltool get-bam_file-from-directory__1.0.2.cwl --input_dir inputs-test --bam_nameroot foo

CWL-TS-AUTO alternatives

Rabix CWLTS

Source code: rabix/cwl-ts

I find this one easier to use over cwl-ts-auto for the following reasons:

  • files and directories are already set as interfaces
  • class attribute exists for files and directories rather than needing to use class_
  • class attribute is a string rather than an attribute of File_class.FILE
  • Nested listing is supported
  • Less sed 'magic' is required to convert from js to cwljs when using rabix-cwlts over cwl-ts-auto

I hope these features are soon available in the cwl-ts-auto repository.

To get started with the rabix-cwlts module have created an example under expressions/get-bam-file-from-directory/1.0.1-rabix to showcase the rabix-cwlts repo.

You will need to also perform the following steps.

Run the following code under the top directory / CWL_ICA_REPO_PATH

yarn add cwlts --dev

You should expect to see the package.json and yarn.lock files have been updated.

After creating the typescript expression directory with cwl-ica append-typescript-expression-to-cwl-expression-tool you will need to also run yarn add cwlts --dev in this directory. Again, you should expect to see updates in the package.json and yarn.lock files in your typescript-expression directory.

If a node_modules directory is created under your typescript-expression directory, please feel free to remove it.

Configuring your IDE 🚧

This is still a work in progress but I will share what I have so far.

Step 1: Install yarn into a virtualenv / conda env

For cwl-ica users, just make sure your conda env is up-to-date and re-run the install.sh script in the cwl-ica cli repo.

For non cwl-ica users, this install.sh script performs the following tasks on the cwl-ica conda repo that you will need to perform manually.

  1. Checks yarn is NOT installed by conda (removes it if so)
  2. Installs npm (v18+)
  3. Updates npm
  4. Checks npm prefix is the same as the conda prefix
  5. Runs corepack enable and corepack prepate yarn@stable --activate
  6. Instead installs yarn via npm
  7. Installs pnpm via npm

You may use the instructions under #initialising-the-typescript-directory to guide you.

Step 2: Add a package.json to the top of your git repo

This is CWL_ICA_REPO_PATH for cwl-ica users

The package.json should have the following contents

{
  "name": "cwl-ica",
  "packageManager": "[email protected]",
  "devDependencies": {
    "@types/jest": "^29.0.2",
    "@types/node": "^18.7.18",
    "cwl-ts-auto": "^0.1.3",
    "cwlts": "^1.23.6",
    "jest": "^29.0.3",
    "ts-jest": "^29.0.1",
    "typescript": "^4.8.3"
  }
}

Step 3: Run yarn install at the top of your repo

yarn install

This will create a few files under .yarn, please ensure that this is ignored by git.

WebStorm Users

VSCode users should head to the vscode section

Step 4: Create a project in Webstorm

This is ideally NOT in the same directory, saves having an extra .idea file in the directory.
I usually place mine in the default WebStormProjects in documents.

Step 5: Configure Webstorm

Configure Webstorm - make sure to click Apply after changing your settings!

Step 6: Set directories

Set the only content root to the top of your git repo

img.png

Step 7: Confirm Webstorm is working correctly

You should see the autocompletion of variable my_file similar to below

webstorm_confirm_working_correctly

Set JavaScript version

Set JavaScript version to 5.1

img_1.png

Set NodeJS

Set NodeJS binaries based on the installation inside your virtualenv / conda env.

img_2.png

Set TypeScript

Set node to the same value it was in the previous command.

But set TypeScript to yarn:package.json:typescript.

img_3.png

Invalidate caches

Then give webstorm a big all-mighty refresh by going File -> Invalidate Caches and ensuring that the Clear file system cache and Local History is checked.

VSCode Users

Step 4: Using npm to install dev requirements

Unfortunately, much of the yarn work comes unstuck for VSCode users,
hopefully this is just a temporary thing, for now you will need to let loose on node modules in the top of your directory

In the CLI use npm to install all the required items (and their dependencies) at the top of your cwl-ica repo.

npm install

Please ensure 'node_modules' is added to your .gitignore file.

Step 5: VSCode npm package manager settings

In VSCode to Settings -> Extensions -> Workspace -> Npm

Ensure that Package Manager is set to yarn

vscode_set_npm

Step 6: VSCode TypeScript Auto Type Acquisition

In VSCode to Settings -> Extensions -> Workspace -> TypeScript

Use npm for TypeScript Auto Type Acquisition

vscode_set_typescript

Step 7: Confirm VSCode is working correctly

Create a new file foo.ts in the top directory and see if the autocompletion for the variable my_file is working.

vscode_confirm_working_correctly

Clone this wiki locally