Merge branch 'kenny/sm-gatk' of github.com:latchbio/latch into kenny/…

…sm-gatk

.github/workflows/test.yaml

@@ -1,56 +1,13 @@
 name: Test
-on: [push, pull_request_target]
-
+on: push
 jobs:
   macos:
-    name: Test MacOs
+    name: Test
     strategy:
       fail-fast: false
       matrix:
-        python: ["3.8", "3.9"]
-        platform: ["macos-10.15"]
-    runs-on: ${{ matrix.platform }}
-    steps:
-      - name: Checkout
-        uses: actions/checkout@v2
-      - name: Setup Python
-        uses: actions/setup-python@v2
-        with:
-          python-version: ${{ matrix.python }}
-      - name: Download Docker
-        run: |
-          brew install docker docker-machine
-
-          # Latest virtualbox has breaking change for mac
-          brew uninstall virtualbox
-          cd $(brew --repo homebrew/cask)
-          git checkout 8670a72380c57c606d6582b645421e31dad2eee2
-          brew install --cask virtualbox
-
-          # Avoids throttling git api in `docker-machine create` cmd
-          curl --create-dirs -Lo /Users/runner/.docker/machine/cache/boot2docker.iso https://github.com/boot2docker/boot2docker/releases/download/v18.09.1-rc1/boot2docker.iso
-
-          docker-machine create --driver virtualbox default
-          docker-machine env default
-      - name: Install dev
-        run: |
-          python3 -m pip install -r dev-requirements.txt
-      - name: Install latch
-        run: |
-          python3 -m pip install -e .
-      - name: Test
-        env:
-          TEST_TOKEN: ${{ secrets.TEST_TOKEN }}
-        run: |
-          eval $(docker-machine env default)
-          cd tests; python3 -m pytest -s .
-  linux:
-    name: Test Linux
-    strategy:
-      fail-fast: false
-      matrix:
-        python: ["3.8", "3.9"]
-        platform: ["ubuntu-18.04"]
+        python: ["3.8", "3.9", "3.10", "3.11"]
+        platform: ["macos-12", "ubuntu-22.04"]
     runs-on: ${{ matrix.platform }}
     steps:
       - name: Checkout
@@ -61,13 +18,12 @@ jobs:
           python-version: ${{ matrix.python }}
       - name: Install dev
         run: |
-          python3 -m pip install -r dev-requirements.txt
+          pip install -r dev-requirements.txt
       - name: Install latch
         run: |
-          python3 -m pip install -e .
+          pip install .
       - name: Test
         env:
           TEST_TOKEN: ${{ secrets.TEST_TOKEN }}
         run: |
-          eval $(docker-machine env default)
-          cd tests; python3 -m pytest -s .
+          pytest -s

.gitignore

@@ -18,5 +18,4 @@ docs/build
 .vscode
 scratch.py
 /scratch
-test_*
 .latch_report.tar.gz

.pre-commit-config.yaml

@@ -36,7 +36,7 @@ repos:
       # - id: sort-simple-yaml
       # - id: trailing-whitespace
   - repo: https://github.com/psf/black
-    rev: 23.7.0
+    rev: 23.9.1
     hooks:
       - id: black
         args: [--preview]

CHANGELOG.md

@@ -16,7 +16,7 @@ Types of changes
 
 # Latch SDK Changelog
 
-## 2.33.0 - 2023-10-04
+## 2.34.0 - 2023-10-04
 
 ### Added
 
@@ -34,6 +34,12 @@ temporary because they are deleted at the end of each job on Latch.
 * Detect use of `conda` keyword and install in image. This effectively supports wrapper/conda keywords.
 * `Iterable, Generator` cause issues as type hints when imported from `collections.abc` rather than `typing`
 
+## 2.33.0 - 2023-09-29
+
+### Added
+
+* Add `latch sync` for synchronization from local to remote directories that only uploads modified content
+
 ## 2.32.8 - 2023-09-07
 
 ### Fixed

docs/source/index.md

@@ -149,6 +149,7 @@ cli/mv
 :maxdepth: 2
 :caption: Manual
 manual/snakemake.md
+manual/tutorial.md
 ```
 
 ```{toctree}

docs/source/manual/snakemake.md

@@ -1,10 +1,4 @@
-# [Alpha Pre-release] Snakemake Integration
-
-## Pre-release Disclaimer - Currently not ready for production use
-
-Latch support for Snakemake is in active development. Some workflows already work but a lot of common use cases need minor work. This documentation is also in active development.
-
-This pre-release was created to integrate miscellaneous improvements that accumulated over the course of developing the integration and to prevent the codebase from further diverging from the main branch.
+# Snakemake Integration
 
 ## Getting Started
 

docs/source/manual/tutorial.md

@@ -0,0 +1,144 @@
+# A simple Snakemake example
+
+In this guide, we will walk through how you can upload a simple Snakemake workflow to Latch.
+
+The example being used here comes from the [short tutorial in Snakemake's documentation](https://snakemake.readthedocs.io/en/stable/tutorial/short.html).
+
+## Prerequisites
+
+* Install the [Latch SDK](https://github.com/latchbio/latch#installation) and [snakemake](https://snakemake.readthedocs.io/en/stable/getting_started/installation.html) with:
+
+```console
+pip install latch[snakemake]
+```
+
+* Install [Docker](https://www.docker.com/get-started/) and have Docker run locally
+
+## Step 1
+
+First, initialize an example Snakemake workflow:
+
+```console
+latch init snakemake-wf --template snakemake
+```
+
+The workflow generated contains what is typically seen in a Snakemake workflow, such as python scripts and a Snakefile.
+
+```console
+snakemake-wf
+├── Dockerfile # Latch specific
+├── Snakefile
+├── data
+│   ├── genome.fa
+│   ├── genome.fa.amb
+│   ├── genome.fa.ann
+│   ├── genome.fa.bwt
+│   ├── genome.fa.fai
+│   ├── genome.fa.pac
+│   ├── genome.fa.sa
+│   └── samples
+│       ├── A.fastq
+│       ├── B.fastq
+│       └── C.fastq
+├── environment.yaml
+├── scripts
+│   ├── __pycache__
+│   │   └── plot-quals.cpython-39.pyc
+│   └── plot-quals.py
+├── version
+└── wf
+```
+
+To make the workflow compatible to execute on Latch, two additional files are needed:
+
+* `Dockerfile` to specify dependencies the workflow needs to run
+* `latch_metadata.py` to specify workflow parameters to expose on the user interface.
+
+In this tutorial, we will walk through how these two files can be constructed.
+
+## Step 2: Add a metadata file
+
+The `latch_metadata.py` is used to specify the input parameters that the Snakemake workflow needs to run.
+
+For example, by examining the Snakefile, we determine there are two parameters that the workflow needs: a reference genome and a list of samples to be aligned against the reference genome.
+
+```python
+# latch_metadata.py
+from latch.types.metadata import SnakemakeMetadata, SnakemakeFileParameter
+from latch.types.directory import LatchDir
+from latch.types.metadata import LatchAuthor, LatchMetadata, LatchParameter
+from pathlib import Path
+
+SnakemakeMetadata(
+    display_name="snakemake_tutorial_workflow",
+    author=LatchAuthor(
+            name="latchbio",
+    ),
+    parameters={
+        "samples" : SnakemakeFileParameter(
+                display_name="Sample Input Directory",
+                description="A directory full of FastQ files",
+                type=LatchDir,
+                path=Path("data/samples"),
+        ),
+        "ref_genome" : SnakemakeFileParameter(
+                display_name="Indexed Reference Genome",
+                description="A directory with a reference Fasta file and the 6 index files produced from `bwa index`",
+                type=LatchDir,
+                path=Path("genome"),
+        ),
+    },
+)
+```
+
+For each `LatchFile`/`LatchDir` parameter, the `path` keyword specifies the path where files will be copied before the Snakemake workflow is run and should match the paths of the inputs for each rule in the Snakefile.
+
+## Step 3: Add dependencies
+
+Next, create an `environment.yaml` file to specify the dependencies that the Snakefile needs to run successfully:
+
+```python
+# environment.yaml
+channels:
+  - bioconda
+  - conda-forge
+dependencies:
+  - snakemake=7.25.0
+  - jinja2
+  - matplotlib
+  - graphviz
+  - bcftools =1.15
+  - samtools =1.15
+  - bwa =0.7.17
+  - pysam =0.19
+  - pygments
+```
+
+A Dockerfile can be automatically generated by typing:
+```console
+latch dockerfile snakemake-wf --snakemake
+```
+
+## Step 3: Upload the workflow to Latch
+
+Finally, type the following command to register the workflow to Latch:
+
+```console
+cd snakemake-wf &&\ 
+latch register . --snakefile Snakefile
+```
+
+During registration, a workflow image is built based on dependencies specified in the `environment.yaml` file. Once the registration finishes, the `stdout` provides a link to your workflow on Latch.
+
+![Snakemake workflow interface on Latch](../assets/snakemake/tutorial.png)
+
+## Step 4: Run the workflow
+
+Snakemake support is currently uses JIT (Just-In-Time) registration. This means that the workflow produced by `latch register` will register a second workflow, which will run the actual Snakemake jobs.
+
+Once the workflow finishes running, results will be deposited to [Latch Data](https://console.latch.bio/data) under the `Snakemake Outputs` folder.
+
+## Next Steps
+
+* Learn more in-depth about how Snakemake integration works on Latch by reading our [manual](../manual/snakemake.md).
+* Visit the repository of [public examples](https://github.com/latchbio/latch-snakemake-examples) of Snakemake workflows on Latch.

latch/account.py

@@ -155,14 +155,12 @@ def load(self) -> None:
     @overload
     def list_registry_projects(
         self, *, load_if_missing: Literal[True] = True
-    ) -> List[Project]:
-        ...
+    ) -> List[Project]: ...
 
     @overload
     def list_registry_projects(
         self, *, load_if_missing: bool
-    ) -> Optional[List[Project]]:
-        ...
+    ) -> Optional[List[Project]]: ...
 
     def list_registry_projects(
         self, *, load_if_missing: bool = True

latch/registry/project.py

@@ -85,12 +85,10 @@ def load(self) -> None:
     # get_display_name
 
     @overload
-    def get_display_name(self, *, load_if_missing: Literal[True] = True) -> str:
-        ...
+    def get_display_name(self, *, load_if_missing: Literal[True] = True) -> str: ...
 
     @overload
-    def get_display_name(self, *, load_if_missing: bool) -> Optional[str]:
-        ...
+    def get_display_name(self, *, load_if_missing: bool) -> Optional[str]: ...
 
     def get_display_name(self, *, load_if_missing: bool = True) -> Optional[str]:
         """Get the display name of this project.
@@ -116,12 +114,10 @@ def get_display_name(self, *, load_if_missing: bool = True) -> Optional[str]:
     # list_tables
 
     @overload
-    def list_tables(self, *, load_if_missing: Literal[True] = True) -> List[Table]:
-        ...
+    def list_tables(self, *, load_if_missing: Literal[True] = True) -> List[Table]: ...
 
     @overload
-    def list_tables(self, *, load_if_missing: bool) -> Optional[List[Table]]:
-        ...
+    def list_tables(self, *, load_if_missing: bool) -> Optional[List[Table]]: ...
 
     def list_tables(self, *, load_if_missing: bool = True) -> Optional[List[Table]]:
         """List Registry tables contained in this project.

latch/registry/record.py

@@ -167,12 +167,10 @@ def load(self) -> None:
     # get_name
 
     @overload
-    def get_name(self, *, load_if_missing: Literal[True] = True) -> str:
-        ...
+    def get_name(self, *, load_if_missing: Literal[True] = True) -> str: ...
 
     @overload
-    def get_name(self, *, load_if_missing: bool) -> Optional[str]:
-        ...
+    def get_name(self, *, load_if_missing: bool) -> Optional[str]: ...
 
     def get_name(self, *, load_if_missing: bool = True) -> Optional[str]:
         """Get the name of this record.
@@ -200,16 +198,14 @@ def get_columns(
         self,
         *,
         load_if_missing: Literal[True] = True,
-    ) -> Dict[str, Column]:
-        ...
+    ) -> Dict[str, Column]: ...
 
     @overload
     def get_columns(
         self,
         *,
         load_if_missing: bool,
-    ) -> Optional[Dict[str, Column]]:
-        ...
+    ) -> Optional[Dict[str, Column]]: ...
 
     def get_columns(
         self,
@@ -238,16 +234,14 @@ def get_values(
         self,
         *,
         load_if_missing: Literal[True] = True,
-    ) -> Dict[str, RecordValue]:
-        ...
+    ) -> Dict[str, RecordValue]: ...
 
     @overload
     def get_values(
         self,
         *,
         load_if_missing: bool,
-    ) -> Optional[Dict[str, RecordValue]]:
-        ...
+    ) -> Optional[Dict[str, RecordValue]]: ...
 
     def get_values(
         self,