Merge pull request #419 from sunbeam-labs/418-prep-for-v410-release

Prep v4.1.0 release

.github/PULL_REQUEST_TEMPLATE.md

@@ -5,6 +5,6 @@
 
 If this is for a release:
 * [ ] I have updated documentation
-* [ ] I have updated all conda pin files (`snakedeploy update-conda-envs workflow/envs/*.yml`)
+* [ ] I have updated all conda pin files (`snakedeploy pin-conda-envs workflow/envs/*.yml`)
 * [ ] I have updated the hardcoded version at the top of `install.sh` to match what this release's version will be
-* [ ] I have created a release archive that will be attached to this release
+* [ ] I have created a release archive that will be attached to this release (`bash dev_scripts/generate_archive.sh`)

.github/workflows/test_tar_release.yml

@@ -0,0 +1,33 @@
+name: Test Tarball on Release
+
+on:
+  release:
+    types: [published]
+
+
+jobs:
+  test-tarball:
+    name: Test Tarball
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: setup-conda
+        uses: s-weigand/[email protected]
+        with:
+          # Additional channels like 'conda-forge' which can be used to install packages
+          conda-channels: '' # optional, default is 
+
+      - name: Download and Extract
+        shell: bash
+        run: |
+          wget https://github.com/sunbeam-labs/sunbeam/releases/latest/download/sunbeam.tar.gz
+          mkdir sunbeam
+          tar -zxf sunbeam.tar.gz -C sunbeam
+          cd sunbeam && ./install.sh -e sunbeam
+      
+      - name: Run Tests
+        shell: bash
+        run: |
+          source activate sunbeam
+          cd sunbeam
+          pytest -vvl tests/

dev_scripts/generate_archive.sh

@@ -1,4 +1,4 @@
 #!/bin/bash
 # $1 is the version for this release (e.g. 4.0.0 or 4.1.3-rc.2)
 
-tar -czvf sunbeam$1.tar.gz etc/ extensions/.placeholder sunbeamlib/ tests/ workflow/ environment.yml install.sh Readme.md setup.py MANIFEST.in pytest.ini
+tar -czv --exclude '__pycache__' -f sunbeam$1.tar.gz etc/ extensions/.placeholder src/sunbeamlib/ stable_env/ tests/ workflow/ environment.yml install.sh README.md pyproject.toml MANIFEST.in pytest.ini

docs/quickstart.rst

@@ -18,9 +18,9 @@ There are two installation methods available, installing via git or via tar. We
 
       .. code-block:: shell
 
-         wget https://github.com/sunbeam-labs/sunbeam/releases/download/v4.0.0/sunbeam.tar.gz
+         wget https://github.com/sunbeam-labs/sunbeam/releases/latest/download/sunbeam.tar.gz
          mkdir sunbeam4.0.0
-         tar -zxf sunbeam4.0.0.tar.gz -C sunbeam4.0.0
+         tar -zxf sunbeam.tar.gz -C sunbeam4.0.0
          cd sunbeam4.0.0 && ./install.sh
 
    .. tab:: git install

docs/structure.rst

@@ -26,7 +26,7 @@ sunbeam/ (root directory)
 -------------------------
 
 The root sunbeam directory holds a few important files including 
-``environment.yml``, ``setup.py``, and ``install.sh``. The environment file defines 
+``environment.yml``, ``pyproject.toml``, and ``install.sh``. The environment file defines 
 the dependencies required to run sunbeam and is used to create the main sunbeam 
 environment. The setup file defines the structure and dependencies of the 
 sunbeamlib_ and makes it installable via pip. The install script is used to install 
@@ -50,8 +50,7 @@ The additional files are all involved in the setup and deployment of the docs
 to ReadTheDocs using Sphinx. Most of these are autogenerated by Sphinx. The one 
 bit of trickiness comes from importing the version of sunbeam into the docs 
 build. This is done in ``conf.py`` by adding the sunbeam root to ``sys.path`` and 
-then importing ``sunbeamlib`` which stores the version tag in a ``__version__`` 
-variable using ``semantic_version``.
+then importing ``sunbeamlib`` which stores the version tag in ``__version__``.
 
 .. _envs:
 workflow/envs/
@@ -66,7 +65,7 @@ by default.
 
 The accompanying files named something like ``ENV_NAME.ARCH.pin.txt`` are generated 
 with ``snakedeploy``. They list all the packages and exact versions in a given 
-environment (and for the architecture they were generated on, e.g. linux-64) so that 
+environment (for the architecture they were generated on, e.g. linux-64) so that 
 snakemake can first try to use that exact environment and only if it fails, try to 
 solve the ``.yml`` file for itself.
 
@@ -93,14 +92,14 @@ rules. Again they are organized into subdirectories to match function and each
 is named according to the rule that calls it.
 
 .. _sunbeamlib:
-sunbeamlib/
+src/sunbeamlib/
 -----------
 
-This directory contains the python library that acts as a wrapper for 
-snakemake. The python files in the root contain a number of utility functions 
-whiles those in ``scripts/`` define the commands for sunbeam. 
-``scripts/command.py`` takes in ``sunbeam [cmd]`` and then routes it to the file 
-matching the given command. The ``data/`` directory contains the default config 
+This directory contains the python library that acts as a runner/utility for the 
+underlying snakemake. Many python files contain utility functions 
+whiles those prefixed by ``script_`` define the commands for sunbeam. 
+``script_sunbeam.py`` takes in ``sunbeam [cmd]`` and then routes it to the file 
+matching the given command. The ``.yml/.yaml`` data files include the default config 
 file as well as some sample config templates for running on a cluster. It also 
 contains the default profile template and one for slurm.
 
@@ -129,8 +128,8 @@ run by CircleCI as well as any scripts that are included in those jobs.
 
 This directory contains the ``PULL_REQUEST_TEMPLATE.md`` file which defines a 
 template for any pull requests on the sunbeam repository and ``ISSUE_TEMPLATE/`` 
-which contains issue templates for the repository. It is also where 
-the SuperLinter CI job definition lives.
+which contains issue templates for the repository. It is also where some
+CI job workflows live.
 
 .snakemake/
 ***********

docs/usage.rst

@@ -31,9 +31,9 @@ versions you want)).
 
       .. code-block:: shell
 
-         wget https://github.com/sunbeam-labs/sunbeam/releases/download/v4.0.0/sunbeam.tar.gz
+         wget https://github.com/sunbeam-labs/sunbeam/releases/latest/download/sunbeam.tar.gz
          mkdir sunbeam4.0.0
-         tar -zxf sunbeam4.0.0.tar.gz -C sunbeam4.0.0
+         tar -zxf sunbeam.tar.gz -C sunbeam4.0.0
          cd sunbeam4.0.0 && ./install.sh
 
    .. tab:: git install
@@ -77,13 +77,6 @@ installing or updating Sunbeam:
 If the tests fail, you should either refer to our troubleshooting guide or file
 an issue on our `Github page <https://github.com/sunbeam-labs/sunbeam/issues>`_.
 
-.. tip::
-
-  You can speed up the testing process by using the environment created during 
-  the install process with something like this 
-  'bash tests/run_tests.bash -e SUNBEAM_ENV_NAME'. Without this argument the 
-  script will create a temporary environment.
-
 .. _updating:
 Updating
 --------
@@ -101,9 +94,9 @@ upgrades, we will increment the patch or minor numbers (e.g. 1.0.0 ->
    git pull
    ./install.sh --upgrade all
 
-Sunbeam v3+ is designed to be installable separately on a system that already 
+Sunbeam is designed to be installable separately on a system that already 
 has sunbeam installed. This means multiple versions of sunbeam can be installed 
-on the same machine in different repositories.
+on the same machine in different repositories and different environments.
 
 .. _uninstall:
 Uninstalling or reinstalling
@@ -260,6 +253,8 @@ qc
   using cutadapt. Replace with ``""`` to skip.
 * ``cutadapt_opts``: (cutadapt) options to pass to cutadapt. Replace with ``""`` to pass no extra options.
 * ``kz_threshold``: a value between 0 and 1 to determine the low-complexity boundary (1 is most stringent). Ignored if not masking low-complexity sequences.
+* ``pct_id``: the percent identity threshold for filtering mapped reads.
+* ``frac``: the minimum fraction of the read that must be mapped to the reference to be kept.
 * ``host_fp``: the path to the folder with host/contaminant genomes (ending in
   *.fasta)
 

environment.yml

@@ -4,4 +4,4 @@ channels:
 dependencies:
   - snakemake
   - git # Ensure sunbeam extend works even with tar installation of main pipeline
-  - python>=3.10
+  - python =3.12.0

install.sh

@@ -1,7 +1,7 @@
 #!/usr/bin/env bash
 
 __conda_url=https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
-__version_tag=$(if git describe --tags >/dev/null 2>&1 ; then git describe --tags; else echo v4.0.0; fi) # <--- Update this on each version release
+__version_tag=$(if git describe --tags >/dev/null 2>&1 ; then git describe --tags; else echo v4.1.0; fi) # <--- Update this on each version release
 __version_tag="${__version_tag:1}" # Remove the 'v' prefix
 
 read -r -d '' __usage <<-'EOF'

workflow/envs/cutadapt.linux-64.pin.txt

@@ -5,26 +5,25 @@
 https://conda.anaconda.org/conda-forge/linux-64/_libgcc_mutex-0.1-conda_forge.tar.bz2#d7c89558ba9fa0495403155b64376d81
 https://conda.anaconda.org/conda-forge/linux-64/ca-certificates-2023.7.22-hbcca054_0.conda#a73ecd2988327ad4c8f2c331482917f2
 https://conda.anaconda.org/conda-forge/linux-64/ld_impl_linux-64-2.40-h41732ed_0.conda#7aca3059a1729aa76c597603f10b0dd3
-https://conda.anaconda.org/conda-forge/linux-64/libstdcxx-ng-13.1.0-hfd8a6a1_0.conda#067bcc23164642f4c226da631f2a2e1d
-https://conda.anaconda.org/conda-forge/linux-64/libgomp-13.1.0-he5830b7_0.conda#56ca14d57ac29a75d23a39eb3ee0ddeb
+https://conda.anaconda.org/conda-forge/linux-64/libstdcxx-ng-13.2.0-h7e041cc_2.conda#9172c297304f2a20134fc56c97fbe229
+https://conda.anaconda.org/conda-forge/linux-64/libgomp-13.2.0-h807b86a_2.conda#e2042154faafe61969556f28bade94b9
 https://conda.anaconda.org/conda-forge/linux-64/_openmp_mutex-4.5-2_gnu.tar.bz2#73aaf86a425cc6e73fcf236a5a46396d
-https://conda.anaconda.org/conda-forge/linux-64/libgcc-ng-13.1.0-he5830b7_0.conda#cd93f779ff018dd85c7544c015c9db3c
+https://conda.anaconda.org/conda-forge/linux-64/libgcc-ng-13.2.0-h807b86a_2.conda#c28003b0be0494f9a7664389146716ff
 https://conda.anaconda.org/conda-forge/linux-64/libffi-3.4.2-h7f98852_5.tar.bz2#d645c6d2ac96843a2bfaccd2d62b3ac3
-https://conda.anaconda.org/conda-forge/linux-64/libnsl-2.0.0-h7f98852_0.tar.bz2#39b1328babf85c7c3a61636d9cd50206
+https://conda.anaconda.org/conda-forge/linux-64/libnsl-2.0.0-hd590300_1.conda#854e3e1623b39777140f199c5f9ab952
 https://conda.anaconda.org/conda-forge/linux-64/libzlib-1.2.13-hd590300_5.conda#f36c115f1ee199da648e0597ec2047ad
 https://conda.anaconda.org/conda-forge/linux-64/ncurses-6.4-hcb278e6_0.conda#681105bccc2a3f7f1a837d47d39c9179
-https://conda.anaconda.org/conda-forge/linux-64/openssl-3.1.2-hd590300_0.conda#e5ac5227582d6c83ccf247288c0eb095
+https://conda.anaconda.org/conda-forge/linux-64/openssl-3.1.3-hd590300_0.conda#7bb88ce04c8deb9f7d763ae04a1da72f
 https://conda.anaconda.org/conda-forge/linux-64/xz-5.2.6-h166bdaf_0.tar.bz2#2161070d867d1b1204ea749c8eec4ef0
-https://conda.anaconda.org/conda-forge/linux-64/libsqlite-3.42.0-h2797004_0.conda#fdaae20a1cf7cd62130a0973190a31b7
+https://conda.anaconda.org/conda-forge/linux-64/libsqlite-3.43.2-h2797004_0.conda#4b441a1ee22397d5a27dc1126b849edd
+https://conda.anaconda.org/conda-forge/linux-64/pigz-2.8-h2797004_0.conda#1832561770273ca7cf52b989dd83e6c3
 https://conda.anaconda.org/conda-forge/linux-64/readline-8.2-h8228510_1.conda#47d31b792659ce70f470b5c82fdfb7a4
-https://conda.anaconda.org/conda-forge/linux-64/tk-8.6.12-h27826a3_0.tar.bz2#5b8c42eb62e9fc961af70bdd6a26e168
-https://conda.anaconda.org/conda-forge/linux-64/zlib-1.2.13-hd590300_5.conda#68c34ec6149623be41a1933ab996a209
-https://conda.anaconda.org/conda-forge/linux-64/pigz-2.6-h27826a3_0.tar.bz2#677798489006b14291e8617223e34348
-https://conda.anaconda.org/conda-forge/linux-64/sqlite-3.42.0-h2c6b66d_0.conda#1192f6ec654a5bc4ee1d64bdc4a3e5cc
+https://conda.anaconda.org/conda-forge/linux-64/tk-8.6.13-h2797004_0.conda#513336054f884f95d9fd925748f41ef3
+https://conda.anaconda.org/conda-forge/linux-64/sqlite-3.43.2-h2c6b66d_0.conda#c37b95bcd6c6833dacfd5df0ae2f4303
 https://conda.anaconda.org/conda-forge/linux-64/python-3.7.12-hf930737_100_cpython.tar.bz2#416558a6f46b7a1fa8db7d0e98aff56a
 https://conda.anaconda.org/conda-forge/noarch/bz2file-0.98-py_0.tar.bz2#784154d757e590f1f26c2427207bfe7a
-https://conda.anaconda.org/conda-forge/noarch/setuptools-68.0.0-pyhd8ed1ab_0.conda#5a7739d0f57ee64133c9d32e6507c46d
-https://conda.anaconda.org/conda-forge/noarch/wheel-0.41.1-pyhd8ed1ab_0.conda#8f467ba2db2b5470d297953d9c1f9c7d
+https://conda.anaconda.org/conda-forge/noarch/setuptools-68.2.2-pyhd8ed1ab_0.conda#fc2166155db840c634a1291a5c35a709
+https://conda.anaconda.org/conda-forge/noarch/wheel-0.41.2-pyhd8ed1ab_0.conda#1ccd092478b3e0ee10d7a891adbf8a4f
 https://conda.anaconda.org/conda-forge/noarch/pip-23.2.1-pyhd8ed1ab_0.conda#e2783aa3f9235225eec92f9081c5b801
 https://conda.anaconda.org/bioconda/noarch/xopen-0.7.3-py_0.tar.bz2#ab0a57604100736fb19c421c0c0828c1
 https://conda.anaconda.org/bioconda/linux-64/dnaio-0.3-py37h14c3975_1.tar.bz2#d58d26d5cac1482ffbb60c91a31a9222

workflow/envs/cutadapt.yml

@@ -1,7 +1,6 @@
 channels:
-  - conda-forge
   - bioconda
 dependencies:
   - cutadapt
-  - python =3.10.12
+  #- python =3.12.0
 name: cutadapt

workflow/envs/komplexity.linux-64.pin.txt

@@ -4,24 +4,25 @@
 @EXPLICIT
 https://conda.anaconda.org/eclarke/linux-64/komplexity-0.3.6-musl.tar.bz2#991933bd25d6344b08011f622a1187c8
 https://conda.anaconda.org/conda-forge/linux-64/_libgcc_mutex-0.1-conda_forge.tar.bz2#d7c89558ba9fa0495403155b64376d81
-https://conda.anaconda.org/conda-forge/linux-64/ca-certificates-2022.12.7-ha878542_0.conda#ff9f73d45c4a07d6f424495288a26080
+https://conda.anaconda.org/conda-forge/linux-64/ca-certificates-2023.7.22-hbcca054_0.conda#a73ecd2988327ad4c8f2c331482917f2
 https://conda.anaconda.org/conda-forge/linux-64/ld_impl_linux-64-2.40-h41732ed_0.conda#7aca3059a1729aa76c597603f10b0dd3
-https://conda.anaconda.org/conda-forge/noarch/tzdata-2023b-h71feb2d_0.conda#6c80c63151d7f493dab457a0ba6c2523
-https://conda.anaconda.org/conda-forge/linux-64/libgomp-12.2.0-h65d4601_19.tar.bz2#cedcee7c064c01c403f962c9e8d3c373
+https://conda.anaconda.org/conda-forge/noarch/tzdata-2023c-h71feb2d_0.conda#939e3e74d8be4dac89ce83b20de2492a
+https://conda.anaconda.org/conda-forge/linux-64/libgomp-13.2.0-h807b86a_2.conda#e2042154faafe61969556f28bade94b9
 https://conda.anaconda.org/conda-forge/linux-64/_openmp_mutex-4.5-2_gnu.tar.bz2#73aaf86a425cc6e73fcf236a5a46396d
-https://conda.anaconda.org/conda-forge/linux-64/libgcc-ng-12.2.0-h65d4601_19.tar.bz2#e4c94f80aef025c17ab0828cd85ef535
+https://conda.anaconda.org/conda-forge/linux-64/libgcc-ng-13.2.0-h807b86a_2.conda#c28003b0be0494f9a7664389146716ff
 https://conda.anaconda.org/conda-forge/linux-64/bzip2-1.0.8-h7f98852_4.tar.bz2#a1fd65c7ccbf10880423d82bca54eb54
+https://conda.anaconda.org/conda-forge/linux-64/libexpat-2.5.0-hcb278e6_1.conda#6305a3dd2752c76335295da4e581f2fd
 https://conda.anaconda.org/conda-forge/linux-64/libffi-3.4.2-h7f98852_5.tar.bz2#d645c6d2ac96843a2bfaccd2d62b3ac3
-https://conda.anaconda.org/conda-forge/linux-64/libnsl-2.0.0-h7f98852_0.tar.bz2#39b1328babf85c7c3a61636d9cd50206
-https://conda.anaconda.org/conda-forge/linux-64/libuuid-2.32.1-h7f98852_1000.tar.bz2#772d69f030955d9646d3d0eaf21d859d
-https://conda.anaconda.org/conda-forge/linux-64/libzlib-1.2.13-h166bdaf_4.tar.bz2#f3f9de449d32ca9b9c66a22863c96f41
-https://conda.anaconda.org/conda-forge/linux-64/ncurses-6.3-h27087fc_1.tar.bz2#4acfc691e64342b9dae57cf2adc63238
-https://conda.anaconda.org/conda-forge/linux-64/openssl-3.1.0-h0b41bf4_0.conda#2d833be81a21128e317325a01326d36f
+https://conda.anaconda.org/conda-forge/linux-64/libnsl-2.0.0-hd590300_1.conda#854e3e1623b39777140f199c5f9ab952
+https://conda.anaconda.org/conda-forge/linux-64/libuuid-2.38.1-h0b41bf4_0.conda#40b61aab5c7ba9ff276c41cfffe6b80b
+https://conda.anaconda.org/conda-forge/linux-64/libzlib-1.2.13-hd590300_5.conda#f36c115f1ee199da648e0597ec2047ad
+https://conda.anaconda.org/conda-forge/linux-64/ncurses-6.4-hcb278e6_0.conda#681105bccc2a3f7f1a837d47d39c9179
+https://conda.anaconda.org/conda-forge/linux-64/openssl-3.1.3-hd590300_0.conda#7bb88ce04c8deb9f7d763ae04a1da72f
 https://conda.anaconda.org/conda-forge/linux-64/xz-5.2.6-h166bdaf_0.tar.bz2#2161070d867d1b1204ea749c8eec4ef0
-https://conda.anaconda.org/conda-forge/linux-64/libsqlite-3.40.0-h753d276_0.tar.bz2#2e5f9a37d487e1019fd4d8113adb2f9f
+https://conda.anaconda.org/conda-forge/linux-64/libsqlite-3.43.2-h2797004_0.conda#4b441a1ee22397d5a27dc1126b849edd
 https://conda.anaconda.org/conda-forge/linux-64/readline-8.2-h8228510_1.conda#47d31b792659ce70f470b5c82fdfb7a4
-https://conda.anaconda.org/conda-forge/linux-64/tk-8.6.12-h27826a3_0.tar.bz2#5b8c42eb62e9fc961af70bdd6a26e168
-https://conda.anaconda.org/conda-forge/linux-64/python-3.11.0-he550d4f_1_cpython.conda#8d14fc2aa12db370a443753c8230be1e
-https://conda.anaconda.org/conda-forge/noarch/setuptools-67.6.0-pyhd8ed1ab_0.conda#e18ed61c37145bb9b48d1d98801960f7
-https://conda.anaconda.org/conda-forge/noarch/wheel-0.40.0-pyhd8ed1ab_0.conda#49bb0d9e60ce1db25e151780331bb5f3
-https://conda.anaconda.org/conda-forge/noarch/pip-23.0.1-pyhd8ed1ab_0.conda#8025ca83b8ba5430b640b83917c2a6f7
+https://conda.anaconda.org/conda-forge/linux-64/tk-8.6.13-h2797004_0.conda#513336054f884f95d9fd925748f41ef3
+https://conda.anaconda.org/conda-forge/linux-64/python-3.12.0-hab00c5b_0_cpython.conda#7f97faab5bebcc2580f4f299285323da
+https://conda.anaconda.org/conda-forge/noarch/setuptools-68.2.2-pyhd8ed1ab_0.conda#fc2166155db840c634a1291a5c35a709
+https://conda.anaconda.org/conda-forge/noarch/wheel-0.41.2-pyhd8ed1ab_0.conda#1ccd092478b3e0ee10d7a891adbf8a4f
+https://conda.anaconda.org/conda-forge/noarch/pip-23.2.1-pyhd8ed1ab_0.conda#e2783aa3f9235225eec92f9081c5b801

workflow/envs/komplexity.yml

@@ -1,7 +1,7 @@
-name: komplexity
 channels:
   - eclarke
   - conda-forge
 dependencies:
   - komplexity
-  - python>=3.10
+  - python =3.12.0
+name: komplexity