Skip to content

Commit

Permalink
Merge pull request #56 from Sage-Bionetworks-Workflows/bwmac/orca-256…
Browse files Browse the repository at this point in the history
…/failure_prototyping

[ORCA-253] Return Failure Reasons to `suites.json` and Failed Tests to `output.csv`
  • Loading branch information
BWMac authored May 10, 2024
2 parents b682d1e + cfea929 commit 1febf2e
Show file tree
Hide file tree
Showing 33 changed files with 1,063 additions and 691 deletions.
1 change: 1 addition & 0 deletions .github/CODEOWNERS
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
* @Sage-Bionetworks-Workflows/dpe
34 changes: 19 additions & 15 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,20 +2,20 @@
# MODIFY IT ACCORDING TO YOUR NEEDS!
# Reference: https://docs.github.com/en/actions

name: tests
name: test

on:
push:
# Avoid using all the resources/limits available by checking only
# relevant branches and tags. Other branches can be checked via PRs.
branches: [main]
tags: ['v[0-9]*', '[0-9]+.[0-9]+*'] # Match tags that resemble a version
pull_request: # Run in every PR
workflow_dispatch: # Allow manually triggering the workflow
tags: ["v[0-9]*", "[0-9]+.[0-9]+*"] # Match tags that resemble a version
pull_request: # Run in every PR
workflow_dispatch: # Allow manually triggering the workflow
schedule:
# Run roughly every 15 days at 00:00 UTC
# (useful to check if updates on dependencies break the package)
- cron: '0 0 1,16 * *'
- cron: "0 0 1,16 * *"

concurrency:
group: >-
Expand All @@ -31,9 +31,9 @@ jobs:
tarball-path: ${{ steps.distribution-paths.outputs.tarball }}
steps:
- uses: actions/checkout@v3
with: {fetch-depth: 0} # deep clone for setuptools-scm
with: { fetch-depth: 0 } # deep clone for setuptools-scm
- uses: actions/setup-python@v4
with: {python-version: "3.11"}
with: { python-version: "3.11" }
- name: Run static analysis and format checkers
run: pipx run pre-commit run --all-files --show-diff-on-failure
- name: Install tox-gh plugin
Expand All @@ -54,17 +54,21 @@ jobs:
name: python-distribution-files
path: dist/
retention-days: 1
- name: Keepalive Workflow
uses: gautamkrishnar/[email protected]
with:
time_elapsed: 44

test:
needs: prepare
strategy:
matrix:
python:
- "3.9" # oldest Python that is supported
- "3.11" # newest Python that is stable
- "3.9" # oldest Python that is supported
- "3.11" # newest Python that is stable
platform:
- ubuntu-latest
- macos-latest
- ubuntu-latest
- macos-latest
# TODO: Debug the Windows issues
# - windows-latest
env:
Expand All @@ -77,7 +81,7 @@ jobs:
with:
python-version: ${{ matrix.python }}
- uses: actions/download-artifact@v3
with: {name: python-distribution-files, path: dist/}
with: { name: python-distribution-files, path: dist/ }
- name: Install tox-gh plugin
run: python -m pip install tox-gh>=1.2
- name: Setup test suite
Expand Down Expand Up @@ -106,9 +110,9 @@ jobs:
steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
with: {python-version: "3.11"}
with: { python-version: "3.11" }
- uses: actions/download-artifact@v3
with: {name: python-distribution-files, path: dist/}
with: { name: python-distribution-files, path: dist/ }
- name: Publish Python Package to PyPI
env:
TWINE_REPOSITORY: pypi
Expand All @@ -123,7 +127,7 @@ jobs:
steps:
- uses: actions/checkout@v3
- uses: actions/download-artifact@v3
with: {name: python-distribution-files, path: dist/}
with: { name: python-distribution-files, path: dist/ }
- uses: docker/setup-qemu-action@v2
- uses: docker/setup-buildx-action@v2
- uses: docker/login-action@v2
Expand Down
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -57,3 +57,5 @@ MANIFEST
.venv*/
.conda*/
.python-version

debugging/
88 changes: 67 additions & 21 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,18 +43,18 @@ that any documentation update is done in the same way was a code contribution.
The documentation is written using [CommonMark] with [MyST] extensions.

:::{tip}
Please notice that the [GitHub web interface] provides a quick way of
propose changes in `dcqc`'s files. While this mechanism can
be tricky for normal code contributions, it works perfectly fine for
contributing to the docs, and can be quite handy.

If you are interested in trying this method out, please navigate to
the `docs` folder in the source [repository], find which file you
would like to propose changes and click in the little pencil icon at the
top, to open [GitHub's code editor]. Once you finish editing the file,
please write a message in the form at the bottom of the page describing
which changes have you made and what are the motivations behind them and
submit your proposal.
Please notice that the [GitHub web interface] provides a quick way of
propose changes in `dcqc`'s files. While this mechanism can
be tricky for normal code contributions, it works perfectly fine for
contributing to the docs, and can be quite handy.

If you are interested in trying this method out, please navigate to
the `docs` folder in the source [repository], find which file you
would like to propose changes and click in the little pencil icon at the
top, to open [GitHub's code editor]. Once you finish editing the file,
please write a message in the form at the bottom of the page describing
which changes have you made and what are the motivations behind them and
submit your proposal.
:::

When working on documentation changes in your local machine, you can
Expand Down Expand Up @@ -91,7 +91,7 @@ This often provides additional considerations and avoids unnecessary work.

1. Create an user account on GitHub if you do not already have one.

2. Fork the project [repository]: click on the *Fork* button near the top of the
2. Fork the project [repository]: click on the _Fork_ button near the top of the
page. This creates a copy of the code under your account on GitHub.

3. Clone this copy to your local disk:
Expand All @@ -108,7 +108,7 @@ This often provides additional considerations and avoids unnecessary work.
```

to create an isolated virtual environment containing package dependencies,
including those needed for development (*e.g.* testing, documentation).
including those needed for development (_e.g._ testing, documentation).

5. Install [pre-commit] hooks:

Expand Down Expand Up @@ -186,34 +186,79 @@ This often provides additional considerations and avoids unnecessary work.
the PR as a draft first and mark it as ready for review after the feedbacks
from the continuous integration (CI) system or any required fixes.

### Special Considerations for Contributing External Tests
### Contributing Internal Tests

In `py-dcqc`, any test where the primary business logic is executed within the package itself is considered internal. One example is the `Md5ChecksumTest`.

When contributing an internal test be sure to do the following:

1. Follow the steps above to set up `py-dcqc` and create your contribution.

1. Include a class docstring that describes the purpose of the test.

1. Include the following class attributes:

- `tier`: A `TestTier` enum describing the complexity of the test contributed. Valid `tier` values include:
- `FILE_INTEGRITY`
- `INTERNAL_CONFORMANCE`
- `EXTERNAL_CONFORMANCE`
- `SUBJECTIVE_CONFORMANCE`
- `target`: The target class that the test will be applied to. This value will be `SingleTarget` for individual files and `PairedTarget` for paired files.

1. Implement the major logic of the test in the `compute_status` method. This should include a condition for returning a `status` of `TestStatus.PASS` when the test conditions are met and `TestStatus.FAIL` when they are not.
- For failing cases be sure to include a line setting the class' `status_reason` to a helpful string that will tell users why the test failed before returning the `status`.

### Contributing External Tests

In `py-dcqc`, any test where the primary business logic is executed outside of this package itself is considered to be external. One example is the `LibTiffInfoTest`. For these tests, `py-dcqc` is responsible for packaging up a Nextflow process which is then executed in an [nf-dcqc](https://github.com/Sage-Bionetworks-Workflows/nf-dcqc) workflow run. Such tests are not possible to run in `py-dcqc` alone at this time. This makes contributing, testing, debugging, and using external tests a little more complicated that internal tests such as the `Md5ChecksumTest` which has all of its logic built into this package.

When contributing an external test, following these steps may be helpful:
When contributing an internal test be sure to do the following:

1. Follow the steps above to set up `py-dcqc` and create your contribution.

2. Follow the instructions in the [README.md](https://github.com/Sage-Bionetworks-Workflows/nf-dcqc/blob/dev/README.md)
1. Include a class docstring that describes the purpose of the test.

1. Include the following class attributes:

- `tier`: A `TestTier` enum describing the complexity of the test contributed. Valid `tier` values include:
- `FILE_INTEGRITY`
- `INTERNAL_CONFORMANCE`
- `EXTERNAL_CONFORMANCE`
- `SUBJECTIVE_CONFORMANCE`
- `pass_code`: The exit code that will be returned by the command indicating a passed test.
- `fail_code`: The exit code that will be returned by the command indicating a failed test.
- `failure_reason_location`: The file (either `"std_out"` or `"std_err"`) that will contain the reason for a failed test.
- `target`: The target class that the test will be applied to. This value will be `SingleTarget` for individual files and `PairedTarget` for paired files.

1. If possible, contribute an external test that returns different codes when it fails and when it errors out. Currently, a limitation of DCQC is that several external tests return the same `exit_code` when they fail and encounter an error. This will be addressed in future work that will add finer grained result interpretation.

### Testing Your Changes

1. Follow the instructions in the [README.md](https://github.com/Sage-Bionetworks-Workflows/nf-dcqc/blob/dev/README.md)
file in the `nf-dcqc` respository to set up the workflow on your local machine.

- Run `git checkout dev` to switch to the developer branch

3. Build your local version of `py-dcqc` with your new changes with:
1. Build your local version of `py-dcqc` with your new changes with:

```console
src/docker/build.sh
```

NOTE: This step assumes that you have docker installed and that it is running, and that you have `pipx` installed.

4. Follow `nf-dcqc` instructions to create a `nextflow run` command that tests your contribution.
1. Follow `nf-dcqc` instructions to create a `nextflow run` command that tests your contribution.

- You should include at least two files in your `nf-dcqc` input file ([example](https://github.com/Sage-Bionetworks-Workflows/nf-dcqc/blob/dev/testdata/input_full.csv)), one that you expect to pass your contributed test, and one that you expect to fail.
- Include the `local` profile so that the workflow leverages your locally built `py-orca` container

Example command (executed from within your local `nf-dcqc` repo clone):

```
nextflow run main.nf -profile local,docker --input path/to/your/input.csv -- outdir output --required_tests <YOUR_TEST_NAME>
```

5. Examine the final `suites.json` that is exported by the Nextflow workflow, if your contributed test bahaved as
1. Examine the final `output.csv` and `suites.json` files exported by the Nextflow workflow, if your contributed test bahaved as
expected, you're done! If not, debug and make changes to your contribution and re-run the workflow.

### Troubleshooting
Expand Down Expand Up @@ -291,7 +336,8 @@ on [PyPI], the following steps can be used to release a new version for
6. Run `tox -e publish -- --repository pypi` and check that everything was
uploaded to [PyPI] correctly.

[^contrib1]: Even though, these resources focus on open source projects and
[^contrib1]:
Even though, these resources focus on open source projects and
communities, the general ideas behind collaborating with other developers
to collectively create software are general and can be applied to all sorts
of environments, including private companies and proprietary code bases.
Expand Down
Loading

0 comments on commit 1febf2e

Please sign in to comment.