Skip to content

Commit

Permalink
Release 1.6.0 (#399)
Browse files Browse the repository at this point in the history
* WGS assay 4.0 (#366)

* cnv-purple-plugin
* other updates for WGS 4.0
---------

Co-authored-by: Felix Beaudry <[email protected]>
Co-authored-by: Aqsa Alam <[email protected]>
Co-authored-by: Aqsa Alam <[email protected]>
Co-authored-by: Felix Beaudry, Ph.D <[email protected]>

* Replace input_params_tools with read_maybe_input_params in workspace; update tests (#367)

* LOH data added to snv indel plugin

* Changed rendering of LOH

* Removed useless comments

* Update all curated gene list for oncoKB

* delete old curated genes file 2023116

* Fix gcgi 1317 write purple copy states (#369)

* Fix indentation, imports and return statements
* Fix imports, indentation, and return statements in main plugin
* Add write_copy_states to purple plugin
* Read by column headers instead of numbers
* Refactoring: Less fragile handling of paths and validation of purity/ploidy, general cleanup

* Fix gcgi 1320 cnv dependency (#370)

* Create new wgts.common package and move tools.py there
* move CNV code to common.cnv; simplify import path in process_CNA_data.r

* Added if statement for X chromosome

* Fixed tests for LOH

* Update genomic landscape to read purity from PURPLE output; check for existence of input path in TMB calculation (#371)

* Added test for when cnv data is missing

* Added conditional for if MACN is greater than 0.5 it needs to be False

* Fixed minor bug

* Dev gcgi 993 multiple plugin packages (#373)

* Support multiple top-level package names

* Added r directory

* Fixed metrix_cell to metric_cell

* updated changelog

* Support PWGS in mini-Djerba by making summary optional (#376)

* Warning instead of error on inconsistent sample names (#377)

* Added ARID2 to swi snf gene list

* Removed copy state from snv indel plugin

* Rewrite case_overview config; new core config method and assay constants; update changelog (#381)

* Updated CHANGELOG

* Remove obsolete provenance_tools from genomic_landscape (#384)

* Import the traceback module in core.render (#383)

* GCGI-1318, pWGS report improvements (#374)

* first draft of changes

* add detection cutoff constant to results_dict

* Add introductory sentence

* introductory sentence tweaks

* introductory sentence tweaks

* import case overview plugins

* no message

* no message

* Add extraction step to plugin.py

* include data from different plugin to the results dictionary

* Pull parameters from full.config.ini

* get path to the workspace directory

* pull pc.results from json output file

* write data to json

* Different pattern for extracted fields

* no message

* using built in method to write to json

* Use wrapper to access configuration parameters

* Added file not found exception

* file exist check and adjusting test

* checksum update and testing for scenario when file exists

* update summary/plugin_test.py checksum

---------

Co-authored-by: Oumaima Hamza <[email protected]>

* Edited definitions and disclaimers to include LOH

* Edited md5sums for supplement and snv indel

* Fixed sensitivity of CNVs from 30% to 100%

* pWGS formatting fixes (#385)

* Formatting fixes

* Summary tweak

* Update checksum

* Extra formatting tweaks

* Adjust padding

* Adjust padding

* Conditional page break

* Syntax fix

* no top margin

* Adjust padding

* no message

* Adjust bottom margin

* checksums update

---------

Co-authored-by: Oumaima Hamza <[email protected]>

* Write updated JSON by default, with a more informative name; update changelog and tests (#387)

* Dev gcgi 1344 checksum fix (#388)

* Update checksums after merging changes from release branch

* Update checksums; tests on 1.6.0 release branch now pass (#389)

* Update plugin lists for djerba.py setup mode

* Added HRDetect to assay description

* Updated CHANGELOG.md

* Edited md5sums for supplementary

* Added links for references

* Fix gcgi 1348 benchmark tests (#391)

* Update checksums; tests on 1.6.0 release branch now pass

* First stage of fixes to benchmarking; can now generate GSICAPBENCH_1219 report

* update changelog

* Reinstate all but one GSICAPBENCH samples; update tests

* Adjusting pWGS e-signature spacing (#392)

* adjust space for signature

* Checksum update

* adjust padding bottom

* no message

* Checksum update

---------

Co-authored-by: Oumaima Hamza <[email protected]>

* Merge changes from release v1.5.6 and fix conflicts (#396)

* GCGI-1346 setup.py and test/benchmark fixes  (#397)

* Update checksums

* Add init file for alternate Djerba package

* Add another init file

* Update CNV plugins in setup script

* More edits to package data list in setup script

* Update setup script and data README

* Add pwgs_cardea_helper to setup.py

* Reinstate Sequenza CNV plugin to install list

* Add HRD to list of plots for redaction

* Fix gcgi 1352 update examples (#398)

* Changed TERT protein annotation

* Changed always to auto for whitespace option in gene information merger

* Update CHANGELOG.md

* Update version.py

* Update WGTS examples

* Update changelog and version for release 1.6.0

---------

Co-authored-by: Aqsa Alam <[email protected]>
Co-authored-by: Aqsa Alam <[email protected]>

---------

Co-authored-by: Felix Beaudry <[email protected]>
Co-authored-by: Aqsa Alam <[email protected]>
Co-authored-by: Aqsa Alam <[email protected]>
Co-authored-by: Felix Beaudry, Ph.D <[email protected]>
Co-authored-by: Oumaima Hamza <[email protected]>
Co-authored-by: Oumaima Hamza <[email protected]>
  • Loading branch information
7 people authored May 9, 2024
1 parent 1cc0dfb commit 3aa3435
Show file tree
Hide file tree
Showing 150 changed files with 4,022 additions and 84,487 deletions.
41 changes: 40 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,54 @@
# CHANGELOG

## v1.6.0: 2024-05-08

### WGS 4.0

- Removed file list in expression data
- added HRD to genomic landscape
- GCGI-1173: genomic landscape plugin uses provenance_helper
- GCGI-1220: fixed MSI table formatting
- GCGI-1190: Removed obsolete data values/files
- GCGI-1012: Moved R testing to formal tests
- GCGI-1304: Merge updates to genomic landscape plugin and resolve conflicts
- GCGI-1319: Update genomic landscape plugin to read purity from PURPLE output file
- GCGI-1349: Update plugin lists for djerba.py setup mode
- GCGI-1334: Add HRDetect text to Djerba report footer
- new PURPLE CNV calling plugin
- removed sequenza support
- new `djerba.plugins.wgts.common` package for code shared between multiple WGTS/WGS plugins
- Added "r" directory to setup, bug fix (metrix_cell -> metric_cell)
- Added ARID2 to SWI/SNF gene list in CAPTIV8 plugin
- Removed copy state from SNVs and Indels plugin
- Added LOH to SNVs and Indels plugin

### External plugin support

- GCGI-993: Support for plugins outwith the main `djerba` package

### Other

- GCGI-1257: Fix import of traceback module
- GCGI-1313: Refactor reading input_params.json
- GCGI-1322: Warning instead of error if manually configured sample name does not match provenance
- GCGI-1323: Support PWGS in mini-Djerba by making summary optional
- GCGI-1325: Rewrite case_overview config method to fix bugs and make it clearer
- GCGI-1344: Write updated JSON by default, with a more informative name
- GCGI-1346: Fixes and updates to GSICAPBENCH
- GCGI-1352: Update WGTS example INI and PDF

## v.1.5.6: 2024-05-06

- Fix to TERT protein annotation (G>A -> C>T)
- Removed extra white space in gene information section

## v.1.5.5: 2024-03-05

- Further fixes to pWGS Cardea helper to allow multiple projects for one donor

## v1.5.4: 2024-03-01

- Fixed pWGS Cardea helper, pWGS provenance helper
- Fixed pWGS Cardea helper, pWGS provenance helper

## v1.5.3: 2024-02-22

Expand Down
8 changes: 8 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,14 @@ Reports are generated by the following types of component:

The report document is made up of one or more _sections_, generated by running plugins, helpers, and mergers in a specific order.

### Installation and package locations

Djerba is primarily written in Python. It has a `setup.py` script and can be installed using [pip](https://pypi.org/project/pip/). PDF conversion requires the [wkhtmltopdf](https://wkhtmltopdf.org/) binary, and individual plugins may have dependencies on other languages such as `R`.

The Djerba core code is contained in a Python package called `djerba`. Djerba components (plugins, helpers, and mergers) are also Python packages. As of version `1.6.0`, Djerba supports loading components from locations other than the main `djerba` package.

See [ReadTheDocs](https://djerba.readthedocs.io/en/latest/) for details.

## History

See the [changelog](./CHANGELOG.md) for a detailed development history. In brief:
Expand Down
60 changes: 0 additions & 60 deletions doc/old/update.md

This file was deleted.

Binary file modified examples/WGTS/PLACEHOLDER-v1_report.clinical.pdf
Binary file not shown.
48 changes: 22 additions & 26 deletions examples/WGTS/config.ini
Original file line number Diff line number Diff line change
Expand Up @@ -13,21 +13,6 @@ primary_cancer = Ovarian
oncotree_code = HGSOC
tcgacode = OV
requisition_approved = 2024/02/02
purity = 0.61
ploidy = 2.2
sequenza_reviewer_1 = NONE
sequenza_reviewer_2 = ALSO NONE

[wgts.snv_indel]
maf_path = example.mutect2.filtered.maf.gz

[cnv]
sequenza_gamma = 400
sequenza_solution = _primary_
sequenza_path = example.sequenza_results.zip

[expression_helper]
rsem_genes_results = example.gep.genes.results

[provenance_helper]
sample_name_whole_genome_normal=PLACEHOLDER
Expand All @@ -36,30 +21,41 @@ sample_name_whole_transcriptome=PLACEHOLDER
tumour_id=PLACEHOLDER
normal_id=PLACEHOLDER

[patient_info]
[report_title]

[genomic_landscape]
msi_file = example.filter.deduped.realigned.recalibrated.msi.booted
ctdna_file = example.SNP.count.txt
[patient_info]

[case_overview]
assay = WGTS

[treatment_options_merger]

[summary]
summary_file = example.results_summary.txt

[sample]
mean_coverage=101
callability=92

[summary]
summary_file = example.results_summary.txt
[genomic_landscape]
msi_file = PLACEHOLDER.filter.deduped.realigned.recalibrated.msi.booted
ctdna_file = PLACEHOLDER.SNP.count.txt
hrd_path = PLACEHOLDER.signatures.json

[supplement.header]
[expression_helper]
rsem_genes_results = PLACEHOLDER.genes.results

[supplement.body]
[wgts.snv_indel]
maf_path = PLACEHOLDER.filter.deduped.realigned.recalibrated.mutect2.filtered.maf.gz

[gene_information_merger]
[wgts.cnv_purple]
purple_zip = PLACEHOLDER.purple.zip

[treatment_options_merger]
[fusion]
mavis path = PLACEHOLDER.mavis_summary.tab
arriba_path = PLACEHOLDER.fusions.tsv

[report_title]
[gene_information_merger]

[supplement.body]

28 changes: 11 additions & 17 deletions setup.py
100755 → 100644
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
'html/*',
'resources/*',
'R/*',
'r/*',
'Rscripts/*'
]

Expand All @@ -45,27 +46,17 @@
package_data={
'djerba': [
'data/20200818-oncoKBcancerGeneList.tsv',
'data/20231116-allCuratedGenes.tsv',
'data/20201201-OncoTree.txt',
'data/20240116-translocation_annotations.txt',
'data/20240315-allCuratedGenes.tsv',
'data/OncoTree.json',
'data/NCCN_annotations.txt',
'data/benchmark_config.ini',
'data/benchmark_params.json',
'data/civic/01-Jun-2020-GeneSummaries.tsv',
'data/civic/01-Jun-2020-VariantGroupSummaries.tsv',
'data/civic/01-Jun-2020-VariantSummaries.tsv',
'data/cytoBand.txt',
'data/ensemble_conversion_hg38.txt',
'data/ensemble_conversion.txt',
'data/entrez_conversion.txt',
'data/filter_flags.exclude',
'data/gencode_v33_hg38_genes.bed',
'data/genomic_summary.txt',
'data/technical_notes.txt',
'data/gencode.v31.ensg_annotation_w_entrez.bed',
'data/hg38_centromeres.txt',
'data/pgacomp-tcga.txt',
'data/mutation_types.exonic',
'data/mutation_types.nonsynonymous',
'data/targeted_genelist.txt',
'data/tcga_code_key.txt',
'data/tmbcomp-externaldata.txt',
'data/tmbcomp-tcga.txt',
Expand All @@ -74,6 +65,7 @@
'djerba.helpers.expression_helper': install_wildcards,
'djerba.helpers.input_params_helper': install_wildcards,
'djerba.helpers.provenance_helper': install_wildcards,
'djerba.helpers.pwgs_cardea_helper': install_wildcards,
'djerba.helpers.tar_input_params_helper': install_wildcards,
'djerba.mergers.gene_information_merger': install_wildcards,
'djerba.mergers.treatment_options_merger': install_wildcards,
Expand All @@ -86,29 +78,31 @@
'djerba.plugins.failed_report': install_wildcards,
'djerba.plugins.fusion': install_wildcards,
'djerba.plugins.genomic_landscape': install_wildcards,
'djerba.plugins.hrd': install_wildcards,
'djerba.plugins.patient_info': install_wildcards,
'djerba.plugins.pwgs.analysis': install_wildcards,
'djerba.plugins.pwgs.case_overview': install_wildcards,
'djerba.plugins.pwgs.sample': install_wildcards,
'djerba.plugins.pwgs.summary': install_wildcards,
'djerba.plugins.report_title': install_wildcards,
'djerba.plugins.sample': install_wildcards,
'djerba.plugins.summary': install_wildcards,
'djerba.plugins.supplement': install_wildcards,
'djerba.plugins.supplement.body': install_wildcards,
'djerba.plugins.tar.sample': install_wildcards,
'djerba.plugins.tar.snv_indel': install_wildcards,
'djerba.plugins.tar.snv_indel.snv_indel_tools': install_wildcards,
'djerba.plugins.tar.swgs': install_wildcards,
'djerba.plugins.wgts.cnv_purple': install_wildcards,
'djerba.plugins.wgts.common.cnv': install_wildcards,
'djerba.plugins.wgts.snv_indel': install_wildcards,
'djerba.plugins.report_title': install_wildcards
'alternate_djerba.plugins.demo4': install_wildcards,
},
install_requires=[
'configparse',
'email_validator',
'jsonschema',
'mako',
'markdown',
'numpy',
'pandas',
'pdfkit',
'pyinstaller',
Expand Down
1 change: 0 additions & 1 deletion src/bin/djerba.py
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,6 @@ def get_parser():
update_parser.add_argument('-j', '--json', metavar='PATH', required=True, help='Path for JSON input')
update_parser.add_argument('-o', '--out-dir', metavar='DIR', required=True, help='Directory for output files')
update_parser.add_argument('-p', '--pdf', action='store_true', help='Generate PDF output from HTML')
update_parser.add_argument('-u', '--write-json', action='store_true', help='Write updated JSON to the output directory')
update_parser.add_argument('-w', '--work-dir', metavar='PATH', help='Path to workspace directory; optional, defaults to value of --out-dir')
update_parser.add_argument('--no-archive', action='store_true', help='Do not archive the JSON report file')
return parser
Expand Down
5 changes: 5 additions & 0 deletions src/lib/alternate_djerba/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# alternate_djerba

This package exists solely to test Djerba's functionality for loading plugins from multiple packages, added in release 1.6.0. It contains a single `demo4` plugin, which is a renamed version of the old `demo1` plugin.

It is *not* intended for production plugins, which should go either go into the main `djerba` package, or an alternate package with a more descriptive name.
File renamed without changes.
File renamed without changes.
Empty file.
58 changes: 58 additions & 0 deletions src/lib/alternate_djerba/plugins/demo4/plugin.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
"""Simple Djerba plugin for demonstration and testing: Example 1"""

import logging
from djerba.plugins.base import plugin_base
import djerba.core.constants as core_constants

class main(plugin_base):

PRIORITY = 200
PLUGIN_VERSION = '1.0.0'

# __init__ is inherited from the parent class

def configure(self, config):
config = self.apply_defaults(config)
return config

def extract(self, config):
wrapper = self.get_config_wrapper(config)
attributes = wrapper.get_my_attributes()
self.check_attributes_known(attributes)
data = {
'plugin_name': self.identifier+' plugin',
'version': self.PLUGIN_VERSION,
'priorities': wrapper.get_my_priorities(),
'attributes': attributes,
'merge_inputs': {
'gene_information_merger': [
{
"Gene": "KRAS",
"Gene_URL": "https://www.oncokb.org/gene/KRAS",
"Chromosome": "12p12.1",
"Summary": "KRAS, a GTPase which functions as an upstream regulator of the MAPK and PI3K pathways, is frequently mutated in various cancer types including pancreatic, colorectal and lung cancers."
},
{
"Gene": "PIK3CA",
"Gene_URL": "https://www.oncokb.org/gene/PIK3CA",
"Chromosome": "3q26.32",
"Summary": "PIK3CA, the catalytic subunit of PI3-kinase, is frequently mutated in a diverse range of cancers including breast, endometrial and cervical cancers."
}
]
},
'results': {},
}
question = 'What do you get if you multiply six by nine?'
self.workspace.write_string('question.txt', question)
return data

def render(self, data):
super().render(data) # validate against schema
return "<h3>TODO demo1 plugin output goes here</h3>"

def specify_params(self):
self.logger.debug("Specifying params for plugin demo1")
self.add_ini_required('question')
self.set_ini_default(core_constants.ATTRIBUTES, 'clinical')
self.set_ini_default('dummy_file', None)
self.set_priority_defaults(self.PRIORITY)
Empty file.
4 changes: 4 additions & 0 deletions src/lib/alternate_djerba/plugins/demo4/test/demo_1.ini
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
[core]

[demo1]
question = 'Who will win the Stanley Cup?'
23 changes: 23 additions & 0 deletions src/lib/alternate_djerba/plugins/demo4/test/demo_1.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
{
"attributes": ["clinical"],
"priorities": {"configure": 200, "extract": 200, "render": 200},
"merge_inputs": {
"gene_information_merger": [
{
"Chromosome": "12p12.1",
"Gene": "KRAS",
"Gene_URL": "https://www.oncokb.org/gene/KRAS",
"Summary": "KRAS, a GTPase which functions as an upstream regulator of the MAPK and PI3K pathways, is frequently mutated in various cancer types including pancreatic, colorectal and lung cancers."
},
{
"Chromosome": "3q26.32",
"Gene": "PIK3CA",
"Gene_URL": "https://www.oncokb.org/gene/PIK3CA",
"Summary": "PIK3CA, the catalytic subunit of PI3-kinase, is frequently mutated in a diverse range of cancers including breast, endometrial and cervical cancers."
}
]
},
"plugin_name": "demo1 plugin",
"results": {},
"version": "1.0.0"
}
Loading

0 comments on commit 3aa3435

Please sign in to comment.