Report updates #64

fasterius · 2024-01-23T08:27:00Z

This PR contains substantial changes and fixes to the downstream analysis reports. Changes include:

Changing the documentation from scanpy to custom explanations and contexts
Harmonisation of input/output file/process/parameter names
Renaming of variables/parameters to better describe what they are used for and to adhere to PEP8
Merge the pre/post-processing subworkflows into one downstream subworkflow
Moving the normalization from the QC report to the clustering report
Store raw data in the AnnData object so that a user may re-do analyses from scratch if so desired
Simplified input file staging in reports
Addition of filtering of mitochondrial, ribosomal and haemoglobin content
Additional figures
Removal of redundant figures
Optimisation of figure sizes to fit the report format, where possible
Minor formatting and language changes throughout
Updated documentation

Hopefully this PR makes the reports and their related pipeline content are a bit more fully-featured and better documented.

PR checklist

This comment contains a description of changes (with reason).
If you've fixed a bug or added code that should be tested, add tests!
Make sure your code lints (nf-core lint).
Ensure the test suite passes (nextflow run . -profile test,docker --outdir <OUTDIR>).
Check for unexpected warnings in debug mode (nextflow run . -profile debug,test,docker --outdir <OUTDIR>).
Usage Documentation in docs/usage.md is updated.
Output Documentation in docs/output.md is updated.
CHANGELOG.md is updated.

Remove the redundant total counts/genes by counts histogram plots in the QC report, as violin plots show the same thing. Also replace the post-filtering plots with additional violin plots for consistency.

Change `cell` variables and params to `spot`, as that is actually what is being done in the spatial data analysis. Previously the usage was mixed in the reports and parameters, and should now be unified to use `spot`.

github-actions · 2024-01-23T08:28:16Z

`nf-core lint` overall result: Passed ✅ ⚠️

Posted for pipeline commit ecffa90

+| ✅ 158 tests passed       |+
#| ❔   3 tests were ignored |#
!| ❗   5 tests had warnings |!

❗ Test warnings:

readme - README contains the placeholder zenodo.XXXXXXX. This should be replaced with the zenodo doi (after the first release).
pipeline_todos - TODO string in README.md: Add citation for pipeline after first release. Uncomment lines below and update Zenodo doi and badge at the top of this file.
pipeline_todos - TODO string in WorkflowMain.groovy: Add Zenodo DOI for pipeline after first release
pipeline_todos - TODO string in methods_description_template.yml: #Update the HTML below to your preferred methods description, e.g. add publication citation for this pipeline
pipeline_todos - TODO string in awsfulltest.yml: You can customise AWS full pipeline tests as required

❔ Tests ignored:

files_exist - File is ignored: conf/igenomes.config
files_unchanged - File ignored due to lint config: .gitattributes
actions_ci - actions_ci

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .nf-core.yml
files_exist - File found: .editorconfig
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: CITATIONS.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/config.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: assets/nf-core-spatialtranscriptomics_logo_light.png
files_exist - File found: conf/modules.config
files_exist - File found: conf/test.config
files_exist - File found: conf/test_full.config
files_exist - File found: docs/images/nf-core-spatialtranscriptomics_logo_light.png
files_exist - File found: docs/images/nf-core-spatialtranscriptomics_logo_dark.png
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: lib/nfcore_external_java_deps.jar
files_exist - File found: lib/NfcoreTemplate.groovy
files_exist - File found: lib/Utils.groovy
files_exist - File found: lib/WorkflowMain.groovy
files_exist - File found: main.nf
files_exist - File found: assets/multiqc_config.yml
files_exist - File found: conf/base.config
files_exist - File found: .github/workflows/awstest.yml
files_exist - File found: .github/workflows/awsfulltest.yml
files_exist - File found: lib/WorkflowSpatialtranscriptomics.groovy
files_exist - File found: modules.json
files_exist - File found: pyproject.toml
files_exist - File not found check: Singularity
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: pipeline_template.yml
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: docs/images/nf-core-spatialtranscriptomics_logo.png
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: .travis.yml
nextflow_config - Config variable found: manifest.name
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: manifest.homePage
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: params.validationShowHiddenParams
nextflow_config - Config variable found: params.validationSchemaIgnoreParams
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config manifest.name began with nf-core/
nextflow_config - Config variable manifest.homePage began with https://github.com/nf-core/
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config manifest.version ends in dev: 1.0dev
nextflow_config - Config params.custom_config_version is set to master
nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Lines for loading custom profiles found
nextflow_config - nextflow.config contains configuration profile test
files_unchanged - .prettierrc.yml matches the template
files_unchanged - CODE_OF_CONDUCT.md matches the template
files_unchanged - LICENSE matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/CONTRIBUTING.md matches the template
files_unchanged - .github/ISSUE_TEMPLATE/bug_report.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/config.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - .github/PULL_REQUEST_TEMPLATE.md matches the template
files_unchanged - .github/workflows/branch.yml matches the template
files_unchanged - .github/workflows/linting_comment.yml matches the template
files_unchanged - .github/workflows/linting.yml matches the template
files_unchanged - assets/email_template.html matches the template
files_unchanged - assets/email_template.txt matches the template
files_unchanged - assets/sendmail_template.txt matches the template
files_unchanged - assets/nf-core-spatialtranscriptomics_logo_light.png matches the template
files_unchanged - docs/images/nf-core-spatialtranscriptomics_logo_light.png matches the template
files_unchanged - docs/images/nf-core-spatialtranscriptomics_logo_dark.png matches the template
files_unchanged - docs/README.md matches the template
files_unchanged - lib/nfcore_external_java_deps.jar matches the template
files_unchanged - lib/NfcoreTemplate.groovy matches the template
files_unchanged - .gitignore matches the template
files_unchanged - .prettierignore matches the template
files_unchanged - pyproject.toml matches the template
actions_awstest - '.github/workflows/awstest.yml' is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml does not use -profile test
readme - README Nextflow minimum version badge matched config. Badge: 23.04.0, Config: 23.04.0
pipeline_name_conventions - Name adheres to nf-core convention
template_strings - Did not find any Jinja template strings (130 files)
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
schema_lint - Input mimetype lint passed: 'text/csv'
schema_params - Schema matched params returned from nextflow config
system_exit - No System.exit calls found
actions_schema_validation - Workflow validation passed: linting_comment.yml
actions_schema_validation - Workflow validation passed: fix-linting.yml
actions_schema_validation - Workflow validation passed: awsfulltest.yml
actions_schema_validation - Workflow validation passed: release-announcements.yml
actions_schema_validation - Workflow validation passed: awstest.yml
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: clean-up.yml
actions_schema_validation - Workflow validation passed: branch.yml
actions_schema_validation - Workflow validation passed: ci.yml
merge_markers - No merge markers found in pipeline files
modules_json - Only installed modules found in modules.json
multiqc_config - 'assets/multiqc_config.yml' contains report_section_order
multiqc_config - 'assets/multiqc_config.yml' contains export_plots
multiqc_config - 'assets/multiqc_config.yml' contains report_comment
multiqc_config - 'assets/multiqc_config.yml' follows the ordering scheme of the minimally required plugins.
multiqc_config - 'assets/multiqc_config.yml' contains a matching 'report_comment'.
multiqc_config - 'assets/multiqc_config.yml' contains 'export_plots: true'.
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'

Run details

nf-core/tools version 2.11.1
Run at 2024-01-26 08:35:16

cavenel

Some minor comments and then all good for me!

bin/st_quality_controls.qmd

nextflow_schema.json

cavenel

All good!

fasterius added 30 commits January 16, 2024 15:58

Add nf-core Quarto template as asset

c190b7a

Use nf-core Quarto template in reports

812f462

Fix broken QC report link

662e68c

Echo QC report import code chunk

41e601d

Formatting and clean-up of QC report

3ecbe91

Remove unused st_data_plain QC report output

7b4c9d0

Do not check SCSS files with Prettier

7842683

Fix QC report figure sizes

1924ce7

Add parameter for number of highly variable genes

e256687

Fix QC report header capitalisation

2116395

Formatting and clean-up of clustering report

7df9db4

Formatting and clean-up of DE report

adbd9ca

Update tests

29d8bec

Update untar module

d5bd0e0

Also check ribosomal content in QC report

e7934f0

Add A vs B QC metrics plots

eb9eaf9

Fix QC report header levels

f99bfc6

Add QC sub-headers and more explanations

2af8d6c

Print all warnings in QC report

e02baee

Remove redundant QC plots

0e4f0c6

Remove the redundant total counts/genes by counts histogram plots in the QC report, as violin plots show the same thing. Also replace the post-filtering plots with additional violin plots for consistency.

Prettify filtering output

ae566ca

Change cell variable/params to spot

7b4b43c

Change `cell` variables and params to `spot`, as that is actually what is being done in the spatial data analysis. Previously the usage was mixed in the reports and parameters, and should now be unified to use `spot`.

Move top expressed genes section to pre-filtering

c2bfaad

Move normalisation and HVG to cluster report

bcfd236

Prettify and format clustering figures

8050b56

Format spatial visualisations in cluster report

086b836

Rename files and processes with new report content

2dd37b4

Add nf-core TODO-string

fb92ccb

Increase spatial plot point size

b9c8ed9

Do not print number of FASTQ files found for SR

c12b455

fasterius added 5 commits January 22, 2024 16:33

Update schema variable doc

909ebdf

Remove redundant analysis usage doc section

ffd10c5

Update FastQC module

48ca093

Change Quarto template files into a value channel

5457f29

Update changelog

4416fcb

fasterius added 2 commits January 23, 2024 11:46

Update tests

4af9545

Prettier fixes

d3cd6c2

fasterius marked this pull request as ready for review January 23, 2024 11:27

fasterius requested a review from cavenel January 23, 2024 11:27

fasterius added 13 commits January 24, 2024 08:21

Properly print all AnnData objects in reports

d94db35

Store the raw data in the AnnData object

04cee16

Fix header levels

c719bea

Clarify filtering fallback if clause

8a5607d

Move quality controls code chunk

87cbb97

Add mitochondrial content filtering

cf03cab

Update parameter names for clarity

965b643

Add ribosomal filtering

482cab2

Clarify filtering fallback callout

007e17e

Fix counting of spots during filtering

ef0e69e

Add haemoglobin filtering

7faa00a

Rename header; minor formatting

b65e1e5

Update tests

0d8364b

cavenel reviewed Jan 25, 2024

View reviewed changes

bin/st_quality_controls.qmd Outdated Show resolved Hide resolved

nextflow_schema.json Show resolved Hide resolved

fasterius added 2 commits January 26, 2024 09:04

Fix grammar

a5691c6

Change QC threshold type from integer to number

ecffa90

fasterius force-pushed the reports branch from 10e4ffe to ecffa90 Compare January 26, 2024 08:25

cavenel approved these changes Jan 26, 2024

View reviewed changes

fasterius merged commit 8c69d02 into dev Jan 26, 2024
8 checks passed

fasterius deleted the reports branch January 26, 2024 08:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Report updates #64

Report updates #64

fasterius commented Jan 23, 2024 •

edited

Loading

github-actions bot commented Jan 23, 2024 •

edited

Loading

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

cavenel left a comment

cavenel left a comment

Report updates #64

Report updates #64

Conversation

fasterius commented Jan 23, 2024 • edited Loading

PR checklist

github-actions bot commented Jan 23, 2024 • edited Loading

nf-core lint overall result: Passed ✅ ⚠️

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

cavenel left a comment

Choose a reason for hiding this comment

cavenel left a comment

Choose a reason for hiding this comment

fasterius commented Jan 23, 2024 •

edited

Loading

github-actions bot commented Jan 23, 2024 •

edited

Loading

`nf-core lint` overall result: Passed ✅ ⚠️