All notable changes to this project will be documented in this file. This project adheres to Semantic Versioning.
About changelog here
- Added median target coverage to mip:s pydantic model
- Added step to mips metric deliverables model
- cg transfer pool: skip samples without pool name
- Show warnings when priorities differ on a case in an order
- RML priority on pool-case
- Database migration order
- Support Orderform 1508:25, replaces Orderform 1541:6
- New Balsamic delivery types
- fastq-qc + analysis + cram
- fastq-qc + analysis + cram + Scout
- Handle empty values for synopsis in excel orders
- Add proxy-protocol to dockerfile
- Synopsis from old and new samples are combined
- Added required ENV declarations in dockerfile
- pyyaml dependency required to run app in container
- Fluffy samplesheet add control field
- Balsamic delivery report exception text
- Mark a NIPT/Fluffy sample as control
- Accept synopsis both as list and as string in ordering json
- Compatability of fluffy and new headers in dragen samplesheets
- Case opencow and stillant to MIP validation cases
- New fields, subject_id, phenotype_groups in clinical samples json orders for Scout
- improve avatar search
- include dragen samplesheet in HK
- Action to clean up workflow artifact cache
- error in post demux process
- demultiplexed runs project checks
- Novaseq Dragen Demultiplexing
- Display number of samples in order ticket
- Raised sbatch memory parameter for demultiplexing
- Delivery type also works for one type
- Added support for multiple delivery flags for the command cg deliver analysis.
- Function for downloading external data from caesar
- Methods in mutant config to AM doc number
- Removed balsamic bam and bam index files from cg deliver analysis command
- MIP command to start after a given step
- Add new model for qc_metrics
- Add new model for mip_analysis
- Use new metrics deliverables format, which is already part of qc_metrics file
- Use new models when parsing mip analysis files
- Show more errors from parsing orderforms
- Update deployment instructions
- Updated bump2version action to v3
- Removed build container action (for time being) since we never use the containers
- New families can be created without the --panel flag
- Increased flowcells ondisk cap
- Fixed mean Q score added to cgstats by
cg demultiplex add <flowcell>being off by a factor of
- Integrate with bump2version-ci workflow
- Changed how qos for rsync is handled
- Cases removed from FOHM batch properly
- Remove KS validation cases from upload to GISAID/FOHM
- Added UMI validation cases to skip spring compress
- Send multiple emails properly
- Progress bar when uploading to FOHM
- Samples that lack data during delivery is reported
- Statina load use latest file
- Sftp upload before sending email to FOHM
- Tooltips for fohm upload functions
- Data delivery export to LIMS
- Fail graceful if Avatar url lookup fails
- Upload gisaid uses once again single fasta
- Upload gisaid uses pandas
- Upload gisaid uploads only qc pass files
- Added support for FOHM batch upload
- email functionality
- gisaid upload command: gisaid cli log file will be saved in housekeeper with tag gisaid-log
- gisaid upload command: the log file will be appended to if the upload is run again for the same case.
- gisaid upload command: Accession numbers will be parsed from the log file
- gisaid upload command: The completion file which already exists in housekeeper will be updated with accession numbers
- gisaid upload command: If files are missing or not all samples within the case were uploaded to gisiad, an email will be sent to logwatch.
- Add deepvariant for input file in upload to loqusDB
- Added rsync processes into trailblazer
- Added cleanup of rsync directories
- mip check if analysis is complete
- Avatar name lookup problem
- Avatars for cases
- Statina upload api call to raise error when server response is not ok
- cg will no longer run rsync on the gpu nodes
- Fixed statina load bug where missing headers prevented data from being read
- Add a command to upload both to Statina and ftp
- Adds a command to run the first command for all available cases
- Header in indexreport to versal letters
- Add more files to the mip-rna delivery
- Add cg clean fluffy_past_run_dirs command
- Remove dependency to archived pypi colorclass
- Use slurm qos class and constants instead of hard coding priority options
- Check for case name uniqueness before storing order to avoid half stored orders
- Rsync now run on slurm instead of the login node
- Added two more cases to BALSAMIC's validation
- Divided VALIDATION_CASES into their own workflow lists to find them easier
- Adds store and store-available commands that use Hermes
- Add pydantic models for mip config and mip sample info
- Add constants for getting housekeeper MIP analysis tags
- Remove old MIP-DNA store command using cg internally
- Remove old unused fixtures
- cli for upload of data to Statina
- NIPT upload to sftp server
- Clarified the code for automatic decompression
- Deliver, concatenate (if needed) and rsync in one command
- Support for showing demultiplexing in CiGRID
- bugfix
- gisaid -fixing bug accidentally inserted in the previous pr.
- gisaid get region and lab from sample udfs
- gisaid adjusting for new fasta consensus file in housekeeper
- gisaid removing failing tests
- Support for Orderform 1508:24 for Clinical Samples
- Add Fluffy analysis results -> ftp upload functionality for cust002
- Set memory allocation to 95G when demultiplexing on slurm
- Add batch-ref flag to command string for fluffy analyses
- Changed so that barcode report is written to the correct path
- Command
cg demultplex reportto generate barcode report for demuxed flowcells
- Automatically generate barcode report when post processing demultiplexed flowcell
- Copy sample sheet to demuxed flowcell dir
- Create
copycomplete.txtafter demux post-processing is done
- Fixed bug when checking if flowcell projects has been renamed
- Added support for SARS-CoV-2 Orderform 2184.5 with fields for GISAID upload
- Check if versions are larger than known version when determining reverse complement in demultiplexing
- Validate that flowcell name is correct when demultiplexing
- Fixed linking sample files from case bundle
- Check that logfile exists before doing demux post-processing
- Check if Unaligned dir exists before doing demux post-processing
- Fix bugs in create sample sheet all command
- Fix so that delivery will not break if fastq bundle is missing when delivering results with ticket id
- Fix bug in sqlalchemy models
- If a boolean value is passed to
cg set sample -kv <key> <value>a boolean is passed to the db
- Fix bug in mip and balsamic crontab
- Functionality to do demultiplexing post processing from CG
- Set status to analyze when resolving decompression
- Use only the first item from region and lab code values in mutant workflow.
- Fix tag to deliver correct mutant result files to KS inbox
- Block orders unintentionally reusing case names
- Set correct fluffy analysis finish path
- Fixed content of fluffy samplesheet according to customer specification
- By default fetch related flowcell information using
cg get sample <sample_id>
- User field for allowing order portal login
- Delivery/Invoicing/Primary Contacts are now relations from Customer to User in admin
- Propagate all samples to microsalt, even those without reads
- Display invoice contact on invoice
- Remove default option for mip-dna priority
- Change how samples are fetched for cgstats select command
- Bug when fetching info with
cg demultiplex select-command
- A column in customer-group admin view in to show customers in the group
- PDC backup destination server for 2500 flowcells
- Remove dependency
cgstatsfrom requirements.txt and move used functionality into CG
- Functionality for the cgstats api in CG
Select analyses to be uploaded to vogue based on analysis completed date (before or after a date, of between two dates) Add uploaded to vogue date to analysis table Only select potential analyses to upload that have not been uploaded
- Add support for balsamic 7.x.x
- Rework Balsamic server configurations
- Upload to scout now possible for all analysis types through cg upload scout
- Added more DNA and RNA positive control cases to VALIDATION_CASES
- Workflow linking also links undetermined files when
- Cases to analyze query checks if any samples in case are newer than latest analysis to start topups
- Microsalt config-case wont include samples that dont pass sequencing qc
- Changes that customer contact persons are referred to as emails instead of users. This removes the need to add each contact as a user in the system and it becomes easier to manage the user list
- Show customers in user admin view
- gisaid uppload support via cli
- gisaid API
- Add command
cg get analysisto view analysis information
- Fix subprocess wildcard issue in deliver rsync
- Added cg deliver rsync <ticket_id>
- bug in cg clean scout-finished-cases
- CLI command to deploy Loqusdb via shipping
- fixed bug where CgError was raised with wrong arguments in AnalysisAPI
- fixed bug that cgstats configs was saved as cg_stats in CGConfig
- Update sequenced at timestamp of sample whenever sample has been sequenced and flowcell transferred
- Bug when instantiating analysis api in upload
- Bug when fetching the api in mip workflow cli base
- use pydantic to control CLI context
- Trailblazer support for mutant/sars-cov-2
- Bugfix allowing orders of single samples from existing families
- Added new covid prep method to lims constants
- Adds support to use original sample lims id in downsampled cases (affects balsamic)
- Fix duplicate Housekeeper session in scout upload
- Status of mutant cases is set to running before running analysis, and revoked if start fails. This prevents users and cron jobs from starting the case which already started, given they check in statusDB first.
- Deliver sarscov2 cli
- Demultiplexing functionality to CG
- cg to rename mutant fastq files directly
- CLI methods for starting mutant
- Add RNA validation cases to blacklist, which defines cases to not compress fastq files for
- So that existing samples don't get added to lims again
- Possibility to associate more than one customer to each user
- Update orderform 2184 to latest version 4
- Fix skip MIP-DNA (for MAF) for tumours with wgs applications for fastq delivery
- Add support for fastq delivery
- functionality to upload synopsis, cohort and phenotype information to scout
- Mip start-available command
- Panel command to mip-rna analysis workflow
- Add genome build 38 as default for mip-rna when exporting gene panels from Scout
- Add genome build 37 as default for mip-dna when exporting gene panels from Scout
- Add genome build 37 as default when exporting gene panels from Scout
- Refactor start and start-available for mip
- Use pydantic models for handling lims samples
- Fix bug that that prevented wgs fastq samples from being genotyped
- Move synopsis and cohorts from sample to case in the database since that is how they are used
- Fixed bug in upload delivery report automation
- Fixed context in upload vogue
- Added missing pipeline option in database for SARS-CoV-2 on case in database
- Use
coverage_qc_reportinstead ofdelivery_reportwhen uploading balsamic cases to scout
- Balsamic crontab start-available auto-disables balsamic dry run
- Fix bug that pending path was not created
- Bug in automation of delivery report upload
- Bug when updating crunchy metadata files
- Bug preventing MicroSALT to start automatically
- SlurmAPI to handle all communication with slurm from cg
- Deletes unused files .gitlint, housekeeper, status, status_db
- Support for creating delivery reports for analyses that are not the latest ones
- DIAB and NBS-M to master list
- Alphabetical order of master list
- Use cgmodels for crunchy metadata files
- Change ending of lintjob from .py to .yml (accidentaly changed in previous pr)
- Support for SARS-CoV-2 Orderform 2184:3
- Remove pipfile and pipfile.lock since they are not used
- Removes linting gh actions job
- Add support for Microbial Orderform 1603:10
- Add support for Metagenome Orderform 1605:09
- Support for SARS-CoV-2 Orderform 2184:1
- Set concentration and concentration sample to str in json orderform sample since this is expected in frontend
- Fixed cases_to_store for microbial workflow.
- Fixed process call for all workflows to not create a new process object
- HousekeeperAPI to reuse db connection from context instead of creating new one for each call
- Refactored AnalysisAPI anf FastHandler classes into one class
- Fixed json orderform special cases
- Use pydantic models to validate orderforms
- delivery_report handling from BALSAMIC case import to Scout config export
- Genes are optional when exporting scout variants
- Refactored and removed code from cg.store.utils in backwards compatible way
- Updates issue template
- Fix automatic decompression to also work for mixed cases
- Removed old unused scripts and scripts directory
- Moved crunchy query from store into CompressAPI
- cg workflow mip-rna link command to take case as positional arg
- Refactor ticket handling from orders API
- Fix dry run flag when resolving decompression state
- fix container name when publishing branch builds on dockerhub
- changelog
- alembic functionality
- destination server PDC retrieval novaseq flowcells thalamus -> hasta
- flowcell status after PDC retrieval ondisk -> retrieved
- Support for storing cohorts, phenotype_terms and synopsis from order json
- Dockerhub build app container for release and pull requests
- Changed CI actions that run on pull request on push to only run on pull request
- Dockerfile declaration for running the cg app. Dockerfile should not be used for the cli toolkit
- New options for balsamic report deliver to propagate delivery report data to Balsamic
- Cases that decompression is started for will have the action set to "analyze"
- Allow price update files for application-version import to have empty prices
- Cases that decompression is started for will have the action set to "analyze"
- Cli command to deliver old balsamic analyses which were stored with old hk tags
- Adds command to start decompression for spring compressed files if needed
- Refactors MIP cli slightly to always expect a case id
- workflow start now called start-available
- Checking if flowcells are on disk moved to its own cli command
- Add prep-category 'cov' for applications
- Install package in gihub-jobs via pip==21.0.1
- Upgraded insecure cryptography dependency
- Chromograph image tags back on track
- Fix message in order ticket that says what type of project it is
- Fix so that variants gets uploaded to scout for balsamic samples
- Fix so that upload breaks if balsamic case is WGS
- New Fluffy workflow for preparing, starting and storing of analyses
- FLUFFY now have a validation schema and can be submitted in the Order Portal again
- Samples of pools are now marked not to be invoiced, only the pool is invoiced
- Allow existing trio-samples to be re-run as single samples
- Support for delivery type in the Order Portal
- cg upload vogue bioinfo-all uploads both BALSAMIC as well.
- Add functionality to upload balsamic analyses to scout
- cg workflow microsalt upload-analysis-vogue [case_id] to upload the latest analysis from specific case
- cg workflow microsalt upload-latest-analyses-vogue to upload all latest analyses what haven't been uploaded
- Skip deliver fastq files when delivering balsamic analysis
- clean_fastq command now also skips validation cases when cleaning fastq files
- Added customer name in order tickets
- Fix bug with name clash for created case when submitting RML-orders via Orderportal
- Add command 'delete case' to remove case from the database
- Add command 'delete cases' to remove multiple cases from the database
- Fix bug with microsalt deliverables path where it only returns the path only if it exists. This caused errors in some cases when submitting to Trailblazer
- Added MHT to gene panel master-list
- Added submission of microsalt cases for tracking in Trailblazer
- cg workflow mip-dna --panel-bed and --dry-run options can be set when executing full workflow
- Changed logic for how case links are retrieved in order to support linking of very old cases
- Analysis not submitted to Trailblazer if executing MIP workflow in dry-run
- Remove option to specify delivery path when delivering data
- Improved usage of
cg deliver analysiscommand
- cg workflow mip-rna link command
- Better help text for microsalt cli commands
- deliver filtered cnvkit file for balsamic
- cg workflow mip-rna config-case command
- Updates balsamic deliver tags to align with the ones specified in hermes
- Customer in the ticket created for an order from the Orderportal
- New commands for running cron jobs
- Changed cli commands for starting and storing microsalt workflows
- Full Microsalt can now be executed through cg interface
- HermesApi added to context when running cg clean
- Using HermesApi to save Balsamic deliverables in Housekeeper
- Added support for Balsamic-orderforms as json-file
- Lims ID is sent to Balsamic during case config
- Fixed exporting reduced mt_bam to scout config
- Update Scout config output for images from chromograph
- This PR fixes the problem handling wells in json orders without ":" as separator, e.g. A1 instead of A:1
- Use hermes for generating balsamic deliverables
- The problem handling mip-json without specified data_analysis
- Validation models for excel files in
cg/store/api/import_func.py
- Removes dependency on deprecated excel-parser
xlrd
- cg deploy hermes
- cg deploy fluffy
- Stop on bad values for analysis (pipeline) when adding family through CLI
- Save in Family and Analysis Admin views
- Added support for RML-orderforms as json-file
- Lock dependency for xlrd so that we can parse modern excel files
- cg set family [CASEID] now has the option --data-delivery
- Bug when building tag definition for balsamic-analysis delivery
- Use
--jsonwhen exporting causative variants from scout
- Use correct pipeline name when cleaning mip analysis dirs
- Adds new mip-dna tags to hk
- Adds new delivery type to balsamic-analysis
- column percent_reads_guaranteed to application table
- Bug in the interactions with Scout when cleaning Scout cases
- Bug in the interaction with scout in command export_causatives
- Adds adding samplesheet to HK flowcell bundle to cg transfer flowcell
- Mutacc looks for 'id' instead of '_id' in case export
- Convert 'other' to '0' for sex in case export
- Show sample priorities in created ticket
- Split generating config into its own command
- Delete old load config when running
cg upload scout --re-upload
- Functionality to deploy scout with cg
- Bug that madeline output files where not uploaded to scout
- Bug when exporting panels with
cg workflow mip-dna panel
- Bug that display_name was used instead of sample_name
- Change compression query to be both satisfactory syntax for flake and to be working on our version of sql server
- Use Pydantic models to validate Scout input/output
- Decouples scout from CG
- Fixed documentation on Trailblazers purpose
- Fixed setting of priority in statusDB and LIMS for samples
- Fixed setting of apptag in LIMS for samples
- Fixed a bug in naming of "default_gene_panels" in Scout load config
- Changed logic for which cases are to be compressed. Now compression will be run on all cases older then 60 days provided their fastq files have not been decompressed less than 21 days prior
- Use CLI to upload to Scout
- Accreditation logotype only shown on new delivery reports for accredited analyses
- Changed the way cg cleans cases. Now it only uses StatusDB and family status
- Added one-time script to iterate over mip directories, and set cleaned_at timestamp on very old cases that were already cleaned
- 'cg upload auto --pipeline' to accept 'mip-dna' as pipeline
- Trailblazer integration fixed
- Case database entities (Family) can only have specific values for data_analysis
- Analysis database entities can only have specific values for pipeline
- Enum used for pipeline as arguments
- Microbial config-case now correctly imports reference from customer provided reference
- Added case intentmayfly to list of cases to except from SPRING compression
- Updated PR template to include implementation plan
- Deliver analysis based on case-id or ticket
- Deliver commands merged into new command
cg deliver analysis
- fixed failing
cg microsalt store completedcronjob
- Fixed path where microsalt deliverables files are located
- Wrap more cg workflow mip-dna store code in try-except in order to not cause future production blocks
- Fix bug in compress clean command
- New command: cg store ticket <ticket_id>
- New command: cg store flowcell <flowcell_id>
- New command: cg store case <case_id>
- New command: cg store sample <sample_id>
- Old command: cg store fastq <case_id>
- Removed unused options form cg workflow balsamic base command
- New command: cg decompress ticket <ticket_id>
- New command: cg decompress flowcell <flowcell_id>
- New command: cg decompress case <case_id>
- New command: cg decompress sample <sample_id>
- Old command: cg compress decompress spring <case_id>
- Changed condition for which cases should be stored in CG. This fixes a bug where cg would try to store cases which already have been stored due to mismatch in timestamp stored in Trailblazer and Housekeeper
- Only fastq files older than three weeks will be compressed
- Added new value to lims constants
- Moved lims constants to a constants directory
- Workflow mip-dna store no longer needs analysisrunstatus to be completed to attempt storing bundle
- Fixed bug where parse_mip_config() only returned values for primary analysis, breaking Upload Delivery Report
- Fix bug where cg workflow mip store still relied on Trailblazer to find case_config.yaml (Where it can no longer be found)
- Fix bug where microsalt cli lost its store command in merge conflict
- New REST-based TrailblazerAPI
- Trailblazer support for Balsamic
- Naming convention for API harmonized
- New query to get all cases in ticket
- Refactors constants file
- Fixed bug where CalledProcessError class could not be represented as string, and broke workflows.
- Rephrased query used for compression. The query output is unchanged
- Fixed typo in query name
- Remove data_analysis from sample since it is deprecated
- Move data_analysis from sample level to case level to enable samples to be analysed differently in different cases
- Store all available completed microbial analyses in HK
- Balsamic always skips mutect when application is WES
- SPRING compression is set to run on oldest families first
- Format print statements
- Storing chromograph, upd and rhocall files in housekeeper
- Repaired automation query for storing Balsamic cases in Housekeeper
- Functionality to deploy
genotypewith CG on hasta
- Stored completed not parsing through all completed entries
- Deployment command
- Functionality to deploy
shippingwith CG
- Functionality to change multiple families in one go, e.g. cg set families --sample-identifier ticket_number 123456 --priority research
- Set flowcell status to
ondiskwhen retrieving archived flowcell from PDC has finished.
- Store microsalt analyses in Housekeeper with a provided deliverables file
- Possibility to give case-id as argument when setting values on multiple samples by the CLI
- Updated changelog with correct release version
- Microbial order table
- Microbial order model
- Microbial sample table
- Microbial sample model
- Microbial Flowcell-sample table
- Microbial Flowcell-sample model
Show customer name instead of id in invoices view. Made customer name searchable in invoices view.
Made unidirectional links to Customer (instead of bi) to speed up customer view Made unidirectional links to ApplicationVersion (instead of bi) to speed up view
- Exclude analysis older than hasta in production (2017-09-27) from delivery report generation
- Added new cases to skip during compression
- Only talk to genotype via subprocess and CLI
- Added cases for all microbial samples
- Add a case when a new microbial order is received
- Moved all microbial samples into samples table and the depending logic
- Removed store-housekeeper one-time script that was used to store balsamic results in Housekeeper
- Moved queries in TrailblazerAPI to Trailblazer
- Added support for MIP9
- Adds support for MIP 9.0 files and tags in HK:
- telomerecat
- cyrius star-caller
- smncopynumber caller on case level
- tiddit_coverage bigwig track
- Adds smncopynumbercaller file to scout load config
- Removed TB mip start source code from cg
- Return when using mip-dry-run option so that the dry-run is not added to TB
- Now ignores errors while cleaning old Balsamic data with cg clean
- Providing a name of panel bed file in MIP cli now overrides getting capture kit through LimsAPI during case config
- Providing panel bed path or capture kit shortname in BALSAMIC cli now overrides getting capture kit through LimsAPI during case config
- Fix Balsamic automation functions to exit with 1 if any of the errors are raised while looping through cases
- Fixes bug where scout_api was sent into the compression_api in cli/store. The compression_api does not have scout_as an argument.
- Added @Mropat as codeowner
- Fixes bug where upload_started_at and uploaded_at timestamps were not being updated in StatusDB upon successful Scout upload. This bug was happening because several instances of Store were instantiated in the same context
- Fixes but where linking MIP trio samples only linked the first sample instead of the whole family (Introduced in recent PR)
- Fixes bug where linking by SampleID was not linking the entire family (Old)
- Fixes bug where linking by SampleID would not be able to generate correct linking path (Old)
- Microbial Samples are now treated as ordinary samples in a case
- Upload delivery report should now only happen for mip analyses
- Re-use the same API within all upload context
- Handle unspecified exceptions in order to keep the cron running when unexpected exception occurs for one case
- When linking file without data analysis set, warn about it and link file correctly
- Fixed bug where AnalysisAPI in cg upload auto was not updated to recent class changes
- Class ConfigHandler moved from Trailblazer codebase into CG codebase
- FastqHandler methods moved from Trailblazer to AnalysisAPI
- Merged MipAPI and AnalysisAPI for mip_rna and mip_dna into one meta-api class
- Adjusted tests to support the new architecture
- Removed (unused/deprecated)run method which was used to execute MIP through Trailblazer
- MIP workflow once again performs check to skip evaluation
- MIP workflow once again updates StatusDB about the case status
- Merged methods cases_to_mip_analyze and cases_to_balsamic_analyze, now called cases_to_analyze for any pipeline.
- Made method cases_to_deliver pipeline aware
- Made method cases_to_store pipeline aware
- Made method cases_to_clean pipeline aware
- Added option to apply read count threshold for cases_to_analyze based on panel in ClinicalDB
- Updated MIP and BALSAMIC workflows to utilize the new methods
- Added tests for new methods in balsamic workflow
- Removed class FastqAPI. FastqAPI was only used by BALSAMIC, and contained one method. The method is now moved to FastqHandler class.
- Create crunchy pending path outside batch script
- support current orderform RML-1604:9 again
- Use long description from setup.py on PyPI
- Use another parameter in build and publish
- Syntax in github action build and publish workflow
- Build and publish on pypi with github actions
- Create a meta-API (BalsamicAnalysisAPI) to handle communication between balsamic and other cg applications. The API will handle the following:
- Calling BalsamicAPI to execute balsamic commands
- Query Lims and StatusDB to decide what arguments to pass to Balsamic
- Read (new version of) deliverables report generated by Balsamic and store bundle in Housekeeper + StatusDB
- More info in #687
- Reduce number of options that can/should be passed to run the workflow. Most of the logic for determining the options will be handled by BalsamicAnalysisAPI.
- Every command now requires sample family name as argument.
- No longer support using sample_id to link files for sake of consistency.
- Removes all interactions with the beacon software
- Fixed so that empty gzipped files are considered empty considering metadata
- Adds a CHANGELOG.md