Validate "reference_radiation_treatment_id" against previously submitted or currently submitting "submitter_treatment_id"

[hknahal]

Description:

This is related to icgc-argo/argo-dictionary#375
Cross-validation required between Radiation and Treatment tables, specifically the submitter_donor_id and reference_radiation_treatment_id in the Radiation table, and the submitter_donor_id, submitter_treatment_id and treatment_type fields in the Treatment table.

Background:

Two new optional fields will be added to the "Radiation" schema:
radiation_boost
reference_radiation_treatment_id

The reference_radiation_treatment_id is only required if radiation_boost = Yes (this is taken care by validation script). The field tells us which treatment the radiation boost was linked to.

Requirements for reference_radiation_treatment_id

• The reference_radiation_treatment_id requires a submitter_treatment_id that has either been submitted previously in the Radiation table or is currently being submitted in the Radiation table.
• The submitted submitter_treatment_id must belong to the same submitter_donor_id
• The submitted submitter_treatment_id must be associated with radiation treatment (ie. treatment_type = Radiation therapy).

Example of validation that will pass:

Example 1: If reference_radiation_treatment_id matches a previous or existing submitter_treatment_id and it is associated with treatment_type = Radiation therapy, then validation will pass.

Test files
sample_registration.txt
donor.txt
primary_diagnosis.txt
chemotherapy.txt
radiation_pass.txt
treatment_pass.txt

The submitter_treatment_id "TR-4" submitted in the reference_radiation_treatment_id exists in the Treatment table.

Radiation table:

Treatment table:

Examples of validations that will fail:

Example 1: If reference_radiation_treatment_id does not match a previous or existing submitter_treatment_id, then validation will fail with an error message ("The submitter_treatment_id submitted in the "reference_radiation_treatment_id" field does not exist.")

Test files:
sample_registration.txt
donor.txt
primary_diagnosis.txt
chemotherapy.txt
radiation_fail1.txt
treatment_fail1.txt

The submitter_treatment_id "TR-9" submitted in the reference_radiation_treatment_id does not exist in the Treatment table.

Radiation table:

Treatment table:

Example 2: If reference_radiation_treatment_id matches a previous or existing submitter_treatment_id but it is associated with a different treatment type other than Radiation therapy, then validation will fail with an error message ("The submitter_treatment_id submitted in the "reference_radiation_treatment_id" field is not for radiation treatment.")

Test files:
sample_registration.txt
donor.txt
primary_diagnosis.txt
chemotherapy.txt
radiation_fail2.txt
treatment_fail2.txt

The submitter_treatment_id "TR-1" submitted in the reference_radiation_treatment_id is not associated with radiation treatment in the Treatment table (it is for Chemotherapy)

Radiation table:

Treatment table:

Example 3: The reference_radiation_treatment_id exists in the Treatment table, but it belongs to a different submitter_donor_id. (The reference_radiation_treatment_id "TR-5" belongs to submitter_donor_id "DN425", not "DN290"). This should fail validation.

Test files:
sample_registration.txt
donor.txt
primary_diagnosis.txt
chemotherapy.txt
radiation_fail3.txt
treatment_fail3.txt

Radiation table:

Treatment table:

[puneet-oicr]

Dev needed from the clinical submission side. @Buwujiu @b-f-chan - who can we assign this to?
Release pending till we get a validation in.

[Buwujiu]

@hknahal will make a new dictionary in qa and test cases.

[demariadaniel]

Testing, but unit tests are working properly:
#928

[demariadaniel]

Bug fix from Dev testing:
#1008

[hknahal]

Test cases that fail

1. Tested "Example 2" from Examples of validations that will fail from above on QA using TEST-GR

Validation passed but expected validation to fail because the submitter_treatment_id submitted in the reference_radiation_treatment_id is not for radiation treatment. Instead it is for chemotherapy treatment.

Expected error message: The submitter_treatment_id "TR-1" submitted in the reference_radiation_treatment_id field is not for radiation treatment in the Treatment table

2. Tested "Example 3" from Examples of validations that will fail from above on QA using TEST-GR

Validation passed but expected validation to fail because the submitter_treatment_id submitted in the reference_radiation_treatment_id field does not belong to the same submitter_donor_id

Expected error message: The reference_radiation_treatment_id "TR-5" belongs to submitter_donor_id DN425, not DN290

[Buwujiu]

This feature will be hidden in staging and prod until the issues are fixed.

[demariadaniel]

Tested and working on localhost:

[demariadaniel]

[hknahal]

This feature will be hidden in staging and prod until the issues are fixed.

Tested in Staging using TEST-CA by submitting radiation file with the reference_radiation_treatment_id and radiation_boost fields. Validation failed as expected. Confirmed feature is hidden in Staging