Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Validation fixes #115

Merged
merged 23 commits into from
Jul 31, 2024
Merged

Validation fixes #115

merged 23 commits into from
Jul 31, 2024

Conversation

johngrimes
Copy link

@johngrimes johngrimes commented Jun 28, 2024 •
edited
Loading

Resolves #95, #96, #97, #98, #99, #100, #101, #102, #103, #104, #105, #106, #107, #108, #109, #110, #111, #112, #113, #114, #92, #91.

Depends upon the corresponding PR kind-lab/mimic-profiles#26.

Summary of Changes

This pull request includes several updates and fixes to the mimic-fhir repository. Below is a detailed list of the changes made:

  1. Switching mimic-profiles to the aehrc repo (commit: cc90af7)

    • Updated terminology resources.

  2. Added Python script (commit: 6a6c5f9)

    • Generates code systems and value set resources from the mimic database (fhir_trm schema).

  3. Removed unnecessary code systems and value sets (commit: 873ad6f)

    • Cleaned up terminology description tables.

  4. Fixed source code system issues (commit: e4303e4)

    • Added emar_detail.product_unit and emar_detail.dose_given_unit as sources for the cs_unit code system.

  5. Aligned coding display names (commit: aec446c)

    • Fixed display names for cs-diagnosis-icd9, cs-diagnosis-icd10, and cs-medication-etc code systems.
    • Addresses issues: #113, #114

  6. Corrected whitespace in pharmacy.frequency (commit: b3149dd)

    • Ensured compliance with coding whitespace rules.
    • Addresses issue: #112

  7. Updated dosageInstruction.timing in fhir_medication_dispense.sql (commit: b03ef06)

    • Ensured inclusion of code or repeat elements.
    • Addresses issue: #102

  8. Handled blank medication in medication_dispense generation (commit: ce0739a)

    • Treated blank medications as NULL values.
    • Addresses issue: #111

  9. Produced NULL for blank lab_VALUE in fhir_observation_labevents.sql (commit: 0ff87cb)

    • Addresses issue: #110

  10. Handled blank dose_given_unit in fhir_medication_administration.sql (commit: 5f6c995)

    • Addresses issue: #109

  11. Corrected display name for LOINC code: 2708-6 (commit: 7f51b40)

    • Updated fhir_observation_vitalsigns.sql.
    • Addresses issue: #108

  12. Changed extraction of quantity from pharmacy.fill_quantity (commit: 18fb04a)

    • Improved extraction logic for numerical values and units.
    • Addresses issues: #106, #107

  13. Handled NULL values in BP vital signs observations (commit: 6e9306a)

  14. Changed JSON export format to 'csv' (commit: ba8387d)

    • Fixed backslash escaping issues.
    • Addresses issue: #104

  15. Added null value checks (commit: cb72cfe)

    • Prevented creation of empty objects in fhir_medication_administration.sql and fhir_specimen.sql.
    • Addresses issues: #99, #100

  16. Removed array wrapping for medicationCodeableConcept (commit: 0f1a940)

    • Updated fhir_medication_request.sql.
    • Addresses issue: #98

  17. Renamed mimic-lab-priority to lab-priority (commit: 81e8b3e)

    • Updated extension URL in fhir_observation_labevents.sql.
    • Addresses issue: #97

  18. Fixed unit codes for respiratory and heart rate (commit: 12211b4)

    • Updated fhir_observation_vitalsigns.sql.
    • Addresses issue: #96

  19. Removed leading whitespace from display names (commit: b2beb9e)

    • Updated map_race_omb.sql.
    • Addresses issue: #95

piotrszul added 23 commits June 27, 2024 17:11
@piotrszul
Syntactic changes to simplify recognition of implicit unpivots
8f7282f
@piotrszul
Updating setup details for python package.
f3ba2da
@piotrszul
Removing leading ' ' from display name ' Black ...' in map_race_omb.s…
b2beb9e 
…ql to fix:

kind-lab#95

The Coding provided (urn:oid:2.16.840.1.113883.6.238#2054-5) was not found in the value set 'OMB Race Categories' (http://hl7.org/fhir/us/core/ValueSet/omb-race-category|4.0.0), and a code is required from this value set. (error message = Wrong whitespace in Display Name ' Black or African American' for urn:oid:2.16.840.1.113883.6.238#2054-5. Valid display is 'Black or African American' (for the language(s) '--'))
@piotrszul
Fixing the codes of unit for respiratory and heart rate in fhir_obser…
12211b4 
…vation_vitalsigns.sql, to [/min] to fix:

kind-lab#96

- Value is '{Beats}/min' but must be '/min'
- Value is '{Breaths}/min' but must be '/min'
@piotrszul
Renaming 'mimic-lab-priority' to 'lab-priority' (as per mimic profile…
81e8b3e 
…) in the extension url in fhir_observation_labevents.sql to fix:

kind-lab#97

The extension http://mimic.mit.edu/fhir/mimic/StructureDefinition/mimic-lab-priority could not be found so is not allowed here
@piotrszul
Removing array wrapping for 'medicationCodeableConcept' in fhir_medic…
0f1a940 
…ation_request.sql to fix:

kind-lab#98

The property medicationCodeableConcept must be an Object, not an Array (at MedicationRequest.medication[x])
@piotrszul
Adding check for null values to avoid creation of empty object in fhi…
cb72cfe 
…r_medication_administration.sql (request) and fhir_specimen.sql (collection) to fix:

kind-lab#99
kind-lab#100

Object must have some content
(e.g: status": "completed", "request": {}, "subject": {"reference": )
@piotrszul
Only generate the hasMember array if there is at last one non-NULL re…
d6fe23c 
…ference (mi_ORGANISM) present in fhir_observation_micro_test.sql to fix:

kind-lab#101

Object must have some content (e.g: "hasMember": [{}])

Note: this may not be not the ultimate fix - it's still possible for empty object to be produced if there is a mix of NULL and non-NULL values in the group.
@piotrszul
Adding http://www.w3.org/1999/xhtml namespace to the <div> element in…
75631e0 
… medication_mix.sql to fix:

kind-lab#103

Wrong namespace on the XHTML ('null', should be 'http://www.w3.org/1999/xhtml')
@piotrszul
Changing the json export format to 'csv' to fix the backslash escapin…
ba8387d 
…g issue:

kind-lab#104

invalid	Error parsing JSON: Error parsing JSON source: JSON syntax error - found String expecting Colon at Line 1 (path=[//medicationCodeableConceptConcentrated])

eg: "medicationCodeableConcept": {"text": "Humulin R U-500 \\"Concentrated\\"", "coding":
@piotrszul
Adding correct null handling for bp vital signs observations in fhir_…
6e9306a 
…observation_vitalsigns.sql to conform with http://hl7.org/fhir/StructureDefinition/bp|4.0. Always producing the required bp components, with not values if not available.

Fixing:

kind-lab#105

Observation.component: minimum required = 2, but only found 0 (from http://hl7.org/fhir/StructureDefinition/bp|4.0.1)

Slice 'Observation.component:DiastolicBP': a matching slice is required, but not found (from http://hl7.org/fhir/StructureDefinition/bp|4.0.1). Note that other slices are allowed in addition to this required slice

Slice 'Observation.component:SystolicBP': a matching slice is required, but not found (from http://hl7.org/fhir/StructureDefinition/bp|4.0.1). Note that other slices are allowed in addition to this required slice
@piotrszul
Changing extraction of quantity from pharmacy.fill_quantity. When pos…
18fb04a 
…sible extract the numerical value and the unit. Otherwise, do not provide the quantity.

Fixing:

kind-lab#106
kind-lab#107

- Error parsing JSON: the primitive value must be a number	(e.g: quantity": {"value": "3"})
- The value '60 ml' is not a valid decimal (e.g:   {"value": "60ml"})
- The value 'PO' is not a valid decimal	 ({"value": "PO"})
@piotrszul
Correcting display name for loinc code: 2708-6 in fhir_observation_vi…
7f51b40 
…talsigns.sql to fix:

kind-lab#108

Wrong Display Name 'Oxgyen saturation in Arterial blood' for http://loinc.org#2708-6. Valid display is 'Oxygen saturation in Arterial blood' (en-US) (for the language(s) '--')
@piotrszul
Producing NULL for blank dose_given_unit in fhir_medication_administr…
5f6c995 
…ation.sql to fix:

kind-lab#109

Invalid attribute value "": Attribute value must not be empty ("")
(e.g: {"dose": {"code": "", "unit": "", "value": 1,)
@piotrszul
Producing NULL for blank lab_VALUE in fhir_observation_labevents.sql …
0ff87cb 
…to fix:

kind-lab#110

Invalid attribute value "": Attribute value must not be empty ("")
(e.g: [{"value": "51307513", "system": "http://mimic.mit.edu/fhir/mimic/identifier/observation-labevents"}], "valueString": "",)
@piotrszul
Skipping generation of 'medication_dispense' for pharmacy records wit…
ce0739a 
…h blank medication (treating them same as NULL values) to fix:

kind-lab#111

Invalid attribute value "": Attribute value must not be empty ("")
(e.g: "medicationCodeableConcept": {"coding": [{"code": "", "system": "http://mimic.mit.edu/fhir/mimic/CodeSystem/mimic-medication-name )
@piotrszul
Making 'dosageInstruction.timing' to include 'code' or 'repeat' in fh…
b03ef06 
…ir_medication_dispense.sql to fix:

kind-lab#102

Object must have some content
(e.g.: MedicationDispense", "dosageInstruction": [{}], "authorizingPrescription": [{"reference": "MedicationReq
@piotrszul
Replacing repated whitespaces with ' ' in pharmacy.frequency to make …
b3149dd 
…it comply with coding whitespacing rules and the cs_medication_frequency code system in fhir_medication_dispense.sql, to fix:

kind-lab#112

- None of the codings provided are in the value set 'MimicMedicationFrequency' (http://mimic.mit.edu/fhir/mimic/ValueSet/mimic-medication-frequency|2.2.0), and a coding from this value set is required) (codes = http://mimic.mit.edu/fhir/mimic/CodeSystem/mimic-medication-frequency#EVERY 4 WEEKS)
- Unknown code 'EVERY 4 WEEKS' in the CodeSystem 'http://mimic.mit.edu/fhir/mimic/CodeSystem/mimic-medication-frequency' version '2.2.0'
- The code 'EVERY 4 WEEKS' is not valid (whitespace rules)

MedicationDispensation:(timing": {"code": {"coding": [{"code": "EVERY 4  WEEKS", "system": "http://mimic.mit.edu/fhir/mimic/CodeSystem/mimic-medication-frequency"}]}
@piotrszul
Aligning coding display names in fhir resources with the display name…
aec446c 
…s in the code system for cs-diagnosis-icd9, cs-diagnosis-icd10 and cs-medication-etc to fix:

kind-lab#113
kind-lab#114

'Wrong Display Name  XXX' errors, e.g:
Wrong Display Name 'Acne Therapy Systemic - Tetracycline antibiotic' for http://mimic.mit.edu/fhir/mimic/CodeSystem/mimic-medication-etc#00005953. Valid display is 'Acne Therapy Systemic - Tetracyclines' (en) (for the language(s) '--')
@piotrszul
Adding 'emar_detail.product_unit' and 'emar_detail.dose_given_unit' a…
e4303e4 
…s source sources for cs_unit code system to fix:

Unknown code 'TPN Bag' in the CodeSystem 'http://mimic.mit.edu/fhir/mimic/CodeSystem/mimic-units' version '2.2.0'

(eg: ": {"dose": {"code": "TPN Bag", "unit": "TPN Bag", ) in MedicationAdministation.
@piotrszul
Removing the code system and values sets with not sql scripts from te…
873ad6f 
…rminology description tables.
@piotrszul
Adding the python script for generating code systems and value set re…
6a6c5f9 
…sorces from mimic database (fhir_trm schema).
@piotrszul
Switching mimic-profiles to the aehrc repo with updated terminlogy re…
cc90af7 
…sources
@alistairewj alistairewj merged commit 16d1cda into kind-lab:main Jul 31, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Wrong whitespace in Display Name ' Black or African American'
3 participants
@johngrimes @alistairewj @piotrszul