MHub / GC - Add the Node21 baseline model #41

silvandeleemput · 2023-08-02T15:42:43Z

This PR adds the GC STOIC baseline model (from the STOIC2021 challenge) to MHub.
GitHub Repo: https://github.com/node21challenge/node21_detection_baseline
GC page: https://node21.grand-challenge.org/

Algorithm I/O

Input should be a Chest X-Ray image (MHA) (This MHub implementation expects a Dicom which is converted to MHA using panimg MhaConverter)
Outputs are:
- A JSON file with the bounding boxes and probabilities for the nodules
Test DICOM images for this algorithm can be downloaded from IDC:

# To download the files in this manifest, first install s5cmd (https://github.com/peak/s5cmd),
# then run the following command:
# s5cmd --no-sign-request --endpoint-url https://storage.googleapis.com run file_manifest_gcs.s5cmd
cp s3://public-datasets-idc/8fb5dfd4-0bd1-4dc2-a8a7-7a867c40450e/* .
cp s3://public-datasets-idc/3c0bd6df-1772-4bb2-abe5-570665311253/* .

Caveats

I added the CR = Meta(mode="CR") # CR Computed Radiography label, which should move to mhubio/core/templates.py, not really sure if this is the best label for Chest X-Rays (CXR)
~~The PR target is main, but should be something like m-gc-node21-baseline~~
~~In the Dockerfile the MHub model repo integration is currently marked TODO and should be uncommented/edited~~
~~Added a TODO to the MHAConverter since it doesn't have the panimg backend yet (without this backend it fails for my test image), and it uses a run.py script to use the panimg mhaconverter.~~

* Dockerfile * Use mhubio.run as entrypoint * Fixed commit for node21 repo * Updated TODO on MHub model code integration * default.yml * Changed mhaconverter backend to panimg * Node21BaselineRunner.py * Added comments * Removed * run.py * PanImgConverters.py

silvandeleemput · 2023-09-14T14:05:37Z

Updated this PR:

Dockerfile
- Use mhubio.run as entrypoint
- Fixed commit for node21 repo
- Updated TODO on MHub model code integration
default.yml
- Changed mhaconverter backend to panimg
Node21BaselineRunner.py
- Added comments
Removed
- run.py
- PanImgConverters.py

…_models into m-gc-node21-baseline

LennyN95 · 2023-11-22T11:30:11Z

NOTE: This PR proposes a model currently using a dynamic length output. We are working on a method to export dynamic length outputs and will likely introduce a @IO.Out.Data.Many decorator (similar to our @IO.Output -> @IO.Outputs decorators).

models/gc_node21_baseline/utils/Node21BaselineRunner.py

silvandeleemput · 2024-04-18T11:36:47Z

gc_node21_baseline_output.zip

silvandeleemput · 2024-04-18T11:36:55Z

/test

sample:
  idc_version: 17.0
  data:
    - SeriesInstanceUID: 1.3.6.1.4.1.14519.5.2.1.6279.6001.210102868760281756294235082201
      aws_url: s3://idc-open-data/b1c859c3-7471-43e7-9882-0bb5420ec18a/*
      path: dicom

reference:
  url: https://github.com/MHubAI/models/files/15024198/gc_node21_baseline_output.zip

Test Results (24.04.18_22.31.12_E3krCTzQUg)

id: e8377a1b-338f-4fab-8647-fe4b72a250be
date: '2024-04-18 22:36:27'
checked_files:
- file: nodules.json
  path: /app/test/src/1.3.6.1.4.1.14519.5.2.1.6279.6001.210102868760281756294235082201/nodules.json
  checks:
  - checker: DataFileCheck
    notes:
    - label: Value Match
      description: These keys have identical values
      info: type,boxes.[0].corners.[0].[0],boxes.[0].corners.[0].[1],boxes.[0].corners.[0].[2],boxes.[0].corners.[1].[0],boxes.[0].corners.[1].[1],boxes.[0].corners.[1].[2],boxes.[0].corners.[2].[0],boxes.[0].corners.[2].[1],boxes.[0].corners.[2].[2],boxes.[0].corners.[3].[0],boxes.[0].corners.[3].[1],boxes.[0].corners.[3].[2],boxes.[0].probability,boxes.[1].corners.[0].[0],boxes.[1].corners.[0].[1],boxes.[1].corners.[0].[2],boxes.[1].corners.[1].[0],boxes.[1].corners.[1].[1],boxes.[1].corners.[1].[2]
        (+ 31 more)
    findings:
    - label: Value similar
      description: Value of key 'boxes.[2].corners.[0].[0]' is similar in source file
      subpath: boxes.[2].corners.[0].[0]
      info:
        scale: 10
        precision: -1
        src: 56.17
        ref: 52.17
    - label: Value similar
      description: Value of key 'boxes.[2].corners.[0].[1]' is similar in source file
      subpath: boxes.[2].corners.[0].[1]
      info:
        scale: 100
        precision: -1
        src: 210.43
        ref: 259.47
    - label: Value similar
      description: Value of key 'boxes.[2].corners.[1].[0]' is similar in source file
      subpath: boxes.[2].corners.[1].[0]
      info:
        scale: 1
        precision: -1
        src: 48.52
        ref: 31.0
    - label: Value similar
      description: Value of key 'boxes.[2].corners.[1].[1]' is similar in source file
      subpath: boxes.[2].corners.[1].[1]
      info:
        scale: 100
        precision: -1
        src: 210.43
        ref: 259.47
    - label: Value similar
      description: Value of key 'boxes.[2].corners.[2].[0]' is similar in source file
      subpath: boxes.[2].corners.[2].[0]
      info:
        scale: 1
        precision: -1
        src: 48.52
        ref: 31.0
    - label: Value similar
      description: Value of key 'boxes.[2].corners.[2].[1]' is similar in source file
      subpath: boxes.[2].corners.[2].[1]
      info:
        scale: 100
        precision: -1
        src: 201.47
        ref: 237.66
    - label: Value similar
      description: Value of key 'boxes.[2].corners.[3].[0]' is similar in source file
      subpath: boxes.[2].corners.[3].[0]
      info:
        scale: 10
        precision: -1
        src: 56.17
        ref: 52.17
    - label: Value similar
      description: Value of key 'boxes.[2].corners.[3].[1]' is similar in source file
      subpath: boxes.[2].corners.[3].[1]
      info:
        scale: 100
        precision: -1
        src: 201.47
        ref: 237.66
    - label: Value similar
      description: Value of key 'boxes.[3].corners.[0].[0]' is similar in source file
      subpath: boxes.[3].corners.[0].[0]
      info:
        scale: 10
        precision: -1
        src: 52.16
        ref: 56.17
    - label: Value similar
      description: Value of key 'boxes.[3].corners.[0].[1]' is similar in source file
      subpath: boxes.[3].corners.[0].[1]
      info:
        scale: 100
        precision: -1
        src: 259.48
        ref: 210.43
    - label: Value similar
      description: Value of key 'boxes.[3].corners.[1].[0]' is similar in source file
      subpath: boxes.[3].corners.[1].[0]
      info:
        scale: 1
        precision: -1
        src: 31.0
        ref: 48.52
    - label: Value similar
      description: Value of key 'boxes.[3].corners.[1].[1]' is similar in source file
      subpath: boxes.[3].corners.[1].[1]
      info:
        scale: 100
        precision: -1
        src: 259.48
        ref: 210.43
    - label: Value similar
      description: Value of key 'boxes.[3].corners.[2].[0]' is similar in source file
      subpath: boxes.[3].corners.[2].[0]
      info:
        scale: 1
        precision: -1
        src: 31.0
        ref: 48.52
    - label: Value similar
      description: Value of key 'boxes.[3].corners.[2].[1]' is similar in source file
      subpath: boxes.[3].corners.[2].[1]
      info:
        scale: 100
        precision: -1
        src: 237.66
        ref: 201.47
    - label: Value similar
      description: Value of key 'boxes.[3].corners.[3].[0]' is similar in source file
      subpath: boxes.[3].corners.[3].[0]
      info:
        scale: 10
        precision: -1
        src: 52.16
        ref: 56.17
    - label: Value similar
      description: Value of key 'boxes.[3].corners.[3].[1]' is similar in source file
      subpath: boxes.[3].corners.[3].[1]
      info:
        scale: 100
        precision: -1
        src: 237.66
        ref: 201.47
    - label: Value similar
      description: Value of key 'boxes.[4].probability' is similar in source file
      subpath: boxes.[4].probability
      info:
        scale: 1
        precision: 1
        src: 0.06
        ref: 0.05
  - checker: SizeCheck
summary:
  files_missing: 0
  files_extra: 0
  checks:
    DataFileCheck:
      files: 1
      findings:
        Value similar: 17
    SizeCheck:
      files: 1
conclusion: false

Test Results (24.04.25_08.24.29_0ZZ5RYzDlk)

id: 12aafd5a-a277-4f27-a5c7-708d5670cf91
date: '2024-04-25 08:30:39'
checked_files:
- file: nodules.json
  path: /app/test/src/1.3.6.1.4.1.14519.5.2.1.6279.6001.210102868760281756294235082201/nodules.json
  checks:
  - checker: DataFileCheck
    notes:
    - label: Value Match
      description: These keys have identical values
      info: type,boxes.[0].corners.[0].[0],boxes.[0].corners.[0].[1],boxes.[0].corners.[0].[2],boxes.[0].corners.[1].[0],boxes.[0].corners.[1].[1],boxes.[0].corners.[1].[2],boxes.[0].corners.[2].[0],boxes.[0].corners.[2].[1],boxes.[0].corners.[2].[2],boxes.[0].corners.[3].[0],boxes.[0].corners.[3].[1],boxes.[0].corners.[3].[2],boxes.[0].probability,boxes.[1].corners.[0].[0],boxes.[1].corners.[0].[1],boxes.[1].corners.[0].[2],boxes.[1].corners.[1].[0],boxes.[1].corners.[1].[1],boxes.[1].corners.[1].[2]
        (+ 43 more)
    findings:
    - label: Value similar
      description: Value of key 'boxes.[2].corners.[0].[0]' is similar in source file
      subpath: boxes.[2].corners.[0].[0]
      info:
        scale: 1
        precision: 1
        src: 52.16
        ref: 52.17
    - label: Value similar
      description: Value of key 'boxes.[2].corners.[0].[1]' is similar in source file
      subpath: boxes.[2].corners.[0].[1]
      info:
        scale: 1
        precision: 1
        src: 259.48
        ref: 259.47
    - label: Value similar
      description: Value of key 'boxes.[2].corners.[1].[1]' is similar in source file
      subpath: boxes.[2].corners.[1].[1]
      info:
        scale: 1
        precision: 1
        src: 259.48
        ref: 259.47
    - label: Value similar
      description: Value of key 'boxes.[2].corners.[3].[0]' is similar in source file
      subpath: boxes.[2].corners.[3].[0]
      info:
        scale: 1
        precision: 1
        src: 52.16
        ref: 52.17
    - label: Value similar
      description: Value of key 'boxes.[4].probability' is similar in source file
      subpath: boxes.[4].probability
      info:
        scale: 1
        precision: 1
        src: 0.06
        ref: 0.05
  - checker: SizeCheck
summary:
  files_missing: 0
  files_extra: 0
  checks:
    DataFileCheck:
      files: 1
      findings:
        Value similar: 5
    SizeCheck:
      files: 1
conclusion: false

silvandeleemput · 2024-04-18T11:39:45Z

/review

We updated the meta.json file, we implemented the suggestion for the OutputDatas (probabilities, and bounding boxes as JSON), and we have wrapped the algorithm/model in a CLI to avoid stdout capture issues. It should be as good as done, please have a look.

LennyN95

To keep the structure clean, the utils folder ideally only contains any MHub-IO related utilities (e.g., custom Modules such as the Node21BaselineRunner) but no model implementation code. All model implementation should go into an src folder, all additional resources into an res folder.

Making such a change is purely organizational (although for the utils folder, it is somewhat meaningful as we parse this folder to detect available Modules).

However, in case you have to make any code adjustments and we have to redo the test, I'd ask for this change.

LennyN95

The test failed and showed (high) variation in the bounding boxes. This needs to be investigated before we can proceed.

silvandeleemput · 2024-04-23T13:34:23Z

I investigated this issue and found the root cause for the large bounding box differences.

Rerunning on my machine consistently gave me the same output as I have uploaded before.
I have 5 detected nodules with each 4 corners (3 values x,y,z=0) and a probability.

Comparing these with your test results I found:

The number of detected bounding boxes matches the number I found on my side (5). This seems ok.
All found probabilities match pretty closely. (some small deviations are expected because of GPU hardware/driver differences and rounding errors). These seem fine.
Bounding boxes 0, 1, and 4 seem fine and the same as what I found.
Bounding boxes 2 & 3 are swapped, having the same probability! Giving rise to the big found differences. It seems to be a sorting issue.

I can fix this by applying a better sorting algorithm after the output is generated to produce a more consistent output JSON file. This can be added to the cli.py and go together with a small refactor of the cli.py file to a /src directory. This should be under /models/gc_node21_baseline/src/ then ?

… added improved output nodule sorting

silvandeleemput · 2024-04-24T08:14:23Z

@LennyN95 alright, I have added the improved sorting, moved the cli.py to a src directory, and updated the model version to 1.1, both in the meta.json and the Dockerfile. The previous test can be redone and should produce better results because the sorting issue is fixed. This model should be as good as done.

silvandeleemput · 2024-04-25T09:26:14Z

I looked at the new result generated after adding the sorting fix, and the sorting works now: the same bounding boxes are found and sorted in the correct order. The found differences this time around are only small rounding errors for both the coordinates of the bounding boxes and for the probabilities, which are expected. This result is acceptable.

LennyN95

Test passed (rounding errors expected, see comment from the model contributor).

silvandeleemput added 2 commits August 2, 2023 17:11

added working MHub implementation of the node21 baseline

a0eac94

minor modification Node21 output JSON description

890f2e6

LennyN95 self-assigned this Aug 13, 2023

silvandeleemput and others added 2 commits September 14, 2023 15:29

Merge branch 'MHubAI:main' into m-gc-node21-baseline

125f251

silvandeleemput and others added 4 commits October 10, 2023 16:03

Merge branch 'MHubAI:main' into m-gc-node21-baseline

4940d9a

fix revert to default workdir

75cea2b

Merge branch 'm-gc-node21-baseline' of github.com:DIAGNijmegen/MHubAI…

8e9ea1c

…_models into m-gc-node21-baseline

add meta.json

8e17ccc

LennyN95 added the Requires MHub-IO Update Only for PR with model suggestions which require extended MHub-IO functionality. label Nov 22, 2023

LennyN95 reviewed Nov 23, 2023

View reviewed changes

models/gc_node21_baseline/utils/Node21BaselineRunner.py Outdated Show resolved Hide resolved

LennyN95 added +Model: ACTION REQUIRED and removed Requires MHub-IO Update Only for PR with model suggestions which require extended MHub-IO functionality. labels Nov 23, 2023

silvandeleemput and others added 2 commits November 23, 2023 21:30

add mhub model definition import to Dockerfile

9e9c1df

Merge branch 'MHubAI:main' into m-gc-node21-baseline

3111827

LennyN95 removed the +Model: ACTION REQUIRED label Jan 8, 2024

miriam-groeneveld and others added 7 commits April 16, 2024 09:49

Merge branch 'MHubAI:main' into m-gc-node21-baseline

67fc8b5

Updated meta.json

06e1e25

meta.json - updated version number

325e8a5

meta.json - changed table entries from float into string

777ff39

Dockerfile - fixed source pull to version tag

5cc1190

runner.py - add outputdatas for nodule probabilities and bounding boxes

2bafc52

wrap algorithm in cli to allow proper capture within mhub framework

fb17009

github-actions bot added the TEST REQUESTED label Apr 18, 2024

github-actions bot added the REQUEST REVIEW Attach this label to your PR when your submission is "in progress" and is ready to be reviewed by us label Apr 18, 2024

LennyN95 approved these changes Apr 18, 2024

View reviewed changes

LennyN95 reviewed Apr 19, 2024

View reviewed changes

LennyN95 requested changes Apr 19, 2024

View reviewed changes

Dockerfile, meta.json - Updated source to v1.1, moved cli to src dir,…

e27ec43

… added improved output nodule sorting

LennyN95 removed REQUEST REVIEW Attach this label to your PR when your submission is "in progress" and is ready to be reviewed by us TEST REQUESTED labels Apr 25, 2024

LennyN95 approved these changes Apr 25, 2024

View reviewed changes

LennyN95 merged commit 2df5376 into MHubAI:main Apr 25, 2024
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MHub / GC - Add the Node21 baseline model #41

MHub / GC - Add the Node21 baseline model #41

silvandeleemput commented Aug 2, 2023 •

edited

Loading

silvandeleemput commented Sep 14, 2023

LennyN95 commented Nov 22, 2023

silvandeleemput commented Apr 18, 2024

silvandeleemput commented Apr 18, 2024 •

edited by LennyN95

Loading

silvandeleemput commented Apr 18, 2024

LennyN95 left a comment

LennyN95 left a comment

silvandeleemput commented Apr 23, 2024

silvandeleemput commented Apr 24, 2024

silvandeleemput commented Apr 25, 2024

LennyN95 left a comment

MHub / GC - Add the Node21 baseline model #41

MHub / GC - Add the Node21 baseline model #41

Conversation

silvandeleemput commented Aug 2, 2023 • edited Loading

Algorithm I/O

Caveats

silvandeleemput commented Sep 14, 2023

LennyN95 commented Nov 22, 2023

silvandeleemput commented Apr 18, 2024

silvandeleemput commented Apr 18, 2024 • edited by LennyN95 Loading

silvandeleemput commented Apr 18, 2024

LennyN95 left a comment

Choose a reason for hiding this comment

LennyN95 left a comment

Choose a reason for hiding this comment

silvandeleemput commented Apr 23, 2024

silvandeleemput commented Apr 24, 2024

silvandeleemput commented Apr 25, 2024

LennyN95 left a comment

Choose a reason for hiding this comment

silvandeleemput commented Aug 2, 2023 •

edited

Loading

silvandeleemput commented Apr 18, 2024 •

edited by LennyN95

Loading