[STORY] Configure XNAT-ingest workflow from TBP to AIS

[tclose]

Metadata

Epic: #271
Feature: #273

Description

• The ghcr.io/australian-imaging-service/xnat-ingest:latest image needs to be regularly pulled to the staging server k8s cluster to pick up updates
• Configure the staging command, docker run ghcr.io/australian-imaging-service/xnat-ingest:latest stage, to run regularly:
  • Daily @ 12am
  • Environment vars:
    • XNAT_INGEST_STAGE_DICOMS_PATH=/vol/vmhost/kubernetes/<path>/<to>/<dicom>/<store>/**/*.IMA (@fxusyd)
    • XNAT_INGEST_STAGE_DIR=/vol/vmhost/usyd-data-export/STAGING
    • XNAT_INGEST_STAGE_PROJECT=ProtocolName
    • XNAT_INGEST_STAGE_SUBJECT=PatientID
    • XNAT_INGEST_STAGE_VISIT=AccessionNumber
    • XNAT_INGEST_STAGE_ASSOCIATED="/vol/vmhost/usyd-data-export/RAW-DATA-EXPORT/{PatientName.family_name}_{PatientName.given_name}/.ptd","./[^\.]+.[^\.]+.[^\.]+.(?P\d+).[A-Z]+_(?P[^\.]+)."
    • XNAT_INGEST_STAGE_DELETE=0 (for now, will eventually be 1 when we are satisfied everything is working)
    • XNAT_INGEST_STAGE_LOGFILE=<somewhere-sensible>,INFO
    • XNAT_INGEST_STAGE_DEIDENTIFY=1
• Configure the transfer command, docker run ghcr.io/australian-imaging-service/xnat-ingest:latest transfer, to run regularly:
  • Daily @ 6am
  • Environment vars:
    • XNAT_INGEST_STAGE_DIR=/vol/vmhost/usyd-data-export/STAGING
    • XNAT_INGEST_TRANSFER_REMOTE_STORE=<s3-bucket>
    • XNAT_INGEST_TRANSFER_STORE_CREDENTIALS=<s3-bucket-access-key>,<s3-bucket-access-secret> (@fxusyd)
    • XNAT_INGEST_TRANSFER_LOGFILE=<somewhere-else-sensible>,INFO
    • XNAT_INGEST_TRANSFER_DELETE=0 (for now, will eventually be 1 when we are satisfied everything is working)
    • XNAT_INGEST_TRANSFER_XNAT_LOGIN=https://xnat.sydney.edu.au,<role-account-user>,<role-account-pass> (@fxusyd)

@fxusyd: I have put your handle alongside the parts I think you will need to help out with.

Acceptance Criteria

• 1. DICOM data is copied from the DICOMs store to STAGING dir and deidentified
• 2. Associated raw-data files (i.e. *.ptd) are copied/moved from RAW-DATA-EXPORT to STAGING dir alongside matching DICOMs
• 3. Staged data is sync'd from STAGING dir to S3 bucket

Blocked by

• [STORY] Ensure that GHCR is accessible from staging server #283

[tclose]

@dean-taylor does this make sense?

[fxusyd]

Testing it in K8s on the staging server now

[fxusyd]

S3 bucket used by xnat-ingest transfer is not accessible. Emailed RNSH to add it to whitelist.

[fxusyd]

Two issues were identified.

1. Fixed pod within K8s cluster couldn't make external connections due to proxy.
2. Waiting on new container image with script to bypass SSL verification due to self-signed certificate is used in RNSH.