[ETL-674] Add script to compress intermediate JSON

[philerooski]

Recombines the NDJSON part sets which we wrote to our intermediate bucket and uploads a gzipped NDJSON to an adjacent location.

Data Input

Each part set is located at:

s3://<bucket-name>/<namespace>/json/dataset={dataset}/cohort={cohort}/

where {dataset} is the dataset identifier (datatype), and {cohort} is either 'adults_v1' or 'pediatric_v1'. Each file in the part set is named:

{file_identifier}.part{part_number}.ndjson

Data Output

Each combined and gzipped part set is written to:

s3://<bucket-name>/<namespace>/compressed_json/dataset={dataset}/cohort={cohort}/{file_identifier}.ndjson.gz

[sonarqubecloud]

Quality Gate passed

Issues
17 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

[philerooski]

@BryanFauble I amended my commit but it seems like SonarCloud ran its analysis on the previous version of the code again.

[BryanFauble]

@BryanFauble I amended my commit but it seems like SonarCloud ran its analysis on the previous version of the code again.

I am not sure if SonarCloud would work properly with amended commits. I suspect that is needs a new commit hash for sonarcloud to run.

[thomasyu888]

🔥 LGTM!