[ETL-653] Add corresponding deleted data type table for all HealthKit data types #116
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
thomasyu888
reviewed
Jun 3, 2024
| "HealthKitV2Samples", | ||
| "HealthKitV2Statistics", | ||
| "HealthKitV2Samples_Deleted", | ||
| "HealthKitV2Statistics_Deleted" |
There was a problem hiding this comment.
Should we add all the other HealthKit data columns here?
There was a problem hiding this comment.
No, these are the only data types with a subtype.
rxu17
approved these changes
Jun 3, 2024
|
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We didn't have
*_Deletedtables for some HealthKit data types, which caused L255 ofsrc/glue/jobs/json_to_parquet.pyto raise a caught exception rather than removing deleted records for that data type. This only affected theHealthKitV2Heartbeatdata type since we haven't received deleted records for any other HealthKit data type besidesHealthKitV2Samples, which we already had a corresponding*_Deletedtable for.In addition to adding these new tables, I updated a few lines in
src/glue/jobs/s3_to_json.py. In particular, the addition of theHealthKitV2Samples_DeletedandHealthKitV2Statistics_Deleteddata types to theDATA_TYPES_WITH_SUBTYPEhas the effect of adding the subtype included in the JSON file name into the JSON itself as aTypefield. This field is not relevant when dropping deleted records from the data type, but it is file metadata which we were previously ignoring and not including in the resulting JSON datasets.