[ETL-644] Update drop_table_duplicates function to use new method

[philerooski]

Primary changes are L192-208 where we swap the old (and faulty) sort + drop duplicate method of removing duplicates with the new window + sort + rank + filter method.

I also did some refactoring to make things a little more intuitive:

• drop_table_duplicates and drop_deleted_healthkit_data now return Spark DataFrames rather than Glue DynamicFrames. These functions already produced DataFrames so I removed the final cast as a DynamicFrame to simplify things. We cast back to a DynamicFrame on L662 since both branches of the conditional (relationalize+write else write) take a DynamicFrame as input.
• Mostly as a consequence of the above refactor, rather than deriving the data type from the table name in non-main functions I derive the data type once in main and pass that as an argument to functions which reference the data type. I think this consolidates the number of places where a data type is derived from its table name (or from the job arguments, where the table name is derived), making things more consistent generally.

[sonarqubecloud]

Quality Gate passed

Issues
13 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud

[BryanFauble]

Excellent work!

[thomasyu888]

Fun fact, this is exactly what I do to create de-duplicated tables within the synapse data warehouse: https://github.com/Sage-Bionetworks/snowflake/blob/b4755ad226abe29d30308708539c99d632dd2337/synapse_data_warehouse/synapse/tables/V1.11.0__initial_creation_of_latest_tables.sql#L3-L17

[philerooski]

I think this way may be more common in databases, versus sorting and dropping as you would an in-memory dataframe. Perhaps it has something to do with the distributed (across partitions) nature of the data.

[thomasyu888]

🔥 LGTM!

[rxu17]

LGTM!

[philerooski]

As a sanity check, I compared the Fitbit Intraday Combined Parquet data produced by the main branch (which uses the older method of dropping duplicates) to this feature branch. Both were produced from the same set of JSON pilot data in the dev environment. While each contained the same number of unique records, I saw differences in which exports the records were sourced from, confirming that these two methods were dropping duplicates differently:

>>> fitbit_branch.export_end_date.value_counts() 
export_end_date
2023-01-12T00:00:00    1996228
2023-01-03T00:00:00     173251
2023-01-14T00:00:00      59005
2023-06-27T00:00:00      10050
Name: count, dtype: int64
>>> fitbit_main.export_end_date.value_counts()
export_end_date
2023-01-12T00:00:00    1918724
2023-01-03T00:00:00     251167
2023-01-14T00:00:00      58593
2023-06-27T00:00:00      10050
Name: count, dtype: int64

I then read the JSON data as a pandas dataframe and dropped duplicates. I understand and trust how duplicates are dropped in an in-memory dataframe more so than in a Spark dataframe, so this just a way to independently verify that we are dropping duplicates correctly. I saw the same distribution of records as the feature branch:

>>> fitbit_json_df_no_dups.export_end_date.value_counts()
2023-01-12T00:00:00    1996228
2023-01-03T00:00:00     173251
2023-01-14T00:00:00      59005
2023-06-27T00:00:00      10050
Name: export_end_date, dtype: int64