Epic: Historic Data Migration

[sambodeme]

Description

This epic focuses on developing a structured model for logging changes in historic data and executing the initial migration of this data.
The goal is to establish a model for tracking modifications made to historic data during the migration process and to identify potential improvement areas of the current migration code without altering the data.

Tasks

Tasks

Preview Give feedback

1. Create a Django command to trigger the migration process for a specific year #2840
   eng +1
   
2. Define Change Log Model #2838
   eng +1
3. Record Successful and Failed Migration Attempts (Logic) #2841
   eng +1
   
4. Implement Django Model for Tracking Migration Attempts #2843
   eng +1
   
5. Improve Data Migration Code Base #2842
   eng +1
   
6. Update Data Migration Logic to Use Census Models #2854
   13 of 13
   eng +1
   
7. Update CSV to Postgres Logic to Prevent Empty Values Turning Into nan #2874
   eng +1
   
8. Enforce Audit Year in All Queries #2879
   eng +1
9. Add support to migrate by year with pagination #2895
   eng +1
   
10. Add support to specify dbkey range for run_migration_for_year #2896
    eng +1
    
11. Create ADR for Historical Data Migration Transformation Tracking #2909
    +0
    
12. Improve the New Migration Logic #2911
    +0
13. Compile List of Questions around Data Discrepancy #2912
    census eng +2
    
14. Create ADR for Historical Data Migration Iteration #2920
    eng +1
    
15. Run migrations for tribal data #2894
    +0
    
16. Mapping / Transformation functions #2798
    3 of 3
    census +1
    
17. Implement an orchestrator to run end to end with parallelism and restartability #2582
    eng +1
18. Implement user creation based on an email address #2581
    eng +1
19. Implement a parking load for submissions that fail validations #2580
    eng +1
20. Update Validation Logic and Schema for Data Migration #2945
    eng +1
    
21. Record migration errors in MigrationErrorDetail table #2949
    +0
    
22. Record transformations in MigrationChangeRecord table #2950
    5 of 5
    eng +1
    

    
23. E2E refactor: demonstrate execution of code via GH Action #2671
    eng +1
    
24. Data Migration Logic Enhancement: Compiling Questions for Team Huddle #2832
    eng +1
    
25. Analyze string lengths for DBKEY and zipcode #2988
    +0
    
26. Split Census data by year excluding tribal data #2989
    +0
    
27. Add Test Table to General Information and Improve Existing Test Tables #3066
    eng +1
    
28. Replace All Print Statements with Log
29. Remove/clean old comments
30. Always read data as text strings from Census csv files #2993
    +0
    
31. Implement Transformation for Invalid ALNs #3023
    eng +1
    
32. Update end_to_end state transitions to match new transitions in dissemination #3030
    +0
    
33. Insert consent for tribal audit during migration #3033
    +0
    
34. Summarize ELECNOTES CONTENT data for TYPE_ID=2 #3060
    eng +1
    
35. Modify MigrationChangeRecord table #3094
    +0
    
36. Analyze Final Census data #3134
    census +1
    
37. Code Improvement #3163
    eng +1
    
38. Test 2019 through 2022 migrations in Preview #3166
    5 of 5
    +0
    
39. Enforce Correct Sorting when Migrating Reports #3178
    eng +1
    
40. Update Data Migration to Include Cognizant and Oversight Agency Data #3202
    eng +1
    
41. Add Index to Historical Database Tables to Mitigate Queries Performance Issues #3206
    eng +1
    
42. Questions regarding Data Migration #3215
    +0
    
43. Data Migration Tests #3219
    +0
44. Add index to Dissemination tables to improve query performance #3220
    +0
    
45. Add Command for Re-running Failed Migration by Error Tag #3232
    eng +1
    
46. Backup 2019 through 2022 dissemination tables in Preview #3289
    +0
    
47. Historic data go-live #3287
    23 of 26
    +0
48. Backup Preview database #3385
    +0
    

    

[benjaminrobinson]

Hello!
Do you have an estimated time of completion for the migration of the Historic Data from the Census?

My team loves the API as it is a huge improvement over using manual data uploads from the Census for our work! Thanks so much for developing this in the public so we can understand what is going on with the FAC project. I deeply appreciate it!

Sincerely, Ben Robinson.
Manager of Data Strategy and Insights
Universal Service Administrative Co.
Email: [email protected]

[jadudm]

We don't generally communicate timelines here; we're deep in trying to finish this right now. We're doing final testing through our various staging environments before we commit things to production.

We'll post to

https://www.fac.gov/info/updates/

in conjunction with that completion.

Thank you for the interest and enthusiasm!

[benjaminrobinson]

Thanks, @jadudm! Appreciate the quick reply. Will look to the Updates page for more information going forward, there just hasn't been anything posted there since mid-December.

Sincerely, Ben Robinson.
Manager of Data Strategy and Insights
Universal Service Administrative Co.
Email: [email protected]

[jadudm]

We've... been buried in the final details of the migration. We should have done an interim update. We'll get back on a good cadence.

[jadudm]

Closing this ticket. The individual sub-tickets can be picked up in backlog or other epics. We have moved the historic data to production, and now have more work to do around it. Congrats to the data migration subteam!

[sambodeme]

Link to next #3364