Epic: Initial end-to-end workbook code refactor

[jadudm]

At a glance

In order to migrate historical data
as a product owner
I want to know the process held the data to the same (or higher) standard as our intake.

The code: a brief and likely inaccurate history

Months ago, I developed a workbook generator. It began life in order to exercise our early JSON Schema validations. It helped find many logical errors, and drove a great deal of improvement in our intake.

It grew and evolved, serving in a one-off capacity for generating workbooks. As the API came online, I extended it to run end-to-end. Or, I discovered I could generate workbooks in memory, and feed them to the Django application. This rapidly became an end-to-end test: a set of workbooks could be generated, fed to the app, run through all of our validations (including cross-val), and then the data at time of input is compared with what is available via the API.

It is this code that we are basing our historical data migration on.

Approach

The initial refactoring will clone the existing code to another part of the tree, and then walk it forward.

1. The code will always generate workbooks that validate end-to-end.
2. Unit tests for critical/core functions will be developed as needed along the way.
3. Cleanup/refactoring can and should happen as needed.
4. No massive/architectural rewrites are part of this work.

At all points in the refactoring the code should continue to generate workbooks that validate.

Tasks

Preview Give feedback

1. E2E refactor: Clone E2E code from dissemination to census_historical_migration #2723
   eng +1
   
2. E2E refactor: Clone tests all_should_pass and all_should_fail, plus workbooks #2727
   eng +1
   
3. E2E refactor: Update import statements for new E2E code
4. E2E refactor: demonstrate execution of code via GH Action #2671
   eng +1
   
5. E2E refactor: document execution of code locally, in GH Actions
6. Add a README for running the end-to-end code
7. Add a README for running the historical migration code

#2672 is broken out as its own epic for refactoring/improving this code.

When these steps are complete, the team should have a tool that can generate workbooks using the "messy" historical data loaded into our environments via Docker container. It does not yet work with new/"authoritative" Census data.