Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Create a process to backfill all tables from HARs older than March 2022 #138

Closed
giancarloaf opened this issue Sep 1, 2022 · 0 comments · Fixed by HTTPArchive/dataform#10
Closed

Create a process to backfill all tables from HARs older than March 2022 #138

giancarloaf opened this issue Sep 1, 2022 · 0 comments · Fixed by HTTPArchive/dataform#10
Assignees
@giancarloaf
Milestone
M4: Improving dis...

Comments

@giancarloaf
Copy link
Collaborator

giancarloaf commented Sep 1, 2022 •
edited
Loading

There are some attributes necessary for the pipeline to function properly that are only available in recent HARs (i.e. March 2022 and on).

We will need a methodology to backfill older tables on request.

A suggestion from @rviscomi - this might include nulling out fields where the necessary data is missing from HARs, and supplementing from the historical tables.

Reference this method used previously for generating all.pages from legacy tables.
#15 (comment)

Originally posted by @rviscomi in #136 (comment)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@giancarloaf giancarloaf

Labels
None yet
Projects
None yet
Milestone
M4: Improving discoverability
Development

Successfully merging a pull request may close this issue.

Backfills & deprecating legacy tables HTTPArchive/dataform
2 participants
@rviscomi @giancarloaf