Prepare for migration of report queries

[max-ostapenko]

We want to replace legacy reports script https://github.com/HTTPArchive/bigquery/tree/master/sql
This is the implementation that follows previous discussion.

TODO list for reports:

• run aggregated table updates with fresh reports data (dataset reports) on crawl_complete
• create reports configuration file with a timeseries and a histogram
• trigger GCS upload whenever data is updated in BQ

for tech reports:

• run aggregated table updates on cwv_tech_report tag after CrUX update
• create reports configuration file
• update Firestore collections whenever data was updated in BQ

Supports features:

• Run monthly histograms SQLs when crawl is finished
  SQLs and exports run as part of a tag workflow invocation (when CrUX data is updated)

• [?] Run longer term time series SQLs when crawl is finished
  Example needed

• Be able to run the time series in an incremental fashion
  We can rerun any previous month by redefining the data variable in the corresponding actions.

• [?] Handle different lenses (Top X, WordPress, Drupal, Magento)
  To clarify

• [?] Handle CrUX reports (monthly histograms and time series) having to run later.
  Need to clarify

• Be able to upload to cloud storage in GCP to allow it to be hosted on our CDN
  Implemented with dataform-export function (together with Firestore export)

• Be able to run and only run reports missing (histograms) or missing dates (time series)
  The actions to run can be selected manually in Dataform console.

• Be able to force rerun (to override any existing reports).
  Use crawl-data repo in Dataform to rerun

• Be able to run a subset of reports.
  Select actions manually and rerun via Dataform console.

Resolves:

• Update report JSON exports to crawl dataset httparchive.org#938
• Document how the new analysis pipeline works, including metrics data-pipeline#21
• Monitor for missing reports data-pipeline#24

[max-ostapenko]

I found reports config file - seems a good idea to keep all the configs in one place (more transparent for future contributors).

I copied it over here (to experiment with) and added the queries.
I wouldn't be able to add the queries unless the format supports multiline strings - so just saved in JS.
Actually it is required to be readable with python - YAML?

[max-ostapenko]

I couldn't make it work with remote config (e.g. reading from httparchive repo).
The file needs to be stored in this repo at the runtime.

[max-ostapenko]

In reports_* datasets we could store intermediate aggregated data - it's easier to check for data issues in BQ than in GCS.
Cloud Function then will pick fresh row batches and save them to GCS.

Currently it's configured to have a table per metric per chart type, e.g httparchive.reports_timeseries.totalBytes
We could (but it seems a bit more complicated for maintaining and querying), store all the metrics for one chart type in a single table (and cluster by metric).

[max-ostapenko]

Query parameters.
I found only date.
Need to list all the required and add the queries to test them with.

[max-ostapenko]

@tunetheweb here is a demo version that needs to be discussed.
Once we see that it covers all the requirements and agree on feasibility of the 3 topics in comments above - I'll finalise the part with uploading to GCS.

And I have no idea what to do with lenses and 2 more requests (see in description)..

[max-ostapenko]

I'm using this part of SQL query as a trigger even metadata for BQ to GCS (Firestore) exports.
Every time the table update query succeeds - it triggers the export.