Deploy new gardener and k8s etl parser to prod

[gfr10598]

Looks like prod mostly runs in us-central instead of east region. So the new k8s cluster should probably be there too.

There is some documentation in the README.md file from January.

Steps:

1. Create data-processing cluster, with appropriate networking options
2. Create node-pools for etl and gardener.
3. Add cloud builder rule for etl prod- tags.

[gfr10598]

Based on info in README.md, added create-cluster.sh in new branch, which has all the gcloud commands to set up the network, subnet, firewall rules, cluster, and node-pools.

[gfr10598]

Manually added cloud build trigger. Note that gcloud beta builds now supports creating triggers, too.

gcloud beta builds triggers create github
--repo-name=[REPO_NAME]
--repo-owner=[REPO_OWNER]
--branch-pattern=".*"
--build-config=[BUILD_CONFIG_FILE] \

[gfr10598]

bq --project=mlab-oti mk tmp_ndt
bq --project=mlab-oti mk raw_ndt

Need to add the table creation and schema updates to etl-schema.

[gfr10598]

CREATE OR REPLACE TABLE mlab-oti.raw_ndt.ndt7
PARTITION BY date CLUSTER BY metro
AS
SELECT date, REGEXP_EXTRACT(parser.ArchiveURL , ".-mlab[1-4]-([a-z]{3})[0-9]{2}.") AS metro, id, * EXCEPT(date,id)
FROM mlab-sandbox.tmp_ndt.ndt7
WHERE date > CURRENT_DATE()

[gfr10598]

CREATE OR REPLACE TABLE mlab-oti.raw_ndt.annotation
PARTITION BY date CLUSTER BY metro
AS
SELECT date, REGEXP_EXTRACT(parser.ArchiveURL , ".-mlab[1-4]-([a-z]{3})[0-9]{2}.") AS metro, id, * EXCEPT(date,id)
FROM mlab-sandbox.tmp_ndt.annotation
WHERE date > CURRENT_DATE()

[gfr10598]

NOTE: bigquery does not store data in us-central. This may mean that we will get network egress charges for the BQ loads?

Probably should specify the BQ dataset data_location=US to make it multi-regional. See https://cloud.google.com/bigquery/docs/locations#multi-regional-locations

The documentation is not crystal clear, so we should probably just look for these charges in billing.