Reenable advanced observatory features like search

.gitattributes

@@ -0,0 +1,2 @@
+# R wants HTML files to have UNIX line endings ie. just \n or LF 
+*.html text eol=lf

.gitignore

@@ -144,11 +144,22 @@ vignettes/*.pdf
 rsconnect/
 
 /dags/r_scripts/credentials.json
-/dags/galileo/*.csv
+*.csv
 /data
 /.idea/
 airflow.cfg
 *.avro
 /shiny/observatory/htpasswd
-/tools/warcraider/*.warc
-/shiny/observatory/*.csv
+htpasswd
+.htpasswd
+*.warc
+*.csv
+*.rdata
+*.xls
+*.xlsx
+*.gexf
+*.graphml
+Staticfile.auth
+*.iml
+*.ipr
+*.iws

.vscode/launch.json

@@ -0,0 +1,15 @@
+{
+    // Use IntelliSense to learn about possible attributes.
+    // Hover to view descriptions of existing attributes.
+    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
+    "version": "0.2.0",
+    "configurations": [
+        {
+            "name": "Python: Current File",
+            "type": "python",
+            "request": "launch",
+            "program": "${file}",
+            "console": "integratedTerminal"
+        }
+    ]
+}

.vscode/settings.json

@@ -0,0 +1,6 @@
+{
+    "python.linting.pylintEnabled": false,
+    "python.linting.pep8Enabled": false,
+    "python.linting.enabled": true,
+    "python.linting.flake8Enabled": true
+}

AIRFLOW101.md

@@ -0,0 +1,130 @@
+# Airflow 101
+
+## Our Environment
+
+Our analytics pipeline runs on open source [Apache Airflow](http://airflow.apache.org/tutorial.html) which is written in Python. This means we can deploy it to other clouds or inhouse if we need.
+
+We have some special configuration:
+- using latest beta version composer-1.6.1-airflow-1.10.1
+- using Python 3
+- google-api-python-client, tablib, python-igraph, plotly pypi packages preinstalled
+- slack_default and bigquery_default connection
+- special docker image that includes google cloud SDK and R with tidyverse/ggplot2
+- sendgrid enabled to allow warning and other messages to be sent.
+
+## Getting Started 
+To access "Cloud Composer" (the Google branding for Airflow), visit https://console.cloud.google.com/composer/environments
+From this page you can access the Airflow webserver and the DAGs folder
+
+Read https://cloud.google.com/composer/docs/ for more information.
+
+## How to write a new workflow
+
+### DAGs
+Each pipeline is defined in a DAG file. (A Directed Acyclical Graph is a graph describing a process that goes step by step forward only with no infinite recursion or "cycles".)
+DAG files are technically Python code but use some special keywords and operators to describe data processes. Each pipeline can have a schedule and a SLA (maximum expected run time).
+
+DAG files are made up of Tasks that run Operators and can draw from Connections (via Hooks) and Variables. Definitions @ http://airflow.apache.org/concepts.html
+
+Tutorials http://airflow.apache.org/tutorial.html and https://cloud.google.com/composer/docs/how-to/using/writing-dags and https://cloud.google.com/blog/products/gcp/how-to-aggregate-data-for-bigquery-using-apache-airflow
+
+Tips for designing a workflow: https://en.wikipedia.org/wiki/SOLID
+
+### Header 
+
+
+Set email_on_failure to True to send an email notification when an operator in the DAG fails. 
+```
+default_dag_args = {
+
+    # Email whenever an Operator in the DAG fails.
+    'email': models.Variable.get('email'),
+    'email_on_failure': True,
+    'email_on_retry': False,
+    'retries': 1,
+    'retry_delay': datetime.timedelta(minutes=5)
+}
+```
+
+Schedule, start date and SLA
+
+**WARNING: if the start date is in the past, it will try to catch up running jobs for the schedule period (eg. daily) the first time the DAG is loaded **
+
+If the task takes longer than the SLA, an alert email is triggered.
+
+```
+with models.DAG(
+        'ga_quarterly_reporter',
+        schedule_interval=datetime.timedelta(days=90),
+        sla=datetime.timedelta(hours=1),
+        default_args=default_dag_args) as dag:
+```
+
+ ### Variables
+
+ Variables are configured via the webserver under Admin -> Variables. A variable can be a string, Python list/array or Python dict.
+ The second parameter of the .get() function is the default value if the variable isn't found.
+ You can use variables in the python string formatting functions https://docs.python.org/3/library/string.html#formatexamples 
+```
+from airflow import models
+
+'dataflow_default_options': {
+    'project': models.Variable.get('GCP_PROJECT','dta-ga-bigquery'),
+    'tempLocation': 'gs://staging.%s.appspot.com/' % models.Variable.get('GCP_PROJECT','dta-ga-bigquery')
+ }
+```
+
+### Operators
+
+Full listing at http://airflow.apache.org/_api/airflow/operators/index.html and http://airflow.apache.org/_api/airflow/contrib/operators/index.html includes operators for Bash scripts, JIRA, S3, SQL databases etc. 
+
+**Our favourite operators:**
+
+- PythonOperator
+http://airflow.apache.org/howto/operator/python.html
+
+- BigQueryOperator and BigQueryToCloudStorageOperator
+
+Our environment automatically has a connection to bigquery so no credentials are needed.
+
+http://airflow.apache.org/_api/airflow/contrib/operators/bigquery_operator/index.html
+
+- KubernetesPodOperator 
+Perfect for running an R script or a Python script that needs system packages like chart/graph rendering.
+
+We run a custom docker image with extra R packages described in docker/Dockerfile
+
+https://airflow.apache.org/_api/airflow/contrib/operators/kubernetes_pod_operator/index.html#airflow.contrib.operators.kubernetes_pod_operator.KubernetesPodOperator
+
+**Honorable Mentions:**
+
+- DataFlowOperator
+
+uses the google cloud branded implementation of Apache Beam, another 
+- SlackWebHookOperator
+
+- EmailOperator
+Gmail seems to take 5 or 6 minutes to virus scan attachments before they appear.
+
+
+## Dependencies and Deployment
+
+At the end of the file, in the indented "with DAG:" section you can define dependencies between operators (else they will all run concurrently):
+```
+A >> B >> C
+
+or 
+A >> B
+B >> C
+
+or 
+
+A >> B
+C << B
+
+```
+
+Once you have a DAG, can drag drop it into the folder via the web browser and soon it will be visible in the webserver. When updating a DAG, there is also a Refresh (recycling icon) button.
+You can either trigger the whole DAG or "clear" a task to make that task and all dependent tasks be retried.
+
+Once it is good, check it into Git!

README.md

@@ -1,130 +1,2 @@
-# Airflow 101
-
-## Our Environment
-
-Our analytics pipeline runs on open source [Apache Airflow](http://airflow.apache.org/tutorial.html) which is written in Python. This means we can deploy it to other clouds or inhouse if we need.
-
-We have some special configuration:
-- using latest beta version composer-1.6.1-airflow-1.10.1
-- using Python 3
-- google-api-python-client, tablib, python-igraph, plotly pypi packages preinstalled
-- slack_default and bigquery_default connection
-- special docker image that includes google cloud SDK and R with tidyverse/ggplot2
-- sendgrid enabled to allow warning and other messages to be sent.
-
-## Getting Started 
-To access "Cloud Composer" (the Google branding for Airflow), visit https://console.cloud.google.com/composer/environments
-From this page you can access the Airflow webserver and the DAGs folder
-
-Read https://cloud.google.com/composer/docs/ for more information.
-
-## How to write a new workflow
-
-### DAGs
-Each pipeline is defined in a DAG file. (A Directed Acyclical Graph is a graph describing a process that goes step by step forward only with no infinite recursion or "cycles".)
-DAG files are technically Python code but use some special keywords and operators to describe data processes. Each pipeline can have a schedule and a SLA (maximum expected run time).
-
-DAG files are made up of Tasks that run Operators and can draw from Connections (via Hooks) and Variables. Definitions @ http://airflow.apache.org/concepts.html
-
-Tutorials http://airflow.apache.org/tutorial.html and https://cloud.google.com/composer/docs/how-to/using/writing-dags and https://cloud.google.com/blog/products/gcp/how-to-aggregate-data-for-bigquery-using-apache-airflow
-
-Tips for designing a workflow: https://en.wikipedia.org/wiki/SOLID
-
-### Header 
-
-
-Set email_on_failure to True to send an email notification when an operator in the DAG fails. 
-```
-default_dag_args = {
-
-    # Email whenever an Operator in the DAG fails.
-    'email': models.Variable.get('email'),
-    'email_on_failure': True,
-    'email_on_retry': False,
-    'retries': 1,
-    'retry_delay': datetime.timedelta(minutes=5)
-}
-```
-
-Schedule, start date and SLA
-
-**WARNING: if the start date is in the past, it will try to catch up running jobs for the schedule period (eg. daily) the first time the DAG is loaded **
-
-If the task takes longer than the SLA, an alert email is triggered.
-
-```
-with models.DAG(
-        'ga_quarterly_reporter',
-        schedule_interval=datetime.timedelta(days=90),
-        sla=datetime.timedelta(hours=1),
-        default_args=default_dag_args) as dag:
-```
-
- ### Variables
-
- Variables are configured via the webserver under Admin -> Variables. A variable can be a string, Python list/array or Python dict.
- The second parameter of the .get() function is the default value if the variable isn't found.
- You can use variables in the python string formatting functions https://docs.python.org/3/library/string.html#formatexamples 
-```
-from airflow import models
-
-'dataflow_default_options': {
-    'project': models.Variable.get('GCP_PROJECT','dta-ga-bigquery'),
-    'tempLocation': 'gs://staging.%s.appspot.com/' % models.Variable.get('GCP_PROJECT','dta-ga-bigquery')
- }
-```
-
-### Operators
-
-Full listing at http://airflow.apache.org/_api/airflow/operators/index.html and http://airflow.apache.org/_api/airflow/contrib/operators/index.html includes operators for Bash scripts, JIRA, S3, SQL databases etc. 
-
-**Our favourite operators:**
-
-- PythonOperator
-http://airflow.apache.org/howto/operator/python.html
-
-- BigQueryOperator and BigQueryToCloudStorageOperator
-
-Our environment automatically has a connection to bigquery so no credentials are needed.
-
-http://airflow.apache.org/_api/airflow/contrib/operators/bigquery_operator/index.html
-
-- KubernetesPodOperator 
-Perfect for running an R script or a Python script that needs system packages like chart/graph rendering.
-
-We run a custom docker image with extra R packages described in docker/Dockerfile
-
-https://airflow.apache.org/_api/airflow/contrib/operators/kubernetes_pod_operator/index.html#airflow.contrib.operators.kubernetes_pod_operator.KubernetesPodOperator
-
-**Honorable Mentions:**
-
-- DataFlowOperator
-
-uses the google cloud branded implementation of Apache Beam, another 
-- SlackWebHookOperator
-
-- EmailOperator
-Gmail seems to take 5 or 6 minutes to virus scan attachments before they appear.
-
-
-## Dependencies and Deployment
-
-At the end of the file, in the indented "with DAG:" section you can define dependencies between operators (else they will all run concurrently):
-```
-A >> B >> C
-
-or 
-A >> B
-B >> C
-
-or 
-
-A >> B
-C << B
-
-```
-
-Once you have a DAG, can drag drop it into the folder via the web browser and soon it will be visible in the webserver. When updating a DAG, there is also a Refresh (recycling icon) button.
-You can either trigger the whole DAG or "clear" a task to make that task and all dependent tasks be retried.
-
-Once it is good, check it into Git!
+# Observatory
+To run the HTML version, download augov.gexf from /data on Google Cloud Storage, put it in html/observatory/data and run html/observatory/run.sh

dags/ga_benchmark.py

@@ -14,15 +14,12 @@
     'start_date': yesterday,
 }
 
-
 with models.DAG(
         'ga_benchmark',
         schedule_interval=datetime.timedelta(days=1),
         default_args=default_dag_args) as dag:
-    project_id = models.Variable.get('GCP_PROJECT','dta-ga-bigquery')
+    project_id = models.Variable.get('GCP_PROJECT', 'dta-ga-bigquery')
 
-    view_id = '69211100'
-    timestamp = '20190425'
     temp_table = 'benchmark_%s_%s' % (view_id, timestamp)
     query = """
     CREATE TABLE `{{params.project_id}}.tmp.{{ params.temp_table }}`
@@ -58,6 +55,7 @@
     export_benchmark_to_gcs = bigquery_to_gcs.BigQueryToCloudStorageOperator(
         task_id='export_benchmark_to_gcs',
         source_project_dataset_table="%s.tmp.%s" % (project_id, temp_table),
-        destination_cloud_storage_uris=["gs://us-central1-maxious-airflow-64b78389-bucket/data/%s.csv" % (temp_table,)],
+        destination_cloud_storage_uris=["gs://%s/data/%s.csv" % (
+        models.Variable.get('AIRFLOW_BUCKET', 'us-east1-dta-airflow-b3415db4-bucket'), temp_table)],
         export_format='CSV')
     query_benchmark >> export_benchmark_to_gcs