Skip to content

Google Cloud Function to push json files from GC Storage to Big Query

License

Notifications You must be signed in to change notification settings

tomerr-ma/GCS-To-Big-Query

 
 

Repository files navigation

GCS To Big Query Cloud Function

Google Cloud Function for processing json log files in Google Cloud Storage and creating BigQuery tables.

Deploy from gcloud CLI

Open Google Cloud Shell

curl -LO "https://github.com/cloudflare/GCS-To-Big-Query/archive/master.zip" && unzip master.zip && cd GCS-To-Big-Query-master

Update the environment variables in deploy.sh (e.g. nano deploy.sh)

BUCKET_NAME="" # required – The name of Google Cloud Storage bucket used for Cloudflare Logpush logs.
DATASET="" # optional – BigQuery dataset to write to. Will be created if necessary.
TABLE="" # optional – BigQuery table to write to. Will be created if necessary.

Deploy function

sh ./deploy.sh

Note: You can also deploy this function locally using the same steps below. The only requirement is the gcloud SDK.#Test1234

About

Google Cloud Function to push json files from GC Storage to Big Query

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages

  • JavaScript 79.6%
  • Shell 20.4%