Google Analytics Extractor

A CLI tool for extracting Google Analytics data using Google Reporting API. Can be also used to transform data to various formats suitable for migration to other analytics platforms.

Also see - Goodbye, Google Analytics - Why and How You Should Leave The Platform for more context.

---

If you find this useful, you can support me on Ko-Fi (Donations are always appreciated, but never required):

Setup

You will need Google Cloud API access for run the CLI:

• Navigate to Cloud Resource Manager and click Create Project

  • alternatively create project with gcloud projects create $PROJECT_ID

• Navigate to Reporting API and click Enable

• Create credentials:

  • Go to credentials page

  • Click Create credentials, select Service account

  • Give it a name and make note of service account email. Click Create and Continue

  • Open Service account page

  • Select previously created service account, Open Keys tab

  • Click Add Key and Create New Key. Choose JSON format and download it. (store this securely)

• Give SA permissions to GA - guide

  • email: SA email from earlier
  • role: Viewer

Alternatively see following setup.

To install and run:

pip install ga-extractor
ga-extractor --help

Running

ga-extractor --help
# Usage: ga-extractor [OPTIONS] COMMAND [ARGS]...
# ...

# Create config file:
ga-extractor setup \
  --sa-key-path="analytics-api-24102021-4edf0b7270c0.json" \
  --table-id="123456789" \
  --metrics="ga:sessions" \
  --dimensions="ga:browser" \
  --start-date="2022-03-15" \
  --end-date="2022-03-19"
  
cat ~/.config/ga-extractor/config.yaml  # Optionally, check config

ga-extractor auth  # Test authentication
# Successfully authenticated with user: ...

ga-extractor setup --help  # For options and flags

• Value for --table-id can be found in GA web console - Click on Admin section, View Settings and see View ID field
• All configurations and generated extracts/reports are stored in ~/.config/ga-extractor/...
• You can also use metrics and dimensions presets using --preset with FULL or BASIC, if you're not sure which data to extract

Extract

ga-extractor extract
# Report written to /home/some-user/.config/ga-extractor/report.json

extract perform raw extraction of dimensions and metrics using the provided configs

Migrate

You can directly extract and transform data to various formats. Available options are:

• JSON (Default option; Default API output)
• CSV
• SQL (compatible with Umami Analytics PostgreSQL backend)

ga-extractor migrate --format=CSV
# Report written to /home/user/.config/ga-extractor/02c2db1a-1ff0-47af-bad3-9c8bc51c1d13_extract.csv

head /home/user/.config/ga-extractor/02c2db1a-1ff0-47af-bad3-9c8bc51c1d13_extract.csv
# path,browser,os,device,screen,language,country,referral_path,count,date
# /,Chrome,Android,mobile,1370x1370,zh-cn,China,(direct),1,2022-03-18
# /,Chrome,Android,mobile,340x620,en-gb,United Kingdom,t.co/,1,2022-03-18

ga-extractor migrate --format=UMAMI
# Report written to /home/user/.config/ga-extractor/cee9e1d0-3b87-4052-a295-1b7224c5ba78_extract.sql

# IMPORTANT: Verify the data and check test database before inserting into production instance 
# To insert into DB (This should be run against clean database):
cat cee9e1d0-3b87-4052-a295-1b7224c5ba78_extract.sql | psql -Upostgres -a some-db

You can verify the data is correct in Umami web console and GA web console:

• Umami extract
• GA Pageviews

Note: Some data in GA and Umami web console might be little off, because GA displays many metrics based on sessions (e.g. Sessions by device), but data is extracted/migrated based on page views. You can however confirm that percentage breakdown of browser or OS usage does match.

Development

Setup

Requirements:

• Poetry (+ virtual environment)

poetry install
python -m ga_extractor --help

Testing

pytest

Building Package

poetry install
ga-extractor --help

# Usage: ga-extractor [OPTIONS] COMMAND [ARGS]...
# ...