Skip to content

This repository showcases my implementation of an end-to-end ETL pipeline

Notifications You must be signed in to change notification settings

KomnisEvangelos/ETL-Data-Pipeline-on-Google-Cloud-Platform

Folders and files

NameName
Last commit message
Last commit date

Latest commit

1533f63 · May 18, 2024

History

8 Commits
May 18, 2024
May 18, 2024
May 18, 2024
May 18, 2024
May 18, 2024

Repository files navigation

Weather API - ETL Data Pipeline on Google Cloud Platform

This repository showcases my implementation of an end-to-end ETL pipeline, leveraging Google Cloud Platform (GCP) to manage and analyze weather data. This project emphasizes my capabilities in using cloud technologies and data engineering practices to handle real-world data workflows effectively.

Architecture

project acrhitecture

Looker Studio Report

report

Skills Demonstrated

  • API Interaction: Automated data extraction from OpenWeather API.
  • Data Manipulation: Utilized Python and Pandas to clean and transform data.
  • Cloud Services: Deployed automated ETL workflows to extract, process, and store data on production using GCP services like Cloud Functions, Cloud Scheduler, BigQuery, and Cloud Storage.
  • Data Visualization: Created an interactive dashboard using Looker Studio.

About

This repository showcases my implementation of an end-to-end ETL pipeline

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages