Add airbyte-dbt-prefect-snowflake quickstart

README.md

@@ -17,6 +17,7 @@ To empower data teams by providing ready-to-use code templates, enabling the swi
 - [Airbyte, dbt, Airflow and BigQuery E-commerce Stack](./airbyte_dbt_airflow_bigquery)
 - [Airbyte, dbt, Dagster and BigQuery Basic Stack](./airbyte_dbt_dagster)
 - [Airbyte, dbt, Dagster and Snowflake Basic Stack](./airbyte_dbt_dagster_snowflake)
+- [Airbyte, dbt, Prefect and Snowflake Basic Stack](./airbyte_dbt_prefect_snowflake)
 - [Airbyte, dbt, Prefect and BigQuery (PAD) Stack](./airbyte_dbt_prefect_bigquery)
 - [API to Data Warehouse Integration Stack](./api_to_warehouse)
 - [Customer Satisfaction Analytics Stack With Zendesk Support, Airbyte, dbt, Dagster and BigQuery](./satisfaction_analytics_zendesk_support)

airbyte_dbt_prefect_snowflake/.gitignore

@@ -0,0 +1,163 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+#poetry.lock
+
+# pdm
+#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
+#pdm.lock
+#   pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
+#   in version control.
+#   https://pdm.fming.dev/#use-with-ide
+.pdm.toml
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# PyCharm
+#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#  and can be added to the global gitignore or merged into this file.  For a more nuclear
+#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
+#.idea/
+
+#Desktop Services Store
+.DS_Store

airbyte_dbt_prefect_snowflake/README.md

@@ -0,0 +1,210 @@
+# Airbyte-dbt-Prefect-Snowflake Integration
+
+Welcome to the "Airbyte-dbt-Prefect-Snowflake Integration" repository! This repo provides a quickstart template for building a full data stack using Airbyte, Prefect, dbt, and Snowflake. Easily extract data from Postgres and load it into Snowflake using Airbyte, and apply necessary transformations using dbt, all orchestrated seamlessly with Prefect. While this template doesn't delve into specific data or transformations, its goal is to showcase the synergy of these tools.
+
+This quickstart is designed to minimize setup hassles and propel you forward.
+
+## Table of Contents
+
+- [Airbyte-dbt-Prefect-Snowflake Integration](#airbyte-dbt-prefect-snowflake-integration)
+  - [Table of Contents](#table-of-contents)
+  - [Infrastructure Layout](#infrastructure-layout)
+  - [Prerequisites](#prerequisites)
+  - [1. Setting an environment for your project](#1-setting-an-environment-for-your-project)
+  - [2. Setting Up Airbyte Connectors with Terraform](#2-setting-up-airbyte-connectors-with-terraform)
+  - [3. Setting Up the dbt Project](#3-setting-up-the-dbt-project)
+  - [4. Orchestrating with Prefect](#4-orchestrating-with-Prefect)
+  - [Next Steps](#next-steps)
+
+## Infrastructure Layout
+![insfrastructure layout](images/infrastructure.png)
+
+## Prerequisites
+
+Before you embark on this integration, ensure you have the following set up and ready:
+
+1. **Python 3.10 or later**: If not installed, download and install it from [Python's official website](https://www.python.org/downloads/).
+
+2. **Docker and Docker Compose (Docker Desktop)**: Install [Docker](https://docs.docker.com/get-docker/) following the official documentation for your specific OS.
+
+3. **Airbyte OSS version**: Deploy the open-source version of Airbyte. Follow the installation instructions from the [Airbyte Documentation](https://docs.airbyte.com/quickstart/deploy-airbyte/).
+
+4. **Terraform**: Terraform will help you provision and manage the Airbyte resources. If you haven't installed it, follow the [official Terraform installation guide](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli).
+
+
+## 1. Setting an environment for your project
+
+Get the project up and running on your local machine by following these steps:
+
+1. **Clone the repository (Clone only this quickstart)**:  
+   ```bash
+   git clone --filter=blob:none --sparse  https://github.com/airbytehq/quickstarts.git
+   ```
+
+   ```bash
+   cd quickstarts
+   ```
+
+   ```bash
+   git sparse-checkout add airbyte_dbt_Prefect_snowflake
+   ```
+
+
+2. **Navigate to the directory**:  
+   ```bash
+   cd airbyte_dbt_prefect_snowflake
+   ```
+
+3. **Set Up a Virtual Environment**:  
+   - For Mac:
+     ```bash
+     python3 -m venv venv
+     source venv/bin/activate
+     ```
+   - For Windows:
+     ```bash
+     python -m venv venv
+     .\venv\Scripts\activate
+     ```
+
+4. **Install Dependencies**:  
+   ```bash
+   pip install -e ".[dev]"
+   ```
+
+## 2. Setting Up Airbyte Connectors with Terraform
+
+Airbyte allows you to create connectors for sources and destinations, facilitating data synchronization between various platforms. In this project, we're harnessing the power of Terraform to automate the creation of these connectors and the connections between them. Here's how you can set this up:
+
+1. **Navigate to the Airbyte Configuration Directory**:
+
+   Change to the relevant directory containing the Terraform configuration for Airbyte:
+   ```bash
+   cd infra/airbyte
+   ```
+
+2. **Modify Configuration Files**:
+
+   Within the `infra/airbyte` directory, you'll find three crucial Terraform files:
+    - `provider.tf`: Defines the Airbyte provider.
+    - `main.tf`: Contains the main configuration for creating Airbyte resources.
+    - `variables.tf`: Holds various variables, including credentials.
+
+   Adjust the configurations in these files to suit your project's needs. Specifically, provide credentials for your Postgres and Snowflake connections. You can utilize the `variables.tf` file to manage these credentials.
+
+3. **Initialize Terraform**:
+
+   This step prepares Terraform to create the resources defined in your configuration files.
+   ```bash
+   terraform init
+   ```
+
+4. **Review the Plan**:
+
+   Before applying any changes, review the plan to understand what Terraform will do.
+   ```bash
+   terraform plan
+   ```
+
+5. **Apply Configuration**:
+
+   After reviewing and confirming the plan, apply the Terraform configurations to create the necessary Airbyte resources.
+   ```bash
+   terraform apply
+   ```
+
+6. **Verify in Airbyte UI**:
+
+   Once Terraform completes its tasks, navigate to the Airbyte UI. Here, you should see your source and destination connectors, as well as the connection between them, set up and ready to go.
+
+## 3. Setting Up the dbt Project
+
+[dbt (data build tool)](https://www.getdbt.com/) allows you to transform your data by writing, documenting, and executing SQL workflows. Setting up the dbt project requires specifying connection details for your data platform, in this case, Snowflake. Here’s a step-by-step guide to help you set this up:
+
+1. **Navigate to the dbt Project Directory**:
+
+   Change to the directory containing the dbt configuration:
+   ```bash
+   cd ../../dbt_project
+   ```
+
+2. **Update Connection Details**:
+
+   You'll find a `profiles.yml` file within the directory. This file contains configurations for dbt to connect with your data platform. Update this file with your Snowflake connection details.
+
+3. **Utilize Environment Variables (Optional but Recommended)**:
+
+   To keep your credentials secure, you can leverage environment variables. An example is provided within the `profiles.yml` file.
+
+4. **Test the Connection**:
+
+   Once you’ve updated the connection details, you can test the connection to your Snowflake instance using:
+   ```bash
+   dbt debug
+   ```
+
+   If everything is set up correctly, this command should report a successful connection to Snowflake.
+
+## 4. Orchestrating with Prefect
+
+[Prefect](https://prefect.io/) is an orchestration workflow tool that makes it easy to build, run, and monitor data workflows by writing Python code. In this section, we'll walk you through creating a Prefect flow to orchestrate both Airbyte extract and load operations, and dbt transformations with Python:
+
+1. **Navigate to the Orchestration Directory**:
+
+   Switch to the directory containing the Prefect orchestration configurations:
+   ```bash
+   cd ../orchestration
+   ```
+
+2. **Set Environment Variables**:
+
+   Prefect requires certain environment variables to be set to interact with other tools like dbt and Airbyte. Set the following variables:
+
+   ```bash
+   export AIRBYTE_PASSWORD=password
+   ```
+
+3. **Connect to Prefect's API**:
+
+   Open a new terminal window. Start a local Prefect server instance in your virtual environment:
+
+   ```bash
+   prefect server start
+   ```
+
+4. **Deploy the Flow**:
+
+   When we run the flow script, Prefect will automatically create a flow deployment that you can interact with via the UI and API. The script will stay running so that it can listen for scheduled or triggered runs of this flow; once a run is found, it will be executed within a subprocess.
+
+   ```bash
+   python my_elt_flow.py
+   ```
+
+5. **Access Prefect UI in Your Browser**:
+
+   Open your browser and navigate to:
+   ```
+   http://127.0.0.1:4200
+   ```
+   You can now begin interacting with your newly created deployment!
+
+## Next Steps
+
+Once you've set up and launched this initial integration, the real power lies in its adaptability and extensibility. Here’s a roadmap to help you customize and harness this project tailored to your specific data needs:
+
+1. **Create dbt Sources for Airbyte Data**:
+
+   - Your raw data extracted via Airbyte can be represented as sources in dbt. Start by [creating new dbt sources](https://docs.getdbt.com/docs/build/sources) to represent this data, allowing for structured transformations down the line.
+
+2. **Add Your dbt Transformations**:
+
+   - With your dbt sources in place, you can now build upon them. Add your custom SQL transformations in dbt, ensuring that you treat the sources as an upstream dependency. This ensures that your transformations work on the most up-to-date raw data.
+
+### 3. **Extend the Prefect Pipeline**:
+
+   - You can create flow runs from this deployment via API calls to be triggered by new data sync in Airbyte rather than on a schedule. You can customize your dbt   runs based on the results got from AirbyteSyncResult. You can also migrate the deployment to the Prefect cloud.
+
+4. **Extend the Project**:
+
+   - The real beauty of this integration is its extensibility. Whether you want to add more data sources, integrate additional tools, or enhance your transformation logic – the floor is yours. With the foundation set, sky's the limit for how you want to extend and refine your data processes.
+

airbyte_dbt_prefect_snowflake/dbt_project/.gitignore

@@ -0,0 +1,10 @@
+
+target/
+dbt_packages/
+logs/
+
+#Desktop Services Store
+.DS_Store
+
+#User cookie
+.user.yml