AIRFLOW_MLFLOW_DOCKER

Table of content

• Background
• Tools Overview
• Getting started
  • Docker Compose configuration
  • Airflow
  • MLflow
• Connect Airflow to MLflow
• References

Background

The goal of this project is to create an ecosystem where to run Data Pipelines and monitor Machine Learning Experiments.

Tools Overview

From Airflow documentation:

Apache Airflow is an open-source platform for developing, scheduling, and monitoring batch-oriented workflows

From MLflow documentation:

MLflow is an open source platform for managing the end-to-end machine learning lifecycle

From Docker documentation:

Docker Compose is a tool for defining and running multi-container Docker applications.

Getting Started

The first step to structure this project is connecting Airflow and MLflow together: docker compose.

Docker Compose Configuration

Create docker-compose.yaml, which contains the configuration of those docker containers responsible for running Airflow and MLflow services. Each of those services runs on a different container:

• airflow-webserver
• airflow-scheduler
• airflow-worker
• airflow-triggerer
• mlflow

To create and start multiple container, from terminal run the following command:

docker compose up -d

Airflow

In order to access to Airflow server visit the page: localhost:8080

And take a step into Airflow world!

To start creating DAGS initialize an empty folder named dags and populate it with as many scripts as you need.

└── dags
     └── example_dag.py

MLFlow

In order to monitor MLflow experiments through its server, visit the page: localhost:600

Connect Airflow to MLflow

To establish a connection between Airflow and MLflow, define the URI of the MLflow server:

mlflow.set_tracking_uri('http://mlflow:600')

After that, create a new connection on Airflow that points to that port.

References

• Airflow Docker
• What is Airflow?
• MLflow