03-orchestration

3. Orchestration and ML Pipelines

Note: orchestration.py is a ready final version. The rest of the files were worked on together during the video tutorials.

Note With Prefect version 2.2.1 or later DeploymentSpec's are now just Deployment's.

3.1 Negative engineering and workflow orchestration

3.2 Introduction to Prefect 2.0

3.3 First Prefect flow and basics

3.4 Remote Prefect Orion deployment

3.5 Deployment of Prefect flow

Note: There are several changes to deployment in Prefect 2.3.1 since 2.0b8:

• DeploymentSpec in 2.0b8 now becomes Deployment.
• work_queue_name is used instead of tags to submit the deployment to the a specific work queue.
• You don't need to create a work queue before using the work queue. A work queue will be created if it doesn't exist.

from prefect.deployments import Deployment
from prefect.orion.schemas.schedules import IntervalSchedule
from datetime import timedelta

deployment = Deployment.build_from_flow(
    flow=main,
    name="model_training",
    schedule=IntervalSchedule(interval=timedelta(minutes=5)),
    work_queue_name="ml"
)

deployment.apply()

Links:

• Instructions for Hosting Prefect Orion

3.6 MLOps Zoomcamp 3.6 - (Optional) Work queues and agents

3.7 Homework

More information here: homework.md

Notes

Did you take notes? Add them here:

• Week 3, Prefect Introduction and S3 Bucket configuration with Prefect by M. Ayoub C.
• Notes from froukje
• Minimalist code notes from Anna V
• Getting Started on Prefect 2.0 + Deploying worfklows for MLflow Staging by Ron Medina (Jupyter Book)
• Quickstart your homework by Zioalex
• Notes from Maxime M
• Week3: Prefect introduction and homework notes by Bhagabat
• Week 3: Orchestration notes by Ayoub.B
• Send a PR, add your notes above this line