Skip to content

Latest commit

 

History

History

05-monitoring

README.md

5. Model Monitoring

5.1 Intro to ML monitoring

5.2 Environment setup

5.3 Prepare reference and model

5.4 Evidently metrics calculation

5.5 Dummy monitoring

5.6 Data quality monitoring

5.7 Save Grafana Dashboard

5.8 Debugging with test suites and reports

5.8 Homework

More information here

Notes

Did you take notes? Add them here:

Monitoring example

Prerequisites

You need following tools installed:

  • docker
  • docker-compose (included to Docker Desktop for Mac and Docker Desktop for Windows )

Preparation

Note: all actions expected to be executed in repo folder.

  • Create virtual environment and activate it (eg. python -m venv venv && source ./venv/bin/activate or conda create -n venv python=3.11 && conda activate venv)
  • Install required packages pip install -r requirements.txt
  • Run baseline_model_nyc_taxi_data.ipynb for downloading datasets, training model and creating reference dataset

Monitoring Example

Starting services

To start all required services, execute:

docker-compose up

It will start following services:

  • db - PostgreSQL, for storing metrics data
  • adminer - database management tool
  • grafana - Visual dashboarding tool

Sending data

To calculate evidently metrics with prefect and send them to database, execute:

python evidently_metrics_calculation.py

This script will simulate batch monitoring. Every 10 seconds it will collect data for a daily batch, calculate metrics and insert them into database. This metrics will be available in Grafana in preconfigured dashboard.

Accsess dashboard

  • In your browser go to a localhost:3000 The default username and password are admin

  • Then navigate to General/Home menu and click on Home.

  • In the folder General you will see New Dashboard. Click on it to access preconfigured dashboard.

Ad-hoc debugging

Run debugging_nyc_taxi_data.ipynb to see how you can perform a debugging with help of Evidently TestSuites and Reports

Stopping services

To stop all services, execute:

docker-compose down