model_monitoring_stream/model_monitoring_stream.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "# Model Monitoring\n",
    "\n",
    "## Initial set up (and pre-requisites)\n",
    "1. Make sure you have the `mlrun-api` datasource available in your Grafana instance, otherwise add it by:\n",
    "   1. Open your grafana instance\n",
    "   2. Navigate to `Configuration -> Data Sources`\n",
    "   3. Press `Add data source` and configure the following parameters\n",
    "    ```\n",
    "    Name: mlrun-api\n",
    "    URL: http://mlrun-api:8080/api/grafana-proxy/model-endpoints\n",
    "    Access: Server (default)\n",
    "\n",
    "    ## Add a custom header of:\n",
    "    X-V3io-Session-Key: <YOUR ACCESS KEY>\n",
    "    ```\n",
    "   4. Press `Save & Test` to make sure it works, a confirmation message should appear when this button is pressed\n",
    "\n",
    "2. Import the available dashboards `(./dashboards/*)` to you Grafana instance\n",
    "3. To allow the system to utilize drift measurement, make sure you supply the train set when logging the model on the\n",
    "   training step\n",
    "\n",
    "    ```python\n",
    "    # Log model\n",
    "    context.log_model(\n",
    "        \"model\",\n",
    "        body=dumps(model),\n",
    "        artifact_path=context.artifact_subpath(\"models\"),\n",
    "        extra_data=eval_metrics,\n",
    "        model_file=\"model.pkl\",\n",
    "        metrics=context.results,\n",
    "        training_set=X_test,  # <- make sure this is passed into log_model\n",
    "        labels={\"class\": \"sklearn.linear_model.LogisticRegression\"}\n",
    "    )\n",
    "    ```\n",
    "4. When serving a model, make sure that the Nuclio function is deployed with tracking enabled by applying\n",
    "   `fn.set_tracking()`\n",
    "\n",
    "## Configuration\n",
    "The stream processing portion of the model monitoring, can be deployed under multiple configuration options. The\n",
    "available configurations can be found under `stream.Config`. Once configured it should be supplied as environment\n",
    "parameters to the Nuclio function by setting `fn.set_envs`\n",
    "\n",
    "```python\n",
    "project: str                        # project name\n",
    "sample_window: int                  # The sampling window for the data that flows into the TSDB and the KV\n",
    "kv_path_template: str               # Path template for the kv table\n",
    "tsdb_path_template: str             # Path template for the tsdb table\n",
    "parquet_path_template: str          # v3io parquets path template, assumes v3io is mounted\n",
    "tsdb_batching_max_events: int       # The max amount of event to batch before writing the batch to tsdb\n",
    "tsdb_batching_timeout_secs: int     # The max amount of seconds a given batch can be gathered before being emitted\n",
    "parquet_batching_max_events: int    # The max amount of event to batch before writing the batch to parquet\n",
    "parquet_batching_timeout_secs: int  # The max amount of seconds, a given batch can be gathered before being written to parquet\n",
    "container: str                      # container name\n",
    "v3io_access_key: str                # V3IO Access key\n",
    "v3io_framesd: str                   # V3IO framesd URL\n",
    "time_format: str                    # The time format into which time related fields will be converted\n",
    "aggregate_count_windows: List[str]  # List of window sizes for predictions count\n",
    "aggregate_count_period: str         # Period of predictions count windows\n",
    "aggregate_avg_windows: List[str]    # List of window sizes for average latency\n",
    "aggregate_avg_period: str           # Period of average latency windows\n",
    "```"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Export function yaml"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "from mlrun import code_to_function\n",
    "from mlrun.runtimes import RemoteRuntime\n",
    "\n",
    "\n",
    "fn: RemoteRuntime = code_to_function(\n",
    "    name=\"model-monitoring-stream\",\n",
    "    kind=\"nuclio\",\n",
    "    image=\"mlrun/mlrun\",\n",
    "    filename=\"model_monitoring_stream.py\",\n",
    "    handler=\"handler\",\n",
    ")\n",
    "fn.export(\"model_monitoring_stream.yaml\")\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Deploy Stream Processing"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "from mlrun import import_function\n",
    "from mlrun.platforms import mount_v3io\n",
    "from mlrun.runtimes import RemoteRuntime\n",
    "import json\n",
    "\n",
    "# Set project name\n",
    "project = \"\"\n",
    "\n",
    "fn: RemoteRuntime = import_function(\"hub://model_monitoring_stream\")\n",
    "\n",
    "fn.add_v3io_stream_trigger(\n",
    "    stream_path=f\"projects/{project}/model-endpoints/stream\",\n",
    "    name=\"monitoring_stream_trigger\",\n",
    ")\n",
    "\n",
    "fn.set_env(\"MODEL_MONITORING_PARAMETERS\", json.dumps({\"project\": project, \"v3io_framesd\": os.environ.get(\"V3IO_FRAMESD\")}))\n",
    "\n",
    "fn.metadata.project = project\n",
    "fn.apply(mount_v3io())\n",
    "fn.deploy()"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}