README.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MLFlow and Seldon\n",
    "\n",
    "End to end example integrating MLFlow and Seldon, with A/B testing of the models.\n",
    "The slides accompanying this demo can be [found here](https://docs.google.com/presentation/d/1QXiOZkd_XNw6PbUalhYDajljKYQjgKczzNncTyLk9uA/edit?usp=sharing)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pre-requisites\n",
    "\n",
    "### Python\n",
    "\n",
    "The training part of the example assumes that you are able to run `mlflow` on your local environment.\n",
    "To set it up, you can run:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: mlflow in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from -r requirements.txt (line 1)) (1.11.0)\n",
      "Requirement already satisfied: jupyter in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from -r requirements.txt (line 2)) (1.0.0)\n",
      "Requirement already satisfied: pandas in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from -r requirements.txt (line 3)) (1.1.3)\n",
      "Requirement already satisfied: numpy in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from -r requirements.txt (line 4)) (1.19.2)\n",
      "Requirement already satisfied: conda in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from -r requirements.txt (line 5)) (4.3.16)\n",
      "Requirement already satisfied: entrypoints in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from mlflow->-r requirements.txt (line 1)) (0.3)\n",
      "Requirement already satisfied: protobuf>=3.6.0 in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from mlflow->-r requirements.txt (line 1)) (3.13.0)\n",
      "Requirement already satisfied: Flask in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from mlflow->-r requirements.txt (line 1)) (1.1.2)\n",
      "Requirement already satisfied: six>=1.10.0 in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from mlflow->-r requirements.txt (line 1)) (1.15.0)\n",
      "Requirement already satisfied: pyyaml in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from mlflow->-r requirements.txt (line 1)) (5.3.1)\n",
      "Requirement already satisfied: requests>=2.17.3 in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from mlflow->-r requirements.txt (line 1)) (2.24.0)\n",
      "Requirement already satisfied: prometheus-flask-exporter in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from mlflow->-r requirements.txt (line 1)) (0.18.1)\n",
      "Requirement already satisfied: azure-storage-blob>=12.0 in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from mlflow->-r requirements.txt (line 1)) (12.5.0)\n",
      "Requirement already satisfied: sqlalchemy<=1.3.13 in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from mlflow->-r requirements.txt (line 1)) (1.3.13)\n",
      "Requirement already satisfied: sqlparse in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from mlflow->-r requirements.txt (line 1)) (0.4.1)\n",
      "Requirement already satisfied: python-dateutil in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from mlflow->-r requirements.txt (line 1)) (2.8.1)\n",
      "Requirement already satisfied: alembic<=1.4.1 in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from mlflow->-r requirements.txt (line 1)) (1.4.1)\n",
      "Requirement already satisfied: docker>=4.0.0 in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from mlflow->-r requirements.txt (line 1)) (4.3.1)\n",
      "Requirement already satisfied: click>=7.0 in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from mlflow->-r requirements.txt (line 1)) (7.1.2)\n",
      "Requirement already satisfied: gunicorn; platform_system != \"Windows\" in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from mlflow->-r requirements.txt (line 1)) (20.0.4)\n",
      "Requirement already satisfied: querystring-parser in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from mlflow->-r requirements.txt (line 1)) (1.2.4)\n",
      "Requirement already satisfied: databricks-cli>=0.8.7 in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from mlflow->-r requirements.txt (line 1)) (0.13.0)\n",
      "Requirement already satisfied: gorilla in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from mlflow->-r requirements.txt (line 1)) (0.3.0)\n",
      "Requirement already satisfied: cloudpickle in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from mlflow->-r requirements.txt (line 1)) (1.6.0)\n",
      "Requirement already satisfied: gitpython>=2.1.0 in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from mlflow->-r requirements.txt (line 1)) (3.1.11)\n",
      "Requirement already satisfied: notebook in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from jupyter->-r requirements.txt (line 2)) (6.1.4)\n",
      "Requirement already satisfied: qtconsole in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from jupyter->-r requirements.txt (line 2)) (4.7.7)\n",
      "Requirement already satisfied: jupyter-console in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from jupyter->-r requirements.txt (line 2)) (6.2.0)\n",
      "Requirement already satisfied: nbconvert in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from jupyter->-r requirements.txt (line 2)) (6.0.7)\n",
      "Requirement already satisfied: ipykernel in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from jupyter->-r requirements.txt (line 2)) (5.3.4)\n",
      "Requirement already satisfied: ipywidgets in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from jupyter->-r requirements.txt (line 2)) (7.5.1)\n",
      "Requirement already satisfied: pytz>=2017.2 in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from pandas->-r requirements.txt (line 3)) (2020.1)\n",
      "Requirement already satisfied: ruamel.yaml>=0.11.14 in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from conda->-r requirements.txt (line 5)) (0.16.12)\n",
      "Requirement already satisfied: pycosat>=0.6.1 in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from conda->-r requirements.txt (line 5)) (0.6.3)\n",
      "Requirement already satisfied: setuptools in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from protobuf>=3.6.0->mlflow->-r requirements.txt (line 1)) (50.2.0)\n",
      "Requirement already satisfied: Jinja2>=2.10.1 in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from Flask->mlflow->-r requirements.txt (line 1)) (2.11.2)\n",
      "Requirement already satisfied: itsdangerous>=0.24 in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from Flask->mlflow->-r requirements.txt (line 1)) (1.1.0)\n",
      "Requirement already satisfied: Werkzeug>=0.15 in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from Flask->mlflow->-r requirements.txt (line 1)) (1.0.1)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from requests>=2.17.3->mlflow->-r requirements.txt (line 1)) (2020.6.20)\n",
      "Requirement already satisfied: idna<3,>=2.5 in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from requests>=2.17.3->mlflow->-r requirements.txt (line 1)) (2.10)\n",
      "Requirement already satisfied: chardet<4,>=3.0.2 in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from requests>=2.17.3->mlflow->-r requirements.txt (line 1)) (3.0.4)\n",
      "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from requests>=2.17.3->mlflow->-r requirements.txt (line 1)) (1.25.11)\n",
      "Requirement already satisfied: prometheus-client in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from prometheus-flask-exporter->mlflow->-r requirements.txt (line 1)) (0.8.0)\n",
      "Requirement already satisfied: msrest>=0.6.10 in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from azure-storage-blob>=12.0->mlflow->-r requirements.txt (line 1)) (0.6.19)\n",
      "Requirement already satisfied: cryptography>=2.1.4 in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from azure-storage-blob>=12.0->mlflow->-r requirements.txt (line 1)) (3.2)\n",
      "Requirement already satisfied: azure-core<2.0.0,>=1.6.0 in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from azure-storage-blob>=12.0->mlflow->-r requirements.txt (line 1)) (1.8.2)\n",
      "Requirement already satisfied: Mako in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from alembic<=1.4.1->mlflow->-r requirements.txt (line 1)) (1.1.3)\n",
      "Requirement already satisfied: python-editor>=0.3 in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from alembic<=1.4.1->mlflow->-r requirements.txt (line 1)) (1.0.4)\n",
      "Requirement already satisfied: websocket-client>=0.32.0 in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from docker>=4.0.0->mlflow->-r requirements.txt (line 1)) (0.57.0)\n",
      "Requirement already satisfied: tabulate>=0.7.7 in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from databricks-cli>=0.8.7->mlflow->-r requirements.txt (line 1)) (0.8.7)\n",
      "Requirement already satisfied: gitdb<5,>=4.0.1 in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from gitpython>=2.1.0->mlflow->-r requirements.txt (line 1)) (4.0.5)\n",
      "Requirement already satisfied: tornado>=5.0 in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from notebook->jupyter->-r requirements.txt (line 2)) (6.0.4)\n",
      "Requirement already satisfied: traitlets>=4.2.1 in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from notebook->jupyter->-r requirements.txt (line 2)) (5.0.5)\n",
      "Requirement already satisfied: pyzmq>=17 in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from notebook->jupyter->-r requirements.txt (line 2)) (19.0.2)\n",
      "Requirement already satisfied: nbformat in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from notebook->jupyter->-r requirements.txt (line 2)) (5.0.8)\n",
      "Requirement already satisfied: Send2Trash in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from notebook->jupyter->-r requirements.txt (line 2)) (1.5.0)\n",
      "Requirement already satisfied: jupyter-client>=5.3.4 in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from notebook->jupyter->-r requirements.txt (line 2)) (6.1.7)\n",
      "Requirement already satisfied: terminado>=0.8.3 in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from notebook->jupyter->-r requirements.txt (line 2)) (0.9.1)\n",
      "Requirement already satisfied: ipython-genutils in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from notebook->jupyter->-r requirements.txt (line 2)) (0.2.0)\n",
      "Requirement already satisfied: jupyter-core>=4.6.1 in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from notebook->jupyter->-r requirements.txt (line 2)) (4.6.3)\n",
      "Requirement already satisfied: argon2-cffi in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from notebook->jupyter->-r requirements.txt (line 2)) (20.1.0)\n",
      "Requirement already satisfied: pygments in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from qtconsole->jupyter->-r requirements.txt (line 2)) (2.7.2)\n",
      "Requirement already satisfied: qtpy in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from qtconsole->jupyter->-r requirements.txt (line 2)) (1.9.0)\n",
      "Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from jupyter-console->jupyter->-r requirements.txt (line 2)) (3.0.8)\n",
      "Requirement already satisfied: ipython in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from jupyter-console->jupyter->-r requirements.txt (line 2)) (7.18.1)\n",
      "Requirement already satisfied: nbclient<0.6.0,>=0.5.0 in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from nbconvert->jupyter->-r requirements.txt (line 2)) (0.5.1)\n",
      "Requirement already satisfied: testpath in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from nbconvert->jupyter->-r requirements.txt (line 2)) (0.4.4)\n",
      "Requirement already satisfied: mistune<2,>=0.8.1 in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from nbconvert->jupyter->-r requirements.txt (line 2)) (0.8.4)\n",
      "Requirement already satisfied: jupyterlab-pygments in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from nbconvert->jupyter->-r requirements.txt (line 2)) (0.1.2)\n",
      "Requirement already satisfied: pandocfilters>=1.4.1 in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from nbconvert->jupyter->-r requirements.txt (line 2)) (1.4.3)\n",
      "Requirement already satisfied: defusedxml in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from nbconvert->jupyter->-r requirements.txt (line 2)) (0.6.0)\n",
      "Requirement already satisfied: bleach in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from nbconvert->jupyter->-r requirements.txt (line 2)) (3.2.1)\n",
      "Requirement already satisfied: widgetsnbextension~=3.5.0 in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from ipywidgets->jupyter->-r requirements.txt (line 2)) (3.5.1)\n",
      "Requirement already satisfied: ruamel.yaml.clib>=0.1.2; platform_python_implementation == \"CPython\" and python_version < \"3.9\" in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from ruamel.yaml>=0.11.14->conda->-r requirements.txt (line 5)) (0.2.2)\n",
      "Requirement already satisfied: MarkupSafe>=0.23 in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from Jinja2>=2.10.1->Flask->mlflow->-r requirements.txt (line 1)) (1.1.1)\n",
      "Requirement already satisfied: requests-oauthlib>=0.5.0 in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from msrest>=0.6.10->azure-storage-blob>=12.0->mlflow->-r requirements.txt (line 1)) (1.3.0)\n",
      "Requirement already satisfied: isodate>=0.6.0 in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from msrest>=0.6.10->azure-storage-blob>=12.0->mlflow->-r requirements.txt (line 1)) (0.6.0)\n",
      "Requirement already satisfied: cffi!=1.11.3,>=1.8 in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from cryptography>=2.1.4->azure-storage-blob>=12.0->mlflow->-r requirements.txt (line 1)) (1.14.3)\n",
      "Requirement already satisfied: smmap<4,>=3.0.1 in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from gitdb<5,>=4.0.1->gitpython>=2.1.0->mlflow->-r requirements.txt (line 1)) (3.0.4)\n",
      "Requirement already satisfied: jsonschema!=2.5.0,>=2.4 in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from nbformat->notebook->jupyter->-r requirements.txt (line 2)) (3.2.0)\n",
      "Requirement already satisfied: ptyprocess; os_name != \"nt\" in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from terminado>=0.8.3->notebook->jupyter->-r requirements.txt (line 2)) (0.6.0)\n",
      "Requirement already satisfied: wcwidth in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->jupyter-console->jupyter->-r requirements.txt (line 2)) (0.2.5)\n",
      "Requirement already satisfied: decorator in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from ipython->jupyter-console->jupyter->-r requirements.txt (line 2)) (4.4.2)\n",
      "Requirement already satisfied: pexpect>4.3; sys_platform != \"win32\" in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from ipython->jupyter-console->jupyter->-r requirements.txt (line 2)) (4.8.0)\n",
      "Requirement already satisfied: pickleshare in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from ipython->jupyter-console->jupyter->-r requirements.txt (line 2)) (0.7.5)\n",
      "Requirement already satisfied: backcall in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from ipython->jupyter-console->jupyter->-r requirements.txt (line 2)) (0.2.0)\n",
      "Requirement already satisfied: jedi>=0.10 in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from ipython->jupyter-console->jupyter->-r requirements.txt (line 2)) (0.17.2)\n",
      "Requirement already satisfied: async-generator in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from nbclient<0.6.0,>=0.5.0->nbconvert->jupyter->-r requirements.txt (line 2)) (1.10)\n",
      "Requirement already satisfied: nest-asyncio in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from nbclient<0.6.0,>=0.5.0->nbconvert->jupyter->-r requirements.txt (line 2)) (1.4.2)\n",
      "Requirement already satisfied: packaging in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from bleach->nbconvert->jupyter->-r requirements.txt (line 2)) (20.4)\n",
      "Requirement already satisfied: webencodings in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from bleach->nbconvert->jupyter->-r requirements.txt (line 2)) (0.5.1)\n",
      "Requirement already satisfied: oauthlib>=3.0.0 in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from requests-oauthlib>=0.5.0->msrest>=0.6.10->azure-storage-blob>=12.0->mlflow->-r requirements.txt (line 1)) (3.1.0)\n",
      "Requirement already satisfied: pycparser in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from cffi!=1.11.3,>=1.8->cryptography>=2.1.4->azure-storage-blob>=12.0->mlflow->-r requirements.txt (line 1)) (2.20)\n",
      "Requirement already satisfied: pyrsistent>=0.14.0 in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from jsonschema!=2.5.0,>=2.4->nbformat->notebook->jupyter->-r requirements.txt (line 2)) (0.17.3)\n",
      "Requirement already satisfied: attrs>=17.4.0 in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from jsonschema!=2.5.0,>=2.4->nbformat->notebook->jupyter->-r requirements.txt (line 2)) (20.2.0)\n",
      "Requirement already satisfied: importlib-metadata; python_version < \"3.8\" in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from jsonschema!=2.5.0,>=2.4->nbformat->notebook->jupyter->-r requirements.txt (line 2)) (2.0.0)\n",
      "Requirement already satisfied: parso<0.8.0,>=0.7.0 in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from jedi>=0.10->ipython->jupyter-console->jupyter->-r requirements.txt (line 2)) (0.7.1)\n",
      "Requirement already satisfied: pyparsing>=2.0.2 in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from packaging->bleach->nbconvert->jupyter->-r requirements.txt (line 2)) (2.4.7)\n",
      "Requirement already satisfied: zipp>=0.5 in /home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages (from importlib-metadata; python_version < \"3.8\"->jsonschema!=2.5.0,>=2.4->nbformat->notebook->jupyter->-r requirements.txt (line 2)) (3.4.0)\n",
      "\u001b[33mWARNING: You are using pip version 20.1.1; however, version 20.2.4 is available.\n",
      "You should consider upgrading via the '/home/agm/.virtualenvs/mlflow-talk/bin/python -m pip install --upgrade pip' command.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!pip install -r requirements.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Kubernetes\n",
    "\n",
    "The serving side of the example assumes that you've got access to a Kubernetes cluster where Seldon Core is installed.\n",
    "If you don't have access to a local cluster, feel free to use [`kind`](https://kind.sigs.k8s.io/).\n",
    "\n",
    "For instructions on how to install Seldon Core, please check their [setup docs](https://docs.seldon.io/projects/seldon-core/en/latest/workflow/install.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Analytics\n",
    "\n",
    "Additionally, after we deploy the models, we will compare their performance using Seldon Core's integration with Prometheus and Grafana.\n",
    "For that part to work, we will need to install Prometheus and Grafana.\n",
    "\n",
    "To speed things up, we can do this through the [`seldon-core-analytics` chart](https://docs.seldon.io/projects/seldon-core/en/latest/charts/seldon-core-analytics.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME: seldon-core-analytics\n",
      "LAST DEPLOYED: Mon Oct 26 17:46:16 2020\n",
      "NAMESPACE: seldon-system\n",
      "STATUS: deployed\n",
      "REVISION: 1\n"
     ]
    }
   ],
   "source": [
    "!helm install seldon-core-analytics \\\n",
    "    seldon-core-analytics \\\n",
    "    --namespace seldon-system \\\n",
    "    --repo https://storage.googleapis.com/seldon-charts \\\n",
    "    --set grafana.adminPassword=password \\\n",
    "    --set grafana.adminUser=admin"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "toc-hr-collapsed": false
   },
   "source": [
    "## Training\n",
    "\n",
    "This first section will cover how to train models using MLFlow."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### MLflow Project\n",
    "\n",
    "The MLproject file defines:\n",
    "- The environment where the training runs.\n",
    "- The hyperparameters that can be tweaked. In our case, these are $\\{\\alpha, l_{1}\\}$.\n",
    "- The interface to train the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting ./training/MLproject\n"
     ]
    }
   ],
   "source": [
    "%%writefile ./training/MLproject\n",
    "name: mlflow-talk\n",
    "\n",
    "conda_env: conda.yaml\n",
    "\n",
    "entry_points:\n",
    "  main:\n",
    "    parameters:\n",
    "      alpha: float\n",
    "      l1_ratio: {type: float, default: 0.1}\n",
    "    command: \"python train.py {alpha} {l1_ratio}\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This allows us to have a single command to train the model. \n",
    "\n",
    "``` bash\n",
    "$ mlflow run ./training -P alpha=... -P l1_ratio=...\n",
    "```\n",
    "\n",
    "For our example, we will train two versions of the model, which we'll later compare using A/B testing.\n",
    "\n",
    "- $M_{1}$ with $\\alpha = 0.5$\n",
    "- $M_{2}$ with $\\alpha = 0.75$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020/10/27 12:54:06 INFO mlflow.projects.utils: === Created directory /tmp/tmpk8nx2_0w for downloading remote URIs passed to arguments of type 'path' ===\n",
      "2020/10/27 12:54:06 INFO mlflow.projects.backend.local: === Running command 'source /opt/miniconda3/bin/../etc/profile.d/conda.sh && conda activate mlflow-8dcff9dffe20ef7df93f638f77ce4b6257ec7b18 1>&2 && python train.py 0.1 0.1' in run with ID 'c047eddcdc2d4a08963f08516fd18d74' === \n",
      "Elasticnet model (alpha=0.100000, l1_ratio=0.100000):\n",
      "  RMSE: 0.7792546522251949\n",
      "  MAE: 0.6112547988118587\n",
      "  R2: 0.2157063843066196\n",
      "2020/10/27 12:54:08 INFO mlflow.projects: === Run (ID 'c047eddcdc2d4a08963f08516fd18d74') succeeded ===\n"
     ]
    }
   ],
   "source": [
    "!mlflow run ./training -P alpha=0.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020/10/27 12:54:12 INFO mlflow.projects.utils: === Created directory /tmp/tmp82tamhfc for downloading remote URIs passed to arguments of type 'path' ===\n",
      "2020/10/27 12:54:12 INFO mlflow.projects.backend.local: === Running command 'source /opt/miniconda3/bin/../etc/profile.d/conda.sh && conda activate mlflow-8dcff9dffe20ef7df93f638f77ce4b6257ec7b18 1>&2 && python train.py 1.0 0.1' in run with ID '1aab766b2d9246ed85f8dbfec4e8743d' === \n",
      "Elasticnet model (alpha=1.000000, l1_ratio=0.100000):\n",
      "  RMSE: 0.8107373707184711\n",
      "  MAE: 0.6241295925236751\n",
      "  R2: 0.15105362812007328\n",
      "2020/10/27 12:54:13 INFO mlflow.projects: === Run (ID '1aab766b2d9246ed85f8dbfec4e8743d') succeeded ===\n"
     ]
    }
   ],
   "source": [
    "!mlflow run ./training -P alpha=1.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### MLflow Tracking\n",
    "\n",
    "The `train.py` script uses the `mlflow.log_param()` and `mlflow.log_metric()` commands to track each experiment. These are part of the `MLtrack` API, which tracks experiments parameters and results. These can be stored on a remote server, which can then be shared across the entire team. However, on our example we will store these locally on a `mlruns` folder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "06d9317c0c6945f5a0e57321acb5385b  9735daaa839b4934abeaf2768fc1be93\n",
      "169119a7fe1e4b31a746e891499552b0  9cf21e1060974725936e4489ce75c640\n",
      "182d08b34ff042c0b5f572dfa2e754db  ed0705646b8e40008cbfe440d74e4fb2\n",
      "25396a9f87fd42bfadf98fb1802917c9  ef3d34fa48ac4ac29a85b1d6bd7236d0\n",
      "373c06380a55446caa7ced6c5b5a1b96  f34e5dd1221b4d6da41268c8cf1691ff\n",
      "45631cca7b8c4200aaee8b84a3c1f03d  meta.yaml\n",
      "4f7b5bfa0db7404d915d6429902bd1c2  tags\n",
      "5a6be5a1ef844783a50a6577745dbdc3\n"
     ]
    }
   ],
   "source": [
    "!ls mlruns/0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also run `mlflow ui` to show these visually. This will start the MLflow server in http://localhost:5000.\n",
    "\n",
    "```bash\n",
    "$ mlflow ui\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![MLFlow UI](./images/mlflow-ui.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### MLflow Model\n",
    "\n",
    "The `MLmodel` file allows us to version and share models easily. Below we can see an example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "artifact_path: model\n",
      "flavors:\n",
      "  python_function:\n",
      "    data: model.pkl\n",
      "    env: conda.yaml\n",
      "    loader_module: mlflow.sklearn\n",
      "    python_version: 3.6.9\n",
      "  sklearn:\n",
      "    pickled_model: model.pkl\n",
      "    serialization_format: cloudpickle\n",
      "    sklearn_version: 0.19.1\n",
      "run_id: 5a6be5a1ef844783a50a6577745dbdc3\n",
      "utc_time_created: '2019-10-02 14:21:15.783806'\n"
     ]
    }
   ],
   "source": [
    "!cat ./mlruns/0/5a6be5a1ef844783a50a6577745dbdc3/artifacts/model/MLmodel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see above the `MLmodel` keeps track, between others, of\n",
    "\n",
    "- The experiment id, `5a6be5a1ef844783a50a6577745dbdc3`\n",
    "- Date \n",
    "- Version of `sklearn` \n",
    "- How the model was stored\n",
    "\n",
    "As we shall see shortly, the pre-packaged Seldon's model server will use this file to serve this model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Upload models (optional)\n",
    "\n",
    "As a last step, we will persist the models we have just trained using `MLflow`. For that, we will upload them into Google Cloud Storage. Note that to run these commands you need write access into the `gs://seldon-models` bucket and you need to have `gsutil` set up.\n",
    "\n",
    "Note that in a production setting, MLflow would be configured to log models against a persistent data store (e.g. GCS, Minio, etc.). In that case, this manual step wouldn't be needed.\n",
    "\n",
    "We will upload both versions of the model to:\n",
    "\n",
    "- `gs://seldon-models/mlflow/model-a`\n",
    "- `gs://seldon-models/mlflow/model-b`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Copying file://mlruns/0/c047eddcdc2d4a08963f08516fd18d74/artifacts/model/conda.yaml [Content-Type=application/octet-stream]...\n",
      "Copying file://mlruns/0/c047eddcdc2d4a08963f08516fd18d74/artifacts/model/MLmodel [Content-Type=application/octet-stream]...\n",
      "Copying file://mlruns/0/c047eddcdc2d4a08963f08516fd18d74/artifacts/model/model.pkl [Content-Type=application/octet-stream]...\n",
      "\\ [3 files][  1.1 KiB/  1.1 KiB]                                                \n",
      "Operation completed over 3 objects/1.1 KiB.                                      \n",
      "Copying file://mlruns/0/1aab766b2d9246ed85f8dbfec4e8743d/artifacts/model/conda.yaml [Content-Type=application/octet-stream]...\n",
      "Copying file://mlruns/0/1aab766b2d9246ed85f8dbfec4e8743d/artifacts/model/MLmodel [Content-Type=application/octet-stream]...\n",
      "Copying file://mlruns/0/1aab766b2d9246ed85f8dbfec4e8743d/artifacts/model/model.pkl [Content-Type=application/octet-stream]...\n",
      "\\ [3 files][  1.1 KiB/  1.1 KiB]                                                \n",
      "Operation completed over 3 objects/1.1 KiB.                                      \n"
     ]
    }
   ],
   "source": [
    "!gsutil cp -r mlruns/0/c047eddcdc2d4a08963f08516fd18d74/artifacts/model/* gs://seldon-models/mlflow/model-a\n",
    "!gsutil cp -r mlruns/0/1aab766b2d9246ed85f8dbfec4e8743d/artifacts/model/* gs://seldon-models/mlflow/model-b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "toc-hr-collapsed": false
   },
   "source": [
    "## Serving\n",
    "\n",
    "To serve this model we will use Seldon."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deploy models\n",
    "\n",
    "Once the cluster is set up, the next step will to upload these models into a common repository and to deploy two `SeldonDeployment` specs to `k8s`. As we can see below, we will route 50% of the traffic to each of the models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting ./serving/model-a-b.yaml\n"
     ]
    }
   ],
   "source": [
    "%%writefile ./serving/model-a-b.yaml\n",
    "---\n",
    "apiVersion: machinelearning.seldon.io/v1alpha2\n",
    "kind: SeldonDeployment\n",
    "metadata:\n",
    "  name: wines-classifier\n",
    "spec:\n",
    "  annotations:\n",
    "    seldon.io/executor: \"false\" \n",
    "  predictors:\n",
    "  - graph:\n",
    "      children: []\n",
    "      implementation: MLFLOW_SERVER\n",
    "      modelUri: gs://seldon-models/mlflow/model-a\n",
    "      name: wines-classifier\n",
    "    name: model-a\n",
    "    replicas: 1\n",
    "    traffic: 50\n",
    "    componentSpecs:\n",
    "    - spec:\n",
    "        # We are setting high failureThreshold as installing conda dependencies\n",
    "        # can take long time and we want to avoid k8s killing the container prematurely\n",
    "        containers:\n",
    "        - name: wines-classifier\n",
    "          livenessProbe:\n",
    "            initialDelaySeconds: 60\n",
    "            failureThreshold: 100\n",
    "            periodSeconds: 5\n",
    "            successThreshold: 1\n",
    "            httpGet:\n",
    "              path: /health/ping\n",
    "              port: http\n",
    "              scheme: HTTP\n",
    "          readinessProbe:\n",
    "            initialDelaySeconds: 60\n",
    "            failureThreshold: 100\n",
    "            periodSeconds: 5\n",
    "            successThreshold: 1\n",
    "            httpGet:\n",
    "              path: /health/ping\n",
    "              port: http\n",
    "              scheme: HTTP\n",
    "  - graph:\n",
    "      children: []\n",
    "      implementation: MLFLOW_SERVER\n",
    "      modelUri: gs://seldon-models/mlflow/model-b\n",
    "      name: wines-classifier\n",
    "    name: model-b\n",
    "    replicas: 1\n",
    "    traffic: 50\n",
    "    componentSpecs:\n",
    "    - spec:\n",
    "        # We are setting high failureThreshold as installing conda dependencies\n",
    "        # can take long time and we want to avoid k8s killing the container prematurely\n",
    "        containers:\n",
    "        - name: wines-classifier\n",
    "          livenessProbe:\n",
    "            initialDelaySeconds: 60\n",
    "            failureThreshold: 100\n",
    "            periodSeconds: 5\n",
    "            successThreshold: 1\n",
    "            httpGet:\n",
    "              path: /health/ping\n",
    "              port: http\n",
    "              scheme: HTTP\n",
    "          readinessProbe:\n",
    "            initialDelaySeconds: 60\n",
    "            failureThreshold: 100\n",
    "            periodSeconds: 5\n",
    "            successThreshold: 1\n",
    "            httpGet:\n",
    "              path: /health/ping\n",
    "              port: http\n",
    "              scheme: HTTP"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io/wines-classifier created\n"
     ]
    }
   ],
   "source": [
    "!kubectl apply -f ./serving/model-a-b.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can verify these have been deployed by checking the pods and `SeldonDeployment` resources in the cluster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME                                                           READY   STATUS    RESTARTS   AGE\n",
      "wines-classifier-model-a-0-wines-classifier-676985589d-97rh5   0/2     Running   0          24s\n",
      "wines-classifier-model-b-0-wines-classifier-678d9477b9-zt4k6   0/2     Running   0          24s\n"
     ]
    }
   ],
   "source": [
    "!kubectl get pods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME               AGE\n",
      "wines-classifier   3m46s\n"
     ]
    }
   ],
   "source": [
    "!kubectl get sdep"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test models\n",
    "\n",
    "We will now run a sample query to test that the inference graph is working."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"meta\": {\n",
      "    \"puid\": \"2a05tajkudskg2lpav076ilsee\",\n",
      "    \"tags\": {\n",
      "    },\n",
      "    \"routing\": {\n",
      "    },\n",
      "    \"requestPath\": {\n",
      "      \"wines-classifier\": \"seldonio/mlflowserver_rest:1.3.0\"\n",
      "    },\n",
      "    \"metrics\": []\n",
      "  },\n",
      "  \"data\": {\n",
      "    \"names\": [],\n",
      "    \"ndarray\": [5.554168965299016]\n",
      "  }\n",
      "}"
     ]
    }
   ],
   "source": [
    "!curl \\\n",
    "    -X POST \\\n",
    "    -H 'Content-Type: application/json' \\\n",
    "    -d '{\\\n",
    "        \"data\": { \\\n",
    "            \"names\": [\"fixed acidity\",\"volatile acidity\",\"citric acid\",\"residual sugar\",\"chlorides\",\"free sulfur dioxide\",\"total sulfur dioxide\",\"density\",\"pH\",\"sulphates\",\"alcohol\"], \\\n",
    "            \"ndarray\": [[7,0.27,0.36,20.7,0.045,45,170,1.001,3,0.45,8.8]] \\\n",
    "        } \\\n",
    "    }' \\\n",
    "    http://localhost:8083/seldon/default/wines-classifier/api/v1.0/predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analytics\n",
    "\n",
    "To access Grafana, it will be necessary to forward the port to the respective pod as we did previously to access the Seldon Core deployment.\n",
    "The credentials will be simply `admin` // `password`.\n",
    "\n",
    "This command needs to run constantly on the background, so **please make sure you run it on a separate terminal**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```bash\n",
    "$ kubectl port-forward \\\n",
    "    $(kubectl get pods \\\n",
    "        -l app=grafana-prom-server -o jsonpath='{.items[0].metadata.name}') \\\n",
    "    3000:3000\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have both models running in production, we can analyse their performance using Seldon Core's integration with Prometheus and Grafana.\n",
    "To do so, we will iterate over the training set (which can be foud in `./training/wine-quality.csv`), making a request and sending the feedback of the prediction.\n",
    "\n",
    "Since the `/feedback` endpoint requires a `reward` signal (i.e. higher better), we will simulate one as\n",
    "\n",
    "$$\n",
    "  R(x_{n})\n",
    "    = \\begin{cases}\n",
    "        \\frac{1}{(y_{n} - f(x_{n}))^{2}} &, y_{n} \\neq f(x_{n}) \\\\\n",
    "        500 &, y_{n} = f(x_{n})\n",
    "      \\end{cases}\n",
    "$$\n",
    "\n",
    ", where $R(x_{n})$ is the reward for input point $x_{n}$, $f(x_{n})$ is our trained model and $y_{n}$ is the actual value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting feedback.py\n"
     ]
    }
   ],
   "source": [
    "### %%writefile feedback.py\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from seldon_core.seldon_client import SeldonClient\n",
    "\n",
    "sc = SeldonClient(\n",
    "    gateway=\"istio\", \n",
    "    namespace=\"default\",\n",
    "    gateway_endpoint=\"localhost:8083\",\n",
    "    deployment_name='wines-classifier')\n",
    "\n",
    "df = pd.read_csv(\"./training/wine-quality.csv\")\n",
    "\n",
    "def _get_reward(y, y_pred):\n",
    "    if y == y_pred:\n",
    "        return 500    \n",
    "    \n",
    "    return 1 / np.square(y - y_pred)\n",
    "\n",
    "def _test_row(row):\n",
    "    input_features = row[:-1]\n",
    "    feature_names = input_features.index.to_list()\n",
    "    X = input_features.values.reshape(1, -1)\n",
    "    y = row[-1].reshape(1, -1)\n",
    "    \n",
    "    r = sc.predict(\n",
    "        data=X,\n",
    "        names=feature_names)\n",
    "    \n",
    "    y_pred = r.response['data']['tensor']['values']\n",
    "    reward = _get_reward(y, y_pred)\n",
    "    sc.feedback(\n",
    "        prediction_request=r.request,\n",
    "        prediction_response=r.response,\n",
    "        reward=reward)\n",
    "    \n",
    "    return reward[0]\n",
    "\n",
    "df.apply(_test_row, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "^C\n",
      "Traceback (most recent call last):\n",
      "  File \"feedback.py\", line 39, in <module>\n",
      "    df.apply(_test_row, axis=1)\n",
      "  File \"/home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages/pandas/core/frame.py\", line 7548, in apply\n",
      "    return op.get_result()\n",
      "  File \"/home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages/pandas/core/apply.py\", line 180, in get_result\n",
      "    return self.apply_standard()\n",
      "  File \"/home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages/pandas/core/apply.py\", line 271, in apply_standard\n",
      "    results, res_index = self.apply_series_generator()\n",
      "  File \"/home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages/pandas/core/apply.py\", line 300, in apply_series_generator\n",
      "    results[i] = self.f(v)\n",
      "  File \"feedback.py\", line 27, in _test_row\n",
      "    names=feature_names)\n",
      "  File \"/home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages/seldon_core/seldon_client.py\", line 387, in predict\n",
      "    return rest_predict_gateway(**k)\n",
      "  File \"/home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages/seldon_core/seldon_client.py\", line 1631, in rest_predict_gateway\n",
      "    url, json=payload, headers=req_headers, verify=verify, cert=cert\n",
      "  File \"/home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages/requests/api.py\", line 119, in post\n",
      "    return request('post', url, data=data, json=json, **kwargs)\n",
      "  File \"/home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages/requests/api.py\", line 61, in request\n",
      "    return session.request(method=method, url=url, **kwargs)\n",
      "  File \"/home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages/requests/sessions.py\", line 530, in request\n",
      "    resp = self.send(prep, **send_kwargs)\n",
      "  File \"/home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages/requests/sessions.py\", line 643, in send\n",
      "    r = adapter.send(request, **kwargs)\n",
      "  File \"/home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages/requests/adapters.py\", line 449, in send\n",
      "    timeout=timeout\n",
      "  File \"/home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages/urllib3/connectionpool.py\", line 677, in urlopen\n",
      "    chunked=chunked,\n",
      "  File \"/home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages/urllib3/connectionpool.py\", line 426, in _make_request\n",
      "    six.raise_from(e, None)\n",
      "  File \"<string>\", line 3, in raise_from\n",
      "  File \"/home/agm/.virtualenvs/mlflow-talk/lib/python3.7/site-packages/urllib3/connectionpool.py\", line 421, in _make_request\n",
      "    httplib_response = conn.getresponse()\n",
      "  File \"/home/agm/.asdf/installs/python/3.7.8/lib/python3.7/http/client.py\", line 1354, in getresponse\n",
      "    response.begin()\n",
      "  File \"/home/agm/.asdf/installs/python/3.7.8/lib/python3.7/http/client.py\", line 306, in begin\n",
      "    version, status, reason = self._read_status()\n",
      "  File \"/home/agm/.asdf/installs/python/3.7.8/lib/python3.7/http/client.py\", line 267, in _read_status\n",
      "    line = str(self.fp.readline(_MAXLINE + 1), \"iso-8859-1\")\n",
      "  File \"/home/agm/.asdf/installs/python/3.7.8/lib/python3.7/socket.py\", line 589, in readinto\n",
      "    return self._sock.recv_into(b)\n",
      "KeyboardInterrupt\n"
     ]
    }
   ],
   "source": [
    "!python feedback.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " We can now access the Grafana dashboard in http://localhost:3000 (credentials are `admin` // `password`). Inside the portal, we will go to the Prediction Analytics dashboard.\n",
    " \n",
    " \n",
    "We can see a snapshot below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Seldon Analytics](./images/seldon-analytics.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.6"
  },
  "toc-showmarkdowntxt": false,
  "toc-showtags": false
 },
 "nbformat": 4,
 "nbformat_minor": 4
}