2. Bias Variance Trade-Off.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>\n",
       "\n",
       "div.cell { /* Tunes the space between cells */\n",
       "margin-top:1em;\n",
       "margin-bottom:1em;\n",
       "}\n",
       "\n",
       "div.text_cell_render h1 { /* Main titles bigger, centered */\n",
       "font-size: 2.2em;\n",
       "line-height:1.4em;\n",
       "text-align:center;\n",
       "}\n",
       "\n",
       "div.text_cell_render h2 { /*  Parts names nearer from text */\n",
       "margin-bottom: -0.4em;\n",
       "}\n",
       "\n",
       "\n",
       "div.text_cell_render { /* Customize text cells */\n",
       "font-family: 'Arial';\n",
       "font-size:1.5em;\n",
       "line-height:1.4em;\n",
       "padding-left:3em;\n",
       "padding-right:3em;\n",
       "}\n",
       "\n",
       ".MathJax {\n",
       "    font-size: 70%;\n",
       "}\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import HTML\n",
    "css_file = './custom.css'\n",
    "HTML(open(css_file, \"r\").read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bias-Variance Trade-Off\n",
    "\n",
    "© 2018 Daniel Voigt Godoy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Definition\n",
    "\n",
    "Every time you train a model and then test it on some ***unseen data***, you're going to get ***prediction errors***.\n",
    "\n",
    "These errors can be ***decomposed*** into three distinct terms: ***bias***, ***variance*** and ***noise***.\n",
    "\n",
    "Let's start with the last term: ***noise***. This is the ***irreducible*** part of the error. Data is ***noisy***. Sometimes the noise can be cleaned up (removing outliers, for instance), but more often than not, there will be a level of noise you simply cannot get rid of.\n",
    "\n",
    "Since we cannot do much about the noise, let's check the other two:\n",
    "\n",
    "1. ***Bias***: \n",
    "\n",
    "    1.1 If a model has high ***bias***, it means it is ***consistently*** either ***under-*** or ***over-*** shooting the actual values (on unseen data!)\n",
    "    \n",
    "    1.2 This is due to ***wrong assumptions*** about the model, i.e., the model is ***too simple*** to represent the data - like ***fitting a line*** to ***quadratic data***.\n",
    "\n",
    "    1.3 ***High Bias*** is usually a sign of a model ***underfitting*** the training data.\n",
    "    \n",
    "    1.4 If you add ***more data*** and retrain the model, its coefficients ***do not change much***.\n",
    "\n",
    "    1.5 ***IMPORTANT: DO NOT*** confuse this bias with the ***bias term (b0)*** from a linear model!\n",
    "    \n",
    "\n",
    "2. ***Variance***: \n",
    "\n",
    "    2.1 If a model has high ***variance***, it means it is ***too sensitive*** to variations in the training data.\n",
    "    \n",
    "    2.2 This is due to a ***too complex*** model - it is fitting the noisy data too good!\n",
    "    \n",
    "    2.3 ***High Variance*** is usually a sign of a model ***overfitting*** the training data.\n",
    "    \n",
    "    2.4 If you add ***more data*** and retrain the models, its coefficients are likely ***changing a lot***.\n",
    "    \n",
    "    \n",
    "The plot below illustrates the difference:\n",
    "\n",
    "![bias variance](bias_variance.png)\n",
    "<center>Source: Scott Fortmann-Roe - Understanding the Bias-Variance Tradeoff</center>\n",
    "\n",
    "\n",
    "### Talking about model complexity...\n",
    "![](https://imgs.xkcd.com/comics/curve_fitting.png)\n",
    "<center>Source: <a href=\"https://xkcd.com/2048/\">XKCD</a></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Experiment\n",
    "\n",
    "Time to try it yourself!\n",
    "\n",
    "This is a dataset with 12 points only. We start with two of them and fit a line to it.\n",
    "\n",
    "The controls below allow you to:\n",
    "- change the ***degree*** of the polynomial to fit the data to (using degree + 1 points)\n",
    "- add ***more samples*** to the training data (up to 10 samples for 1st degree down to 4 samples for 8th degree), ***fitting it again***\n",
    "- include ***validation*** and ***test*** data\n",
    "\n",
    "The ***left plot*** contains the data, and the ***fitted model*** to it (green line).\n",
    "\n",
    "The ***upper right*** plot has the model's coefficients, as many as the polynomial degree you chose.\n",
    "\n",
    "The ***lower right*** plot has the ***average error*** when fitting the model with as ***many samples*** as shown in the horizontal axis. It starts with ***zero*** error for ***two points***.\n",
    "\n",
    "Use the controls to play with different configurations and answer the ***questions*** below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from intuitiveml.supervised.regression.BiasVariance import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train, y_train, x_val, y_val, x_test, y_test = data()\n",
    "f = build_figure(x_train, y_train, x_val, y_val, x_test, y_test)\n",
    "vb = VBox(f, layout={'align_items': 'center'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7d7b8f1220644e3b897f757413038603",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(FigureWidget({\n",
       "    'data': [{'marker': {'color': 'green'},\n",
       "              'mode': 'markers',\n",
       "   …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "vb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Questions\n",
    "\n",
    "1. For a ***degree of 1***:\n",
    "    - increase the number of extra samples to 5 \n",
    "        - what happens to the ***training error***?\n",
    "    - keep increasing the number of samples up to 10 \n",
    "        - what happens to the ***training error***? \n",
    "        - what about the ***coefficients***?\n",
    "    - what is this behavior a sign of? High ***bias*** or high ***variance***? Why?\n",
    "    \n",
    "    \n",
    "2. For a ***degree of 5***:\n",
    "    - increase the number of extra samples one by one up to 6 (to the total of 12 data points)\n",
    "        - what happens to the ***training error***?\n",
    "        - what about the ***coefficients***?\n",
    "    - what is this behavior a sign of? High ***bias*** or high ***variance***? Why?\n",
    " \n",
    "\n",
    "3. Looking at the ***training error*** only? Which model (degree) is the best? Why?\n",
    "\n",
    "\n",
    "4. Is this enough information to settle on a given model? If not, why?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 Train - Validation - Test Split\n",
    "\n",
    "- ***Training Set***: the data you use to, obviously, ***train*** your model - you can use and abuse this data!\n",
    "\n",
    "\n",
    "- ***Validation Set***: the data you should only use to ***hyper-parameter tuning***, that is, comparing differently parameterized models trained on the training data, to decide which parameters are best. \n",
    "\n",
    "    You should use, but ***not*** abuse this data, as it is intended to provide an ***unbiased*** evaluation of your model and, if you mess around with it too much, you'll end up incorporating knowledge about it in your model without even noticing.\n",
    "\n",
    "\n",
    "- ***Test Set***: the data you should use only ***once***, when you are done with everything else, to check if your model is still performing well.\n",
    "\n",
    "    I like to pretend this is data from the ***\"future\"*** - that particular day in the future when my model is ready to give it a go in the real world! So, until that day, I cannot know this data, as the future hasn't arrived yet :-)\n",
    "    \n",
    "This is a nice representation of the split:\n",
    "\n",
    "![](train-validate-test.png)\n",
    "<center>Source: David Ziganto - Model Tuning (Part 2 - Validation & Cross-Validation) </center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you can go back to the ***experiment*** to answer the questions below.\n",
    "\n",
    "#### More questions:\n",
    "\n",
    "5. Now, set ***extra samples*** to 10 and add ***validation data***:\n",
    "    - for a ***degree of 1***:\n",
    "        - what happens to the ***validation error***?\n",
    "        - how ***far apart*** both errors, training and validation, are?\n",
    "        - what is this behavior a sign of? High ***bias*** or high ***variance***? Why?\n",
    "    - increase the ***degree*** one by one:\n",
    "        - what happens to the ***distance*** between the two error curves?\n",
    "\n",
    "\n",
    "6. Looking at both ***training*** and ***validation*** errors, which model (degree) is the best? Why?\n",
    "\n",
    "\n",
    "7. Set ***degree*** to your choice of best model and add ***test data***:\n",
    "    - how does ***test error*** compare to ***validation*** and ***training*** errors?\n",
    "    - are you happy with your choice of best model? Why?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Learning Curves\n",
    "\n",
    "After performing the experiment and answering the question, you probably observed some differences in the training, validation and test curves as the number of samples got increased.\n",
    "\n",
    "There are typical patterns, depending on how well your model is tuned:\n",
    "\n",
    "![](bias-variance2.png)\n",
    "\n",
    "<center>Source: Utku Ufuk - Learning Curves in Linear & Polynomial Regression </center>\n",
    "\n",
    "- High Bias (left plot): training error is ***high*** even if you add more samples, test error is ***high**** but there is ***little gap*** between the two\n",
    "\n",
    "- High Variance (right plot): training error is ***very low***, but test error is ***high*** - there is a ***huge gap***\n",
    "\n",
    "- \"Just right\" (center plot): training error sits between the other two situations, and test error converges to a similar level as new samples are added to the training - there is a ***little gap***, but the overall level is lower than the \"high bias\" situation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. The Trade-Off\n",
    "\n",
    "To summarize it, as the ***model complexity*** increases:\n",
    "\n",
    "- ***variance*** increases\n",
    "- ***bias*** decreases\n",
    "- ***total error*** goes ***down*** up to a point and then ***up*** again\n",
    "\n",
    "![](bias_vs_variance_error.png)\n",
    "<center>Source: Scott Fortmann-Roe - Understanding the Bias-Variance Tradeoff</center>\n",
    "\n",
    "There is a ***sweet spot*** of model complexity where the ***total error*** is at its ***minimum***."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Scikit-Learn\n",
    "\n",
    "[Train Test Split](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html)\n",
    "\n",
    "[Cross-validation](https://scikit-learn.org/stable/modules/cross_validation.html#cross-validation)\n",
    "\n",
    "[Learning Curves](https://scikit-learn.org/stable/modules/learning_curve.html#learning-curve)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. More Resources\n",
    "\n",
    "[Model Tuning and the Bias-Variance Tradeoff](http://www.r2d3.us/visual-intro-to-machine-learning-part-2/)\n",
    "\n",
    "[Understanding the Bias-Variance Tradeoff](http://scott.fortmann-roe.com/docs/BiasVariance.html)\n",
    "\n",
    "[Learning Curves in Linear & Polynomial Regression](https://utkuufuk.github.io/2018/05/04/learning-curves/)\n",
    "\n",
    "[Model Evaluation & Selection](https://heartbeat.fritz.ai/model-evaluation-selection-i-30d803a44ee)\n",
    "\n",
    "[Learning Curves for Machine Learning](https://www.dataquest.io/blog/learning-curves-machine-learning/)\n",
    "\n",
    "[Model Tuning (Part 2 - Validation & Cross-Validation)](https://dziganto.github.io/cross-validation/data%20science/machine%20learning/model%20tuning/python/Model-Tuning-with-Validation-and-Cross-Validation/)\n",
    "\n",
    "[How (and Why) to create a good validation set](https://www.fast.ai/2017/11/13/validation-sets/)\n",
    "\n",
    "[How (dis)similar are my train and test data?](https://towardsdatascience.com/how-dis-similar-are-my-train-and-test-data-56af3923de9b)\n",
    "\n",
    "[Validation strategies for your machine learning model](https://heartbeat.fritz.ai/model-evaluation-selection-i-30d803a44ee)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### This material is copyright Daniel Voigt Godoy and made available under the Creative Commons Attribution (CC-BY) license ([link](https://creativecommons.org/licenses/by/4.0/)). \n",
    "\n",
    "#### Code is also made available under the MIT License ([link](https://opensource.org/licenses/MIT))."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<script>\n",
       "  function code_toggle() {\n",
       "    if (code_shown){\n",
       "      $('div.input').hide('500');\n",
       "      $('#toggleButton').val('Show Code')\n",
       "    } else {\n",
       "      $('div.input').show('500');\n",
       "      $('#toggleButton').val('Hide Code')\n",
       "    }\n",
       "    code_shown = !code_shown\n",
       "  }\n",
       "\n",
       "  $( document ).ready(function(){\n",
       "    code_shown=false;\n",
       "    $('div.input').hide()\n",
       "  });\n",
       "</script>\n",
       "<form action=\"javascript:code_toggle()\"><input type=\"submit\" id=\"toggleButton\" value=\"Show Code\"></form>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import HTML\n",
    "HTML('''<script>\n",
    "  function code_toggle() {\n",
    "    if (code_shown){\n",
    "      $('div.input').hide('500');\n",
    "      $('#toggleButton').val('Show Code')\n",
    "    } else {\n",
    "      $('div.input').show('500');\n",
    "      $('#toggleButton').val('Hide Code')\n",
    "    }\n",
    "    code_shown = !code_shown\n",
    "  }\n",
    "\n",
    "  $( document ).ready(function(){\n",
    "    code_shown=false;\n",
    "    $('div.input').hide()\n",
    "  });\n",
    "</script>\n",
    "<form action=\"javascript:code_toggle()\"><input type=\"submit\" id=\"toggleButton\" value=\"Show Code\"></form>''')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}