14. OGrisel Pandas.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Predictive Modeling with heterogeneous data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "import warnings\n",
    "warnings.simplefilter('ignore', DeprecationWarning)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"files/images/predictive_modeling_data_flow.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading tabular data from the Titanic kaggle challenge in a pandas Data Frame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us have a look at the Titanic dataset from the Kaggle Getting Started challenge at:\n",
    "\n",
    "https://www.kaggle.com/c/titanic-gettingStarted\n",
    "\n",
    "We can load the CSV file as a pandas data frame in one line:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PassengerId,Survived,Pclass,Name,Sex,Age,SibSp,Parch,Ticket,Fare,Cabin,Embarked\n",
      "\n",
      "1,0,3,\"Braund, Mr. Owen Harris\",male,22,1,0,A/5 21171,7.25,,S\n",
      "\n",
      "2,1,1,\"Cumings, Mrs. John Bradley (Florence Briggs Thayer)\",female,38,1,0,PC 17599,71.2833,C85,C\n",
      "\n",
      "3,1,3,\"Heikkinen, Miss. Laina\",female,26,0,0,STON/O2. 3101282,7.925,,S\n",
      "\n",
      "4,1,1,\"Futrelle, Mrs. Jacques Heath (Lily May Peel)\",female,35,1,0,113803,53.1,C123,S\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#!curl -s https://dl.dropboxusercontent.com/u/5743203/data/titanic/titanic_train.csv | head -5\n",
    "with open('titanic_train.csv', 'r') as f:\n",
    "    for i, line in zip(range(5), f):\n",
    "        print(line)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#data = pd.read_csv('https://dl.dropboxusercontent.com/u/5743203/data/titanic/titanic_train.csv')\n",
    "data = pd.read_csv('titanic_train.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "pandas data frames have a HTML table representation in the IPython notebook. Let's have a look at the first 5 rows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 3</td>\n",
       "      <td>                           Braund, Mr. Owen Harris</td>\n",
       "      <td>   male</td>\n",
       "      <td> 22</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td>        A/5 21171</td>\n",
       "      <td>  7.2500</td>\n",
       "      <td>  NaN</td>\n",
       "      <td> S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td> 2</td>\n",
       "      <td> 1</td>\n",
       "      <td> 1</td>\n",
       "      <td> Cumings, Mrs. John Bradley (Florence Briggs Th...</td>\n",
       "      <td> female</td>\n",
       "      <td> 38</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td>         PC 17599</td>\n",
       "      <td> 71.2833</td>\n",
       "      <td>  C85</td>\n",
       "      <td> C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td> 3</td>\n",
       "      <td> 1</td>\n",
       "      <td> 3</td>\n",
       "      <td>                            Heikkinen, Miss. Laina</td>\n",
       "      <td> female</td>\n",
       "      <td> 26</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> STON/O2. 3101282</td>\n",
       "      <td>  7.9250</td>\n",
       "      <td>  NaN</td>\n",
       "      <td> S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td> 4</td>\n",
       "      <td> 1</td>\n",
       "      <td> 1</td>\n",
       "      <td>      Futrelle, Mrs. Jacques Heath (Lily May Peel)</td>\n",
       "      <td> female</td>\n",
       "      <td> 35</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td>           113803</td>\n",
       "      <td> 53.1000</td>\n",
       "      <td> C123</td>\n",
       "      <td> S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td> 5</td>\n",
       "      <td> 0</td>\n",
       "      <td> 3</td>\n",
       "      <td>                          Allen, Mr. William Henry</td>\n",
       "      <td>   male</td>\n",
       "      <td> 35</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td>           373450</td>\n",
       "      <td>  8.0500</td>\n",
       "      <td>  NaN</td>\n",
       "      <td> S</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   PassengerId  Survived  Pclass  \\\n",
       "0            1         0       3   \n",
       "1            2         1       1   \n",
       "2            3         1       3   \n",
       "3            4         1       1   \n",
       "4            5         0       3   \n",
       "\n",
       "                                                Name     Sex  Age  SibSp  \\\n",
       "0                            Braund, Mr. Owen Harris    male   22      1   \n",
       "1  Cumings, Mrs. John Bradley (Florence Briggs Th...  female   38      1   \n",
       "2                             Heikkinen, Miss. Laina  female   26      0   \n",
       "3       Futrelle, Mrs. Jacques Heath (Lily May Peel)  female   35      1   \n",
       "4                           Allen, Mr. William Henry    male   35      0   \n",
       "\n",
       "   Parch            Ticket     Fare Cabin Embarked  \n",
       "0      0         A/5 21171   7.2500   NaN        S  \n",
       "1      0          PC 17599  71.2833   C85        C  \n",
       "2      0  STON/O2. 3101282   7.9250   NaN        S  \n",
       "3      0            113803  53.1000  C123        S  \n",
       "4      0            373450   8.0500   NaN        S  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PassengerId    891\n",
       "Survived       891\n",
       "Pclass         891\n",
       "Name           891\n",
       "Sex            891\n",
       "Age            714\n",
       "SibSp          891\n",
       "Parch          891\n",
       "Ticket         891\n",
       "Fare           891\n",
       "Cabin          204\n",
       "Embarked       889\n",
       "dtype: int64"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data frame has 891 rows. Some passengers have missing information though: in particular Age and Cabin info can be missing. The meaning of the columns is explained on the challenge website:\n",
    "\n",
    "https://www.kaggle.com/c/titanic-gettingStarted/data\n",
    "\n",
    "A data frame can be converted into a numpy array by calling the `values` attribute:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['PassengerId',\n",
       " 'Survived',\n",
       " 'Pclass',\n",
       " 'Name',\n",
       " 'Sex',\n",
       " 'Age',\n",
       " 'SibSp',\n",
       " 'Parch',\n",
       " 'Ticket',\n",
       " 'Fare',\n",
       " 'Cabin',\n",
       " 'Embarked']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(data.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(891, 12)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 0, 3, ..., 7.25, nan, 'S'],\n",
       "       [2, 1, 1, ..., 71.2833, 'C85', 'C'],\n",
       "       [3, 1, 3, ..., 7.925, nan, 'S'],\n",
       "       ..., \n",
       "       [889, 0, 3, ..., 23.45, nan, 'S'],\n",
       "       [890, 1, 1, ..., 30.0, 'C148', 'C'],\n",
       "       [891, 0, 3, ..., 7.75, nan, 'Q']], dtype=object)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However this cannot be directly fed to a scikit-learn model:\n",
    "\n",
    "\n",
    "- the target variable (survival) is mixed with the input data\n",
    "\n",
    "- some attribute such as unique ids have no predictive values for the task\n",
    "\n",
    "- the values are heterogeneous (string labels for categories, integers and floating point numbers)\n",
    "\n",
    "- some attribute values are missing (nan: \"not a number\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Predicting survival"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The goal of the challenge is to predict whether a passenger has survived from others known attribute. Let us have a look at the `Survived` columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dtype('int64')"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "survived_column = data['Survived']\n",
    "survived_column.dtype"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`data.Survived` is an instance of the pandas `Series` class with an integer dtype:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.series.Series"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(survived_column)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `data` object is an instance pandas `DataFrame` class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.frame.DataFrame"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Series` can be seen as homegeneous, 1D columns. `DataFrame` instances are heterogenous collections of columns with the same length.\n",
    "\n",
    "The original data frame can be aggregated by counting rows for each possible value of the `Survived` column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PassengerId</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Survived</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td> 549</td>\n",
       "      <td> 549</td>\n",
       "      <td> 549</td>\n",
       "      <td> 549</td>\n",
       "      <td> 424</td>\n",
       "      <td> 549</td>\n",
       "      <td> 549</td>\n",
       "      <td> 549</td>\n",
       "      <td> 549</td>\n",
       "      <td>  68</td>\n",
       "      <td> 549</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td> 342</td>\n",
       "      <td> 342</td>\n",
       "      <td> 342</td>\n",
       "      <td> 342</td>\n",
       "      <td> 290</td>\n",
       "      <td> 342</td>\n",
       "      <td> 342</td>\n",
       "      <td> 342</td>\n",
       "      <td> 342</td>\n",
       "      <td> 136</td>\n",
       "      <td> 340</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          PassengerId  Pclass  Name  Sex  Age  SibSp  Parch  Ticket  Fare  \\\n",
       "Survived                                                                    \n",
       "0                 549     549   549  549  424    549    549     549   549   \n",
       "1                 342     342   342  342  290    342    342     342   342   \n",
       "\n",
       "          Cabin  Embarked  \n",
       "Survived                   \n",
       "0            68       549  \n",
       "1           136       340  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.groupby('Survived').count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.61616161616161613"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean(survived_column == 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From this the subset of the full passengers list, about 2/3 perished in the event. So if we are to build a predictive model from this data, a baseline model to compare the performance to would be to always predict death. Such a constant model would reach around 62% predictive accuracy (which is higher than predicting at random):"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "pandas `Series` instances can be converted to regular 1D numpy arrays by using the `values` attribute:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "target = survived_column.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "numpy.ndarray"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dtype('int64')"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "target.dtype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 1, 1, 1, 0])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "target[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training a predictive model on numerical features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`sklearn` estimators all work with homegeneous numerical feature descriptors passed as a numpy array. Therefore passing the raw data frame will not work out of the box.\n",
    "\n",
    "Let us start simple and build a first model that only uses readily available numerical features as input, namely `data.Fare`, `data.Pclass` and `data.Age`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Fare</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Age</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>  7.2500</td>\n",
       "      <td> 3</td>\n",
       "      <td> 22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td> 71.2833</td>\n",
       "      <td> 1</td>\n",
       "      <td> 38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>  7.9250</td>\n",
       "      <td> 3</td>\n",
       "      <td> 26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td> 53.1000</td>\n",
       "      <td> 1</td>\n",
       "      <td> 35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>  8.0500</td>\n",
       "      <td> 3</td>\n",
       "      <td> 35</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Fare  Pclass  Age\n",
       "0   7.2500       3   22\n",
       "1  71.2833       1   38\n",
       "2   7.9250       3   26\n",
       "3  53.1000       1   35\n",
       "4   8.0500       3   35"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "numerical_features = data.get(['Fare', 'Pclass', 'Age'])\n",
    "numerical_features.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unfortunately some passengers do not have age information:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Fare      891\n",
       "Pclass    891\n",
       "Age       714\n",
       "dtype: int64"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "numerical_features.count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use pandas `fillna` method to input the median age for those passengers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Fare      15.7417\n",
       "Pclass     2.0000\n",
       "Age       28.0000\n",
       "dtype: float64"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "median_features = numerical_features.dropna().median()\n",
    "median_features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Fare      891\n",
       "Pclass    891\n",
       "Age       891\n",
       "dtype: int64"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "imputed_features = numerical_features.fillna(median_features)\n",
    "imputed_features.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Fare</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Age</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>  7.2500</td>\n",
       "      <td> 3</td>\n",
       "      <td> 22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td> 71.2833</td>\n",
       "      <td> 1</td>\n",
       "      <td> 38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>  7.9250</td>\n",
       "      <td> 3</td>\n",
       "      <td> 26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td> 53.1000</td>\n",
       "      <td> 1</td>\n",
       "      <td> 35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>  8.0500</td>\n",
       "      <td> 3</td>\n",
       "      <td> 35</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Fare  Pclass  Age\n",
       "0   7.2500       3   22\n",
       "1  71.2833       1   38\n",
       "2   7.9250       3   26\n",
       "3  53.1000       1   35\n",
       "4   8.0500       3   35"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "imputed_features.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that the data frame is clean, we can convert it into an homogeneous numpy array of floating point values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[  7.25  ,   3.    ,  22.    ],\n",
       "       [ 71.2833,   1.    ,  38.    ],\n",
       "       [  7.925 ,   3.    ,  26.    ],\n",
       "       ..., \n",
       "       [ 23.45  ,   3.    ,  28.    ],\n",
       "       [ 30.    ,   1.    ,  26.    ],\n",
       "       [  7.75  ,   3.    ,  32.    ]])"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "features_array = imputed_features.values\n",
    "features_array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dtype('float64')"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "features_array.dtype"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take the 80% of the data for training a first model and keep 20% for computing is generalization score:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Library/Python/2.7/site-packages/sklearn/utils/sparsetools/__init__.py:3: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility\n",
      "  from ._min_spanning_tree import minimum_spanning_tree\n",
      "/Library/Python/2.7/site-packages/sklearn/utils/sparsetools/__init__.py:3: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility\n",
      "  from ._min_spanning_tree import minimum_spanning_tree\n",
      "/Library/Python/2.7/site-packages/sklearn/utils/sparsetools/_graph_validation.py:5: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility\n",
      "  from ._graph_tools import csgraph_to_dense, csgraph_from_dense,\\\n",
      "/Library/Python/2.7/site-packages/sklearn/utils/sparsetools/_graph_validation.py:5: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility\n",
      "  from ._graph_tools import csgraph_to_dense, csgraph_from_dense,\\\n",
      "/Library/Python/2.7/site-packages/sklearn/utils/sparsetools/__init__.py:4: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility\n",
      "  from ._traversal import connected_components\n",
      "/Library/Python/2.7/site-packages/sklearn/utils/sparsetools/__init__.py:4: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility\n",
      "  from ._traversal import connected_components\n",
      "/Library/Python/2.7/site-packages/sklearn/utils/extmath.py:20: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility\n",
      "  from ._logistic_sigmoid import _log_logistic_sigmoid\n",
      "/Library/Python/2.7/site-packages/sklearn/utils/extmath.py:20: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility\n",
      "  from ._logistic_sigmoid import _log_logistic_sigmoid\n",
      "/Library/Python/2.7/site-packages/sklearn/utils/extmath.py:22: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility\n",
      "  from .sparsefuncs_fast import csr_row_norms\n",
      "/Library/Python/2.7/site-packages/sklearn/utils/extmath.py:22: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility\n",
      "  from .sparsefuncs_fast import csr_row_norms\n",
      "/Library/Python/2.7/site-packages/sklearn/metrics/cluster/supervised.py:18: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility\n",
      "  from .expected_mutual_info_fast import expected_mutual_information\n",
      "/Library/Python/2.7/site-packages/sklearn/metrics/cluster/supervised.py:18: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility\n",
      "  from .expected_mutual_info_fast import expected_mutual_information\n",
      "/Library/Python/2.7/site-packages/sklearn/metrics/pairwise.py:56: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility\n",
      "  from .pairwise_fast import _chi2_kernel_fast, _sparse_manhattan\n",
      "/Library/Python/2.7/site-packages/sklearn/metrics/pairwise.py:56: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility\n",
      "  from .pairwise_fast import _chi2_kernel_fast, _sparse_manhattan\n"
     ]
    }
   ],
   "source": [
    "from sklearn.cross_validation import train_test_split\n",
    "\n",
    "features_train, features_test, target_train, target_test = train_test_split(\n",
    "    features_array, target, test_size=0.20, random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(712, 3)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "features_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(179, 3)"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "features_test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(712,)"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "target_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(179,)"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "target_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start with a simple model from sklearn, namely `LogisticRegression`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Library/Python/2.7/site-packages/sklearn/linear_model/least_angle.py:24: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility\n",
      "  from ..utils import array2d, arrayfuncs, as_float_array, check_arrays\n",
      "/Library/Python/2.7/site-packages/sklearn/linear_model/least_angle.py:24: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility\n",
      "  from ..utils import array2d, arrayfuncs, as_float_array, check_arrays\n",
      "/Library/Python/2.7/site-packages/sklearn/linear_model/coordinate_descent.py:26: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility\n",
      "  from . import cd_fast\n",
      "/Library/Python/2.7/site-packages/sklearn/linear_model/coordinate_descent.py:26: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility\n",
      "  from . import cd_fast\n",
      "/Library/Python/2.7/site-packages/sklearn/linear_model/__init__.py:21: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility\n",
      "  from .sgd_fast import Hinge, Log, ModifiedHuber, SquaredLoss, Huber\n",
      "/Library/Python/2.7/site-packages/sklearn/linear_model/__init__.py:21: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility\n",
      "  from .sgd_fast import Hinge, Log, ModifiedHuber, SquaredLoss, Huber\n",
      "/Library/Python/2.7/site-packages/sklearn/svm/base.py:8: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility\n",
      "  from . import libsvm, liblinear\n",
      "/Library/Python/2.7/site-packages/sklearn/svm/base.py:8: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility\n",
      "  from . import libsvm, liblinear\n",
      "/Library/Python/2.7/site-packages/sklearn/svm/base.py:9: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility\n",
      "  from . import libsvm_sparse\n",
      "/Library/Python/2.7/site-packages/sklearn/svm/base.py:9: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility\n",
      "  from . import libsvm_sparse\n",
      "/Library/Python/2.7/site-packages/sklearn/utils/random.py:9: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility\n",
      "  from ._random import sample_without_replacement\n",
      "/Library/Python/2.7/site-packages/sklearn/utils/random.py:9: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility\n",
      "  from ._random import sample_without_replacement\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, penalty='l2', random_state=None, tol=0.0001)"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "logreg = LogisticRegression(C=1)\n",
    "logreg.fit(features_train, target_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "target_predicted = logreg.predict(features_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.73184357541899436"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "accuracy_score(target_test, target_predicted)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This first model has around 73% accuracy: this is better than our baseline that always predicts death."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.73184357541899436"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "logreg.score(features_test, target_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model evaluation and interpretation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Interpreting linear model weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `coef_` attribute of a fitted linear model such as `LogisticRegression` holds the weights of each features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index([u'Fare', u'Pclass', u'Age'], dtype='object')"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feature_names = numerical_features.columns\n",
    "feature_names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.0043996 , -0.80916725, -0.03348064]])"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "logreg.coef_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": [
       "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEJCAYAAABv6GdPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\n",
       "AAALEgAACxIB0t1+/AAADzZJREFUeJzt3X+Q3PVdx/HnJlcKkYTL0ZIfQgTapgUsCAoTKGNWIYKh\n",
       "pNERpozalhZLtQ6gVgnDqOtvk07tD6EdhIqptiKltRXBNkd0wygKrUChBQwwxrZgLhh+JKIowvrH\n",
       "+3vkurd3e8n39tf7no+Zm/1+dz+738/M5/Z1n+/78909kCRJkiRJkiRJkiRJkiRJGliVXndg3OrV\n",
       "qxvbtm3rdTckadBsA6rNd/ZNuAONRqPR6z50TK1Wo1ar9bobOgCO3WDLPn6VSgVaZPm87ndFktRp\n",
       "hrskJWS4d0m1Wu11F3SAHLvBNlfHz5q7JA0wa+6SNIcY7pKUkOEuSQkZ7pKUkOEuSQkZ7pKUkOEu\n",
       "SQkZ7pKUkOEuSQkZ7pKUkOEuSQkZ7pKU0GyE+7nAI8CjwJVTtPlY8fjXgJNn4ZiSpGmUDff5wDVE\n",
       "wB8PXAQc19RmLfB64A3Ae4FPlDymJKmNsuF+GvAYsAN4EbgJeFtTm3XA5mL7bmAYWFLyuJKkaZQN\n",
       "9+8GvjVh/9vFfe3aHFnyuJKkaQyVfP5M/7tG8xfJt3xe8aXz6pCFCxezZ8/Tve6GpC4oG+5PAEdN\n",
       "2D+KmJlP1+bI4j4lsWjRCHv3PtPrbqTXqT/Ojl/nzebY1et16vV623Zlp8pDwL8AZwFPAvcQi6oP\n",
       "T2izFvj54nYV8JHitpn/Zm9AxRmXY9d5FTrxHnH8uqEzYwdT/5u9sjP3/yOC+8vElTOfJIL90uLx\n",
       "64DbiWB/DHgeuLjkMSVJbfRTkduZ+4By5tctztwHV/dn7n5CVZISMtwlKSHDXZISKrugKmnALVy4\n",
       "mL17+2n5LZ+FCxd3/Zj9NKIuqA4oF+S6pXOLchpcLqhK0hxiuEtSQoa7JCVkuEtSQoa7JCVkuEtS\n",
       "Qoa7JCVkuEtSQoa7JCVkuEtSQoa7JCVkuEtSQoa7JCVkuEtSQoa7JCVkuEtSQoa7JCVkuEtSQoa7\n",
       "JCVkuEtSQoa7JCVkuEtSQoa7JCVkuEtSQoa7JCVkuEtSQoa7JCVkuEtSQoa7JCVkuEtSQmXDfQQY\n",
       "BbYDW4DhFm2OAv4O+AbwdeCykseUJLVRNtw3EOG+Etha7Dd7EfgF4ARgFfB+4LiSx5UkTaNsuK8D\n",
       "Nhfbm4H1LdrsBO4vtv8TeBhYXvK4kqRplA33JcBYsT1W7E/naOBk4O6Sx5UkTWNoBm1GgaUt7r+6\n",
       "ab9R/EzlUOAW4HJiBi9J6pCZhPuaaR4bI4J/J7AM2DVFu1cBnwP+DPjCVC9Wq9Ve2a5Wq1Sr1Rl0\n",
       "T5Lmjnq9Tr1eb9uuUvI4m4DdwEZiMXWYyYuqFaIev5tYWJ1Ko9GYbuKvflWpVJj+pE2zo4LvETWL\n",
       "99/kLC8b7iPAzcAKYAdwIfAssWB6PXAecCZwJ/AA+xLgKuBLTa9luA8ow71bDHdN1qlwn02G+4Ay\n",
       "3LvFcNdkU4W7n1CVpIQMd0lKyHCXpIQMd0lKyHCXpIQMd0lKyHCXpIQMd0lKyHCXpIQMd0lKyHCX\n",
       "pIQMd0lKyHCXpIQMd0lKyHCXpIQMd0lKyHCXpIQMd0lKyHCXpIQMd0lKyHCXpIQMd0lKyHCXpIQM\n",
       "d0lKyHCXpIQMd0lKyHCXpIQMd0lKyHCXpIQMd0lKyHCXpIQMd0lKyHCXpIQMd0lKyHCXpIQMd0lK\n",
       "qEy4jwCjwHZgCzA8Tdv5wH3ArSWOJ0maoTLhvoEI95XA1mJ/KpcDDwGNEseTJM1QmXBfB2wutjcD\n",
       "66dodySwFrgBqJQ4niRphsqE+xJgrNgeK/Zb+TDwy8DLJY4lSdoPQ20eHwWWtrj/6qb9Bq1LLm8F\n",
       "dhH19ur+dk6SdGDahfuaaR4bI4J/J7CMCPFmZxDlm7XAwcAi4FPAO1q9YK1We2W7Wq1SrVbbdE+S\n",
       "5pZ6vU69Xm/brkwNfBOwG9hILKYOM/2i6mrgA8D5UzzeaDRcbx1ElUoF18q7oYLvETWL99/kLC9T\n",
       "c/99Yma/HfjhYh9gOXDbFM/xN1OSuqCfrl5x5j6gnLl3izN3TdaJmbskqU8Z7pKUkOEuSQkZ7pKU\n",
       "kOEuSQkZ7pKUkOEuSQkZ7pKUkOEuSQkZ7pKUkOEuSQkZ7pKUkOEuSQkZ7pKUkOEuSQkZ7pKUkOEu\n",
       "SQkZ7pKUkOEuSQkZ7pKUkOEuSQkZ7pKUkOEuSQkZ7pKUkOEuSQkZ7pKUkOEuSQkZ7pKUkOEuSQkZ\n",
       "7pKUkOEuSQkZ7pKUkOEuSQkZ7pKUkOEuSQmVCfcRYBTYDmwBhqdoNwzcAjwMPASsKnFMSdIMlAn3\n",
       "DUS4rwS2FvutfBS4HTgOOJEIeUlSB1VKPPcRYDUwBiwF6sCbmtocBtwHHDuD12s0Go0S3VGvVCoV\n",
       "wLHrvAq+R9Qs3n+Ts7zMzH0JEewUt0tatDkGeAq4EbgXuB5YUOKYkqQZaBfuo8CDLX7WNbVr0Hrq\n",
       "NgScAny8uH2eqcs3kqRZMtTm8TXTPDZejtkJLAN2tWjz7eLnK8X+LUwT7rVa7ZXtarVKtVpt0z1J\n",
       "mlvq9Tr1er1tuzI1903AbmAjEdjDtA7uO4FLiKtqasAhwJUt2llzH1DW3LvFmrsmm6rmXibcR4Cb\n",
       "gRXADuBC4FlgOVFbP69odxJwA3AQ8DhwMfBci9cz3AeU4d4thrsm60S4zzbDfUAZ7t1iuGuyTlwt\n",
       "I0nqU4a7JCVkuEtSQoa7JCVkuEtSQoa7JCVkuEtSQoa7JCVkuEtSQoa7JCVkuEtSQoa7JCVkuEtS\n",
       "Qoa7JCVkuEtSQoa7JCVkuEtSQoa7JCVkuEtSQoa7JCVkuEtSQoa7JCVkuEtSQoa7JCVkuEtSQoa7\n",
       "JCVkuEtSQoa7JCVkuEtSQoa7JCVkuEtSQoa7JCVkuEtSQoa7JCVkuEtSQoa7JCVUJtxHgFFgO7AF\n",
       "GJ6i3VXAN4AHgc8Ary5xTEnSDJQJ9w1EuK8Ethb7zY4GfgY4BXgzMB94e4ljSpJmoEy4rwM2F9ub\n",
       "gfUt2uwBXgQWAEPF7RMljilJmoEy4b4EGCu2x4r9Zk8DHwK+CTwJPAvcUeKYkqQZGGrz+CiwtMX9\n",
       "VzftN4qfZq8DriDKM88BnwV+Evj0fvVSkrRf2oX7mmkeGyOCfyewDNjVos0PAHcBu4v9zwNnMEW4\n",
       "12q1V7ar1SrVarVN9yRpbqnX69Tr9bbtKiWOsYkI7Y3EYuowkxdVTyKC/FTgBeBPgHuAa1u8XqPR\n",
       "aDX5V7+rVCq0PnHT7Krge0TN4v03OcvLhPsIcDOwAtgBXEjU1JcD1wPnFe1+BXgn8DJwL3AJscja\n",
       "zHAfUIsWjbB37zO97kZ6CxcuZs+ep3vdDfWZToT7bDPcJWk/TRXufkJVkhIy3CUpIcNdkhIy3CUp\n",
       "IcNdkhIy3CUpIcNdkhIy3CUpIcNdkhIy3CUpIcNdkhIy3CUpIcNdkhIy3CUpIcO9S2byn1PUnxy7\n",
       "wTZXx89w75K5+guWgWM32Obq+BnukpSQ4S5JCfXTv9mrA6t73QlJGjDbgGqvOyFJkiRJkiRJkjSb\n",
       "5ve6A5pVjudgGB+nfrqQRIlMvPz0e3rWC3XCRcChve6EJmkO80OmeSwNZxzdMw9oFD9vBG4CfghY\n",
       "CYwBu3vXNe2n8bEcD4YLgBuBI4BR4L971C9N78eB64FjgTcD/9Tb7mjQDU3YPghYDHwSOLvY/haw\n",
       "ATi4+13TAZg4IRo/C7sROHfC/RPHXN1XYfLE9TTgC8DxwIXADuAt3e1Wdzlz77yXi9sLgE8QHzj4\n",
       "fuDVwK8DdwK/BbzQk95pfzWAw4E/BI4BHgbWAKuAM4D3FLc7ix9113ziPdcgJlMnAE8B5wPfBFYA\n",
       "lwMfAf6yR33UgJrHd9bVDwY+A/w1MWtYTMwgthKnhuNOxa+D6EfNi3CnAtuBXwOWFPedBLwXOBNY\n",
       "C1xDhIl652Lgr4BbgWHgR4AXgT8gJlYQZbQTetK7LjBMZtcQMWt4GVhABMMLwL8B3wc8BDxDzPbu\n",
       "JmYWrwG+CFzGvl869d74H+mXiv1GcbsK+CjwIeC7iED/GvBHwN8DzxElAM/EuuMs4gxq3GJiMnUu\n",
       "sa51InHW/Cjw58T61v8QZ1ufJSZcaRdVVc584HfZV78bImYHtxe3ELP3rwI/VewfA7yPmFXcC1zZ\n",
       "rc6qrQVN+99LLMJdRMwA30KM5ZeJU/tdwNXA0cC1wD3ELFGdNwI8AdxBnDlBhPsdwGuL/XcQf3hP\n",
       "IS5e+Efg88BdwE90s7MaLJcQX3h2E/FLtRj4W+A3idnAo8DGou0FxOr8xHWO1+Clc/3kR4mxW1bs\n",
       "XwbcB7yLmKlfS1xGdzgRLAA/CPxpsX1StzoqIP7Y3gr8NPAPRJAfR6yHnFW0qQBfAT7IvrOx1ze9\n",
       "TsqZuwuqB+4IYnF0HfBx4jS8QZRYHgGuI8ozZxKhvgVYD5xcbAP8F/C/Xe21WplPjN0biNP4l4jS\n",
       "2Qjwq8XtLxZtjiBmhocB7ywev50Y47Fud3yOe4H4g7yHmJ2/lbjMcZh4Xz0FPEuU0l4L3E9ccvx0\n",
       "8fzxcU/JcD9wzwNHETXWh4jZ22lEuP8O8BjwbqLW/nbgj4nA+A9iRq/eG79kbryu/hhx6n488Dgx\n",
       "43sbcXXF+4ixO6e4/wTgx4AriDFX7ywBbibG7lJiLes54v33s8CTxGdLniICfnymnjbYwQXVsi4j\n",
       "Fm8eAL5O1F7nETP2fy/a7AKWEr949xGzPPVehXhzv0SsgVwHnA7cRqybnF20O5sou/0zsYB6OBH4\n",
       "f0OU2h7oaq/V7FCinv4XRN39N4jZ+TJi/eMK4OeICdW/Fs9JHeqaPe8CPjdh/yDg/cRM4nHgt9m3\n",
       "uKPemk/UZ49l34fG1hNv/CuBRcV97yauiFlBLKQ+TtRxv8q+Wq76w2FEmeWaCfe9iVgLqRDBfj+w\n",
       "qftd06CrEJ8yPXHCffOK/dN70iO18h6iLj5KfHjll4r7P0aUVyZaDvwe8IFi/xzgKqLerv7zYfZd\n",
       "odRcaj6f+CMtHZDT2fc9FSlX3gfcEUSpbGWxfwHwKeLKig8SX/8A8KoJz1kHfJpYR1F/+yIxXhPL\n",
       "zJacNWvu4jtn7+ovNxDlMoiyzJeIWvulRJ12/NPCryNKMRUSf3oxmcW97oBy88qj/raAuNrlEGIh\n",
       "/Dai7v5G4nLGe4lSzIPE1TEaPM7WpTnqEuLqmGuZ/F365xCh7nfsS9KAqRAfV19a7B+Msz1JSuF0\n",
       "4kvbJEnJuPgtSQm5+C1JkiRJkiRJkiRJkiRJkiRJktRF/w8QvJTGmfHuDwAAAABJRU5ErkJggg==\n"
      ],
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11395b290>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = np.arange(len(feature_names))\n",
    "plt.bar(x, logreg.coef_.ravel())\n",
    "_ = plt.xticks(x + 0.5, feature_names, rotation=30)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, survival is slightly positively linked with Fare (the higher the fare, the higher the likelyhood the model will predict survival) while passenger from first class and lower ages are predicted to survive more often than older people from the 3rd class.\n",
    "\n",
    "First-class cabins were closer to the lifeboats and children and women reportedly had the priority. Our model seems to capture that historical data. We will see later if the sex of the passenger can be used as an informative predictor to increase the predictive accuracy of the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Alternative evaluation metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is possible to see the details of the false positive and false negative errors by computing the confusion matrix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[98 12]\n",
      " [36 33]]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "cm = confusion_matrix(target_test, target_predicted)\n",
    "print(cm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The true labeling are seen as the rows and the predicted labels are the columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": [
       "iVBORw0KGgoAAAANSUhEUgAAAToAAAEaCAYAAACB7ptqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\n",
       "AAALEgAACxIB0t1+/AAAHsdJREFUeJzt3Xm8E9X5x/FPclkFWVwLVERR3EABBQVUtFqLVKtW60Yt\n",
       "1q11r/vPuqG1tbVaa7HaVm1V3FArWrQKboiIsoggqLiiWAEXQAEVRLy/P54zZO6QzE1yk5lk8n33\n",
       "Na+bmcxkzr01D+ecOec8ICIiIiIiIiIiIiIiIiIiIiIiEpnWwFjgM2B0Ez5nGDCuJCWK3x7A3LgL\n",
       "IVKLjgamA8uBBcB/gUEl+NxjgClAugSfVQ2+BbaMuxASrlb+Y5SGzgauA64ENgE2A/4K/KgEn705\n",
       "8CYWAGpFKuS9ZpGVQkTWao/V4g4NOacl8GfgQ7ddB7Rw7+0F/A8Llh9htcFj3XuXA6uAr909jgNG\n",
       "AKN8n90NC4LeP7LHAu8Ay4B3sZqmd/w533UDgWlYk3gqMMD33gTgCmCS+5xxwIY5fjev/OcBH7vy\n",
       "HwwMxQL0YuD/fOf3B14AlrpzRwLN3XsT3e+ywv2+P/F9/vnAQuB2d+wDd013d48+br8z8AmwZ47y\n",
       "ikgRhgCrCa/NXwFMBjZy2/PuGNiXdjUWwOqA/YEvsAAKcBlwh++zLiN3oGsDfA5s7d7bFNjevT6W\n",
       "TKDbAAs0w9x1RwJLgI7u/QnAW8BWQCvgGeCqHL+bV/6LXflPAD4F7nLl2R74EquZAvTFgl3aHXsN\n",
       "ONP3ecGmq/f5V2EBsRUNAx3unq9i/ZnjgKtzlFVEijQMq2mEeRsLiJ79gHnu9V5YIPAHyo+wYADr\n",
       "1uCC+91oGOiWAj/GvvR+x5IJdMcALwbenwwMd6+fAX7te+9k4LHgLxUov9fcXN+Vp5/vnOnAQTmu\n",
       "/xXwoG8/W6BbRaYG7B3zBzqAh4HZwEwyNUQpE/XR1Z7FWC0t7P/7zsD7vv357pj/M/x9cF8CbYso\n",
       "yxfAEcAvsWbhI8A2OcozP3Ds/UCZFvlef9VIeRYD9b5zwYK1//o27nUPV66FWO3zt+RuFns+wZrv\n",
       "YW4BdsCawqsbOVeaSIGu9ryA1TgOCTlnAVbz8nR1x4qxAljPt/+dwPvjsRrjd7AhGDdn+YwPyTQl\n",
       "PZu74+V2E9Zc3Qprnl9E49+b+kbeb4v1gd6C9Wt2DD9dmkqBrvZ8DlyKPWU9CAtCzbG+tj+4c+7B\n",
       "+rC8PrpLadj8LMRMrKN9MyxQXOh7bxNXhjZYreYLYE2Wz3gMq1kdhT3FPALYFqtpecKefDZFW+xB\n",
       "w5funicH3v8Ie8BQiOuxByonAY8Cf2tiGaURCnS16U/YU9OLsSeP84FTgDHu/SuxfqpX3DbdHfOE\n",
       "1VjqA+8/iQ0cfgV7ajrW934aOAurmS3GBtaenOVzFgMHAOdgDw7OdftLcpQpWIZsZQzb9zsXexK8\n",
       "DPgHcG/g/BHYk9WlwGEh9/aOHYTVYL3f82zsgcdRIWUQEREREREREREREckt1aaz1zmtTVvVbXvs\n",
       "ObieUqlrmc89l+S8vgnK9UheMupb9T417jLkbfXCqTTv1L/xEyvE0mk3xF2Eglx5xQguvnRE3MXI\n",
       "W+vmKShdnKhv1ef00BNWvjyylPdbSysriEh0UvHUrRToRCQ6qXiG7irQSQPptl3iLkKi7Tl4r7iL\n",
       "EK90XSy3VaCTBurWV6Arp5oPdGq6ikjiqUYnIomnPjoRSTzV6EQk8dRHJyKJl44n5CjQiUh00qrR\n",
       "iUjSxdRHpxWGRSQ6qXT4lt2ZWMa0OWRSTW4APIHl4h0PdAi7rQKdiEQnlQrf1tUTy4PbD9gJW0K/\n",
       "O5Zk/Aksl8hTNEw6vg4FOhGJTroufFvXtsAUYCWWOOlZ4FDgR1iuDtzPg0NvW7JfQESkMYU3Xedg\n",
       "SZM2wDLWDQW+C2xKJhfvR24/Jz2MEJHoBGptaxa/xbdL3g67Yi6WhnM8lg5zJuumxPQW7cxJgU5E\n",
       "ohPoh6vbqAd1G/VYu7/m7XHZrvqn2wB+C/wPq8V9B1gEdMLSduakpquIRCfdLHzLbhP3syvwY+Bu\n",
       "4D/AcHd8OPBQ2G1VoxOR6BQ3BewBYENgNZZo/XPg98B9wPHAe8DhYR+gQCci0SluwPCeWY4tAfbN\n",
       "9wMU6EQkOlqmSUQST6uXiEjSpdOq0YlI0sWUSVqBTkQioxqdiCReSn10IpJ0KS28KSJJpxqdiCSe\n",
       "+uhEJPFUoxORxIurj06rl4hIZFKpVOiWw4XAq1jeiLuBlihnhIhUqiICXTfgRKAv0AuoA45EOSNE\n",
       "pFKl0qnQLYtl2PJM62FdbesBC1DOCBGpVEXU6JYA1wLzsQD3GVaTU84IEalMweElqxbM4esFr4Zd\n",
       "0h34FdaE/Ry4H/hp4BzljBCRChKotLXs0pOWXXqu3V8x477gFbsAk4HFbv9BYACWK0I5I0Sk8qTT\n",
       "6dAti7nAbkBrLEzuC7wGjKWAnBGVFuiGY9E5Ss+X6HNuwxLrikgORfTRzQLuAKYDr7hj/8ByRnwf\n",
       "G17yPbefU6U1XY/FEtYuLOFnen+9XG34QSW6T6P9BCK1rsgBw1e7za+gnBHlrNF1A17Hou8cYBzQ\n",
       "yr3XG3gRi9YPYoP9DsPa43cBM3znes7ABg3OwgYNAowAzvGdMwdLidYNeAN77DwbuISGf6hjgZHu\n",
       "9Qr3814sC7jnNiy1Whr4IzDV3fsk934KuAGrWj+BpWSLaVlBkepQ5IDhJit303UrLBj0xB4Le027\n",
       "O4DzgJ2wQHQZltJsOnA0NjhwZeCzLsAC5E7AL92xYA3Kv78V8Fd37xuBQ3zvHQHcE7jmXjIp01pg\n",
       "1eFHgRNc2fu77UQskB6CDVbcDvgZMDBLeUTEp4g+upIod9N1Hpl29UtYgGgHtAeec8dvxx4Ze3KF\n",
       "9VewmtxDNNLx6LyP1cIAPgXeBXYF3ga2wZ7k+D0OXI8Fuf2BZ4FVwH7YiOzD3HntgK2BPVx56rGm\n",
       "9tO5CrJ64dS1r9Ntu1C3fpc8ii8SvYnPTmDisxPKd4OELqW+yvd6Des2R2HdXz1XreiHWH7HA4GL\n",
       "sODzDQ1rpf7P/yJwvVdjm4s1l4NWAhOAH7jz7vG9dxrWPPUbmqXsWTXv1D+f00Rit+fgvdhz8F5r\n",
       "93/7m8tL+vlxrV4S9VPXFDalYymwuzt2DBZgAJZjNaZs13V15/0fViNsg2Xo7uvO6QtsEXLvMdg0\n",
       "kaOwoJfNaOA4rLb2uDs2DssO7v2j0AObhjIRawKnsSfFe4fcW0SAdDoVupVLuWt0ufrQhgN/wwLG\n",
       "O8DP3fHb3PEvsT4vr5+uDhiFBbgU1sRcBvwb6x+bA0zBHkDkuvdn2Pib7bC+wGznjXf3eQirLQLc\n",
       "gjW5Z7h7f4wFzDFYP95r2PSUYFNYRALiqtHpKWH51bfqfWrcZUispdNuiLsIida6eQpKFyfqe5z/\n",
       "eOgJb149pJT3W6vSxtGJSILFVKFToBOR6NTVaSl1EUk45YwQkcQr55PVMAp0IhKZWhlHJyI1LJUK\n",
       "37LYBnjZt32OzXtXchwRqUxFDBh+A+jjtp2xMbZjUHIcEalUTVy9ZF9srvoHFJgcR310IhKZJj6M\n",
       "OJLMHHQlxxGRyhSstC17dybL583M59IW2IIeF2R5T8lxRKRyBGt0HbbqQ4et+qzdX/D07cFLPPtj\n",
       "S7194vY/QslxRKQSNaGP7igaLp32HwpIjqManYhEpsg+ujbYg4gTfcd+D9wHHI8t13b4updlKNCJ\n",
       "SGSKHC/8BbBR4FhByXEU6EQkMuXMCxFGgU5EIqNlmkQk8Spx9ZKRIe/VY/PNRETyVomrl7xEZhCe\n",
       "P9t9CuUvFZEiVGLT9bbAfhvWTSEoIpK3uphqdPk8AhmIZbqa6/Z7Y5nvRUQK0sRJ/UXLJ9D9GRiC\n",
       "ZbsHmAkMLluJRCSx6tKp0K1c8n3qOj+w/03Ws0REQlRiH51nPjDIvW6BPW19vWwlEpHEqqvgpdRP\n",
       "Bk4FugAfYit9KiOziBQsrj66fGp0nwBHl60EIlIzioxlHYBbgB2woW0/B94CRgObk5nU/1muD8in\n",
       "RtcdGIs9jPgEeBjYsqjiikhNK/JhxPXAf4HtgB2xESAlzxlxN7YcSiegM3A/DdeFEhHJSxFN1/bA\n",
       "HsA/3f43WCawgnJG5BPoWgOjgNVuuxNolcd1IiINFFGj2wJrSf4LmAHcjE1eKFnOiA2w6V6PAReS\n",
       "qcUd4Y6JiBQkGMo+fn06H8+dHnZJM6AvcBowDRvXG2ymNilnxIzAxSf5ylqf5WYiIqGCtbZOO/Sj\n",
       "0w791u6/+vA/gpf8z23T3P4DWMVrEQXkjAgLdN0aLbWISAGKGEKyCMvj2gN4E1tV+FW3DQf+QAlz\n",
       "RvQEtqdh39wdhZVXRGpdkcs0nQ7chU1YeAcbXlJHiXNGjMDmtu4APIqlHZuEAp2IFKjI6ayzgH5Z\n",
       "juedMyKfp66HuQ9ciEXSnbABfCIiBUmnUqFbueRTo/sKWIONX2mPdfptVrYSiUhilTOYhckn0E0D\n",
       "OmLjV6Zji29OLmehRCSZKnn1klPcz78B44B2WJtZRKQglZgzYmdyD8Lri42zExHJWyU2Xa8lfLTx\n",
       "3iUuS2I9cs+IuIuQWDPfy7lghVSgSqzR7RVVIUSkNuQzzKMclMBaRCITVxYwBToRiUxMcU6BTkSi\n",
       "U8l5XdPAMcClbr8r0L9sJRKRxEqlwrdyySfQ3QgMIJM3YgVKYC0iRWiWSoVuZbtvHufsimX+etnt\n",
       "LwGal61EIpJYRcay94Bl2FTU1ViLcgNKnBzna2xJFM/GwLfFlFZEaluRk/rrseFufch0m5U8Oc5I\n",
       "YAywCfA74HngqjyuExFpoC4dvoUIRsGCkuPk03S9E3gJ2MftHwS8nsd1IiINFDkFrB54Emu6/h1b\n",
       "YKRkyXE8XbEVS8b6btoVmF94eUWklgVrbe/OnMK8WVMau2wQth7mxlhzdW7g/SYlx/H81/chrbD0\n",
       "Y29gKw6LiOQtFWiBdu+9G91777Z2/5lRI7NdttD9/ATrRuuP1eLyTo6TTx9dT6CX27Z2N3kxj+tE\n",
       "RBpolg7fslgPWN+9bgPsB8wG/oMlxYESJsfxm4ENORERKUgRWcA2xWpxYPHqLmA8tghwSZPjnON7\n",
       "ncbWovuwsLKKiDT6ZDWbeUDvLMeXUEBynHwCXVvf62+AR4B/53sDERFPJS68CTZQuB0Na3UiIkWp\n",
       "xNVLmmE1uEHYYL3Qx7ciIo2pq8Aa3VSsP24m8DBwP/Cle68eeLC8RRORpKnELGBekVoBi4HvBd5X\n",
       "oBORgjSrwBWGNwbOxsasiIg0WSXW6OrIDNQTEWmySswZsQi4PKqCiEjyKQuYiCReJY6jy3vUsYhI\n",
       "Piox0C2OrBQiUhPiGjAcV5NZRGpQKpUK3ULUYXlrvHUxN8DWpnsTm+TfIexiBToRiUy6kS3EmcBr\n",
       "ZGZolTxnhIhISRSZHOe7wFDgFjITGUqeM0JEpCSKWI8O4DrgPGyBEU/Jc0aIiJREcFL/nGmTmTN9\n",
       "ctglB2DLpL+MpTzMpiQ5I0RESiJYn+vVbyC9+g1cu3/f364NXjIQa6YOxebdtwNGUYacESIiJVGX\n",
       "SoVuWfwa2AxLynUk8DRwDBHkjBARKUoJxgt7TdTfU+KcESIiJdHEmRHPug3KkDNCRKQk0uv00kVD\n",
       "gU5EIlOJ69GJiJRUJU7qFxEpqUrMAiYiUlJx1ehqfRzdgcAFJfqsFSX6HJHESjXyv3KphRqdl582\n",
       "m7Fkln1pKuW9FWlEXHldq6lG1wZ4FMszOxsbIDgPW5cKYBfgGfd6BDZNZBJwB/ACsL3vsyYAOwPH\n",
       "AiOxaSXvBe41H1sDqzvwGDAdmAhs487Zwn3uK8CVTf/1RJIvlQrfyqWaAt0Q4EOgN9ALeLyR87cF\n",
       "9gGOBkaTGTndCZsj95Lv3GVYAN3L7R/gPn8N8A/gdCyQngfc6M65HvgrsCOwoLhfSaS2FDEFrCSq\n",
       "qen6CnANNvXjEay2lks9Nhduldu/D1uFdAQW8O7Pcs1o4AistnckcAPQFptU7D+/hfs5EDjEvb4T\n",
       "+EOuwtx+w9VrX+/UfxC9+w8KKbpIfGZMmcSMKWFfraaJ6aFrVQW6t4A+wA+xpuLTWN+bVyttFTj/\n",
       "S9/rBVgOjF5YoPuFO+7vVxsL/A7oCPR1n78+sNTdt2jDTzu/KZeLRKbvrrvTd9fd1+7fOjLnv99F\n",
       "KXI9uiarpqZrJ2AlcBdWs+uD9dHt4t4/1Hdutr/maOwJaztgTpbzVgDTgL9gQa8ea9LOAw7znb+j\n",
       "e/08VvMDGFbMLyRSa9RH17hewBRsAb5LgN8AV2B9ZdOw2p1XQ8u2EN8DWNP0Pt+x4HmjyfTpeYZh\n",
       "KyTMxALkj9zxM4FTsSZ15yz3E5GAVCNbFq2w7/1MLGfEVe54Qclx4moy15L6J1//JO4yJFabZtXU\n",
       "+1J9BmzdEUoXJ+qnvft56An9tmyf7X7rYV1RzbC++XOxCsenwNVYS60jIQlyqqlGJyJVrsimq9ff\n",
       "3gIb8rWUApPjKNCJSGSKDHRprOn6ETZW9lWUHEdEKlVwmtf0F55j+ovPNXbZt9j42fbAOGDvwPuN\n",
       "JsdRH135qY+ujNRHV16l7qOb+f6y0BN6b96usftdAnwFnIAN8PeS4zyDTRLISk1XEYlO4Y9dNyLz\n",
       "RLU18H1s5IWS44hIZSpimaZO2MOGtNtGAU9hwU7JcUSk8hTRBp6NzVQKUnIcEalMcU0BU6ATkcgo\n",
       "OY6IJJ5WLxGRxFPTVUQST01XEUk8BToRSbxyZvoKo0AnIpFRAmsRST4FOhFJuiKmgJWEAp2IRCau\n",
       "cXRavUREolP46iWbkVlscw5whjteUM4IBToRiUw6lQrdslgNnAXsAOyGJaTaDssP8QTQA1vNJGe+\n",
       "CFCgE5EIFZEFbBG2jDpYStLXgS4UmDNCfXQiEpkmTgHrhuVznoJyRohIpQrGucmTnuWFSRPzubQt\n",
       "8G8sn/LywHvKGVEBlDOijJQzorxKnTNiwWerQk/o3KFltvs1Bx4BHgP+7I7NRTkjRKQSpRr5X9ZL\n",
       "4FbgNTJBDpQzQkQqVRFddIOAnwKvYHkiAC4Efo9yRohIJSoi0E0id8tTOSNEpPJo4U0RSTwtpS4i\n",
       "iaeFN0Uk8eJavUTDS0Qk8VSjE5HIaD06EUk89dGJSOLFFejURycNzJz6fNxFSLQZUybFXYRYFTEF\n",
       "rCQU6KSBWQp0ZVXrgS6dCt/KRU1XEYmO+uhEJOnieuqq9ejKbwIwOO5CiBTpWWzdt1IIXRzTWYol\n",
       "vhEREREREREREYmDhnuJVJDvAocAneMuSAJcB4wBWvqOKeCVWF3cBZCqdBAwDGiHfSk/xTKqS+Fe\n",
       "wpIxXwMsA2aQeTqpUREiMdsDGAe8AZwN9MbS0kn+vFrcTljO0iVYGr/9YitRQulfDClUM+AbYCT2\n",
       "xVwO9MT+WxoPTAXeiq101WdD4AXgMGA+1iXwR+zveDywML6iidS2zcmkngPLon4d8AnWpJX8DQYe\n",
       "da+9mUrDgfexoCciMUgBbYDRwPnApr73ngA2i6NQVawVMBZr/nuGB/alidR0lXylgW99+98HDgRe\n",
       "xR5q9cc608+IvmhVrT025ekhrEvgMaxWfDgwLcZyJYoCneTDH+SOBLoAM4FOQHfsi/oVMAJYGUP5\n",
       "qkkdsAZ7cj0Ye6hzL3AtcBT29/sAmB5XAZNIq5dIIa7B/pvZChgK7AOsB3wZZ6GqzBr38wLgdGyI\n",
       "znfdscexSe0iEpONgInu9c3ASe71UdjwCMnf/sCfsCeuM7BgB/B3oG9chRKpdSmseXoZcAXwpO+9\n",
       "WWgZqnz4Zzs0A24FJgMHu2P7oeZq2WhmhIRJY6P0N3U/uwHHATcBnwGnAS2wcV8SzpvtcDIwGxuS\n",
       "MwxYhTX/LwYuR2MQy0IPIyQfl2FPVK8DzgF2wYZFLMa+nB/EV7SqkXLbvdh4wzOwhzojgAVYE/bB\n",
       "uAonIjAAeAU4z+1vDnRAU76K0RG4HqvZeVThEImBv0vD61vaApv2NSD64iTCer7X22OT+X+L1Yyl\n",
       "zNRHJ9l4/UkHYvkCtgNeBPoAl2L9SO/EUrLqsjE2vjCF/f0GANsC6wOjsDGJ72BzXKWMFOgk6Hjs\n",
       "i/kRMAT7Uh6OTTp/E5t/2QxbQ03C/RL4H/b3HI0tZbUSe9I6FFvxZS6aASESuSFY/9t52KwHz67A\n",
       "QCzg9YmhXNWmBVZ7aw78EziLhvOCOwNdYyiXSM3z1/CvBD4HbkP9SE3RFTgCm1VyM3Ai9g+JiMTs\n",
       "AvdzE+A/2ArC58dXnKq1I/Bn9/o7wLHYEJ3bgZ1jKpOIYB3oN2IDgz2DgEXAKbGUqHptiK01Nwob\n",
       "VtIK+1ue5fZFJEbfw5ZfugZ7GCH5yzYm7hpsjTnPelnOkTLSU1eBzJezBzbbYR7WZO0FfIGGP+Qr\n",
       "hQ3NaYYtdLA1NgtiEbYMU1tsifRVcRWwVinQSR221lxn7Enr77DVNLoAe2Md6bOwYRISzpsb/Gts\n",
       "5sOmwFVYsFsOnIoNM9FSTBHTenTirY92LXAL9kUcgq0xl8YCoNaba5y3OOnWWLKgn2E1ue7AD7Gx\n",
       "cq3QQGuRyHk1+oHA3TnO2SKisiTFpdhSVqA5rCIV5SlgEg27MlrmOFdy647lfpgB/ICGLSZ1E8VI\n",
       "f/za5a9tLMKaqydgSwa9iTVpVSNpnP9vtBRL6N0am7jfDZvr+jGZ+cMiErEOWIAD+8IeBzyP5S7o\n",
       "nusiyeoQ4BFgB6y/7mBsufSb3TGJkWp0tW134F/YF/FtYDw2rGRbrGb3fnxFqzorsUHApwDbAHdi\n",
       "S6U3x7oGRCRCwebo+ti0JG+A8MaRl6h6Bf+WrbFpXzcBE7B5rSISo6HAlr79nbFa3GRsYKuE84Lc\n",
       "tsC5ZFpHzbH+uTEol0bFUNO1Nm2NDV7tig0O/hQb39UCG+/1fHxFqzr7YQOtj8TW8JuL5dc4ErgI\n",
       "GygsMdNTtdrhTU/y9Af2xUbvL8eesh4C9ENTlBpTh/29hmH9m78GjsYSBU3AVipZQsP5rSISAa/2\n",
       "fjGZZZhaYYOFz8X66Q6NoVzVbAqZ5N1DgDuwTF67oXGIFUVTwGpDCquBbInVQr7vjh+DNVmvialc\n",
       "1WwA1kTdEFuktCc2vGRTLABq3JxITE7GanMtsS/nU9hKJYfHWagqdhY2C+Jit78/9jBHRGK0O/Zk\n",
       "dTpwujt2BpkvqhSmJZkn1GksheH+8RVHRDybAfu419tiiak1cb9pmmF9nRfFXRCRWuUloK6j4XCi\n",
       "FNZ8/U3kJUouDdcSiVid76d/toOGFIlIYvTHngj+HVtUM6h5tMURESmtFthKJDOwhw893XENJ5Ka\n",
       "pD6FZFoDvIwtg94R6IPV4N51718DPAesjqV0IhFTf03yeNOTwBJQf4wNED4Xy+bVFku3d1AspROJ\n",
       "gZoyyeLNgACrtbUENsfWnNsfS9iyGhgbS+lERErAG0pyHpa7YG/gAOBJ4LLAOSIiVasZ8DD21NXT\n",
       "CxiFraohUnP0r3vyfAO8AJzpOzYb2MptIjVHDyOSwUue7GkP3ICtLXcH9gCiL5lEOCI1RYEuWU7G\n",
       "HjbMB6Zii0L+AngaWxDyvbgKJhInBbrq59XmDsPmrT6DNV+/wvrqppB5EitSkzRguPp5CzyeiI2V\n",
       "uwNYgQ0U3hdL1DIZLQQpIlXuAKwWd6HvWHvgJ9iy3iIiVa8F1hc3B0uzt1P46SIi1asjcAkwC7gV\n",
       "zXwRkQTrScNxdCIiIiIiIiIiIiIiIiIiIiIiEfHyUcwG7gNaN+GzbgMOda9vBrYLOXcwMKCIe7wH\n",
       "bFDAcb8VBd5rBHBOgddIBLQenRTqSyzZTi/ga+CXgfcLGaRcT8O5uq+HnLs3MLCAz/bfo5DjhZ7T\n",
       "lPMlIgp00hTPYYt5DnavH8amoaWBP2JLRc0CTnLnp7B18uYCT2DJezwTgJ3d6yHAS8BMd97m2BS3\n",
       "s7Da5CAsKfcD7h5TyQTBDYHxrhw3k98KPWOA6e6aEwPv/ckdfxLYyB3rDjzmrpkIbJPHPUSkiix3\n",
       "P70l23+BBboVWEACC2wXudctgWlAN+DHWBBKAZ2Ape4Y2PJSfbEANt/3WR3cz8uAs33luBsLeABd\n",
       "gdfc678AF7vXQ7ElrLI1Uef5jnd0P1tjTXJv/1vgKPf6EmCke/0UmdWad3X7XhnVdK1AmgsphWqN\n",
       "1arAajP/xALOVOB9d3w/rGl7mNtvB2wN7IEFqHpgIbYgqF8KW21lou+zPgu879mXhn166wNt3D0O\n",
       "ccf+iwXTxpwJHOxeb+bKOhULdKPd8TuBB909BgL3+65vkcc9JEYKdFKor7A+uqAvAvunYc1Ov6E0\n",
       "3pTMt58rhdWmvs7xXr72AvbBAuxKrGbZKsdn1mPN8qVk/xtIhVIfnZTDOOAUMv+Q9sCSZk8EjsD+\n",
       "u+uEPWDwqwdeBPbEmrqQaV4ux2ptnvHAGb59b2mqicDR7vX+ZJqhubTDAtdKYFsart+Xxtb0w33m\n",
       "c64c88jUVlPAjo3cQ2KmQCeFylbjqg8cvwXrM5uB9XndhK1mPQZ4y713O7bycdCnWB/fg9jDiHvc\n",
       "8bFYk9R7GHEGsAv2sONVrK8Q4HIsUM5x53tN4Fy/x+NYQH4NuArLoOb5AksbORur+V3hjg8Djnfl\n",
       "mwP8KMvnioiIiIiIiIiIiIiIiIiIiIiIiIhUhP8H8IJ50+xjwvYAAAAASUVORK5CYII=\n"
      ],
      "text/plain": [
       "<matplotlib.figure.Figure at 0x113968dd0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def plot_confusion(cm):\n",
    "    plt.imshow(cm, interpolation='nearest', cmap=plt.cm.binary)\n",
    "    plt.title('Confusion matrix')\n",
    "    plt.set_cmap('Blues')\n",
    "    plt.colorbar()\n",
    "\n",
    "    target_names = ['not survived', 'survived']\n",
    "\n",
    "    tick_marks = np.arange(len(target_names))\n",
    "    plt.xticks(tick_marks, target_names, rotation=60)\n",
    "    plt.yticks(tick_marks, target_names)\n",
    "    plt.ylabel('True label')\n",
    "    plt.xlabel('Predicted label')\n",
    "    # Convenience function to adjust plot parameters for a clear layout.\n",
    "    plt.tight_layout()\n",
    "    \n",
    "plot_confusion(cm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[98 12]\n",
      " [36 33]]\n"
     ]
    }
   ],
   "source": [
    "print(cm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can normalize the number of prediction by dividing by the total number of true \"survived\" and \"not survived\" to compute false and true positive rates for survival (in the second column of the confusion matrix)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0.89090909  0.17391304]\n",
      " [ 0.32727273  0.47826087]]\n"
     ]
    }
   ],
   "source": [
    "print(cm.astype(np.float64) / cm.sum(axis=1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can therefore observe that the fact that the target classes are not balanced in the dataset makes the accuracy score not very informative.\n",
    "\n",
    "scikit-learn provides alternative classification metrics to evaluate models performance on imbalanced data such as precision, recall and f1 score:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "not survived       0.73      0.89      0.80       110\n",
      "    survived       0.73      0.48      0.58        69\n",
      "\n",
      " avg / total       0.73      0.73      0.72       179\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import classification_report\n",
    "\n",
    "print(classification_report(target_test, target_predicted,\n",
    "                            target_names=['not survived', 'survived']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another way to quantify the quality of a binary classifier on imbalanced data is to compute the precision, recall and f1-score of a model (at the default fixed decision threshold of 0.5)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Logistic Regression is a probabilistic models: instead of just predicting a binary outcome (survived or not) given the input features it can also estimates the posterior probability of the outcome given the input features using the `predict_proba` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.75263264,  0.24736736],\n",
       "       [ 0.75824771,  0.24175229],\n",
       "       [ 0.58542437,  0.41457563],\n",
       "       [ 0.25224882,  0.74775118],\n",
       "       [ 0.75817844,  0.24182156]])"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "target_predicted_proba = logreg.predict_proba(features_test)\n",
    "target_predicted_proba[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default the decision threshold is 0.5: if we vary the decision threshold from 0 to 1 we could generate a family of binary classifier models that address all the possible trade offs between false positive and false negative prediction errors.\n",
    "\n",
    "We can summarize the performance of a binary classifier for all the possible thresholds by plotting the ROC curve and quantifying the Area under the ROC curve:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import roc_curve\n",
    "from sklearn.metrics import auc\n",
    "\n",
    "def plot_roc_curve(target_test, target_predicted_proba):\n",
    "    fpr, tpr, thresholds = roc_curve(target_test, target_predicted_proba[:, 1])\n",
    "    \n",
    "    roc_auc = auc(fpr, tpr)\n",
    "    # Plot ROC curve\n",
    "    plt.plot(fpr, tpr, label='ROC curve (area = %0.3f)' % roc_auc)\n",
    "    plt.plot([0, 1], [0, 1], 'k--')  # random predictions curve\n",
    "    plt.xlim([0.0, 1.0])\n",
    "    plt.ylim([0.0, 1.0])\n",
    "    plt.xlabel('False Positive Rate or (1 - Specifity)')\n",
    "    plt.ylabel('True Positive Rate or (Sensitivity)')\n",
    "    plt.title('Receiver Operating Characteristic')\n",
    "    plt.legend(loc=\"lower right\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": [
       "iVBORw0KGgoAAAANSUhEUgAAAYYAAAEZCAYAAACTsIJzAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\n",
       "AAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8E3X6wPFPKVCuQsuNXFVEBRVUThWhIgIVbxEURHA9\n",
       "f+tNRVRQWBVdXEXXW1AQ1lVWBAVUQEWrKCpXyw0KIjcKFClnS+n398czadI0TaclyeR43q9XXs1M\n",
       "JjNPpsl853uDUkoppZRSSimllFJKKaWUUkoppZRSSimllFJKOWoV0NXpIMLIo8AEh479LvCUQ8cO\n",
       "tIHAvHK+V7+TSnn4HTgMHAB2Af8BajoZUIAlAM8Cm5HP+QvwkIPxpAJbQ3i8OOA+YCVw0Dr2h8BZ\n",
       "1uuTgCdDGE9JRiPfvVB4l+hJDCNCBacDUGVmgMuBRKAtcDYw0tGIyqdiCeunARcDaUANYBBwB/Dv\n",
       "IMQQZz3Cyb+RhOFeIBk4DfgEuCwIx4oPwj4j4dhKRZ1NQHeP5eeAzzyWOwMLgX1AFtDN47XayB3n\n",
       "diAb+Njjtcut7fcBPyAJjsvv1jFPQu7ikz1eOxfYjfuH/jdgjbX/uUAzj20LgL8DvwIbfXy2S4Aj\n",
       "QGOv9R2BfOAUazkDyVX8DOxHLpyeMfk7BxnA09ZnPAy0AG6xYs6x4rrD2ra6Fc9xJIeWAzSi6N1y\n",
       "ivW5bkZyObuBxzyOVxWYbJ2PNcDDlJwDaWl9zvYlvA7y/3sV+NSK5yfc5wUkYdmCnJclQBeP10YD\n",
       "H1mx70f+Vx2AH5FztQN4Bajk8Z4zgS+BvUgO9VGgF5AL5CHnJdPathbwjrWfbchdvuvmcwhyzscB\n",
       "e6zXhgALrNfjgBeBP6zYVljHvsM6Tq51rJnW9r8j3xeQ795jwAbrnCwBmvg4d0pFrU24fxBNkB/Q\n",
       "E9ZyY+RH19ta7mEt17GWPwM+QH7AFYGLrPXnIj/IDsgP9GbrOK4LhGdiNB+4zSOefwGvW8+vQi76\n",
       "pyMXhBHIxcClAClTTkKKjLz9E/imhM/9O3C79TwDufC0BqrhvthB6ecgw9pXKyvGisjd+MnW612B\n",
       "Q8g5AUlUvC/koyieMLxlfaY2wFHkHHh+plpWbCuQC7cvdyHn2p93rc/THrkgvof8T10GIolkBWAo\n",
       "sBOobL02GrnIXmktVwHOQxLeCkBzJPG633o90Xr/g9Y+aljbus7BFK/YPgbeQBLDekjC7UpkhwDH\n",
       "gLutY1WhaMLQC7mgu4pFTwcaWs99FZ95fieHIee1pbV8NnITpFTM+B333WsB8mN03ZUNp/iPdS5y\n",
       "oW+E3PnW8rHPNyj+w1uHO+Hw/BHeiiQOIInIFtx3pXOQu1CXCshFtqm1XICU2ZfkbYpe5Dz9iNyt\n",
       "glxon/F4rRVyR1kB/+fA9d7RfmIAOaf3Wc9TKZ4wjKZ4wnCSx+s/A/2s5xuBSz1eu9XH/lxGIJ/T\n",
       "n0nAeI/lNGCtn+2zcef+RiMJoz8PADOs5zcCS0vYbjRF6xgaIAliFY91NwJfW8+HIDkqT0NwJwzd\n",
       "gfVAJ4oXcU+ieB2D53dyPXBFCXGqctA6hshjkDvzmshFqzvuoofmwPVIsYDrcSFy59UUuUjs97HP\n",
       "5kC61/uaUPRi5zIDON/aZ1fkovi9x37+7bGPvdZ6z6IhfxW5u5EEzJeTkDtlX/vZguRu6uL/HJQU\n",
       "QxpSJLPX2v4y3DkMu3Z5PD+M3F274vY83jY/+9hLyZ/f0x8ez494HAukon4N8BfyWWoh56Wk45+G\n",
       "FEvtRL4bY3B/9qbAbzbiATnvlaz9uM77m0jOwcXf//5rpIjsNeTzvYXkWOxogu+iSVVOmjBEtu+Q\n",
       "MuGx1vIW5C4u2eORiNRDbEWy175yDFuQC4Ln+2oA//Ox7T7gC6A/MICid/hbkKIDz/1URy66LsbP\n",
       "5/kKuWP0Lh92rfvaY10zr+fHkITF3znwFUMCMN16vb61/ee4K6V9xevvM3jbiTvHhNdzb/ORz9mu\n",
       "DPv3dBFSrHI9UlyXjFzsPSvYvWN/A0lITkW+GyNwXxe2ULT+wlOB1/JWJNdWB/d5r0XRuqrSztsr\n",
       "yE1OayTBGmbzfVut+FWAaMIQ+V5Cyn07IeXNVwA9kfLnKkiuojFygZqD1AckIXd3rnbgE5Dy7Y7I\n",
       "RaQ60Ieid6Ke3gcGA9dZz13eRCoBW1vLtZCLlF3zrcd0ax/xSEXyf6y4XXeFccBNSBFSNaQYbBpy\n",
       "AfF3DvB4v0tl67EHudilWe91+QO52Hk2CS5LS6YPkSKwJCuGeyj5Qvcr8jk/QOo2Klvx34AUkZV2\n",
       "7ESk8nqP9d4nKL0pcw2kaPIwcAbwfx6vfYbkYO5HEtBE3HUMfyDFaK54diI3DOOs7SogFft2+xq0\n",
       "R77DlaxYjiJFn65jlZRAgRRBPoUkDnFIPY/WMZwATRgi3x6k1ctwpJjgKuTi/Cdyx5eO+/88CLmz\n",
       "Xof82Fzl6EuRit1XkeKmX5Ey+ZIuYLOQH+FOpL29yydI7mUqcqe6EqlUdLFzp30dUg8wF7lg/Qf5\n",
       "4d/rtZ//IBWxrspV12cp6RyUdNd8wHrvh8hnvxF3yxeQc/UBUqSSjVwojdc+/H2uJ62YNiEXzmlI\n",
       "BXBJ7sNdpLIPaWlzFXLOXcfyPp5rea71+AWpizpC0YpuX+99CMn55SB1F1M9tjmA1I9cgZznX3DX\n",
       "EU2z/u5FKo1BvjOVcbdKm4a7CK+kuF3ralrHz7Zi34M0bABp6dQaOR8zKG4c8v/7AvneTaBoXYcK\n",
       "MxORC9BKP9u8jFyIluNuCaKUP99QtJI7kvwfJbe8UiosBDvHMAl3s0FfLkPuPFsiZdNvBDkeFT3C\n",
       "rWNaSRoild8VkCaYQynaf0SpsBPshGEBkv0ryZVIMQhIE78kpNmbUqUpSwWwkyojdS85SP3JJ7j7\n",
       "fSgVlkoaliBUGlO8KV8TijbHU8rbxU4HUAZbKNoyR6mwFw6Vz95FApFyJ6iUUlHJ6RzDdoq2625i\n",
       "rSuiRYsWZuNG7b+ilFJltJFy9PFwOscwC/dQBZ2R3prFipE2btyIMUYfxjBq1CjHYwiXh54LPRd6\n",
       "LtyPZcuW0aZNG/r06cP27dvp1s2A9CUps2DnGFwddeoidQmjcA/M9hbSw/QypK32IWSUS6WUihn5\n",
       "+XDHHXDwYPn38csvL7Ju3bO0afM8VasO4oEH4li9uvz7C3bCcKONbe4JcgxKKRW2Dh+GDz6AyZNL\n",
       "37Yk69Z1oH79LGrXdg9v1r8/9O1bvv05Xcegyig1NdXpEMKGngs3PRdukXguKlWCfv1K365kXUrf\n",
       "pAwipZOQMUYbKymlwt+hQ/Duu3D8eKmbAnD0KDz9NOTkBD6WuLg4KMd1XhMGpZQKoB9+kCKc68sw\n",
       "fORJJ8Ejj/jfJi8vjzFjxpCUlMSDDz5oa7/lTRi0KEkppQLslFPg5ZcDt7/MzEyGDBlC06ZNGT9+\n",
       "fOlvOEGaMCillB9Hj8KaNfa3X78+cMd25RLeeOMNnn/+eQYNGuTKBQSVJgxKKeXHe+9JMU+zZqVv\n",
       "69K9e+nb2PHAAw+wZcsWsrKyOOkkXxMqBofWMSillB9vvglZWfI31A4cOECNGjXKnUvQOgallIoy\n",
       "iYl2p70OLKeHxFBKqZiXl5fH3r17nQ6jkCYMSinloMzMTDp06MDrr4fPNB2aMCillAPy8vIYNWoU\n",
       "vXr1Ij09nZEjRzodUiGtY1BKRTVj4Jln4Ntvy/f+rVvh4gBPDeXZLyHULY7s0FZJSqmoNno0fPwx\n",
       "jB0LFcpZRnLWWdI7OVDGjRtH3bp1g94vQYfEUEopLy++CG+8AQsWQIMYnE1em6sqpWLS7NmQmVl8\n",
       "/e7dMGtW7CYKJ0Irn5VSEe3552HDBpnwxvNRuzZ8/XXZeiwHWmZmJt98841zAZST5hiUUhHv1luh\n",
       "Wzeno3DzHOMonJqh2qUJg1JR6o8/OKHpHSPFvn1OR1BUuLc4skMTBqWi1JgxMHcuNGnidCTB1aAB\n",
       "NG/udBTi1Vdf5cknnwzpSKjBEClRa6skpcro7ruhdWv5q0Jj6dKlNGrUKGxyCdoqSSmlHNauXTun\n",
       "QwgIbZWklFKqCM0xKKVUGbhaHFWoUIFRo0Y5HU5QaI5BKaVsco2EunTpUm6//XanwwkazTEoFeH+\n",
       "9S/46KPi6zdtgqeeCn080cipuZedogmDUhHu55/h6qt9zzPcpk3o44lGI0aMYO3atRHbL6GsypLk\n",
       "VQEMkBukWPzR5qpKlaBvX7jhBvmrguPIkSNUqVIl4nIJwWiuWgG4GrgRuMBajgOOAz8C/wU+QRIL\n",
       "pVQITZ8OCxfK8+XLJWFQwVO1alWnQwgpfynJd8ACYBaQhTunkACcC1wJdAG6BjNAi+YYlPKQlgaN\n",
       "G0OrVhAXB4MGQb16TkcV+fLy8sjOzqZhw4ZOhxIQwcgxXIrvYqNc4CfrkVDWAyqlAuO66ySBUIHh\n",
       "GuOoT58+PPPMM06H4yh/zVVdicI44MxStlFKqYjkPffymDFjnA7JcXZaJa0FxgOVgInAB8D+YAal\n",
       "lFKhEA0joQaDnQ5uE4ALgZuBFGAl8D4Q4OmxlVIqtJYuXUp6ejqzZ8/WRMGD3X4M8cAZQCtgN7Ac\n",
       "GArcBfQPTmhKxabDh+HQIf/b5OWFJpZod9tttzkdQliykzC8CFwBfA2MARZZ68cC64MUl1Ixq0sX\n",
       "6bVc0c+vs0IFqF8/dDGp2GInYVgBjAR83cN0Cmw4SqnDh+HHH+GMM5yOJHpkZmayfft2Lr/8cqdD\n",
       "iQh26hgGUTxRmG/9/auU9/YG1gG/AsN9vF4XmIv0k1gFDLERj1JK2eLZ4uhQaeVzqpC/HENVoBpy\n",
       "8a7tsb4m0NjGvuOBV4EewHZgMdJZbq3HNvcAmcCj1nHWA+8B+fbCVyq8FRTA+efDX6XdQnnYtAkq\n",
       "Vw5eTLFCWxyVn7+E4U7gfuAkYKnH+gPIBb80HYENwO/W8lTgKoomDDsB1zBfNYG9aKKgosjx47B0\n",
       "Kaxebf89lStDSkrQQooJ48ePZ+TIkTExEmow2Dlb9wKvlGPffYFegGvQ8puQOol7PbapgFRqnwYk\n",
       "Av2AOT72pUNiqIh07BhUqyZ/VeisWbOGpKSkmM8lBGNIjO7IRXsHcK2P12eUsm87V/LHkPqFVKAF\n",
       "8CXQFsmVFDF69OjC56mpqaSmptrYvVKhkZsLQ4fC0aNF1xcUOBNPrGvdurXTITgiIyODjIyME96P\n",
       "v5TkH8Ao4F18X+RvKWXfnYHRSAU0SD1CAdLM1eVzpAnsD9byfKSSeonXvjTHoMLa9u1w5pnwwgvF\n",
       "X6tTR+ZLUMFhjNGiohKUN8dg5w0VKV+5f0WkMvkSJNexCBnC27OOYRwyvMY/gAZIXUYbINtrX5ow\n",
       "qLC2fTt07Ch/VWi4ZlU7cOAA48aNczqcsBSMoiSX35Ampf9DipbsXqHzkVZH85AWSu8gicKd1utv\n",
       "Ac8Ak5Ce1BWAhymeKCgVNvLy4H//g3yvW6V9+5yJJ1Z5tjgaP3680+FEHTspSXXgcuAG4DxgNpJI\n",
       "LAhiXN40x6DCwrJl0KMHXHVV8ddatICRI0MfUyyJtbmXT1QwcwyHkITgf0Ay8DKQgeQClIo5KSkw\n",
       "aZLTUcSmZ555hqVLl2q/hCCzO4heKjJYXm+ko1q/YAWklFMKCuC338Bf5nTLltDFo4p77LHHqFSp\n",
       "kuYSgsxOwvA70qT0f8Aw4GAwA1LKKZ99JnMnl3YjetFFoYlHFVdZu4SHhJ2EoQ2QE+xAlHJaXh70\n",
       "7g3TpzsdicrLy2PXrl00a9bM6VBikr9B9FyD3o1Bej57Pl4OclxKqRiVmZlJhw4deOmll5wOJWb5\n",
       "yzGssf4upWgT1TjsN1lVKihGj4Y33gjsPo8ehSuuCOw+lX2+WhwpZ/hLGGZbfw8DH3q9ppXPylGb\n",
       "NsGIEdA/wPMHJiUFdn/KHh0JNbzYqWN4lOIJg691SoVUrVrQoIHTUahA+PXXX0lPT9d+CWHCX8KQ\n",
       "BlyGzL3wMu5OEomAjhWpTtjo0bByZfneu3gxXHJJQMNRDurXTwshwom/pLktcC7wJPC4x7Y5wDdA\n",
       "KAcB0J7PUej00+Huu6GxnWmffOjRQ3INSinfgjmIXiWczyFowhCFTj8dZs2Svyo2ZGZmsn79em64\n",
       "4QanQ4kJ5U0Y/DVXnWb9XQas9HqsKOuBlFKxy3Pu5QKdpCLs+atjuN/6qw34lFLlpi2OIo+/HMMO\n",
       "6+9uYCsyNEYC0hNaR51XSpXq3XffpVevXqSnpzN79mxNFCKEneaqC4AuyMiq85BB9PoDA4MYl4pA\n",
       "69bBwTKMpOU9DaaKPl27dtVcQgSyUymRibROuheoCjyHTKzTNohxedPK5zB37BhUrQrnnGP/PZUr\n",
       "w+zZMvWlUirwgjkfA8D5SA7hVmvZXxGUikHGQIUKsMR7tm4VM3Tu5ehh5wL/ANLT+WNgNdAC6ceg\n",
       "lFKFLY5uv/12p0NRARIpybsWJTloxw444wwZlrokxkDNmrB7d+jiUs7znntZ6xLCSzCLkk4HHgJS\n",
       "PLY3QPeyHkxFpgMHZEyi0oaviNfJXmOGzr0c3ewkDNOAN4C3gePBDUeFqwoVoEoVp6NQ4eKVV17R\n",
       "uZejmJ0kfinQLtiBlEKLkkJg82a44w447pX8HzoE+/fDmjW+36diT35+PvHx8ZpLCHPBLEqaDdwN\n",
       "zAByPdZnl/VgKrxt3iz1CS++WPw1vSlUnipWtNugUUUiO//dIUidwkNe608OeDTKccnJMmqpUiB1\n",
       "CZs3b6Zly5ZOh6JCyE7CkBLsIFTZLVkCn34a2H1u3hzY/anI5mpxdOGFF/L66687HY4KITv9GKoj\n",
       "8zFMsJZbApcHLSJly9SpsHBhYPfZvDmkpwd2nyry5OXl8cQTT9CrVy8eeughXnvtNadDUiFmJ8cw\n",
       "CamAvsBa3gF8BAT4flWVVc+e8JB3AZ9SJyAzM5PBgwfTvHlzbXEUw+wkDC2AfoBrZo1DwQtH+bNz\n",
       "J6xeLc83b4aGDZ2NR0WfXbt2MWzYMG666SZtcRTD7CQMucjgeS4tKNo6SYXI00/D/PnQpIksn3uu\n",
       "s/Go6JOWluZ0CCoM2EkYRgNzgSbA+8CFSEslFWIFBXD//fB//+d0JEqpaGan8vkL4DrgFiRhaIcO\n",
       "oqdURFu2bBlvv/2202GoMOUvYUgBkqzne4DDQE/gZqBycMNS3g4dgp9/hqSk0rdVqiSuFke9e/em\n",
       "atWqpb9BxSR/CcOHQDXr+TnImEmbrefaqDmEcnPhmmugbVvo39/paFSkWrZsGe3btyczM5OsrCwG\n",
       "DtRJGJVv/podrEDmdwZ4HigAHkYSk+XA2cENrYiYHSspPx/69ZNB7KZOBR2JQJXHf//7Xx588EFe\n",
       "eOEFbXEUQ8o7VpK/N6zEffHPRCbrmevjtVCIqYThn/+Ejz6S5zk5cPLJMGsWJCQ4G5eKXDt27ADQ\n",
       "fgkxJhiD6H2DFB/tROoavrbWn4T95qq9gZeAeGTY7rE+tkkFXgQqIXUZqTb3HbUWL4brr4dLLpHl\n",
       "Nm1kfmSlyksTBFUW/hKGB4D+QEOgC+Cav6sBMMLGvuOBV4EewHZgMTALWOuxTRLwGtAL2AbULUPs\n",
       "Ue3UU6F9e6ejUJGooKCAChV0WnZVfv4SBgN84GN9psfzOGs7XzoCG4DfreWpwFUUTRgGANORRAEk\n",
       "xxCTJk2C5cvl+fLlMGCAs/GoyJOXl8fTTz/NL7/8wtSpU50OR0Uwf7cVGcAw4DQfr50ODAe+9fP+\n",
       "xsBWj+Vt1jpPLYHaSLHVEmCQ/3Cj00svwbPPQkqKPO69F7p1czoqFUlcLY6WLVvGuHHjnA5HRTh/\n",
       "OYaewECkqOcs4ACSQ6gBrAL+ixQTlcRObXEl4DzgEqRp7I/AT8CvNt4bFSZOlIThu++gWTOno1GR\n",
       "RudeVsHgL2HIBSZaj3jc5f97sDf383agqcdyU9xFRi5brf0dsR7fAW3xkTCMHj268Hlqaiqpqak2\n",
       "Qghv06fDyJGQkaGJgiqfiRMn6tzLqlBGRgYZGRknvJ9g3lpUBNYjuYEdwCLgRorWMZyBVFD3AhKA\n",
       "n5EKb+/ZhaOyuepFF8HDD8MVVzgdiYpUBQUFxMXFaS5B+RTMOZ/LKx+4B5iH5DjeQRKFO63X3wLW\n",
       "IX0jViAd6CZQPFGIWsbIVJpKlZe2PlLBEOx+tHOsh6e3vJaftx5R5eBBGd/In7w8/68r5ZKXl8ev\n",
       "v/7KmWee6XQoKgaUljBUBL4ELg5BLFHlnHPgr78gPr7kbeLjoa723FClyMrKYsiQIbRr14533nnH\n",
       "6XBUDCgtYchHiniSgL+CH070OHJE+iM09m6gq5RNvlocKRUKdoqSDiFjI32Je1pPA9wXrKCUinUr\n",
       "Vqzg5ptvpkmTJtriSIWcnYRhhvVwNQvy19tZKRUA+/fvZ+jQodovQTnC7jcuAXcP6HXAseCEU6KI\n",
       "a67auDEsWqRFSUop5wSzuWoqMBmZpAegGTAY/8NhKKWUilB2GkGPQ4bH6Go9eiLDZCulTlBWVhYv\n",
       "vfSS02EoVYSdhMHVg9nlF4Lf/0GpqJaXl8eoUaPo2bMnderUcTocpYqwc4Ffikyy8x5SVjUQGQlV\n",
       "KVUOrn4J2uJIhSs7OYb/Q4ayuA+4F1htrVN+RFhduQqR6dOn07NnT4YOHcrs2bM1UVBhKVLawUVU\n",
       "q6QXXoAJE6SDm87TrDzt3buX3NxcTRBUSITjIHoxacIEeOUVWLBAEwVVnNYnqEigOYYAmjoV0tNl\n",
       "foWWLZ2ORjnt+PHjxPsbLEupICtvjqEsb6gGHC7rAQIkIhKGevVgzhxo397pSJSTXGMcLVmyhM8+\n",
       "+8zpcFQMK2/CYKfy+QJkjgRXk9VzgNfLeqBYcOwYnHqq01EoJ2VlZdGxY0eWLl3KhAkTnA5HqXKx\n",
       "kzC8BPRGpuAEyAJ0qnqlPHj2S9AWRyrS2a183uK1nB/oQCLVmWfCn3/K88OHoVIlZ+NRzpg2bZrO\n",
       "vayihp2yp4+QITBeBToh/RnaAzcEMS5vYVvHULEibN0qfxMSoGZNpyNSTnB9P3UkVBVOgln5XA/4\n",
       "N9DD2v4LJHHYW9aDnYCwThiOHpW/SikVToJZ+XwaMACojyQSA4EzynogpaJBXl4ey5YtczoMpYLK\n",
       "TsLwqs11SkU1V4ujcePGOR2KUkHlrwDkfKSpaj1gKO7sSCL2EhSlooLOvaxijb+EoTKSCMRbf11y\n",
       "gL7BDCqcGQPvvAP798tyQYGz8ajgWrlyJYMGDdKRUFVMsVMpkQL8HtwwShU2lc+5uVCtGjzwgCwn\n",
       "JcHIkaCNUaJTVlYWK1as0LmXVUQKZquk+sDDQGugqrXOAN3LerATEFYJQ82a8lcppcJZMEdX/S/w\n",
       "P+By4E5gCLC7rAeKdCtXSge2vDynI1FKqeCyk5IsA84DVgBtrHVLkE5uoeJojmH/fqhTB9q1k+UG\n",
       "DWDWLMfCUUGQlZXF7Nmzefzxx50ORamACWY/Btc98i4k13AekFzWA0Wy48el+Ojnn+WhiUL08Bzj\n",
       "qHnz5k6Ho1RYsFOUNAZIAtKBV4CawIPBDEqpUNC5l5XyrbzNLDoCiwIZSCmCVpT066/Qti3k+xkW\n",
       "0Bho0gQ2bQpKCMoBn332GbfcckthvwRtcaSiUTBaJVUArgFaAKuAz5F6hWeQlkrnlDnK8gtawrB0\n",
       "Kdx2mxQR+RMfLw8VHQ4cOMCBAwc0l6CiWjBaJY0HTkZyBiOBW5ExkkYAM8seYviqUAEqV3Y6ChVK\n",
       "iYmJJCYmlr6hUjHIX8LQGWmFVABUQSqfWxDaUVUDZv16uPvu4j2Vc3I0JxDtjh07RiWdKEMp2/wl\n",
       "DMeQRAHgKLCJCE0UADZulETgn/8s/lrTpqGPRwWfa4yjjIwMMjIytB5BKZv8JQxnACs9llt4LBvc\n",
       "fRoiRt260D2U/bWVYzxbHH3wwQeaKChVBv4ShlYhiyJIduyAt96SVkW//up0NCoUfI2EqomCUmXj\n",
       "L2H4PQD77w28hIzQ+jYwtoTtOgA/Av2AGQE4LgA//AAffQT9+0OrVtCpU6D2rMLVvHnzdO5lpU5Q\n",
       "MG+l4oH1yJSg24HFwI3AWh/bfQkcBiYB033sq1zNVadNgw8/lL8qNujcy0q5BXNIjPLqCGxAch7H\n",
       "gKnAVT62uxf4iBgcmE8FXlxcnCYKSp0guwlDNeD0Mu67MbDVY3mbtc57m6uAN6zl8BhbW4W9vLw8\n",
       "Fi5c6HQYSkUlOwnDlUAmMM9aPhewM4ycnYv8S8Aj1rZxBLdoS0UJ19zLL774IuEyT4dS0cTOIHqj\n",
       "gU7AN9ZyJnCKjfdtBzx7CDRFcg2e2iFFTAB1gTSk2KlYwjN69OjC56mpqaSmptoIQUUTbXGklH+u\n",
       "Pjsnys6v6mckYchEcgtQdG6GklREKp8vAXYgQ2v4qnx2mQTMxnerJK18jnFr1qxhwIABNGnShPHj\n",
       "x2uLI6VsCOYMbquBgda2LYH7ADuFu/nAPUgRVDzwDpIo3Gm9/lZZg1Wxq3LlygwdOlRzCUqFgJ1f\n",
       "WHVk4Lye1vI84ClkmIxQ0RyDUkqVUTBzDKcDj1kPpZRSUc5Oq6RxwDokl3BWcMNRsS4rK4thw4Zp\n",
       "ayOlHGQnYUgFLgb2IPUCKwGdMV0FlOfcy2effbbT4SgV0+x2cNsJ/Bu4C1gOPBG0iFTMcfVLcI1x\n",
       "dPPNN2sFs1IOspMwtEb6MqwCXkVaJHn3YFaqXObPn0/Pnj0ZOnQos2fP1maoSoUBO5XPE5FOaL2Q\n",
       "TmtKBUyXLl10JFSlwoydhKFz0KNQMSshIUETBaXCjL+EYRpwPUVncXOJyBnclLOOHj1KlSpVnA5D\n",
       "KVUKfwnD/dbfyyneQULbEirbXGMcffbZZyxevFgrlpUKc/4qn3dYf/+OzKng+fh7EGNSUcSzxdGs\n",
       "WbM0UVAqAthpldTTx7rLAh2Iii6e/RK0xZFSkcVfUdL/ITmDFhStZ0gEfghmUCry/fjjjyxbtkxb\n",
       "HCkVgfwlDO8Dc4B/AsNx1zMcAPYGOS4V4bp160a3bt2cDkMpVQ7+EgaD1CfcTfHK5tpAdpBiOiFf\n",
       "fw233y7PDx6EHj2cjUcppSKNv4ThA6APsBTfrZBODkpEJ2jLFmjbFv71L1lu0MDZeKJdXl4eCxYs\n",
       "4JJLLnH7ODQ2AAAbX0lEQVQ6FKVUgPhLGPpYf1NCEEdA1awJLVo4HUX0y8rKYsiQIZx88slcfPHF\n",
       "VKhgd+gtpVQ4s/NLvhCoYT0fhAzD3TxoEamw593iaMaMGZooKBVF7AyJ8SbQ1noMRabonAJozWIM\n",
       "WrduHTfccANNmjTRFkdKRSk7t3n5QAFwNfAaMsJqYjCDUuGrZs2apKena78EpaKYnW6o3wFzgVuA\n",
       "i4DdQBYQytlU/M75vHMnfPqpPP/B6mHx7rvBD0oppcJZeed8tpNj6A/kAn8DdiFzMfyrrAcKphkz\n",
       "4MUXYdEiqFQJrr3W6YiUUipy2U1JGgIdkGari4A/gxaRb35zDK+9BmvWyF8VGFlZWbz55pu8/vrr\n",
       "WrGsVIQKZo6hH/AzMgR3PyRhuL6sB1KRwbPF0QUXXKCD3ikVg+y0ShqJ5BZcuYR6wHxkvgYVRVz9\n",
       "ErTFkVKxzU6OIQ6pcHbZSzmyJiq8LVy4UEdCVUoB9nIMc4F5yKB6cUhl9JxgBqVCr1OnTqxYsYKG\n",
       "DRs6HYpSymF2EoZhwLVAF2v5LeDjoEWkHBEfH6+JglIK8J8wnIY0Sz0VWIEkENtCEZQKrkOHDlG9\n",
       "enWnw1BKhSl/dQwTgU+B64BlwMshiUgFjavFUceOHTl+/LjT4SilwpS/HEMNYIL1fB2QGfxwVLB4\n",
       "tjj68ssviY+PdzokpVSY8pcwVAHOs57HAVWt5Tiko9uy4IamAiEvL48xY8bwxhtv8PzzzzNo0CDt\n",
       "m6CU8stfwrALeMHP8sVBiUgF1MqVK8nKytJ+CUop2/wlDKmhCkIFT7t27Zg5c6bTYSilIogOgqOU\n",
       "UqoITRiiRF5eHp+6xh5XSqkToAlDFMjKyqJjx46MHz+e/Px8p8NRSkU4OwlDBWSu5yes5WZAxzIc\n",
       "ozfS3PVXYLiP1wcCy5FOdD8Abcqw75jmPffyzJkzqVjRTmd2pZQqmZ2ryOvI1J7dgSeBg9a69jbe\n",
       "G49MBdoD2A4sBmYBaz22+Q3oCuxHEpHxQGd74ceuDRs20LdvXx0JVSkVcHYShk7Aubg7uGUDlWzu\n",
       "vyOwAfjdWp4KXEXRhOFHj+c/A01s7jum1alTh4cffpgbb7xR+yUopQLKTlFSHnLn71IPyUHY0RjY\n",
       "6rG8zVpXkluBz+3seOJEiI+Xxz33QHKyzYiiRHJyMgMGDNBEQSkVcHZyDK8go6nWB54B+iKT99hR\n",
       "8nycxV2MzCt9oa8XR48eXfg8NTWVPXtSefBBGDtW1unsk0qpWJeRkUFGRsYJ78fu7WYr4BLr+XyK\n",
       "FgX50xkYjdQdADyK5DbGem3XBphhbbfBx36Kzfn83HOwZ4/8jWZZWVk8//zzTJo0iUqV7JbgKaVU\n",
       "cOd8bgYcAmZbj0PWOjuWAC2BFKAyMsnPLB/7nwHchO9EISZ5tjjq2bOntjZSSoWMnavN57iLhKoA\n",
       "JwPrgTNtvDcfuAeZAS4eeAfJbdxpvf4W0gw2GXjDWneMsjWHjTo697JSykl2EoazvJbPA+4uwzHm\n",
       "UHwq0Lc8nt9mPRSQmZlJr169dCRUpZRjylM+sQxpwqqC4JxzzmH16tXUq1fP6VCUUjHKTsKQ7vG8\n",
       "ApJj2B6ccFRcXJwmCkopR9mpfK7h8aiMTPd5VTCDihX79+93OgSllCqmtBxDPFCTorkGdYJcs6q9\n",
       "9957rF27lsqVKzsdklJKFfKXY6gIHEc6nGkNaIBkZmbSoUMHli5dyoIFCzRRUEqFHX85hkVIfUIW\n",
       "MBOYBhy2XjNI3wNlk+fcyy+88AI33XSTtjhSSoUlfwmD66pVBdiLjK7qSROGMti4cSOrVq3SfglK\n",
       "qbDnL2GoBwwFVoYolqjWqlUrpk+f7nQYSilVKn8JQzyQGKpAlFJKhQd/CcMu4B+hCiRa5OXlMXPm\n",
       "TK6//nqnQ1FKqXLRwaoDyNXiaMqUKeTm5jodjlJKlYu/HEOPkEUR4bTFUejUrl2bffv2OR2GUmEl\n",
       "OTmZ7OzsgO3PX8KwN2BHiWKbNm3i6quvplmzZtriKAT27duH99wcSsW6QN+I6iD/J6h+/fo89thj\n",
       "9OvXT3MJSqmooAnDCapevTr9+/d3OgyllAoYrXxWSilVhCYMNmVmZnLttddy9OhRp0NRSqmg0oSh\n",
       "FK65l3v16sU111xDQkKC0yEpFRHWrFlDhw4dnA4jKvTt25e5c+eG7HiaMPjh6pewbNkysrKydKpN\n",
       "5VdKSgrVqlUjMTGRhg0bMmjQIHJycopss3DhQrp3707NmjVJSkriyiuvZO3atUW2ycnJ4YEHHqB5\n",
       "8+YkJiZy6qmn8uCDD7J3b2Q1FHz88ccZNmyY02GckN9//52LL76Y6tWr06pVK+bPn1/itmlpaSQm\n",
       "JhY+EhISaNOmTeHrnt+PxMREevfuXeT9u3fvZsCAASQlJVG7dm1uuummwteGDx/OyJEjA/8BI5zx\n",
       "NnasMcOGFVsdMOvWrTP16tUzU6ZMMQUFBcE7kCoTX9+FcJGSkmLmz59vjDFm165dpm3btmaYx5d0\n",
       "4cKFpkaNGubll182Bw8eNNnZ2WbkyJEmOTnZ/Pbbb8YYY3Jzc0379u1Nz549zdq1a40xxvz555/m\n",
       "6aefNp9//nnQYj927FhA97djxw5Tu3Ztk5ubW6735+fnBzSe8urcubNJT083R48eNdOnTzdJSUlm\n",
       "9+7dtt6bmppqnnrqqcJlz++HL126dDHp6ekmJyfH5Ofnm6ysrCKvt2zZ0ixZssTne0v6XSAjYUet\n",
       "Yh842AmDMcZkZ2cH9wCqzHx9F8KF9w9/2LBh5rLLLitc7tKli7n77ruLvS8tLc3cfPPNxhhjJkyY\n",
       "YBo0aGAOHTpk+7irVq0yPXr0MLVr1zYNGjQwzz77rDHGmMGDB5uRI0cWbvfNN9+YJk2aFC43b97c\n",
       "jB071px99tkmISHBjB071vTt27fIvu+77z5z3333GWOM+euvv8zf/vY306hRI9O4cWMzcuRIc/z4\n",
       "cZ8xTZ482Vx66aVF1j377LOmRYsWJjEx0bRu3dp8/PHHha9NmjTJXHDBBebBBx80derUMY8//rjJ\n",
       "zc016enpplmzZqZBgwbmrrvuMkeOHDHGGLNv3z7Tp08fU69ePZOcnGwuv/xys23bNtvnzI7169eb\n",
       "hIQEc/DgwcJ1Xbt2NW+++Wap7920aZOJj483mzdvLlyXkpJivvrqK5/bz5s3z6SkpJR4Po0x5vbb\n",
       "bzf/+Mc/fL5W0u+CciYMWpTkR3JystMhqAhjrM5327ZtY+7cuXTq1AmAw4cP8+OPP/ocQ6tfv358\n",
       "+eWXAHz11VekpaVRrVo1W8c7cOAAPXr04LLLLmPnzp1s2LCBSy65BJBOT6UVfU6dOpU5c+awf/9+\n",
       "brjhBj7//HMOHjwIwPHjx5k2bRoDBw4EYMiQIVSuXJmNGzeSmZnJF198wdtvv+1zvytXruT0008v\n",
       "su7UU0/l+++/Jycnh1GjRnHTTTfxxx9/FL6+aNEiWrRowZ9//sljjz3G8OHD2bBhA8uXL2fDhg1s\n",
       "376dJ598EoCCggJuvfVWtmzZwpYtW6hatSr33HNPiZ/z8ssvJzk52efjyiuv9Pme1atXc8opp1C9\n",
       "evXCdW3btmX16tV+zynAlClT6Nq1K82aNSuyfuDAgdSvX59evXqxYsWKwvU//fQTp59+OoMHD6Zu\n",
       "3bp07NiR7777rsh7W7VqxfLly0s9diwplhIGMsewZ8+ewOxIBZ2v70LR1wPzKI/mzZubGjVqmMTE\n",
       "RBMXF2euvvrqwjvArVu3mri4OLN+/fpi75szZ46pVKmSMcaYHj16mEcffdT2Md9//31z3nnn+Xxt\n",
       "yJAhfnMMKSkpZtKkSUXe06VLFzNlyhRjjDFffPGFadGihTFGisYSEhIK79hdx7744ot9Hvv22283\n",
       "jzzyiN/YzznnHDNz5kxjjOQYmjVrVvhaQUGBqV69utm4cWPhuoULF5qTTz7Z574yMzNNcnKy3+OV\n",
       "1ZQpU0znzp2LrBsxYoQZMmRIqe9t0aKFmTx5cpF1CxcuNEePHjWHDx82zz77rGnYsKHZv3+/MUbO\n",
       "V1xcnJk4caLJz883U6dONUlJSUWuTePHjzfdu3f3ebySfhdojqHsXC2Ozj33XA4fPlz6G1TYC1TS\n",
       "UB5xcXHMnDmTnJwcMjIy+Prrr1myZAkguc8KFSqwc+fOYu/buXMn9erVA6Bu3brs2LHD9jG3bt3K\n",
       "KaecUr6AgaZNmxZZHjBgAB988AEA77//fmFuYfPmzRw7doxGjRoV3mnfdddd7N692+d+k5OTOXDg\n",
       "QJF1U6ZM4dxzzy18/6pVq4pUqHvGsnv3bg4fPky7du0Kt09LS2PPnj2A5MDuvPNOUlJSqFWrFt26\n",
       "dWP//v0BHS6lRo0axRoP/PXXX9SsWdPv+77//nv++OMP+vbtW2T9+eefT0JCAlWrVuWRRx4hKSmJ\n",
       "BQsWAFC1alVOPvlkbrnlFuLj4+nfvz9Nmzblhx9+KHz/gQMHSEpKCtCn8y9mEwbPFkc//fST7ay7\n",
       "UnZ07dqVe++9l+HDhwPSQ/7888/nww8/LLbthx9+WFj806NHD+bNm2f7RqVZs2b89ttvPl+rXr16\n",
       "kf3s2rWr2DbeRU19+/YlIyOD7du388knnzBgwABALtoJCQns3buXffv2sW/fPvbv38/Klb7n8WrT\n",
       "pg2//PJL4fLmzZu54447eO2118jOzmbfvn2cddZZRS7knrHUrVuXqlWrsmbNmsLj/fXXX4UX6hde\n",
       "eIFffvmFRYsWsX//fr799luMMSUmDN4thjwfffr08fmeM888k99++62waA1g+fLlnHnmmT63d5k8\n",
       "eTLXXXddqdeUuLi4wnjbtm3r83XPc7J27VrOOeccv/uMNcWySOUtSsrNzTVPPPGEtjiKUL6+C+HC\n",
       "u/J59+7dplq1auann34yxhjz/fffm+rVq5uXX37Z5OTkmOzsbDNixAiTnJxsNmzYYIyR72eHDh1M\n",
       "7969zbp168zx48fNnj17zJgxY3y2Sjpw4IBp1KiReemll8zRo0dNTk6O+fnnn40xUpF9xhlnmOzs\n",
       "bLNz507TqVOnYkVJvlrJpKWlmR49ehQrorrqqqvM/fffb3Jycszx48fNhg0bzLfffuvzXOzatcvU\n",
       "qVOnsFXS6tWrTZUqVcz69etNfn6+mThxoqlYsaJ55513jDFSlNSlS5ci+7j//vtNv379zJ9//mmM\n",
       "MWbbtm1m3rx5xhhjHn74YZOWlmaOHj1q9u7da66++moTFxfnt/K2PDp37mweeughc+TIkcJWSf6K\n",
       "ng8fPmxq1aplvvnmmyLrt2zZYr7//nuTm5trjhw5Yp577jlTv379wgYu2dnZJjk52UyePNnk5+eb\n",
       "adOmmTp16pi9e/cW7uO0004zixcv9nnckn4XaFGSPTt37mTdunXaL0EFXd26dRk8eDBjx44F4MIL\n",
       "L2TevHnMmDGDk046iZSUFJYvX873339PixYtAKhcuTJfffUVZ5xxBpdeeim1atWiU6dOZGdn07lz\n",
       "52LHqFGjBl9++SWzZ8+mUaNGnHbaaWRkZAAwaNAg2rZtS0pKCr179+aGG26w9X0fMGAA8+fPL8wt\n",
       "uEyZMoW8vDxat25N7dq1uf76633mQgAaNGhA9+7d+eSTTwBo3bo16enpnH/++TRs2JBVq1bRpUuX\n",
       "wu19VZSPHTuWU089lc6dO1OrVi0uvfTSwlzIAw88wJEjR6hbty4XXHABaWlpQfktT506lSVLllC7\n",
       "dm1GjBjB9OnTqVOnDgALFiwgMbHoJJeffPIJycnJpKamFll/4MAB/v73v1O7dm2aNGnCF198wZw5\n",
       "cwobuCQnJzNr1iyef/55kpKSeO6555g5cya1a9cGYPHixSQmJtK+ffuAf0ZfIuWqaCV+bs89B3v2\n",
       "yF8VOzyz3yq8rV27lsGDB7No0SKnQ4l4ffv25bbbbivWKc6lpN+FlViW+TqvCYOKKJowKFVcoBOG\n",
       "iCpK2rgR3nxTHj/+6H/bvLw8Jk+erBcRpZQqo4hKGCZPhrffhqwsaNAALrvM93auFkcfffSRNkNV\n",
       "SqkyiriJeq68Ep54wvdrOveyUkqduIhLGEqybds2+vTpo3MvK6XUCYqahKF+/fqMGjWKa665RnMJ\n",
       "Sil1AqImYahcuTLXXnut02GoIEtOTtaEXykvgR7wM9gJQ2/gJSAeeBsY62Obl4E04DAwBMgMckwq\n",
       "gmVnZzsdglJRL5itkuKBV5HEoTVwI9DKa5vLgFOBlsAdwBsl7Sw5Wfos/PVXJmlpacUGt4oVrl6t\n",
       "Ss+FJz0XbnouTlwwE4aOwAbgd+AYMBW4ymubK4HJ1vOfgSSgga+drVuXx333jeK993oxYMCAYl3R\n",
       "Y4V+6d30XLjpuXDTc3HiglmU1BjY6rG8DehkY5smwB9e29GzZwdtcaSUUiEQzITBbpdj75pEn+9L\n",
       "T0/XQe+UUioEgnmV7QyMRuoYAB4FCihaAf0mkIEUMwGsA7pRPMewAWgRpDiVUipabUTqccNGRSSo\n",
       "FKAykIXvyufPreedgZ9CFZxSSilnpAHrkTv+R611d1oPl1et15cD54U0OqWUUkoppVRk6Y3UM/wK\n",
       "DC9hm5et15cD54YoLieUdi4GIudgBfAD0CZ0oYWcne8FQAcgH4jWLvB2zkMq0kl0FVJ/F61KOxd1\n",
       "gblIEfYqpPNstJqI1Mv6noBbROx1Mx4pUkoBKlF6nUQnordOws65OB+oZT3vTWyfC9d2XwOfAteF\n",
       "KrgQsnMekoDVSJNvkItjNLJzLkYDz1rP6wJ7iaIhgLxchFzsS0oYynzdDKf5GALaIS7C2TkXPwL7\n",
       "rec/474YRBs75wLgXuAjYHfIIgstO+dhADAd6Q8EsCdUwYWYnXOxE6hpPa+JJAz5IYov1BYA+/y8\n",
       "XubrZjglDL46uzW2sU00XhDtnAtPt+K+I4g2dr8XV+EeUiUap+2zcx5aArWBb4AlwKDQhBZyds7F\n",
       "BOBMYAdSfHJ/aEILS2W+boZT1iqgHeIiXFk+08XA34ALgxSL0+yci5eAR6xt44icuczLws55qIS0\n",
       "7LsEqIbkKn9CypajiZ1z8RhSxJSK9IH6EmgLHAheWGGtTNfNcEoYtgNNPZab4s4Sl7RNE2tdtLFz\n",
       "LkAqnCcgdQz+spKRzM65aIe7k2RdpJn0MWBW0KMLHTvnYStSfHTEenyHXAyjLWGwcy4uAMZYzzcC\n",
       "m4DTkZxUrIno66Z2iHOzcy6aIeWsnUMaWejZOReeJhGdrZLsnIczgK+QytlqSGVk69CFGDJ2zsU4\n",
       "YJT1vAGScNQOUXxOSMFe5XNEXje1Q5xbaefibaRCLdN6LAp1gCFk53vhEq0JA9g7Dw8hLZNWAveF\n",
       "NLrQKu1c1AVmI9eJlUjFfLT6AKlLyUNyjX8jdq+bSimllFJKKaWUUkoppZRSSimllFJKKaWUUkqp\n",
       "8HEcd5+HTKSDXEkOBuB47wK/WcdaSvk64k1AOm2BDHHg6YdyR1aU67ysAGYANUrZvi3Shj5UvgIS\n",
       "red2hlf253JgGdIhbDVwxwlHV9Q/kOE4QEb8XG0d7yRgmrXe7vm7j+gd60mpsFGWMWICMZ6MZ0ez\n",
       "S5GONSciWGPceO73XSC9lO2HAK8EIQ5fw9N0B17zWC5teGV/KiHDIJzksXxaOfZj15vInCHehmDv\n",
       "/CUS3Z02lQoL3hfW6sjd6FLkbvlKH9s2QsbbyUQuRl2s9T2BhdZ7P7T25W0S7nkRqgCHrOdDrX2t\n",
       "xD3iZXXgM+ROdiVwvbU+AxkH6Z/IkMmZwH+s11y5mqlIl3+Xd5EEqQLwL+TispyS7449z8udwOvW\n",
       "847WZ1yG5E5OQ4Zf2AL8acVyvRX7RGQ442UUPY+e/mV9thVAP2tdKjJk8kykF6+3t5Fz7SmF8iUM\n",
       "tZHcRhUfr72LXMgXW3H0sdbHU/I5HI58lizgGY/9XIeM9rsXyTH+B2huxVwJ9/lbhpyHX3DPG1EB\n",
       "6Z1bx1qeg4yQqpQKEteFNRMZsz8edxFFXYoOtOa6WKbjLsKpgBSz1AW+Bapa64cDj/s4nmfCcD0y\n",
       "0ud5yMWkKnJBXQWcY2033uO9rnH0v8Hdfd87YXMtX41ckMB94U5ALmIjrPUJyEUvxUecrv3EI+fl\n",
       "79ZyorUOoAcy1wPAYGQ2LJdncN8ZJyEX1mpex7gO+AIZ4bI+sBloiCQMB5ELpy9rKT6+TwrlL0qa\n",
       "gCQO7yNDRLhG3JyEeyydU5FhFfydwzQksXQlMkke+7nWx3PPmL3P3xO4bxB64i5yAima+r+yfEAV\n",
       "GOE0uqoKriMUndKvEjLD1UVAAVLEUB+5m3NZhNwNVwI+Qe4aU5GB2RZa21T2eO4pDrnbHGnt81ak\n",
       "SGmGFQvW84uQKRifR3IGnwLfl+FzzQX+bcWRhiRauchF5mygr7VdTeSi97vX+6siiWVj67U3rfVJ\n",
       "wBTrPQb3b8V7WO+ewBXIGEUgF9CmFM0BXIhcjA1yLr5FpiHNQc7x5hI+20lAdkkfvBxuR85VDyve\n",
       "S4FbrNc+tP5uQO70z8D3OWyJ1CNMBI5a6/8q4Xi+hj/3Pn8TkRzTv5ExfiZ5vLYDOKX0j6UCTROG\n",
       "2DUQufs/D6mA3UTxYoYFyIX7cuSufBwyvPeXlD4omUEuPjM81vWg6EUhztruVyTR6gM8DcwHnrL5\n",
       "OY4iRU69kKKJDzxeu8eK1R9XglkVmIdM+POxdfz5wDXIHX2Gn31cS+lDW5c0Hv4h7w1PQDzuYaVn\n",
       "ItNbeltlPf6D/M9v8bGNZ3y+zmEvAjfnxTYkF9MdSSxv9HjN9f1QIRZOM7ip0KqJ3L0eRyb78VWc\n",
       "0QyZKvNt63EuMmTvhcjkJyBFQi1LOIb3xWMBUvTjKkq62lrXCLnA/xfJOfiarPwYJd/I/A+523Tl\n",
       "PkAu8n/3eM9pFC/i8XQEaQkzxoq7JnLHCkUvnjm4i+Bcx/EcxdRX7AuA/sjvrR7QFckplHZx3YG7\n",
       "vN2O49bxz6V4olAdye15xvm79TwOKe6LQ/6vpwDrKPkcfomcE1dxYnIZYvQ+fyDfrfeQXItnQtCI\n",
       "4jk8pVQA5Xgt10GKgFYg2fnVuJuwurYdjJQNL0OKP1yJx8W4KySXIzkKbyUNf/0g7spn1wW1p7Uf\n",
       "1/DhrnoFzzqGfwJrcFc+e36eikhl5zse6+KQi/wK61jzcdddePI+L7OQi3hnpDhoGZJ7+M16PdmK\n",
       "0VX5XAUpflqB3ImXNDnQc7grn12V6938bA9SJ9DLY9k1vHIuUg9Q0t2+LzWQCv51VuwLcJ/bSci0\n",
       "qK7KZ1dlvq9z6LqoD0e+M5lILs+1n5LqGFZYzz3Pn6sSvhIyf7l3KymtfFZKKS+puOexDian57Bo\n",
       "j9x4eKqJJFTKAVqUpFT4ykCK6byLXqLJI0iLr0e91g9BKqSVUkoppZRSSimllFJKKaWUUkoppZRS\n",
       "SimllFLR5f8BGMhjk754DIMAAAAASUVORK5CYII=\n"
      ],
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1139dd210>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_roc_curve(target_test, target_predicted_proba)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here the area under ROC curve is 0.756 which is very similar to the accuracy (0.732). However the ROC-AUC score of a random model is expected to 0.5 on average while the accuracy score of a random model depends on the class imbalance of the data. ROC-AUC can be seen as a way to callibrate the predictive accuracy of a model against class imbalance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cross-validation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We previously decided to randomly split the data to evaluate the model on 20% of held-out data. However the location randomness of the split might have a significant impact in the estimated accuracy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.73184357541899436"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "features_train, features_test, target_train, target_test = train_test_split(\n",
    "    features_array, target, test_size=0.20, random_state=0)\n",
    "\n",
    "logreg.fit(features_train, target_train).score(features_test, target_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.67039106145251393"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "features_train, features_test, target_train, target_test = train_test_split(\n",
    "    features_array, target, test_size=0.20, random_state=1)\n",
    "\n",
    "logreg.fit(features_train, target_train).score(features_test, target_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.66480446927374304"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "features_train, features_test, target_train, target_test = train_test_split(\n",
    "    features_array, target, test_size=0.20, random_state=2)\n",
    "\n",
    "logreg.fit(features_train, target_train).score(features_test, target_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So instead of using a single train / test split, we can use a group of them and compute the min, max and mean scores as an estimation of the real test score while not underestimating the variability:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.63128492,  0.68715084,  0.70224719,  0.73033708,  0.71751412])"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.cross_validation import cross_val_score\n",
    "\n",
    "scores = cross_val_score(logreg, features_array, target, cv=5)\n",
    "scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.63128491620111726, 0.69370682962933028, 0.7303370786516854)"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scores.min(), scores.mean(), scores.max()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`cross_val_score` reports accuracy by default be it can also be used to report other performance metrics such as ROC-AUC or f1-score:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.61093544137022393, 0.72123181651091761, 0.78776737967914512)"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scores = cross_val_score(logreg, features_array, target, cv=5,\n",
    "                         scoring='roc_auc')\n",
    "scores.min(), scores.mean(), scores.max()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise**:\n",
    "\n",
    "- Compute cross-validated scores for other classification metrics ('precision', 'recall', 'f1', 'accuracy'...).\n",
    "\n",
    "- Change the number of cross-validation folds between 3 and 10: what is the impact on the mean score? on the processing time?\n",
    "\n",
    "Hints:\n",
    "\n",
    "The list of classification metrics is available in the online documentation:\n",
    "\n",
    "  http://scikit-learn.org/stable/modules/model_evaluation.html#common-cases-predefined-values\n",
    "  \n",
    "You can use the `%%time` cell magic on the first line of an IPython cell to measure the time of the execution of the cell. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.285714285714 0.415882352941 0.558823529412 0.0873266135482\n",
      "CPU times: user 28.8 ms, sys: 1.29 ms, total: 30.1 ms\n",
      "Wall time: 29.1 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "scores = cross_val_score(logreg, features_array, target, cv=10, scoring='recall')\n",
    "\n",
    "print scores.min(), scores.mean(), scores.max(), scores.std()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## More feature engineering and richer models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us now try to build richer models by including more features as potential predictors for our model.\n",
    "\n",
    "Categorical variables such as `data.Embarked` or `data.Sex` can be converted as boolean indicators features also known as dummy variables or one-hot-encoded features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Sex_female</th>\n",
       "      <th>Sex_male</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Sex_female  Sex_male\n",
       "0           0         1\n",
       "1           1         0\n",
       "2           1         0\n",
       "3           1         0\n",
       "4           0         1"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#one hot encoding of the sex variables\n",
    "pd.get_dummies(data.Sex, prefix='Sex').head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Embarked_C</th>\n",
       "      <th>Embarked_Q</th>\n",
       "      <th>Embarked_S</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Embarked_C  Embarked_Q  Embarked_S\n",
       "0           0           0           1\n",
       "1           1           0           0\n",
       "2           0           0           1\n",
       "3           0           0           1\n",
       "4           0           0           1"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.get_dummies(data.Embarked, prefix='Embarked').head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can combine those new numerical features with the previous features using `pandas.concat` along `axis=1`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Fare</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Age</th>\n",
       "      <th>Sex_female</th>\n",
       "      <th>Sex_male</th>\n",
       "      <th>Embarked_C</th>\n",
       "      <th>Embarked_Q</th>\n",
       "      <th>Embarked_S</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>  7.2500</td>\n",
       "      <td> 3</td>\n",
       "      <td> 22</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td> 71.2833</td>\n",
       "      <td> 1</td>\n",
       "      <td> 38</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>  7.9250</td>\n",
       "      <td> 3</td>\n",
       "      <td> 26</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td> 53.1000</td>\n",
       "      <td> 1</td>\n",
       "      <td> 35</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>  8.0500</td>\n",
       "      <td> 3</td>\n",
       "      <td> 35</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Fare  Pclass  Age  Sex_female  Sex_male  Embarked_C  Embarked_Q  \\\n",
       "0   7.2500       3   22           0         1           0           0   \n",
       "1  71.2833       1   38           1         0           1           0   \n",
       "2   7.9250       3   26           1         0           0           0   \n",
       "3  53.1000       1   35           1         0           0           0   \n",
       "4   8.0500       3   35           0         1           0           0   \n",
       "\n",
       "   Embarked_S  \n",
       "0           1  \n",
       "1           0  \n",
       "2           1  \n",
       "3           1  \n",
       "4           1  "
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rich_features = pd.concat([data.get(['Fare', 'Pclass', 'Age']),\n",
    "                           pd.get_dummies(data.Sex, prefix='Sex'),\n",
    "                           pd.get_dummies(data.Embarked, prefix='Embarked')],\n",
    "                          axis=1)\n",
    "rich_features.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By construction the new `Sex_male` feature is redundant with `Sex_female`. Let us drop it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Fare</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Age</th>\n",
       "      <th>Sex_female</th>\n",
       "      <th>Embarked_C</th>\n",
       "      <th>Embarked_Q</th>\n",
       "      <th>Embarked_S</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>  7.2500</td>\n",
       "      <td> 3</td>\n",
       "      <td> 22</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td> 71.2833</td>\n",
       "      <td> 1</td>\n",
       "      <td> 38</td>\n",
       "      <td> 1</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>  7.9250</td>\n",
       "      <td> 3</td>\n",
       "      <td> 26</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td> 53.1000</td>\n",
       "      <td> 1</td>\n",
       "      <td> 35</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>  8.0500</td>\n",
       "      <td> 3</td>\n",
       "      <td> 35</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Fare  Pclass  Age  Sex_female  Embarked_C  Embarked_Q  Embarked_S\n",
       "0   7.2500       3   22           0           0           0           1\n",
       "1  71.2833       1   38           1           1           0           0\n",
       "2   7.9250       3   26           1           0           0           1\n",
       "3  53.1000       1   35           1           0           0           1\n",
       "4   8.0500       3   35           0           0           0           1"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rich_features_no_male = rich_features.drop('Sex_male', 1)\n",
    "rich_features_no_male.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Fare          891\n",
       "Pclass        891\n",
       "Age           714\n",
       "Sex_female    891\n",
       "Embarked_C    891\n",
       "Embarked_Q    891\n",
       "Embarked_S    891\n",
       "dtype: int64"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rich_features_no_male.count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us not forget to imput the median age for passengers without age information:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Fare</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Age</th>\n",
       "      <th>Sex_female</th>\n",
       "      <th>Embarked_C</th>\n",
       "      <th>Embarked_Q</th>\n",
       "      <th>Embarked_S</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>  7.2500</td>\n",
       "      <td> 3</td>\n",
       "      <td> 22</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td> 71.2833</td>\n",
       "      <td> 1</td>\n",
       "      <td> 38</td>\n",
       "      <td> 1</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>  7.9250</td>\n",
       "      <td> 3</td>\n",
       "      <td> 26</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td> 53.1000</td>\n",
       "      <td> 1</td>\n",
       "      <td> 35</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>  8.0500</td>\n",
       "      <td> 3</td>\n",
       "      <td> 35</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Fare  Pclass  Age  Sex_female  Embarked_C  Embarked_Q  Embarked_S\n",
       "0   7.2500       3   22           0           0           0           1\n",
       "1  71.2833       1   38           1           1           0           0\n",
       "2   7.9250       3   26           1           0           0           1\n",
       "3  53.1000       1   35           1           0           0           1\n",
       "4   8.0500       3   35           0           0           0           1"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rich_features_final = rich_features_no_male.fillna(rich_features_no_male.dropna().median())\n",
    "rich_features_final.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can finally cross-validate a logistic regression model on this new data an observe that the mean score has significantly increased:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Logistic Regression CV scores:\n",
      "min: 0.770, mean: 0.786, max: 0.810\n",
      "CPU times: user 17.1 ms, sys: 1.12 ms, total: 18.2 ms\n",
      "Wall time: 17.2 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.cross_validation import cross_val_score\n",
    "\n",
    "logreg = LogisticRegression(C=1)\n",
    "scores = cross_val_score(logreg, rich_features_final, target, cv=5, scoring='accuracy')\n",
    "print(\"Logistic Regression CV scores:\")\n",
    "print(\"min: {:.3f}, mean: {:.3f}, max: {:.3f}\".format(\n",
    "    scores.min(), scores.mean(), scores.max()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise**:\n",
    "\n",
    "- change the value of the parameter `C`. Does it have an impact on the score?\n",
    "\n",
    "- fit a new instance of the logistic regression model on the full dataset.\n",
    "\n",
    "- plot the weights for the features of this newly fitted logistic regression model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%load solutions/04A_plot_logistic_regression_weights.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": [
       "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEeCAYAAACJ266bAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\n",
       "AAALEgAACxIB0t1+/AAAGStJREFUeJzt3XmcJGV9x/FP787CAi7sLscih+4uR2BREQ0rRwiDIHIo\n",
       "inKIBDkUVgiHJiZ4ILsYo3igEZBDQAQPFrwQAUEOWzEoisAGjcthQgIaIQq4q8QEZPLHr4uu6Z2Z\n",
       "nZk+qqqfz/v1mtdMT1d3P11d/a2nnuepp0CSJEmSJEmSJEmSJEmSJEkl8VngUeDeUe4fBH4H3N34\n",
       "Oa03xZIkddpuwA6MHfjX9Kw0kqQRTenAc9wGPLGaZWodeB1JUhs6EfirMwTsAiwDrgcW9OA1JUkt\n",
       "BnrwGncBmwNPAfsCVwNbty60/fbbDy1btqwHxZGkvrIMeOl4FuxFDX8lEfYA3wKmAbNbF1q2bBlD\n",
       "Q0Nd+1m8eHFXn9+yW/6y/lj+/i4/sP14w7gXgT+HZhv+wsbfj/fgdSVJOZ1o0rkC2B3YAHgYWEzU\n",
       "4gEuBA4CjgeeIWr6b+rAa0qSJqgTgX/Yau7/dOOnUIODg0UXYdKqXHaw/EWz/MUqU/nLNFxyqNEe\n",
       "JUkap1qtBuPM8l604UuSSsDAl6REGPiSlAgDX5ISYeBLUiIMfElKhIEvSYkw8CUpEQa+JCXCwJek\n",
       "RBj4kpQIA1+SEmHgS1IiDHxJSoSBL0mJMPAlKREGviQlwsCXpEQY+JKUiE5cxFwq1LrrzmblyieK\n",
       "LsYqZsyYxYoVjxddDOk5XsRclRcXcS7jtlPDbVrd5kXMJUmrMPAlKREGviQlwsCXpEQY+JKUCANf\n",
       "khJh4EtSIgx8SUqEgS9JiTDwJSkRBr4kJcLAl6REGPiSlAgDX5IS0YnA/yzwKHDvGMucDTwALAN2\n",
       "6MBrSpImqBOBfymwzxj37wdsCWwFHAec34HXlCRNUCcC/zZgrMsNHQBc1vj7DmAmMKcDrytJmoBe\n",
       "tOFvCjycu/0IsFkPXleSlNOra9q2Xn5rxOu+LVmy5Lm/BwcHGRwc7F6JJKmC6vU69Xp9Uo/t1DVt\n",
       "5wLfBF48wn0XAHVgaeP2cmB3oqM3z2vaalK8pq1SVrZr2l4DvKXx907Ak6wa9pKkLutEk84VRI19\n",
       "A6KtfjEwrXHfhcD1xEidB4E/AEd34DUlSRPUqSadTrBJR5Nik45SVrYmHUlSCRj4kpQIA1+SEmHg\n",
       "S1IiDHxJSoSBL0mJMPAlKREGviQlwsCXpEQY+JKUCANfkhJh4EtSIgx8SUqEgS9JiTDwJSkRBr4k\n",
       "JcLAl6REGPiSlAgDX5ISYeBLUiIMfElKhIEvSYkw8CUpEQa+JCXCwJekRBj4kpQIA1+SEmHgS1Ii\n",
       "DHxJSoSBL0mJMPAlKREGviQlwsCXpEQY+JKUCANfkhLRicDfB1gOPACcOsL9g8DvgLsbP6d14DUl\n",
       "SRM00ObjpwLnAnsBvwR+DFwD/Lxlue8CB7T5WpKkNrRbw18IPAg8BDwNLAVeN8JytTZfR5LUpnYD\n",
       "f1Pg4dztRxr/yxsCdgGWAdcDC9p8TUnSJLTbpDM0jmXuAjYHngL2Ba4Gtm7zdSVJE9Ru4P+SCPPM\n",
       "5kQtP29l7u9vAecBs4HHW59syZIlz/09ODjI4OBgm8WTpP5Sr9ep1+uTemy7besDwH3AnsCvgB8B\n",
       "hzG803YO8BhxNLAQuAqYO8JzDQ0NjeeAQRquVqsxvoPNXqvhNq1ui+1/fFnebg3/GeBE4EZixM4l\n",
       "RNgvatx/IXAQcHxj2aeAN7X5mpKkSSjT6Blr+JoUa/hK2URq+J5pK0mJMPAlKREGviQlwsCXpEQY\n",
       "+JKUCANfkhJh4EtSIgx8SUqEgS9JiTDwJSkRBr4kJcLAl6REGPiSlAgDX5ISYeBLUiIMfElKhIEv\n",
       "SYkw8CUpEQa+JCXCwJekRBj4kpQIA1+SEmHgS1IiDHxJSoSBL0mJMPAlKREGviQlwsCXpEQY+JKU\n",
       "CANfkhJh4EtSIgx8SUqEgS9JiTDwJSkRBr4kJcLAl6REdCLw9wGWAw8Ap46yzNmN+5cBO3TgNSVJ\n",
       "E9Ru4E8FziVCfwFwGLBtyzL7AVsCWwHHAee3+ZqSpEloN/AXAg8CDwFPA0uB17UscwBwWePvO4CZ\n",
       "wJw2X1eSNEHtBv6mwMO52480/re6ZTZr83UlSRM00Objh8a5XG08j6vVWhcr3owZs1ix4vHVLrfu\n",
       "urNZufKJHpRoYsZT/rKWHcZX/hkzZrFyZTm3nfEo6/pPYduH6pW/Xq9Tr9cn9Zztfkt2ApYQbfgA\n",
       "7wGeBT6SW+YCoE4090B08O4OPNryXOPdefTUeDea2FmV8S3UGBoau1zlLTuMp/xVV971P751b/m7\n",
       "ZSLlH1+Wt1vDv5PojJ0L/Ao4lOi4zbsGOJEI/J2AJ1k17AH6/ostSUVqN/CfIcL8RmLEziXAz4FF\n",
       "jfsvBK4nRuo8CPwBOLrN15QkTUKZGj6HqlzDr/JhYXnLDjbpFCmlJpGql783TTqSElf1TvOUlOlT\n",
       "sobfFdbwy66867//1z1Uf/1bw5cqxBqyeqVMW5k1/K6whi+Npbzbf+dr+M6WKUmJMPAlKREGviQl\n",
       "wsCXpEQY+JKUCANfkhJh4EtSIgx8SUqEgS9JiTDwJSkRBr4kJcLAl6REGPiSlAgDX5ISYeBLUiIM\n",
       "fElKhIEvSYkw8CUpEQa+JCXCi5hLSlpKF5Ev07v0IuZd4UXMpX7mRcwlSasw8CUpEQa+JCXCwJek\n",
       "RBj4kpQIA1+SEmHgS1IiDHxJSoRn2qq0ZxpCd842lFJVpm+5Z9p2hWeqSv3MM20lSatop0lnNnAl\n",
       "8ELgIeAQ4MkRlnsIWAH8CXgaWNjGa0qSJqmdGv67gZuArYFbGrdHMgQMAjtg2EtSYdoJ/AOAyxp/\n",
       "Xwa8foxly9RXIElJaifw5wCPNv5+tHF7JEPAzcCdwLFtvJ4kqQ2ra8O/Cdh4hP+/r+X2EKMPUdkV\n",
       "+C9gw8bzLQduG2nBJUuWPPf34OAgg4ODqymeJKWlXq9Tr9cn9dh2mlqWE23zvwaeD3wH2GY1j1kM\n",
       "/B44a4T7HJbZFQ7LlPpZr4ZlXgMc2fj7SODqEZZZG5jR+HsdYG/g3jZeU5I0Se3U8GcDVwEvYPiw\n",
       "zE2Ai4D9gfnA1xrLDwBfBD48yvNZw+8Ka/hSP5tIDb9Mo2cM/K4w8KV+5pm2kqRVGPiSlAgDX5IS\n",
       "YeBLUiIMfElKhIEvSYkw8CUpEQa+JCXCwJekRBj4kpQIA1+SEmHgS1IiDHxJSoSBL0mJMPAlKREG\n",
       "viQlwsCXpEQY+JKUCANfkhJh4EtSIgx8SUqEgS9JiTDwJSkRBr4kJcLAl6REGPiSlAgDX5ISYeBL\n",
       "UiIMfElKhIEvSYkw8CUpEQa+JCXCwJekRBj4kpQIA1+SEtFO4B8M/Az4E/CyMZbbB1gOPACc2sbr\n",
       "SZLa0E7g3wscCHxvjGWmAucSob8AOAzYto3XnLR6vd7V558xYxZQK91PlKtY3V733Wb5i2X5O6ed\n",
       "wF8O3L+aZRYCDwIPAU8DS4HXtfGak9btlb5ixeMMDQ115Wfx4sWTfuyKFY939X2PR5k2+Mmw/MWy\n",
       "/J3T7Tb8TYGHc7cfafxPktRjA6u5/yZg4xH+/17gm+N4/qEJl0iSVFrfYfRO252AG3K338PoHbf3\n",
       "EDsIf/zxxx9/xv9zDz30HeDlo9w3APwCmAusQRSskE5bSdLkHUi0z/8P8GvgW43/bwJcl1tuX+A+\n",
       "ovP2Pb0soCRJkiRJUn+aWnQBCjKV6OzoF1V6P1lZa0UXRG1ZA3g98L/AEwWXpV1V3BYHgPWBpyby\n",
       "oBTn0plCTAcB8MIiC9JB2fs5DHhekQUZQ/alyso6fYT7UrC6odBVcALwfWAvYGdgRrHFmZQ1gO0a\n",
       "v6tSWcocS8xwsBfpVtpXK79z+zPiHIOvAe9v3K6S7L1kQXkwcAdwBbAB5Q7QNxBhcSbwzoLL0kv5\n",
       "7a9GfE5Tc7er4khiZF72nali4JwE/BT4MrEt7kcEf9mtA1xNDIpZSFQeplGt7afr8jWqNYBZwCXE\n",
       "3nEWMdLo3QyvcZZZ/guWhcilxHxFmTLUImusGgYLiQ12AXAIMeXGrr0tVs8taLl9GBE2/wR8ruel\n",
       "mbxsWzsTeGvuf9nETVVxHHArMJ8o/9uIk0gPLbJQ4/Qi4Cu522u23F+lz6HrDib25tsB5wPvAr5L\n",
       "TO62VoHlmoz1gYuBdwDrNf6+Cvhk4/engJcWVrrhQb8G8BLiy3U8cDLRJHA3sKj3Reup3YArgb9s\n",
       "3N4e+ALwAuDFwLPA7sUUbVIGiO/Qbo3bVardZzusjxNHKdAMyJOI78wOvS7UBM0E/gjsQnyHzgDO\n",
       "Af6+yEIVbQrDD5+nA18CriVqW7OIWuYtxJcusyPl7NNoPezfkZi07nRgTuN/2xM1l78gDk/PBV7b\n",
       "wzKO5mjgGqIGNRPYm5hE7xM0aycbETvhfpJtR5sSR49nEp/jIcSX8yPAj4j1UxXZdviPRFNIZoDY\n",
       "Ntdn+PepjAaA22nusLJmnBcS7+lVRRRqnLL1fwJwF/DPwD8Q29KDjf9DYrX8fFPG2jRX0oeJidvI\n",
       "3f4QcYbwBsA3gM9Trpp+644rcxLw10R73nwi5PN2JcKklxvvnsC83O1ZxE72SuDNwH8QHU3zgMtp\n",
       "Tq/xKuIo62D6Y0Mdqca7B/AZ4JXEFCT/RxxdZu93PvCKnpRufOYCBwDrNm63fi7bAHXgNS3/35/Y\n",
       "oZWx0gSj77Cy0L+KqDEXbS5jr3+IzIJmZ/kRxPcqCVOJ8M7aggeIGuT1jd8Qtfw7gb9q3J4HvJ2o\n",
       "ed5FuS7MsnbL7RcBFxFtvzOJ93kncCPRDvwY8D5iQ/k0EfZ796isALOBXwI3E0cZEIF/M7Bh4/Zb\n",
       "iNB7GbA18AOiw/x24KAelrVXDgAOJ2q9U4id9PnEl/cGoma2GXE0dgdwCuXZ4R1KBOJOYyxzJNH/\n",
       "ciyxA7iQmDZll24XbhzmMr4dVusR8EdoNr0VaTzrH5o71tnA14nvWN97G/HhLSVCZhbRIfMB4oN+\n",
       "gPggIWqRP2R4LWwDyjWMcV+i7M9v3D6ZaOs+CjiLCPS1iCCZ3VjmL4mjE4imnV6bSew4jyAOM99C\n",
       "zJd0DlHzh/gsfgx8jOaRy5Ytz1OWwJuI1jJvRlQ0riUqEVcQATSfCPxDiCasTxBf0pvo7c55LPn3\n",
       "cjHwN8R21npf5nDiSPkG4v2UYaAAjC8wjyJ2WEcBWxGDHlYQR6Wtney9MpH1XwP+nOh3uBf4INXq\n",
       "T5mUjYhOr3y74ZrESppPHKLViQ82+/CvJTptyib7sPYlgvKNudtrEaHwADFfUdZBszHRtLOM6Lwt\n",
       "0uXEEMsdiZ3SGUSN/u00z3W4hGji2arlsVXdUEdquhgkapcQX8b7idovwJuIUTnbNG5vkntcbZTn\n",
       "65UsUPYnmtiuI5pA9xxh2dZy5itMRYb+RHdYRwIfJY5EzyIGOSwaZdlum8j6z2xBlD/flFrW5rSO\n",
       "uZiouUPUcj/W+Ps84LTG358jRhZA9MLv16vCjcNIQxdPId7H/MbtNxI1wZfT7ATdktgZfJkYAVO0\n",
       "A4nOSYidz78TJ4Z8iNjJ3kZ8JrcTtSqoZo0eVv28TiJq7lnz1cbEgIDziCasOtGOP5OobOzbWC57\n",
       "/0Xs8Eba7tYjauvZ8N5LiaPjeayq9fGj9Tf1Sjs7rHVHWKbb2l3/rY+dSgJhD9He/TTwL8Qh9BrE\n",
       "Gz+X5ljhjxK92FsXUcAx5ANvHlET3JkI80/SbA8/n+Z7OZFoNvnbHpVxvI4gjqiuBP6VCPWvExvt\n",
       "6TSnz76Mag1DzGsNtZnA3xEBcx7xXtclQv78xjLrEW3032s8trV/pmjrEBWG6URN/Rs0BwFsQQTn\n",
       "oTRr7q1BNZXiasT9sMPqxPovS1NazxwFfDV3ew2iqeMqYi7+D9KsfRVtKhGO82me6PV6ornmVJq1\n",
       "jWOIJoEXEJ21vyCaeu5k7MO8oqwHPE7saDPbEP0LNWLI2D3Ezrfq5hFtvRfRbB6cQ4T+ScQO+9+I\n",
       "M4q/THSo75h7fFEnKrUO7z2O2O4uJHZQLwHOJq45vU5jmaxZdKTx6e8kjkSL3IlVaYfVj+u/EDXi\n",
       "bNl808aUxu2dCynRyN5KdBrfRNR+s1r62USTSN4mRIfYuxq3X01cS2Cj7hdz0j5JswOytfb1WmLn\n",
       "VTWt0yFkTWqLiBFSdxAd/wPEENNriXbjfYk+iw/0srCjaB1qvDbRuXwxcVr+TsQ1LXYn+lw+Q7R/\n",
       "LyQ6499P7NAzexNNVotZ9UzPbqpqYPbL+i+VnYkwhXK2DWcdzFmz0sFER+e2xEaXtX9Pyz3mAOCL\n",
       "xAdfBd8gypwPyaq2K2YnE+XNIj6POxq3pxOd6Ic0bs8kdgJZc04Z1sM7iIpDNl77DCLodgUuIJrg\n",
       "vk9zkMBaRH/DF4hmqP1zzzWNOOpcSvOEv17oZmB2e/6cflj/pXU75ejAHM3FRFMTRJPODUTzwCJi\n",
       "Q8hGG21BNOPUqNYZqLOKLkAHvILhNb69iKabtzX+vzPxBc3Gmx9JNCdmI5G2ptnZDs25Znotqzi8\n",
       "mBimnB3pfh7YnNiufkSEXmYBMcoIVp39MmsS2bjTBV2NqgZmv6z/Uiv78L61gd8QG+XJRPvidGLW\n",
       "wfcTJ4B9mBhXe0pBZeyEKtbqpxJfvvuJHW6N6CC/mdgJ3Eo0F2xBNLNdmnvsrTQ72DNFHmXmX3tb\n",
       "osnjQ8R0Ap/J3X8xzVFDhxMTumUnIGafYVHfqSoHZqnXf9lDciLKPqf100TH5g+IcwPeC/y28fO9\n",
       "xv+mE6NabiykhJ1R9s9hJDOItt/Tiea33xBh81XiBLeDiSarm4DfEe316xAd0XUilMqgRqz/7Yia\n",
       "8B7EGPO3E1M7PEu8t4eI/oaNiP6jHYhmkGzKgaGW371Uo3nNhPWJo6bNiSkpFhAd5o8RO+IhIsNe\n",
       "QeyQf0uMYnsmd98Q8b4Bft+Dsld9/auDasQ0BFlNYzrVrBH3o+uJwLiAaCM+H7iPqFVu1ljmecQR\n",
       "2iLiCwrNz6+oz7F1WN5somJxOc2a8iHEkeNZxNDYW4j3uSXD27HLMp5+O6JWfAFxot4txBz8VxEh\n",
       "ClHOI4jmm6uJM0+L0E/rX12wM82OP5XDBkTn+d3EOHqIppvP0ZzmYjeiGW4zyjn2+ZU026bfTRxJ\n",
       "5ocdfoXoc5hBHL1k7d5FngAG/ROYVV3/6oGydzD3q9WFwUlEjXEGETwfI5rXriRO7ntry/JFhEtr\n",
       "/8BBRPvv5UQNOJu64qfEOR6Z/YkQ2rTbBZykqgRmv65/dZF782JNa7mdfR7TiYDPZlVdk9gxHzLC\n",
       "Y3ptpI7gLYkTvuYSJxc+RjRFrU3zspf5cm8xjufstqoGZr+sf6mv5dvYNydmdtyKVb9sWegfREyX\n",
       "0DqbZ36ZXqoRZ15nQ3o3JDqYs0nLBogTwu4hRntdSYz6gGiiOmmE5ytCVQOzX9a/1Ldav1SbN37P\n",
       "AX7C6s+u/DjNsfWjPWevTCMuInMdcYnIe4jhoktpjjP/Cs1pfb9OjByaS/Q/FN3fUPXArPr6H8am\n",
       "BfWbAZrD8CCC5VZijpWtgV8TZ2CuSXP4XyYbxvdtYvhlUbITtrIhhf9N7LQOJ85mPocIzlcS8yzt\n",
       "RJzP8cfG/64l+ol+23j8FIob4jetUbY3Eme8fopog9+LOEP2PmJm2xOJYbCHEydW1Ylmnu8y/PPs\n",
       "hX5a/1Jfaj0F/xjipJb1iC/vHkTt61lixM3s3LJTWHUiraKab/Lm0Txzd1diJzTYuL0FMWrlDUSA\n",
       "Xk1MQ12G67K2jpbZmLjK1zKacyodT5x8tB0x9PUsYrK9G4gzbPPTFveqc7xf1r/Ut/YkavCvIWrt\n",
       "WxAn33yBOKP5ZprhvoiYvvk0on34RKIGmg+UVxNB1OupIlpD7XTiWsC3EVM7QJxZeklumatoXg9i\n",
       "fYYrwwiiKgVmP6x/qW+tRVxd64fEcL21iOac1xI1+ynEZGd1mh2wBxIn9EB8UfMn68wlappL6f1I\n",
       "kPzRxCKibTg7zX4XotlgQ6JJ6k7gTKJz+Wc0A7To8dxVDsx+WP9SX8vmO89kAXEGsJzYEZyeu79G\n",
       "jNAZaU7+bL7+Xs5Mugdxtm7Webk90ezxfeKs32W5+75EtH1DTKv9EDGVQFnO5ahiYPbT+pf63qZE\n",
       "U84gMeXtyUTIHEVc4exluWWPJ5p+tmF4U00WLs+ndzXKecQZvD8hmo8g3stKmqGyAxEox+TK9yti\n",
       "h7Qhwy+0XdSMnFUNzH5Z/1JSphG1yf8kaudnEc03FxHzFV1HNPV8mzgVPz/Msqj21blEp/Gxuf9l\n",
       "UwOcQUzOBnE26dHEmPTsRKTFNM8uzRTxPqocmHOp/vqXkrYN0X6f1dyPIS5reSTRdPDmgso1mnto\n",
       "nkB0AjEL55bEe3iAGK4IMXrlXIZPAVy0uVQ/MKu8/iW1+CIx+qZVWTrT5gGPEmO1v0mM4c6cQByl\n",
       "ZAZpHpnUcj9FqnpgVn39S0kbIL7EJxIXxbiM4RetL+Nh9znESUYjuZ+oHeeVKWT6ITCrvP6l5L2E\n",
       "uJrQYO5/Zf6S5q9+BlHWU4i27yrMplj1wKz6+pfUUKM8zTdjWURMHLYrcRLYpcS8/JkyHplk+iEw\n",
       "q7z+JVGtL2kNeJI4K3i3gssyGVUPzKqvf0kVkz+7tCpHJpl+CMwqr39JFVWqKXQnoF8Cs6rrX5J6\n",
       "zsCUJEmSJEmSJEmSJEmSJAD+H5gyvst3xuJTAAAAAElFTkSuQmCC\n"
      ],
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1139e1b90>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.preprocessing import scale\n",
    "logreg_new = LogisticRegression(C=1).fit(scale(rich_features_final), target)\n",
    "\n",
    "feature_names = rich_features_final.columns.values\n",
    "x = np.arange(len(feature_names))\n",
    "plt.bar(x, logreg_new.coef_.ravel())\n",
    "_ = plt.xticks(x + 0.5, feature_names, rotation=30)\n",
    "\n",
    "# Rich young women like Kate Winslet tend to survive the Titanic better\n",
    "# than poor men like Leonardo.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training Non-linear models: ensembles of randomized trees"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`sklearn` also implement non linear models that are known to perform very well for data-science projects where datasets have not too many features (e.g. less than 5000).\n",
    "\n",
    "In particular let us have a look at Random Forests and Gradient Boosted Trees:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Library/Python/2.7/site-packages/sklearn/tree/tree.py:28: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility\n",
      "  from ._tree import Criterion\n",
      "/Library/Python/2.7/site-packages/sklearn/tree/tree.py:28: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility\n",
      "  from ._tree import Criterion\n",
      "/Library/Python/2.7/site-packages/sklearn/ensemble/gradient_boosting.py:48: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility\n",
      "  from ._gradient_boosting import predict_stages\n",
      "/Library/Python/2.7/site-packages/sklearn/ensemble/gradient_boosting.py:48: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility\n",
      "  from ._gradient_boosting import predict_stages\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Random Forest CV scores:\n",
      "min: 0.777, mean: 0.801, max: 0.831\n",
      "CPU times: user 54.1 ms, sys: 21.2 ms, total: 75.3 ms\n",
      "Wall time: 382 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "rf = RandomForestClassifier(n_estimators=100)\n",
    "scores = cross_val_score(rf, rich_features_final, target, cv=5, n_jobs=4,\n",
    "                         scoring='accuracy')\n",
    "print(\"Random Forest CV scores:\")\n",
    "print(\"min: {:.3f}, mean: {:.3f}, max: {:.3f}\".format(\n",
    "    scores.min(), scores.mean(), scores.max()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Gradient Boosted Trees CV scores:\n",
      "min: 0.793, mean: 0.827, max: 0.848\n",
      "CPU times: user 46.8 ms, sys: 16.4 ms, total: 63.2 ms\n",
      "Wall time: 360 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "from sklearn.ensemble import GradientBoostingClassifier\n",
    "\n",
    "gb = GradientBoostingClassifier(n_estimators=100, learning_rate=0.1,\n",
    "                                subsample=.8, max_features=.5)\n",
    "scores = cross_val_score(gb, rich_features_final, target, cv=5, n_jobs=4,\n",
    "                         scoring='accuracy')\n",
    "print(\"Gradient Boosted Trees CV scores:\")\n",
    "print(\"min: {:.3f}, mean: {:.3f}, max: {:.3f}\".format(\n",
    "    scores.min(), scores.mean(), scores.max()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Both models seem to do slightly better than the logistic regression model on this data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise**:\n",
    "\n",
    "- Change the value of the learning_rate and other `GradientBoostingClassifier` parameter, can you get a better mean score?\n",
    "\n",
    "- Would treating the `PClass` variable as categorical improve the models performance?\n",
    "\n",
    "- Find out which predictor variables (features) are the most informative for those models.\n",
    "\n",
    "Hints:\n",
    "\n",
    "Fitted ensembles of trees have `feature_importances_` attribute that can be used similarly to the `coef_` attribute of linear models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%load solutions/04B_more_categorical_variables.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Logistic Regression CV scores:\n",
      "min: 0.770, mean: 0.789, max: 0.816\n",
      "Gradient Boosting Trees CV scores:\n",
      "min: 0.804, mean: 0.827, max: 0.848\n"
     ]
    }
   ],
   "source": [
    "features = pd.concat([data.get(['Fare', 'Age']),\n",
    "                      pd.get_dummies(data.Sex, prefix='Sex'),\n",
    "                      pd.get_dummies(data.Pclass, prefix='Pclass'),\n",
    "                      pd.get_dummies(data.Embarked, prefix='Embarked')],\n",
    "                     axis=1)\n",
    "features = features.drop('Sex_male', 1)\n",
    "features = features.fillna(features.dropna().median())\n",
    "features.head(5)\n",
    "\n",
    "\n",
    "logreg = LogisticRegression(C=1)\n",
    "scores = cross_val_score(logreg, features, target, cv=5, scoring='accuracy')\n",
    "print(\"Logistic Regression CV scores:\")\n",
    "print(\"min: {:.3f}, mean: {:.3f}, max: {:.3f}\".format(\n",
    "    scores.min(), scores.mean(), scores.max()))\n",
    "\n",
    "gb = GradientBoostingClassifier(n_estimators=100, learning_rate=0.1,\n",
    "                                subsample=.8, max_features=.5)\n",
    "scores = cross_val_score(gb, features, target, cv=5, n_jobs=4,\n",
    "                         scoring='accuracy')\n",
    "print(\"Gradient Boosting Trees CV scores:\")\n",
    "print(\"min: {:.3f}, mean: {:.3f}, max: {:.3f}\".format(\n",
    "    scores.min(), scores.mean(), scores.max()))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%load solutions/04C_feature_importance.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": [
       "iVBORw0KGgoAAAANSUhEUgAAAYAAAAEeCAYAAACDq8KMAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\n",
       "AAALEgAACxIB0t1+/AAAH6BJREFUeJzt3Xu4HWV96PHvTjZRCIkkiqAhuoEEEQqorSGY2m4k0ijV\n",
       "oFAickcxSgO2ak/keAqJp+2RVixHaTHUHBRKjZcCInLXriMqt1DYYEswQdMDkVO5VS5PbROz+8dv\n",
       "xjV77bWzLnv2mtmZ7+d51rPXrJlZ8+5ZM+9v3su8A5IkSZIkSZIkSZIkSZIkSdpJLQE2ABuBlTtY\n",
       "7o3ANuC4LtaVJJXMVGATMADsAtwPvHaM5b4DXE89ALS7riSpAFNazF9AZOKbga3AOmBpk+XOAb4O\n",
       "PNHFupKkArQKAHOARzPTjyWfNS6zFLg0mR7uYF1JUkH6W8wfbjEf4GLg48myfcmr3XU57LDDhoeG\n",
       "htpZVJJUNwS8bjxf0KoEsAWYm5meS1zJZ/06Ub3zE6L+/6+Bd7a5LkNDQwwPD+fyuuCCC3L7LtNk\n",
       "msqaLtNkmoaHhwEO6ySzb6ZVCWA9MJ9oyP0psAw4sWGZ/TLvLwe+CVyXfHerdSVJBWkVALYBK4Cb\n",
       "iV49a4GHgOXJ/DVdrCtJKoFWAQDgxuSVNVbGf0Yb606YwcHBXm2qbaapPWVME5QzXaapPaaptb7W\n",
       "i0y44aQ+S5LUpr6+PhhnHt6qEViStJMyAEhSRRkAJKmiDACSVFEGAEmqKAOAJFWUAUCSKsoAIEkV\n",
       "ZQCQpIoyAEhSRRkAJKmiDACSVFEGAEmqKAOAJFWUAUCSKqqdB8JMuGRc656aMWMWzz77dM+3K0ll\n",
       "UYoHwkARD4TpwwfRSJqsfCCMJKlr7QSAJcAGYCOwssn8pcAQcB9wL/CWzLzNwAPJvLvHk1BJUr5a\n",
       "FR+mAg8Di4EtwD3AicBDmWWmAy8k7w8BrgHmJdM/AX4d2FFlu1VAktShXlQBLQA2EVfyW4F1xBV/\n",
       "1guZ97sDTzbML0M7gySpQasAMAd4NDP9WPJZo2OJUsGNwLmZz4eB24D1wFndJ1OSlLdW3UDbrSO5\n",
       "Nnm9GbgSeE3y+SLgcWBP4FaiLeH20auvyrwfTF6SpFStVqNWq+X6na2qZxYSufOSZPo8YDtw4Q7W\n",
       "eYSoOnqq4fMLgOeBixo+tw1AkjrUizaA9cB8YACYBiwDrmtYZv9MIt6Q/H0K2A2YkUxPB44GHhxP\n",
       "YiVJ+WlVBbQNWAHcTPQIWkvU9S9P5q8BjgNOJRqJnwfek8zbG7g6s52rgFvySrgkaXzK0EPHKiBJ\n",
       "6pB3AkuSumYAkKSKMgBIUkUZACSpogwAklRRBgBJqigDgCRVlAFAkirKACBJFWUAkKSKMgBIUkUZ\n",
       "ACSpogwAklRRBgBJqigDgCRVlAFAkiqq1RPBKmnmzNk899wzPd/ujBmzePbZp3u+XUnV5BPBms3p\n",
       "66NsaZKkLJ8IJknqWjsBYAmwAdgIrGwyfykwBNwH3Au8pYN1JUkFaVV8mAo8DCwGtgD3ACcCD2WW\n",
       "mQ68kLw/BLgGmNfmumAVUHbLVgFJaksvqoAWAJuAzcBWYB1xxZ/1Qub97sCTHawrSSpIqwAwB3g0\n",
       "M/1Y8lmjY4kr+xuBcztcV5JUgFbdQNutj7g2eb0ZuBI4sLNkrMq8H0xekqRUrVajVqvl+p2t6o8W\n",
       "ErnzkmT6PGA7cOEO1nmEqP6Z3+a6tgHUt2wbgKS29KINYD2RkQ8A04BlwHUNy+yfScQbkr9Ptbmu\n",
       "JKkgraqAtgErgJuJXj1ribr+5cn8NcBxwKlEQ+/zwHtarCtJKgHvBG42xyogSSXnncCSpK4ZACSp\n",
       "ogwAklRRBgBJqigDgCRVlAFAkirKACBJFWUAkKSKMgBIUkUZACSpogwAklRRBgBJqigDgCRVlAFA\n",
       "kirKACBJFWUAkKSKMgBIUkUZACSpogwAklRR7QSAJcAGYCOwssn8k4Ah4AHg+8ChmXmbk8/vA+4e\n",
       "T0IlSfnqbzF/KnAJsBjYAtwDXAc8lFnmx8BvAT8ngsVlwMJk3jAwCDydW4olSbloVQJYAGwiruS3\n",
       "AuuApQ3L3EFk/gB3Afs0zB/XU+slSROjVQCYAzyamX4s+Wws7wNuyEwPA7cB64GzukmgJGlitKoC\n",
       "Gu7gu44EzgQWZT5bBDwO7AncSrQl3D561VWZ94PJS5KUqtVq1Gq1XL+zVfXMQiJ3XpJMnwdsBy5s\n",
       "WO5Q4OpkuU1jfNcFwPPARQ2fD3cWZ/LSx/Bw8+329fVRtjRJUlbkU+OrYm9VBbQemA8MANOAZUQj\n",
       "cNariMz/ZEZm/rsBM5L304GjgQfHk1hJUn5aVQFtA1YANxM9gtYSPYCWJ/PXAOcDs4BLk8+2Eo3H\n",
       "exOBId3OVcAteSVckjQ+ZeihYxVQfctWAUlqSy+qgCRJOykDgCRVlAFAkirKACBJFWUAkKSKMgBI\n",
       "UkUZACSpogwAklRRBgBJqigDgCRVlAFAkirKACBJFWUAkKSKMgBIUkUZACSpogwAklRRBgBJqigD\n",
       "gCRVlAFAkiqqnQCwBNgAbARWNpl/EjAEPAB8Hzi0g3UlSQVp9UDhqcDDwGJgC3APcCLwUGaZI4B/\n",
       "Bn5OZPirgIVtrgs+FD67ZR8KL6ktvXgo/AJgE7AZ2AqsA5Y2LHMHkfkD3AXs08G6kqSCtAoAc4BH\n",
       "M9OPJZ+N5X3ADV2uK0nqof4W8zupjzgSOBNY1Pm6qzLvB5OXJClVq9Wo1Wq5fmer+qOFRO68JJk+\n",
       "D9gOXNiw3KHA1clymzpc1zaA+pZtA5DUll60AawH5gMDwDRgGXBdwzKvIjL/k6ln/u2uK0kqSKsq\n",
       "oG3ACuBmolfPWqIXz/Jk/hrgfGAWcGny2VaiAXisdSVJJTCu4kNOrAKqb9kqIElt6UUVkCRpJ2UA\n",
       "kKSKMgBIUkUZACSpogwAklRRBgBJqigDgCRVlAFAkirKACBJFWUAmERmzpxNX19fT18zZ84u+t+W\n",
       "NEEcCqLZnJIOBVFMuhyeQiojh4KQJHXNACBJFWUAkKSKMgBIUkUZACSpogwAklRRBgBJqigDgCRV\n",
       "VDsBYAmwAdgIrGwy/0DgDuAXwEcb5m0GHgDuA+7uOpWSpNz1t5g/FbgEWAxsAe4BrgMeyizzFHAO\n",
       "cGyT9YeBQeDp8SZUkpSvViWABcAm4kp+K7AOWNqwzBPA+mR+M2UYbkKS1KBVAJgDPJqZfiz5rF3D\n",
       "wG1EgDirs6RJkiZSqyqg8Y4Ctgh4HNgTuJVoS7h99GKrMu8Hk5ckKVWr1ajVarl+Z6vqmYVE7rwk\n",
       "mT4P2A5c2GTZC4DngYvG+K6x5jsaaH3LjgYqqS29GA10PTAfGACmAcuIRuCm6WmY3g2YkbyfDhwN\n",
       "PNhVKiVJuWtVBbQNWAHcTPQIWkv0AFqezF8D7E30DppJlA4+DBwEvBy4OrOdq4Bbcky7JGkcytBD\n",
       "xyqg+patApLUFh8II0nqmgFAkirKACBJFWUAkKSKMgBIUkUZACSpogwAklRRBgBJqigDgCRVlAFA\n",
       "kirKACBJFWUAkKSKMgBIUkUZACSpogwAklRRBgBJqigDgCRVlAFAkirKACBJFdVOAFgCbAA2Aiub\n",
       "zD8QuAP4BfDRDteVJBWk1QOFpwIPA4uBLcA9wInAQ5ll9gReDRwLPANc1MG64EPhs1v2ofCS2tKL\n",
       "h8IvADYBm4GtwDpgacMyTwDrk/mdritJKkirADAHeDQz/VjyWTvGs64kaYL1t5g/nrJ/B+uuyrwf\n",
       "TF6SpFStVqNWq+X6na0CwBZgbmZ6LnEl344O1l3V5ldKUjUNDg4yODj4q+nVq1eP+ztbVQGtB+YD\n",
       "A8A0YBlw3RjLNjZGdLKuJKnHWpUAtgErgJuJXj1riV48y5P5a4C9iR4+M4HtwIeBg4Dnx1hXklQC\n",
       "4+pClBO7gda3bDdQSW3pRTdQSdJOygAgSRVlAJCkijIASFJFGQAkqaIMAJJUUQYASaooA4AkVZQB\n",
       "QJIqygAgSRVlAJCkijIASFJFGQAkqaIMAJJUUQYASaooA4AkVZQBQJIqqtUjISXlYObM2Tz33DM9\n",
       "3+6MGbN49tmne75dTQ7tlACWABuAjcDKMZb5bDJ/CHh95vPNwAPAfcDdXadSmuQi8x/u+auIoKPJ\n",
       "o1UJYCpwCbAY2EI8/P06Rj7c/e3APGA+cDhwKbAwmTcMDAJegkhSybQqASwANhFX8luBdcDShmXe\n",
       "CXwpeX8XsAewV2Z+GR48L0lq0CoAzAEezUw/lnzW7jLDwG3AeuCs7pMpScpbqyqg4Ta/Z6yr/N8E\n",
       "fgrsCdxKtCXc3uZ3SpImUKsAsAWYm5meS1zh72iZfZLPIDJ/gCeAa4gqpSYBYFXm/WDykiSlarUa\n",
       "tVot1+9sVT/fDzwMHEVk5ncDJzK6EXhF8nchcHHydzeiEfk5YDpwC7A6+Zs13H5BI099DA83325f\n",
       "Xx9lSxMUla4dp0ntKesxpckrjqnxtbG2KgFsIzL3m4nMfC2R+S9P5q8BbiAy/03AC8AZyby9gasz\n",
       "27mK0Zm/JKkgZeihYwmgvmVLADupsh5TmrzyKAE4FIQkVZQBQJIqygAgSRVlAJCkijIASFJFORy0\n",
       "xsVhjqXJy26gzeaUtMteGbuBlnVflY37SXmzG6gkqWsGAEmqKAOAJFWUAUCSKsoAIEkVZQCQpIry\n",
       "PgBJamFnvd/F+wCazSlpn23vAxix5UnVv939NLmV8ffzPgBJUtesApIqamet1lD7rAJqNqeExT2w\n",
       "Cqhhy2Omq4wZWxn3UxnTBMX8fq2CUhn3VR5VQAaAZnNK+GODAaBhy5Pq9zNNI7ZcwmOqjGmCMrQB\n",
       "LAE2ABuBlWMs89lk/hDw+g7XlSQVoFUAmApcQmTkBwEnAq9tWObtwDxgPvAB4NIO1s1ZbWK/viu1\n",
       "ohPQRK3oBDRRKzoBY6gVnYAmakUnoIla0QloolZ0ApqoFZ2AEVoFgAXAJmAzsBVYByxtWOadwJeS\n",
       "93cBewB7t7luzmoT+/VdqRWdgCZqRSegiVrRCRhDregENFErOgFN1IpOQBO1ohPQRK3oBIzQKgDM\n",
       "AR7NTD+WfNbOMq9sY11JUkFaBYB2Wz3K0JgsScrRQuCmzPR5jG7M/Tzwnsz0BmCvNtcFuJ8INL58\n",
       "+fLlq/3X/UywfuARYACYlmywWSPwDcn7hcCdHawrSSqxtwEPEw265yWfLU9eqUuS+UPAG1qsK0mS\n",
       "JEmSJOVkatEJaJCmxx5TE6NMI9u+hPIdfy8Hdik6EV2YBhwH7Fd0Qpoo07ncD+zZ6UplOmnyNAX4\n",
       "ZfL+1UUmhPpBkqbnxU3mTXZFjCo7neiB9t8L2PZYpgNrgC8DnyMy3aLtDvwVcDWRpgOLTU5Hzga+\n",
       "BywGjgBmFJscIALSwcnf4YLTkjoL+C6xn8p24dFT2YD2GuBW4sD/42S6SO8mDuZPAX9YcFrykt3f\n",
       "fcDL6E1J56XAdcTv+8/Ue5cVefDPA+4BPgPMBL5BBKepFBfo9wMeAv4S2JXorffRZF7ZLz5OA/6B\n",
       "+nlbhoztHOCHwNeIc/ntRCAoynTgWuBbxMgL/UQpr+y/be6yV6DTgFnAWiIiziLuSP44I6++J0of\n",
       "ow/WBcQPdRBwAjE8xqIepGWiHNQwfSJxYlwMfLFHaTiK+N0/QZyQRZsFHJOZPoLIfIuWvfA5H/gj\n",
       "is20WkkvKj4FvC/zWR/FZmwfAL5DBNUpwPuBbwLLCkzTrwFfz0y/qGF+5QLB7xGR+WBiULqPAf+X\n",
       "6Ka6aw+2n834pwGHEgfLh4BziSLtfYzsQjvZvBn4CvBbyfRhwN8CrwIOAbYDvz2B208P6jSj2IsY\n",
       "g+p3k+miHnLUT/0EnEbc//I14rgr+kTsJwLzU8SFyFrgN4tMUAv9xHn85mS6yKv/9Dj7NFEqgfrv\n",
       "eQ7wvxk5AnIv7QH8AngTkbesJqr5/ltB6emZKYysfngx8HfA9cTV6SziQP82kSml3khv2j3OIKoo\n",
       "vkn8SEcTA+J9hnom8XIiUE0W6X6bQ5SmPkWcmCcQB9yFwN3E/95rZwLfz0wXneECvAP4atGJoL4v\n",
       "Dkv+7gqsIqohy/hEwDSz/1NGluz6if/lpYw8p3uhH/gB9YCUlqBeTaTxrT1OD9T309nAPxLH//8k\n",
       "zsNNyefQ4lyYjI3A/cRV5nZgN2JH/AL4F+B1RJ3wM0Tx+y7ix3oZUSd7LqOLSeNxFLBvZnoWEYiW\n",
       "EKOfHkqUSjYSDYP/CvwHccB8jQhWZcisdiQ90LYnf7cQ+3U2caW/CfgT4AngcOByoph8eA7bbWff\n",
       "TCH27U+JktVbqZ+oeeukPv9o4gIA4FhGV5vlpVWa0obKB5K//w78G/AKYNsEpakdA8RIwjOT6cbO\n",
       "ElcSvVrSkt024n9ZSLT59Crvmpps+x+I/APgP4l85V+SNE1kaWqAHe+nvyaOtUXAnxPD7awm9hOU\n",
       "p6F6XKYCf0a93ryfuJq+IfkLUQpYD5ycTO8LfJA4Cf+R/B9IM5vIDG8j6gchAsBt1LtjnQpcRtwd\n",
       "fQBwB9Eo/QPg+JzTM9HeCZxEXIFNIYq/lxIH5E3E1cc+RMPYXcCH6T64ZYv8+9NeFcBKIkjdSZw0\n",
       "eWs3TelV9V8BFxENdDcRGW5Raco6nKjLPmUC0tOJZcRF0MIdLHMa0V52FtF7aQ0xpMybJiA9AzTP\n",
       "aFMHEmM5v6Ph8wupV4dOhHb2E9QD4mzgGiLv2Sm8n9jx64gMdhZxAH+S+JE2Ej8CxNX2nYw8EV5G\n",
       "dIXL2x5EcDmFKH6dSlyZfI4oGZCk7x7gL6hXW81r+J6ylQAa07MPEWivJzLZLxMny35EADiBqM76\n",
       "DHHg3UpckYzXK4EriGAy0CRdWWcRjdDvzWG7eaXpcaIUemxJ0rQX8eS++4hAXpRs+r4AfIS4qGic\n",
       "lzoJ+F9EEP0ME1dt1U5GezoRkE4nHoB1OfAsUerPu4TXyX7qA36DaI94kCiRl6HX1Li9nLiqy9b5\n",
       "vYjYEfsRdaw14kdJf7jriQabXriCqEt9I3HFt5q44v8g9fsP1hKNpvMb1i3jD9SsWD1IXBlBHGA/\n",
       "Iq7EIEaB/SL1vuWvzKzXN8b3NdO4L6YQJaVPtFgvPRH2bvg8j0yi2zRBVA8c1fBZHr/3eNLUTxyn\n",
       "jev3Uvp7HUN0zvgW8ZyQxn0Fo9OWvYjLMwh0GpBOI6pabiNKeK8jqh7zvJDrZD+l9k/Sla2Sbvn7\n",
       "ljETynoBmAv8nKjbv5LoUvkNopFoE9EA+DoiM/o/xFXXk0TJoBf2IgLRAcSBMC1J75lE75+fEl3x\n",
       "niCKr+mPW6a6uanUh5iFqN4ZAH5G7PefE/v8GaJv+0lE/ef3iCuPZ4jf4vnM922n/f8xXe69RAlp\n",
       "M9GN8kkiuLyN2L+PJ9tI92Fa//1cMt2ffFfaXjEe3aZpSrL9HyfTu9DZvpioNA0TVZZQb0eb6GMw\n",
       "7RKd3c5LiBLxp4lS/CFEAH+EaJtIDTes/5/Uu4P+knykT3o/hsj8XwS8i6hF+EnDsuk+HCJKudcS\n",
       "NQD/H7g3h3R0u59I1n0qSde/UT8vWp4Hk6ER+FyiiPUAUcz/BPWT7PFkmZ8RO+cAooh7w+ivmRC7\n",
       "E/X7XyHaAVYTP8QriB4xf0C0xm+kfkCVKeNPq6XSE2oPop/48cSV/58TdaKvJK78zyYOwF2J//VZ\n",
       "om/5jcn66f/W6gRtvFqaS9zJeDqRmf87cDtx0O9B1Gsuod64m54wvySOgwOSz9KGwm7klaa0g8IB\n",
       "xL7d2mV6JjJNvWr8HSZ+o+lEh4gXJ9P/Qf1i4U+I/uzpjUzZdKfrZzO08QT3xnt0homM9hyimukY\n",
       "6lWY+zasu71h/WcZ3RuxW+PdT+n5NjWZl54XO43Tgb/PTE8Dfp+48n6E2Dkdj4ORg5cATxP3GaQO\n",
       "JBqF+ogM834iIy2zfYkg+zfUq8/2InoYnENcbf6YuJv5a0QQzlYndHKTTmPxvY8oUXws89luTdb5\n",
       "BqPbFmYT+/4O6o133TBN+Wm8E/wDxAXQGqLN6FCiLWIpkeFBvSq3WX/6PySuhBv/1/HIZrS7E/ss\n",
       "7cmzP1HlsozRGW0qj7u7J8N+Ko0+4m7eQzOfTUmmjygkRXV/Sf2Ea6xSewdxg1SZNA7fkN6zsJy4\n",
       "k/cuouG8n+hSeT1RJ/o2oj3jkzlt+1Si6g4iwN9DBPmLidLSMqJU91HiKXN/ysj9+3EiQ/udcaTH\n",
       "NOWn8UbL3YgOBF8gqsEWEiWW3ybayC4j6tsXEFUpf0xcUKWOJu7juYDxdd0uW0Zb1v1UekdQf9pY\n",
       "mXrOfINoJM2etGWsWktvpMmaBVxFZPoQV0Q3Ej17IKoVLiZOFOjufzyIaA9JT5ijiG65VxK9Ov4o\n",
       "+XwxUe20JzEEwBXEAX0yo0eCfDtxInQ7uqVpytcfEFUo6WBtq4mMchExYN9XiPai45L5uwJHEneQ\n",
       "f5eRQ2jsQnQ2WEeUQrs1kRltt0NplHE/TSo/YGQpoAxmFZ2AFg5n5NXKYqKq5/3J50cQB13at/o0\n",
       "4goz7cV0ACMzlrQhrpV0mZOJk+uo5LNziDrr6cTveTsxllBqHnHAX8zIIDOV8QdW05SvNLAcQnTN\n",
       "TkvjVxLtFQcTbWEXZNY5iAhgMHp0z7TKpbFXV6fKltGWdT9NOmXutVS2q/6pxAH1I6Jesw9YQXRf\n",
       "S28G+mwy72NEn+bUd6jf3JbqpNSVrb+ekaRjFREs+4irrgeJISQ+BHwpmXc80WbykXFs2zT1RnZb\n",
       "ryWqVP6MaIC+LDP/C0S70tuINowfUr8pMz1n8jqvy5jRlnE//UqZM9RmytSDplHZ0jaDqNc8n7if\n",
       "4knixPh7olHw94jqq1uJbp5vJa427yfqP7/TxTbTgzntgXAK0StqC3Ey9gP/RFR1XEOcDL9B3Cy1\n",
       "lbjJ7HLiKgxGd43rhmnKX9p98mDiyvpIok/8B4G3EP/Xk0Q31euJ4+9dRJ36R6iP8TPc8He8aUp7\n",
       "w7yUKLnOJbqPHkR0cvgZcfEzTOyzw4mLoKeImzm3ZeZluxOnPXG6SVPZ9pMq5Abi4P48Uf95KfAw\n",
       "cUW0T7LM7kQReDn1K8opDX9beQ0jLyYOI660riGuevqIRsoLiXrV1US7w5HEHZ7LM+mBfHpZmKZ8\n",
       "NfZKmk30gLuC+pX3CUSJ5SKitPJt4tibx8h687y6T6bSfXAwcXX9eeLGy28TY/h8ldiH6bZPIap7\n",
       "riUCa57KvJ9UIS8jeivcR9yrAFHV80Xq49K8mejytg/d3V2Zdh8dItoV3p18fj5RZZH1a0R9dToG\n",
       "zSeJBuf3ZJbJIzMzTRPrLdTrwtMeRtlA9HWiHWkGUeJM69mzN+/loewZbVn2k3ZyrQ7cc4irnRnE\n",
       "SfIXwM1EQ9gD1B+60e73pQaIIuwpRAniI0RweRVxR3baXz3bG+MM4gQ9jNHyOAFNU34ag8zxRL30\n",
       "FcQVdTq0yQ8ZOd7RMURmN2eiE5goOqOdLPtJO7nGrn7pgf1iIsNPR0l9EdGT6oQm63RqiHpD2+uJ\n",
       "7nOHEHdvr6T+TNwpRD3sjGS72Sez5X2lY5rGp1npYh5xU+AA0f30Z0R14m5EW9JdjDyW9m/jO8eb\n",
       "rqIz2rLuJ1VAto5+LtFPfD6jD6A00zieGFSqcSTS7DLdGCB6GUE0qN1CHOyLiGqMrxJ3WN5EDJbX\n",
       "iweQmKbu9BGNzr+fTO9JdCJIB2HrJ0om9xN95L9CfVTR+4iSZuP35ZWuRkVmtGXdT9rJNRsXBqL4\n",
       "ey+t70z8NPW+/WN9Zze+QAwY913i5Eu/dy+iu92XiTtY896uacrXLsQgc98i2iXuJ7oMr6PeL/7r\n",
       "1Ic9voboLTZAtCnlHbTKmtGWbT+pAhoPmt2J+vuriHGQLk4+b3Yr+ERXHexKdE1MA9CObkTqVTWG\n",
       "aWpPY+Pn3sSDfYaoD2HyISJ4HUz0HruIGOvqJuKGq+zYQnm2T5Qpoy3zftJOrPF29jOJG0VeQlzR\n",
       "HEkc+NuJHj2zM8tOYXQ3wYnKWNJb6aH5iVfEAW+axtZ4Nbwv9bu9FxH3hQwm0/sTmdm7iXaLa4m7\n",
       "kifiObhly2jLup+0kzuKuBHrd4mr+v2JG1X+lmg4vI16Zr+cqFr4H0Td5wri6il78P8OcdJM1LAV\n",
       "fcQ45I11rUUyTc01ZornE892uJ0YHgSiSmptZpmvUq+2eikj5RG0ypjRlnE/aSe3K9EIeCfRdW1X\n",
       "4krxHcSV/xSi2qdGvUH3XcTNLxAHX/bGlgHiKmkdE9/dLO3JUqZGLdM0UrYEuJyoSkmHG3gTMbT6\n",
       "nsRdtOuBTxEdCP6Jegacdz/1Mma0ZdxPqoB0LPJUejCvJob6vZM4QVJ9RA+gZs8bSJ9FsCD/ZO5Q\n",
       "mTLbVJXTdCRx70HaeHoYUa3yPeJO8aHMvL8jBj6DuCN5MzFUwkQNwFimjLbM+0kVMYeo+hkkhp9N\n",
       "+4mfTjxy8Q2ZZT9EVBUdyMiqnfREeAUWO6tsX+KGs3upPwtgDvEksTTzej2RcZ2ZTL+CeITpAiLj\n",
       "zT7ovN1RYFspW0Zb1v2kCtqFuBL6f8TV+0VEdc/fEAOFfYuoGrqFuK09263TzF6pAaJjwFmZz9Kh\n",
       "D1YTAwBC3HB2BtGHPr2R6gLqd8um8ji2ypjRDlC+/SRxIFH/n17Zn0l09zyNKAa/t6B0afK4n3pj\n",
       "89nEyK/ziONqI/FcCIjeNJcwcojkvA1Q3oy2TPtJauoqondPIxuYNJZ9gX8lhhn+JvEErNTZRMky\n",
       "NUi9NNmXeeWprBlt2faTRD9xYK4ghgv+EiMfem9RU+34HPH8h2Z+RFxtZ01kZlbmjLZM+0kCoqHr\n",
       "Muq9HcADT53ZjXjgSHpDYR/wYaKuvYhRKMua0ZZtP0kj9GF1j7qznBg4bRFxo+DlxLMiUr0sTZY5\n",
       "oy3TfpJ+xQNP45Heffx9YriQopU1oy3bfpKkXGTvli26NFnmjLZM+0mSclWWoYjLntGWZT9J0k7L\n",
       "jFaSJEmSJEmSJEmSJEmSxuW/AHI+ShBWQ3tsAAAAAElFTkSuQmCC\n"
      ],
      "text/plain": [
       "<matplotlib.figure.Figure at 0x113a2d890>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "gb_new = GradientBoostingClassifier(n_estimators=100, learning_rate=0.1,\n",
    "                                    subsample=.8, max_features=.5)\n",
    "gb_new.fit(features, target)\n",
    "feature_names = features.columns.values\n",
    "x = np.arange(len(feature_names))\n",
    "plt.bar(x, gb_new.feature_importances_)\n",
    "_ = plt.xticks(x + 0.5, feature_names, rotation=30)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "GradientBoostingClassifier?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Automated parameter tuning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instead of changing the value of the learning rate manually and re-running the cross-validation, we can find the best values for the parameters automatically (assuming we are ready to wait):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 301 ms, sys: 51 ms, total: 352 ms\n",
      "Wall time: 5.09 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "from sklearn.grid_search import GridSearchCV\n",
    "\n",
    "gb = GradientBoostingClassifier(n_estimators=100, subsample=.8)\n",
    "\n",
    "params = {\n",
    "    'learning_rate': [0.05, 0.1, 0.5],\n",
    "    'max_features': [0.5, 1],\n",
    "    'max_depth': [3, 4, 5],\n",
    "}\n",
    "gs = GridSearchCV(gb, params, cv=5, scoring='roc_auc', n_jobs=4)\n",
    "gs.fit(rich_features_final, target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us sort the models by mean validation score:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[mean: 0.87356, std: 0.02462, params: {'max_features': 0.5, 'learning_rate': 0.1, 'max_depth': 3},\n",
       " mean: 0.87107, std: 0.02576, params: {'max_features': 0.5, 'learning_rate': 0.05, 'max_depth': 5},\n",
       " mean: 0.87079, std: 0.02837, params: {'max_features': 1, 'learning_rate': 0.1, 'max_depth': 4},\n",
       " mean: 0.87055, std: 0.02073, params: {'max_features': 0.5, 'learning_rate': 0.1, 'max_depth': 5},\n",
       " mean: 0.86975, std: 0.02440, params: {'max_features': 0.5, 'learning_rate': 0.1, 'max_depth': 4},\n",
       " mean: 0.86962, std: 0.02766, params: {'max_features': 0.5, 'learning_rate': 0.05, 'max_depth': 4},\n",
       " mean: 0.86866, std: 0.02616, params: {'max_features': 1, 'learning_rate': 0.05, 'max_depth': 5},\n",
       " mean: 0.86830, std: 0.03029, params: {'max_features': 1, 'learning_rate': 0.5, 'max_depth': 3},\n",
       " mean: 0.86756, std: 0.01717, params: {'max_features': 0.5, 'learning_rate': 0.5, 'max_depth': 3},\n",
       " mean: 0.86647, std: 0.01785, params: {'max_features': 1, 'learning_rate': 0.05, 'max_depth': 4},\n",
       " mean: 0.86573, std: 0.02612, params: {'max_features': 1, 'learning_rate': 0.1, 'max_depth': 3},\n",
       " mean: 0.86552, std: 0.02676, params: {'max_features': 0.5, 'learning_rate': 0.05, 'max_depth': 3},\n",
       " mean: 0.86469, std: 0.02907, params: {'max_features': 1, 'learning_rate': 0.1, 'max_depth': 5},\n",
       " mean: 0.86075, std: 0.01981, params: {'max_features': 1, 'learning_rate': 0.05, 'max_depth': 3},\n",
       " mean: 0.86039, std: 0.01578, params: {'max_features': 0.5, 'learning_rate': 0.5, 'max_depth': 4},\n",
       " mean: 0.85591, std: 0.02954, params: {'max_features': 1, 'learning_rate': 0.5, 'max_depth': 4},\n",
       " mean: 0.83984, std: 0.01910, params: {'max_features': 0.5, 'learning_rate': 0.5, 'max_depth': 5},\n",
       " mean: 0.83798, std: 0.02771, params: {'max_features': 1, 'learning_rate': 0.5, 'max_depth': 5}]"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sorted(gs.grid_scores_, key=lambda x: x.mean_validation_score, reverse=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.87355571493611861"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gs.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'learning_rate': 0.1, 'max_depth': 3, 'max_features': 0.5}"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gs.best_params_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We should note that the mean scores are very close to one another and almost always within one standard deviation of one another. This means that all those parameters are quite reasonable. The only parameter of importance seems to be the `learning_rate`: 0.5 seems to be a bit too high."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Avoiding data snooping with pipelines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When doing imputation in pandas, prior to computing the train test split we use data from the test to improve the accuracy of the median value that we impute on the training set. This is actually cheating. To avoid this we should compute the median of the features on the training fold and use that median value to do the imputation both on the training and validation fold for a given CV split.\n",
    "\n",
    "To do this we can prepare the features as previously but without the imputation: we just replace missing values by the -1 marker value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Fare</th>\n",
       "      <th>Age</th>\n",
       "      <th>Sex_female</th>\n",
       "      <th>Pclass_1</th>\n",
       "      <th>Pclass_2</th>\n",
       "      <th>Pclass_3</th>\n",
       "      <th>Embarked_C</th>\n",
       "      <th>Embarked_Q</th>\n",
       "      <th>Embarked_S</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>  7.2500</td>\n",
       "      <td> 22</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td> 71.2833</td>\n",
       "      <td> 38</td>\n",
       "      <td> 1</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>  7.9250</td>\n",
       "      <td> 26</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td> 53.1000</td>\n",
       "      <td> 35</td>\n",
       "      <td> 1</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>  8.0500</td>\n",
       "      <td> 35</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> 1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Fare  Age  Sex_female  Pclass_1  Pclass_2  Pclass_3  Embarked_C  \\\n",
       "0   7.2500   22           0         0         0         1           0   \n",
       "1  71.2833   38           1         1         0         0           1   \n",
       "2   7.9250   26           1         0         0         1           0   \n",
       "3  53.1000   35           1         1         0         0           0   \n",
       "4   8.0500   35           0         0         0         1           0   \n",
       "\n",
       "   Embarked_Q  Embarked_S  \n",
       "0           0           1  \n",
       "1           0           0  \n",
       "2           0           1  \n",
       "3           0           1  \n",
       "4           0           1  "
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "features = pd.concat([data.get(['Fare', 'Age']),\n",
    "                      pd.get_dummies(data.Sex, prefix='Sex'),\n",
    "                      pd.get_dummies(data.Pclass, prefix='Pclass'),\n",
    "                      pd.get_dummies(data.Embarked, prefix='Embarked')],\n",
    "                     axis=1)\n",
    "features = features.drop('Sex_male', 1)\n",
    "\n",
    "# Because of the following bug we cannot use NaN as the missing\n",
    "# value marker, use a negative value as marker instead:\n",
    "# https://github.com/scikit-learn/scikit-learn/issues/3044\n",
    "features = features.fillna(-1)\n",
    "features.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now use the `Imputer` transformer of scikit-learn to find the median value on the training set and apply it on missing values of both the training set and the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.cross_validation import train_test_split\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(features.values, target, random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Imputer(axis=0, copy=True, missing_values=-1, strategy='median', verbose=0)"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.preprocessing import Imputer\n",
    "\n",
    "imputer = Imputer(strategy='median', missing_values=-1)\n",
    "\n",
    "imputer.fit(X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The median age computed on the training set is stored in the `statistics_` attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 14.5,  29. ,   0. ,   0. ,   0. ,   1. ,   0. ,   0. ,   1. ])"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "imputer.statistics_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Imputation can now happen by calling  the transform method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "X_train_imputed = imputer.transform(X_train)\n",
    "X_test_imputed = imputer.transform(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.any(X_train == -1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.any(X_train_imputed == -1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.any(X_test == -1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.any(X_test_imputed == -1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now use a pipeline that wraps an imputer transformer and the classifier itself:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(0.77653631284916202, 0.82160982372072788, 0.8595505617977528)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "\n",
    "imputer = Imputer(strategy='median', missing_values=-1)\n",
    "\n",
    "classifier = GradientBoostingClassifier(n_estimators=100, learning_rate=0.1,\n",
    "                                        subsample=.8, max_features=.5)\n",
    "\n",
    "pipeline = Pipeline([\n",
    "    ('imp', imputer),\n",
    "    ('clf', classifier),\n",
    "])\n",
    "\n",
    "scores = cross_val_score(pipeline, features.values, target, cv=5, n_jobs=4,\n",
    "                         scoring='accuracy', )\n",
    "print(scores.min(), scores.mean(), scores.max())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The mean cross-validation is slightly lower than we used the imputation on the whole data as we did earlier although not by much. This means that in this case the data-snooping was not really helping the model cheat by much.\n",
    "\n",
    "Let us re-run the grid search, this time on the pipeline. Note that thanks to the pipeline structure we can optimize the interaction of the imputation method with the parameters of the downstream classifier without cheating:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 285 ms, sys: 45.4 ms, total: 330 ms\n",
      "Wall time: 3.44 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "params = {\n",
    "    'imp__strategy': ['mean', 'median'],\n",
    "    'clf__max_features': [0.5, 1],\n",
    "    'clf__max_depth': [3, 4, 5],\n",
    "}\n",
    "gs = GridSearchCV(pipeline, params, cv=5, scoring='roc_auc', n_jobs=4)\n",
    "gs.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[mean: 0.87521, std: 0.02455, params: {'clf__max_features': 0.5, 'clf__max_depth': 3, 'imp__strategy': 'median'},\n",
       " mean: 0.86977, std: 0.02696, params: {'clf__max_features': 0.5, 'clf__max_depth': 4, 'imp__strategy': 'mean'},\n",
       " mean: 0.86968, std: 0.03220, params: {'clf__max_features': 0.5, 'clf__max_depth': 3, 'imp__strategy': 'mean'},\n",
       " mean: 0.86581, std: 0.02859, params: {'clf__max_features': 0.5, 'clf__max_depth': 5, 'imp__strategy': 'mean'},\n",
       " mean: 0.86568, std: 0.03090, params: {'clf__max_features': 0.5, 'clf__max_depth': 4, 'imp__strategy': 'median'},\n",
       " mean: 0.86162, std: 0.02233, params: {'clf__max_features': 0.5, 'clf__max_depth': 5, 'imp__strategy': 'median'},\n",
       " mean: 0.86124, std: 0.02853, params: {'clf__max_features': 1, 'clf__max_depth': 4, 'imp__strategy': 'median'},\n",
       " mean: 0.85985, std: 0.02975, params: {'clf__max_features': 1, 'clf__max_depth': 5, 'imp__strategy': 'mean'},\n",
       " mean: 0.85929, std: 0.02333, params: {'clf__max_features': 1, 'clf__max_depth': 3, 'imp__strategy': 'median'},\n",
       " mean: 0.85704, std: 0.02546, params: {'clf__max_features': 1, 'clf__max_depth': 3, 'imp__strategy': 'mean'},\n",
       " mean: 0.85508, std: 0.02140, params: {'clf__max_features': 1, 'clf__max_depth': 4, 'imp__strategy': 'mean'},\n",
       " mean: 0.85469, std: 0.02320, params: {'clf__max_features': 1, 'clf__max_depth': 5, 'imp__strategy': 'median'}]"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sorted(gs.grid_scores_, key=lambda x: x.mean_validation_score, reverse=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8752060496598677"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gs.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": [
       "iVBORw0KGgoAAAANSUhEUgAAAYYAAAEZCAYAAACTsIJzAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\n",
       "AAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd4VFX6wPFvCD2FhCLSQQTEhqAgKmpABCLYARUFUde2\n",
       "a6etgoKuWH6LiFhQUFDsfQVU0EXjiigoJDTpiBRBgVBCy0Byfn+8dzIzyczkTpg+7+d55sncMve+\n",
       "c5Occ0+554BSSimllFJKKaWUUkoppZRSSimllFJKKaWUUkpF1HLggkgHEUUeBKZE6NyvA/+K0LmD\n",
       "7XpgTgU/q3+TSrnZCBwECoDtwJtAeiQDCrJqwJPA78j3XAMMjWA8WcDmMJ4vCbgHWAbst879AXCq\n",
       "tX0a8FgY4/FlDPK3Fw6vEz+ZYUyoFOkAVMAM0AdIA9oBpwGjIhpRxVT2sf5DoCuQDaQCA4HbgOdC\n",
       "EEOS9YomzyEZw91AJtAa+A9wSQjOlRyCY8bCuZWKO78B3dyW/w/43G25MzAf2A3kARe6bauN3HFu\n",
       "BfKBT9229bH23w38gGQ4ThutczZE7uIz3ba1B3bg+ke/GfjVOv5soKnbvsXA34G1wHov3+0i4BDQ\n",
       "qNT6TsBR4ARrOQcpVSwA9iIJp3tM/q5BDvC49R0PAi2Bm6yY91lx3Wbtm2LFU4SU0PYBDfC8W25u\n",
       "fa9BSClnB/CQ2/lqAG9Y1+NXYDi+SyCtrO95lo/tIL+/F4BZVjw/4bouIBnLJuS6/AJ0cds2BvjI\n",
       "in0v8rvqCPyIXKs/gOeBKm6fOQX4GtiFlFAfBHoChYADuS651r61gNes42xB7vKdN5+DkWs+Hthp\n",
       "bRsMfG9tTwKeBf60Yltqnfs26zyF1rk+s/bfiPy9gPztPQSss67JL0BjL9dOqbj1G65/iMbIP9Aj\n",
       "1nIj5J+ul7Xc3VquYy1/DryL/ANXBs631rdH/iE7Iv+gg6zzOBMI98xoLvA3t3j+Dbxkvb8cSfTb\n",
       "IAnCSCQxcCpG6pQzkCqj0p4CvvXxvTcCt1rvc5CE52SgJq7EDsq/BjnWsdpaMVZG7sZbWNsvAA4g\n",
       "1wQkUymdkI+mbMbwivWdTgcOI9fA/TvVsmJbiiTc3tyBXGt/Xre+z1lIgvgW8jt1uh7JJCsBDwDb\n",
       "gKrWtjFIInuZtVwd6IBkvJWAZkjmda+1Pc36/P3WMVKtfZ3XYHqp2D4FJiGZYT0k43ZmsoOBI8A/\n",
       "rHNVxzNj6Ikk6M5q0TbA8dZ7b9Vn7n+Tw5Dr2spaPg25CVIqYWzEdfdajPwzOu/KRlD2n3U2ktA3\n",
       "QO58a3k55iTK/uOtwpVxuP8T3oJkDiCZyCZcd6VfInehTpWQRLaJtVyM1Nn78iqeiZy7H5G7VZCE\n",
       "9gm3bW2RO8pK+L8Gzs+O8RMDyDW9x3qfRdmMYQxlM4aGbtsXAP2t9+uBi9223eLleE4jke/pzzRg\n",
       "sttyNrDSz/75uEp/Y5CM0Z/7gE+s99cBi3zsNwbPNob6SIZY3W3ddcA31vvBSInK3WBcGUM3YDVw\n",
       "NmWruKdRto3B/W9yNXCpjzhVBWgbQ+wxyJ15OpJodcNV9dAM6IdUCzhf5yF3Xk2QRGKvl2M2A4aU\n",
       "+lxjPBM7p0+Ac6xjXoAkivPcjvOc2zF2Wevdq4b8NeTuQDIwbxoid8rejrMJKd3Uxf818BVDNlIl\n",
       "s8va/xJcJQy7tru9P4jcXTvjdj/fFj/H2IXv7+/uT7f3h9zOBdJQ/yuwB/kutZDr4uv8rZFqqW3I\n",
       "38ZYXN+9CbDBRjwg172KdRzndX8ZKTk4+fvdf4NUkb2IfL9XkBKLHY3xXjWpKkgzhtj2P6RO+Glr\n",
       "eRNyF5fp9kpD2iE2I8VrbyWGTUiC4P65VOB9L/vuBr4CrgEG4HmHvwmpOnA/TgqS6DoZP9/nv8gd\n",
       "Y+n6Yee6b9zWNS31/giSsfi7Bt5iqAZ8bG0/ztr/C1yN0t7i9fcdStuGq8REqfelzUW+55kBHN/d\n",
       "+Ui1Sj+kui4TSezdG9hLxz4JyUhORP42RuJKFzbh2X7hrrjU8mak1FYH13WvhWdbVXnX7XnkJudk\n",
       "JMMaZvNzm634VZBoxhD7JiD1vmcj9c2XAj2Q+ufqSKmiEZJAfYm0B2Qgd3fOfuBTkPrtTkgikgL0\n",
       "xvNO1N07wI3A1dZ7p5eRRsCTreVaSCJl11zr9bF1jGSkIflNK27nXWEScANShVQTqQb7EElA/F0D\n",
       "3D7vVNV67UQSu2zrs05/Iomde5fgQHoyfYBUgWVYMdyF74RuLfI930XaNqpa8V+LVJGVd+40pPF6\n",
       "p/XZRyi/K3MqUjV5EDgJuNNt2+dICeZeJANNw9XG8CdSjeaMZxtywzDe2q8S0rBv91mDs5C/4SpW\n",
       "LIeRqk/nuXxlUCBVkP9CMockpJ1H2xiOgWYMsW8n0utlBFJNcDmSOP+F3PENwfV7HojcWa9C/tmc\n",
       "9eiLkIbdF5DqprVInbyvBGwG8k+4Delv7/QfpPTyHnKnugxpVHSyc6d9NdIOMBtJsN5E/vHvLnWc\n",
       "N5GGWGfjqvO7+LoGvu6aC6zPfoB89+tw9XwBuVbvIlUq+UhCaUodw9/3esyK6Tck4fwQaQD25R5c\n",
       "VSq7kZ42lyPX3Hmu0udzLs+2XmuQtqhDeDZ0e/vsUKTktw9pu3jPbZ8CpH3kUuQ6r8HVRvSh9XMX\n",
       "0mgM8jdTFVevtA9xVeH5itu5Lt06f74V+06kYwNIT6eTkevxCWWNR35/XyF/d1PwbOtQUWYqkgAt\n",
       "87PPRCQhWoKrJ4hS/nyLZyN3LLkT3z2vlIoKoS4xTMPVbdCbS5A7z1ZI3fSkEMej4ke0PZjmy/FI\n",
       "43clpAvmA3g+P6JU1Al1xvA9Uvzz5TKkGgSki18G0u1NqfIE0gAcSVWRtpd9SPvJf3A996FUVPI1\n",
       "LEG4NKJsV77GeHbHU6q0rpEOIACb8OyZo1TUi4bG59JVArFyJ6iUUnEp0iWGrXj2625srfPQsmVL\n",
       "s369Pr+ilFIBWk8FnvGIdMYwA+nX/R7SX30PXqqR1q9fjzFakAAYM2YMY8aMiXQYUUGvhYteC5do\n",
       "vhZHj8KCBVBUVP6+gVqzJpennhrMoUNNOPXUyXz+eUOqVElqWZFjhTpjcD6oUxdpSxiNa2C2V5An\n",
       "TC9B+mofQEa5VEqpuPTTT9C7N7RrF9zjbt36LJs3P0mLFuPo2nUgU6YkUfkYUvdQZwzX2djnrhDH\n",
       "oJRSUaGoCM44A777LrjHnTevIyeckEfDht6GNwtcpKuSVICysrIiHULU0GvhotfCxf1arF4Nz4Vi\n",
       "iqcK2lqmBTU4unTpUv5OAYiVh4SMtjEopQL1+uvw8sswaFC5u4bNqafCBWGarTopKQkqkM5riUEp\n",
       "FddOOgn+/vdIR3HsHA4HY8eOJSMjg/vvvz+k59KMQSkVMW+/DXffXf5+FVVYCIMHh+744ZKbm8vg\n",
       "wYNp0qQJkydPLv8Dx0gzBqVUxGzdCtdfD48+GrpzpJc38HgUc5YSJk2axLhx4xg4cKCzeiikNGNQ\n",
       "Svn055+wdGnojr9mDdSuLS9V1n333cemTZvIywtejyM7tPFZKeXT8OHw6afQvHnoznHHHXD11aE7\n",
       "fiwrKCggNTW1wqUEbXxWSgVdcTHcfjsMHRrpSBJTWprdaa+DSzMGpSJk3jyYNi3SUfi3YEF8NN5G\n",
       "O4fDQUFBAXXq1Il0KIBWJSkVEcbA6afDVVdB06aRjsa/Xr2gUaPy91MV4+xx1LdvXx5++OGgHlur\n",
       "kpSKIV99BUlJMGaM/FSJx1uPo2ihGYNSETBunNTba6aQmNyfSwh3jyM7YuXPUquSVNzIy5MRNn/7\n",
       "DapWjXQ0KhLGjx9P3bp1Q/5cQkWrkjRjUAHbty/4o0MmksmToUsXGDEi0pGoeKdtDCpsPvoIRo+W\n",
       "4YNV4NLSpAuoUtFKMwYVMGOgRw947bVIR6JUdMvNzWXPnj107do10qEERDOGBPbUU7BsWeCfW7cO\n",
       "Tjst+PEoFS/cexy99NJLkQ4nYJoxJLA33oCbbqpYH/Wzzgp+PErFg2jvcWSHZgwJ7tJLoW3bSEeh\n",
       "VHx44YUXeOyxx8I6EmooaMYQxw4ehAYN4MAB79uTkiAlJbwxKRXPzjnnnJgtJbiLlexMu6tWwO7d\n",
       "0KIF7NzpfXtSEiQnhzcmpVT4aHdV5VVSElTW37JSKgCaZCilVACcPY4qVarE6NGjIx1OSFSKdABK\n",
       "KRUrcnNz6dixI4sWLeLWW2+NdDghoxmDUkqVw+FwMHr0aHr27MmQIUOYOXNmzDcw+6NVSUopVY6R\n",
       "I0eycuXKuOhxZEcgrdXVAQMUhigWfxK6V5IxUFQU+Od274bWreWnUqriDh06RPXq1WPuuYSK9kry\n",
       "V5VUCbgK+BDYCvwG/G69/wi4siInVIF75hkZnrl69cBeDRrAccdFOnqlYl+NGjViLlM4Fv4yhhzg\n",
       "TGAccALQADjeej8O6Ajo4MthsGcPPPYYHD0a+Gv16khHr1TscDgcbN++PdJhRJy/jOFiYCSwAM/q\n",
       "o0LgJ+Ahax8VInPnwttvV2ygO6VUYJw9jiZOnBjpUCLOX8bgzAzGA6eUs48KgWuugU8/hdRUuPDC\n",
       "SEejVHwq3eNo7NixkQ4p4uz0SloJTAaqAFOBd4G9oQxKCWPglVegTp1IR6JUfIqHkVBDIZDWlJOA\n",
       "wcAAYB4wBfg2BDF5kzC9klasgAcflExh9mz46y/IzIx0VErFp1dffZWqVavG9Eio/oR6zudk4FLg\n",
       "JqAx8AHQBTgIXBPoSSsgYTKGDz6AiRNlPuCaNeGiiyIdkVIqVoVyEL1nkUzhG2AssNBa/zSgfV5C\n",
       "oGFDmSdBKaUiwc6QGEuBdsBtuDIFp7ODHpFSSgVZbm4us2bNinQYMcNOxjAQKD3Vy1zr555yPtsL\n",
       "WAWsBUZ42V4XmA3kAcuRNgyllAoK9x5HB3zNWKXK8FeVVAOoiSTetd3WpwN2ZglOBl4AuiNPS/8M\n",
       "zEB6OTndBeQCD1rnWQ28BRy1F378+OknaXj++edIR6JUfNAeRxXnL2O4HbgXaAgscltfgCT45ekE\n",
       "rAM2WsvvAZfjmTFsA0633qcDu0jATAHgoYegWjVpX+jbN9LRKBXbJk+ezKhRo2J+7uVI8ZcxTLBe\n",
       "dwPPV+DYjYDNbstbKNsmMQVp1P4DSAP6V+A8cWPYMOjWLdJRKBX7unTpoqWEY+AvY+iGK9G+ysv2\n",
       "T8o5tp3+pQ8h7QtZQEvga6Shu6D0jmPGjCl5n5WVRVZWlo3DK6US0cknnxzpECIiJyeHnJycYz6O\n",
       "v4zhQiRjuBTviXx5GcNWoInbchOk1ODuXKQLLMB6ZATXNsAvpQ/mnjEopZSTMUariiylb5offfTR\n",
       "Ch3HX8bgnMz0b1Ss3v8XoBXQHCl1XANcV2qfVUjj9A9AfSRT2FCBc8WEgwfh8GHv244cCW8sSsU6\n",
       "59zLBQUFjB8/PtLhxBU7D7htQLqUvo+UIOw+gnwU6XU0B+mh9BrS8Hy7tf0V4AlgGrAE6To7HMi3\n",
       "efyY06QJFBeDt5ub5GSoWzf8MSkVi9x7HE2ePDnS4cQdO+WvFKAPcC3QAZiJZBLfhzCu0uJiSIzq\n",
       "1WVuherVIx2JUrHJWUqYNGmS9jiyIZRDYhxAMoL3gUxgIjKJT3KgJ4tXS5fCvHnl73c0ITviKhU8\n",
       "TzzxBIsWLdIeRyFmNyfJQtoIeiEPqr0PfByimLyJ6hLDbbfJTGmn+Jq1wlKrFjzxhPeqJKVU+RwO\n",
       "B1WqVNFSgk2hLDFsRLqUvg8MA/YHepJEcP31kkEopUKnatWqkQ4hIdjJGE4H9oU6EKWUcnLOvdy0\n",
       "adNIh5KQ/A2i5xz0bizy5LP7SydFVUqFhHPu5QkTJkQ6lITlr8Twq/VzEZ5dVJOw32VVKaVs8dbj\n",
       "SEWGv4xhpvXzIDJjm7uEHtNIKRVcOhJqdLEzH8ODNtcppVSFrF27liFDhjBz5kzNFKKAvxJDNnAJ\n",
       "MkrqRFxdntIAHcABmDZNHlhbuhTOOivS0SgVu/r310qIaOKvxPAH0r5w2PrpfM0AeoY+tOhWVAQ3\n",
       "3wybNsE558CFF0Y6IqWUCg47Dz5UIfIlhKh7wK2oCKpWlZ9KKXtyc3NZvXo11157baRDSQiheMDt\n",
       "Q6AfsNjLNoNr5rWY9vnn8NhjgX/OGKhs5ykQpZRHjyPthhr9/CVt91o/Lw1HIJGyfDm0bg133RX4\n",
       "Z2vVCn48SsUb7XEUe/xlDH9YP3cg7QxFyHwJbYAvQxxXWDVsCGeXnnRUKXXMXn/9dYYPH64jocYY\n",
       "O7+lxUAXZGTVH5BB9BzA9SGMq7SQtDEUFkLPnnDFFXDffUE/vFIJb8OGDVSvXl1LCRFS0TYGOx/I\n",
       "BdoDdwM1gP9DJtZpF+jJjkHQM4biYrjuOhkK+4MPZKIcpZSKJ6EcXRXgHKSEcIu1bOfBuKg2fDhs\n",
       "3w5z5mimoFQw6NzL8cNOAn8f8qTzp8AKoCXwbSiDCoePP4bJk3U2NaWOlcPhYPTo0dx6662RDkUF\n",
       "iZ0Sw3fWy2k9cE9owgkvHdpdqWOjcy/HJzsZQxtgKNDcbX8DdAtRTEqpKKdzL8c3OxnDh8Ak4FWk\n",
       "y6pSKsE9//zzOvdyHLOTxS8Czgx1IOUIeq+kFi3gm2/kp1IqMEePHiU5OVlLCVGuor2S7DQ+zwT+\n",
       "ATQAaru9lFIJqnLlypopxDE7VUmDkTaFoaXW6722UnHO4XDw+++/06pVq0iHosLITomhOZIJlH4p\n",
       "peKYc+7lZ599NtKhqDCzkzGkAA8DU6zlVkCfkEWklIooh8PBI488Qs+ePRk6dCgvvvhipENSYWan\n",
       "Kmka0gB9rrX8B/ARMCtUQSmlIiM3N5cbb7yRZs2aaY+jBGanxNASeBoZOA/gQOjCUUpF0vbt2xk2\n",
       "bBgzZszQTCGB2SkxFCKD5zm1tNYppeJMdnZ2pENQUcBOxjAGmA00Bt4BzkN6KsWMnTthyxbPdYWa\n",
       "tSmllFd2OyLXBTpb738CdoYmHJ+O6QG3vn1h0SLPGdeqVZORVTMyghCdUjFm8eLFLF68mL/97W+R\n",
       "DkWFUCgecGsOOJPNncBBoAcwCIip4eeOHIEJEyAvz/VasEAzBZV4nD2OevXqRY0aNcr/gEpI/jKG\n",
       "D4Ca1vszkDGTfrfevxTiuJRSQbZ48WLOOusscnNzycvL4/rrwzkJo4ol/toYquOa9/kG4DXgGSQz\n",
       "WRLiuJRSQfT2229z//3388wzz3DDDTfocBbKL38Zg/tfzkXIZD0AxaELRykVCl27dtXnEpRt/qqS\n",
       "vkWqjyYibQ3fWOsbYr+7ai9gFbAWGOFjnyxkXunlQI7N45aroADatpXRU7/+WhqblUpUDRs21ExB\n",
       "2eavPFkJuAY4Hmlv2Gqtbw8cB8wp59jJwGqgu/XZn4HrgJVu+2QAPwA9gS1I7ydvPZ4C7pW0bRuc\n",
       "fjosXAiVKkHTpqClZ5UIiouLqVQp5qdlV0FQ0V5J/qqSDPCul/W57ue19vOmE7AO2GgtvwdcjmfG\n",
       "MAD4GMkUIMjdYCtX1vkWVOJwOBw8/vjjrFmzhvfeey/S4agY5u+2IgcYBrT2sq0NUjX0nZdtTo2A\n",
       "zW7LW6x17lohczt8C/wCDPQfrlLKG2ePo8WLFzN+/PhIh6NinL+MoQewC3gR2AasQdoKtgEvAH8i\n",
       "1US+2Kn7qQJ0AC5BqpMeRjILpZQNDoeD0aNH06tXL4YOHcrMmTO1LUEdM39VSYXAVOuVjNT/g1T3\n",
       "2Jn7eSvQxG25Ca4qI6fN1vEOWa//Ae2QDMjDmDFjSt5nZWWRlZVlIwSl4tvUqVN17mVVIicnh5yc\n",
       "nGM+TiibYysjjc8XIc9DLKRs4/NJSOmjJ1ANWIA0eP9a6lgVanzu0EF+KhWviouLSUpK0ucSlFeh\n",
       "aHw+VkeBu5DeS8nIA3Irgdut7a8gXVlnA0uR5yOmUDZTUEr5oL2PVCjEym2GlhhUQnM4HKxdu5ZT\n",
       "Tjkl0qGoGBKKQfRAShTfViQgpVRw5OXl0alTJ+1tpMKmvIzhKFLFo+OQKhVmzh5HPXr04IEHHuDV\n",
       "V1+NdEgqQdhpYzgALAO+xjWtpwHuCVVQwfDKK9Da2xMYSsWApUuXMmjQIBo3bqw9jlTY2ckYPrFe\n",
       "zkp+f087R4WXX4a33oIffoh0JEpVzN69e3nggQcYOHCg9jhSYWf3L64ariegVwFHQhOOT7Ybn7/6\n",
       "Cm66Cf73P2jZMsRRKaVUFAtld9Us4A1kkh6ApsCN+B8OI2LmzYNbb9VMQSmlKspOJ+jxyPAYF1iv\n",
       "HsCzoQzqWGnXbhUr8vLymDBhQqTDUMqDnSTU+QSz0xpC+2CcUnHPvcdRnTp1Ih2OUh7sJPCLgFeB\n",
       "t5C6quuRkVCVUhWQl5fH4MGDtceRilp2Sgx3IkNZ3APcDayw1imlAvTxxx+XPJegI6GqaBUr/eBs\n",
       "9UrasQPOOw+eeAL69g1DVEoFaNeuXRQWFmqGoMKior2S4iZj2L8funaFXr3gX/8KU1RKKRXFEj5j\n",
       "uOIKqFcPJk/WuZ1VdCgqKiI5OTnSYagEFo6MoSZwMNATBEm5GcNxx8GyZVC/fpgiUsoHh8PB2LFj\n",
       "+eWXX/j8888jHY5KYKEaXRXgXGSOBGeX1TOAlwI9UThoSUFFmnMk1EWLFjFlypRIh6NUhdjJGCYA\n",
       "vZApOAHygAtDFpFSMaj0SKja40jFMrsPqm0qtXw02IEoFcs+/PBDnXtZxQ07GcMm4DzrfVXkeYaV\n",
       "vncPn4MH4ZtvwBgoLIx0NCqRDRgwgAEDBuhIqCou2MkY7gSeAxoBW4GvgH+EMii75syBO++Ejh2l\n",
       "m2qtWpGOSCUqzRBUPLGTMbQGBpRadx4Q8dkOiovlgbaPP450JCpROBwOli9fTocOHSIdilIhY6fx\n",
       "+QWb65SKazr3skoU/koM5yBdVesBD+DqC5uGvQxFqbjgfC5h0qRJjBs3joEDB0Y6JKVCyl/GUBXJ\n",
       "BJKtn077AB2JSCWEZcuWMXDgQB0JVSUUfxnDd9brdWBjOIJRKtoUFRXp3Msq4dhpfD4IjANOBmpY\n",
       "6wzQLVRBKRUtzjjjDM4444xIh6FUWNlpK3gbWAWcAIxBSg86UY9SSsUpOxlDHWQGNwdStXQTWlpQ\n",
       "cSYvL49/6XjtSgH2MgaH9XM70AfoAGSGLCKlwsh9jKNmzZpFOhylooKdNoaxQAYwBHgeSAfuD2VQ\n",
       "SoWDzr2slHd2MoaZ1s89QJb1vlNIolEqTD7//HNuuummkucStMeRUi7+MoZKwJVAS2A58AVwFvAE\n",
       "cBwyL4NSMemCCy7QUoJSPvjLGCYDLYCFwCjgFuAkYCTwWehDUyp00tLSSEtLK39HpRKQv4yhM3A6\n",
       "UAxURxqfWwK7whCXUkFz5MgRqlSpEukwlIoZ/nolHUEyBYDDwG9opqBiiLPHUffu3SlvznCllIu/\n",
       "EsNJwDK35ZZuywYpTSgVldx7HL377rvauKxUAPxlDG3DFoVSQeJtJFTNFJQKjL+MYWMQjt8LmICM\n",
       "0Poq8LSP/ToCPwL9gU+CcF6VoObMmaNzLyt1jOw8x1BRyciEPt2RKUF/BmZQdr7oZCTDmI1rzgel\n",
       "KqRPnz706dNHSwlKHYNQTrjTCViHlDyOAO8Bl3vZ727gI2BHCGNRCSIpKUkzBaWOkd2MoSbQJsBj\n",
       "NwI2uy1vsdaV3udyYJK1rF1HlC0Oh4P58+dHOgyl4pKdjOEyIBeYYy23R6qEymMnkZ8A/NPaNwmt\n",
       "SlI2OOdefvbZZ7UbqlIhYKeNYQxwNvCttZyLzM1Qnq1AE7flJkipwd2ZSBUTQF0gG6l2KpPxPPLI\n",
       "GJYtg6IiaNMmi5NOymLRIhtRqLihPY6U8i8nJ4ecnJxjPo6d/6oFSMaQi5QWAJZS/nMMlYHVwEXA\n",
       "H8jQGtdRtvHZaRoyYJ+3Xklm7VpD+/bQv7/nhosvhmuvLf9LqNj266+/MmDAABo3bszkyZO1x5FS\n",
       "Nlg3TgHfPdkpMawArrf2bQXcA9ip3D0K3IVUQSUDryGZwu3W9lcCDbZ+fXjttUA/peJB1apVde5l\n",
       "pcLEzn9YCjJwXg9reQ7wL2SYjHAxa9caevWCdevCeFallIphoSwxtAEesl5KKaXinJ1eSeOBVUgp\n",
       "4dTQhqMSXV5eHsOGDdPeRkpFkJ2MIQvoCuxE2gWWAQ+HMCaVgNznXj7ttNMiHY5SCc3uA27bgOeA\n",
       "O4AlwCMhi0glHOdzCc4xjgYNGqQNzEpFkJ2M4WTkWYblyNhH8yn7BLNSFTJ37lx69OjBAw88wMyZ\n",
       "M7UbqlJRwE7j81TkIbSeyENrSgVNly5ddCRUpaKMnYyhc8ijUAmrWrVqmikoFWX8ZQwfAv3wnMXN\n",
       "SWdwUwE7fPgw1atXj3QYSqly+MsY7rV+9qHsAxLal1DZ5hzj6PPPP+fnn3/WhmWlopy/xuc/rJ9/\n",
       "R+ZUcH/9PYQxqTji3uNoxowZmikoFQPs9Erq4WXdJcEORMUX9+cStMeRUrHFX1XSnUjJoCWe7Qxp\n",
       "wA+hDErFvh9//JHFixdrjyOlYpC/cn0tIBN4Chjhtm8BsCvEcZWmg+gppVSAQjGInkHaE/5B2cbm\n",
       "2kB+oCdTSikV/fy1Mbxr/Vzk46UUDoeDuXPnRjoMpVQQ+Ssx9LZ+Ng9DHCoG5eXlMXjwYFq0aEHX\n",
       "rl2pVMnu0FtKqWhm5z/5PCDVej8QGYa7WcgiUlGvdI+jTz75RDMFpeKInSExXgbaWa8HkCk6pwMX\n",
       "hjAuFaVWrVrFtddeS+PGjbXHkVJxys5t3lGgGLgCeBEZYTUtlEGp6JWens6QIUP0uQSl4pidEkMB\n",
       "Mq3nDcD5QDJQJZRBqejVsGFDBg4cGOkwlFIhZKfEcA1QCNwMbEfmYvh3KIPy5uWXw31GpZRKTHYy\n",
       "hm3A20AGMqDeYaSNIayqVIGHHgr3WRNXXl4ed9xxB8XFxZEORSkVZnYyhv7AAmQI7v7AQut9WD35\n",
       "JNx8c7jPmnjcexyde+65OuidUgnIThvDKKAj8Je1XA+Yi8zXoOKI87kE7XGkVGKzU2JIAna4Le+i\n",
       "AmNvqOg2f/58HQlVKQXYS+D/jTzD8I61/zXAUmB4COMqzRijcwOFUlFRETt27OD444+PdChKqSCp\n",
       "6CB6dj/6fsKxAAAaLElEQVRwFdDFev898GmgJzpGmjEopVSAQjG6amuktHAiUkIYBmypSHAquhw4\n",
       "cICUlJRIh6GUilL+2himArOAq4HFwMSwRKRCxtnjqFOnThQVFUU6HKVUlPJXYkgFpljvVwG5oQ9H\n",
       "hYp7j6Ovv/6a5OTkSIeklIpS/jKG6kAH630SUMNaTkIm7lkc2tBUMDgcDsaOHcukSZMYN24cAwcO\n",
       "1GcTlFJ++csYtgPP+FnuGpKIVFAtW7aMvLw8fS5BKWVbrNw6aq8kpZQKUEV7JensKkoppTxoxhAn\n",
       "HA4Hs2bNinQYSqk4oBlDHMjLy6NTp05MnjyZo0ePRjocpVSMs5MxVELmen7EWm4KdArgHL2Q7q5r\n",
       "gRFetl8PLEEeovsBOD2AYye00nMvf/bZZ1SubGdcRKWU8s1OKvISMrVnN+AxYL+17iwbn01GpgLt\n",
       "DmwFfgZmACvd9tkAXADsRTKRyUBne+EnrnXr1tG3b18dCVUpFXR2Moazgfa4HnDLx/7Unp2AdcBG\n",
       "a/k94HI8M4Yf3d4vABrbPHZCq1OnDsOHD+e6667T5xKUUkFlpyrJgdz5O9VDShB2NAI2uy1vsdb5\n",
       "cgvwhc1jJ7TMzEwGDBigmYJSKujslBieR0ZTPQ54AuiLTN5jRyAPH3RF5pU+z9vGMWPGlLzPysoi\n",
       "KysrgEMrpVT8y8nJIScn55iPY/d2sy1wkfV+Lp5VQf50BsYgbQcADyKljadL7Xc68Im13zovx0nY\n",
       "B9zy8vIYN24c06ZNo0oVuzV4SikV2gfcmgIHgJnW64C1zo5fgFZAc6AqMsnPDC/H/wS4Ae+ZQkJy\n",
       "73HUo0cP7W2klAobO6nNF7iqhKoDLYDVwCk2PnsUuAuYg7RTvIaUNm63tr+CdIPNBCZZ644QWHfY\n",
       "uKNzLyulIqkiLZcdgH8gDcXhkjBVSbm5ufTs2VNHQlVKHbNQT+1Z2nLg1Ap+tiISJmMwxrBz507q\n",
       "1asX6VCUUjEuFFN7Og1xe18JKTFsDfREyp6kpCTNFJRSEWWn8TnV7VUVme7z8lAGlSj27t0b6RCU\n",
       "UqqM8koMyUA6nqUGdYycs6q99dZbrFy5kqpVq0Y6JKWUKuGvxFAZKEIeONMW0CDJzc2lY8eOLFq0\n",
       "iO+//14zBaVU1PFXYliItCfkAZ8BHwIHrW0GefZA2eQ+9/IzzzzDDTfcoD2OlFJRyV/G4Ey1qgO7\n",
       "kNFV3WnGEID169ezfPlyfS5BKRX1/N2ybgHG+9nnmeCH41PCdFdVSqlgCUV31WQgraIBKaWUik3+\n",
       "cpJcZB6GaBAzJQaHw8Fnn31Gv379Ih2KUirBhXIQPWWTs8fR9OnTKSwsjHQ4SilVIf5ykjpIo3M0\n",
       "iOoSg/Y4Cp/atWuze/fuSIehVFTJzMwkPz+/zPpQtDFES6YQ1X777TeuuOIKmjZtqj2OwmD37t1E\n",
       "802CUpEQ7BvRWLmtjdoSw4EDB5g1axb9+/fXUkIYJCUlacagVCm+/i/CPbpquEVtxqDCSzMGpcoK\n",
       "dsagjc9KKaU8aMZgU25uLldddRWHDx+OdChKKRVSmjGUwzn3cs+ePbnyyiupVq1apENSKib8+uuv\n",
       "dOzYMdJhxIW+ffsye/bssJ1PMwY/nM8lLF68mLy8PJ1qU/nVvHlzatasSVpaGscffzwDBw5k3759\n",
       "HvvMnz+fbt26kZ6eTkZGBpdddhkrV6702Gffvn3cd999NGvWjLS0NE488UTuv/9+du2KrY6CDz/8\n",
       "MMOGDYt0GMdk48aNdO3alZSUFNq2bcvcuXN97nv06FHuvvtuGjRoQJ06dbjsssv4448/SravWLGC\n",
       "rKwsMjIyaNKkCY8//rjH5ydPnsyJJ55IrVq16NixIz/88EPJthEjRjBq1Kjgf8EYZ8Jt1apVpl69\n",
       "emb69OmmuLg47OdX3kXib8Gu5s2bm7lz5xpjjNm+fbtp166dGTZsWMn2+fPnm9TUVDNx4kSzf/9+\n",
       "k5+fb0aNGmUyMzPNhg0bjDHGFBYWmrPOOsv06NHDrFy50hhjzF9//WUef/xx88UXX4Qs9iNHjgT1\n",
       "eH/88YepXbu2KSwsrNDnjx49GtR4Kqpz585myJAh5vDhw+bjjz82GRkZZseOHV73fe6550y7du3M\n",
       "X3/9ZQ4fPmwGDRpkrrrqqpLt7du3N6NGjTLFxcVm/fr1pkGDBmbGjBnGGGNyc3NNamqqWbx4sTHG\n",
       "mEmTJpl69ep5pD2tWrUyv/zyi9dz+/q/QEbCjlsV+qUeq/z8/IicV/kWqb8FO9wzBmOMGTZsmLnk\n",
       "kktKlrt06WL+8Y9/lPlcdna2GTRokDHGmClTppj69eubAwcO2D7v8uXLTffu3U3t2rVN/fr1zZNP\n",
       "PmmMMebGG280o0aNKtnv22+/NY0bNy5ZbtasmXn66afNaaedZqpVq2aefvpp07dvX49j33PPPeae\n",
       "e+4xxhizZ88ec/PNN5sGDRqYRo0amVGjRpmioiKvMb3xxhvm4osv9lj35JNPmpYtW5q0tDRz8skn\n",
       "m08//bRk27Rp08y5555r7r//flOnTh3z8MMPm8LCQjNkyBDTtGlTU79+fXPHHXeYQ4cOGWOM2b17\n",
       "t+ndu7epV6+eyczMNH369DFbtmyxfc3sWL16talWrZrZv39/yboLLrjAvPzyy173v+2228zw4cNL\n",
       "lmfNmmXatGlTslytWrWSzN4YY/r162eeeuopY4wxb7/9tunUqVPJtv3795ukpCSzffv2knW33nqr\n",
       "efTRR72e29f/BRXMGLQqyY/MzMxIh6BijLG6DG7ZsoXZs2dz9tlnA3Dw4EF+/PFHr2No9e/fn6+/\n",
       "/hqA//73v2RnZ1OzZk1b5ysoKKB79+5ccsklbNu2jXXr1nHRRRcB0lWxvKrP9957jy+//JK9e/dy\n",
       "7bXX8sUXX7B//34AioqK+PDDD7n++usBGDx4MFWrVmX9+vXk5uby1Vdf8eqrr3o97rJly2jTpo3H\n",
       "uhNPPJF58+axb98+Ro8ezQ033MCff/5Zsn3hwoW0bNmSv/76i4ceeogRI0awbt06lixZwrp169i6\n",
       "dSuPPfYYAMXFxdxyyy1s2rSJTZs2UaNGDe666y6f37NPnz5kZmZ6fV122WVeP7NixQpOOOEEUlJS\n",
       "Sta1a9eOFStWeN2/R48efPnll2zbto2DBw/y9ttvc8kll3hsf+ONNzh69CirVq3ixx9/pHv37gCc\n",
       "f/75/PbbbyxcuJCioiKmTp1K+/btqV+/fsnn27Zty5IlS3x+x0TkJ18/djt37gzp8VXwlPe3AMF5\n",
       "VUSzZs1MamqqSUtLM0lJSeaKK64ouaPevHmzSUpKMqtXry7zuS+//NJUqVLFGGNM9+7dzYMPPmj7\n",
       "nO+8847p0KGD122DBw/2W2Jo3ry5mTZtmsdnunTpYqZPn26MMearr74yLVu2NMZI1Vi1atVK7tid\n",
       "5+7atavXc996663mn//8p9/YzzjjDPPZZ58ZY6TE0LRp05JtxcXFJiUlxaxfv75k3fz5802LFi28\n",
       "His3N9dkZmb6PV+gpk+fbjp37uyxbuTIkWbw4ME+PzNo0CCTlJRkKleubDp06OBR67Bu3TrTokUL\n",
       "U7lyZZOUlGTGjBnj8dlXXnnFVK5c2VSuXNnUq1fP/Pzzzx7bJ0+ebLp16+b1vL7+L9ASQ+CcPY7a\n",
       "t2/PwYMHy/+AinrByhoqIikpic8++4x9+/aRk5PDN998wy+//AJI6bNSpUps27atzOe2bdtGvXr1\n",
       "AKhbt65Hg2V5Nm/ezAknnFCxgIEmTZp4LA8YMIB3330XgHfeeaektPD7779z5MgRGjRoUHKnfccd\n",
       "d7Bjxw6vx83MzKSgoMBj3fTp02nfvn3J55cvX+7RoO4ey44dOzh48CBnnnlmyf7Z2dns3LkTkBLY\n",
       "7bffTvPmzalVqxYXXnghe/fuDerDj6mpqWU6D+zZs4f09HSv+w8dOpSCggLy8/M5cOAAV155JdnZ\n",
       "2SXxduvWjccee4zCwkI2b97M7NmzmTRpEgAzZszgmWeeYeXKlRw5coQ333yTPn36ePy9FBQUkJGR\n",
       "EbTv50/CZgzuPY5++ukn20V3pey44IILuPvuuxkxYgQAKSkpnHPOOXzwwQdl9v3ggw9Kqn+6d+/O\n",
       "nDlzbN+oNG3alA0bNnjdlpKS4nGc7du3l9mndFVT3759ycnJYevWrfznP/9hwIABgCTa1apVY9eu\n",
       "XezevZvdu3ezd+9eli1b5vXcp59+OmvWrClZ/v3337ntttt48cUXyc/PZ/fu3Zx66qkeCbl7LHXr\n",
       "1qVGjRr8+uuvJefbs2dPSUL9zDPPsGbNGhYuXMjevXv57rvvMMb4zBiys7NJS0vz+urdu7fXz5xy\n",
       "yils2LChpGoNYMmSJZxyyile9589ezY33XQTGRkZVK1albvuuouFCxeSn5/PihUrKCgo4IYbbqBS\n",
       "pUo0atSIa665hi+++AKAOXPm0Lt3b0488UQAevbsSYMGDfjxxx9Ljr9y5UrOOOMMr+dOVD6LboEq\n",
       "LCw0jzzyiPY4ilHB/FsIttKNzzt27DA1a9Y0P/30kzHGmHnz5pmUlBQzceJEs2/fPpOfn29Gjhxp\n",
       "MjMzzbp164wx8vfZsWNH06tXL7Nq1SpTVFRkdu7cacaOHeu1V1JBQYFp0KCBmTBhgjl8+LDZt2+f\n",
       "WbBggTFGGrJPOukkk5+fb7Zt22bOPvvsMlVJ7vE6ZWdnm+7du5eporr88svNvffea/bt22eKiorM\n",
       "unXrzHfffef1Wmzfvt3UqVOnpFfSihUrTPXq1c3q1avN0aNHzdSpU03lypXNa6+9ZoyRqqQuXbp4\n",
       "HOPee+81/fv3N3/99ZcxxpgtW7aYOXPmGGOMGT58uMnOzjaHDx82u3btMldccYVJSkry2RheUZ07\n",
       "dzZDhw41hw4dKumV5Kvq+brrrjNXX3212bt3r3E4HGbs2LEl1zs/P9+kpKSYd955xxQVFZlt27aZ\n",
       "zp07m5EjRxpjpBqpdevWZsOGDaa4uNh89dVXpmbNmh5Vj61bty5TveTk6/8C7ZVkz8aNG03//v3N\n",
       "1q1bg3ZMFT7B/FsINm8J7Z133mmuvPLKkuV58+aZrKwsk5qaatLT002fPn3MihUrPD6zd+9ec999\n",
       "95kmTZqY1NRU07JlSzNkyBCfveSWL19uLrroIpOZmWmOP/548/TTTxtjjDl8+LC55pprTHp6umnX\n",
       "rp159tlnTZMmTfzGa4wxb775pklKSjLjxo0rE9edd95pGjdubGrVqmXat29v3n//fZ/Xo1+/fh7b\n",
       "R44caWrXrm3q1q1rHnjgAZOVlVWSMbz++uvm/PPP9/j84cOHzUMPPWROOOEEk56ebtq2bWuef/55\n",
       "Y4x0h3VexzZt2phXXnnFVKpUKegZw8aNG01WVpapUaOGOemkkzyu1//+9z+Tmppasrx9+3bTr18/\n",
       "U7duXZORkWHOP/98j4T8iy++MO3btzfp6enm+OOPN7fddltJm01RUZEZNmyYady4cUmvrbfeeqvk\n",
       "swsXLjRnnnmmzzh9/V9QwYwhVp7Wsr6jSnQ6iF7sWLlyJTfeeCMLFy6MdCgxr2/fvvztb3+jV69e\n",
       "Xrfr6KoqoWnGoFRZOrqqTQ6HgzfeeEMTEaWUClBcZgzOHkcfffSRdkNVSqkAxVXG4D4S6tChQ5kx\n",
       "Y4bHU4tKKaXK52/O55iyZcsWevfurXMvK6XUMYqbxmeHw8GsWbO48sordWjsOKaNz0qVpb2SVEKr\n",
       "Xbs2u3fvjnQYSkWVzMxM8vPzy6yP1oyhFzABSAZeBZ72ss9EIBs4CAwGcr3soxmDUkoFKBq7qyYD\n",
       "LyCZw8nAdUDbUvtcApwItAJuAyaVd9Dc3Fyys7PLDG6VKHJyciIdQtTQa+Gi18JFr8WxC2XG0AlY\n",
       "B2wEjgDvAZeX2ucy4A3r/QIgA6iPF+49jgYMGEBaWlpIgo52+kfvotfCRa+Fi16LYxfKXkmNgM1u\n",
       "y1uAs23s0xj4s9R+dOzYUXscKaVUGIQyY7DbKFC6/svr54YMGcLAgQO1x5FSSoVYKFPZzsAYpI0B\n",
       "4EGgGM8G6JeBHKSaCWAVcCFlSwzrgJYhilMppeLVeqQdN2pURoJqDlQF8vDe+PyF9b4z8FO4glNK\n",
       "KRUZ2cBq5I7/QWvd7dbL6QVr+xKgQ1ijU0oppZRSSsWWXkg7w1pghI99JlrblwDtwxRXJJR3La5H\n",
       "rsFS4Afg9PCFFnZ2/i4AOgJHgavCEVQE2LkOWchDosuR9rt4Vd61qAvMRqqwlyMPz8arqUi7rPcJ\n",
       "uEXMppvJSJVSc6AK5bdJnE38tknYuRbnALWs971I7Gvh3O8bYBZwdbiCCyM71yEDWIF0+QZJHOOR\n",
       "nWsxBnjSel8X2EUcDRpayvlIYu8rYwg43YymYbeD+kBcjLNzLX4E9lrvF+BKDOKNnWsBcDfwEbAj\n",
       "bJGFl53rMAD4GHkeCGBnuIILMzvXYhuQbr1PRzKGo2GKL9y+B/wNIBZwuhlNGYO3h90a2dgnHhNE\n",
       "O9fC3S247gjijd2/i8txDakSjwNr2bkOrYDawLfAL8DA8IQWdnauxRTgFOAPpPrk3vCEFpUCTjej\n",
       "qWgV1AfiYlwg36krcDNwXohiiTQ712IC8E9r3yRiZ9TgQNi5DlWQnn0XATWRUuVPSN1yPLFzLR5C\n",
       "qpiykGegvgbaAQWhCyuqBZRuRlPGsBVo4rbcBFeR2Nc+ja118cbOtQBpcJ6CtDHE61jUdq7Fmbge\n",
       "kqyLdJM+AswIeXThY+c6bEaqjw5Zr/8hiWG8ZQx2rsW5wFjr/XrgN6ANUpJKNDGdbuoDcS52rkVT\n",
       "pJ61c1gjCz8718LdNOKzV5Kd63AS8F+kcbYm0hh5cvhCDBs712I8MNp6Xx/JOGqHKb5IaI69xueY\n",
       "TDf1gTiX8q7Fq0iDWq71WhjuAMPIzt+FU7xmDGDvOgxFeiYtA+4Ja3ThVd61qAvMRNKJZUjDfLx6\n",
       "F2lLcSClxptJ3HRTKaWUUkoppZRSSimllFJKKaWUUkoppZRSSikVPYpwPfOQizwg58v+IJzvdWCD\n",
       "da5FVOxBvCnIQ1sgQxy4+6HCkXlyXpelwCdAajn7t0P60IfLf4E0672d4ZX96QMsRh4IWwHcdszR\n",
       "eXoUGY4DZMTPFdb5GgIfWuvtXr97iN+xnpSKGoGMEROM8WTcHzS7GHmw5liEaowb9+O+DgwpZ//B\n",
       "wPMhiMPb8DTdgBfdlssbXtmfKsgwCA3dlltX4Dh2vYzMGVLaYOxdvzTi+6FNpaJC6YQ1BbkbXYTc\n",
       "LV/mZd8GyHg7uUhi1MVa3wOYb332A+tYpU3DNS9CdeCA9f4B61jLcI14mQJ8jtzJLgP6WetzkHGQ\n",
       "nkKGTM4F3rS2OUs17yGP/Du9jmRIlYB/I4nLEnzfHbtfl9uBl6z3nazvuBgpnbRGhl/YBPxlxdLP\n",
       "in0qMpzxYjyvo7t/W99tKdDfWpeFDJn8GfIUb2mvItfaXXMqljHURkob1b1sex1JyH+24uhtrU/G\n",
       "9zUcgXyXPOAJt+NcjYz2uwspMb4JNLNiroLr+i1GrsMaXPNGVEKezq1jLX+JjJCqlAoRZ8Kai4zZ\n",
       "n4yriqIungOtORPLIbiqcCoh1Sx1ge+AGtb6EcDDXs7nnjH0Q0b67IAkJjWQBHU5cIa132S3zzrH\n",
       "0f8W1+P7pTM25/IVSIIEroS7GpKIjbTWV0MSveZe4nQeJxm5Ln+3ltOsdQDdkbkeAG5EZsNyegLX\n",
       "nXEGkrDWLHWOq4GvkBEujwN+B45HMob9SMLpzUrKju/TnIpXJU1BMod3kCEinCNuTsM1ls6JyLAK\n",
       "/q5hNpJZOjOZDLfjXOXlvXvMpa/fI7huEHrgqnICqZq6M5AvqIIjmkZXVaF1CM8p/aogM1ydDxQj\n",
       "VQzHIXdzTguRu+EqwH+Qu8YsZGC2+dY+Vd3eu0tC7jZHWce8BalS+sSKBev9+cgUjOOQksEsYF4A\n",
       "32s28JwVRzaSaRUiicxpQF9rv3Qk0dtY6vM1kMyykbXtZWt9BjDd+ozB9b9SeljvHsClyBhFIAlo\n",
       "EzxLAOchibFBrsV3yDSk+5Br/LuP79YQyPf1xSvgVuRadbfivRi4ydr2gfVzHXKnfxLer2ErpB1h\n",
       "KnDYWr/Hx/m8DX9e+vpNRUpMzyFj/Exz2/YHcEL5X0sFm2YMiet65O6/A9IA+xtlqxm+RxLuPshd\n",
       "+XhkeO+vKX9QMoMkPp+4reuOZ6KQZO23Fsm0egOPA3OBf9n8HoeRKqeeSNXEu27b7rJi9ceZYdYA\n",
       "5iAT/nxqnX8ucCVyR5/j5xhXUf7Q1r7Gwz9QesdjkIxrWOnPkOktS1tuvd5Efuc3ednHPT5v17An\n",
       "wZvzYgtSiumGZJbXuW1z/n2oMIumGdxUeKUjd69FyGQ/3qozmiJTZb5qvdojQ/aeh0x+AlIl1MrH\n",
       "OUonHt8jVT/OqqQrrHUNkAT+baTk4G2y8iP4vpF5H7nbdJY+QBL5v7t9pjVlq3jcHUJ6woy14k5H\n",
       "7ljBM/Hch6sKznke91FMvcX+PXAN8v9WD7gAKSmUl7j+gau+3Y4i6/ztKZsppCClPfc4N1rvk5Dq\n",
       "viTk93oCsArf1/Br5Jo4qxMzA4ix9PUD+dt6Cym1uGcEDShbwlNKBdG+Ust1kCqgpUhxfgWuLqzO\n",
       "fW9E6oYXI9UfzsyjK64GySVIiaI0X8Nf34+r8dmZoPawjuMcPtzZruDexvAU8Cuuxmf371MZaex8\n",
       "zW1dEpLIL7XONRdX24W70tdlBpKId0aqgxYjpYcN1vZMK0Zn43N1pPppKXIn7mtyoP/D1fjsbFy/\n",
       "0M/+IG0CPd2WncMrFyLtAL7u9r1JRRr4V1mxf4/r2k5DpkV1Nj47G/O9XUNnoj4C+ZvJRUp5zuP4\n",
       "amNYar13v37ORvgqyPzlpXtJaeOzUkqVkoVrHutQivQcFmchNx7u0pGMSkWAViUpFb1ykGq60lUv\n",
       "8eSfSI+vB0utH4w0SCullFJKKaWUUkoppZRSSimllFJKKaWUUkoppeLL/wObym+6KoomMgAAAABJ\n",
       "RU5ErkJggg==\n"
      ],
      "text/plain": [
       "<matplotlib.figure.Figure at 0x113aa7c90>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_roc_curve(y_test, gs.predict_proba(X_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'clf__max_depth': 3, 'clf__max_features': 0.5, 'imp__strategy': 'median'}"
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gs.best_params_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From this search we can conclude that the imputation by the 'mean' strategy is generally a slightly better imputation strategy when training a GBRT model on this data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Further integrating sklearn and pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Helper tool for better sklearn / pandas integration: https://github.com/paulgb/sklearn-pandas by making it possible to embed the feature construction from the raw dataframe directly inside a pipeline."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Credits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thanks to:\n",
    "\n",
    "- Kaggle for setting up the Titanic challenge.\n",
    "\n",
    "- This blog post by Philippe Adjiman for inspiration:\n",
    "\n",
    "http://www.philippeadjiman.com/blog/2013/09/12/a-data-science-exploration-from-the-titanic-in-r/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}