LMDSF2018 [French] top 10%.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Le Meilleur Data Scientist de France 2018\n",
    "\n",
    "\n",
    "## Préambule \n",
    "\n",
    "Label Emmaüs propose à la vente en ligne des objets rénovés ou créés par le mouvement Emmaüs. Le but de ce challenge est d’estimer le délai de vente de chaque objet. \n",
    "\n",
    "## Description de la compétition \n",
    "\n",
    "Le catalogue d'objets du Label Emmaüs est en croissance régulière. L'ajout d'un objet au catalogue depuis la réception jusqu’à sa désignation (images, descriptif, entrepôt) n'est pas automatisé et prend de plus en plus de temps. \n",
    " \n",
    "La détermination d’un prix et la rédaction d’une description ne sont pas toujours simples. Il faut un peu de temps et d’expérience pour traiter rapidement un objet. L’ajout d’un produit prend aujourd’hui 40 minutes jusqu’à la mise en ligne. Beaucoup d’objets restent longtemps sur le site avant de trouver acquéreur. Et l’augmentation du catalogue nécessite de plus en plus d’espace de stockage, qui n’est pas toujours disponible. \n",
    " \n",
    "Certains magasins vendent mieux que d'autres. Ceux qui vendent moins bien créent moins d'annonces : leur espace de stockage tend à saturer et l'activité n'est pas perçue comme assez performante pour investir. A la question \"pourquoi mes produits ne se vendent pas\", Label Emmaüs ne sait pas répondre (sauf par du ressenti métier). \n",
    " \n",
    "L'objectif est donc de prédire la durée entre la mise en ligne et la vente d’un objet. \n",
    " \n",
    "Les données contiennent environ 10 000 objets (avec la description, le prix, la catégorie, etc.). Le délai de vente n’est connu que pour un sous-ensemble des produits (la base d’apprentissage). \n",
    " \n",
    "Il faudra prédire la durée pour les produits de la base de tests.\n",
    "\n",
    "La cible est catégorisée en 3 modalités: \n",
    " \n",
    "- 0 : entre 0 et 10 jours \n",
    "- 1 : entre 10 jours et 60 jours \n",
    "- 2 : plus de 60 jours\n",
    " \n",
    "Vous avez 2 heures pour résoudre le challenge. Les soumissions réalisées avant et après la fin du concours ne seront pas prises en compte.\n",
    "\n",
    "## Format de soumission\n",
    "\n",
    "Le fichier soumis doit contenir 4 colonnes : \n",
    "\n",
    "- colonne 1 : id produit \n",
    "- colonne 2 : probabilité que le produit appartient à la catégorie delai_de_vente = 0 (entre 0 et 10 jours) \n",
    "- colonne 3 : probabilité que le produit appartient à la catégorie delai_de_vente = 1 (entre 11 et 60 jours) \n",
    "- colonne 4 : probabilité que le produit appartient à la catégorie delai_de_vente = 2 (plus de 60 jours) \n",
    "\n",
    " \n",
    "Le séparateur de colonnes est la “,” (virgule). \n",
    "Le séparateur décimal est le “.” (point). \n",
    "\n",
    "## Métrique d'évaluation\n",
    "\n",
    "Log Loss\n",
    "\n",
    "http://wiki.fast.ai/index.php/Log_Loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# import modules and data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline \n",
    "pd.set_option('display.max_columns', 500)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Skipping line 2168: expected 31 fields, saw 33\n",
      "Skipping line 4822: expected 31 fields, saw 37\n",
      "Skipping line 4859: expected 31 fields, saw 37\n",
      "Skipping line 7342: expected 31 fields, saw 37\n",
      "\n"
     ]
    }
   ],
   "source": [
    "X_train = pd.read_csv(\"X_train.csv\", index_col=0, error_bad_lines=False)\n",
    "X_test = pd.read_csv(\"X_test.csv\", index_col=0, error_bad_lines=False)\n",
    "y_train = pd.read_csv(\"y_train.csv\", index_col=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_index = X_train.index\n",
    "X_test_index = X_test.index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# exploration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('Dimension X_train:', (8880, 30))\n",
      "('Dimension X_test:', (2960, 30))\n"
     ]
    }
   ],
   "source": [
    "print(\"Dimension X_train:\", X_train.shape)\n",
    "print(\"Dimension X_test:\", X_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>nb_images</th>\n",
       "      <th>longueur_image</th>\n",
       "      <th>largeur_image</th>\n",
       "      <th>url_image</th>\n",
       "      <th>description_produit</th>\n",
       "      <th>taille</th>\n",
       "      <th>matiere</th>\n",
       "      <th>age</th>\n",
       "      <th>garantie</th>\n",
       "      <th>annee</th>\n",
       "      <th>couleur</th>\n",
       "      <th>largeur_produit</th>\n",
       "      <th>wifi</th>\n",
       "      <th>etat</th>\n",
       "      <th>longueur_produit</th>\n",
       "      <th>pointure</th>\n",
       "      <th>vintage</th>\n",
       "      <th>marque</th>\n",
       "      <th>auteur</th>\n",
       "      <th>editions</th>\n",
       "      <th>hauteur_produit</th>\n",
       "      <th>poids</th>\n",
       "      <th>prix</th>\n",
       "      <th>categorie</th>\n",
       "      <th>sous_categorie_1</th>\n",
       "      <th>sous_categorie_2</th>\n",
       "      <th>sous_categorie_3</th>\n",
       "      <th>sous_categorie_4</th>\n",
       "      <th>nom_produit</th>\n",
       "      <th>nom_magasin</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3</td>\n",
       "      <td>3458.0</td>\n",
       "      <td>2552.0</td>\n",
       "      <td>https://d1kvfoyrif6wzg.cloudfront.net/assets/i...</td>\n",
       "      <td>Superbe petit top bustier avec explosion de co...</td>\n",
       "      <td>44</td>\n",
       "      <td>100 % polyester</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Multicolore</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>bon état</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>200.0</td>\n",
       "      <td>4.5</td>\n",
       "      <td>mode</td>\n",
       "      <td>tops, t-shirts, débardeurs femme</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Top bustier multicolore</td>\n",
       "      <td>Emmaüs 88 Neufchateau</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>2486.0</td>\n",
       "      <td>2254.0</td>\n",
       "      <td>https://d1kvfoyrif6wzg.cloudfront.net/assets/i...</td>\n",
       "      <td>Radio ITT Océnic Flirt, année 70\\nPour déco</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Plastique</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Jaune</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>en l'état</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>True</td>\n",
       "      <td>ITT Océanic</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1000.0</td>\n",
       "      <td>15.0</td>\n",
       "      <td>mobilier - deco</td>\n",
       "      <td>bibelots et objets déco</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Radio ITT Océanic</td>\n",
       "      <td>Communauté Emmaüs Thouars (magasin Parthenay)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1536.0</td>\n",
       "      <td>1536.0</td>\n",
       "      <td>https://d1kvfoyrif6wzg.cloudfront.net/assets/i...</td>\n",
       "      <td>Veste boléro à manches courtes NÛMPH. Gris chi...</td>\n",
       "      <td>40</td>\n",
       "      <td>Polyester, coton, laine</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Gris</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>neuf</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>False</td>\n",
       "      <td>Nûmph</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>360.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>label selection</td>\n",
       "      <td>mode</td>\n",
       "      <td>mode femme</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Label Emmaüs Chambéry</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    nb_images  longueur_image  largeur_image  \\\n",
       "id                                             \n",
       "0           3          3458.0         2552.0   \n",
       "1           2          2486.0         2254.0   \n",
       "2           3          1536.0         1536.0   \n",
       "\n",
       "                                            url_image  \\\n",
       "id                                                      \n",
       "0   https://d1kvfoyrif6wzg.cloudfront.net/assets/i...   \n",
       "1   https://d1kvfoyrif6wzg.cloudfront.net/assets/i...   \n",
       "2   https://d1kvfoyrif6wzg.cloudfront.net/assets/i...   \n",
       "\n",
       "                                  description_produit taille  \\\n",
       "id                                                             \n",
       "0   Superbe petit top bustier avec explosion de co...     44   \n",
       "1         Radio ITT Océnic Flirt, année 70\\nPour déco    NaN   \n",
       "2   Veste boléro à manches courtes NÛMPH. Gris chi...     40   \n",
       "\n",
       "                    matiere  age garantie  annee      couleur  \\\n",
       "id                                                              \n",
       "0          100 % polyester   NaN      NaN    NaN  Multicolore   \n",
       "1                 Plastique  NaN      NaN    NaN        Jaune   \n",
       "2   Polyester, coton, laine  NaN      NaN    NaN         Gris   \n",
       "\n",
       "    largeur_produit wifi       etat  longueur_produit  pointure vintage  \\\n",
       "id                                                                        \n",
       "0               NaN  NaN   bon état               NaN       NaN   False   \n",
       "1               NaN  NaN  en l'état               NaN       NaN    True   \n",
       "2               NaN  NaN       neuf               NaN       NaN   False   \n",
       "\n",
       "         marque auteur editions  hauteur_produit   poids  prix  \\\n",
       "id                                                               \n",
       "0           NaN    NaN      NaN              NaN   200.0   4.5   \n",
       "1   ITT Océanic    NaN      NaN              NaN  1000.0  15.0   \n",
       "2         Nûmph    NaN      NaN              NaN   360.0  16.0   \n",
       "\n",
       "          categorie                  sous_categorie_1 sous_categorie_2  \\\n",
       "id                                                                       \n",
       "0              mode  tops, t-shirts, débardeurs femme              NaN   \n",
       "1   mobilier - deco           bibelots et objets déco              NaN   \n",
       "2   label selection                              mode       mode femme   \n",
       "\n",
       "   sous_categorie_3 sous_categorie_4               nom_produit  \\\n",
       "id                                                               \n",
       "0               NaN              NaN  Top bustier multicolore    \n",
       "1               NaN              NaN         Radio ITT Océanic   \n",
       "2               NaN              NaN                       NaN   \n",
       "\n",
       "                                      nom_magasin  \n",
       "id                                                 \n",
       "0                           Emmaüs 88 Neufchateau  \n",
       "1   Communauté Emmaüs Thouars (magasin Parthenay)  \n",
       "2                           Label Emmaüs Chambéry  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "      <th>unique</th>\n",
       "      <th>top</th>\n",
       "      <th>freq</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>min</th>\n",
       "      <th>25%</th>\n",
       "      <th>50%</th>\n",
       "      <th>75%</th>\n",
       "      <th>max</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>nb_images</th>\n",
       "      <td>8880</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.63345</td>\n",
       "      <td>2.04857</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>5</td>\n",
       "      <td>29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>longueur_image</th>\n",
       "      <td>8823</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1807.82</td>\n",
       "      <td>1025.25</td>\n",
       "      <td>58</td>\n",
       "      <td>1000</td>\n",
       "      <td>1536</td>\n",
       "      <td>2448</td>\n",
       "      <td>5472</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>largeur_image</th>\n",
       "      <td>8823</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1801.77</td>\n",
       "      <td>1101.21</td>\n",
       "      <td>64</td>\n",
       "      <td>970.5</td>\n",
       "      <td>1536</td>\n",
       "      <td>2448</td>\n",
       "      <td>5472</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>url_image</th>\n",
       "      <td>8823</td>\n",
       "      <td>8775</td>\n",
       "      <td>https://d1kvfoyrif6wzg.cloudfront.net/assets/i...</td>\n",
       "      <td>4</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>description_produit</th>\n",
       "      <td>8880</td>\n",
       "      <td>8836</td>\n",
       "      <td>Relié, 48 pages, couverture usagée</td>\n",
       "      <td>6</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>taille</th>\n",
       "      <td>2414</td>\n",
       "      <td>33</td>\n",
       "      <td>38</td>\n",
       "      <td>402</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>matiere</th>\n",
       "      <td>3947</td>\n",
       "      <td>1722</td>\n",
       "      <td>Coton</td>\n",
       "      <td>144</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>age</th>\n",
       "      <td>120</td>\n",
       "      <td>18</td>\n",
       "      <td>4a</td>\n",
       "      <td>14</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>garantie</th>\n",
       "      <td>101</td>\n",
       "      <td>2</td>\n",
       "      <td>6 mois</td>\n",
       "      <td>100</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>annee</th>\n",
       "      <td>1497</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>14810.1</td>\n",
       "      <td>496237</td>\n",
       "      <td>0</td>\n",
       "      <td>1979</td>\n",
       "      <td>1998</td>\n",
       "      <td>2007</td>\n",
       "      <td>1.92019e+07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>couleur</th>\n",
       "      <td>5321</td>\n",
       "      <td>18</td>\n",
       "      <td>Multicolore</td>\n",
       "      <td>1063</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>largeur_produit</th>\n",
       "      <td>107</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>42.0589</td>\n",
       "      <td>20.5453</td>\n",
       "      <td>10</td>\n",
       "      <td>30</td>\n",
       "      <td>40</td>\n",
       "      <td>46.5</td>\n",
       "      <td>140</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>wifi</th>\n",
       "      <td>83</td>\n",
       "      <td>2</td>\n",
       "      <td>True</td>\n",
       "      <td>62</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>etat</th>\n",
       "      <td>8856</td>\n",
       "      <td>5</td>\n",
       "      <td>bon état</td>\n",
       "      <td>4537</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>longueur_produit</th>\n",
       "      <td>90</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>65.2</td>\n",
       "      <td>36.2889</td>\n",
       "      <td>24</td>\n",
       "      <td>35.125</td>\n",
       "      <td>50</td>\n",
       "      <td>80</td>\n",
       "      <td>160</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>pointure</th>\n",
       "      <td>223</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>39.139</td>\n",
       "      <td>2.63781</td>\n",
       "      <td>25</td>\n",
       "      <td>37</td>\n",
       "      <td>39</td>\n",
       "      <td>41</td>\n",
       "      <td>48</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>vintage</th>\n",
       "      <td>5890</td>\n",
       "      <td>2</td>\n",
       "      <td>False</td>\n",
       "      <td>3832</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>marque</th>\n",
       "      <td>4329</td>\n",
       "      <td>2658</td>\n",
       "      <td>tmp</td>\n",
       "      <td>93</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>auteur</th>\n",
       "      <td>1412</td>\n",
       "      <td>1021</td>\n",
       "      <td>Hergé</td>\n",
       "      <td>21</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>editions</th>\n",
       "      <td>1437</td>\n",
       "      <td>655</td>\n",
       "      <td>Pocket</td>\n",
       "      <td>80</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hauteur_produit</th>\n",
       "      <td>113</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>64.2788</td>\n",
       "      <td>33.4202</td>\n",
       "      <td>9</td>\n",
       "      <td>45.5</td>\n",
       "      <td>61</td>\n",
       "      <td>72</td>\n",
       "      <td>198</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>poids</th>\n",
       "      <td>8878</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7794.86</td>\n",
       "      <td>82201.3</td>\n",
       "      <td>0.1</td>\n",
       "      <td>252</td>\n",
       "      <td>500</td>\n",
       "      <td>1000</td>\n",
       "      <td>2.2e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>prix</th>\n",
       "      <td>8880</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>21.5033</td>\n",
       "      <td>44.0243</td>\n",
       "      <td>0.5</td>\n",
       "      <td>6</td>\n",
       "      <td>10</td>\n",
       "      <td>20</td>\n",
       "      <td>2200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>categorie</th>\n",
       "      <td>8879</td>\n",
       "      <td>11</td>\n",
       "      <td>mode</td>\n",
       "      <td>2982</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sous_categorie_1</th>\n",
       "      <td>7292</td>\n",
       "      <td>169</td>\n",
       "      <td>mode</td>\n",
       "      <td>511</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sous_categorie_2</th>\n",
       "      <td>3620</td>\n",
       "      <td>145</td>\n",
       "      <td>label selection</td>\n",
       "      <td>1132</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sous_categorie_3</th>\n",
       "      <td>927</td>\n",
       "      <td>124</td>\n",
       "      <td>les coups de coeur des vendeurs</td>\n",
       "      <td>65</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sous_categorie_4</th>\n",
       "      <td>70</td>\n",
       "      <td>37</td>\n",
       "      <td>les coups de coeur des vendeurs</td>\n",
       "      <td>7</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>nom_produit</th>\n",
       "      <td>8064</td>\n",
       "      <td>7921</td>\n",
       "      <td>carte postale</td>\n",
       "      <td>6</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>nom_magasin</th>\n",
       "      <td>8880</td>\n",
       "      <td>46</td>\n",
       "      <td>Emmaüs Lyon</td>\n",
       "      <td>895</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    count unique  \\\n",
       "nb_images            8880    NaN   \n",
       "longueur_image       8823    NaN   \n",
       "largeur_image        8823    NaN   \n",
       "url_image            8823   8775   \n",
       "description_produit  8880   8836   \n",
       "taille               2414     33   \n",
       "matiere              3947   1722   \n",
       "age                   120     18   \n",
       "garantie              101      2   \n",
       "annee                1497    NaN   \n",
       "couleur              5321     18   \n",
       "largeur_produit       107    NaN   \n",
       "wifi                   83      2   \n",
       "etat                 8856      5   \n",
       "longueur_produit       90    NaN   \n",
       "pointure              223    NaN   \n",
       "vintage              5890      2   \n",
       "marque               4329   2658   \n",
       "auteur               1412   1021   \n",
       "editions             1437    655   \n",
       "hauteur_produit       113    NaN   \n",
       "poids                8878    NaN   \n",
       "prix                 8880    NaN   \n",
       "categorie            8879     11   \n",
       "sous_categorie_1     7292    169   \n",
       "sous_categorie_2     3620    145   \n",
       "sous_categorie_3      927    124   \n",
       "sous_categorie_4       70     37   \n",
       "nom_produit          8064   7921   \n",
       "nom_magasin          8880     46   \n",
       "\n",
       "                                                                   top  freq  \\\n",
       "nb_images                                                          NaN   NaN   \n",
       "longueur_image                                                     NaN   NaN   \n",
       "largeur_image                                                      NaN   NaN   \n",
       "url_image            https://d1kvfoyrif6wzg.cloudfront.net/assets/i...     4   \n",
       "description_produit                 Relié, 48 pages, couverture usagée     6   \n",
       "taille                                                              38   402   \n",
       "matiere                                                          Coton   144   \n",
       "age                                                                 4a    14   \n",
       "garantie                                                        6 mois   100   \n",
       "annee                                                              NaN   NaN   \n",
       "couleur                                                    Multicolore  1063   \n",
       "largeur_produit                                                    NaN   NaN   \n",
       "wifi                                                              True    62   \n",
       "etat                                                          bon état  4537   \n",
       "longueur_produit                                                   NaN   NaN   \n",
       "pointure                                                           NaN   NaN   \n",
       "vintage                                                          False  3832   \n",
       "marque                                                             tmp    93   \n",
       "auteur                                                           Hergé    21   \n",
       "editions                                                        Pocket    80   \n",
       "hauteur_produit                                                    NaN   NaN   \n",
       "poids                                                              NaN   NaN   \n",
       "prix                                                               NaN   NaN   \n",
       "categorie                                                         mode  2982   \n",
       "sous_categorie_1                                                  mode   511   \n",
       "sous_categorie_2                                       label selection  1132   \n",
       "sous_categorie_3                       les coups de coeur des vendeurs    65   \n",
       "sous_categorie_4                       les coups de coeur des vendeurs     7   \n",
       "nom_produit                                              carte postale     6   \n",
       "nom_magasin                                                Emmaüs Lyon   895   \n",
       "\n",
       "                        mean      std  min     25%   50%   75%          max  \n",
       "nb_images            3.63345  2.04857    0       2     3     5           29  \n",
       "longueur_image       1807.82  1025.25   58    1000  1536  2448         5472  \n",
       "largeur_image        1801.77  1101.21   64   970.5  1536  2448         5472  \n",
       "url_image                NaN      NaN  NaN     NaN   NaN   NaN          NaN  \n",
       "description_produit      NaN      NaN  NaN     NaN   NaN   NaN          NaN  \n",
       "taille                   NaN      NaN  NaN     NaN   NaN   NaN          NaN  \n",
       "matiere                  NaN      NaN  NaN     NaN   NaN   NaN          NaN  \n",
       "age                      NaN      NaN  NaN     NaN   NaN   NaN          NaN  \n",
       "garantie                 NaN      NaN  NaN     NaN   NaN   NaN          NaN  \n",
       "annee                14810.1   496237    0    1979  1998  2007  1.92019e+07  \n",
       "couleur                  NaN      NaN  NaN     NaN   NaN   NaN          NaN  \n",
       "largeur_produit      42.0589  20.5453   10      30    40  46.5          140  \n",
       "wifi                     NaN      NaN  NaN     NaN   NaN   NaN          NaN  \n",
       "etat                     NaN      NaN  NaN     NaN   NaN   NaN          NaN  \n",
       "longueur_produit        65.2  36.2889   24  35.125    50    80          160  \n",
       "pointure              39.139  2.63781   25      37    39    41           48  \n",
       "vintage                  NaN      NaN  NaN     NaN   NaN   NaN          NaN  \n",
       "marque                   NaN      NaN  NaN     NaN   NaN   NaN          NaN  \n",
       "auteur                   NaN      NaN  NaN     NaN   NaN   NaN          NaN  \n",
       "editions                 NaN      NaN  NaN     NaN   NaN   NaN          NaN  \n",
       "hauteur_produit      64.2788  33.4202    9    45.5    61    72          198  \n",
       "poids                7794.86  82201.3  0.1     252   500  1000      2.2e+06  \n",
       "prix                 21.5033  44.0243  0.5       6    10    20         2200  \n",
       "categorie                NaN      NaN  NaN     NaN   NaN   NaN          NaN  \n",
       "sous_categorie_1         NaN      NaN  NaN     NaN   NaN   NaN          NaN  \n",
       "sous_categorie_2         NaN      NaN  NaN     NaN   NaN   NaN          NaN  \n",
       "sous_categorie_3         NaN      NaN  NaN     NaN   NaN   NaN          NaN  \n",
       "sous_categorie_4         NaN      NaN  NaN     NaN   NaN   NaN          NaN  \n",
       "nom_produit              NaN      NaN  NaN     NaN   NaN   NaN          NaN  \n",
       "nom_magasin              NaN      NaN  NaN     NaN   NaN   NaN          NaN  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.describe(include='all').T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    3027\n",
       "2    2953\n",
       "1    2900\n",
       "Name: delai_vente, dtype: int64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train.delai_vente.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "bon état         4537\n",
       "en l'état        1577\n",
       "comme neuf       1110\n",
       "reconditionné     885\n",
       "neuf              747\n",
       "Name: etat, dtype: int64"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.etat.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# feature engineering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import LabelEncoder\n",
    "from sklearn.impute import SimpleImputer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "feat_num = [\"poids\", \"prix\", \"nb_images\", \"longueur_image\", \"largeur_image\"]\n",
    "feat_cat = [\"garantie\",\"couleur\",\"wifi\",\"etat\",\"vintage\",\"categorie\",\"taille\",\"age\",\"sous_categorie_1\",\"sous_categorie_2\",\"sous_categorie_3\",\"sous_categorie_4\",\"nom_magasin\",\"matiere\",\"auteur\",\"editions\",\"marque\"]\n",
    "feat_engineered = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# etat to ordinal\n",
    "def ordinalEtat(string):\n",
    "    if(string==\"en l'état\"):\n",
    "        return(1)\n",
    "    elif(string==\"bon état\"):\n",
    "        return(2)\n",
    "    elif(string==\"reconditionné\"):\n",
    "        return(3)\n",
    "    elif(string==\"comme neuf\"):\n",
    "        return(4)\n",
    "    elif(string==\"neuf\"):\n",
    "        return(5)\n",
    "    \n",
    "X_train[\"ordinaletat\"] = X_train[\"etat\"].apply(lambda x: ordinalEtat(x))\n",
    "X_test[\"ordinaletat\"] = X_test[\"etat\"].apply(lambda x: ordinalEtat(x))\n",
    "feat_engineered.append(\"ordinaletat\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# dummify categorical columns\n",
    "\n",
    "dummies_train = pd.get_dummies(X_train[feat_cat])\n",
    "dummies_test = pd.get_dummies(X_test[feat_cat])\n",
    "\n",
    "feat_cat_dummified = dummies_train.columns\n",
    "\n",
    "X_train = pd.concat([X_train, dummies_train], axis=1)\n",
    "X_train = X_train.drop(feat_cat, axis=1)\n",
    "\n",
    "X_test = pd.concat([X_test, dummies_test], axis=1)\n",
    "X_test = X_test.drop(feat_cat, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# nb characters description_produit\n",
    "X_train[\"nbChardescription_produit\"] = X_train[\"description_produit\"].apply(lambda x: len(x))\n",
    "X_test[\"nbChardescription_produit\"] = X_test[\"description_produit\"].apply(lambda x: len(x))\n",
    "feat_engineered.append(\"nbChardescription_produit\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# has largeur_produit\n",
    "X_train[\"haslargeur_produit\"] = X_train[\"largeur_produit\"].apply(lambda x: 0 if pd.isnull(x) else 1).astype('bool')\n",
    "X_test[\"haslargeur_produit\"] = X_test[\"largeur_produit\"].apply(lambda x: 0 if pd.isnull(x) else 1).astype('bool')\n",
    "feat_engineered.append(\"haslargeur_produit\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# has longueur_produit\n",
    "X_train[\"haslongueur_produit\"] = X_train[\"longueur_produit\"].apply(lambda x: 0 if pd.isnull(x) else 1).astype('bool')\n",
    "X_test[\"haslongueur_produit\"] = X_test[\"longueur_produit\"].apply(lambda x: 0 if pd.isnull(x) else 1).astype('bool')\n",
    "feat_engineered.append(\"haslongueur_produit\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# has pointure\n",
    "X_train[\"haspointure\"] = X_train[\"pointure\"].apply(lambda x: 0 if pd.isnull(x) else 1).astype('bool')\n",
    "X_test[\"haspointure\"] = X_test[\"pointure\"].apply(lambda x: 0 if pd.isnull(x) else 1).astype('bool')\n",
    "feat_engineered.append(\"haspointure\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# has hauteur_produit\n",
    "X_train[\"hashauteur_produit\"] = X_train[\"hauteur_produit\"].apply(lambda x: 0 if pd.isnull(x) else 1).astype('bool')\n",
    "X_test[\"hashauteur_produit\"] = X_test[\"hauteur_produit\"].apply(lambda x: 0 if pd.isnull(x) else 1).astype('bool')\n",
    "feat_engineered.append(\"hashauteur_produit\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# nb characters nom_produit\n",
    "X_train[\"nbCharnom_produit\"] = X_train[\"nom_produit\"].apply(lambda x: 0 if pd.isnull(x) else len(str(x)))\n",
    "X_test[\"nbCharnom_produit\"] = X_test[\"nom_produit\"].apply(lambda x: 0 if pd.isnull(x) else len(str(x)))\n",
    "feat_engineered.append(\"nbCharnom_produit\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# nom_magasin contains Emmaüs\n",
    "X_train[\"nom_magasinContainsEmmaus\"] = X_train[\"description_produit\"].apply(lambda x: 0 if pd.isnull(x) else \"Emmaüs\" in x).astype('bool')\n",
    "X_test[\"nom_magasinContainsEmmaus\"] = X_test[\"description_produit\"].apply(lambda x: 0 if pd.isnull(x) else \"Emmaüs\" in x).astype('bool')\n",
    "feat_engineered.append(\"nom_magasinContainsEmmaus\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# has image nb_images\n",
    "X_train[\"hasimage\"] = X_train[\"nb_images\"].apply(lambda x: 0 if x == 0 else 1).astype('bool')\n",
    "X_test[\"hasimage\"] = X_test[\"nb_images\"].apply(lambda x: 0 if x == 0 else 1).astype('bool')\n",
    "feat_engineered.append(\"hasimage\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# select only features used for modelling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "6682\n"
     ]
    }
   ],
   "source": [
    "features = feat_num+list(feat_cat_dummified)+feat_engineered\n",
    "print(len(features))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/jonathan/anaconda2/lib/python2.7/site-packages/pandas/core/indexing.py:1472: FutureWarning: \n",
      "Passing list-likes to .loc or [] with any missing label will raise\n",
      "KeyError in the future, you can use .reindex() as an alternative.\n",
      "\n",
      "See the documentation here:\n",
      "https://pandas.pydata.org/pandas-docs/stable/indexing.html#deprecate-loc-reindex-listlike\n",
      "  return self._getitem_tuple(key)\n"
     ]
    }
   ],
   "source": [
    "X_train = X_train.loc[:, features]\n",
    "X_test = X_test.loc[:, features]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# missing data imputation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "# train\n",
    "imp = SimpleImputer(strategy='median')\n",
    "imp.fit(X_train)\n",
    "X_train = imp.transform(X_train)\n",
    "\n",
    "# test\n",
    "X_test = imp.transform(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# model selection and hyperparameter tuning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## random forest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.model_selection import cross_val_score\n",
    "from sklearn.metrics import log_loss "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "# max_depth values to experiment\n",
    "max_depth_values = [5,10,20,100,200]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "----------------------------------------------------------------------\n",
      "('max_depth:', 5)\n",
      "('CV logloss', 1.0776594232835266)\n",
      "----------------------------------------------------------------------\n",
      "('max_depth:', 10)\n",
      "('CV logloss', 1.0582727287365181)\n",
      "----------------------------------------------------------------------\n",
      "('max_depth:', 20)\n",
      "('CV logloss', 1.0291053118601903)\n",
      "----------------------------------------------------------------------\n",
      "('max_depth:', 100)\n",
      "('CV logloss', 0.9748831673859627)\n",
      "----------------------------------------------------------------------\n",
      "('max_depth:', 200)\n",
      "('CV logloss', 0.9854028233592302)\n",
      "CPU times: user 36.6 s, sys: 11.9 s, total: 48.6 s\n",
      "Wall time: 2min 29s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "# for each possible value of max_depth: print CV logloss\n",
    "for depth in max_depth_values:\n",
    "    rf = RandomForestClassifier(n_estimators=50,n_jobs=-1,max_depth=depth)\n",
    "    neg_log_losses = cross_val_score(rf, X_train, np.ravel(y_train), cv=5, scoring=\"neg_log_loss\")\n",
    "    log_losses = [-x for x in neg_log_losses]\n",
    "    print(\"----------------------------------------------------------------------\")\n",
    "    print(\"max_depth:\",depth)\n",
    "    print(\"CV logloss\",np.mean(log_losses))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('CV logloss', 0.9504948728835766)\n"
     ]
    }
   ],
   "source": [
    "# check results with a higher number of estimators\n",
    "rf = RandomForestClassifier(n_estimators=1000,n_jobs=-1,max_depth=100)\n",
    "neg_log_losses = cross_val_score(rf, X_train, np.ravel(y_train), cv=5, scoring=\"neg_log_loss\")\n",
    "log_losses = [-x for x in neg_log_losses]\n",
    "print(\"CV logloss\",np.mean(log_losses))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# final predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 7min 15s, sys: 1.1 s, total: 7min 17s\n",
      "Wall time: 2min 8s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "rf = RandomForestClassifier(n_estimators=1000,n_jobs=-1,max_depth=100)\n",
    "rf.fit(X_train, np.ravel(y_train))\n",
    "pred_test = rf.predict_proba(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# submit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_submission = pd.DataFrame(pred_test, index=X_test_index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.172846</td>\n",
       "      <td>0.164243</td>\n",
       "      <td>0.662911</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.318958</td>\n",
       "      <td>0.266533</td>\n",
       "      <td>0.414509</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.247665</td>\n",
       "      <td>0.721040</td>\n",
       "      <td>0.031295</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.129430</td>\n",
       "      <td>0.238071</td>\n",
       "      <td>0.632499</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.512244</td>\n",
       "      <td>0.289359</td>\n",
       "      <td>0.198398</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.473997</td>\n",
       "      <td>0.495044</td>\n",
       "      <td>0.030959</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.266829</td>\n",
       "      <td>0.298948</td>\n",
       "      <td>0.434222</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.335390</td>\n",
       "      <td>0.583538</td>\n",
       "      <td>0.081072</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0.272147</td>\n",
       "      <td>0.197345</td>\n",
       "      <td>0.530508</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0.438600</td>\n",
       "      <td>0.286839</td>\n",
       "      <td>0.274561</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>0.578887</td>\n",
       "      <td>0.200673</td>\n",
       "      <td>0.220440</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>0.289501</td>\n",
       "      <td>0.354937</td>\n",
       "      <td>0.355562</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>0.310926</td>\n",
       "      <td>0.310004</td>\n",
       "      <td>0.379070</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>0.390874</td>\n",
       "      <td>0.297914</td>\n",
       "      <td>0.311212</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>0.283791</td>\n",
       "      <td>0.407367</td>\n",
       "      <td>0.308842</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>0.473209</td>\n",
       "      <td>0.261661</td>\n",
       "      <td>0.265130</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>0.358057</td>\n",
       "      <td>0.262348</td>\n",
       "      <td>0.379594</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>0.363759</td>\n",
       "      <td>0.279154</td>\n",
       "      <td>0.357087</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>0.218555</td>\n",
       "      <td>0.290945</td>\n",
       "      <td>0.490500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>0.335005</td>\n",
       "      <td>0.295589</td>\n",
       "      <td>0.369406</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>0.264791</td>\n",
       "      <td>0.138298</td>\n",
       "      <td>0.596911</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>0.128667</td>\n",
       "      <td>0.465756</td>\n",
       "      <td>0.405577</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>0.394153</td>\n",
       "      <td>0.179650</td>\n",
       "      <td>0.426196</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>0.423803</td>\n",
       "      <td>0.283496</td>\n",
       "      <td>0.292702</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>0.295278</td>\n",
       "      <td>0.674237</td>\n",
       "      <td>0.030486</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>0.605816</td>\n",
       "      <td>0.231002</td>\n",
       "      <td>0.163181</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>0.265986</td>\n",
       "      <td>0.683239</td>\n",
       "      <td>0.050776</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>0.008563</td>\n",
       "      <td>0.008779</td>\n",
       "      <td>0.982657</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>0.210744</td>\n",
       "      <td>0.161490</td>\n",
       "      <td>0.627765</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>0.220030</td>\n",
       "      <td>0.389194</td>\n",
       "      <td>0.390776</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2930</th>\n",
       "      <td>0.518725</td>\n",
       "      <td>0.328563</td>\n",
       "      <td>0.152712</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2931</th>\n",
       "      <td>0.815707</td>\n",
       "      <td>0.142412</td>\n",
       "      <td>0.041881</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2932</th>\n",
       "      <td>0.284363</td>\n",
       "      <td>0.214700</td>\n",
       "      <td>0.500936</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2933</th>\n",
       "      <td>0.218986</td>\n",
       "      <td>0.451039</td>\n",
       "      <td>0.329975</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2934</th>\n",
       "      <td>0.434320</td>\n",
       "      <td>0.360642</td>\n",
       "      <td>0.205038</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2935</th>\n",
       "      <td>0.286300</td>\n",
       "      <td>0.324245</td>\n",
       "      <td>0.389455</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2936</th>\n",
       "      <td>0.398947</td>\n",
       "      <td>0.570149</td>\n",
       "      <td>0.030904</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2937</th>\n",
       "      <td>0.119878</td>\n",
       "      <td>0.340339</td>\n",
       "      <td>0.539783</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2938</th>\n",
       "      <td>0.195139</td>\n",
       "      <td>0.229911</td>\n",
       "      <td>0.574949</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2939</th>\n",
       "      <td>0.438520</td>\n",
       "      <td>0.353587</td>\n",
       "      <td>0.207893</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2940</th>\n",
       "      <td>0.140377</td>\n",
       "      <td>0.397664</td>\n",
       "      <td>0.461959</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2941</th>\n",
       "      <td>0.259668</td>\n",
       "      <td>0.356077</td>\n",
       "      <td>0.384255</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2942</th>\n",
       "      <td>0.233752</td>\n",
       "      <td>0.281232</td>\n",
       "      <td>0.485016</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2943</th>\n",
       "      <td>0.115347</td>\n",
       "      <td>0.143454</td>\n",
       "      <td>0.741199</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2944</th>\n",
       "      <td>0.445903</td>\n",
       "      <td>0.248308</td>\n",
       "      <td>0.305789</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2945</th>\n",
       "      <td>0.406415</td>\n",
       "      <td>0.253331</td>\n",
       "      <td>0.340253</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2946</th>\n",
       "      <td>0.162862</td>\n",
       "      <td>0.299849</td>\n",
       "      <td>0.537289</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2947</th>\n",
       "      <td>0.291093</td>\n",
       "      <td>0.261266</td>\n",
       "      <td>0.447641</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2948</th>\n",
       "      <td>0.393044</td>\n",
       "      <td>0.326760</td>\n",
       "      <td>0.280196</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2949</th>\n",
       "      <td>0.320106</td>\n",
       "      <td>0.296386</td>\n",
       "      <td>0.383508</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2950</th>\n",
       "      <td>0.361709</td>\n",
       "      <td>0.624140</td>\n",
       "      <td>0.014151</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2951</th>\n",
       "      <td>0.069248</td>\n",
       "      <td>0.097855</td>\n",
       "      <td>0.832897</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2952</th>\n",
       "      <td>0.345922</td>\n",
       "      <td>0.266377</td>\n",
       "      <td>0.387700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2953</th>\n",
       "      <td>0.056033</td>\n",
       "      <td>0.153197</td>\n",
       "      <td>0.790770</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2954</th>\n",
       "      <td>0.202945</td>\n",
       "      <td>0.771184</td>\n",
       "      <td>0.025871</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2955</th>\n",
       "      <td>0.437002</td>\n",
       "      <td>0.309652</td>\n",
       "      <td>0.253346</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2956</th>\n",
       "      <td>0.322678</td>\n",
       "      <td>0.360354</td>\n",
       "      <td>0.316968</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2957</th>\n",
       "      <td>0.256939</td>\n",
       "      <td>0.197650</td>\n",
       "      <td>0.545411</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2958</th>\n",
       "      <td>0.541785</td>\n",
       "      <td>0.283079</td>\n",
       "      <td>0.175136</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2959</th>\n",
       "      <td>0.114544</td>\n",
       "      <td>0.215854</td>\n",
       "      <td>0.669602</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2960 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "             0         1         2\n",
       "id                                \n",
       "0     0.172846  0.164243  0.662911\n",
       "1     0.318958  0.266533  0.414509\n",
       "2     0.247665  0.721040  0.031295\n",
       "3     0.129430  0.238071  0.632499\n",
       "4     0.512244  0.289359  0.198398\n",
       "5     0.473997  0.495044  0.030959\n",
       "6     0.266829  0.298948  0.434222\n",
       "7     0.335390  0.583538  0.081072\n",
       "8     0.272147  0.197345  0.530508\n",
       "9     0.438600  0.286839  0.274561\n",
       "10    0.578887  0.200673  0.220440\n",
       "11    0.289501  0.354937  0.355562\n",
       "12    0.310926  0.310004  0.379070\n",
       "13    0.390874  0.297914  0.311212\n",
       "14    0.283791  0.407367  0.308842\n",
       "15    0.473209  0.261661  0.265130\n",
       "16    0.358057  0.262348  0.379594\n",
       "17    0.363759  0.279154  0.357087\n",
       "18    0.218555  0.290945  0.490500\n",
       "19    0.335005  0.295589  0.369406\n",
       "20    0.264791  0.138298  0.596911\n",
       "21    0.128667  0.465756  0.405577\n",
       "22    0.394153  0.179650  0.426196\n",
       "23    0.423803  0.283496  0.292702\n",
       "24    0.295278  0.674237  0.030486\n",
       "25    0.605816  0.231002  0.163181\n",
       "26    0.265986  0.683239  0.050776\n",
       "27    0.008563  0.008779  0.982657\n",
       "28    0.210744  0.161490  0.627765\n",
       "29    0.220030  0.389194  0.390776\n",
       "...        ...       ...       ...\n",
       "2930  0.518725  0.328563  0.152712\n",
       "2931  0.815707  0.142412  0.041881\n",
       "2932  0.284363  0.214700  0.500936\n",
       "2933  0.218986  0.451039  0.329975\n",
       "2934  0.434320  0.360642  0.205038\n",
       "2935  0.286300  0.324245  0.389455\n",
       "2936  0.398947  0.570149  0.030904\n",
       "2937  0.119878  0.340339  0.539783\n",
       "2938  0.195139  0.229911  0.574949\n",
       "2939  0.438520  0.353587  0.207893\n",
       "2940  0.140377  0.397664  0.461959\n",
       "2941  0.259668  0.356077  0.384255\n",
       "2942  0.233752  0.281232  0.485016\n",
       "2943  0.115347  0.143454  0.741199\n",
       "2944  0.445903  0.248308  0.305789\n",
       "2945  0.406415  0.253331  0.340253\n",
       "2946  0.162862  0.299849  0.537289\n",
       "2947  0.291093  0.261266  0.447641\n",
       "2948  0.393044  0.326760  0.280196\n",
       "2949  0.320106  0.296386  0.383508\n",
       "2950  0.361709  0.624140  0.014151\n",
       "2951  0.069248  0.097855  0.832897\n",
       "2952  0.345922  0.266377  0.387700\n",
       "2953  0.056033  0.153197  0.790770\n",
       "2954  0.202945  0.771184  0.025871\n",
       "2955  0.437002  0.309652  0.253346\n",
       "2956  0.322678  0.360354  0.316968\n",
       "2957  0.256939  0.197650  0.545411\n",
       "2958  0.541785  0.283079  0.175136\n",
       "2959  0.114544  0.215854  0.669602\n",
       "\n",
       "[2960 rows x 3 columns]"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_submission"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import io, math, requests\n",
    "\n",
    "# Ne fonctionne qu'en Python3, voir commentaire ci-dessous pour Python2\n",
    "def submit_prediction(df, sep=',', **kwargs):\n",
    "    # TOKEN a recuperer sur la plateforme: \"Submissions\" > \"Submit from your Python Notebook\"\n",
    "    TOKEN='37760905e1b8e6e98e0c5ce188a74eae75cb3750b26031fae91aba16e5ed87f5e5138fd1c56166329d2b80a603f9a5a105287f3fc7a02d8accbfd8f9a136e862'  \n",
    "    URL='https://qscore.meilleurdatascientistdefrance.com/api/submissions'\n",
    "    buffer = io.BytesIO() # Python 2\n",
    "    df.to_csv(buffer, sep=sep, **kwargs)\n",
    "    buffer.seek(0)\n",
    "    r = requests.post(URL, headers={'Authorization': 'Bearer {}'.format(TOKEN)},files={'datafile': buffer})\n",
    "    if r.status_code == 429:\n",
    "        raise Exception('Submissions are too close. Next submission is only allowed in {} seconds.'.format(int(math.ceil(int(r.headers['x-rate-limit-remaining']) / 1000.0))))\n",
    "    if r.status_code != 200:\n",
    "        raise Exception(r.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "submit_prediction(df_submission, sep=',', index=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}