week_02_cnn_for_texts.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "13pL--6rycN3"
   },
   "source": [
    "# Practice: Dealing with texts using CNN\n",
    "_Reference: Based on YSDA [materials](https://github.com/yandexdataschool/nlp_course/blob/master/week02_classification/seminar.ipynb). Special thanks to [Oleg Vasilev](https://github.com/Omrigan/) for the core assignment idea._\n",
    "\n",
    "Today we're gonna apply the newly learned tools for the task of predicting job salary.\n",
    "\n",
    "<img src=\"https://storage.googleapis.com/kaggle-competitions/kaggle/3342/media/salary%20prediction%20engine%20v2.png\" width=400px>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "34x92vWQycN_"
   },
   "source": [
    "## About the challenge\n",
    "For starters, let's download and unpack the data.\n",
    "\n",
    "You can also get it from [Yandex.Disk](https://yadi.sk/d/vVEOWPFY3NruT7) or [the competition page](https://www.kaggle.com/c/job-salary-prediction/data) (pick `Train_rev1.*`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "vwN72gd4ycOA",
    "outputId": "0e802d91-32b7-402f-fc4d-bd40dadcb10a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "100   145    0   145    0     0    436      0 --:--:-- --:--:-- --:--:--   436\n",
      "100   342  100   342    0     0    475      0 --:--:-- --:--:-- --:--:--   475\n",
      "100  119M  100  119M    0     0  20.3M      0  0:00:05  0:00:05 --:--:-- 30.4M\n"
     ]
    }
   ],
   "source": [
    "!curl -L \"https://www.dropbox.com/s/5msc5ix7ndyba10/Train_rev1.csv.tar.gz?dl=1\" -o Train_rev1.csv.tar.gz\n",
    "!tar xzf Train_rev1.csv.tar.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "xIjzhbXEpHfT",
    "outputId": "1227e443-9056-434b-e02a-cb2bc68fb550"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(244768, 12)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "\n",
    "data = pd.read_csv(\"Train_rev1.csv\", index_col=None)\n",
    "data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "z7kznuJfycOH"
   },
   "source": [
    "One problem with salary prediction is that it's oddly distributed: there are many people who are paid standard salaries and a few that get tons o money. The distribution is fat-tailed on the right side, which is inconvenient for MSE minimization.\n",
    "\n",
    "There are several techniques to combat this: using a different loss function, predicting log-target instead of raw target or even replacing targets with their percentiles among all salaries in the training set. We gonna use logarithm for now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 265
    },
    "id": "UuuKIKfrycOH",
    "outputId": "8837fb06-fc83-42f2-d353-4ab5b002aa76"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfMAAAD4CAYAAAD4vw88AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAcNElEQVR4nO3db7Bd1Xnf8e8vEsHENpg/glElqLCtpsbMWJg7qjJ0PE6UGMVkAu5AI09rNKmmyrg4wdN0GuG8sPOCGdGOTcO0JpUNRVDHoGJ7YGywrQi7bmawsOxgkBAExahGQUWyIRhPxiSSn74468ZHV/deHUn3377n+5k5c/Z5zl77rn3u3fc5e+2110pVIUmSuuvnZrsCkiTp1JjMJUnqOJO5JEkdZzKXJKnjTOaSJHXcwtmuwMk677zzatmyZbNdDWnO+/a3v/2Dqlo02/WYjMezdHyTHcudTebLli1j586ds10Nac5L8n9nuw7H4/EsHd9kx7LN7JIkdZzJXJKkjjOZS5LUcSZzSZI6zmQuSVLHmcwlSeo4k7kkSR1nMpckqeNM5pIkdVxnR4CbSss2fum46+zbdNUM1ESS5gb/L3aLyXxA/mFLkuYqm9klSeo4k7kkSR1nMpckqeNM5pIkdZzJXJKkjjOZS5LUcd6aJg2JJK8DvgGcTu/Yv7+qPprkY8C/BQ61VT9SVQ+1MjcB64EjwO9V1Vda/HLgLuAM4CHgxqqqJKcDdwOXAz8Efquq9s3IDmpgg9xqq27xzFwaHq8Bv1JV7wBWAGuSrGrv3VpVK9pjNJFfAqwF3g6sAT6ZZEFb/3ZgA7C8Pda0+Hrg5ap6K3ArcMv075Ykk7k0JKrnx+3lae1RkxS5Gri3ql6rqueAvcDKJIuBM6vq0aoqemfi1/SV2dKW7wdWJ8kU74qkMQZK5knelOT+JE8n2ZPkl5Kck2Rbkmfb89l969+UZG+SZ5Jc2Re/PMmT7b3bRg/yJKcnua/FdyRZNuV7KokkC5I8DhwEtlXVjvbWh5I8keTOvmN5CfB8X/H9LbakLY+NH1Wmqg4DrwDnTlCXDUl2Jtl56NCh8VaRNKBBz8z/GPhyVf1T4B3AHmAjsL2qlgPb22ub5qQ5rKqOVNUKYCm9s+xL6R2Xb6HX9H4A+Hhbfbwz6pokPlmZ8eqyuapGqmpk0aJFA++DpGMdN5knORN4F3AHQFX9XVX9DUc3p23h6GY2m+akOawdw18H1lTViy3J/xT4FLCyrbYfuLCv2FLghRZfOk78qDJJFgJnAS9Nz15IGjXImfmb6fVy/R9J/iLJp5O8Hrigqg4AtOfz2/rT1jRns5x08pIsSvKmtnwG8KvA0+2L9qj3Abva8oPA2nYZ7GJ6rWmPteP91SSr2pfu64EH+sqsa8vXAo+0L++SptEgt6YtBN4J/G5V7Ujyx7Qm9QlMW9NcVW0GNgOMjIz4D0I6MYuBLe2y188BW6vqi0nuSbKC3jG3D/gdgKranWQr8BRwGLihqo60bX2Qn92a9nB7QK8F754ke+mdka+dgf2Sht4gyXw/sL+vo8z99JL5i0kWV9WB9s3+YN/6J9s0t9+mOWl6VNUTwGXjxD8wSZmbgZvHie8ELh0n/hPgulOrqaQTddxm9qr6f8DzSX6xhVbT+6be35y2jqOb2WyakyRphgw6AtzvAp9J8vPA94DfpjXTJVkPfJ/2bdymOUmSZtZAybyqHgdGxnlr9QTr2zQnSdIMcQQ4SZI6zolWJGkecRKV4eSZuSRJHWcylySp40zmkiR1nMlckqSOM5lLktRxJnNJkjrOZC5JUseZzCVJ6jiTuSRJHWcylySp40zmkiR1nMlckqSOM5lLktRxJnNJkjrOZC4NiSSvS/JYku8m2Z3kj1r8nCTbkjzbns/uK3NTkr1JnklyZV/88iRPtvduS5IWPz3JfS2+I8myGd9RaQiZzKXh8RrwK1X1DmAFsCbJKmAjsL2qlgPb22uSXAKsBd4OrAE+mWRB29btwAZgeXusafH1wMtV9VbgVuCWGdgvaeiZzKUhUT0/bi9Pa48Crga2tPgW4Jq2fDVwb1W9VlXPAXuBlUkWA2dW1aNVVcDdY8qMbut+YPXoWbuk6WMyl4ZIkgVJHgcOAtuqagdwQVUdAGjP57fVlwDP9xXf32JL2vLY+FFlquow8Apw7gR12ZBkZ5Kdhw4dmoK9k4aXyVwaIlV1pKpWAEvpnWVfOsnq451R1yTxycqMV5fNVTVSVSOLFi2apBqSjsdkLg2hqvob4Ov0rnW/2JrOac8H22r7gQv7ii0FXmjxpePEjyqTZCFwFvDSdOyDpJ8xmUtDIsmiJG9qy2cAvwo8DTwIrGurrQMeaMsPAmtbD/WL6XV0e6w1xb+aZFW7Hn79mDKj27oWeKRdV5c0jQZK5kn2tdtQHk+ys8W8nUXqlsXA15I8AXyL3jXzLwKbgF9L8izwa+01VbUb2Ao8BXwZuKGqjrRtfRD4NL1OcX8FPNzidwDnJtkL/Htaz3hJ02vhCaz7y1X1g77Xo7ezbEqysb3+gzG3s/wj4M+S/JP2T2D0dpZvAg/Ra+J7mL7bWZKspXc7y2+d4r5J6lNVTwCXjRP/IbB6gjI3AzePE98JHHO9vap+Alx3ypWVdEJOpZnd21kkSZoDBk3mBXw1ybeTbGixGb+dxVtZJEk61qDN7FdU1QtJzge2JXl6knWn7XaWqtoMbAYYGRmxU40kSQx4Zl5VL7Tng8AXgJV4O4skSXPCcZN5ktcneePoMvAeYBfeziJJ0pwwSDP7BcAXWn+0hcCfVtWXk3wL2JpkPfB9Wg/WqtqdZPR2lsMcezvLXcAZ9Hqx99/Ock+7neUler3hJUlz2LKNXzruOvs2XTUDNdFxk3lVfQ94xzhxb2eRJGkOcAQ4SZI6zmQuSVLHmcwlSeo4k7kkSR1nMpckqeNM5pIkdZzJXJKkjjOZS5LUcSZzSZI6zmQuSVLHmcwlSeo4k7kkSR1nMpeGRJILk3wtyZ4ku5Pc2OIfS/LXSR5vj/f2lbkpyd4kzyS5si9+eZIn23u3tWmNaVMf39fiO5Ism/EdlYaQyVwaHoeB36+qtwGrgBuSXNLeu7WqVrTHQwDtvbXA24E1wCeTLGjr3w5sAJa3x5oWXw+8XFVvBW4FbpmB/ZKGnslcGhJVdaCqvtOWXwX2AEsmKXI1cG9VvVZVzwF7gZVJFgNnVtWjVVXA3cA1fWW2tOX7gdWjZ+2Spo/JXBpCrfn7MmBHC30oyRNJ7kxydostAZ7vK7a/xZa05bHxo8pU1WHgFeDcCeqwIcnOJDsPHTp06jslDTGTuTRkkrwB+Bzw4ar6Eb0m87cAK4ADwMdHVx2neE0Sn6zMscGqzVU1UlUjixYtGnwHJB3DZC4NkSSn0Uvkn6mqzwNU1YtVdaSqfgp8CljZVt8PXNhXfCnwQosvHSd+VJkkC4GzgJemZ28kjTKZS0OiXbu+A9hTVZ/oiy/uW+19wK62/CCwtvVQv5heR7fHquoA8GqSVW2b1wMP9JVZ15avBR5p19UlTaOFs12B+WTZxi8dd519m66agZpI47oC+ADwZJLHW+wjwPuTrKDXHL4P+B2AqtqdZCvwFL2e8DdU1ZFW7oPAXcAZwMPtAb0vC/ck2UvvjHzttO6RJMBkLg2Nqvpzxr+m/dAkZW4Gbh4nvhO4dJz4T4DrTqGakk6CzeySJHWcyVySpI4bOJknWZDkL5J8sb0+J8m2JM+257P71nUISEmSZsiJnJnfSG/EqFEbge1VtRzY3l47BKQkSTNsoGSeZClwFfDpvnD/sI1bOHo4R4eAlCRphgx6Zv5fgP8I/LQvdkG735T2fH6LT9sQkA7/KEnSsY57a1qS3wAOVtW3k7x7gG1O2xCQVbUZ2AwwMjLiQBSShsYg41hoeA1yn/kVwG+2OY5fB5yZ5H8CLyZZXFUHWhP6wbb+qQwBud8hICVJOjHHbWavqpuqamlVLaPXse2RqvrXHD1s4zqOHs7RISAlSZohpzIC3CZga5L1wPdpoz45BKQkSTPrhJJ5VX0d+Hpb/iGweoL1HAJSkqQZ4ghwkiR1nMlckqSOM5lLktRxJnNJkjrOZC5JUseZzCVJ6jiTuSRJHWcylySp40zm0pBIcmGSryXZk2R3khtb/Jwk25I8257P7itzU5K9SZ5JcmVf/PIkT7b3bhudsrgN43xfi+9IsmzGd1QaQiZzaXgcBn6/qt4GrAJuSHIJsBHYXlXLge3tNe29tcDbgTXAJ5MsaNu6HdhAb+6F5e19gPXAy1X1VuBW4JaZ2DFp2JnMpSFRVQeq6jtt+VVgD7AEuBrY0lbbAlzTlq8G7q2q16rqOWAvsLLNknhmVT3aJkS6e0yZ0W3dD6wePWuXNH1M5tIQas3flwE7gAvarIa05/PbakuA5/uK7W+xJW15bPyoMlV1GHgFOHeCOmxIsjPJzkOHDk3BXknDy2QuDZkkbwA+B3y4qn402arjxGqS+GRljg1Wba6qkaoaWbRo0WRVlnQcpzIFqqSOSXIavUT+mar6fAu/mGRxVR1oTegHW3w/cGFf8aXACy2+dJx4f5n9SRYCZ9Gb1liTWLbxS7NdBXWcZ+bSkGjXru8A9lTVJ/reehBY15bXAQ/0xde2HuoX0+vo9lhrin81yaq2zevHlBnd1rXAI+26uqRp5Jm5NDyuAD4APJnk8Rb7CLAJ2JpkPfB94DqAqtqdZCvwFL2e8DdU1ZFW7oPAXcAZwMPtAb0vC/ck2UvvjHztNO+TJEzm0tCoqj9n/GvaAKsnKHMzcPM48Z3ApePEf0L7MiBp5tjMLklSx5nMJUnqOJO5JEkdZzKXJKnjTOaSJHWcyVySpI47bjJP8rokjyX5bps28Y9a3GkTJUmaAwY5M38N+JWqegewAliTZBVOmyhJ0pxw3EFj2lCMP24vT2uPojfV4btbfAvwdeAP6Js2EXiujQS1Msk+2rSJAElGp018uJX5WNvW/cB/TRKHgZSk4TDI+PT7Nl01AzXppoGumSdZ0IZ/PAhsq6pZmzZRkiQdbaBkXlVHqmoFvdmRViY5ZhjHPtM2baLzH0uSdKwT6s1eVX9Drzl9DW3aRIApnDaRyaZNdP5jSZKONUhv9kVJ3tSWzwB+FXgap02UJGlOGGTWtMXAltYj/eeArVX1xSSP4rSJkiTNukF6sz8BXDZO/Ic4baIkaRKD9FLXqXMEOEmSOs5kLklSx5nMJUnqOJO5JEkdZzKXhkiSO5McTLKrL/axJH+d5PH2eG/fe06aJHWAyVwaLnfxswmO+t1aVSva4yFw0iSpS0zm0hCpqm8wzuiKE/iHSZOq6jlgdNKkxbRJk9rgTqOTJo2W2dKW7wdWj561S5o+JnNJAB9K8kRrhj+7xaZ10iTnWpCmjslc0u3AW4AVwAHg4y0+bZMmgXMtSFPJZC4Nuap6sc2M+FPgU8DK9ta0TZokaWqZzKUhNzr7YfM+YLSnu5MmSR0xyEQrkuaJJJ8F3g2cl2Q/8FHg3UlW0GsO3wf8DjhpktQlJnNpiFTV+8cJ3zHJ+k6aJHWAzeySJHWcZ+YzbJDpAPdtumoGaiJJmi88M5ckqeNM5pIkdZzJXJKkjvOauSRNo0H6yUinyjNzSZI6zmQuSVLHmcwlSeo4k7kkSR133GSe5MIkX0uyJ8nuJDe2+DlJtiV5tj2f3VfmpiR7kzyT5Mq++OVJnmzv3dYmaaBN5HBfi+9Ismwa9lWSpHlpkDPzw8DvV9XbgFXADUkuATYC26tqObC9vaa9txZ4O7AG+GSSBW1btwMb6M2+tLy9D7AeeLmq3grcCtwyBfsmSdJQOG4yr6oDVfWdtvwqsAdYAlwNbGmrbQGuactXA/dW1WtV9RywF1jZplk8s6oebVMi3j2mzOi27gdWj561S5KkyZ3QNfPW/H0ZsAO4oM1rTHs+v622BHi+r9j+FlvSlsfGjypTVYeBV4BzT6RukiQNq4GTeZI3AJ8DPlxVP5ps1XFiNUl8sjJj67Ahyc4kOw8dOnS8KkuSNBQGSuZJTqOXyD9TVZ9v4Rdb0znt+WCL7wcu7Cu+FHihxZeOEz+qTJKFwFnAS2PrUVWbq2qkqkYWLVo0SNUlSZr3BunNHuAOYE9VfaLvrQeBdW15HfBAX3xt66F+Mb2Obo+1pvhXk6xq27x+TJnRbV0LPNKuq0uSpOMYZGz2K4APAE8mebzFPgJsArYmWQ98H7gOoKp2J9kKPEWvJ/wNVXWklfsgcBdwBvBwe0Dvy8I9SfbSOyNfe2q7JUnS8DhuMq+qP2f8a9oAqycoczNw8zjxncCl48R/QvsyIEmSTowjwElDJMmdSQ4m2dUXcwAoqeNM5tJwuYufDdY0ygGgpI4zmUtDpKq+wbF3ijgAlNRxg3SA66xlG78021WQuuCoAaCS9A8A9c2+9UYHevp7BhwAKsnoAFA/GPtDk2ygd3bPRRddNGU7Iw0jz8wlTWTaBoACx42QppLJXNKMDwAlaWqZzCU5AJTUcfP6mrmkoyX5LPBu4Lwk+4GP4gBQUueZzKUhUlXvn+AtB4CSOsxmdkmSOs4z8zlo0Fvq9m26apprIknqAs/MJUnqOJO5JEkdZzKXJKnjTOaSJHWcyVySpI4zmUuS1HEmc0mSOs5kLklSxzlojCSpEwYZUGtYB9PyzFySpI4zmUuS1HEmc0mSOs5kLklSxx03mSe5M8nBJLv6Yuck2Zbk2fZ8dt97NyXZm+SZJFf2xS9P8mR777YkafHTk9zX4juSLJvifZQkaV4b5Mz8LmDNmNhGYHtVLQe2t9ckuQRYC7y9lflkkgWtzO3ABmB5e4xucz3wclW9FbgVuOVkd0aSpGF03GReVd8AXhoTvhrY0pa3ANf0xe+tqteq6jlgL7AyyWLgzKp6tKoKuHtMmdFt3Q+sHj1rlyRJx3ey18wvqKoDAO35/BZfAjzft97+FlvSlsfGjypTVYeBV4Bzx/uhSTYk2Zlk56FDh06y6pIkzS9TPWjMeGfUNUl8sjLHBqs2A5sBRkZGxl1H0slJsg94FTgCHK6qkSTnAPcBy4B9wL+sqpfb+jfRu0x2BPi9qvpKi19O7/LcGcBDwI2tRW7eGWQQE82sYR1Y5mTPzF9sTee054Mtvh+4sG+9pcALLb50nPhRZZIsBM7i2GZ9STPjl6tqRVWNtNdT2T9G0jQ52WT+ILCuLa8DHuiLr2091C+mdyA/1priX02yql0Pv35MmdFtXQs8Ml+/xUsdNJX9YyRNk+M2syf5LPBu4Lwk+4GPApuArUnWA98HrgOoqt1JtgJPAYeBG6rqSNvUB/lZ09vD7QFwB3BPkr30zsjXTsmeSTpRBXw1SQH/vV3WOqp/TJL+/jHf7Cs72g/m75m4f8xRkmygdwbPRRddNJX7IQ2d4ybzqnr/BG+tnmD9m4Gbx4nvBC4dJ/4T2pcBSbPqiqp6oSXsbUmenmTdk+kfc3TQPjDSlHEEOEkAVNUL7fkg8AVgJVPbP0bSNDGZSyLJ65O8cXQZeA+wi6ntHyNpmjifeYcN6y0YmhYXAF9o4zUtBP60qr6c5FtMXf8YSdPEZC6Jqvoe8I5x4j9kivrHSJo+NrNLktRxJnNJkjrOZC5JUseZzCVJ6jiTuSRJHWcylySp40zmkiR1nMlckqSOc9CYec5R4iRp/vPMXJKkjjOZS5LUcTazy6Z4Seo4z8wlSeo4k7kkSR1nM7skaagMcmkRunV50TNzSZI6zjNzDWQ+fpOVpPnCM3NJkjrOM3NJGmPQlihprpgzyTzJGuCPgQXAp6tq0yxXSdJJmo7jearGQzBRaz6aE8k8yQLgvwG/BuwHvpXkwap6anZrJulEzebxbKLWsJoTyRxYCeytqu8BJLkXuBowmUvd4/GseWGqvhzORMfguZLMlwDP973eD/yzsSsl2QBsaC9/nOSZMaucB/xgWmo4/eZF3XPLLNfkxHX5c4fB6v+PZ6IifabqeJ5pXf9bGOV+zC3n5ZYp248Jj+W5kswzTqyOCVRtBjZPuJFkZ1WNTGXFZop1nx1drjvM2fpPyfE80+boZ3nC3I+5Zab2Y67cmrYfuLDv9VLghVmqi6RT4/EszbC5ksy/BSxPcnGSnwfWAg/Ocp0knRyPZ2mGzYlm9qo6nORDwFfo3cpyZ1XtPolNzZkmu5Ng3WdHl+sOc7D+U3g8z7Q591meJPdjbpmR/UjVMZeyJElSh8yVZnZJknSSTOaSJHXcvEjmSdYkeSbJ3iQbZ7ku+5I8meTxJDtb7Jwk25I8257P7lv/plbvZ5Jc2Re/vG1nb5LbkqTFT09yX4vvSLLsFOp6Z5KDSXb1xWakrknWtZ/xbJJ1U1T3jyX56/bZP57kvXO07hcm+VqSPUl2J7mxxTvx2c83SW5Msqv9Lj482/U5ESd6DM9VE+zHde138tMknbhFbYL9+M9Jnk7yRJIvJHnTtPzwqur0g14Hm78C3gz8PPBd4JJZrM8+4Lwxsf8EbGzLG4Fb2vIlrb6nAxe3/VjQ3nsM+CV69+w+DPx6i/874E/a8lrgvlOo67uAdwK7ZrKuwDnA99rz2W357Cmo+8eA/zDOunOt7ouBd7blNwJ/2erYic9+Pj2AS4FdwC/Q6xD8Z8Dy2a7XCdR/4GN4Lj8m2I+3Ab8IfB0Yme06nsJ+vAdY2JZvma7fx3w4M/+HoSOr6u+A0aEj55KrgS1teQtwTV/83qp6raqeA/YCK5MsBs6sqker9xdw95gyo9u6H1g9ejZ2oqrqG8BLs1DXK4FtVfVSVb0MbAPWTEHdJzLX6n6gqr7Tll8F9tAbNa0Tn/088zbgm1X1t1V1GPjfwPtmuU4DO8FjeM4abz+qak9VzfaogCdkgv34avvbAvgmvXEXptx8SObjDR25ZJbqAr2Rrr6a5NvpDVcJcEFVHYDeP3Lg/BafqO5L2vLY+FFl2h/IK8C5U1j/majrdP7OPtSas+7sa16cs3Vvzd+XATvo/mffRbuAdyU5N8kvAO/l6AFvumiivyPNvn9DrwVtys2HZD7Q0JEz6Iqqeifw68ANSd41yboT1X2yfZqt/Z3Kuk7XPtwOvAVYARwAPn4K9Zj2uid5A/A54MNV9aPJVj2Jusz0Z99JVbWHXtPnNuDL9C5nHJ60kHQSkvwhvb+tz0zH9udDMp9TQ0dW1Qvt+SDwBXqXAV5sTaK054Nt9Ynqvp+jm2L69+kfyiRZCJzF4M3Ng5iJuk7L76yqXqyqI1X1U+BT9D77OVn3JKfRS+SfqarPt3BnP/suq6o7quqdVfUuep/Rs7Ndp1M00d+RZknraPobwL9ql8Sm3HxI5nNm6Mgkr0/yxtFleh0fdrX6jPYaXgc80JYfBNa2nscXA8uBx1rT2KtJVrXrnNePKTO6rWuBR6b4j2Mm6voV4D1Jzm5N4e9psVMy+g+seR+9z37O1b39rDuAPVX1ib63OvvZd1mS89vzRcC/AD47uzU6ZRP9HWkWJFkD/AHwm1X1t9P2g6arV99MPuhd5/pLer18/3AW6/Fmes103wV2j9aF3rXK7fS+8W8Hzukr84et3s/QeiK3+Ai9ZPRXwH/lZ6P1vQ74X/Q6QT0GvPkU6vtZes3Rf0/vjG39TNWV3rWjve3x21NU93uAJ4En6P1DWzxH6/7P6TVtPwE83h7v7cpnP98ewP+hN9f6d4HVs12fE6z7CR3Dc/UxwX68ry2/BrwIfGW263mS+7GXXj+V0WP9T6bjZzucqyRJHTcfmtklSRpqJnNJkjrOZC5JUseZzCVJ6jiTuSRJHWcylySp40zmkiR13P8HoAwXC5lq4RQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 576x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "\n",
    "data[\"Log1pSalary\"] = np.log1p(data[\"SalaryNormalized\"]).astype(\"float32\")\n",
    "\n",
    "plt.figure(figsize=[8, 4])\n",
    "plt.subplot(1, 2, 1)\n",
    "plt.hist(data[\"SalaryNormalized\"], bins=20)\n",
    "\n",
    "plt.subplot(1, 2, 2)\n",
    "plt.hist(data[\"Log1pSalary\"], bins=20)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Fcu-qmHRycOK"
   },
   "source": [
    "Our task is to predict one number, __Log1pSalary__.\n",
    "\n",
    "To do so, our model can access a number of features:\n",
    "* Free text: __`Title`__ and  __`FullDescription`__\n",
    "* Categorical: __`Category`__, __`Company`__, __`LocationNormalized`__, __`ContractType`__, and __`ContractTime`__."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 364
    },
    "id": "p9vyA_erycOK",
    "outputId": "0fb6d7e0-2234-43d2-ce1b-3a00c4a65b04"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>Title</th>\n",
       "      <th>FullDescription</th>\n",
       "      <th>LocationRaw</th>\n",
       "      <th>LocationNormalized</th>\n",
       "      <th>ContractType</th>\n",
       "      <th>ContractTime</th>\n",
       "      <th>Company</th>\n",
       "      <th>Category</th>\n",
       "      <th>SalaryRaw</th>\n",
       "      <th>SalaryNormalized</th>\n",
       "      <th>SourceName</th>\n",
       "      <th>Log1pSalary</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>125885</th>\n",
       "      <td>69994072</td>\n",
       "      <td>Interim HR Manager</td>\n",
       "      <td>Main Accountabilities:  Partner with the local...</td>\n",
       "      <td>Slough, Berkshire Berkshire South East</td>\n",
       "      <td>Slough</td>\n",
       "      <td>NaN</td>\n",
       "      <td>contract</td>\n",
       "      <td>hyphen</td>\n",
       "      <td>HR &amp; Recruitment Jobs</td>\n",
       "      <td>45000 - 55000 per annum</td>\n",
       "      <td>50000</td>\n",
       "      <td>totaljobs.com</td>\n",
       "      <td>10.819798</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>184658</th>\n",
       "      <td>71658857</td>\n",
       "      <td>HR Manager  Regional Role</td>\n",
       "      <td>My client a leading a retailer is seeking an e...</td>\n",
       "      <td>North West England, England</td>\n",
       "      <td>North West England</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Oakleaf Partnership</td>\n",
       "      <td>HR &amp; Recruitment Jobs</td>\n",
       "      <td>40000 - 45000 per annum + Benefits + car allow...</td>\n",
       "      <td>42500</td>\n",
       "      <td>Personneltoday Jobs</td>\n",
       "      <td>10.657283</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68287</th>\n",
       "      <td>68783693</td>\n",
       "      <td>GRADUATE SALES EXECUTIVE / GRADUATE SALES TRAINEE</td>\n",
       "      <td>GRADUATE SALES EXECUTIVE / GRADUATE SALES TRAI...</td>\n",
       "      <td>Warrington, Cheshire Cheshire North West</td>\n",
       "      <td>Warrington</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>BMS Sales Specialists LLP</td>\n",
       "      <td>Sales Jobs</td>\n",
       "      <td>22000 per annum + bonus, mobile, benefits</td>\n",
       "      <td>22000</td>\n",
       "      <td>salestarget.co.uk</td>\n",
       "      <td>9.998843</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              Id                                              Title  \\\n",
       "125885  69994072                                 Interim HR Manager   \n",
       "184658  71658857                          HR Manager  Regional Role   \n",
       "68287   68783693  GRADUATE SALES EXECUTIVE / GRADUATE SALES TRAINEE   \n",
       "\n",
       "                                          FullDescription  \\\n",
       "125885  Main Accountabilities:  Partner with the local...   \n",
       "184658  My client a leading a retailer is seeking an e...   \n",
       "68287   GRADUATE SALES EXECUTIVE / GRADUATE SALES TRAI...   \n",
       "\n",
       "                                     LocationRaw  LocationNormalized  \\\n",
       "125885    Slough, Berkshire Berkshire South East              Slough   \n",
       "184658               North West England, England  North West England   \n",
       "68287   Warrington, Cheshire Cheshire North West          Warrington   \n",
       "\n",
       "       ContractType ContractTime                    Company  \\\n",
       "125885          NaN     contract                     hyphen   \n",
       "184658          NaN          NaN        Oakleaf Partnership   \n",
       "68287           NaN          NaN  BMS Sales Specialists LLP   \n",
       "\n",
       "                     Category  \\\n",
       "125885  HR & Recruitment Jobs   \n",
       "184658  HR & Recruitment Jobs   \n",
       "68287              Sales Jobs   \n",
       "\n",
       "                                                SalaryRaw  SalaryNormalized  \\\n",
       "125885                            45000 - 55000 per annum             50000   \n",
       "184658  40000 - 45000 per annum + Benefits + car allow...             42500   \n",
       "68287           22000 per annum + bonus, mobile, benefits             22000   \n",
       "\n",
       "                 SourceName  Log1pSalary  \n",
       "125885        totaljobs.com    10.819798  \n",
       "184658  Personneltoday Jobs    10.657283  \n",
       "68287     salestarget.co.uk     9.998843  "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text_columns = [\"Title\", \"FullDescription\"]\n",
    "categorical_columns = [\"Category\", \"Company\", \"LocationNormalized\", \"ContractType\", \"ContractTime\"]\n",
    "target_column = \"Log1pSalary\"\n",
    "\n",
    "data[categorical_columns] = data[categorical_columns].fillna(\"NaN\")\n",
    "data[text_columns] = data[text_columns].fillna(\"NaN\")\n",
    "\n",
    "data.sample(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IUdclucmycON"
   },
   "source": [
    "## Preprocessing text data\n",
    "\n",
    "Just like last week, applying NLP to a problem begins from tokenization: splitting raw text into sequences of tokens (words, punctuation, etc).\n",
    "\n",
    "__Your task__ is to lowercase and tokenize all texts under `Title` and `FullDescription` columns. Store the tokenized data as a __space-separated__ string of tokens for performance reasons.\n",
    "\n",
    "It's okay to use nltk tokenizers. Assertions were designed for WordPunctTokenizer, slight deviations are okay."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "YzeOxD_aycOO",
    "outputId": "cd28cfd1-7c78-4d01-e8de-47a329da0a6e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Raw text:\n",
      "2         Mathematical Modeller / Simulation Analyst / O...\n",
      "100002    A successful and high achieving specialist sch...\n",
      "200002    Web Designer  HTML, CSS, JavaScript, Photoshop...\n",
      "Name: FullDescription, dtype: object\n"
     ]
    }
   ],
   "source": [
    "print(\"Raw text:\")\n",
    "print(data[\"FullDescription\"][2::100000])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "id": "RUWkpd7PycOQ"
   },
   "outputs": [],
   "source": [
    "from nltk.tokenize import WordPunctTokenizer\n",
    "\n",
    "\n",
    "tokenizer = WordPunctTokenizer()\n",
    "\n",
    "def normalize(text):\n",
    "    tokens = tokenizer.tokenize(text.lower())\n",
    "    return ' '.join(tokens)\n",
    "# Lowercase and tokenize titles and descriptions (text_columns).\n",
    "# Store items as space-separated strings of tokens.\n",
    "data[text_columns] = data[text_columns].applymap(normalize)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Gs-6lnS_ycOU",
    "outputId": "3f85acf0-44aa-4ae4-b593-92f78d607592"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tokenized:\n",
      "2         mathematical modeller / simulation analyst / o...\n",
      "100002    a successful and high achieving specialist sch...\n",
      "200002    web designer html , css , javascript , photosh...\n",
      "Name: FullDescription, dtype: object\n"
     ]
    }
   ],
   "source": [
    "print(\"Tokenized:\")\n",
    "print(data[\"FullDescription\"][2::100000])\n",
    "assert data[\"FullDescription\"][2][:50] == \"mathematical modeller / simulation analyst / opera\"\n",
    "assert data[\"Title\"][54321] == \"international digital account manager ( german )\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ouE3L2hyycOX"
   },
   "source": [
    "Not all words are equally useful. Some of them are typos or rare words that are only present a few times. \n",
    "\n",
    "Let's count how many times is each word present in the data so that we can build a \"white list\" of known words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "id": "iC7hBwwjycOX"
   },
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "from tqdm.auto import tqdm\n",
    "\n",
    "\n",
    "\n",
    "# Count how many times does each token occur in both \"Title\" and \"FullDescription\".\n",
    "# Build a dictionary { token -> count }.\n",
    "# Hint: you may or may not want to use collections.Counter\n",
    "token_counts = Counter()\n",
    "for row in data[text_columns].values.flatten():\n",
    "    token_counts.update(row.split(' '))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "GiOWbc15ycOb",
    "outputId": "e5acee26-a871-4868-861a-8675fcf9e055"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total unique tokens : 202704\n",
      "('and', 2657388)\n",
      "('.', 2523216)\n",
      "(',', 2318606)\n",
      "('the', 2080994)\n",
      "('to', 2019884)\n",
      "...\n",
      "('stephanietraveltraderecruitmnt', 1)\n",
      "('ruabon', 1)\n",
      "('lowehays', 1)\n",
      "Correct!\n"
     ]
    }
   ],
   "source": [
    "print(\"Total unique tokens :\", len(token_counts))\n",
    "print(\"\\n\".join(map(str, token_counts.most_common(n=5))))\n",
    "print(\"...\")\n",
    "print(\"\\n\".join(map(str, token_counts.most_common()[-3:])))\n",
    "\n",
    "assert token_counts.most_common(1)[0][1] in range(2600000, 2700000)\n",
    "assert len(token_counts) in range(200000, 210000)\n",
    "print(\"Correct!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "W2ih1LV4YhtL"
   },
   "source": [
    "Let's see how many words are there for each count."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 279
    },
    "id": "nd5v3BNfycOf",
    "outputId": "ffeb9998-f983-4d0d-fb8b-2d58f5f88ded"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEGCAYAAACevtWaAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAARuElEQVR4nO3dfaxlVX3G8e/jUKBiO4BQQ4FxBoegU00Eb0CsbahQHZSRxrYJY60voUzUYGttUiE2rf7RBKt9wZRIR6RYbUGkRBkYg40tRREVqMqLMDoilitWQJqpNfVl9Nc/zh48udyXc+85d849634/yWTOXufsvdeagWf2/e111k5VIUlqy5PG3QFJ0ugZ7pLUIMNdkhpkuEtSgwx3SWrQAePuAMARRxxR69evH3c3JGmi3HHHHY9W1ZGzvbciwn39+vXcfvvt4+6GJE2UJN+Y672xlmWSbEmyfc+ePePshiQ1Z6zhXlU7qmrb2rVrx9kNSWqON1QlqUGGuyQ1yHCXpAYZ7pLUIMNdkhpkuEtSg8b6JaYkW4AtGzduXPIx1l9ww6ztD1z0siUfU5ImnfPcJalBlmUkqUGGuyQ1yHCXpAYZ7pLUIMNdkhpkuEtSgwx3SWqQ4S5JDRp5uCc5Lcmnklya5LRRH1+StLCBwj3J5UkeTnL3jPbNSXYl2Z3kgq65gP8FDgamR9tdSdIgBr1yvwLY3N+QZA1wCXAmsAnYmmQT8KmqOhN4K/CO0XVVkjSogcK9qm4GHpvRfDKwu6rur6ofAlcBZ1fVT7r3/xs4aGQ9lSQNbJhVIY8GHuzbngZOSfIK4CXAocDfzrVzkm3ANoB169YN0Q1J0kzDhHtmaauquha4dqGdq2o7sB1gamqqhuiHJGmGYWbLTAPH9m0fAzy0mAMk2ZJk+549e4bohiRppmHC/Tbg+CQbkhwInANct5gDuJ67JC2PQadCXgncCpyQZDrJuVW1FzgfuBG4F7i6qu5ZzMm9cpek5TFQzb2qts7RvhPYudSTV9UOYMfU1NR5Sz2GJOmJXH5Akho01nC3LCNJy8MHZEtSgyzLSFKDLMtIUoMsy0hSgyzLSFKDLMtIUoMsy0hSgyzLSFKDDHdJapDhLkkN8oaqJDXIG6qS1CDLMpLUIMNdkhpkuEtSgwx3SWqQs2UkqUHOlpGkBlmWkaQGGe6S1CDDXZIaZLhLUoMMd0lqkOEuSQ1ynrskNch57pLUIMsyktQgw12SGmS4S1KDDHdJapDhLkkNMtwlqUGGuyQ1yHCXpAYtS7gnOSTJHUnOWo7jS5LmN1C4J7k8ycNJ7p7RvjnJriS7k1zQ99ZbgatH2VFJ0uAGvXK/Atjc35BkDXAJcCawCdiaZFOSM4AvA98eYT8lSYtwwCAfqqqbk6yf0XwysLuq7gdIchVwNvAU4BB6gf9/SXZW1U9G12VJ0kIGCvc5HA082Lc9DZxSVecDJHkt8OhcwZ5kG7ANYN26dUN0Q5I00zA3VDNLWz3+ouqKqrp+rp2rantVTVXV1JFHHjlENyRJMw0T7tPAsX3bxwAPLeYArucuSctjmHC/DTg+yYYkBwLnANct5gCu5y5Jy2PQqZBXArcCJySZTnJuVe0FzgduBO4Frq6qexZzcq/cJWl5DDpbZusc7TuBnUs9eVXtAHZMTU2dt9RjSJKeyOUHJKlBPiBbkhrkA7IlqUGWZSSpQZZlJKlBlmUkqUGWZSSpQYa7JDXImrskNciauyQ1yLKMJDXIcJekBllzl6QGWXOXpAZZlpGkBhnuktQgw12SGmS4S1KDnC0jSQ1ytowkNciyjCQ1yHCXpAYZ7pLUoAPG3YHlsv6CG2Ztf+Cil+3nnkjS/ueVuyQ1yHCXpAY5z12SGuQ8d0lqkGUZSWqQ4S5JDTLcJalBhrskNchwl6QGGe6S1CDDXZIaZLhLUoNGHu5JnpXk0iTXJHnDqI8vSVrYQKtCJrkcOAt4uKqe3de+GbgYWANcVlUXVdW9wOuTPAl43zL0eShzrRYJrhgpqR2DXrlfAWzub0iyBrgEOBPYBGxNsql77+XAp4FPjqynkqSBDRTuVXUz8NiM5pOB3VV1f1X9ELgKOLv7/HVV9QLgd0bZWUnSYIZ5WMfRwIN929PAKUlOA14BHATsnGvnJNuAbQDr1q0bohuSpJmGCffM0lZVdRNw00I7V9V2YDvA1NRUDdEPSdIMw8yWmQaO7ds+BnhoMQdwPXdJWh7DhPttwPFJNiQ5EDgHuG4xB3A9d0laHgOFe5IrgVuBE5JMJzm3qvYC5wM3AvcCV1fVPYs5uVfukrQ8Bqq5V9XWOdp3Ms9N0wGOuwPYMTU1dd5SjyFJeiKXH5CkBg0zW2ZoSbYAWzZu3DjObjxurm+v+s1VSZPGB2RLUoMsy0hSg8Ya7s6WkaTlYVlGkhpkWUaSGjTW2TKTwlk0kiaNNXdJapA1d0lqkDV3SWqQ4S5JDbLmLkkNsuYuSQ2yLCNJDXKe+xCc/y5ppfLKXZIaZLhLUoOcLSNJDXK2jCQ1yBuqy8AbrZLGzZq7JDXIcJekBhnuktQga+77kbV4SfuLV+6S1CDnuUtSg8ZalqmqHcCOqamp88bZj3GzXCNp1CzLSFKDDHdJapCzZVYwyzWSlsord0lqkOEuSQ2yLDOBLNdIWohX7pLUIK/cG+IVvaR9vHKXpAYtS7gn+Y0k70vysSQvXo5zSJLmNnBZJsnlwFnAw1X17L72zcDFwBrgsqq6qKo+Cnw0yWHAu4FPjLTXWpS5yjVzsYwjTb7FXLlfAWzub0iyBrgEOBPYBGxNsqnvI3/SvS9J2o8GDvequhl4bEbzycDuqrq/qn4IXAWcnZ53Ah+vqv+Y7XhJtiW5PcntjzzyyFL7L0maxbCzZY4GHuzbngZOAd4EnAGsTbKxqi6duWNVbQe2A0xNTdWQ/dAIzVfGsWQjTYZhwz2ztFVVvQd4z4I7J1uALRs3bhyyG5KkfsPOlpkGju3bPgZ4aNCdq2pHVW1bu3btkN2QJPUb9sr9NuD4JBuAbwLnAK8cdGev3CePX5SSJsNipkJeCZwGHJFkGvizqnp/kvOBG+lNhby8qu4Z9Jg+iakdhr60sgwc7lW1dY72ncDOkfVIkjQ0H5AtSQ3yAdkai8WWcSz7SIvjqpBaVotd+mCxn5c0O8syktSgsYa789wlaXm4nrskNchwl6QGjfWGqt9Q1bCcRSPNzqmQatI4Q9+Ho2glcCqkxNKmYBrKWsmsuUtSg6y5SyuU9xM0DGvu0hKttG/T+o+B+lmWkaQGGe6S1CDDXZIaZLhLUoOcLaNVZaXdBJWWi7NlpAmzGv+BcibQ4vkNValxBuPqZM1dkhpkuEtSgyzLSGNmDV3LwXCXVqlRLU0833Gs64+P4S5p2YzrCt2byGOuuSfZkmT7nj17xtkNSWrOWMO9qnZU1ba1a9eOsxuS1BzLMpIG0sJN0NVUrjHcJa16LYa+89wlqUGGuyQ1yLKMJC3SJJRxvHKXpAYZ7pLUIMsykiZWC9Mzl4vhLklzGOU/Hvu7Tj/yskyS45K8P8k1oz62JGkwA4V7ksuTPJzk7hntm5PsSrI7yQUAVXV/VZ27HJ2VJA1m0Cv3K4DN/Q1J1gCXAGcCm4CtSTaNtHeSpCUZqOZeVTcnWT+j+WRgd1XdD5DkKuBs4MuDHDPJNmAbwLp16wbtryStWCvpBu8wNfejgQf7tqeBo5M8NcmlwIlJLpxr56raXlVTVTV15JFHDtENSdJMw8yWySxtVVXfAV4/0AGSLcCWjRs3DtENSdJMw1y5TwPH9m0fAzy0mAO4nrskLY9hwv024PgkG5IcCJwDXLeYA/gkJklaHoNOhbwSuBU4Icl0knOrai9wPnAjcC9wdVXds5iTe+UuSctj0NkyW+do3wnsHGmPJElD8wHZktQgH5AtSQ1yyV9JalCqatx9IMkjwDeWuPsRwKMj7M4kcMyrg2NeHYYZ89OratZvga6IcB9Gkturamrc/difHPPq4JhXh+Uas2UZSWqQ4S5JDWoh3LePuwNj4JhXB8e8OizLmCe+5i5JeqIWrtwlSTMY7pLUoIkO99me4TqJkhyb5N+S3JvkniR/0LUfnuRfkny1+/2wvn0u7Ma9K8lL+tqfl+Su7r33JJlt3f0VI8maJF9Icn233fSYkxya5Jok93V/36eugjH/Yfff9d1JrkxycGtjnu0506McY5KDkny4a/9cnvhkvCeqqon8BawBvgYcBxwIfAnYNO5+LXEsRwEnda9/DvgKvefS/gVwQdd+AfDO7vWmbrwHARu6P4c13XufB06l9zCVjwNnjnt8C4z9LcA/Add3202PGfgA8Hvd6wOBQ1seM70ntn0d+Nlu+2rgta2NGfhV4CTg7r62kY0ReCNwaff6HODDC/Zp3H8oQ/xhngrc2Ld9IXDhuPs1orF9DPh1YBdwVNd2FLBrtrHSW3b51O4z9/W1bwX+btzjmWecxwCfBF7ET8O92TEDP98FXWa0tzzmfY/jPJzeKrTXAy9ucczA+hnhPrIx7vtM9/oAet9ozXz9meSyzKzPcB1TX0am+3HrROBzwNOq6lsA3e+/0H1srrEf3b2e2b5S/Q3wx8BP+tpaHvNxwCPA33elqMuSHELDY66qbwLvBv4T+Bawp6o+QcNj7jPKMT6+T/WepbEHeOp8J5/kcJ/1Ga77vRcjlOQpwD8Db66q/5nvo7O01TztK06Ss4CHq+qOQXeZpW2ixkzviusk4L1VdSLwPXo/rs9l4sfc1ZnPpld++EXgkCSvmm+XWdomaswDWMoYFz3+SQ73oZ/hupIk+Rl6wf6PVXVt1/ztJEd17x8FPNy1zzX26e71zPaV6JeBlyd5ALgKeFGSD9H2mKeB6ar6XLd9Db2wb3nMZwBfr6pHqupHwLXAC2h7zPuMcoyP75PkAGAt8Nh8J5/kcB/6Ga4rRXdH/P3AvVX1V31vXQe8pnv9Gnq1+H3t53R30DcAxwOf7370+26S53fHfHXfPitKVV1YVcdU1Xp6f3f/WlWvou0x/xfwYJITuqbTgS/T8JjplWOen+TJXV9Pp/dYzpbHvM8ox9h/rN+i9//L/D+5jPsmxJA3MF5Kb2bJ14C3jbs/Q4zjhfR+xLoT+GL366X0amqfBL7a/X543z5v68a9i75ZA8AUcHf33t+ywE2XlfALOI2f3lBteszAc4Hbu7/rjwKHrYIxvwO4r+vvB+nNEmlqzMCV9O4p/IjeVfa5oxwjcDDwEWA3vRk1xy3UJ5cfkKQGTXJZRpI0B8NdkhpkuEtSgwx3SWqQ4S5JDTLcteIl+eskb+7bvjHJZX3bf5nkLUs89mnpVqTcn7rVId+4v8+r1cNw1yT4DL1vNZLkScARwC/1vf8C4JZBDpRkzch7tzSH0lvpT1oWhrsmwS104U4v1O+m902+w5IcBDwL+EKS07sFue7q1tc+CCDJA0n+NMmngd9O7zkA93Xbr5jthOmtM//u7lh3JnlT1z7fOY7oXk8lual7/fbuczcluT/J73enuAh4RpIvJnlXkqOS3Nxt353kV5bhz1GryAHj7oC0kKp6KMneJOvohfyt9FbJO5Xe6nh30rtQuQI4vaq+kuQfgDfQW3kS4PtV9cIkB9P7xuCL6H3b78NznHYbvcWuTqyqvek9eOHgBc4xl2cCv0Zvrf5dSd5Lb8GwZ1fVcwGS/BG9Jaz/vPvp4smD/vlIs/HKXZNi39X7vnC/tW/7M8AJ9Bao+kr3+Q/Qe4DCPvtC/Jnd575ava9nf2iO851B7+EIewGq6rEBzjGXG6rqB1X1KL3Fo542y2duA16X5O3Ac6rquwMcV5qT4a5Jsa/u/hx6ZZnP0rty31dvX+iRa9/rez3ImhuZ5XPznWMvP/3/6eAZ7/2g7/WPmeUn5qq6md4/FN8EPpjk1QP0UZqT4a5JcQtwFvBYVf24u5I+lF7A30pvYar1STZ2n/9d4N9nOc59wIYkz+i2t85xvk8Ar++WVyXJ4Quc4wHged3r3xxgPN+lV6ahO/7T6a1v/z56K4SeNMAxpDkZ7poUd9GbJfPZGW17qurRqvo+8DrgI0nuovd0p0tnHqT73Dbghu6G6jfmON9l9JarvTPJl4BXLnCOdwAXJ/kUvavzeVXVd4Bbupun76K3MuYXk3yB3j8OFy90DGk+rgopSQ3yyl2SGmS4S1KDDHdJapDhLkkNMtwlqUGGuyQ1yHCXpAb9P1cCDZoxbT8SAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.hist(list(token_counts.values()), range=[0, 10 ** 4], bins=50, log=True)\n",
    "plt.xlabel(\"Word counts\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "znuXxeghycOh"
   },
   "source": [
    "Now we will filter tokens a list of all tokens that occur at least 10 times."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "id": "SeNFBWx5ycOh"
   },
   "outputs": [],
   "source": [
    "min_count = 10\n",
    "\n",
    "# We shall only consider tokens that had at least min_count occurences.\n",
    "# Create a list of such tokens.\n",
    "tokens = [token for token, counts in token_counts.items() if counts >= min_count]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "j62zdZXw4Lqe",
    "outputId": "6664ab5f-7efe-44be-c2b3-ada9e5187ac7"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "34156"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "id": "RATIRyPKycOk",
    "outputId": "c497897c-0fbe-42e1-a83c-5bec4d8e9856"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vocabulary size: 34158\n",
      "Correct!\n"
     ]
    }
   ],
   "source": [
    "# Add a special tokens for unknown and empty words\n",
    "UNK, PAD = \"UNK\", \"PAD\"\n",
    "tokens = [UNK, PAD] + sorted(tokens)\n",
    "print(\"Vocabulary size:\", len(tokens))\n",
    "\n",
    "assert type(tokens) == list\n",
    "assert len(tokens) in range(32000, 35000)\n",
    "assert \"me\" in tokens\n",
    "assert UNK in tokens\n",
    "print(\"Correct!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cqEsgbjZycOo"
   },
   "source": [
    "Build an inverse token index: a dictionary from token to it's index in `tokens`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "id": "L60lo1l_ycOq"
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "token_to_idx = {token: idx for idx, token in enumerate(tokens)}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "id": "DeAoVo4mycOr",
    "outputId": "e6af3e99-a933-4354-d8fb-0464bee19f38"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Correct!\n"
     ]
    }
   ],
   "source": [
    "assert isinstance(token_to_idx, dict)\n",
    "assert len(token_to_idx) == len(tokens)\n",
    "for tok in tokens:\n",
    "    assert tokens[token_to_idx[tok]] == tok\n",
    "\n",
    "print(\"Correct!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cmJAkq3gycOv"
   },
   "source": [
    "And finally, let's use the vocabulary you've built to map text lines into neural network-digestible matrices."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "id": "JEsLeBjVycOw"
   },
   "outputs": [],
   "source": [
    "UNK_IDX = token_to_idx[UNK]\n",
    "PAD_IDX = token_to_idx[PAD]\n",
    "\n",
    "\n",
    "def as_matrix(sequences, max_len=None):\n",
    "    \"\"\"Convert a list of tokens into a matrix with padding\"\"\"\n",
    "    if isinstance(sequences[0], str):\n",
    "        sequences = [seq.split(\" \") for seq in sequences]\n",
    "\n",
    "    sequences_max_len = max(len(seq) for seq in sequences)\n",
    "    if max_len is None:\n",
    "        max_len = sequences_max_len\n",
    "    else:\n",
    "        max_len = min(sequences_max_len, max_len)\n",
    "\n",
    "    matrix = np.full((len(sequences), max_len), PAD_IDX)\n",
    "    for i, seq in enumerate(sequences):\n",
    "        for j, token in enumerate(seq):\n",
    "            if j >= max_len:\n",
    "                break\n",
    "\n",
    "            matrix[i, j] = token_to_idx.get(token, UNK_IDX)\n",
    "\n",
    "    return matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "id": "JiBlPkdKycOy",
    "outputId": "6b8f8bf2-ab3b-4ba1-8f3b-349a7d7b5605"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Lines:\n",
      "engineering systems analyst\n",
      "hr assistant\n",
      "senior ec & i engineer\n",
      "\n",
      "Matrix:\n",
      "[[10807 30161  2166     1     1]\n",
      " [15020  2844     1     1     1]\n",
      " [27645 10201    16 15215 10804]]\n"
     ]
    }
   ],
   "source": [
    "print(\"Lines:\")\n",
    "print(\"\\n\".join(data[\"Title\"][::100000].values), end=\"\\n\\n\")\n",
    "print(\"Matrix:\")\n",
    "print(as_matrix(data[\"Title\"][::100000]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nGOdZ3-dycO4"
   },
   "source": [
    "Now let's  encode the categirical data we have.\n",
    "\n",
    "As usual, we shall use one-hot encoding for simplicity. Kudos if you implement more advanced encodings: tf-idf, pseudo-time-series, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "id": "pg12nLQijNJg"
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction import DictVectorizer\n",
    "\n",
    "\n",
    "# We only consider top-1k most frequent companies to minimize memory usage\n",
    "company_counts = Counter(data[\"Company\"])\n",
    "top_companies = set(name for name, count in company_counts.most_common(1000))\n",
    "data[\"Company\"] = data[\"Company\"].apply(\n",
    "    lambda company: company if company in top_companies else \"Other\"\n",
    ")\n",
    "\n",
    "categorical_vectorizer = DictVectorizer(dtype=np.float32, sparse=False)\n",
    "categorical_vectorizer.fit(data[categorical_columns].apply(dict, axis=1));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "yk4jmtAYycO8"
   },
   "source": [
    "## The deep learning part\n",
    "\n",
    "Once we've learned to tokenize the data, let's design a machine learning experiment.\n",
    "\n",
    "As before, we won't focus too much on validation, opting for a simple train-test split.\n",
    "\n",
    "__To be completely rigorous,__ we've comitted a small crime here: we used the whole data for tokenization and vocabulary building. A more strict way would be to do that part on training set only. You may want to do that and measure the magnitude of changes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "id": "TngLcWA0ycO_",
    "outputId": "ab393a5b-97b2-4de0-97ac-bdd0980f2343"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train size =  195814\n",
      "Validation size =  48954\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "\n",
    "data_train, data_val = train_test_split(data, test_size=0.2, random_state=42)\n",
    "data_train.index = range(len(data_train))\n",
    "data_val.index = range(len(data_val))\n",
    "\n",
    "print(\"Train size = \", len(data_train))\n",
    "print(\"Validation size = \", len(data_val))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "id": "2PXuKgOSycPB"
   },
   "outputs": [],
   "source": [
    "def make_batch(data, max_len=None, word_dropout=0):\n",
    "    \"\"\"\n",
    "    Creates a neural-network-friendly dict from the batch data.\n",
    "    :param word_dropout: replaces token index with UNK_IDX with this probability\n",
    "    :returns: a dict with {'title' : int64[batch, title_max_len]\n",
    "    \"\"\"\n",
    "    batch = {}\n",
    "    batch[\"Title\"] = as_matrix(data[\"Title\"].values, max_len)\n",
    "    batch[\"FullDescription\"] = as_matrix(data[\"FullDescription\"].values, max_len)\n",
    "    batch[\"Categorical\"] = categorical_vectorizer.transform(\n",
    "        data[categorical_columns].apply(dict, axis=1)\n",
    "    )\n",
    "\n",
    "    if word_dropout != 0:\n",
    "        batch[\"FullDescription\"] = apply_word_dropout(batch[\"FullDescription\"], 1.0 - word_dropout)\n",
    "\n",
    "    if target_column in data.columns:\n",
    "        batch[target_column] = data[target_column].values\n",
    "\n",
    "    return batch\n",
    "\n",
    "\n",
    "def apply_word_dropout(matrix, keep_prob, replace_with=UNK_IDX):\n",
    "    dropout_mask = np.random.choice(2, np.shape(matrix), p=[keep_prob, 1 - keep_prob])\n",
    "    dropout_mask &= matrix != PAD_IDX\n",
    "    return np.choose(dropout_mask, [matrix, np.full_like(matrix, replace_with)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "id": "I6LpEQf0ycPD",
    "outputId": "0bf066fd-7201-4136-d6a7-59e6747ce6b6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Example batch:\n",
      "{'Title': array([[27645, 29893, 33674,     1,     1,     1,     1],\n",
      "       [29239,   197, 19175, 20042, 15554, 23162,  4051],\n",
      "       [10609, 30412, 17746,    33,  8705, 29157,    65]]), 'FullDescription': array([[27645, 29893, 33674, 32939,   982, 27645, 29893, 33674, 16451,\n",
      "        32939],\n",
      "       [29239,   197, 19175, 20042, 15554, 23162,  4051, 25511,   907,\n",
      "           82],\n",
      "       [30746, 21956, 20601,  6409, 16451,  8165, 27493,   982, 30412,\n",
      "        17746]]), 'Categorical': array([[0., 0., 0., ..., 0., 0., 0.],\n",
      "       [0., 0., 0., ..., 0., 0., 0.],\n",
      "       [0., 0., 0., ..., 0., 0., 0.]], dtype=float32), 'Log1pSalary': array([ 9.71154 , 10.463132, 10.71444 ], dtype=float32)}\n",
      "FullDescription shape: (3, 10)\n"
     ]
    }
   ],
   "source": [
    "example_batch = make_batch(data_train[:3], max_len=10)\n",
    "print(\"Example batch:\")\n",
    "print(example_batch)\n",
    "print(\"FullDescription shape:\", example_batch[\"FullDescription\"].shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "id": "nBfN182zpHfd"
   },
   "outputs": [],
   "source": [
    "def iterate_minibatches(data, batch_size=256, shuffle=True, cycle=False, **kwargs):\n",
    "    \"\"\"iterates minibatches of data in random order\"\"\"\n",
    "    while True:\n",
    "        indices = np.arange(len(data))\n",
    "        if shuffle:\n",
    "            indices = np.random.permutation(indices)\n",
    "\n",
    "        for start in range(0, len(indices), batch_size):\n",
    "            batch = make_batch(data.iloc[indices[start : start + batch_size]], **kwargs)\n",
    "            target = batch.pop(target_column)\n",
    "            yield batch, target\n",
    "\n",
    "        if not cycle:\n",
    "            break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "iQAQPx3-pHfd"
   },
   "source": [
    "To generate minibatches we will use simple pyton generator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "id": "fCeb-W9EodjZ"
   },
   "outputs": [],
   "source": [
    "iterator = iterate_minibatches(data_train, 3)\n",
    "batch, target = next(iterator)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0eI5h9UMycPF"
   },
   "source": [
    "## Architecture\n",
    "\n",
    "Our main model consists of three branches:\n",
    "* Title encoder\n",
    "* Description encoder\n",
    "* Categorical features encoder\n",
    "\n",
    "We will then feed all 3 branches into one common network that predicts salary.\n",
    "\n",
    "<img src=\"https://github.com/yandexdataschool/nlp_course/raw/master/resources/w2_conv_arch.png\" width=600px>\n",
    "\n",
    "This clearly doesn't fit into PyTorch __Sequential__ interface. To build such a network, one will have to use [__PyTorch nn.Module API__](https://pytorch.org/docs/stable/nn.html#torch.nn.Module)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SbhrCEl1pHfc"
   },
   "source": [
    "But to start with let's build the simple model using only the part of the data. Let's create the baseline solution using only the description part (so it should definetely fit into the Sequential model)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "id": "_0Ez0xZb4Lqj"
   },
   "outputs": [],
   "source": [
    "import torch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "id": "2jjPFvWZpHfc"
   },
   "outputs": [],
   "source": [
    "import torch.nn as nn\n",
    "\n",
    "class Flatten(nn.Module):\n",
    "    def forward(self, input):\n",
    "        return input.view(input.size(0), -1)\n",
    "\n",
    "# Initially, our FullDescription has a shape [batch_size, seq_len].\n",
    "# After an Embedding layer shape will be [batch_size, seq_len, embedding_size].\n",
    "# However, Conv1d layer expects batches of shape [batch_size, embedding_size, seq_len].\n",
    "# We will use this layer to fix this misunderstanding.\n",
    "class Reorder(nn.Module):\n",
    "    def forward(self, input):\n",
    "        return input.permute((0, 2, 1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "id": "6LPx2UKepHfd"
   },
   "outputs": [],
   "source": [
    "n_tokens = len(token_to_idx)\n",
    "n_cat_features = len(categorical_vectorizer.vocabulary_)\n",
    "hid_size = 64\n",
    "n_maximums = 2\n",
    "\n",
    "simple_model = nn.Sequential(\n",
    "    nn.Embedding(num_embeddings=n_tokens, embedding_dim=hid_size),\n",
    "    Reorder(),\n",
    "    \n",
    "    nn.Conv1d(in_channels=hid_size,\n",
    "              out_channels=2*hid_size,\n",
    "              kernel_size=3),\n",
    "    nn.ReLU(),\n",
    "    \n",
    "    nn.Conv1d(in_channels=2*hid_size,\n",
    "              out_channels=2*hid_size,\n",
    "              kernel_size=3),\n",
    "    nn.ReLU(),\n",
    "    \n",
    "    nn.BatchNorm1d(hid_size*2),\n",
    "    \n",
    "    nn.AdaptiveMaxPool1d(n_maximums),\n",
    "    nn.Flatten(),\n",
    "    nn.Linear(hid_size*2 * n_maximums, 1)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WYU3d7Ci4Lqj"
   },
   "source": [
    "##### Debug start"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "id": "PiywkZNJ4Lqj"
   },
   "outputs": [],
   "source": [
    "example_emb = nn.Embedding(num_embeddings=n_tokens, embedding_dim=hid_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "id": "tbHrDMlw4Lqk",
    "outputId": "a0b54e93-bdd7-44ab-de0f-e93d7b306d39"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3, 362)"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch['FullDescription'].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Fc5jPZlz4Lqk"
   },
   "outputs": [],
   "source": [
    "b = simple_model(torch.tensor(batch['FullDescription'], dtype=torch.long))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Oly5HsRY4Lqk",
    "outputId": "87ec357d-4eba-4c82-849f-e13cc3b9651e"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([3, 1])"
      ]
     },
     "execution_count": 143,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "m2smsO3d4Lqk"
   },
   "source": [
    "##### Degub end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uMu66aNSpHfd"
   },
   "source": [
    "__Remember!__ We are working with regression problem and predicting only one number."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "CerptbpcpHfd",
    "outputId": "af2f38b5-010e-4425-8a2f-88904f2c2309"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-1.6327],\n",
       "        [-0.2792],\n",
       "        [ 2.8286]], grad_fn=<AddmmBackward0>)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check your model. `torch.long` tensors are required for nn.Embedding layers.\n",
    "simple_model(torch.tensor(batch[\"FullDescription\"], dtype=torch.long))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "id": "1G-n-458pHfd"
   },
   "outputs": [],
   "source": [
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8hUa3gBepHfd"
   },
   "source": [
    "And now simple training pipeline:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "id": "efIqs2iL4i0a"
   },
   "outputs": [],
   "source": [
    "from IPython.display import clear_output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "tKiy4otdpHfd",
    "outputId": "d556c3b3-98bd-45ed-8cb6-20fd659a6a08"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD4CAYAAAAXUaZHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAABFq0lEQVR4nO2deZwU1dX+n9vLzDAb27CDDgiCgIgIuICouOGCRk1cXo1oXN4kLkn8RYPRRKOJGk00iTEmxj3uGl83DAYUxRXZEWTfh21m2GaYYZbuur8/um71rapb3VW9d8/5vp+8dFdXV93usZ869Zxzz2WccxAEQRCFhS/bAyAIgiBSD4k7QRBEAULiThAEUYCQuBMEQRQgJO4EQRAFSCDbAwCAqqoqXl1dne1hEARB5BULFy6s55z3UL2WE+JeXV2NBQsWZHsYBEEQeQVjbLPTa2TLEARBFCAk7gRBEAUIiTtBEEQBkhOeO0EQRLK0t7ejpqYGLS0t2R5KyikpKUH//v0RDAZdvyer4s4Ymwpg6uDBg7M5DIIgCoCamhpUVFSguroajLFsDydlcM6xe/du1NTUYODAga7fl1VbhnP+Luf8+s6dO2dzGARBFAAtLS3o3r17QQk7ADDG0L17d893JOS5EwRRMBSasAsS+Vx5Le5frKvHw/9dne1hEARB5Bx5Le5fbdyDv3y0DtSTniCIXKC8vDzbQzDIa3EP+CK3KhppO0EQhIm8Fne/Lu4hTcvySAiCIKJwznHrrbdi5MiROPLII/Hqq68CAHbs2IFJkyZh9OjRGDlyJD799FOEw2FcddVVxr6PPPJISsaQ13XuPj3JQNpOEITMb95dgW+3N6T0mMP7VuKuqSNc7fvmm29iyZIlWLp0Kerr6zFu3DhMmjQJL730Es4880zccccdCIfDaG5uxpIlS7Bt2zYsX74cALBv376UjDerkTtjbCpj7In9+/cn9H6/Pvowee4EQeQQn332GS677DL4/X706tULJ510EubPn49x48bhmWeewd13341vvvkGFRUVGDRoEDZs2ICbbroJM2fORGVlZUrGkNXInXP+LoB3x44de10i7/f7IuoeDpO4EwQRxW2EnS6cijwmTZqEuXPnYsaMGfj+97+PW2+9FVdeeSWWLl2KDz74AI899hhee+01PP3000mPIb89d730kyJ3giByiUmTJuHVV19FOBxGXV0d5s6di/Hjx2Pz5s3o2bMnrrvuOlxzzTVYtGgR6uvroWkaLrroItx7771YtGhRSsaQ1567SKiGqVyGIIgc4oILLsCXX36Jo446CowxPPjgg+jduzeee+45PPTQQwgGgygvL8fzzz+Pbdu24eqrr4amJw/vv//+lIwhz8U9cuOhUeROEEQOcODAAQCRGaUPPfQQHnroIdPr06ZNw7Rp02zvS1W0LpPftow++hBF7gRBECbyWtyjpZAk7gRBEDJ5Le7kuRMEIVOorUgS+VyFIe4F+gclCMI9JSUl2L17d8EJvOjnXlJS4ul9eZ5QpcidIIgI/fv3R01NDerq6rI9lJQjVmLyQl6vxORnJO4EQUQIBoOeVioqdPJ6JSYfRe4EQRBK8tpzD5C4EwRBKMlrcfdRQpUgCEJJXou7n+rcCYIglOS1uJMtQxAEoSavxZ0SqgRBEGryWtxpEhNBEISavBZ3H9W5EwRBKMlrcReeO7X8JQiCMJPX4i5smRAts0cQBGEir8XdaPlLkTtBEISJvBb3aOOwLA+EIAgixygIcQ9ppO4EQRAyWRV3xthUxtgT+/fvT+j9/g6cUK1tbMGCTXuyPQyCIHKUvO4KGW35m8pR5QdTH/0M3/37l9keBkEQOUp+2zL+jttbZldDa7aHQBBEDpPf4s6E597xxJ0gCCIWeS3uPn301H6AIAjCTF6LO7X8JQiCUJPX4h7QQ3fqLUMQBGEmr8XdsGVI3AmCIEzktbhTy1+CIAg1eS3u1PKXIAhCTV6Lu9HytwOLO6e7FoIgFOS1uEd7y3RcgSNtJwhCRV6LO2MMjHXM3jKCjvvJCYKIRV6LOxCxZnLdc1+6dR8u/vuXaA2FU35ssmUIglCR9+LuY7kv7ne89Q2+3rQHa3YeSPmxc/yjEwSRJfJe3P15ELmnE07GDEEQCvJf3Bnr0HXuHfijEwQRg/wXdz/L+VLIdAowiTtBECryX9wZy5tSSH3OVUohW4YgCBV5L+4+H+vYpZAd96MTBBGDvBd3fx5Uy6STjvvJCYKIRcrFnTE2iDH2FGPsjVQfW0WkWiYTZ0qcdEbXHfmuhSAIZ1yJO2PsacZYLWNsuWX7FMbYasbYOsbYdADgnG/gnF+TjsGqiIh7jqt7GiFtJwhChdvI/VkAU+QNjDE/gMcAnAVgOIDLGGPDUzo6F/h9DOGOLHAd+bMTBOGIK3HnnM8FsMeyeTyAdXqk3gbgFQDnuz0xY+x6xtgCxtiCuro61wO24mMduysk2TIEQahIxnPvB2Cr9LwGQD/GWHfG2N8BHM0Yu93pzZzzJzjnYznnY3v06JHwIAI+H0I5bsukU35J2gmCUBFI4r2qqm3OOd8N4IdJHNcTPj2h+sTc9djV0IpfnZtxZ8g1aalzp8idIAgFyUTuNQAGSM/7A9ie3HC84/dFrIn73l+Fpz7bmOnTZ51sSfs/527A7G93ZensBEHEIxlxnw9gCGNsIGOsCMClAN7xcgDG2FTG2BP79+9PeBB+n69D17l78dw/Xl2L6ukzsL4u+e6Uv3t/Ja59fkHSxyEIIj24LYV8GcCXAIYyxmoYY9dwzkMAbgTwAYCVAF7jnK/wcnLO+buc8+s7d+7sddwGfpb7a6im1TrxcOh3l+4AACzcvDdNgyEIIldw5blzzi9z2P4+gPdTOiKP5FPLX6ZMUySHl0+ur0pIPj1BdADyvv2Az9Lytz3Xp6umGC86LdaczZNrIUEQSZBVcU+F5x6wtPxtbkv9Una5jBfPXVTrUG08QRQ+WRX3VHjuPkvL3+a2UCqGljd4kWnGKHIniI5C3tsyfkvL36bW3IvcxfCyXefu1wdAnjtBFD55L+4BH0MonB+Rezo01csxRUK1I7drIIiOQt6Lu7VaJpdXZUrHqklexJ1sGYLoOBRAQtXcWyYXyyKFqKclcvdwwfAZ4p573xFBEKkl7xOqAWvk3sH6/yZiy5C2E0ThUxC2TLsk6LkclaZjaF4+r89HkTtBdBTyXtyDlt4yHc5z97BvtM495cMgCCLHyHtx9/vNde7ZWnJv5/4WLN+mzh2IQDn71TIUuRNERyH/E6o+ZkmoJnacHfsPYs2uRtO2usZWPP7xemze3RT3/RN//xHOffSzmPukQ1K91KxTbxmC6DjkfULV72MIh5OP3I+//yOc8chc4/ncNXUY97vZ+P3MVbjy6a/jvt+NHZQOUfXWOCyi7plsv9MaCqN6+gw8+emGzJ00RbS0h7Fj/8FsD4MgEiLvbZmg32cS1lR57ku37jMet7anRg3TE7m735dlwZY50BKZVPa3j9dn7Jyp4rrnF+D4+z/K9jAIIiHyXtytk5hSVecuKkvEOZJBjCj7de7m8WSCfDaAPl1bn+0hEETC5L24B3wM7QlOYmoLaY6rEsl9YAL+VDWFSb3UeXGhjIRqFspl0tBWhyCIGBSAuPtMEbFsy8Tr7f6//1qAU//4CVpD9mZj8sIagSQjd0GuRO7ZqJbJ5wieIPKR/K+WsUTVIiqds6oWQ+74j2N5IgDMWV0HANh9oM32mqznQX9qvqZc8dwzKbQUsRNEdiiIahmZ1pCGt5dsw6yVuwAAi7fEXy/0hy8stG2TbRkvnruqIkZsy5k69wzaMoUQsVPpKJGPuFpDNZexWiaPzVmH2sZWVBRHPprfF//6tazGHt2bbBkPkXtI4wg6ePTpKYXMj5WY8jmC5zw9vfgJIp0UgOdu/tXVNrYCABpbQ8rX3WJKqHo4RqyEbrZtGQG1H/AGfV1EPpL34u6PE1UnWsbIWGIJ1Vh19qkMmMXw5q6pw93vrHD1HnH+TEbuheBoULsGIh/Je3GPJ7wJi7t8Dg+lkGFFy2GxZdPuppRZM2JEf5y1Bs9+scmVj66l0ft3Ih3N0jINaTuRj5C4K1hX24i6A63SMdx/Te0xCs9vf/MbvL6gRvna72euwph7Z7kfpAVVUtiKuLCkOxK96eXFOPa+2fpJ03qqjFAIFyii45H/CdU4UbVTctMJTeM47eG5pm3BFHnuAPDNtv24eNwA2/bHk5ye/99vd8XdJ1O2zLtLt0fPmdYzZQaK3Il8JO/r3ONF1V6ibgAIK37JXmyZkMbxxNz1eGOhFKFLh0xVzTxLoHxD4+Z/M4G4kORztUmmxZ1zrpxYRxBeyPs693hR9bKafZizqtb18VSRd8DDBSIc5rjv/VX4+etLla8XBVIk7gm8Jyueexp72WeKTNsyD89ag6F3zkSTXvFFEImQ9557e5ww9NGP1uHqZ+e7Pp5S3D1F7rFbHsQT93Qu8C2OTJOYvJHpC9NrC7YCABpbSNyJxMl7cT95aI+UHk9VyuglKRuv5XBRnAtFvH44ySASqmHOsbE+/gIkbpm5fCf2NdtbOADRC0le2zLZHgBBJEDei3tlSRDjq7u52vetxdsw9rezEda4Y/SqEldPde4xSiGB+JG7W3G3imVVeXHc94gI9I2FNTjlDx9jkYvWDPGobWzBD19YiP/9V/xqnXwlW+0HqEqHSIa8F3cAqOwUdLXfHf/3DeoPtOJge1iZOAWA5lZ7IstL+4F4tkq8hGq74uLghoqS+IVP1iqZ7fuSX2VIjHfrnmbl6/nstQtoRi+RjxSGuLsQNitWERYlk01tdp/T2wzV2JF3PIsn5DZyt6RU3Xj11l3KilNXCet09oKY3Zmlj2D9GxOEFwpC3N1ErTKcc4W4R76K5jZ75P7u0u2onj4DLe3O5WnCJoknsvFeb0vQc3cj7tbb/FRUY8STnwKQdrJHiLykIMS93Ku4w17PHhV3u+DtbW4HANQ1ttpeE/h1dVclVGXPNp4Iu7ZlLKrqStwtu9z40mJ353KBU4BeCO1yC+AjEB2QghD3koDf0/6aIqEqxH3h5sSSjGLNVVVCVSa+uCcWubtZGFyVRE62LDKe7RI9fP5aDAVhLREdjryfoQoAxUFvH0PjdpEVJYp/mr02xvucf+Qico/VWwaIL8JtIQ21DS144D+rYl4IrFLpRoBUeyQrXBN/P0c/ttNx8l8Ys/UJyA4ikiHvZ6gCQLHXyJ1zuy3jYuYo55GKmxMf/Mj2mkjItoVii3u8SDmkcfzi38vw90/WY96G3Y77WUsh3SRiVULuVDUERO5iqqfPwLraRtN29WpT6mMUQtCb6c9AiVQiFRSGLeMxcm8NaTYRdlMRwwG8OG8Ltu6xlxAWB/3GsVXvE8SL3NvDmuG7i301jTuWGgrcuCsqkYp1o/HeskgDsE/W1Mc9TjLjynUyHUFTxE6kgoIQd6+R+4QHPsL3/v6laZsbzzpWcrBI9+xjVdQA8W2Q9pBmROViz4/X1OLkP3yM2sYWx/e5S6h6i9xFBGl9n+odzqZMAQhVAXwEouNREOLuNXIHgB37zUIZT5SB2FFosW7rqCJ3mbiee1iLLmSti2p9YxvCGscBqddIKurc5XOocGoZ4KUCphBsmWzdfRTCd0dkj4IQd6+Ru4qWdjdVKjEid13c739/ZcwjPP7xehzQ68u37G7G9H8vM/nl7WEO4RAJEW3VX5dFRhZeH1NH4Cu27zcdWxVFO+UAWtrD2Ly7ST+vi8jd4aspiJa/WQrdSduJZCgQcffFfO6Ggy4i91iRlDinahKU9X0b6yKi+dNXF+OV+VuxZOs+47V2OXLXdbldvxtwipgDfh/CGje9vq62Eef85TM89MFqY5tKx50i/htfWozZK9Wtkr1ElAXR8jdLCdVMdu8kCo/CEHeLLVNR4q7XjMwFo/vF3SfWb81L/xlr8zA5qm1qDWGv3mFRRL1tUuS+p6kNg3/5vukiInray+Or1SdcLa3ZZ2xTXRycPtMX66NJVOvb1JFs4QpR4X4yopApCHEfVFUOALj/wiOx+FenJxS5D+xRhmMO7RpzHznKtUa8Xnxoq7jLb731jWVYtGVfZLu+TUTuGudYtHmvzbcXFxZVXxv52MpqGcXGOatrTRcP6wXAW7VMAdgyGQ7dxcUzn+92iOxTEOLetawImx44B5eNPwRdy4qUbXXj3eL6GBCvGlIWQutMUi930G8uiizBJ97i9F5ui9y5UiRFGaes7apaaWWdu+LkVz9jXtzEjedMde5pOC/dMxBJUBDibqVIYZHEK0H0MWZ43U7IQmiNnmMd3fojffSjda7GJk4hxJ1zdQQsVopSJVXlLarTJNKTRrmPx+35RNbEvRC+PCJrFKS4BwN2BYxVzy2IK+7SMUJhDRvrm7BTlFQm8EsUb3EW98j29hA3nqsicrHGa1jqa6P6KCodT0RAvCVU81+hMh1BGwnVAvjuiOyRuobeOYRqQYw4LV/Aefxe67K10xbWcMofPgYAbHrgnIR+/tz2wD6myLki/rdTkB2UIvc9TW0oLZJKQ02eu7dJTE7vU4mdk4gXQsFH9mwZgkicgozcVbaMq8g9jribbBlL90eN87hL6DnhtMi3NXL/fF29srminFAdc+8sfP+pecruJKqzJGLLeBO7zEvU7gOtOOJXM7E4BcsIAtmLoClwJ5KhILpCWlEmVOP8Ujiik4ec+syYbRlrtQxw4uAqZVI23o+03WFWazRyj7z+0AerlaJtTajO36QWNdV3kEg3SS+FkNkQqK827MHB9jCemLshJcfLnsaSuhOJUxBdIa0oE6pxIlTOo217OxWpZ7zK1o61ta9IdrIEav6cVl+y1rkD6uMHY5VCSgLhtf2AcQxb5O5edMQ53X4rP31lMX4/c5Xr47tl4ea9CS8ITglVIh8pSFtG5bm7sR+EcJYVqVMRcuRuLYXk+vs9LLdq/HqdFugwqmXi9Kvxq0ohLReBNxbWYPk2+x1SIsvzKSP3OOWcbnXqrSXb8fjH613u7Z6LHv8CF/7tiwTfnR2VLYR8BZE9ClPcFbZMY0vs9UI5AHFNKC12itxj2TIcDN4id3GEtxZvU78eR/wFIqEaaxLTz19fio31Tcb2W88cCiB+olk+htPzmO91v2vKSPWEKapzJ/KRghR3lS1z8h8+jtuRUUTAjpG79P6fv77U9JqwZRLx3OesrlNu54rIXWUvGaWQ0mvxBK6yU6RFg6tqmbh7xKqWiWzPxgTVVIkydYUk8pHCFHdFnTsQvx2viLpLHTx3WQhX7bSsTgTuaiKU6T1xfryaInJX2SjiTkW5OLfDscWygq4qQezNZVyfpxCCz6x1hSyA747IHgUp7irP3Q3+OOIeK5GoGZG7O3F3k5RUee4qARd3KvJFIN7hxXfkpvOgrbeMB7ErBH0iW4bIRwpS3FW2jBNTRvQGEPkBC0ultNjJlnE+Dtdnj7oN3DUe/8cbrZbhtm0y4k7FZN+IRKaDMolyUTcJVSvKQ6qiec6zUiOeaguIqmWIfKQgxd3NYtcC0ZcFiE5iKnNhy1jhAOAhcndXgmi3ZVSRu4jCWxXi7oS4ALrz3ONXyyjfxwtDoGgSE5GPFKS4e4ncxQQg4ZkDQKlDQjWmhcFF8zF3540srhF7HyHkckQeVpS3iM8r7xfv2CJydyMg9pa/7rx9jZOxkAz07RHJUJji7ilyj+4rPPeSoEPkrnGleNcfaNWberkvhXQjqsIykSN3Vc170Fi/Ve7BHru+XFwQ/t9rSx32cMZ15C6NI5/7uR9oDWFdbWP8HVMMRe5EMhSkuAf97pVE7Mt51JZxWnA7zLnSdhn729nYtLtZWQoZr0QwFobnLgl6W9j+viKlLRN7DOKCsLOhxfZavLGqhq6M5jmymlFNVeT71Gcbcf5fP8/Coh0EkTgFKe7ebJnovkKYnRbc1jT1YhkCBrPnrmkcA29/H9v2HbTtG+YebBnZc1dkdYsCzp475w7lk9J3NG/Dbjw2Zx32NLUpx2G1o9yKJtf/L9Ok+i5hV0MLmtrCyvVx00khtEsmskdhtvz1YMvIbX7FY6dl+sKc67aL+kfnY+ZqmeYYi25zLX5kpik8d6Uto999tDpMdrJq+8CqMsOCAoBLnvgKADB/0x48e/V42/Ft1xOXde6cu5sBm+vsP9gOIGLPlDlUUqUS8aeh9gNEMlDkLom7iLqtC24LNAfP3cBSLXMgRssDN7ZMSOG5q1oRFPkjdxqt7ep1T+VzjR/YDXN+fjKkGxYMrCoDAMfIXZXEte+jtmUKQZ+EuDe2tGfkfNE/VyF8e0S2KExxTzChaoi7gy0TSag6qzuDeYbqgdbkxF3TOEJhzSTUKs9drDwl2zfy8WXhFRG7fMfSu7IEgPNF0VouqRq5avYvB8+qtZCqUzcY4h67P1GqIVeGSIaCFHfrDNX7LzwSR/XvrNxXaBzn3Ggc5mzLxK5jZwymiDiWuLupLw9zjnaLmKsjd91zb5dLIaPVMrLQC1GXbRnRcGxPs1Pkbn7uVnQ4l1v+ZrJcJrXnEp8h1t8zHZC2E8nQIcR9VP/OjrXro3TRH9630hBup1LIeLaMj5lF7L2l2x335Ty+BIU0buv1/n+L7B0kxZ2KOXKXxx19LDRdLtkUF5C9Lm0ZpySpPfEq///8lyqK3Il8oiDF3eouBP0+x0TYlJF9MPfWUzB5WC+jFDJWQjXWUnwRWyb6/MnPNjru62aCj6ZxWwJVVbooLmYt7fY6dzFugRG5++yR+76D7ep1Vl1G7gctCWTNRUVQPhErh5JKognVAvryiIxToOJu/lgBH3NMkgLAId1LAcilkA7iHs9z99B+IKxxW6Rrrc8Pa/F7uQNx6twt/V38ivGJ3vROZZP2yF1Nk8W2yKQtE9Y4Nu9uMm1LtTQ2ZtqWIW0nkqAgxd26BmrQ73NVQeM3qmW8zVAVMAbXdi/ndt/dOsawpsVdhQmIln6a2w9IpZCSYIs7D6cKHFUuwJrDdUqS2jxpHrVjdja0YMSvZ5pm0aaSP3+4Fic99DE21jelbTZspqtlCsHKIrJHysWdMVbGGHuOMfZPxtjlqT6+G/wWBQ74matZq4zFtmUiE4hiRe7MdZdFjSsid8t5IwnVRCN3OaEa3VdcwJzaCKvGb/PSHT6idZJPpFom+rypLYy9TRGBbA9rqJ4+Aw/PWuPwqbzx1frdACITjtJFpmwZA9J2IglciTtj7GnGWC1jbLll+xTG2GrG2DrG2HR984UA3uCcXwfgvBSP1xWqyN1Nj/d4k5giIuj8i2MA/nTJaPTtXBL3XGHN3g7XHrnzuAuMAJFx+5ilzl16m8pzH9yz3NgmLxmoEnfV8n2Cw3uV44TDugOwR+4RW8Z8PLGPyA889ekGx2N7QUS56TR/Mp5QzejZiELDbeT+LIAp8gbGmB/AYwDOAjAcwGWMseEA+gPYqu+W2fnaOtbIPegzi7uTReM3esuobZmQxmPOGmQMOPqQrrh0/CFxx6ipbBlr5K65i9x9LFKvr6pzj8wSjZ5HWBYlQT+umTgQZUV+0zlUOh5rCIOqynHblGEA7J67KiE4Y9kO03Mva87GQpxKPl6qPetMlUJSQpVIBa7EnXM+F8Aey+bxANZxzjdwztsAvALgfAA1iAh8zOMzxq5njC1gjC2oq1OvIZooAWtC1c9MwnnS0B4OY4r862zLxF58QiQN3cgV59wmpHZxV7cbsJ2XAUEfs9S5R19X1bmLx2HOzbaMslrGklCVdmEMKNcXFLdF7rAL7COzU2PDWBGnSWf3SUqoEvlEMp57P0QjdCAi6v0AvAngIsbY4wDedXoz5/wJzvlYzvnYHj3UYpso1sjd72NGtD68TyX+cunRyveVFQUQ8DF0clqsQ9Ni9nQXp41VLmkci3ObBaJKqFonMalgYM6RO7hyhmpkvAyaXpEjchIqC8aWUIX5TkCUmTa1Wjx3hS0THV/cj5Uw6ZgVe+KQqowlVAWk7UQyJNMFSaVgnHPeBODqJI6bNEN6leOskb1xzqg+2LGvBSVBv2HLHH9Yd3Qq8uOBC4/Elj3NpvdddEx/jOxX6TjhqT3OAhvCEnATPWqaXfisF6UwB9rC8Z0txiLvbXeaxMRlMZYj98hFJhTmKAn40R4OKW2ZWAlVzqPivmL7flM1jDWhaj5G+jzyVF84Sov8KC8OYOd+e7J26dZ9aG4L43g975BKqCskkQzJiHsNgAHS8/4AnKdkZpCg34fHrzjGtE0spycEVOWLlxcHcMyh3RyPG45jy4jX3NS6a4r1Ra2J4MgkJheRO2Pw+5hyDdXI4+i+ctTt16t7woyjrNiPxla1LWON5uU9OAdK9RzFi/O2mBPX3Dn6TLUAyxeLVGuijzFUlASUnvv5j30OANj0wDmpPSkocieSIxlbZj6AIYyxgYyxIgCXAnjHywEYY1MZY0/s378/iWF4w+0kIxUhjcfsCSO8a3eeu70yxWrnhDTNfULVx0y2jNFbxnIeVc17W1gzmqWFFTaQNZqXI8ow56bma99ub4i+jztHn4kszB0L2XOPXthScw4fA8qLg1loP0DyTiSO21LIlwF8CWAoY6yGMXYN5zwE4EYAHwBYCeA1zvkKLyfnnL/LOb++c+fOXsftGSFqbtc4VRHStJgRZ/QciXnu1sjdbUJVnHNDXXSGpnxok7grat6BaBLZa+RutWwO1Wf7Rvaz2zIir2AIV4p8meh5WMojXp+PoVyP3GOuo5ti1u46gHvf+zaj5yQKB1e2DOf8Moft7wN4P6UjShPi92H1tb0Q0swtbLuUBrGvOZpkE8lPV547t5dVWi8KYZeRO+f2njNyKaTJI5fPJ30XYlausv2ANaGq8PPLiyPi17282LSfdZalaAORLr1iLPURr48xVJZEfioH2kKoLAmm9PhO3P+fVQCAayYORN8unTJyTqJwKMj2AyrCHvxwJ6y91YWGTBgcSaYJIXZzjue+2GTzcG3izmHrCumEVZRNkbvJf1eXRYrIXZVTsEeOsi0T+fej/3cSAHMrA2sppI9Fj+Wm5bEXDCOGp8Nzj1y8gCzMUoV5khlBuCWry+wxxqYCmDp48OC0n0vUalutD2/HMCdBuRS1AlFhc3P9eHuJPfccsLRImLumDm7W+lb99OV+7mFzRtVAtmVEjf0Zj8y1HSukcWzd04wB3Ur1Y9vP07OyBF1Lg7b+NvKpS4J+Q9Q1D/kJV0g+u3zXkgp8LGLLAJmfpQoATW2ZPyeR/2Q1cs+k5y4CSjc16E7s2N9iKwMEIsk2IOqPWyNwt8v+qSL+OasTm+ClughZt/sUkbuVM0f0wsodDTjxwTlYuHkPmlpDWL49mgCXLxxFAZ85crfYMp2CfiM5m67Zl1oaInd5noQbmyzVWGf+EoQbOowtI8QkGc99hVQJIh+zTJ+h2WbYMtF9LhrTHyvvNXVucCRRX1XlMQvN5ZbErbn9b3T/7mVFymPLs333H2zHj19chJ+9utTYZhV3c+QObN930HhuitxFPjVV7Qf0fzVFz55k8emlpkDqq3zckOkVoIjCoMOI+8h+kbuDI/pUutr/pWuPxYPfHeVqXzHpSQib3DI4sjpTfGbfMsnUzMsLKrlxqnM3VctIV6HRA7oojy1H952CAczfZO5CIeto0O9DqxTZzlldi8fmrDeelwR9hjiqBDgVidAwT32jXDFJDDB30MwU1m6bBOGGrHrumeS8o/piVL/OqK4qc7X/CYOrAER87/csza4E4mcuequIW/b+UgTuY8yVBz+wqhyfr9utfM0aEbthxbbIXYZ1shR3sGUG9VBfWOQcRVgxQ1c+dpHfPM5lNeb5C6Ihm6ZYqCRyLMePY2Lz7iZs3t2MSYdH21YYPdC5Od+QCuTIPRvNvChyJxIhq5F7picxuRV2mb/+zxjH18Tv3IjcdXEXiUcgsmC2sB6uOqHa8ViRzo7qq0CJgx8eHYh904xvIhckjZutBFmb5ISq07nlPEBI02yljbLYFVttGcu+QtzDUqJVvvC5jdwn//ETXPn016Zt4lyqpf2SvSPw+5jxXWXDlmnOEXHf19xGE6vyiA6TUE0HNs9dF7Y+Uj/3Ml34N9x3Nu6aOtzxWIwxx0qeyk6J11VbI3dVQrUk6FMuvweY16MNK1oey1V6tjsMy76dpFp6ZcmlogpHhWHtKC5aGrffASSrx4xFv6tsTCiqP6BeuDyTrKs9gNH3zMKL87ZkeyiESzqM555KhEhZI3cxiSng92HVvVPwk1OH4GenHw4gIg6MMbxwzbGOx7W2KhaUOyzuLYjlMls7M6pmqJYXBx0TzfJ6tNZJXIBZ7IoCvph1+aV6t01NMTvX+jncaKhcIhgVdx7z7iIRTAnVDESu1r/ERsvasNlgQ90BAMDHq2uzPBLCLSTuCSAibCEiQnzlqLUk6MfPTj/c6JgoiCXETot4x5sRKevNlBG9Ta+FNW5abEM+uxCsypKAY4moNXKP5bkH/T7bJCYZ8V3Ix5HPqpr5GgtVzbkm3V1wzlHb0ILaxta4x4qFL8MJVfkM1d1LsbEu++KeqqomInOQuCeAX/enH7lkNCYd3gNDekWSkW5mk04cXIU/XzoaFxzdz/aaUz28mEDjhgrLvk4Tr4Co1VBeEnC0hPwmz91up5hKIf2xE7+iT77c7pgDuOGlRfhifb2juG+sb0L19Bn4fF195Dx6DqJB6q9ulEJymK4S4+/7EBMe+Ci6n8vI+7fvfWs89rGo5+5ky6TLix7csxz7D2a2j3wsyHLPHzpUQjVRVvzmTJw1MhoRCyEcc0hXPP+D8bZSyFgwxnD+6H7KpfyKHZb3i2vLSD+4CkuUH9I0x1bAgh7lxY4tE2RbJqxptmjcVC0T8JnWfLUKXplsy+ivtYU0zFi2A1c/M99yEQIOtoWhadwov/z3whoA0e9DGblLyVrVxdatOD352Ubjsd+nrnPfVK9u1JZKupUVxVzDNlNQ3J5/UELVBWXFATx+xTF4+4YJmPPzk23euIgkvcxeLFJUpzhF7tZoPBaqyF2e4SjrxA59gtEh3UtjeO7yseziKItap6DfWPgasNsy4iK4bNt+Q/hNDc6kfUMaxxG/nok7315uzJ4VYi28+wYpoo22OI7eXWzdcxBWEvHffYwZFz/5/Rc+/oXxOF1VNEG/L6W9ZeoPtGLc72Zj1c6G+DsTeQ3ZMh44akAXDKwqs0VxZUV+XDimH577wXjXxwoqhNy6hqqgvCSAJb8+HWcf2Vv5uvzTt4p7SOOmOmlZnIQHPq66m6uEqnUtVcBsU5QW+bFDWq3IqqOiqmja019jT1O7fkw9gg9rmProZ8a+rfpF4qV5W4yLXltIw4vzNqNmb0S0W9oV4+HR86rupBLRYLlMVb5+72mKVrGkUtzlv0TQHztJ7ZWPVtWirrEVT366Mf7O8pj0QZErkz90mElMqeTIfp2xbd9BI6JkjOHhi0d7OkZAIe5O/V0qS4LoUlrkWE0j2x/W5GskcldH05eNPwSDepThhMOqsF6vhrAiD1OVTJQvFiUOa88K5OULr3t+ge2YGyWbQxa0gNTX5Y7/Wy7tY5+5KecYVHaGSGgLYe7m0HZBhkmRu5NFkq4qmqCfpbYrJKlzh4HEPQEevuQo/GDbQPSsLIm/swNiQeoB3TrhSL01gpO4C4/ZjYBYk69hjVtKBqPH8PsYTjgsMhPXsc5d2i4Lq3F86XidLDkD62hVtpPTR5KjbmF37WowV720S0sQyqWQ0ffZD756ZyNG9u2MMffOAuBueTzZc3eydVQrWCWKfKSA35cWzz1RD50mMeUPZMskQGlRAOMHOq+16gYRhX9ndD/87fLIeq9iqTsrQtydfliq8kaB1ZY5zKHNgBtbRsUvpgwzHluTxNbxeunIKYu7eGy9u2g1lV1GvXsjclfYGef99XM8/sl62/ZY+BikGarqfdIWufsY2sP2+QUEEY8O08891wgGImIhR5dOnrvw0Z18XaeWAuI9Ta0hHNKtFPddcCTGVndVHkMl7tXdSx3HBADPXD0Opwzt6XjuLXuaLedwPJQN2U8X4t5q8dCtHSgBc/sBVeQOAIu37HM8756mNtv3/IspwyCucU6lkOmqaBG5mZDGjbu9ZEh9WzUiV6FqmSwR9NkrbGIlVAF1MvDyYw/BqUdEBfbI/ubvUoh7WXEAE4dUKUswxX4yw3pXYMbNJ8assbdKjTV6tTYO87IKVrNkJbU6hMvtYQ1rdjWCS50g5RYJiSQix9w7C+N+N9t4ftfU4Rhb3c2403Jq4pWuhKrIN4TCHC3tYc8N5KwYk8c8XidoDlP+QbZMlhBRmCzuTp57hb4YiCpq/N0FR5oqb3qUF9t85LW1Bxz7tQv6d+2EW04/HF9Mn4ybJw/Gk9PGoqw4gIoYNfbWWYuxBO6NHx7vTdylkkqnxSo+X1ePMx6Zixe+2mxssy4QosKLUIk7GhG53yNNbpL5RrqQpbKLo/HfiaZh2K9m4txHP03ZsYnChsQ9S0QrQNS2zEvXRXvQdNYbhx1/WGSt1psnR2ws1axSla+9eXczDuleatsuwxjDzacOQd8unXDLGUPRv2tk/1gTqKxnilXVMba6m6ca82apwqe2Qd0+YIlur6ze1WhsU3WFTAZxQZItJ9VF7Pp/LcSm+iZ8sGInRt71AZZs3ZeS84u/sfhu1+xSVzW5RYyceUypiv07uqlz/fMLcPuby7I9DFdQtUyWKDJut6WSP0mYRRULEKmoAYBrJg7EOaP6oE/nThjSqwLD+zovPNKrshh7m9uN2/hDu8UWdyc6xSlvlBnYI3ZLZS/i/uWGeuPxzOXqfvqNeoRcXhyUJkXFr+jwImsicpdzEu1hDX6f/Xs5+Q8f44rjDgEALKvZ57gAiheCCUyQSwfZ6GOfi/z3210AgPsvdLeQTzYhcc8SZ4/qg5krdhpdI4FI9HzmiF64cEx/ADAmRQn7gzGGPp0jQj/1qL6m4/3nJyeaVkma98vT8M+5G/C791cCAA6NE7k7UWJpZjaoRxk26I2srPbG1FF98OiHa7G2Vh1deqnXfuGraGvZ7dLEKBUVJQGpt4y9LXEyCE2X74jawppj7kLVEC0ZVLmZVODVQ482Y0vpMIg0QuKeJcqLA3j6qnG27f/4/ljj8UnSSkPxOKJPpW0JwXapguOQbt4XKgHs5ZmvXHccxt/3IQD7rT1jDMcO6uYs7mmaoh8KR3sXuLFlvCQ/hS0j31W1x0hqpvoTipmxqj46mSQbi5QQyUGeewEjV1bE89ydsL5Pnlmriv5iReeJiPt3j+kfd5+avc3YoM9udbNAtlxSuaHuQMwFOIyEKpNtGRefw0No3NQawufr6pV1+eL73ttsXrDjm5r9+M83arsqFolG3k5W151vfYPq6TNM295dut1o8kZkD+oKWcAIcS8vDsTtLOlEZUnQVH3TtTTa3kAlXyt3ODekSmQVo3hVPoA1oQp8tUG9Fq1Abm42+Y+f4LE56wAAr863rzKk8tzl81mJruXKlWKt4uWvt+DyJ+fhX1LVj0A0mNvfbG77O/Wvn+FHLy5ydXwVXm0ZUeZq/QvK9pngppcX4/+9vjTBkRGpgurcCxjh0958auomiTHGMKSnPstVIRC3n32E43sTidyti52oqJMW47jnvW8xb+OeGHsDCzbvNT3/fH09WkNh/OLf39j2ZYpqmWlPf42Fm53OEfmMv357BYbf9UHcsQNRy2W3Yjk9UV+/L8s93cWfbvXOBrzy9RZoGsejH641XifbJvcgW6aAEfaBqgNlMgjBVZXTHTeoO1661ryU4ClDI7kDVVfJeJS6qNbZESfhGo/2MMff5qhbEghRt5aYbqpvVu1usj3cTjgSwtjcZm+EJjz3eukClsw6ronMUN1Qd8BYZm9XQyumv/kNvtqwG3+ctcbYJ9nJVUTqoYRqASO85VSLu7B4HCs4LJr/zNWRqp+Rfb3foSVqJ3mhPawZ1owVp3Y4AYdWAIl42mImbWOLOToP+Jjxt9u6N3oxaQnZLwLece/LTP7jJ7Zt1jG0hTRPZbNE+qHIvYAR4uu0CIgXbj1zKP55ZaSSR/Rld5qJ6dRh8oTBVfj6jlOx6YFzXPdJiWfLiLuCZGgLaRjau8LTe5warb26YGtC5wfsFTF+HzNmLW/aHRX3lTvsnn97WMNfPlyL5rYQGlraMWOZOtmaqlJGay/9VPacJ1IDRe4FzDGHdsUbC2s8C5eKG06J+vblejsEJ3GXBdlqq/SsiLRJLvL70K7ox27FqbdN0M+w4jdTsHDzXsxZXRf3OLFoC2mOgayTFrr1mDnncReXFndY4vsURw74GLroCexVUqL6v9/uNI3D72N4fUENHp61Bm0hDat3NWLWt7swvO/JGFhlLoH1qu2rd6qTxwctFhKJe+5BkXsBc+m4Afhi+mQclYKZkjLlInJ3qL2WrZSld52h3CdWt0mnY/345MPw50tHA4hEoEUBn6slCK+dONAoqVT179lQ32Rqd+CGVsUqUMr9QhoWbt6LW19fCk3j+GJ9vSHij364FjOW7TDusIQtIy4cfh9Dl9JItVBDSwij9KZw8tqt0ai/XT9fGFv0KF+uChKIkka31TJn/mmucnuz5djf1OzHi/M2p7SvDpEcFLkXMIwx9O3SKeXHveK4Q/HK/K04fXgv5ety5O7k90cmR8WuAOkU9KNHebHx/LYpw1Cje8+ilt2NJ3/nucPx9pJteGNhDYr8Pjw1bRyueuZrU/XOzgZvSdnb/h3tL+Jjzsv3NbeFccOLi7CzoQUTBlfhp68uQVV5MfY1txnnP0+fbbz/YDt+/vpS1OrJ04Dfhy6doqWnI/pWYvXORuyUEshtYQ2d4DeOFfT7jKSpSsDFhSPZGbQNluqdH76wEED6FgrPFKGwhk27mzC4p7u73YaWdsz+dhcuOLpf3Du0TEORO+GZIb0qsPq3Z2GAQ78aN4JrbWsg07dzCd66YQK+ufsMdLXUuYvSQKEhsVoSy4gl/sKcY+KQKkw/a1icd7jn0O7Os3+b20LGUn6v6X58/YFW04XloB4F72xowRvS5J9/XjkWAb/PEOmBVWXoVOQ3XYhE5C7+Dfh9hsCqJpTN2xAp4fxi/W5TCalX9juUZsqin0xVT7b4zbvf4rSH5+LDlbtsr6k+z9/mrMctry3F7JW1mRieJ0jciZQTS7gFT06zt14Q/PKcIzB6QBcE/D6bjSISmSIxGOtC8s6NE/Cb80YAiC54IiwQeV3ZETEasMkJyBtOOUy5T+dOQfzwJPVr/5y7Ab0qI3cfaxwmP83Sm1HJScpzRvXBMYdGFlYRrQ8qS4LoFPSblhucs6oWmsbxZ73mPOhjhvViXdxE0zhmroj49Rvrm3DJP75UjscN+5rtNfkAUNtgvqvIJ+Zv2mNMJFPNlVCttiX+W483cS4b0AxVIuW4uT0d3FO93B9g9rOFuE8YHGl3bG1zLF6/aIy9TcGo/l0w7YRqALK4R36gB1qjEeZZI3s7jkWuC79QcQ4g4mO3OpQnPvflZqM/T71ikpITpgU79LuV8pKAbZ3a2/69zLRGrt/PsF5v7GatPbf65Bv0FsWJUOsQ9W/fn7/i/r2/Ry92qgllqjshYTvmYp0/zVAlco6DkggxxjD7lpPwhN5QzW8poWSMYeU9U/DgdyMtWG+ePBi/mDIMQ3uZPdPKkqDp+QEpcu9WVgw3DOhaipMVpZdXHl9tRNkyJx3eA4wBexyiXBnrHYrcsE1c0CpKgspulK/Oj5ZfPjhztfG4NRTGVxt2Y/u+g2hpDysXPfnffy20bXvhq814fcFWfLvduZWEUxWNKR+QYsFrbgvhDx+sdryQppJ/L6qxjV+1lKJYMSxda+gmAyVUibRw1QnVGNLLOTqPhbXKQ47yVQuUiMkzcg+cH51stkmsVTWy0E0e1hPHDeqGgVVlePlrc526/JstCvjw7NXjMe3pr/HJmkj55exbJhnJtxtfWmx6749PPgyfrKnD13HaIQBAdfcyU8+aokD0c4oLWnlxQCkwv52xUnnMhpYQrnpmPgDgkG6lePZqZytM5s63lsfdx2lW8I79B43HqRb3xz9ej7/OWYceFcXoUhrE3e+swNd3nGZEzy3tYbSFNduFPFFeW7AVVxx3qPFcLn9dvGUvigN+Y1bxngNt2NfchtKigOtKsHSTG6MgCo67zxuBy489NP6OCqw11DJOk4fiYfXm5UqfLqVBvHL98RjZz34HqYrHpp0Q+VyDe5abqiomWVo0q47nRL+u5qomYcXIjytLAp5WYrr55ejFZsueZkfPX6ahJX4Pm0vGDlBuLy8OmKyntpCG2sYWPDxrDeoPJJ68FYhJXu1hDXe9swJ7m9tNE7/O/vOnGHX3fx3fzzk3chSCzbubcP9/ViqTpda7KTkJfsHfvsDZf/nU+G915oqdGH3PrJxqmEbiTuQcBxX12QJZ9LwgWueeOiyymLi82ImI/IQVUlUe26YR1oh1Ju7jl4/BBz+dZDwvLfK7Si4DkQSnebzRY4u7FacFQtzywxfid5HcukfdM0dmsrQgu0w/S9ntroYWXPrEV/jLh2sxc/lOHGwL47kvNhnlrF4Rdy0BX7SrkTxe0fZ5+bb9qJ4+A0stSx2+vWQ7rn52Pl78OtrJ8saXFuMfn2wwrUHQo0L991d57tZ+QO8u3e7686QbEncirxCR++EJWD5Lfn06Hr/iGMdjikht/MCuOHdUH8fjiKSmtZlYWXHANBuYMYbuLv38QVVl+NW5w43ncsuIW/TVuuSLzgMXHgkAxgzWZJBbE1tbC6s4w2F+Q9cy81gueeIrY9WuxpYQ5q6tw13vrMBdb6+wvbe2oQXvLt1u3F1srG/CU59tRCisGVG1sEU+XBUtOzz/sc9tx/pIf91aziguBDsl60hE43KfpOMGRZL3LS4891iBSLYhz53IOp/edgpOfHAOAODGUwbb/HIrr15/XMxqGyfEbE9Bj4piU623EPe2kIYK3bdVmUDRBTzU53n+B+ONyWPdyoqwbd9B9Y4S//j+MaaeMXLkfvG4Abh4XMQK+fOlo7Fk6z58b+wAaDwiOL9WiKUXdjW2ontZEUqCfsf6dRlrNRRjkdzEiL6d8dUGdX6hsaXdmNG8q9Hs18/+dheufX6B8XzTA+fg128vx6dr63Hve9/itCN64fzRfY18yKdr6xELMTqOSFK5yO8DY8wQcr909yf65cuVPeK/g2c+24iX5m3Bod1K8ffvH6NsOSHKWAWxeibta27D5D9+gienjcWYQ+wJ+FRD4k5knQHdSvHQd0ehe3kRJg9TR4Uyx+qRVbK8d9NErK+L3o6LCVHFAT9+de5wdO4UxJkj7GWS4jfucyj5lL13Ny12375hAgJ+HzoVRUVHVX0DAOeP7ofzR/cDAPzPsYcoF/iwcuKQqpiCOOGBjzCoRxlm/eykuAuAHKnII1w6bgB+de5wfLl+N576bKPyfQ0t7UYpZijM0dwWwvZ9LRjcs9wk7AI5RzJ75S7MVkwqEvz1o7UYIXUcfWvJNgDAutoDGHrnTFx1QjXuPm+EEXkHpauysOTk9hNC3IXNIxagcbMCVyzbcN7GPdjT1Ia/zVmHJ6eNg6ZxNLaE0DkFd18qyJYhssacn5+MV68/DgDwvbEDXAl7KulVWYITDqsynh8/qDtumzIU935nJLqVFeHu80YoKx/668nPS8apE4sywpa5bPwhpu3yguXCyhGev9/HXH8XYT3i/P5x6uT1SYf3cLxQyGyoa8Ln62JHxADw7k0TbdsCPh9KiwI4ZajaiwcitszBtmhC9LrnF+C0hz9RRsO/eGMZ/rPcff39H/67Blc/O994Lur8hcXzvr4coRG5+xm+WFcPTePGHdJ7y6JeuaplRl1jq6ukcKyGcmJymQgKdjS04Kh7/mvMXE41FLkTWWNgVZmta2E2YYzhxyfHX7WqqrwYG+4721XzrV+dOxx7X1uCH510GF7WE3nz7zgNj8xeg827t+A3542IJmj1iLKqPP7SgoKj9dv7ycN6mqL4d26cgOqqMlSWBPG0QzRtZd7GxGZZCjH0+SKrdKkWSH97yXZjwlRI4/h8XeRcv3zTvvpVIm2TVQjLTYwvrEfeby7ahgdnrsbkYT2NyUqvzI99znG/m+3qnG1hDQdaQ8qZ08L5EX/nzfqdQf809H8CKHIniITw+Zjrmbjv3DgRA7pFf8A9KooNa0BeeLpnRTEmDq7Cny452vU4jhrQBSvvmYJThkWj5tm3nIRR/bsY9d5jXETuAPCYw2pUKq7SZ/4C5v4+b/zoBLyniO6BaHsFeY5BqoTcOnMXiNT5A1ExFZH7Ov3i89GqWuWFyLoYuPXPfMVx5rswKyt3NKD+QKuprHTtrkas3hmxd5Zu3YfFW/YaPfoPTVOAQ+JOEBnAeiEQST25djrg9+GFa4/F8Yd5yynIKyAV+X22ZPNoqeXzbVOGGo9/MGGg7Vh+H8NL10WWSZz3y1Mdz3n3eSMMga+QotTOnYIY2a8zeleWGNvuPMe8rm6DQ6voZIjV6qCpNYTVOxsND90LAR+zLXAytLe5F5F1WckHZ67C2N/Oxk/0eQZvLqrB6Y/MxV8+iqz2tX1/Cy742xeYuWInivw+03eVSkjcCSJDXD9pkNGPXni9bpJ0bll452mYf+dpMfeRbaefn3m48XiY7vuXFflxwmFV2PTAOegVR3RE+WCxopb/7RsnAABumjwY1544yNQOwu3M1UE9ynD31OG27c/9YLxtWyyve3dTG87801zMXeN9UZe79cZzgikjemNctflOyNoddf6myALsc1bXoXr6DNzymnpi09w1dehRUZzwxLx4kLgTRIb45dlHGJUul44bgMqSQMx6eq90Ly9G507qyou/XHY0rplojtRFG2QAxmIm1ij18cvHGI+HWVb0EuKuSkD2qizBl7dPNmr0e1bGrvcXHrWcb5gyojeumjAQf79ijJEU7lVZjDGHdDG99/JjY9skXujXpRNunDzEeC6WlBScPaoPhvWuNI1BdXFzi9sVvRIhqwlVxthUAFMHD46fxCKIQmJQj3Isu/vMjJ3vvKP6GouCyLz+w+OxckeDMQfA2jlSRl5qEYjedTgtyNKnczTPEG92rfC5e1SU4M+XHo3Ln5yHc/QL35SRfTCkVwVO/eMnCPh8pmP1riwxlUEmyzNXj0OPimI8NW0syosD2GuZ1CVWIbt47AAs2rIPgPmzDetdgVUOTdVUeF0kxgvUFZIgOjDjqrvhyuOr0VOfcu8USZ45opepZQMQ9bndLHYuZoee5tC6QLSH6FVZjAmDI7aQLNqi1YPPZ76YMAa06V0iAz5mLIwi6FIaxLf3nGm763BCfJJTj+iFYwd1t60BLO525DLYkoAflSUB9OvSCe/ffKKr82QCKoUkiA7Gg98dZasuObxXBXpUFON8i4CLPLCqo+3QXhWYgR3o31W9IpfMdn2W7hkjeitXLaruXorLjz3UcelGYev87LTDTdt9jOGgXoXzg4kDUVrkx59mrzVeP2tkb5QWBVzbH9ZktFXcxbwHOUEe9DMs+tXpYIzZWlIAwP+eNAj/+GQDqsqLbbXyL1xzrG3/VEGeO0F0MC4eO8AWhXcrK8L8O07DneeaE5iijv5yxSSpG04ZjDd/fIKrSVJnHxmxWORVr84+0jz79+JxA2zLKgpKiwLY9MA5tgVTHvruKKOj5vA+lfjJqVG/vF+XTrjn/JEAgD9fejQuHts/5qpbV51Qbatq6mQRd9U9CmMMAb/PSIz+5rwR6CrNOh2idw615goAYOKQKtu2VEGRO0EQjvSqLDH1yZfx+5jrHin3fmckpp81zLgDuHvqcKyRaswfuWR0QuM7YXAVOOfo27kExxza1STOn952ihFJD+9biQe/exTW7GrEv77cjJsmD8b8TXtxw0uL8L1j+mPT7iZcN2mQ7fhy0hmAadLd+Opu+HqTvZfOtBOqcdEx/THyrg8AuFt2Mh2QuBMEkXaCfp+RtF3z27NQFPDhdn126m+/MxKDenhrBPfeTRONKJ8xhrHV3YzX3r5hAnY1tCgtksN7VeDe70Si+bNG9sZf/+doTBnR2/D8rci2jPUi9/w1420LywhE9c/NkwebFpi5e+pwtIU13Pf+KjcfMylI3AmCyCjCtxbtiq2rZLkh1kIoR0mTtmLh8zGcO8peQSRjtWVkSoL+mFVA4mIwU++TwwFcpU8cO++ofq7GmAwk7gRBZIWfnDoEPSuK4wpsNinVxTuRC1AsendOz6xUGRJ3giCyQknQj6sVLRByiYDfh3u/MxITPLaEkIlVcZROSNwJgiBi4NRO2S1iHkCmE6sk7gRBEGnkpMN74kcnH4ZrJ2b2LoXEnSAIIo34fQy/mDIs4+elSUwEQRAFCIk7QRBEAULiThAEUYCQuBMEQRQgJO4EQRAFCIk7QRBEAULiThAEUYCQuBMEQRQgjGe64YFqEIzVAdic4NurANSncDiphsaXPLk+RhpfctD4EudQznkP1Qs5Ie7JwBhbwDkfm+1xOEHjS55cHyONLzlofOmBbBmCIIgChMSdIAiiACkEcX8i2wOIA40veXJ9jDS+5KDxpYG899wJgiAIO4UQuRMEQRAWSNwJgiAKkLwWd8bYFMbYasbYOsbY9CyN4WnGWC1jbLm0rRtjbBZjbK3+b1fptdv18a5mjJ2ZgfENYIzNYYytZIytYIz9JJfGyBgrYYx9zRhbqo/vN7k0PumcfsbYYsbYe7k2PsbYJsbYN4yxJYyxBTk4vi6MsTcYY6v0/w6Pz5XxMcaG6t+b+F8DY+ynuTK+pOCc5+X/APgBrAcwCEARgKUAhmdhHJMAjAGwXNr2IIDp+uPpAH6vPx6uj7MYwEB9/P40j68PgDH64woAa/Rx5MQYATAA5frjIIB5AI7LlfFJ47wFwEsA3svBv/EmAFWWbbk0vucAXKs/LgLQJZfGJ43TD2AngENzcXyeP0+2B5DEH+J4AB9Iz28HcHuWxlINs7ivBtBHf9wHwGrVGAF8AOD4DI/1bQCn5+IYAZQCWATg2FwaH4D+AD4EMFkS91wan0rcc2J8ACoBbIRevJFr47OM6QwAn+fq+Lz+L59tmX4AtkrPa/RtuUAvzvkOAND/7alvz+qYGWPVAI5GJDrOmTHqlscSALUAZnHOc2p8AP4E4DYAmrQtl8bHAfyXMbaQMXZ9jo1vEIA6AM/ottaTjLGyHBqfzKUAXtYf5+L4PJHP4s4U23K9rjNrY2aMlQP4N4Cfcs4bYu2q2JbWMXLOw5zz0YhEyOMZYyNj7J7R8THGzgVQyzlf6PYtim3p/htP4JyPAXAWgBsYY5Ni7Jvp8QUQsS0f55wfDaAJEZvDiaz8RhhjRQDOA/B6vF0V23JSd/JZ3GsADJCe9wewPUtjsbKLMdYHAPR/a/XtWRkzYyyIiLC/yDl/MxfHCACc830APgYwJYfGNwHAeYyxTQBeATCZMfZCDo0PnPPt+r+1AP4PwPgcGl8NgBr9bgwA3kBE7HNlfIKzACzinO/Sn+fa+DyTz+I+H8AQxthA/ap7KYB3sjwmwTsApumPpyHic4vtlzLGihljAwEMAfB1OgfCGGMAngKwknP+cK6NkTHWgzHWRX/cCcBpAFblyvg457dzzvtzzqsR+W/sI875FbkyPsZYGWOsQjxGxDdenivj45zvBLCVMTZU33QqgG9zZXwSlyFqyYhx5NL4vJNt0z/JBMjZiFR/rAdwR5bG8DKAHQDaEbmqXwOgOyIJuLX6v92k/e/Qx7sawFkZGN9ERG4blwFYov/v7FwZI4BRABbr41sO4Nf69pwYn2WsJyOaUM2J8SHiaS/V/7dC/A5yZXz6+UYDWKD/jd8C0DXHxlcKYDeAztK2nBlfov+j9gMEQRAFSD7bMgRBEIQDJO4EQRAFCIk7QRBEAULiThAEUYCQuBMEQRQgJO4EQRAFCIk7QRBEAfL/AS4KGKmbFFmpAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.display import clear_output\n",
    "\n",
    "\n",
    "epochs = 1\n",
    "model = simple_model\n",
    "model.to(device)\n",
    "opt = torch.optim.Adam(model.parameters())\n",
    "criterion = nn.MSELoss().to(device)\n",
    "\n",
    "history = []\n",
    "for epoch in range(epochs):\n",
    "    for batch, target in iterate_minibatches(data_train):\n",
    "        # Preprocessing the batch data and target\n",
    "        batch = torch.tensor(batch[\"FullDescription\"], dtype=torch.long, device=device)\n",
    "        target = torch.tensor(target, device=device)\n",
    "\n",
    "        predictions = model(batch)\n",
    "        predictions = predictions.flatten()\n",
    "\n",
    "        opt.zero_grad()\n",
    "        # Compute loss.\n",
    "\n",
    "        loss = criterion(predictions, target)\n",
    "        \n",
    "        loss.backward()\n",
    "        opt.step()\n",
    "        # Train with backprop.\n",
    "\n",
    "        history.append(loss.item())\n",
    "        if len(history) % 10 == 0:\n",
    "            clear_output(wait=True)\n",
    "            plt.plot(history, label=\"loss\")\n",
    "            plt.yscale('log')\n",
    "            plt.legend()\n",
    "            plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "eOXdOBCnpHfe"
   },
   "source": [
    "To evaluate the model we can switch it to `eval` state. Let's check the model quality."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "id": "FVCKX0MJ4Lql"
   },
   "outputs": [],
   "source": [
    "from tqdm.auto import tqdm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "referenced_widgets": [
      "ec8af72fd11547599bb640ac661c7ee8",
      "5bffe7c135594b3f9aadeaaad0df9f87"
     ]
    },
    "id": "eaiQRrfCpHfe",
    "outputId": "a1f9eb76-4c5d-422d-f7cd-83218ba8eefb"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ec8af72fd11547599bb640ac661c7ee8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train:\n",
      "MSE: 373421321.47838\n",
      "MAE: 13119.37163\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5bffe7c135594b3f9aadeaaad0df9f87",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Val:\n",
      "MSE: 384519130.68301\n",
      "MAE: 13219.02248\n"
     ]
    }
   ],
   "source": [
    "@torch.no_grad()\n",
    "def evaluate_model(model, data, batch_size=256, name=None, **kwargs):\n",
    "    squared_error = abs_error = num_samples = 0.0\n",
    "    data_iterator = iterate_minibatches(data, batch_size, shuffle=False, **kwargs)\n",
    "    for batch, target in tqdm(data_iterator):\n",
    "        batch = torch.tensor(batch[\"FullDescription\"], dtype=torch.long, device=device)\n",
    "        pred = model(batch)[:, 0].cpu().numpy()\n",
    "\n",
    "        squared_error += np.sum(np.square(np.exp(pred) - np.exp(target)))\n",
    "        abs_error += np.sum(np.abs(np.exp(pred) - np.exp(target)))\n",
    "        num_samples += len(target)\n",
    "\n",
    "    if name is not None:\n",
    "        print(f\"{name}:\")\n",
    "\n",
    "    print(f\"MSE: {squared_error / num_samples:.5f}\")\n",
    "    print(f\"MAE: {abs_error / num_samples:.5f}\")\n",
    "    \n",
    "    return squared_error / num_samples, abs_error / num_samples\n",
    "\n",
    "\n",
    "train_errors = evaluate_model(simple_model, data_train, name=\"Train\")\n",
    "test_errors = evaluate_model(simple_model, data_val, name=\"Val\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "yBlasvvu4Lqm",
    "outputId": "2c037ff4-e737-4224-acc1-76d293f7f272"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "19609.159356867138"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.sqrt(384519130.68301)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hUUchHCnpHfe"
   },
   "source": [
    "### Bonus area 1: three-headed network.\n",
    "\n",
    "Now you can try to implement the network we've discussed above. Use [__PyTorch nn.Module API__](https://pytorch.org/docs/stable/nn.html#torch.nn.Module)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "LrSrdSe_4Lqm"
   },
   "outputs": [],
   "source": [
    "iterator = iterate_minibatches(data_train, 3)\n",
    "batch, target = next(iterator)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "i2NcHN554Lqm",
    "outputId": "97dcfeef-2204-48fc-e569-181c886983da"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([10.846966,  9.947122, 11.248973], dtype=float32)"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "id": "SOG-2OK2pHfe"
   },
   "outputs": [],
   "source": [
    "class ThreeInputsNet(nn.Module):\n",
    "    def __init__(\n",
    "        self,\n",
    "        n_tokens=len(tokens),\n",
    "        n_cat_features=len(categorical_vectorizer.vocabulary_),\n",
    "        hid_size=64,\n",
    "    ):\n",
    "        super().__init__()\n",
    "        \n",
    "        # Define modules to process the title.\n",
    "        self.title_emb = nn.Embedding(n_tokens, hid_size)\n",
    "        self.title_enc = nn.LSTM(hid_size, hid_size, 2, batch_first=True)\n",
    "        \n",
    "        # Define modules to process the description.\n",
    "        self.desc_enc = nn.Sequential(\n",
    "            nn.Embedding(n_tokens, hid_size),\n",
    "            Reorder(),\n",
    "            nn.Conv1d(hid_size, hid_size, kernel_size=3),\n",
    "            nn.ReLU(),\n",
    "            nn.Conv1d(hid_size, hid_size, kernel_size=3),\n",
    "            nn.ReLU(),\n",
    "            nn.Conv1d(hid_size, hid_size, kernel_size=3),\n",
    "            nn.BatchNorm1d(hid_size),\n",
    "            nn.ReLU(),\n",
    "            nn.Conv1d(hid_size, hid_size, kernel_size=3),\n",
    "            nn.BatchNorm1d(hid_size),\n",
    "            nn.ReLU(),\n",
    "            nn.AdaptiveMaxPool1d(1),\n",
    "            nn.Flatten()\n",
    "        )\n",
    "        \n",
    "        # Define modules to process the categorical features.\n",
    "        # self.category_out = ...\n",
    "        self.cat_enc = nn.Sequential(\n",
    "            nn.Linear(n_cat_features, hid_size),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(hid_size, hid_size),\n",
    "            nn.ReLU(),\n",
    "        )\n",
    "\n",
    "        # Define fully-connected part which will take outputs of\n",
    "        # three heads and return the result.\n",
    "        self.out = nn.Linear(3 * hid_size, 1)\n",
    "        \n",
    "    def forward(self, title, desc, category):\n",
    "        \n",
    "        # YOUR CODE HERE\n",
    "        # Process the title.\n",
    "        # title = ...\n",
    "        title_beg = self.title_emb(title)\n",
    "        lstm_out, (title_enc, _) = self.title_enc(title_beg)\n",
    "        \n",
    "        assert torch.equal(lstm_out[:, -1],title_enc[-1])\n",
    "        \n",
    "        title_enc = title_enc[-1]\n",
    "\n",
    "        # YOUR CODE HERE\n",
    "        # Process the description.\n",
    "        # desc = ...\n",
    "        desc_enc = self.desc_enc(desc)\n",
    "\n",
    "        # YOUR CODE HERE\n",
    "        # Process the categorical features.\n",
    "        # category = ...\n",
    "        category_enc = self.cat_enc(category)\n",
    "        \n",
    "        #print(title_enc.shape, desc_enc.shape, category_enc.shape)\n",
    "        \n",
    "        concatenated = torch.cat(\n",
    "            [\n",
    "                title_enc.view(title_enc.size(0), -1),\n",
    "                desc_enc.view(desc_enc.size(0), -1),\n",
    "                category_enc.view(category_enc.size(0), -1),\n",
    "            ],\n",
    "            dim=1,\n",
    "        )\n",
    "\n",
    "        # YOUR CODE HERE\n",
    "        # Process the concatenated features to generate network output.\n",
    "        # out = ...\n",
    "        out = self.out(concatenated)\n",
    "\n",
    "        return out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "LTGWuSU14Lqn",
    "outputId": "b7c8077f-c496-4dac-e0a3-6d56bbe3c4f8"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ThreeInputsNet(\n",
       "  (title_emb): Embedding(34158, 64)\n",
       "  (title_enc): LSTM(64, 64, num_layers=2, batch_first=True)\n",
       "  (desc_enc): Sequential(\n",
       "    (0): Embedding(34158, 64)\n",
       "    (1): Reorder()\n",
       "    (2): Conv1d(64, 64, kernel_size=(3,), stride=(1,))\n",
       "    (3): ReLU()\n",
       "    (4): Conv1d(64, 64, kernel_size=(3,), stride=(1,))\n",
       "    (5): ReLU()\n",
       "    (6): Conv1d(64, 64, kernel_size=(3,), stride=(1,))\n",
       "    (7): BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    (8): ReLU()\n",
       "    (9): Conv1d(64, 64, kernel_size=(3,), stride=(1,))\n",
       "    (10): BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    (11): ReLU()\n",
       "    (12): AdaptiveMaxPool1d(output_size=1)\n",
       "    (13): Flatten(start_dim=1, end_dim=-1)\n",
       "  )\n",
       "  (cat_enc): Sequential(\n",
       "    (0): Linear(in_features=3768, out_features=64, bias=True)\n",
       "    (1): ReLU()\n",
       "    (2): Linear(in_features=64, out_features=64, bias=True)\n",
       "    (3): ReLU()\n",
       "  )\n",
       "  (out): Linear(in_features=192, out_features=1, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 169,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = ThreeInputsNet()\n",
    "model.to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "hicVrSMS4Lqn"
   },
   "outputs": [],
   "source": [
    "iterator = iterate_minibatches(data_train, 3)\n",
    "batch, target = next(iterator)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "S68f73Tr4Lqn"
   },
   "outputs": [],
   "source": [
    "title = torch.tensor(batch[\"Title\"], dtype=torch.long, device=device)\n",
    "desc = torch.tensor(batch[\"FullDescription\"], dtype=torch.long, device=device)\n",
    "category = torch.tensor(batch[\"Categorical\"], dtype=torch.float32, device=device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "JsuyHvdY4Lqn",
    "outputId": "b5a11039-8997-4268-87ff-6301273ecb37"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.2095],\n",
       "        [-0.0123],\n",
       "        [ 0.2898]], device='cuda:0', grad_fn=<AddmmBackward0>)"
      ]
     },
     "execution_count": 172,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model(title, desc, category)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "bIpYeZBK4Lqn",
    "outputId": "0acb401f-23d9-4e46-9003-06a752553676"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAD4CAYAAADvsV2wAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAA6gElEQVR4nO2dd5gUVfb3v6fD5AAMmQEGBJWgBAEziLoKKpgDrnFRVl386evqLq6uYVmVNevKqphzWHUNgIABBQXJOWcY0gwDE5jc3ff9o7qqq6qruqunw9R0n8/z8NB9u7rqdE/X95577rnnkhACDMMwTPLjaG4DGIZhmMTAgs8wDJMisOAzDMOkCCz4DMMwKQILPsMwTIrgam4DQtG2bVtRVFTU3GYwDMO0GJYtW3ZICNHO6DVbC35RURGWLl3a3GYwDMO0GIhol9lrHNJhGIZJEVjwGYZhUgQWfIZhmBTB1jF8hmGYaGlsbERxcTHq6uqa25SYkpGRgcLCQrjdbsvvYcFnGCapKS4uRm5uLoqKikBEzW1OTBBCoKysDMXFxejRo4fl93FIh2GYpKaurg4FBQVJI/YAQEQoKCiIeNTCgs8wTNKTTGIv05TPlJSC/9avOzB99b7mNoNhGMZWJEzwiegSInqNiL4iovPiea0PF+3GjNX743kJhmEYy+Tk5DS3CQCiFHwiepOISohora59FBFtIqKtRDQJAIQQXwohbgVwE4Cro7luOLLSnKhp8MbzEgzDMC2OaD38twGMUjcQkRPAVACjAfQFMI6I+qoOedD/etzISnOhpsETz0swDMNEjBAC9913H/r3748TTjgBn3zyCQBg//79GD58OAYOHIj+/ftj/vz58Hq9uOmmm5Rjn3vuuaivH1VaphBiHhEV6ZqHAdgqhNgOAET0MYCLiWgDgCkAvhVCLI/muuHISnNif0VjPC/BMEwL5NFv1mH9vsqYnrNv5zw8PKafpWO/+OILrFy5EqtWrcKhQ4cwdOhQDB8+HB9++CHOP/98PPDAA/B6vaipqcHKlSuxd+9erF0rBVDKy8ujtjUeMfwuAPaonhf72+4EcC6AK4joNrM3E9EEIlpKREtLS0ubZEBWugu1jRzSYRjGXvzyyy8YN24cnE4nOnTogBEjRmDJkiUYOnQo3nrrLTzyyCNYs2YNcnNz0bNnT2zfvh133nknZs2ahby8vKivH4+FV0a5QkII8SKAF8O9WQgxDcA0ABgyZEiTdljPTnOiup5DOgzDaLHqiccLIYwlbfjw4Zg3bx5mzJiB66+/Hvfddx9uuOEGrFq1CrNnz8bUqVPx6aef4s0334zq+vHw8IsBdFU9LwSQ0BzJzDQnannSlmEYmzF8+HB88skn8Hq9KC0txbx58zBs2DDs2rUL7du3x6233orx48dj+fLlOHToEHw+Hy6//HJMnjwZy5dHHwmPh4e/BEBvIuoBYC+AawBcG8kJiGgMgDG9evVqkgHZaS5UN3gghEjKBRcMw7RMLr30UixcuBADBgwAEeHJJ59Ex44d8c477+Cpp56C2+1GTk4O3n33Xezduxc333wzfD4fAOCJJ56I+vpkNsSw9GaijwCcBaAtgIMAHhZCvEFEFwB4HoATwJtCiMeacv4hQ4aIpmyAMnXuVjw1exM2Th6FDLezKZdmGCZJ2LBhA/r06dPcZsQFo89GRMuEEEOMjo82S2ecSftMADOjOXc0ZKdJIl/T4GXBZxiG8WPL0gpENIaIplVUVDTp/VnpUj/GE7cMwzABbCn4QohvhBAT8vPzm/T+LL+Hz6mZDMMA5tkxLZmmfCZbCn60ZKexh88wjERGRgbKysqSSvTlevgZGRkRvS8pN0DJlD18Ts1kmJSnsLAQxcXFaOpCTrsi73gVCbYU/FikZQJANQs+w6Q8brc7ol2hkhlbhnSijuGny1k6HNJhGIaRsaXgR0uWKi2TYRiGkUhSwedJW4ZhGD22FPxo8/Az3NLHqvf4YmkWwzBMi8aWgh9tDD/NyYLPMAyjx5aCHy1EhDSnAw0s+AzDMApJKfgAkOZiwWcYhlGT1IJf7+EsHYZhGBlbCn60k7YAOKTDMAyjw5aCH+2kLQCkux1o8LLgMwzDyNhS8GMBe/gMwzBaklfwedKWYRhGQ3ILPod0GIZhFJJX8J0OXnjFMAyjwpaCH5MsHQ7pMAzDaLCl4MckS4cFn2EYRoMtBT8W8MIrhmEYLUkr+OkuJ8fwGYZhVCSt4GemOVHXyB4+wzCMTNIKfpbbyTteMQzDqEhewU9zorbRCyFEc5vCMAxjC5JW8DPTXBACqGvkOD7DMAxgU8GPRR6+vJF5dQPva8swDAPYVPBjkYef6Rf8Wo7jMwzDALCp4McC2cPniVuGYRiJFBB8DukwDMMASSz4mW4XAA7pMAzDyCSt4HNIh2EYRkvyCz6vtmUYhgGQzIKfLod0OIbPMAwDJLPguzmkwzAMoyZpBT+TY/gMwzAaklbw010OOIizdBiGYWRsKfixKK1ARMhKc7GHzzAM48eWgh+L0gqAFNapbeRJW4ZhGMCmgh8rstK4Jj7DMIxMUgt+usuBei6PzDAMAyDJBd/pcMDj4w1QGIZhgCQXfLeT4PGxh88wDAMkueC7HASPlz18hmEYIOkF38EePsMwjJ/kFnwne/gMwzAySS74DjTypC3DMAyAZBd8B8HLIR2GYRgAKSD4HNJhGIaRSGrBdzsdaPSyh88wDAMkueC7nAQvx/AZhmEAJLngOx2ERg7pMAzDAEhywXdzHj7DMIxCwgSfiHoS0RtE9FmirskhHYZhmABRCT4RvUlEJUS0Vtc+iog2EdFWIpoEAEKI7UKI8dFcL1JcHNJhGIZRiNbDfxvAKHUDETkBTAUwGkBfAOOIqG+U12kSLqcDHs7SYRiGARCl4Ash5gE4rGseBmCr36NvAPAxgIutnpOIJhDRUiJaWlpaGo15UmkFDukwDMMAiE8MvwuAParnxQC6EFEBEb0CYBAR3W/2ZiHENCHEECHEkHbt2kVliMvBgs8wDCPjisM5yaBNCCHKANwWh+uZ4nI44PUJCCFAZGQWwzBM6hAPD78YQFfV80IA+yI5ARGNIaJpFRUVURnidkoizxO3DMMw8RH8JQB6E1EPIkoDcA2AryM5gRDiGyHEhPz8/KgMcTqkj7et9GhU52EYhkkGok3L/AjAQgDHEVExEY0XQngATAQwG8AGAJ8KIdZFb2rkHKysAwDc8cHy5rg8wzCMrYgqhi+EGGfSPhPAzGjOHQuuP7U73l6wE3WN3uY2hWEYptmxZWmFWMXwj2mXgytPKjScRWYYhkk1bCn4sYrhA1IBNa/gSVuGYRhbCn4scTgIvNiWYRjGpoIfq5AOADgI8LGHzzAMY0/Bj2lIh7hiJsMwDGBTwY8lDgfBx4LPMAyT/ILvJJ60ZRiGAVJB8B0c0mEYhgFsKvgxnbR1EE/aMgzDwKaCH49J26U7D6OyrjEG1jEMw7RMbCn4sUTy8IErXlmI299f1tzmMAzDNBtJL/hOVR38dfsqm9EShmGY5iX5BV/1CV0OrqrDMEzqYkvBj/WkrYyTBZ9hmBTGloIf60lbGZfDlh+XYRgmISS9AjrZw2cYhgGQAoLv0Hj4LPgMw6QuKSD4gcfs4TMMk8okveBzSIdhGEbCloIfrywdl9NY8OduKsGyXYejvhbDMIydsaXgxytLx2mSpXPzW0tw+csLo74WwzCMnbGl4McStYfPAR2GYVKZpBd8tYfPMAyTyiS/4Ks8/HBFkqvrPfE1hmEYphlJesFXh3QQpi5+dQMLPsMwyUvSC746pNPoDePj8z4pDMMkMckv+KpP6PH5Qh7LOyEyDJPM2FLwY5qHH4GHz1shMgyTzNhS8GOah6+K4Td4Qnv4LPcMwyQzthT8WKKetA0b0uGYDsMwSUzSC364SVu1yHNEh2GYZCbpBV8TwzcI6XhUgs8xfIZhkpnkF3zVJ2xUhXS2llShaNIMLNxeprSx3DMMk8wkveCbhXQWbJOE/vNlxUobe/gMwyQzSS/4aa7AR/T6BLz+EI68v21to1d5XbDgMwyTxLia24B4k5Ou/YiLdxzGwco6pTZ+nUbwE2oawzBMQkl6wc/WCf64134DADx/9UAAQG1DQPA5K5NhmGQm6UM6esGXcTuDQzocw2cYJpmxpeDHsrRCdprTsF1ej6X28FnvGYZJZmwp+LEsreByGn9EOf+ePXyGYVIFWwp+IvAaCD7rPcMwyUzKCr4s9NpJW1Z8hmGSl5QQ/H9e0h9XnlSoabv/izUAgHpVuQXe4pBhmGQmJQT/ulO6Y1T/jmGPu/b1RQmwhmEYpnlICcEHAmmYDMMwqUrKqCALPsMwqU7KqGCai8IfBKBo0gw8//3mOFvDMAyTeFJG8HPS3ZaPffXn7XG0hGEYpnlIGcHPy7ReNijTZHUuwzBMSyZlBD83w7qHn+lmwWcYJvlIGcE3q6ljBHv4DMMkIykj+ETWJm0BIIsFn2GYJCRlBD8SOKSTWJbtOoyyo/XNbQbDJD0pJfhjBnRGq6zwsXz28BPL5S8vxGUvL2huMxgm6UmY4BNRNhG9Q0SvEdHvE3VdNf8eNwh3nt077HHpLhb8RLOrrKa5TWgWZq3dj6dmb2xuM5gUISrBJ6I3iaiEiNbq2kcR0SYi2kpEk/zNlwH4TAhxK4Cx0Vw3GgpbZ4Y9xumwHu9nmGi47f3lmDp3W0zPuausGrtTtANlQhOth/82gFHqBiJyApgKYDSAvgDGEVFfAIUA9vgP86KZOL9fR/Tvkhf6INZ7pgUz4qmfMPypuc1tBmNDohJ8IcQ8AId1zcMAbBVCbBdCNAD4GMDFAIohiX7I6xLRBCJaSkRLS0tLozHPlFN6FGie6x16RwQZPUx0CN6DgGESRjxi+F0Q8OQBSei7APgCwOVE9DKAb8zeLISYJoQYIoQY0q5duziYBzidWkFPc2m/hlV7ynHx1F9R19hsA5GUIRZ6L4TAf37aisPVDdGfjGGSGOv1Bqxj5B4LIUQ1gJvjcL2IcTu0Au92OlDXGNgIZffhGuw+XINNB6owoGurBFuXWsRil7ElO4/gyVmbsGJ3OV67YUgMrGKY5CQeHn4xgK6q54UA9kVyAiIaQ0TTKioqYmqYjEvn4ae7jL8GL4cb4o4vBl9xo1fqrHnHMoYJTTwEfwmA3kTUg4jSAFwD4OtITiCE+EYIMSE/Pz8O5gEuXdDerFa+LxZqZEO+W38QFTWNzW0GAN5HmGESSbRpmR8BWAjgOCIqJqLxQggPgIkAZgPYAOBTIcS66E2NHS6dwJulYSaj3pdU1eHWd5di4kfLm9sUALGJ4cfjXAyTjEQVwxdCjDNpnwlgZlPPS0RjAIzp1atXU08REr2Hbyb43iRU/LoGKfyxvbS6mS2RiIWHzzlVDGMNW5ZWiHdIJ0NXK8csDdNMjLaVHsWuMnsIZqTUeaTMI7tknsYypCOQfB00w8QSWwp+vDm/X0cAwImFUodiJn4eEw//nGd+xoinfoqHaVFRdrQef/50VcjJS/k1u4Q/YjKIsknnxTB2JyUFv11uOnY8cQEuHyytAzPz8Bs8PsN2u/L0nM34fHkxZqzeb3pMTYO91hbwwiuGSRzxyMOPmnjH8P3XUKpierzGwt5SBH9rSRUqaj044l94lJ1u/me1W+oi6z3DJA5bevjxjuHLyNseqhddqWnwNs0bPlhZh8dnbkjYSt17Pl2Fy19egHX7pXUL+nUGauzm4XNaJsMkDlsKfqKQNzaXJzL1NNXD/2DRbkybtx1v/bqzqaZFxOpiSej3HK4FgJAdTXWDvTz8JEyEYhjbktqC7/fw6808/CYKfpdWGQCA1cXlTXp/pOizSkMJ/tG6yAR/w/7KuMbZOYYvwd8DkwhsKfjxLq0gIwu+mYf/j+nrm3Re2Ws9mqB4uX6/XrMQFRAYDVTWNYYVmTnrDmD0C/Px1cqIKmNERCw9fCuaWdvgxWvztttujQXrPZMIbCn4iYvhu/zXM3690du0u1AWk/lbDuG/S/eEOTr21Ibw8NfukwS/qs6Dg5Wh95HdUnIUALDxQFXsjNOR6Bj+c99vxmMzN+CbVdY7seIjNXj/t11xtIrnMpjEYEvBTxSy4F8ysHNMz6u+eX/cWBLTcxuhn6KtDTEx6/EKtM9NByBVBQ2FnK4azwVNsVlpaz0Rv7JWqiEUyeT1ta8twoNfrkVVXfzqD9lswMEkKbZMy0wULqcDGyePQnW9B182IWxxsLIOHfIygtrV4QJHArZL1C8jMAtRAZLAyumojSbpqEHnjaMYxbSWTuxOpaHsaH1czy+dmxWfiT8p7eEDUpkFs1o64bT6kqm/GrarBV9ftyce6D3cuhDeq9cnlA1fwgm+TDylqCWEMhJhYQv4GpgkIOUFHwie9JTxCfNFWQCwv6LOsF0t+GadSXW9x7LghkV3iVDhCp8A0l3ygrPQKiObHs8y0bE4daIyXEQc1+G1hI6PafnYUvATlaUjE8oJn/jhCs1zK+Ki3jjFadKZ9Ht4Nia8u9SSfe8t3IkXf9hi+JoQIqjjOFJjvtWfEAEP3+MLE9JJQJGaWIh1JJ1GU4rGySbGc0McjuEzicCWgp+oLB0ZMw8fAGatO6B5buXGVHvEoVa9zt1Uasl7/vtX6/Dsd5sNX2vw+oLCAaVHzQXfK4Syw1e4LCT5a4lvSKfp7y2vaUDP+2dg/tb4bHYvI8fX4+mFs4fPJAJbCn6iiSTMHs4rBgC1w20W0pF5a8FO6xc3oN5gcdjWg1W459OVhnVzfD7rHr7yHpsuvFpdXAGfAF79ebv/ZDEySodsYjxDW/EMFzGMDAs+zKtlylTUBtLxwsW9AWshHZnlu4+EPV8o1hQHh72qG7z4YvlefLasOOg1nwDSnFY9/PiHdKLR0ET7xPEN6bCHz8QfFnyEj+tWqgVfp1BFk2bg0FHtAiZfmLRMtVfbKtNt2U4jb3jR9jIAUHLr1Rhtzu5TxfDDrTZVsjLjqEXRCF3CJmv9/8czzm711F+v2oc9YdZPMIwZLPgITE6aRV/UK1eNsnb0u195wqRlqoWjVVZowVfX4zEqmdDgFUhzOpBv0HFkpjmD2rw+ocrSsZaHH09hjU7wY2hIyOv4Y/hxzVaydu7/+2gFxrz0S9zsCMXReo9mtMu0PGwp+InO0pGFzSyEoV65auQVu3WboqtvXqcj+CtWn0Ou56Pn0NF6LNxWhrEvBXL9q+qDb7YGjw9pLodig1s1SSwLuxohoHj420qrE1bgzYxoRDvRi5XiWX/HiuDLHU95TdNFd/2+SmXfhEg5afJ3GPDonCZfOxYcrKzD6/O3c7G5JmJLwU90lo4cZ58wvKfSNuWyE5THag+/0eCmV0/M7jhUjVlrA5k9RqMGtXCYacg1037DuNd+07QZVbps9MqCL10oU7Vfr1FIR52l8/aCnZoORU+gtEL8aBkevvR/PGP4Vk4diw7nghfnY+zUpo0QjBIEEs3t7y/DP2dswI5DLXNP6ebGloKfaBwOws4pF+Kvo45X2tLdga+mttGLZ+dswq3vLoXXYKJTfSOe99zPmho1RveoOjtm2a7DKKkKXsC11V+4TM1LP25VlvnLNHh8cDtJ8fDVu10ZialPJfjhCIR0LB3eJKLRsPHvWFvHoCXy+kDykc0d2rLS4VixUd43oSVS7g8p8SR302DBN0Ed3r75rSV48cet+G79QcNURnW2iz7zxWtwvLrp+w0luOAFax7XFyv24u5PVmraGvwevpzvn6WK2+snmIUQmpCOVRJVPC3cnEI44mWnLKKxWhhthLX1HbE5T3LAO9c3BRZ8HWf2bgvAWKgB44nTUCUSjF7Sdxr6LB/APHOotErn4Xt9SHM6lIlnjYevu/vlpy6DeQU1h6sb4PH6lDmNeDpT6nOfNuVHf5vAVyv3RrxFZLydvnjG8CNdwW0Ge75MKFK6WqYRb988DF6fQLlJeQIjcTbLzW+V5TbsOKwIh8tBhnny+s5FCuk4lA4ilIcvX9fpkOYW5JeFEIq41zZ4MXjyd7j+lO44tkOO9HpYa5uOWuhK/J3Zyj3luOvjlbhqSCGevGJAHK9ujUBaZjPH8C2sAQllYzQhqUTtzxwW7s+igj18HU4HIc3lQPu8DKx6+Lyg140mixpNRgNOoiDRBax5amaLwQ5W1uPXrYcC1/b6NDH5rDTzGL78nIg0Q3/1ZFyNf8/b6av3qTqEsOY2GaO+r8o/Of3LlkPBL/pJZJaGstK2BcTwQ4V9mjpCmbPuAI7/+6wmvTfWyJ8gAWsCkxIW/BCoM15knpy1Mait0S+YczdpNztxOsjwJrOyWtesrPLReg9+//oi5bx6D1+de6+/jiwq+nIPZnv6ytcIJ67frNqHAyaVQ8NhJHSH/WmDDSFCZc0Rq45vWmZsrh+q42hqllEiNvGJFI5cNQ1bCn6i8/DNcDsJOenaqFe1Qelh2Yu/+a0lmnaXieBbuXHDbZwip4rKefiyZ5ersld/Hfmp/tTqtFNtymh4OxfvOIw7P1qBZ+ZsCnusEfpr/La9DAcrpc4j12SNAmC9DlAsiWcnY8nDj1bwLbx/8Y7D2FcuZfHsOFSNsS/9gqowezMv2HoIRZNmYMP+yrDnjxbZ+eA8/KZhS8FPdB6+GUSE1QZhHT1mk7ZOp4ngW/ixhiu6tulAFb5YXqzk4cu7XLXJTjO9jmyLPlykjs/KnZdAQDxCmbvOv0eubK8QAqc8/gM+WbI7pP0y+nNfM+03PPGtNIoKlT6aqE3I5RAX0PxF5KxN2oZ4v4Xv7KpXF+K0KT+i0evD2r0VWF1cgeIwpRxmrt0PAFiy83DY88eKN37ZgfoQO7sxxthS8O2ElS0KzUI0pjF8CzdeuKJrl7+8APd8ugpV9R64nQ4lLFOQE6ipY5SWCRgIvkddOiIg8nI/FkroGvzhLPl78vgEDlTW4a+frwlpv0yoc4fKfjL6Xq0Q6doCpRIn4p2lE/4YK6UdQnUckdg/5duNysgv3Hct/5kSUWxP5uMle/DC98Z7RDDmsODHgEavzzCHPJoYfjgPX6bsaAPSXA7F22mtqs1jlpapP/fYl37Fgm3SBKl6AtqKRyuLsnxGK5/NyCYjQsXwjTJWDhosYDPDakhAbUNz7/xlpZOL1sOXWbWnXBn5hfubyt9lOCdFPvZfszZiW2nwwkIrqC2RQ0+MdVjwm8Dgbq0AANed0g2AVG5hu0H2jsvhQElVHXo/MBOLdwSGu0ZCet3ri3TvtSb4FbWNSHc6lPUBGaqJ5oraRkyevl65cQMhHe05Gjw+TPJ75OqJWp8qvGOG7OHLWmKWsWSGldGDEUbit+dwLZZaDCtYFT+1iLWEGH6oY8K9X9+hyb+bo2Fi+D5h/LsyYn9FHV7+aVvQfFdTCGdXS+L1+dvx2Iz1cb8OC74Fnr1qAF657iTlubxw6Yxe7QBIK0SfmLkh6H1OB2HJziNo9Apc9epCCCEwc81+/PG9ZUHH/rL1EMZNC9TOsRJKklF7+P075+P8fh0ASKUY3vhlBz5eLMXTlZCOwbnlyWl1GMVrIYZf7z/eo/wfmSqG8rRDCb6ZeH279oBhux6r4q3+rpq7Hn4kBdb0j4Hw9qtHMwJAbYP0PNSWmUAgpGPlNyv3n7GIv1ca1JZqqfxzxga8Nn9H3K/Dgm+BywYXYlT/jvh64umYdfeZ6Ns5DwDQIU+Kl9d7fIYevj508tv2w7jjg+XYazIUXbi9DHWNXtQ0eFB8xPpwtX1uuuLh52e58fzVgwAEbuClu46gtsGr3PBGOf45GZLgKzF8BDw+K6IsXyvSjdlDaVBoD9/4tTd+2YGXfgwf27U6AaseacV1x6swpz5YWYfznpsX9jxqE/XnDNcZ60Nocgy/RpeZpv49vPnLDvzi32LSSkhHXhHe1K9S/ZmqkkjwEwULfgScWNgKx3fMw98u6IP/3XEa+nSShP+9hbuwqyw4k0Ev+LWN4X+gh6sb8K9vg3P9Q9EhPwOXDe4CAMhOcwZdd/rq/XjgyzWBGL7BjZknC74spCJwU5rFjreVHsVbv+4EEJjHMNts3YxQN36o6oyhwhNPzzHe/1d7XWve8juqLSibc+HVvM3G+/ZW13s0sWz1efTnDBfS0XewZqtr1af5x/T1OFgprZAOU7FDsiGGewscNSgXzoSGBb8JpLkcGNSttVKhUvbYs3Qbjujj8GSh4NPXq/Zhf4SLmDrmZeDhMf2w9tHz4XI6DCd81+2tVG4youCVinLsX+0Fyjen3AnsLqtB7wdmYvPBKgDQrPidueYAJk9fjw8WWUvHlAkldB6fMBWGpmbpyFgZiOyrqEOZqnZ8c2fpGHHlKwuVGkSA9vvUnzJcSEc9OhNCmAq+2egq3FahQGCy3ezvvmzXkZATuuoCeXsO16Jo0gzL8zYMC35U6IX1v7edqnmeoVupe/Pb4Seqpny7ET+beHNGtM1JQ9/OeXA6AovEjEKpGW6HIpJOBwV1Rt9vOIiVe8o1QirflPKQfsaa/Wj0Cnzu3ytXX4TtnYW7LNsti4tZuEjed/fsZ37C6BfmB70erfhaipkHZTk1bwzfiPW6xU7q00Tr4deaCP7Binr4fMJgYZ/Aewt3osf9M0yvJXcWZq9f/vICnPPMz6Y2Gn1NX63cZ3o8o4UFP0oGFOYrj4/vmKc8vum0Is0iKCMeuqivYXskG00sffB36JSfqWkjChZ0EGHsv6UyzA4ig8VXPlwy9deAECMgePKevnIhOHlyzmom0bzNpZqc6a0lR9H7gW8xa+1+05BOrj/EtLOsJmgF55KdhyMeBcnIFlsJKej/DvpRwao95SiaNAP/XbonYjv0abyxGjxoPHzdOcMJfqNu0tbMwx/+1Fy8/PO2oA7B4xV4+Ot1EML8vYEssJCmRESostjxnHdpidhS8O1SWsEKX008AwDQs202nA7C2ce3x9RrB+ORsf1QkBNa8IcUtcZTV5wY8TV7tZeqWHZplWl6jD5jYtWecmWJvMPAw5dRi4J8/y/fXY61eyvQ4B+Oy++1ulbghjcX47nvpbj63vJaXP3qQgBSRo2ZZ5tnsrn7F8uLceUrC3Hjm4sBBEYCesJlgVjRAb1oqW2dsXo/Lp4q7Rb27Hfh5wzUvD5/O3o98C2q6gIx6GhLBcjCpv77RSr4+g6u1qTGEgAs3FaGlbvLNW2v/LxN+V7NJtzlEWQsM57MPtamA1Xo+beZQTWuUhlblkcWQnwD4JshQ4bc2ty2WGHNI+cpHvObNw1V2vUxfT1OBxluNB6KX/46EnmZbpRU1qF1lnmH4nIQ5OhzmsuhuQEdZJ5CJ5dkPlrvwX+XBTzXi/4d2KRFFoamLKz862erlbi4EOY3qzyJrOeeT1dpj8t04dDR4LTBo3UepOcEvtu95bVo9PgUX9CK4BQf0U7EqwV/S0mV8tjljOyL+OcMKYV356HQO6NFQoPXhwyHU3OeSEM66nLcK3aXY1hRG9Nj22Sn4bo3tGtHtpUGMtXMRqmyDVZDWAu3laFVlltJkDB6m9mp5MWEczeWYORx7S1dL9mxpeC3NMyKfFXXh/YyXQ6HZgP0L/90OpbvOoJ/TDdfgFHYOguA+ebnMnImzu9P7oaSqnp8t/6g5rXe7XOwXOehAdoJObO0N3mvgFBpkzLyGgDAn8nj04YNzDxbdYmIUBht1A5IHZb6HKf7JzavGdpVurbJdWsbvPj3j1vgFUJTVgHQCqZ6hOQOk54ihMA/pq9H9zZZ+Hz5XqV9zxG14Een+Mf/fRbm/2VkyOJ3RhPdjV4fJk9fjz+OOCbo77lYNRl64QmdcFL31spvM9w2mWYjLNkGq+vz5H2dd065MMRRxt+dPPekLhme6vA3EUf+NLIXcjNcmHh2Lxz3oFRPPD/TjQp/TNzlJGW7weHHtsPArq2w+YDkOeZluKJaWCLfAu1UOfoyRIQ3bhyKZbuO4JZ3tfvCTvxwRdhzl9dI9oeba9iwvxKTvgjU1Klp8Gomen1CmArdyT3aWCrLazbKqKrzwOsTKKuu14ig3PF5fQJPztqIEwvzMap/J+X133aU4T8/bTM8p1ovnarPES60daCyTklfVbNHVZQsnN5b6Q7eXrATF5wQ+Cz69xh91wu3leHdhbuwr7wWN5/ew/Tc6S6HZsT6X//EvRnmHr68MjuyDu5ARR2mrzaenPX5gLd/3YHsdBe2lBzFxLN7IS/DrRS+y45wFN0UGjw+/LjxIM7v1zGhNYUihQU/jrTLTcefzzsOgOShyD/Avg/NBgB0zs9UwjLjz5BuNnmLwuM75WnKMQDA3HvPsnxtedl57/a56NwqE58vD9ygDgJaZ6fh3L4dmvCpgGr/5wgXJ9dn15RW1eMXVSonhLmn17NdjqYWUUllnaHXb3ZvHa334Ji/zQxql8NJPiHFnAGt91hjMCp786Yh+MPbS5U4+dF6j8bDd5nMI8jsNVlEt1sj+AIjn/4Jha0z8d74k4OONZp81I9SDlTUacoNCN13a7TwKjDxSiFHbG6TdF8z6ht9OFLdgHqPDx3zM5R2OWzkFQLvLtyJ3/XtEJR0YMSdHy3Hkp1HDF/z+AQe+SYwKm7w+PDI2H7KCFufLXfHB8tw6aBCdGuTheM65lr+TKF44YfNmDpX+j11aZWJXyed3aTzqHefiwcs+AlEP7TMTHMiM82pEZzsdGNv5KTurdGjbXbE1xzdvyOIpDr5t3+wHEDkOexFBVnYqVpYdrTeCyEEHp8Z2QKxL5ZrvcIZa/Zjxpr9hsemuxyaVQvDHv8BL1wzMOg4s7UNZQZxfTWr9pQrj+savchwO1HX6DWs6Z7plv5uByvrMH9LKa5/Y7FSvgIAtpUcxY5D1crfZ9muI2iV5cYx7aTJ9d0m5YX3qDqCl3/ehh2Hqg13VAOCVzD7fALT5mtDTvrvU+1Ff7VyrzKyVBOYmDbPuwcAt4siE3yPF+c++zPKqhs0v291ls5DX63Dh4t2Y9bdw/1t5r/LUKtq9Quwqv2dnuxgqVcQ13u8mLnmAGaukUpw/PDnEcrfKRrUnbrZSnor+AQQ4ZRQRNgySyeVSVPFRi8e2FlJT2xqFofDQSAijOrfUWkzKn+w+G/nmJ6jfV6G5nlpZZ2m9MO2xy/AD38eEdaWyjrzlZHLHjxX81zvlQHAXR+vDGoz06BNB6uMX/CzcHuZ8vj4v8/CxgOV+Mtnq/HS3K1Bx8qhjGe+24yF26T3zV4XmBNp8Pow8umfMOyx73H7+8uUXPJ95bUQQigrUdWkOR2aOvPzTbZzXLD1EA5XNygZUjJ3f7ISU8KsyJbfUe/x4q6PV+Khr9Ypr23xfz+PfC21fb+hRHEIjIjUw69t9Cqjqaq6RmXkoXc2Nh6oUl4z2sNZfX0zSqu0369XNRIDgHpVR6bv9EoM/jYv/bgFy3drRxM+nzBcEPb9+oP4ds1+TYgvGuK9sQ8LfjOw7MFzseSBcw1fUzxWAbxwzSD8dv856JCXjnvPPy6ia3x06ymaEBARKeUXjIbuelFX49a5HPsq6jDiqbnKc6eD0CZExpDM+7+Zr8LVp2GmuxyWRiLtcgNhHjldFZD2YY2EUc/Px9erjGPE6lGXWXwfkDZhVxdvO23Kj3h13nZDD79v5zwUm3iC93+xBi/9uAVen8C1ry/CiCfnBnXSZraq+d8KaYLYaNQwecYGeLw+HKmxVp4gzekIWmgHSEkBRlz/xmLl8QmPzEH/h6UwptdA0Po/PBvlNQ144Qfz9Fb9b1CNfk2G/LuRR3l1qt97he7z7jhUjbs+XqGEJ4UQeHrOZlz2nwWa416bvx3nPPMz1u7Vporf8u5S3P7BcstrUoxQO3ORFh+MFBb8ZqAgJ10jVGoKW0vxzLP7SGlk2ekuLPrbuTjtmLYRXePUYwqCQkByZoXek5J/rOqKoIWtMzGqX0dcf0p3Q+9Fr8WRppfq0d8wVpbpA8DtZx2jPK5Wxa83Hgjt4UeCWSaQFaZ8uxEfLQ7u6I7tkGMaM/9o8W48PWezMldSVe+xXAVUzeTp69Ho9QXlywPSArRIFq9JHr627fx+HTA0ROqmEWaCdtWrC5UYOCB56erJdiMP/40bh2DcsG7Klpjq9wKBKp/qUFW5zsP/2//W4KuV+7B2bwVW7SnH66qKlVPnblWyu5bukjz+PYdrUO/xBqW46tOc1XMuv20vw6TPV2uEfcHWQ5jtd0rUjk20JUPCwTF8m9G1TRYWP3AO2llMS4wE+aZpUE22/nTvWUqlzFH9O2LLY6Ox53ANeqrimn8KMdSXCZemF4p2uelBE1WtskKnncoMUQnO/Rf0wbNzNmnmG6Ll0bH9TDvnaOjdPjBZ2CrLrWQ+qRnw6BzlsXrOIRLOffZnw8J+RNq00HC4nY4g0c3LcGsmZMMhRHA5BpnNB7Xhkqq6Rk1HayT45/TpgLLqhqAOVQ6LHK4OFvwPfjMu//HpkmJ8olsx/dRsaZ/mokkzAp8BUDLu5t03UmnXOyy3vb8Ml59UiPP6dsCfP12FveW1GH9GD/TukIsLX5yPdfukuaLpd56B0qOBsFK8t+5kD9+GtM/NiMtMvXzTqD38orbZaKvqXNxOh0bsgUC4ZfLF/dCvc57mtcmX9AcQvL2d0QSrOgb873GDlMcL/BkNP917FtY9ej6WPnguurbJUl5f9uC5WGQwxzC0qLVmPcLYAZ3x030jNfMggOQJqidZZd75wzD8aeQxQe1qbjytCBlup2kIrqlkqcJEE0f2Qrc2WUrITSYWi1GNxB6QPPzv1h+0HJd3uwitdaVC+nTKw4mF+Ti2g7VJz6p6j2UPduA/vsMLqsqrmuwuFfJmRGrmbT6E8poGJVwlpyXf+dEKfGlSd0cv9maoR5GTVRuW6D/XnPUH8cf3luG/S4vR2//9PPrNeizffUQRe0Ba0KjeDObzZcV4/vvNcduvlwU/hZBDPO3zIvNY8/2CX1HbiD+fdyxaZbmR6Z9UHXtiZ+W44cdKG8LsnHIh+vpXRsqC8scRPbHiod8px44Z0Blf3HEa/nFxP6UjKmqbjex0l9IBtfWXpmiTnYYOeRlYMOls5RoAcPVQKX6cqZvg1YdKerfPxaNj+6N7QZamfVC3Vrjv/OMtfQdtTcpkNLVfPrlHYGTSMT8D8/4yEs9eNRCPjDGur2SEvONaU5i7qRRv/boTlw7qgmevGmB4zGWDuyi1otKcjqBR50ndWyMrzYU5/y/8hD0AHKluwF8+W23ZRjltNhQ92wZ3NrWNXlz6nwWKt1zv8aKqrhHfWJj3CIe641HPK3y5Yq/R4Zi/9RDa+0eIi3ccDpob0PPYzA14/vst+Hhx5PWZrMCCn0L8/uRueOcPwzB2QOfwB6sYWtQagJQbf/bxHbDyofPw3T3D8e1dZyJfFXp566ah2PrYaABAJ3+dn5x0F3ZOuRD3j+4TtDp4cLfWuOHUItPrfjXxDLw3fpgyeujcKlMJHf3n94NxxUmFAIBZd5+JF1UjhnOOl+Y/Lh0kecxdWmeiY34G3lflt2e4HciJYAUmEeHBC/sEtcsVSsMhC7yDgM9uOxW92ucqnWJBdkBIzWoIGTGgsJXlY834y6jjcNngQtx5dq+g1569aiCO8U+Eu52OoNDWMe0jS2f8cPFu0wqcah6OoNMzKxGinqjeW16rKeUdDerKnHJqJ2BeWXTD/kplxW+oPZr1lB0Nzh6KBRzDTyGICCNUHrJVzunTIShfWS7xoEby5qUbMCfdhb+OOh5nHae93vf3jAiZcaGmS6vMoAJxQ7q3xnfrD6KbKuTTvSAb3QsCE9QvjBuEfeW1OKZdDv55SX9llFHYOhPjhnVF5/xM/GlkL0UsstOcqNbt6rT0wXOD8tJvObOnUgdH5n93nI55m0uVkgNtstOU2LGak3sWYNGOwygqyFbmHeQ8+VxV3aCxAzpjw/5KdMrPxGvzt4ecWI12buGCEzqifa4Ug8/XdTQfTzgFADQlt/WpsuE6u4tO7ASfEIow6ktVmCGEZJtaUEPhIG0SwYMX9tH8ndburcRt75vPQ71981DcFIM9do3YWnI0aFLZCvpdxmIFCz5jiaYsTlFn0Mj0itAr1HPrmT1xTp8OIc+Tk+7CsR2kSdFslSgREZ64LLg66ay7h+PMJ6U009H9O+JfV5wYtlaRjNqOTLcT3951JipqG1HYOhO/f30RVuwux8SRvZTtMNV54Of164iNB6rQQZUS63I68MCFkof7yZI9GsH//p4RSHc5FFvV8xwyw4raID/LramdpObM3m2VnP9+nQOlvVvp0mpP6VkAALjrnN5o8PhwRm+p4372qgF477ddyNaNjp69agB2H67B86oy2C9dOxi7y2osC7dMo9eHhy7qZ/l9GW6nRiDVf7u2Oek4FMZbPsuksJp6pXc4Wme5Ue/xaezo1iYLxUdqmrQV4+aSo3FZdcshHaZF4XBQ1J2Gnq5tsvDH4T0BSKOFUGI/ql/HoLYMt3QbuRyEDnkZOLZDLrLSXPjwllOw5bHRuPf84xRP+rBqQ/C7z+mNpQ+ea+qpP3H5CTixMB9/OL0H3rp5KHq1z0HXNll4f/zJeG/8MCWFV03nVhnKYrRhqnmCEwvz8d74YXjjxqFKyOuYdoFRUSu/hz+qX0dNTL8gJx1TLj9Rmf+5bHAh/nfH6Xj/Fm35h8sGF+Luc49VQnoy3QqysO3xCww/HwAs/3tgXqd/FynE1eDxKXFvALh/9PH49I+nBr1XRh6JdW2TiY9uPQXtNHNUWsG+97xj8dil/ZXnp/cqMD3v01caly7//h7tnMX/7jgNKx46T5MxBkhzHEX+7y3Sej5risubvOdDKBLm4RNRTwAPAMgXQlyRqOsyjBXkhWe9w3Qmr1wvrVXweAOlluWsIKcuVKVemyALmDrzxuEgTYaUnsHdWuNr/34Las7o3dZ/rmDv0+MTOO2Ytpi97iAev/QElFTW4drXFyHD7cSZfi/94TH9kJXu0kyAy2mwJxTm47LBhaY2hcOorpBZJlDHvAzNJkFXD+mKtXvXIT/LDYeD8NoNQ3Dru0vRvSBL03kZnd/nFZhz9whkpjk130tehlspn33hCZ0w8ezeWORfZT2wayu8caNUznzO/xuOeZtLNaEgdcef4XYo2T768NegbtIclzo1+bLBXTD54v6YPH09tpdWIzPNFRQ2BKREB6P9imfdPVwz8osVlgSfiN4EcBGAEiFEf1X7KAAvAHACeF0IMcXsHEKI7QDGE9Fn0ZnMMLHnhlO7o11uOi5SVZsMhVrYctMlAbjhlO6mx8fj5iUi/HXU8fjXrECJhT6d8nDDqd1x4Ymd0DYnXZlPUM95dCvI0qTFAgHB14tZU3jlusEoDVPLaOb/nank8H898XSUVNZj5PHtkZXmwiX+yfbf9Q2eO5KZ4B+RAcAXt5+ObaVHlQ6WiPDH4T3x6rzteO7qgfj7V2uxurgCrbOlzzakqA0eHtMXV5xUqMxLHNshF7t1KazqaMqcu0dguH91ufo7ul71N5c7/pO6t8YzVw4AEeHak7vh4yV7DMNK8/8yEt+s3mco+PH4vQDWPfy3AbwE4F25gYicAKYC+B2AYgBLiOhrSOL/hO79fxBC8LYzjG1xOx0RZy/JZKY5sfmfo0NORpuldUbL7Wcdg/5d8uB2OpCd5kK/znkgCowchha1xuOXnoCLB4b+bD3a5uCWM3rgnD7RbxSiLjdtRE66C8d1zFU8/xNV2UaXn6QdXRiJ/bbHL9DUUDqhMB8nqLYaBYD7zj8OVw7pil7tc/D57afh3z9swS3+TsLpIMNS0HIu/YDCfDx4UV/065yHU3q2weSL+6NbQRYuGdgZX67chzSXA/8eNwgenw+XDgrYO6yoDWas3o+bTitSYu/ytqdpTocmS+fvF/VF1zZZSgrxTacVYcLwnnjxhy24/lRzxyFaLAm+EGIeERXpmocB2Or33EFEHwO4WAjxBKTRQJMgogkAJgBAt25NzzNmmESiX+ylJ1wJ5WiQQzVGyF5mOJwOwoMmeyzHgu4FWdhVVoN3/jCsSZliaqwsFnM5Hcpcj9vpwD3nha9FJc8hTDy7t1Iy4uMJgbmDZ64aiCmXS3H9MQbOwY2nFeGUngWauZE0lwPPXz0QvTvkIMPtxJRvN+K79QeR4194J9dIKshOQ+dWmcr540U0MfwuANSrA4oBBBfy9kNEBQAeAzCIiO73dwxBCCGmAZgGAEOGDOEdiJmkoXN+hsabTSU+uOVk/LSpNCqxV28eFA8KW2eF3FnL6SA4HaEnX43q68shKgDKyEueZpA9fHcUpUkiIRrBN+pmTQVaCFEG4LYorscwLZoF95uXoE52Cltn4boQcxxW+Ones0KW2G4J3HvesWj0+nDxQKkTUK8FSQTRCH4xgK6q54UAol+7DICIxgAY06tX8Oo/hmFSk9bZaUH1fFoaBTnpePrKQNrrxJG9IARw1dCuId4VO6IZRywB0JuIehBRGoBrAHwdC6OEEN8IISbk5+eHP5hhGKaFkpvhxt8u6BNVCe5IsCT4RPQRgIUAjiOiYiIaL4TwAJgIYDaADQA+FUKsC3UehmEYpvmwmqUzzqR9JoDgnaKjhEM6DMMwsceWpRU4pMMwDBN7bCn4DMMwTOxhwWcYhkkRbCn4RDSGiKZVVFSEP5hhGIaxhC0Fn2P4DMMwsceWgs8wDMPEHjKqqW0XiKgUwK4mvr0tgNhsZBkf2L7oYPuiw+72Afa30a72dRdCGBYtsrXgRwMRLRVCDGluO8xg+6KD7YsOu9sH2N9Gu9tnBId0GIZhUgQWfIZhmBQhmQV/WnMbEAa2LzrYvuiwu32A/W20u31BJG0Mn2EYhtGSzB4+wzAMo4IFn2EYJkVIOsEnolFEtImIthLRpGa0400iKiGitaq2NkT0HRFt8f/fWvXa/X6bNxHR+XG2rSsRzSWiDUS0jojuspl9GUS0mIhW+e171E72qa7pJKIVRDTdpvbtJKI1RLSSiJbazUYiakVEnxHRRv9v8VS72EdEx/m/N/lfJRHdbRf7mowQImn+AXAC2AagJ4A0AKsA9G0mW4YDGAxgrartSQCT/I8nAfiX/3Ffv63pAHr4P4MzjrZ1AjDY/zgXwGa/DXaxjwDk+B+7ASwCcIpd7FPZeQ+ADwFMt9PfV2XfTgBtdW22sRHAOwBu8T9OA9DKTvap7HQCOACgux3ti+izNLcBMf7DnApgtur5/QDub0Z7iqAV/E0AOvkfdwKwychOSLuInZpAO78C8Ds72gcgC8ByACfbyT5Iezj/AOBsleDbxj7/dYwE3xY2AsgDsAP+xBG72aez6TwAv9rVvkj+JVtIpwuAParnxf42u9BBCLEfAPz/t/e3N5vdRFQEYBAkL9o29vnDJSsBlAD4TghhK/sAPA/gLwB8qjY72QcAAsAcIlpGRBNsZmNPAKUA3vKHxV4nomwb2afmGgAf+R/b0T7LJJvgk0FbS8g7bRa7iSgHwOcA7hZCVIY61KAtrvYJIbxCiIGQPOlhRNQ/xOEJtY+ILgJQIoRYZvUtBm2J+F2eLoQYDGA0gD8R0fAQxybaRhekkOfLQohBAKohhUjMaK57JA3AWAD/DXeoQZvttCfZBL8YQFfV80IA+5rJFiMOElEnAPD/X+JvT7jdROSGJPYfCCG+sJt9MkKIcgA/ARhlI/tOBzCWiHYC+BjA2UT0vo3sAwAIIfb5/y8B8D8Aw2xkYzGAYv/IDQA+g9QB2MU+mdEAlgshDvqf282+iEg2wV8CoDcR9fD3zNcA+LqZbVLzNYAb/Y9vhBQ7l9uvIaJ0IuoBoDeAxfEygogIwBsANgghnrWhfe2IqJX/cSaAcwFstIt9Qoj7hRCFQogiSL+xH4UQ19nFPgAgomwiypUfQ4pDr7WLjUKIAwD2ENFx/qZzAKy3i30qxiEQzpHtsJN9kdHckwhxmGC5AFLWyTYADzSjHR8B2A+gEVLvPx5AAaSJvi3+/9uojn/Ab/MmAKPjbNsZkIabqwGs9P+7wEb2nQhghd++tQAe8rfbwj6drWchMGlrG/sgxchX+f+tk+8Fm9k4EMBS/9/5SwCtbWZfFoAyAPmqNtvY15R/XFqBYRgmRUi2kA7DMAxjAgs+wzBMisCCzzAMkyKw4DMMw6QILPgMwzApAgs+wzBMisCCzzAMkyL8f8iKorrZ6kpsAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "epochs = 1\n",
    "model = ThreeInputsNet()\n",
    "model.to(device)\n",
    "opt = torch.optim.Adam(model.parameters())\n",
    "criterion = nn.MSELoss().to(device)\n",
    "\n",
    "history = []\n",
    "for epoch in range(epochs):\n",
    "    for batch, target in iterate_minibatches(data_train):\n",
    "        # Preprocessing the batch data and target\n",
    "        \n",
    "        title = torch.tensor(batch[\"Title\"], dtype=torch.long, device=device)\n",
    "        desc = torch.tensor(batch[\"FullDescription\"], dtype=torch.long, device=device)\n",
    "        category = torch.tensor(batch[\"Categorical\"], dtype=torch.float32, device=device)\n",
    "        \n",
    "        target = torch.tensor(target, device=device)\n",
    "\n",
    "        predictions = model(title, desc, category)\n",
    "        predictions = predictions.flatten()\n",
    "\n",
    "        opt.zero_grad()\n",
    "        # Compute loss.\n",
    "\n",
    "        loss = criterion(predictions, target)\n",
    "        \n",
    "        loss.backward()\n",
    "        torch.nn.utils.clip_grad_norm_(model.parameters(), 10)\n",
    "        opt.step()\n",
    "        # Train with backprop.\n",
    "\n",
    "        history.append(loss.item())\n",
    "        if len(history) % 10 == 0:\n",
    "            clear_output(wait=True)\n",
    "            plt.plot(history, label=\"loss\")\n",
    "            plt.yscale('log')\n",
    "            plt.legend()\n",
    "            plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "referenced_widgets": [
      "e00a454515cb4cb3b1ec481b8e776433",
      "5267c457a57d47dd9ed7543e6b8c7515"
     ]
    },
    "id": "sS1Axev44Lqn",
    "outputId": "716335f6-4c91-4e6c-dfa9-cf60bc77c60a"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e00a454515cb4cb3b1ec481b8e776433",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train:\n",
      "MSE: 0.11243\n",
      "MAE: 0.25484\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5267c457a57d47dd9ed7543e6b8c7515",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Val:\n",
      "MSE: 0.11965\n",
      "MAE: 0.26394\n"
     ]
    }
   ],
   "source": [
    "@torch.no_grad()\n",
    "def evaluate_model(model, data, batch_size=256, name=None, **kwargs):\n",
    "    squared_error = abs_error = num_samples = 0.0\n",
    "    data_iterator = iterate_minibatches(data, batch_size, shuffle=False, **kwargs)\n",
    "    for batch, target in tqdm(data_iterator):\n",
    "        \n",
    "        title = torch.tensor(batch[\"Title\"], dtype=torch.long, device=device)\n",
    "        desc = torch.tensor(batch[\"FullDescription\"], dtype=torch.long, device=device)\n",
    "        category = torch.tensor(batch[\"Categorical\"], dtype=torch.float32, device=device)\n",
    "        \n",
    "        pred = model(title, desc, category)[:, 0].cpu().numpy()\n",
    "\n",
    "        squared_error += np.sum(np.square(pred - target))\n",
    "        abs_error += np.sum(np.abs(pred - target))\n",
    "        num_samples += len(target)\n",
    "\n",
    "    if name is not None:\n",
    "        print(f\"{name}:\")\n",
    "\n",
    "    print(f\"MSE: {squared_error / num_samples:.5f}\")\n",
    "    print(f\"MAE: {abs_error / num_samples:.5f}\")\n",
    "    \n",
    "    return squared_error / num_samples, abs_error / num_samples\n",
    "\n",
    "\n",
    "train_errors = evaluate_model(model, data_train, name=\"Train\")\n",
    "test_errors = evaluate_model(model, data_val, name=\"Val\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "referenced_widgets": [
      "b06e663dc0094ec1bffb67dcc6c74aff",
      "5a9f3f43d30e4dcbb580ce83d12c8175"
     ]
    },
    "id": "wK-jUxy04Lqo",
    "outputId": "c0fd6770-ba1c-4bba-dbbb-441dd34be758"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b06e663dc0094ec1bffb67dcc6c74aff",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train:\n",
      "MSE: 171929688.19994\n",
      "MAE: 8762.57302\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5a9f3f43d30e4dcbb580ce83d12c8175",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Val:\n",
      "MSE: 183495038.62467\n",
      "MAE: 9110.68273\n"
     ]
    }
   ],
   "source": [
    "@torch.no_grad()\n",
    "def evaluate_model(model, data, batch_size=256, name=None, **kwargs):\n",
    "    squared_error = abs_error = num_samples = 0.0\n",
    "    data_iterator = iterate_minibatches(data, batch_size, shuffle=False, **kwargs)\n",
    "    for batch, target in tqdm(data_iterator):\n",
    "        \n",
    "        title = torch.tensor(batch[\"Title\"], dtype=torch.long, device=device)\n",
    "        desc = torch.tensor(batch[\"FullDescription\"], dtype=torch.long, device=device)\n",
    "        category = torch.tensor(batch[\"Categorical\"], dtype=torch.float32, device=device)\n",
    "        \n",
    "        pred = model(title, desc, category)[:, 0].cpu().numpy()\n",
    "\n",
    "        squared_error += np.sum(np.square(np.exp(pred) - np.exp(target)))\n",
    "        abs_error += np.sum(np.abs(np.exp(pred) - np.exp(target)))\n",
    "        num_samples += len(target)\n",
    "\n",
    "    if name is not None:\n",
    "        print(f\"{name}:\")\n",
    "\n",
    "    print(f\"MSE: {squared_error / num_samples:.5f}\")\n",
    "    print(f\"MAE: {abs_error / num_samples:.5f}\")\n",
    "    \n",
    "    return squared_error / num_samples, abs_error / num_samples\n",
    "\n",
    "\n",
    "train_errors = evaluate_model(model, data_train, name=\"Train\")\n",
    "test_errors = evaluate_model(model, data_val, name=\"Val\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kAigoulWpHfe"
   },
   "source": [
    "### Bonus area 2: comparing RNN to CNN\n",
    "Try implementing simple RNN (or LSTM) and applying it to this task. Compare the quality/performance of these networks. \n",
    "*Hint: try to build networks with ~same number of paremeters.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rQVzJyCMpHfe"
   },
   "source": [
    "*Used LSTM to encode Titles at __Bonus area 1__*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jPn2Q0wXpHfe"
   },
   "source": [
    "### Bonus area 3: fixing the data leaks\n",
    "Fix the data leak we ignored in the beginning of the __Deep Learning part__. Compare results with and without data leaks using same architectures and training time.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "id": "DVgBWPpjpHfe"
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "data = pd.read_csv(\"Train_rev1.csv\", index_col=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 398
    },
    "id": "6Iaf4nfV4Lqp",
    "outputId": "af180f43-fbb7-4cde-9146-21059b91717e"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-d29612ae-e423-4165-83b0-29c1141a21fc\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>Title</th>\n",
       "      <th>FullDescription</th>\n",
       "      <th>LocationRaw</th>\n",
       "      <th>LocationNormalized</th>\n",
       "      <th>ContractType</th>\n",
       "      <th>ContractTime</th>\n",
       "      <th>Company</th>\n",
       "      <th>Category</th>\n",
       "      <th>SalaryRaw</th>\n",
       "      <th>SalaryNormalized</th>\n",
       "      <th>SourceName</th>\n",
       "      <th>Log1pSalary</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>93221</th>\n",
       "      <td>69188446</td>\n",
       "      <td>GRADUATE SALES EXECUTIVE / TRAINEE BUSINESS DE...</td>\n",
       "      <td>GRADUATE SALES EXECUTIVE / TRAINEE BUSINESS DE...</td>\n",
       "      <td>City London South East</td>\n",
       "      <td>London</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>BMS Sales Specialists LLP</td>\n",
       "      <td>Sales Jobs</td>\n",
       "      <td>25000 per annum + 30% Bonus, Car, Mobile</td>\n",
       "      <td>25000</td>\n",
       "      <td>salestarget.co.uk</td>\n",
       "      <td>10.126671</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>143596</th>\n",
       "      <td>70622627</td>\n",
       "      <td>Senior Java Developer</td>\n",
       "      <td>My client are a leading software house and con...</td>\n",
       "      <td>Yorkshire Leeds LS1 8</td>\n",
       "      <td>Horsforth</td>\n",
       "      <td>NaN</td>\n",
       "      <td>permanent</td>\n",
       "      <td>The Bridge Ltd</td>\n",
       "      <td>IT Jobs</td>\n",
       "      <td>40000.00 - 50000.00 GBP Annual</td>\n",
       "      <td>45000</td>\n",
       "      <td>jobserve.com</td>\n",
       "      <td>10.714440</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28108</th>\n",
       "      <td>67946379</td>\n",
       "      <td>Technical Director  Strategic Modelling</td>\n",
       "      <td>Introduction An excellent opportunity to work ...</td>\n",
       "      <td>Woking</td>\n",
       "      <td>Woking</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Matchtech Group plc.</td>\n",
       "      <td>Engineering Jobs</td>\n",
       "      <td>58,000 - 65,000/yr</td>\n",
       "      <td>61500</td>\n",
       "      <td>rengineeringjobs.com</td>\n",
       "      <td>11.026809</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-d29612ae-e423-4165-83b0-29c1141a21fc')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-d29612ae-e423-4165-83b0-29c1141a21fc button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-d29612ae-e423-4165-83b0-29c1141a21fc');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "              Id                                              Title  \\\n",
       "93221   69188446  GRADUATE SALES EXECUTIVE / TRAINEE BUSINESS DE...   \n",
       "143596  70622627                              Senior Java Developer   \n",
       "28108   67946379            Technical Director  Strategic Modelling   \n",
       "\n",
       "                                          FullDescription  \\\n",
       "93221   GRADUATE SALES EXECUTIVE / TRAINEE BUSINESS DE...   \n",
       "143596  My client are a leading software house and con...   \n",
       "28108   Introduction An excellent opportunity to work ...   \n",
       "\n",
       "                   LocationRaw LocationNormalized ContractType ContractTime  \\\n",
       "93221   City London South East             London          NaN          NaN   \n",
       "143596   Yorkshire Leeds LS1 8          Horsforth          NaN    permanent   \n",
       "28108                   Woking             Woking          NaN          NaN   \n",
       "\n",
       "                          Company          Category  \\\n",
       "93221   BMS Sales Specialists LLP        Sales Jobs   \n",
       "143596             The Bridge Ltd           IT Jobs   \n",
       "28108        Matchtech Group plc.  Engineering Jobs   \n",
       "\n",
       "                                       SalaryRaw  SalaryNormalized  \\\n",
       "93221   25000 per annum + 30% Bonus, Car, Mobile             25000   \n",
       "143596            40000.00 - 50000.00 GBP Annual             45000   \n",
       "28108                         58,000 - 65,000/yr             61500   \n",
       "\n",
       "                  SourceName  Log1pSalary  \n",
       "93221      salestarget.co.uk    10.126671  \n",
       "143596          jobserve.com    10.714440  \n",
       "28108   rengineeringjobs.com    11.026809  "
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[\"Log1pSalary\"] = np.log1p(data[\"SalaryNormalized\"]).astype(\"float32\")\n",
    "text_columns = [\"Title\", \"FullDescription\"]\n",
    "categorical_columns = [\"Category\", \"Company\", \"LocationNormalized\", \"ContractType\", \"ContractTime\"]\n",
    "target_column = \"Log1pSalary\"\n",
    "\n",
    "data[categorical_columns] = data[categorical_columns].fillna(\"NaN\")\n",
    "data[text_columns] = data[text_columns].fillna(\"NaN\")\n",
    "\n",
    "data.sample(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "id": "L-OPkZ3O4Lqp"
   },
   "outputs": [],
   "source": [
    "tokenizer = WordPunctTokenizer()\n",
    "\n",
    "def normalize(text):\n",
    "    tokens = tokenizer.tokenize(text.lower())\n",
    "    return ' '.join(tokens)\n",
    "# Lowercase and tokenize titles and descriptions (text_columns).\n",
    "# Store items as space-separated strings of tokens.\n",
    "data[text_columns] = data[text_columns].applymap(normalize)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "id": "9_GiBN7F4Lqp"
   },
   "outputs": [],
   "source": [
    "data_train, data_val = train_test_split(data, test_size=0.2, random_state=42)\n",
    "data_train.index = range(len(data_train))\n",
    "data_val.index = range(len(data_val))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "id": "QIM-lk534Lqp"
   },
   "outputs": [],
   "source": [
    "token_counts = Counter()\n",
    "for row in data_train[text_columns].values.flatten():\n",
    "    token_counts.update(row.split(' '))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 279
    },
    "id": "Ic4Mfr5i4Lqp",
    "outputId": "ce4a6a96-2036-4bac-d14f-67efcf3cf25c"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEGCAYAAACevtWaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAASIElEQVR4nO3df5BdZX3H8ffHpEDVFlAyDg3EBMOgqc4o7qDYH0P9GZRIx9qR1Kk/Sk3RwdbamQpjZ6p/dAar/aEjI6ZAUduCSB0LEie1VoojSAn+IgiRiFiSqgnQRuu0avTbP+6JXtfd7N3de3P3Pvt+zezknOece85z7tl897nf57nPSVUhSWrLI8ZdAUnS8BncJalBBndJapDBXZIaZHCXpAatHHcFAE444YRau3btuKshSRPljjvueLCqVs20bUkE97Vr17Jjx45xV0OSJkqSr822zbSMJDXI4C5JDTK4S1KDDO6S1CCDuyQ1aKzBPcmmJFsPHDgwzmpIUnPGGtyr6oaq2nLssceOsxqS1BzTMpLUoCXxJabFWHvRjTOW33/Ji45wTSRp6bDlLkkNMrhLUoMM7pLUIIO7JDXI4C5JDTK4S1KDDO6S1KChB/ckZyX5VJLLkpw17ONLkuY2UHBPcmWSfUl2TivfmGRXkt1JLuqKC/gf4Bhgz3CrK0kaxKAt96uAjf0FSVYAlwJnAxuAzUk2AJ+qqrOBNwFvHV5VJUmDGii4V9XNwMPTis8AdlfVfVX1PeAa4Nyq+mG3/b+Ao2c7ZpItSXYk2bF///4FVF2SNJvF5NxXAw/0re8BVid5SZL3Ah8A3j3bi6tqa1VNVdXUqlUzPrxbkrRAQ584rKo+DHx4kH2TbAI2rV+/ftjVkKRlbTEt973AyX3rJ3VlA3M+d0kajcUE99uBU5OsS3IUcB5w/XwO4JOYJGk0Bh0KeTVwK3Bakj1Jzq+qg8CFwHbgbuDaqrprPie35S5JozFQzr2qNs9Svg3YNtQaSZIWzQdkS1KDfEC2JDXIicMkqUGmZSSpQaZlJKlBpmUkqUGmZSSpQaZlJKlBpmUkqUEGd0lqkMFdkhpkh6okNcgOVUlqkGkZSWqQwV2SGmRwl6QG2aEqSQ2yQ1WSGmRaRpIaZHCXpAYZ3CWpQQZ3SWqQwV2SGmRwl6QGOc5dkhrkOHdJapBpGUlqkMFdkhpkcJekBhncJalBBndJapDBXZIaZHCXpAYZ3CWpQQZ3SWrQSIJ7kkcl2ZHknFEcX5J0eAMF9yRXJtmXZOe08o1JdiXZneSivk1vAq4dZkUlSYMbtOV+FbCxvyDJCuBS4GxgA7A5yYYkzwO+BOwbYj0lSfOwcpCdqurmJGunFZ8B7K6q+wCSXAOcCzwaeBS9gP+/SbZV1Q+nHzPJFmALwJo1axZaf0nSDAYK7rNYDTzQt74HeEZVXQiQ5FXAgzMFdoCq2gpsBZiamqpF1EOSNM1igvthVdVVc+2TZBOwaf369aOqhiQtS4sZLbMXOLlv/aSubGDO5y5Jo7GY4H47cGqSdUmOAs4Drp/PAXwSkySNxqBDIa8GbgVOS7InyflVdRC4ENgO3A1cW1V3zefkttwlaTQGHS2zeZbybcC2odZIkrRoPiBbkhrkA7IlqUFOHCZJDTItI0kNMi0jSQ0yLSNJDTItI0kNMi0jSQ0yLSNJDTK4S1KDDO6S1CA7VCWpQXaoSlKDTMtIUoMM7pLUIIO7JDXIDlVJapAdqpLUINMyktQgg7skNcjgLkkNMrhLUoMM7pLUoJXjrsCorL3oxhnL77/kRUe4JpJ05DnOXZIa5Dh3SWqQOXdJapDBXZIaZHCXpAYZ3CWpQQZ3SWqQwV2SGmRwl6QGGdwlqUEGd0lq0NCDe5InJbksyXVJXjvs40uS5jbQxGFJrgTOAfZV1ZP7yjcC7wRWAJdX1SVVdTdwQZJHAO8H3jP8ai/cbBOKgZOKSWrHoC33q4CN/QVJVgCXAmcDG4DNSTZ0214M3AhsG1pNJUkDGyi4V9XNwMPTis8AdlfVfVX1PeAa4Nxu/+ur6mzg5bMdM8mWJDuS7Ni/f//Cai9JmtFi5nNfDTzQt74HeEaSs4CXAEdzmJZ7VW0FtgJMTU3VIuohSZpm6A/rqKqbgJsG2TfJJmDT+vXrh10NSVrWFjNaZi9wct/6SV3ZwJzPXZJGYzHB/Xbg1CTrkhwFnAdcP58D+CQmSRqNgYJ7kquBW4HTkuxJcn5VHQQuBLYDdwPXVtVd8zm5LXdJGo2Bcu5VtXmW8m043FGSlhwfkC1JDfIB2ZLUoKEPhZxks01N4LQEkiaNaRlJapBpGUlqkPO5S1KDTMtIUoNMy0hSg0zLSFKDHAo5AIdISpo0ttwlqUF2qEpSg+xQlaQGmZaRpAYZ3CWpQQZ3SWqQHaqS1CA7VCWpQX6JaRH8cpOkpcrgPgIGfUnjZoeqJDXI4C5JDTK4S1KDzLkfQebiJR0pjnOXpAY5zl2SGmRaZgkwXSNp2OxQlaQGGdwlqUEGd0lqkMFdkhpkh+oEsgNW0lwM7kvYbEFckuZiWkaSGmTLvSGmayQdMpLgnuTXgRcBPw9cUVX/PIrzSJJmNnBaJsmVSfYl2TmtfGOSXUl2J7kIoKo+UlWvAS4AXjbcKkuS5jKfnPtVwMb+giQrgEuBs4ENwOYkG/p2+ZNuuyTpCBo4LVNVNydZO634DGB3Vd0HkOQa4NwkdwOXAB+rqs/OdLwkW4AtAGvWrJl/zTUwc/HS8rPY0TKrgQf61vd0Za8Hngu8NMkFM72wqrZW1VRVTa1atWqR1ZAk9RtJh2pVvQt411z7JdkEbFq/fv0oqqE5LGQcva19aTIstuW+Fzi5b/2krmwgzucuSaOx2OB+O3BqknVJjgLOA64f9MU+iUmSRmPgtEySq4GzgBOS7AH+tKquSHIhsB1YAVxZVXcNesyqugG4YWpq6jXzq7bGxc5ZaTLMZ7TM5lnKtwHbhlYjTSSDvrS0+IBsSWrQWOeWMS3TPlv00ng4K6QkNWisLXfHuWtU/MSg5W6sLXfHuUvSaDifu5YUW9zScJiW0UQw6EvzY1pGkhpkWkYTzYeISzMzuGssxhWUTe9ouTC4Szj9sdrj9AOS1CA7VCWpQU4/IEkNMucuLZCds1rKDO7SkI0r6PvHRv38hqo0ZqMOygb95ckOVUlqkB2qktQgc+7SEmU6RYthy12SGmTLXZowTpamQRjcpSPEoKwjyeAu6SfM94/QMPsAxnnu1jjOXVqm/CQxt0nu1B5rcK+qG4AbpqamXjPOekhauMP9kZiEINgq0zKSJtZ8W9aT3BKfL4dCSlKDbLlLOuLM94+eLXdJapAtd0k6Ao50vt+WuyQ1yOAuSQ0yuEtSg8y5SxqZSRkVMyn1nI+ht9yTnJLkiiTXDfvYkqTBDNRyT3IlcA6wr6qe3Fe+EXgnsAK4vKouqar7gPMN7pJaNQnfdB205X4VsLG/IMkK4FLgbGADsDnJhqHWTpK0IAO13Kvq5iRrpxWfAezuWuokuQY4F/jSIMdMsgXYArBmzZoBqytJc2sxhz5fi8m5rwYe6FvfA6xO8tgklwFPS3LxbC+uqq1VNVVVU6tWrVpENSRJ0w19tExVPQRcMMi+zucuSaOxmJb7XuDkvvWTurKBVdUNVbXl2GOPXUQ1JEnTLSa43w6cmmRdkqOA84Dr53OAJJuSbD1w4MAiqiFJmm6g4J7kauBW4LQke5KcX1UHgQuB7cDdwLVVddd8Tm7LXZJGY9DRMptnKd8GbBtqjSRJi+YDsiVpSJbSEMyxThxmWkaSRsNZISWpQWMN7o6WkaTRMC0jSQ0yLSNJDTItI0kNMi0jSQ0yLSNJDTK4S1KDUlXjrgNJ9gNfW+DLTwAeHGJ1JoHXvDx4zcvDYq758VU14wMxlkRwX4wkO6pqatz1OJK85uXBa14eRnXNpmUkqUEGd0lqUAvBfeu4KzAGXvPy4DUvDyO55onPuUuSfloLLXdJ0jQGd0lq0EQH9yQbk+xKsjvJReOuz0IlOTnJJ5N8KcldSf6gK39Mko8nubf79/iuPEne1V33F5Oc3nesV3b735vkleO6pkElWZHkc0k+2q2vS3Jbd20f7B6+TpKju/Xd3fa1fce4uCvfleQF47mSwSQ5Lsl1Se5JcneSM1u/z0n+sPu93pnk6iTHtHafk1yZZF+SnX1lQ7uvSZ6e5M7uNe9KkjkrVVUT+QOsAL4CnAIcBXwB2DDuei3wWk4ETu+Wfw74MrAB+HPgoq78IuBt3fILgY8BAZ4J3NaVPwa4r/v3+G75+HFf3xzX/kbgH4CPduvXAud1y5cBr+2WXwdc1i2fB3ywW97Q3fujgXXd78SKcV/XYa73fcDvdstHAce1fJ+B1cBXgZ/tu7+vau0+A78KnA7s7Csb2n0F/r3bN91rz56zTuN+UxbxZp4JbO9bvxi4eNz1GtK1/RPwPGAXcGJXdiKwq1t+L7C5b/9d3fbNwHv7yn9iv6X2A5wEfAJ4NvDR7hf3QWDl9HsMbAfO7JZXdvtl+n3v32+p/QDHdoEu08qbvc9dcH+gC1gru/v8ghbvM7B2WnAfyn3ttt3TV/4T+832M8lpmUO/NIfs6comWvcx9GnAbcDjqurr3aZvAI/rlme79kl7T/4a+GPgh936Y4H/rqqD3Xp//X90bd32A93+k3TN64D9wN92qajLkzyKhu9zVe0F3gH8B/B1evftDtq+z4cM676u7panlx/WJAf35iR5NPCPwBuq6lv926r3J7uZcatJzgH2VdUd467LEbSS3kf391TV04Dv0Pu4/iMN3ufjgXPp/WH7BeBRwMaxVmoMxnFfJzm47wVO7ls/qSubSEl+hl5g//uq+nBX/M0kJ3bbTwT2deWzXfskvSe/BLw4yf3ANfRSM+8Ejkuystunv/4/urZu+7HAQ0zWNe8B9lTVbd36dfSCfcv3+bnAV6tqf1V9H/gwvXvf8n0+ZFj3dW+3PL38sCY5uN8OnNr1uh9Fr/Pl+jHXaUG6nu8rgLur6i/7Nl0PHOoxfyW9XPyh8ld0ve7PBA50H/+2A89PcnzXYnp+V7bkVNXFVXVSVa2ld+/+tapeDnwSeGm32/RrPvRevLTbv7ry87pRFuuAU+l1Pi05VfUN4IEkp3VFzwG+RMP3mV465plJHtn9nh+65mbvc5+h3Ndu27eSPLN7D1/Rd6zZjbsTYpEdGC+kN7LkK8Cbx12fRVzHL9P7yPZF4PPdzwvp5Ro/AdwL/AvwmG7/AJd2130nMNV3rN8Bdnc/rx73tQ14/Wfx49Eyp9D7T7sb+BBwdFd+TLe+u9t+St/r39y9F7sYYBTBmK/1qcCO7l5/hN6oiKbvM/BW4B5gJ/ABeiNemrrPwNX0+hS+T+8T2vnDvK/AVPf+fQV4N9M65Wf6cfoBSWrQJKdlJEmzMLhLUoMM7pLUIIO7JDXI4C5JDTK4a8lL8ldJ3tC3vj3J5X3rf5HkjQs89lnpZqQ8ktKbHfJ1R/q8Wj4M7poEnwaeBZDkEcAJwC/2bX8WcMsgB0qyYui1W5jj6M2AKI2EwV2T4BZ6MwdCL6jvBL7dfZPvaOBJwGeTPKebkOvObn7towGS3J/kbUk+C/xmes8BuKdbf8lMJ0xvnvl3dHOQfzHJ67vyw53jhG55KslN3fJbuv1uSnJfkt/vTnEJ8IQkn0/y9iQnJrm5W9+Z5FdG8D5qGVk59y7SeFXVfyY5mGQNvVb6rfRmxTuT3qyBd9JrqFwFPKeqvpzk/cBr6c08CfBQVZ2e5Bh63xh8Nr1vAX5wltNuoTeF61Or6mB6D144Zo5zzOaJwK/Rm6t/V5L30Jsw7MlV9VSAJH9E76vmf9Z9unjk4O+Q9NNsuWtS3EIvsB8K7rf2rX8aOI3eBFVf7vZ/H70HKBxyKIg/sdvv3up9PfvvZjnfc+nNmX0QoKoeHuAcs7mxqr5bVQ/SmzzqcTPsczvw6iRvAZ5SVd8e4LjSrAzumhSH8u5PoZeW+Qy9lvug+fbvjK5qABzkx/+fjpm27bt9yz9ghk/MVXUzvT8Ue4GrkrxiFJXU8mFw16S4BTgHeLiqftC1pI+jF+BvoTeZ1Nok67v9fxv4txmOc0+33xO69c2znO/jwO8dmpY2yWPmOMf9wNO75d8Y4Hq+TS9NQ3f8xwPfrKq/AS6nNxWwtGAGd02KO+mNkvnMtLIDVfVgVf0f8GrgQ0nupPd0p8umH6TbbwtwY9ehum/6Pp3L6U1X+8UkXwB+a45zvBV4Z5Id9Frnh1VVDwGf7jpP305vZswvJPkc8DJ6c9tLC+askJLUIFvuktQgg7skNcjgLkkNMrhLUoMM7pLUIIO7JDXI4C5JDfp/UcgPGnGAnMYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.hist(list(token_counts.values()), range=[0, 10 ** 4], bins=50, log=True)\n",
    "plt.xlabel(\"Word counts\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "id": "aBlhaMrI4Lqq"
   },
   "outputs": [],
   "source": [
    "min_count = 10\n",
    "\n",
    "# We shall only consider tokens that had at least min_count occurences.\n",
    "# Create a list of such tokens.\n",
    "tokens = [token for token, counts in token_counts.items() if counts >= min_count]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "IGYEL7Mq4Lqq",
    "outputId": "35ee47ea-abdb-41c2-bc54-9d23c5772445"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vocabulary size: 30715\n"
     ]
    }
   ],
   "source": [
    "UNK, PAD = \"UNK\", \"PAD\"\n",
    "tokens = [UNK, PAD] + sorted(tokens)\n",
    "print(\"Vocabulary size:\", len(tokens))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "id": "Bt4d773N4Lqq"
   },
   "outputs": [],
   "source": [
    "token_to_idx = {token: idx for idx, token in enumerate(tokens)}\n",
    "UNK_IDX = token_to_idx[UNK]\n",
    "PAD_IDX = token_to_idx[PAD]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "id": "GU3OYdTF4Lqq"
   },
   "outputs": [],
   "source": [
    "company_counts = Counter(data_train[\"Company\"])\n",
    "top_companies = set(name for name, count in company_counts.most_common(1000))\n",
    "data_train[\"Company\"] = data_train[\"Company\"].apply(\n",
    "    lambda company: company if company in top_companies else \"Other\"\n",
    ")\n",
    "\n",
    "categorical_vectorizer = DictVectorizer(dtype=np.float32, sparse=False)\n",
    "categorical_vectorizer.fit(data_train[categorical_columns].apply(dict, axis=1));"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "id": "0ubJp3ng4Lqq"
   },
   "outputs": [],
   "source": [
    "def make_batch(data, max_len=None, word_dropout=0):\n",
    "    \"\"\"\n",
    "    Creates a neural-network-friendly dict from the batch data.\n",
    "    :param word_dropout: replaces token index with UNK_IDX with this probability\n",
    "    :returns: a dict with {'title' : int64[batch, title_max_len]\n",
    "    \"\"\"\n",
    "    batch = {}\n",
    "    batch[\"Title\"] = as_matrix(data[\"Title\"].values, max_len)\n",
    "    batch[\"FullDescription\"] = as_matrix(data[\"FullDescription\"].values, max_len)\n",
    "    batch[\"Categorical\"] = categorical_vectorizer.transform(\n",
    "        data[categorical_columns].apply(dict, axis=1)\n",
    "    )\n",
    "\n",
    "    if word_dropout != 0:\n",
    "        batch[\"FullDescription\"] = apply_word_dropout(batch[\"FullDescription\"], 1.0 - word_dropout)\n",
    "\n",
    "    if target_column in data.columns:\n",
    "        batch[target_column] = data[target_column].values\n",
    "\n",
    "    return batch\n",
    "\n",
    "\n",
    "def apply_word_dropout(matrix, keep_prob, replace_with=UNK_IDX):\n",
    "    dropout_mask = np.random.choice(2, np.shape(matrix), p=[keep_prob, 1 - keep_prob])\n",
    "    dropout_mask &= matrix != PAD_IDX\n",
    "    return np.choose(dropout_mask, [matrix, np.full_like(matrix, replace_with)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "id": "_96duuCi4Lqq"
   },
   "outputs": [],
   "source": [
    "def iterate_minibatches(data, batch_size=256, shuffle=True, cycle=False, **kwargs):\n",
    "    \"\"\"iterates minibatches of data in random order\"\"\"\n",
    "    while True:\n",
    "        indices = np.arange(len(data))\n",
    "        if shuffle:\n",
    "            indices = np.random.permutation(indices)\n",
    "\n",
    "        for start in range(0, len(indices), batch_size):\n",
    "            batch = make_batch(data.iloc[indices[start : start + batch_size]], **kwargs)\n",
    "            target = batch.pop(target_column)\n",
    "            yield batch, target\n",
    "\n",
    "        if not cycle:\n",
    "            break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "KJJt__lR4Lqq",
    "outputId": "33409466-24d0-4d65-ffd8-1b13d48c5485"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3579"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(categorical_vectorizer.vocabulary_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "id": "kPT23ZyO4Lqr"
   },
   "outputs": [],
   "source": [
    "class ThreeInputsNet(nn.Module):\n",
    "    def __init__(\n",
    "        self,\n",
    "        n_tokens=len(tokens),\n",
    "        n_cat_features=len(categorical_vectorizer.vocabulary_),\n",
    "        hid_size=64,\n",
    "    ):\n",
    "        super().__init__()\n",
    "        \n",
    "        # Define modules to process the title.\n",
    "        self.title_emb = nn.Embedding(n_tokens, hid_size)\n",
    "        self.title_enc = nn.LSTM(hid_size, hid_size, 2, batch_first=True)\n",
    "        \n",
    "        # Define modules to process the description.\n",
    "        self.desc_enc = nn.Sequential(\n",
    "            nn.Embedding(n_tokens, hid_size),\n",
    "            Reorder(),\n",
    "            nn.Conv1d(hid_size, hid_size, kernel_size=3),\n",
    "            nn.ReLU(),\n",
    "            nn.Conv1d(hid_size, hid_size, kernel_size=3),\n",
    "            nn.ReLU(),\n",
    "            nn.Conv1d(hid_size, hid_size, kernel_size=3),\n",
    "            nn.BatchNorm1d(hid_size),\n",
    "            nn.ReLU(),\n",
    "            nn.Conv1d(hid_size, hid_size, kernel_size=3),\n",
    "            nn.BatchNorm1d(hid_size),\n",
    "            nn.ReLU(),\n",
    "            nn.AdaptiveMaxPool1d(1),\n",
    "            nn.Flatten()\n",
    "        )\n",
    "        \n",
    "        # Define modules to process the categorical features.\n",
    "        # self.category_out = ...\n",
    "        self.cat_enc = nn.Sequential(\n",
    "            nn.Linear(n_cat_features, hid_size),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(hid_size, hid_size),\n",
    "            nn.ReLU(),\n",
    "        )\n",
    "\n",
    "        self.out = nn.Linear(3 * hid_size, 1)\n",
    "        \n",
    "    def forward(self, title, desc, category):\n",
    "        \n",
    "        title_beg = self.title_emb(title)\n",
    "        lstm_out, (title_enc, _) = self.title_enc(title_beg)\n",
    "        \n",
    "        assert torch.equal(lstm_out[:, -1],title_enc[-1])\n",
    "        \n",
    "        title_enc = title_enc[-1]\n",
    "\n",
    "        desc_enc = self.desc_enc(desc)\n",
    "\n",
    "        category_enc = self.cat_enc(category)\n",
    "        \n",
    "        #print(title_enc.shape, desc_enc.shape, category_enc.shape)\n",
    "        \n",
    "        concatenated = torch.cat(\n",
    "            [\n",
    "                title_enc.view(title_enc.size(0), -1),\n",
    "                desc_enc.view(desc_enc.size(0), -1),\n",
    "                category_enc.view(category_enc.size(0), -1),\n",
    "            ],\n",
    "            dim=1,\n",
    "        )\n",
    "\n",
    "        out = self.out(concatenated)\n",
    "\n",
    "        return out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 266
    },
    "id": "EdB6cIz14Lqr",
    "outputId": "3f2d7e52-8dfd-40dc-c1e9-872c8738f521"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAD5CAYAAAAk7Y4VAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO2dd5xU5dXHf2dmtldg6QssHQGluIBIE0UFsUZjxMQIisQ3atRYXtRojF3x1Wg0GgsajSJiiwiCKCio9LKA1KUvbQvsAtt35nn/uGXvvXNn5k7bucye7+ezn515bjvTfs+55znPeUgIAYZhGCb+ccTaAIZhGKZpYMFnGIZpJrDgMwzDNBNY8BmGYZoJLPgMwzDNBBZ8hmGYZoIr1gb4IycnR+Tl5cXaDIZhmNOGtWvXlgohWptts7Xg5+XlYc2aNbE2g2EY5rSBiPb52tZkgk9EVwKYCCATwNtCiG+a6toMwzBMmDF8IppJRMVEtNnQPp6IthNRIRFNBwAhxBdCiFsA3ArgN+Fcl2EYhgmecAdt3wUwXttARE4ArwKYAKAvgElE1Fezy1/k7QzDMEwTElZIRwixlIjyDM1DARQKIXYDABF9BOAKItoK4BkAXwsh1vk6JxFNAzANADp37hyOeQzDMKivr0dRURFqampibUpESU5ORm5uLhISEiwfE40YfkcABzTPiwAMA3AHgHEAsoiohxDidbODhRBvAHgDAPLz87myG8MwYVFUVISMjAzk5eWBiGJtTkQQQqCsrAxFRUXo2rWr5eOabNBWCPEygJeb6noMwzAAUFNTE1diDwBEhFatWqGkpCSo46Ix8eoggE6a57lym2WI6DIieqOioiKihjEM0zyJJ7FXCOU1RUPwVwPoSURdiSgRwHUAvgzmBEKIuUKIaVlZWSEZ8M5Pe/DVxkMhHcswDBOvhJuWOQvAcgC9iaiIiG4WQjQAuB3AQgBbAXwshPglfFOt8+HK/Zi38XBTXpJhGMYn6enpsTYBQPhZOpN8tM8HMD+cc4dDSqIT1fXuWF2eYRjGltiyeFq4MfyUBCeq6ljwGYaxF0II3Hfffejfvz/OPPNMzJ49GwBw+PBhjB49GgMHDkT//v2xbNkyuN1uTJ48Wd33xRdfDPv6tqylI4SYC2Bufn7+LaEcn5LoxLHKughbxTDM6c7f5v6CLYdORPScfTtk4q+X9bO072effYYNGzagoKAApaWlGDJkCEaPHo0PP/wQF198MR566CG43W5UVVVhw4YNOHjwIDZvlgoZlJeXh22rLT38cElNZA+fYRj78eOPP2LSpElwOp1o27YtxowZg9WrV2PIkCF455138Oijj2LTpk3IyMhAt27dsHv3btxxxx1YsGABMjMzw76+LT38cElOcKKaBZ9hGANWPfGmZvTo0Vi6dCnmzZuHyZMn489//jN+//vfo6CgAAsXLsTrr7+Ojz/+GDNnzgzrOrb08MON4afyoC3DMDZk1KhRmD17NtxuN0pKSrB06VIMHToU+/btQ9u2bXHLLbdg6tSpWLduHUpLS+HxeHD11VfjiSeewLp1PivSWMaWHn7YMXz28BmGsSFXXXUVli9fjgEDBoCI8Nxzz6Fdu3b497//jRkzZiAhIQHp6el47733cPDgQUyZMgUejwcA8PTTT4d9fVsKfrikJLpQXe+GxyPgcMTfDDuGYU4vTp06BUCaHTtjxgzMmDFDt/3GG2/EjTfe6HVcJLx6LbYM6YRLaqITAFDTwF4+wzCMgi0FP9wYfmayVC70RHVDJM1iGIY5rbGl4IdbS6dVeiIAoPRUbSTNYhjmNEWI+Ku0HsprsqXgh0tOehIAFnyGYaSFQsrKyuJK9JV6+MnJyUEdF5eDtjmqh8+zbRmmuZObm4uioqKga8fbHWXFq2CIS8FvJXv4ZezhM0yzJyEhIahVoeIZW4Z0wh20TUt0ggiorOVBW4ZhGAVbCn64g7ZEhLREF07VclomwzCMgi0FPxJIBdTYw2cYhlGIW8FPT3LhFId0GIZhVOJW8NOSXBzDZxiG0RC3gp+a6EQlF1BjGIZRiVvBT2cPn2EYRoctBT/ctEwASGXBZxiG0WFLwQ83LRMAklwO1LvjZyo1wzBMuNhS8CNBgtOBOrcn1mYwDMPYhrgV/EQnoZ4Fn2EYRiVuBT/B6UB9Aws+wzCMQvwKPsfwGYZhdMSv4Msx/Hiqgc0wDBMOcSv4iU5p8fIGDws+wzAMYFPBj0QefoJTemk8cMswDCNhS8GPRB6+SxH8BvbwGYZhAJsKfiRQQjrzNx+OsSUMwzD2IG4FXwnpPPDZphhbwjAMYw/iXvAZhmEYibhVRZcc0mEYhmEk4lbwOf2eYRhGT9wKfk09L37CMAyjJW4Fv5br6DAMw+iIW8FnD59hGEZP3Ar+1WfnAgBSEpwxtoRhGMYexK3g56Qn4erBuWiZlhhrUxiGYWyBLQU/ErV0ACDRRbzqFcMwjIwtBT8StXQAeREUFnyGYRgANhX8SJHodKCOs3UYhmEAxLngS6teseAzDMMAcS74iU5pmUNe9YphGCbeBd8lvTweuGUYhol3wVdXvWIPn2EYJq4F3+mQ17VlD59hGKZ5CD6vY84wDBPngi/rPTw8aMswDBPfgk+kePgs+AzDMHEt+A5Z8FnvGYZh4l7wpf9uDuIzDMPEueA7OKTDMAyjEN+CzyEdhmEYlTgXfOk/e/gMwzBxL/iS4nMMn2EYpgkFn4i6EdHbRPRJ011T+s96zzAME6bgE9FMIiomos2G9vFEtJ2IColoOgAIIXYLIW4O53rBosy05WqZDMMw4Xv47wIYr20gIieAVwFMANAXwCQi6hvmdULCQVxagWEYRiEswRdCLAVwzNA8FECh7NHXAfgIwBVWz0lE04hoDRGtKSkpCcc8HrRlGIbREI0YfkcABzTPiwB0JKJWRPQ6gEFE9ICvg4UQbwgh8oUQ+a1btw7LEOJBW4ZhGBVXU11ICFEG4Namuh4AODkPn2EYRiUaHv5BAJ00z3PlNssQ0WVE9EZFRUVYhjjkV8chHYZhmOgI/moAPYmoKxElArgOwJfBnEAIMVcIMS0rKyssQ7haJsMwTCPhpmXOArAcQG8iKiKim4UQDQBuB7AQwFYAHwshfgnf1ODhLB2GYZhGworhCyEm+WifD2B+OOeOBJylwzAM04gtSytEKoavDNp62MVnGIaxp+BHPoYfCasYhmFOb2wp+JFCCekcKq+OrSEMwzA2IL4FX1b8e+YUxOT6K3eXYeBj36Ciuj4m12cYhtFiS8GPWB4+RcigEHn6620or6pHYfHJ2BrCMAwDmwp+pGL4SlpmrCg5WQsAyExOiKkdDMMwgE0FP1LEWvCLT9YA4EFjhmHsAQt+FKl3S0rPxdsYhrEDthT8SMXwtXp/87urw7QqdHjiF8MwdsCWgh+pGL5TM2r73bbicM0KGRZ8hmHsgC0FP1LEOqSjwCEdhmHsQJwLfqwtkGAPn2EYOxDXgk828fDZwWcYxg7YUvDjZeKVAod0GIaxA7YU/GgM2sYSrtbJMIwdsKXgRwq7DNqy3jMMYwfiWvAD6X2fh7/GFa/+FHU73DxoyzCMDYhrwQ/k4dfUe1BwoDzqdnBIh2EYO9CsBb+p4LRMhmHsQHwLvk1eHWfpMAxjB8JaxDxaENFlAC7r0aNHWOfx5eEv2V6MrJSmK1nMHj7DMHbAloIvhJgLYG5+fv4t4ZzHl+BPeadpC6m5PU16OYZhGFNsKfiRwpiGnzd9HiYN7dTkdrCHzzCMHbBJlDs6OEwmXs1adSAq1xJ+RJ0Fn2EYOxDXgt9UOTofrzmArg/Mx+GKal270t/woC3DMHYgrgU/GJmtd3twzWs/4/mF24O+zmfrigAAe0oqde1K8TbWe4Zh7EBcC35CEHmZWw6dwJp9x/HKksKgr6MIujGEpDzjiVcMw9iBuBb8lEQnvrpjJK4enBtw39qG0FNpFEE3ZgUpT7m0AsMwdsCWgh+p8sgA0L9jFjKSAycj1YeRO6kMyjoN7yaBdNsZhmFiiS0FP1LlkRWslEk+XFET8vndsp57LbgiP/3Piv34cOX+kM/PMAwTCWwp+JEmkOB7PAL3zilQnwebVaOkZBpDOspltx4+gQc/3xTUORmGYSJNsxD8QEXUjDH2E9X1QZ1fCdkYQzfUZImhDMMwgWkWgu8K4OEbPfpgB3A9HuW/QfBZ7xmGsRHNQvADhfCNnnmwg6zK/saOwy7lmRmGYYBmIvg1ATx2Y8g+2Bi+KvheIR2GYRj70CwE/3hlnd/tRoEP3sOX/xv7FVZ8hmFsRPMQ/Cr/gm8sfBa0h+9hD59hGPvTTATff9aNt4cf3PnVLB2vQVuWfIZh7EOzEPyUBKff7UbPPNSQTkOAnqK2wR3UeRmGYSJJsxD8F34zAE9ddabP7UZ9D3nQNsBYQO+/LAjqvAzDMJHEloIfyVo6ANAmIxnXD+vsc7tRqIOfaSv997ozMDmNv4VSGIZhooktBT/StXQCYRTqYDVZ6SCMHYXZaerdLPgMw8QGWwp+U2NMpwy2nLGv0gpmYwHV9RzHZxgmNrDgw1vg56w5gLHPf48ai+KsOPaKh7+75BSEEKZ3ClbPyTAME2lY8KGtZy+lUX6wcj/2lFbiWIAJW0YaPAKbD1bg/P/7AW8u223q4a/acyx8gxmGYUKABR+Nnvm1+Z107Q0W4+1Kur3HI7CvrAoAsOFAuWkM/45Z67GxqDxkWxmGYUKFBR/ARS8uBQAkOPUTpeq9aiWYoxzlFkL16onIZ0bOkTAWW2EYhgkVFnwNCYY1CkPx8BXBdxD5zPbhKpoMw8QCFnwNLqOHb3GdW2WhE7dO8M3TMgH/dfIraxuwp7TS0nUZhmGCgQVfQ4LD4OFbnIClCLhbNKZ4Ooh8lmjwd9op76zG2Oe/t3RdhmGYYGhWgv/VHSP9bvcO6Xhw4FhVwFRKxWE/VlmrevVEvidw+Tvfqr2cxcMwTHRoVoLfv6P/mbsJLn2spcEjMOq5Jbj1P2v9HqdUxXx1yS61Yqa/OL2VXHy7lGA4XlmHU7UNsTaDYZgI0KwEPxDGkI4Sw/9+e4nlc6hZOn72Ma7ANfXfqzHpjRWG81i+ZFQZ9PgijHx2cazNYBgmAjQ7wb97XC+f24yDtnUBlkYUQqCqrkE3CGtFqGsNHv63W4uxfHeZri3YEs3RpDzAegIMw5weNDvBv3NcTwzINQ/tuAwx/NoAgv/Byv3o+8hCHCqvVtsa8/B9H2clpBNsxU6GYZhANDvBB4CaenMhT7To4f+wowQNbg8W/nIEgN6rF2pIx18MP3C6p40cfIZh4oRmKfj/uH4QLh/QAed2b6VrdzmMHr63J75sZwlunLkKrywpRJLLeyUtf455n3YZACx6+Kz4DMNEGFdTXYiI0gD8E0AdgO+FEB801bWN9GqbgZcnDcLPu0rx867G2LmVGP7RE7UAgP1lVUhK8O4vldx9s5BOZnICMpJdaPAIVNe5keAkrzCSAod0GIaJNGF5+EQ0k4iKiWizoX08EW0nokIimi43/wrAJ0KIWwBcHs51I0WyYa3bxCBj+MkmHn6Dn9m5DodUkdMjBM54ZIFXuuffv92hPrZLWibDMPFDuCGddwGM1zYQkRPAqwAmAOgLYBIR9QWQC+CAvJstisIbBdvobdeZiLcqxAS4HN5uvJLKaebhOx0EJ5HqvX+7tVi3/e/f7lQfs4fPMEykCUvwhRBLARinhg4FUCiE2C2EqAPwEYArABRBEn2/1yWiaUS0hojWlJRYz38PhWRDSCZQSGfexsNYsl0SaQKZrl7VWI7BW/EdRHA4fJdc0HI66H11nRsPfLYJFZy2yTCnBdEYtO2IRk8ekIS+I4DPAFxNRK8BmOvrYCHEG0KIfCFEfuvWraNgXiPBhnRu+3Ad5m+SMnOIgKo67xmo/ipsOkjv4fvD2CnMWLgNedPnBTyuKZmz9gBmrdqPFzWhqFDZWFSOzQcjs2g908gX6w/i5ndXx9oMxiY02aCtEKISwJSmup4VjIJvDNEEmnhVWevt4QcM6TgIVopwGjuFV5fsAiCFlMqr6lHv8aBNRnLgE0URpYxEJMYbLn/lJwDA3mcmhn0uppG7Zm+ItQmMjYiGh38QgHbpqFy5zTJEdBkRvVFREV2PzxjSSXDpn7/94x6fxxL0C5IrnYUS9zfLwpdCOtZm0fqrtDno8UUY+uR3Ac/BMAyjJRqCvxpATyLqSkSJAK4D8GUwJxBCzBVCTMvK8l/sLFyMg7bGWjr+IAIaNCtiKZU2/d0VOB2wHtLxcRoezGUYJlTCTcucBWA5gN5EVERENwshGgDcDmAhgK0APhZC/BK+qZHHYQjhGAdt/UEgXbxeOVYRfDNddjoIDiJLk6p8e/gs+AzDhEZYMXwhxCQf7fMBzA/n3LHAWA9fi8eg4ET6FbFUD19uc5u46CRn6fjL1Vfw1Smwh88wTKjYsrRCU8XwAeCWUV3Vx8ZFzLXUmJRZ0K6IpcTwa+U6OWarZTnlLB0ra+X6GghlD59hmFCxpeA3VQwfAB6a2Bcds1MA+M99r67TCz6RPgVT8fCV+jvGOwJADuk4yHRClxFfu/iK7TOMP3jmNgPYVPCbmpREafDWLAyj4D3JigwhHcnDL6usA2Du4TuI4HT4z9VX8BW64aJqTChwJJABbCr4TRnSAYA3f5+P3w/vgm456fjLxDNM9zH18D3eHv7REzUAzEMvDpLCOg0W3HRfoZtYx/BveHslHv5ic+AdGVvBoUAGsKngN2VIBwC65qThsSv6wyFn0ZhhrGFP0A/aKnV4Sk5K1TTNvHgiyCEd/bbs1ASvfe2apbNsZyneX7EPAMAScvoQ6+8NYw9sKfixxGlSEA0A6twmg7a6GL50nOKAm3niDnXQVt95mGUH+XLkY+3hm0H+lvdibAHrPQOw4HvhQ++96up4hNCFZoxlGcxi7YqHr+0o3B5hOqDmS9jZU2NCgb83DMCC70X/juZhJOMM2nq3QL1Jlo6CuWATHATUazqKfyzeaerN+wzpNGGWDmd2xA82vDFkYoAtBb+pB221DOrcAusevtCr3Sj4Ro/fKPjLdpZ6nYNIChlpPfxNRRWmnYNZWifQtFk6rPfxgx1DgUzTY0vBb+pBWyMt0xK92oy588Z1aa2UZSBIcXxtDD/R5TD15n0J+1cFhwJeJ1JwGCB+4Ls1BrCp4NsRo4dvFHx/ZRm0+zgNWTqJLoepJ+3r9/l/ixprz/u6C4gUVk/PYmJ/2MFnABZ8yxgF3xiy8VeWQUFZ4tBYZdPUw7fwCzWb3BVJAnn4nJtz+sB3awzAgm+Z6Z9t8rvdiofvcpBXls7JmnrTH6OVH2i047KsEfEDCz4D2FTwYzloGyouH7X0p0/ooz5WPHzteMDCX456TeoCrP1ArczYDQfBU6tCZlfJKeRNn4dtR07E2hQAoXXe24+ctFTZlTl9sKXgx3rQNhR8hXT+MLobxvSS1uZ1yUscWiqPbOF3ZtT7++YU4Df/Wh74QItw3Dd0vt50GADw5YamG2T3R7Ae/t7SSlz896V4dsG2KFnExAJbCv7piK8sHSLCWblSx+V0OOBwkCUh9fUD1U5qNXr4c9YWYeWeY9YMtkAgkeD+wDfKZ+yrVEdTE2znXVYplQhZu+94FKxhYgULfoTwFdIBGgdXXU7yOZPXiHaB8Je/26m292qToT6Oegyf7+ZDRuksjauqxYpQM7q4U48vWPAjRJLL91upCLMSw7eC8vssKKrAC5pUTO0s3Vhn6ShwLR1vGj382NqhEPygrU0MjwJlp2qx9bA9xlaaGhb8MElOkN7CpASnz32UrBwlS8cKysSrekMwX5seGm0PnzM7QkeZm3C6hnQU4vErMP6lZZjw0rJYmxETWPB9MHNyvlfb5HPzvNraZCQDCOThSyLtCsbDl3+hxltxbQcQfQ8/qqePa5TO+HT18JWvaTx+BZQS5s0RWwq+HdIyz+/TFr3bZujaLh/YwWu/Ti2l5RGNM28BICNJWiO+XgnpOB2WPXzlB2oUXWOlzWjCM2hDR/lo7BLuCvaztIfVTKSxpeDbJS1T+a2e11tKq2ydnuS1T252KgDgVG2D17av/jQSAODWhHQszM+SjtEM2mrxFdKJhjg3lYcf7RIRsUCZw3C6h3TiMqbTjLGl4NuNey7sjcX3jEGnlqn46o6R6NEmXd2W20Ly8E/VeAt+l1ZpABrj8cEM2irhGuMPtU4X0vGYtkeKgGmZEdKCeBwr8Jz2IR3J8Pj7ZJo3LPh+ULwzIqBba0nk+3fMQlZK45KE4/q21f03Q5loFcygreLJG3+ouhi+JrxjXHP3mLyYejg0VR5+HDr46mvytYJaODS4PUHP4HV7BGav3o9KkztRM5TPPg77YpXmGLJkwfeDOnDl53txRvtM7H7qElzcr53PfRo0aZlWb/HrGqQf9caicl27Vhy1Hn6VRvAXbD6MwY8vwj0fF+CJr7ZYup4ZgX4PkfrBBON9vrBoB55fuD0i140mymuKRgz/te93Yfzfl2HLIeuiv2L3Mfzvp5vw6Je/WNpfnQcSxz5+PDoagXDF2gA705ipoP9mpBhSMI1e++NX9seI7q3U50qs3eVweB3ri0VbjuLJ+Vv97lPX0GiXVvA/Wn0AAPDpuiIAwPXDOiM9yYWURCcykr0XTPdFICGOVCgmmPP8XFiKBo/AvRf3jsi1o4XykqIR0tldWgkA2HSwHH07ZFo6Rgk5lpyylqHSHBZMafB44HRY+z3GC+zh+yHZJX0ZyJCz8PyvB+A3+Z2w4K5Rpsf1bpuhhoAA/UxbbTjIH6v2Bi6RoA3vaEM6328v0e139EQthj71HS74vx8sXVvB+Js3evSR0oRgzlPb4DHNiLIbnijm4XfIllKBDx6vDtoeq32ruxmEdJpDp2aEBd8PL08ahNvGdkc/gxfVLisZz15zFvq0M/eujHHbKwd2BAD065BpWfD9oSyYrg3pVNb5js1e/9YKAEBxkPnHgQQ+Gh5+oIyd2gY36ho8+GL9QVT5ec2xpjGkE/lzK/38y4sLsfmgtdRlYfgfiKZcOzlWRGMeS1VdA95attu2mWe2FHw75OEDQIfsFNx3cZ+g66EYBX/iWe2x95mJyG2RiswICL4SFtKGdIyDtlpC1eVAAh+xLB3NhQJ1IrUNHuwurcRdszfg4S+sxaNjQTSrCms7upk/7rF2kOqxW/vQmoWH7478i3vm6214Yt5WfLPlaMTPHQlsKfh2ycMPFZefDkIpxRAOShkHbUinyo/gh4q3h28U/Eh5+OaPzdDOQ9h/rDIi148GynsTjbCBds6HlYV3gOAzqhoHbeOXaHj4x6vqAUh3onaEB22jQDRS8bQonYZe8CMf3vCO4fvfHvp1gvPwFYzLTtoJTwQF/1hlHeYWHELnVqlIdjlRVdsoJr7KchsJtm9uDvHtaLzGaGZnRQIW/Cjgz8M/p1srn9uskmzi4VdHYSDTKL6Bnod8Hd2M4cb2vOnzcG1+Lp67ZoDaVqt5nbW2Fnzlf/jv0YOfbcKCX46oz0f1zFEfW/XwQx+0jV/hd0fjtSklNSJ/5ohgy5DO6Yqy6pW/mH9yghODOmeHdZ1y+baxThODrKy1JvjB3Gp6C7xxu/TfnyiUWUgD1J7X+CP8eE2R7rnOw7fx8nu+aiGFgnFAXh/SCa7cttW8ersOOkaSYGL4tQ1uS9lhdiupYYQFP4JMGtoZAJAYwOvyV1lTy43Du+ie920vZQWVyiJa3+BBvduDP36wFqv2lFk6Z02ddZH0DuGYx/B9aUPBgXKc/cS3+GL9Qb/XsRrScXuELu5aW+/Bi4t2YPE2+w2QKS8jEmGD1ER9rnhFdb36OPoevrX9T0eCWRN60GOLMOBv3wTcTzmlTfWeBT+S/PWyfph7+0h0apnqd79ki5OvWmfoi7UpRdwU6t0eHKmowfxNR7DEkHvvi6p667F+rfgKIXQrYAkhNF6suSpskReZWL7Lf2ekKwLn8W5TMMbsD5ZX46XvduK2D9b7PX8sUN+biAi+PvJ6IgTBV947qwIe7B3B6UgwnXFVndtSCFF5v2yq9yz4kcTpIJyZGziz6KK+UhmGT24drrb9cN95XvvlGKpzZqfqUzoPlVfjzo+CEzt/6ZtGtL8HIYwdgDZODXy8+gAWauLMyj5AYG9HK0LKNYwLvwC+w1HRGL8wcriiGoXFpyzvr4hJJCIjKQYPv/RUY52kRIt3i8F4s0DzCOlEI0un8TtvT8nnQdsYMGloJ0zo3w4t0hLVNqWyphajh5+dmqh7/u/l+4K+djDiqBN46OPrbo2HL4TA/Z9uBADsfWai1/GBvvxmIR0z78uXhxWJVNdADH96MQD96/OHUtgu2IHBFbvLsPXwCUwZ0VVtS/VzR2g1I6y+QUmztJiHr5bntrT7aUl0snSk/3apkmqEBT8GEJFO7H1hnKTVQiP4LgeF5KEEU5ZAGIRYK8xVtW5VDHyFdJTWQM6OW3cd6X+DyYBabb254LdIDfxeNjXKgLJVT3nzwQrsK6vCbR+uAyCtrqZ0lP5CgFazgJS1kION4ccz0fHw7Z2WySEdG/HulCG650o2joI2pGM1dmuk2seg7ZJtxdhUpJ/ZrJ8QJXSDr3/6aH3jMoxe+foCi7cdVctCB/J2tB3LyRrpNdebhCB8hXRC+Wl9vr4IFYb3N5IoISkrglxYfAqX/uNHVewB/WfvdyDbYqZJfZCzSpvDxKtoePjKGe26xgMLfox5/XeD8d09YwAA5/Vug11PXYLv7hmD4d1aYXj3Vlh2/1h13+wUreCH5kEYQzobi8oxb+NhTHl3NS575UcAQHlVHQ6WV6NeE0K55KVleGr+NvX5DztKUCWfy/jdXrytGDe9uwb/+mE3AO/iczX1bkz/dCOKT9YA0JchuOHtVQDMPfyCIvNSG4o3/eqSwoADxMprvnt2AR6dG73SDEr4yYqnPO4F76J2B8sbC6OZeaIvTxrkc5sZymdpVYaaRR5+FD18u46B2FLw7VJLpykY3789umsqazodhO6t0zFr2jlIT3LpMn60g7hWB+uMVNe7sWRbMT5cuR8AcPkrP+k8y3kbD2PgYzcctIMAABtoSURBVIsw4pnFunINu0q8yxgo5zCKwtETUtrokROSoBs9/C83HMJHqw/gpW93AtDPElaEzmzQ9t45BaavSclAmbFwOya9uQL7y6pM91PYJ283G8DeW1qJRVuO4tp/Lbf0o739w3XImz4Pry4pNLVJ9ZSFsCSeafIA7QcrG8dnzDq/FqkJcDrIr2hpr6e+n1azdGwqWJEk2IFsKyhvm11DYrYU/NO9lk60yEpJwJ0X9MQ7U4bgjPbW6qAbKSw+hSnvrsaDn28y3a4VfyWtMhDG9E3juY3xzPJqKcskyeXE8l1lWLazVLe9we3x8lz9ZRfVuT06cbt9liTC3201z88/KndE7bKSde1HKmpw3vPf45b31mDVnmN+K5AqfLXxMACps9HZpK5YJj3u+sB8vCh3cP5QJu3NWnVAzQoyE6YEpwNuj8ArSwp9lpjQdgb1fhY0EUKgsPik6bH2lK3IsO2w/jV7PEINQ4aK8n7ZtTSFLQWfMcfhINx9YS+M7d0G487wXlLx2vxc9fHsaeeYnuPl7xpFJ9DqRy8s2mHJLu13u9KPMP9yqAKfry/CoXJJcNOTXZj05gqv6xw9WQu3QeSOVflesrGuwaOLUe+ShfK173eZ7q9MXMpM1ucsKB2BgllBOqs/ZMWjdnuEegfz8nc78XNhKfaUVnpdS+GkZm3kwuJTmLFwm+ndjnYM55stR7y2A/qZyPV+csjf+Wkvxr2wFOv3H8fPhaU4cKwK6ttpT92KCI8ZVoO7c/YG9Hjo67DOGc2ieZGAs3ROU64+OxffbSvG0h3ShKufp5+PDtkpaikCY+62Ge/+vDeka7fLTFbDNQDwZcEh9fFPhaVe+yvTzCe+/KOuvei4PvTy1FVn4sHPN+GH7SU4pIlhj3x2sd8lJD0CurVakxKcph3PF+sPokebdDVTqVYWxNV7j6FdZrJXBtOp2gYYu9W6Bo+l97ZOM2ir7Tiuf2ul+njP05f4zea4e/YGVNe71RnWWrSzubcfOYlLz/I+vl5TPrumwXy8BQAK5GU095ZV4u7ZBUhyOXDvRdKKYmaD5/HKXM332BdCCL+fmdkM6/1lVVi19xiuOTvXx1FNBwv+acDKBy/w8vLSk1x476ahyJs+D4B3uQZloZX8Li3QuVUqPlvnv7xBMCT5yXv/w/trvdpm/rQHo3rleLXvPKqfyDSwk1RjyBgSKjpejbcD1H3Xlxto/EE+/MVmvL9iH+bcOhyPfbUFqYlOFMkrRSlpnr9+fTkA4B1DlpTZgt+1DW5rgi971Ov3l/ssXV16qs5rroUWZYDd1MN3Nb5GZSLWofJqtM9KVgXp8XmNHuz6/ZKom/mdSi6/cpdU2+BRY9C+UmH9sWR7MfK7tAhqOU07UV3nxvTPNuK+i3sjt4V+1nyDR/hNmFBCZtow55X//AnHKutwxcAOIWfXRQoO6ZwGtM1M9vriGVFq5P9qUEfcNa4nurRKw3NXn4XXbzgbf72sH64b0ili9oQyHjXlndVebZsMqzX1aJOOc7q1DMkm7SzfxrIAwPsrpMHPX7++HNV1blXsAe+JXNpwCqAvUubrGC2VtQ1q6qgi0psOVmBfmXnd/qtf+1mXjeOLnSYzfLXC4fEIbDhQjnOfWYw5axuLzX2ytsjrOLOBY6W6q/a9UTzU4pO1+K28YpqWoydqvGZWA8CBY1WY8s5q3Ddno7+XBABYteeY6XscKerdHjz8xWavRWI6y4kQxhpFCt9sOYL/bjhkGvI063y1KG+vdgzqWKXUIRu/X7GABT9OUG7xX/jNQNw1rhcA4NohnZCTnoSslAQ8c7X+nl+71OIN53RBRrL1m71oxCeTXA4kuhwYkBtaJdGnv25MGa2RPeq1+47r9jGmpM5atV8nuNoaNUBjBVLtQJ6/Gvz9/roQ170hiaO2Y9jrI2to/7EqzFiwzXRbILQhnZoGtxpK2xpgoN3cw5fOpR3f0XYMPxWWec2B+N1bK/GH99d6tZ+Q51HsKfW/OE3xyRpc+6/l+N9PN6Kw+BRWW1jDOViW7SzB+yv24bGvtqBcMwakeOgtUhNxsqbeKyFA+YwbNBlWCtr1oncePak7L+BdQ0nbKRq/X7GABf80Z8Fdo3Dfxb2DStP8y8Qz8KymA+jVLkO9A/jjed299t/8t4ux44kJGCsXb4vGpBKlA+rVNiPsc50Mwmt8b/le9XGFl+BL56nRiHdtgxvbj5zEmBlL8Nk6bw9aCZ1oOwZfHj7QmMIaLFoPv7LWrY55tAww61j56E7U1OOiF3/AF+sPYtaq/V77GTtLo3e675jUiZ2oNtwVyfu5nITdJae8zgNIIZP7P5HuAHYVn8K4F35Qw2qhUHaqFoXFJ1FT75bKfHxSgOmfbsSPOxvnZChettsj1BTjqroGXPPacgx76ltd+E4ResWx0SYE/PGDxiy2C19ciitf/Ulni+ILPfzfXzBv42FdiNP4/YoFHMM/zenTLtPnYupGXp40CIfKqzF1VDcAwMZHL8KMBdtx1aCOeHuZdNs7tk8bnNE+E3fMaizKlp4kfU26tU7Hku0lOg//nclD8OT8rV6FxWbdcg4mvekdCvC2PwPbjpxEm0wplq38D0RuixRdCCJUlMlhAFBiWORd6Ti0C4U/9PlmrNwjeaN//th8XoAyrqJQcKDc5/V3lVgvyKZFG0euqmtARbVDZ7MvNh+swPHKOhSWnMKOo6dw1+wNpvsZq6++tWwPpk/og6U7SjBr1X4kOAh1kERMOw7xG/kOx+V04I8frMO2Iyfx18v64uqzc5Epx/Rn/rRH9ZSN73koXPTiUpTJgr7grlFeaygAUgfVrXU6nv+mMX32eFW9uiShMqcEaFxkRwnf+MvX97p70/hC93+i/34odz+xhAW/GXH5gA6655nJCXj8yv4AgGmju2Fc3zbo1yEL32oWYP5g6jD1sbKcXnqyC8XyD7VdVjKuHNgBz3/TmFp5//je6Nm2cTKZlov7tcXCX6Tz/+fmYSgoKse2I9uRnSJ5pq3SAgt+wSMXITXJifNmfI/cFimqAIfLeoMwf7z6AAZ1ysZzmrBLKNfyNUMYgPo+KqQnuSzFtRM0d3Q/a2YXv7F0N9KTXLjj/B6mxzV4BB74bJPlORYKr/+wCx2zk/Hwf/VxbcVrraiqx7Yjjed0ErDtiJTn/re5W/Dsgm04p1srzLxxiE74FKEGgOOVdWqNqR93liI7NQH9O5rPxfF4BK567We0yUjSneN7Q0flIMnrNhtD0vLk/K3q40fnSoPdK3Yfw5q9x9DTcNdZcrLWq3LtiZp6/GfFPl1WkzFTbMo7q1H41CVe1/65sBTts1PQNce7gGKk4ZAOA0BK4+zXQfpxKROOLuzbFiN6aJbTk2O92o7DQQSXJrzw0nUDMW1UN7RKS8RVgzp6XUdbFXRkzxx14Ey5i8jJ0Ick7rygJx68pI+uLSs1AQlOB36afj7uH9+4rWebdHTMTgniVesxeuKbDlbg0n/8iHX7fXvowXD7WHMR1mLMANEuZ7j9ifHqY3+L7LywaAdqNNk1Q7vqB8IX/HIE+4/5n41shlHsAWkh+boGD6a+t1r17gF4vWc19R58v70EFdX1PmfxDnp8ET5dWwSPR+B3b6/Epf+Q0nifnr8VI55ZjFO1Ddhx9CTqGjzYXXoKBQfKsWiLfnLdM1/rx0S0s9hD4ZrXl3stfDLkyW9140FHKmrwyuJCPLdguxrSM6PBIzBnzQGvQeTr31qJsc9/H5adVmEPn/FidM/WOL9PG/zt8n66diV9z+MReP7XA3DvnAJ0yE5W48mTz83DFQMbRf7F3wzEmR2zkOBywEFShc1rh3TCG0sbwyhKeKhXO8mLMsagR/XMQX5eS3RqkYrqejc2Grxl7WzZRX8eg5e/22l5wliwdMxOsZRVY0ZyggP3XNQLrxhKMBhJTXQhJz1Jzcy5dUx3dSZykqsxqyRQet/TXzd6rDeNyMMqC3cmD1/aF48bJiMF4u7ZBZhbcNiv0GmprGvAKT/Lcd4zpwBvLtuta/uX/H2Z+u/VWLE7uDus7q3TTbOcQiE5waF2pNqB3mv/tdxrcSJfx98nj138vKsMlw/sgPwuLSJim1VY8BkvWqQlYubkIV7tivfZ4BG45uxcdSLJBX3a4PGvtuBKE4/+ppFdvdqmjuyKi+SJVFcO6gi3R6j1311OBwbkZqlhEGXwc8KZ7QEAvxqsn7zSxpDHfvvYHhjVMwcDO2XjhUU78I/F/gU2GJTB6pZpiThWWedVy6ZLq1TsK6tCosuBz/7nXPywowQ/7izF8t1lGJCbrZuwc+9FvXCipkHX+QHSGghf3DZCHQfwVRo5UB389zRrJWg7ihapCWrc2sjNI7viSEU13ly2B2d2zEKDRwTM+gGkYnkuE3uU90nL9W+uRItU//n5SihIQQlzWRX7/9w8DL97W5rg1r1NGhCgRt5jV/TDIyZ3L0baZSarMfub3m0MEe0/VoXiAIPvAzpl6+4gv916FN/6KP0RTTikw1hGCd0Y69zk5aRh7zMT1YlTgfjLpX3VMENOehL+MKa7LsvojvN7qo+HdWvl91wJTgd6tU1Xwz4OB2FQ5xYgIgzu7O093TTCuwMCvFcTy22RgltGdcUbN5yttimC/+zVZ2H9wxeqhc5uH9sDt4/toS5Q8/aN+ejfMQu3je2BJ6+SxkjuvrCX7vyXD+iIPu28M5KMq5ylJjoxskcOfjXYuzNVWPngBVj54AUY4OP913YO2jswLT9NPx8A8OAlZ2DnkxMw946R+MvEM9TtwwN8DmZVO41iD0jiWFBUgR5t0jF1ZFfkpAfKKhKWF3kBpLuwkT1z1PeiW07gkM51QzrjIx+lSLS0zWy8m/zlkL4jXGAyJ0F3rJ8Jdgrvr9iH5bvKsKmoAocrwk9IMIMFn7HMtfmdMKxrS9xs4rVHEkVYL+jTxtKP/Zu7x2DaaO900sFdWiA7NQEfTh2mxsJ/nZ+Lyefm4ZXrB+Fqzd2CMZT04/+ej4cm9lXvRCS7pP+ZyS60SEtUxfnGc/Nw78W9USUPtmpFu1vrdOx9ZiLOMQhmcqLDdLzBOPEsNdGJ/0wdhheuHei1b++2GRjcORttM5PRNjPZ1MsGpMqN3XLScMM5XXyut6wILxGp4SKtwM3yIYhTA3wXjGs8KPRtn4m/XNoXc2491+/xD36+Kah0xk/+R1o2NEMeE2pvKJCn0E7z2hJd5p/Fa78drHverbU0/tSvg3lW3J98DJTfOLwLfndOlwCWS7PCJ725Ape98iN+9c+fA+4fChzSYSzTMi0Rs/8wPPCOYaIIa7irBmWlJGDDIxcBAM7tkYMDx6rQqWUqHpXHJi49qwOmT+iDT9cVISPZhYc+34xebdPRPkv/4585OR/ts1JwgxwmSJPFZObkIZi36bAqlr3bZWBn8SkvL92MlAQnOrbQX+fxK/p5CYOxjMM53VqqoY2Fd4/WbTPrHIfktcCwri3x3T1jQESYs+aAbnu3nDTsLq3UhX0U2gZIkb15ZFec36cN3tIMQt41rif+rqkKel7vNtj7zEQUFp/EuBeWApAGnJXXaVyeMsnl0E1am7VKsjcz2YUTAWaqfjh1mPrZzfj1Wfhw5X4M6Wo+c7ve7cEHU4dhzd7j8mv17hhG92qN8f3aqd57npxw0DYzWfXwx53RBt9uLUabjCQM7doKgHcI8X/O6xH03JXDFebF9cKFBZ+xHed0a4k2GUk+UwtDxcy7bZ2RhFvHdIcQAkPzWnql4AHA+X2kEmpKR6TMSs7LScNtmsybZ68+C1NG5Pmtj6OQnOBEiiE+r4SitKQm6n+i7900zOfKX4qH/9J1AzEgNxv1bo/X6zF2IJ//cQQOHDfP2PFVC+fV6wfjWFUdbjini26C2TO/OhPXDe2MqwfnYtRzS3THpCc1nmvHkxPUx20zknFu91ZqaumkoZ3Von6JLgfqGjy47+LeqHd71I5Euz8gpRQ/eElj+AkA2mel4B65AJwZdW4PRvTIUbPQEl0OPH5lf/Rona7OH0lLcuH1G85Wx1Mu6tcOT3+9Db8a3BGLtxUDAHq0ycC3W4tR5/ZgQKcsdGqZggPH9OGY5ASHOgfBSNvMJJ+T707U1Ps8LlSaLKRDRN2I6G0i+qSprsmcnmSnJmLVQ+N8xqSjARGZir0WZYq94uEbSUty4ewu1moBJTgdcDkd2PvMRL9rGxg7hUSXw6cQ33txb3TMTsHYPm2Ql5Nm+nqyDOskZ/nJdQeA1393NhbJdxJTRuThsgEdMPGs9rhB9tATXQ5se3w8HrlUmlwFmHesLdMS0S0nDY9foc/8cjhIlyDw8KV91cdz/jAcr1w/CLeN7YEOsuc+pldrTD43DwBwVm4W7hrXE3eN6wkr/PG87vinHKYxW1TmhnO6YHj3VujRRh/3f//moZg6siu6ymNVl57VAf07Sp/ZiB5SqK68qh4ZyQlYdv/5XudNTnDC4SC8M3kIxp3RRrft7nG98JjhPenTLgNL7xurhqUiiaUzEtFMAJcCKBZC9Ne0jwfwEgAngLeEEM/4OocQYjeAm1nwmdOV64Z2xmvf71LnDEQKZfDXZVKFMZgBy8GdW6iDr74Y0T0H0yf0wZcbDlmKK4/v3ziG8dfL+pnuk5zg9MrG+mDqMF2YJNHlwOJ7z/N5vILTQfjunjEoPVmLAZ2y1U6/T3up89p/rErNge/bPlOtG+WPJ6/qj7X7juP+8X3U2jf+iqB9euu5KDnVGFIZ1bM1RvXUp13OnjYc24+exCDZPn/FCZU5E2P7tMGPhvLhKYlOtE7S3xFmJLvQuZX/YomhYvWb+y6AVwC8pzQQkRPAqwAuBFAEYDURfQlJ/J82HH+TEKI4bGsZJobcd1Fv3HlBT5+pkqHyj+sHYc6aIvSOQB2hQDgchFvHdMetY7wHuSOJdsKeVZSZpt1bp3tNmFJqLF1zdi4u6tsOk8/Ns+zZ/3ZYF/x2WOMdCeB/LeCs1ARkBUgdTUtyqVlg258Yr05KVEhNdOK/t43A8t1l6ipmgHfhwZQEJ1qm6RMGjEX+IoklwRdCLCWiPEPzUACFsucOIvoIwBVCiKch3Q2EBBFNAzANADp37hzqaRgm4jgchGRHeGL/3DVnYWORfpJS+6wU/OkCvXi9+ft8rN/vXXgsXtnwyIV+CwAmJzix66lL1DueRy83v9sIhL8ZyqFiHPCe/6dRaJuZhFbpSV5hNaPgewRwRvtMDOyUjZtHdsUds9bjbJN04kgRzr1pRwDaIf8iAMN87AsiagXgSQCDiOgBuWPwQgjxBoA3ACA/P9+e64QxTIhcm98J1+YHXpvgwr5tcWFf72Us45XsAFU+geDCW75Q5pKc36dNgD1Dp6+PtE2g8Q4j0elAnduDmno3khOc+OK2EQCAnm3To1pTp8mydIQQZQBubarrMQzDmLHqwQuQmRKb1bjuHNcTCU4HbhnVFa//sAsTztQv3Wm18m2ohCP4BwFoXZVcuS1siOgyAJf16BHZtDyGYZg2Jjn3TUVmcgKmT5BmhT80sW+AvSNPOAGt1QB6ElFXIkoEcB2ALyNhlBBirhBiWlaW73QxhmEYJjgsCT4RzQKwHEBvIioiopuFEA0AbgewEMBWAB8LIQJXIGIYhmFigtUsnUk+2ucDmB9Ri8AhHYZhmGhgy+JpHNJhGIaJPLYUfIZhGCbysOAzDMM0E2wp+ER0GRG9UVHhe/FnhmEYJjhsKfgcw2cYhok8JIIszN+UEFEJgH0BdzQnB0BpwL1iB9sXHmxfeNjdPsD+NtrVvi5CCNNV1W0t+OFARGuEEPmxtsMXbF94sH3hYXf7APvbaHf7zLBlSIdhGIaJPCz4DMMwzYR4Fvw3Ym1AANi+8GD7wsPu9gH2t9Hu9nkRtzF8hmEYRk88e/gMwzCMBhZ8hmGYZkLcCT4RjSei7URUSETTY2jHTCIqJqLNmraWRLSIiHbK/1vI7UREL8s2bySiwVG2rRMRLSGiLUT0CxHdaSf75GsmE9EqIiqQbfyb3N6ViFbKtsyW12IAESXJzwvl7XlNYKOTiNYT0Vd2s02+7l4i2kREG4hojdxmp884m4g+IaJtRLSViIbbxT4i6i2/b8rfCSK6yy72hYwQIm7+ADgB7ALQDUAigAIAfWNky2gAgwFs1rQ9B2C6/Hg6gGflx5cA+BoAATgHwMoo29YewGD5cQaAHQD62sU++ZoEIF1+nABgpXztjwFcJ7e/DuB/5Md/BPC6/Pg6ALObwMY/A/gQwFfyc9vYJl9rL4AcQ5udPuN/A5gqP04EkG0n+zR2OgEcAdDFjvYF9VpibUCEP5jhABZqnj8A4IEY2pNnEPztANrLj9sD2C4//heASWb7NZGd/wVwoY3tSwWwDsAwSDMbXcbPG9JCPMPlxy55P4qiTbkAvgNwPoCv5B+6LWzT2Ggm+Lb4jAFkAdhjfB/sYp/BposA/GRX+4L5i7eQTkcABzTPi+Q2u9BWCHFYfnwEQFv5cczslsMLgyB50LayTw6ZbABQDGARpLu3ciGttma0Q7VR3l4BoFUUzfs7gPsBeOTnrWxkm4IA8A0RrSWiaXKbXT7jrgBKALwjh8XeIqI0G9mn5ToAs+THdrTPMvEm+KcNQnIDYpoTS0TpAD4FcJcQ4oR2mx3sE0K4hRADIXnTQwH0iaU9CkR0KYBiIcTaWNsSgJFCiMEAJgC4jYhGazfG+DN2QQp5viaEGASgElKIRMUO30F5HOZyAHOM2+xgX7DEm+AfBNBJ8zxXbrMLR4moPQDI/4vl9ia3m4gSIIn9B0KIz+xmnxYhRDmAJZDCJNlEpCzNqbVDtVHengWgLEomjQBwORHtBfARpLDOSzaxTUUIcVD+Xwzgc0idpl0+4yIARUKIlfLzTyB1AHaxT2ECgHVCiKPyc7vZFxTxJvirAfSUsyUSId2KfRljm7R8CeBG+fGNkGLnSvvv5ZH+cwBUaG4bIw4REYC3AWwVQrxgN/tkG1sTUbb8OAXSGMNWSMJ/jQ8bFduvAbBY9sAijhDiASFErhAiD9J3bLEQ4rd2sE2BiNKIKEN5DCkOvRk2+YyFEEcAHCCi3nLTBQC22MU+DZPQGM5R7LCTfcER60GESP9BGi3fASne+1AM7ZgF4DCAekjezM2Q4rbfAdgJ4FsALeV9CcCrss2bAORH2baRkG5FNwLYIP9dYhf75GueBWC9bONmAI/I7d0ArAJQCOk2O0luT5afF8rbuzXR53weGrN0bGObbEuB/PeL8luw2Wc8EMAa+TP+AkALm9mXBulOLEvTZhv7Qvnj0goMwzDNhHgL6TAMwzA+YMFnGIZpJrDgMwzDNBNY8BmGYZoJLPgMwzDNBBZ8hmGYZgILPsMwTDPh/wGKXvW1EOfeoQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "epochs = 1\n",
    "model = ThreeInputsNet()\n",
    "model.to(device)\n",
    "opt = torch.optim.Adam(model.parameters())\n",
    "criterion = nn.MSELoss().to(device)\n",
    "\n",
    "history = []\n",
    "for epoch in range(epochs):\n",
    "    for batch, target in iterate_minibatches(data_train, word_dropout=0.2):\n",
    "        # Preprocessing the batch data and target\n",
    "        \n",
    "        title = torch.tensor(batch[\"Title\"], dtype=torch.long, device=device)\n",
    "        desc = torch.tensor(batch[\"FullDescription\"], dtype=torch.long, device=device)\n",
    "        category = torch.tensor(batch[\"Categorical\"], dtype=torch.float32, device=device)\n",
    "        \n",
    "        target = torch.tensor(target, device=device)\n",
    "\n",
    "        predictions = model(title, desc, category)\n",
    "        predictions = predictions.flatten()\n",
    "\n",
    "        opt.zero_grad()\n",
    "        # Compute loss.\n",
    "\n",
    "        loss = criterion(predictions, target)\n",
    "        \n",
    "        loss.backward()\n",
    "        torch.nn.utils.clip_grad_norm_(model.parameters(), 10)\n",
    "        opt.step()\n",
    "        # Train with backprop.\n",
    "\n",
    "        history.append(loss.item())\n",
    "        if len(history) % 10 == 0:\n",
    "            clear_output(wait=True)\n",
    "            plt.plot(history, label=\"loss\")\n",
    "            plt.yscale('log')\n",
    "            plt.legend()\n",
    "            plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 185,
     "referenced_widgets": [
      "09f5fa8ca136495db238b788124f2fae",
      "7d189f0386e54da39c49506ca4023958",
      "e6b7ad856643400ea7f410a8276fcb81",
      "68c14164f5774e5fa7c7c393b0c6f92e",
      "332aca2eb6854b13b0a5bbdda4e19f67",
      "0f72a7f7e8584bd8acece5ca4a207c5b",
      "b4a2794c7bd8437e9c8d52d7126fcbda",
      "af5f02089b6243ac8541a6f9ad1bab50",
      "e8c59862f3aa47398c7be3fdcfa3b3c5",
      "4b377fd7194547439aee1ab1330fc71a",
      "394331689a594b3bb3a9a0dd1b8025c8",
      "8e084bb71f8f41549557da0879f9f48b",
      "e82d3d9015f34147bc3020a82737380c",
      "616fc3f2afb8432fb53d2ba1c48606ef",
      "4a3acf47607c4949af93e9815b857b94",
      "dfdbd4c2fb664a949ed9a49f835a7048",
      "b453e7939d5c46a2b82314041ac5b7df",
      "3a1af80cda844908a9da1a0de918ad93",
      "62bcfa339b49409eafd88dc3268fefc6",
      "397857d060654489800fbb286b251161",
      "5122c633168d4d4b901103e020b626b5",
      "bb8f5a7483f54653bc5e8c87aa597eb1"
     ]
    },
    "id": "JjSFZkmX4Lqr",
    "outputId": "0aa2d1fe-6dfe-4c89-8641-3696570cdff6"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "09f5fa8ca136495db238b788124f2fae",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train:\n",
      "MSE: 270671131.15393\n",
      "MAE: 11879.92236\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8e084bb71f8f41549557da0879f9f48b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Val:\n",
      "MSE: 292257295.16526\n",
      "MAE: 12318.27666\n"
     ]
    }
   ],
   "source": [
    "@torch.no_grad()\n",
    "def evaluate_model(model, data, batch_size=256, name=None, **kwargs):\n",
    "    squared_error = abs_error = num_samples = 0.0\n",
    "    data_iterator = iterate_minibatches(data, batch_size, shuffle=False, **kwargs)\n",
    "    for batch, target in tqdm(data_iterator):\n",
    "        \n",
    "        title = torch.tensor(batch[\"Title\"], dtype=torch.long, device=device)\n",
    "        desc = torch.tensor(batch[\"FullDescription\"], dtype=torch.long, device=device)\n",
    "        category = torch.tensor(batch[\"Categorical\"], dtype=torch.float32, device=device)\n",
    "        \n",
    "        pred = model(title, desc, category)[:, 0].cpu().numpy()\n",
    "\n",
    "        squared_error += np.sum(np.square(np.exp(pred) - np.exp(target)))\n",
    "        abs_error += np.sum(np.abs(np.exp(pred) - np.exp(target)))\n",
    "        num_samples += len(target)\n",
    "\n",
    "    if name is not None:\n",
    "        print(f\"{name}:\")\n",
    "\n",
    "    print(f\"MSE: {squared_error / num_samples:.5f}\")\n",
    "    print(f\"MAE: {abs_error / num_samples:.5f}\")\n",
    "    \n",
    "    return squared_error / num_samples, abs_error / num_samples\n",
    "\n",
    "\n",
    "train_errors = evaluate_model(model, data_train, name=\"Train\")\n",
    "test_errors = evaluate_model(model, data_val, name=\"Val\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "V-iOq02ZpHfe"
   },
   "source": [
    "__Terrible start-up idea #1962:__ make a tool that automaticaly rephrases your job description (or CV) to meet salary expectations :)"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "week_02_cnn_for_texts.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "09f5fa8ca136495db238b788124f2fae": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_7d189f0386e54da39c49506ca4023958",
       "IPY_MODEL_e6b7ad856643400ea7f410a8276fcb81",
       "IPY_MODEL_68c14164f5774e5fa7c7c393b0c6f92e"
      ],
      "layout": "IPY_MODEL_332aca2eb6854b13b0a5bbdda4e19f67"
     }
    },
    "0f72a7f7e8584bd8acece5ca4a207c5b": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "332aca2eb6854b13b0a5bbdda4e19f67": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "394331689a594b3bb3a9a0dd1b8025c8": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "397857d060654489800fbb286b251161": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "3a1af80cda844908a9da1a0de918ad93": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "4a3acf47607c4949af93e9815b857b94": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_5122c633168d4d4b901103e020b626b5",
      "placeholder": "​",
      "style": "IPY_MODEL_bb8f5a7483f54653bc5e8c87aa597eb1",
      "value": " 192/? [00:12&lt;00:00, 15.45it/s]"
     }
    },
    "4b377fd7194547439aee1ab1330fc71a": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "5122c633168d4d4b901103e020b626b5": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "616fc3f2afb8432fb53d2ba1c48606ef": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_62bcfa339b49409eafd88dc3268fefc6",
      "max": 1,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_397857d060654489800fbb286b251161",
      "value": 1
     }
    },
    "62bcfa339b49409eafd88dc3268fefc6": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": "20px"
     }
    },
    "68c14164f5774e5fa7c7c393b0c6f92e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_4b377fd7194547439aee1ab1330fc71a",
      "placeholder": "​",
      "style": "IPY_MODEL_394331689a594b3bb3a9a0dd1b8025c8",
      "value": " 765/? [00:50&lt;00:00, 15.31it/s]"
     }
    },
    "7d189f0386e54da39c49506ca4023958": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_0f72a7f7e8584bd8acece5ca4a207c5b",
      "placeholder": "​",
      "style": "IPY_MODEL_b4a2794c7bd8437e9c8d52d7126fcbda",
      "value": ""
     }
    },
    "8e084bb71f8f41549557da0879f9f48b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_e82d3d9015f34147bc3020a82737380c",
       "IPY_MODEL_616fc3f2afb8432fb53d2ba1c48606ef",
       "IPY_MODEL_4a3acf47607c4949af93e9815b857b94"
      ],
      "layout": "IPY_MODEL_dfdbd4c2fb664a949ed9a49f835a7048"
     }
    },
    "af5f02089b6243ac8541a6f9ad1bab50": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": "20px"
     }
    },
    "b453e7939d5c46a2b82314041ac5b7df": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "b4a2794c7bd8437e9c8d52d7126fcbda": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "bb8f5a7483f54653bc5e8c87aa597eb1": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "dfdbd4c2fb664a949ed9a49f835a7048": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "e6b7ad856643400ea7f410a8276fcb81": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_af5f02089b6243ac8541a6f9ad1bab50",
      "max": 1,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_e8c59862f3aa47398c7be3fdcfa3b3c5",
      "value": 1
     }
    },
    "e82d3d9015f34147bc3020a82737380c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_b453e7939d5c46a2b82314041ac5b7df",
      "placeholder": "​",
      "style": "IPY_MODEL_3a1af80cda844908a9da1a0de918ad93",
      "value": ""
     }
    },
    "e8c59862f3aa47398c7be3fdcfa3b3c5": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}