02a_neural_networks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Music machine learning - Neural networks\n",
    "\n",
    "### Author: Philippe Esling (esling@ircam.fr)\n",
    "\n",
    "In this course we will cover\n",
    "1. A [quick introduction](#intro) on the code that we will adress and auxiliary functions\n",
    "2. A simple implementation for a [single neuron](#single)\n",
    "3. An implementation of [regression](#implem) using scikit-learn\n",
    "4. Some [common good practices](#practices) in machine learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introducing neural networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutorial, we will cover a more advanced classification algorithm through the use of *neural networks*. The tutorial starts by performing a simple **single neuron** discrimination of two random distributions. Then, we will study the typical **XOR problem** by using a more advanced 2-layer **perceptron**. Finally, we generalize the use of neural networks in order to perform classification on a given set of audio files."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are going to use relatively _low-level_ libraries to perform the first exercises (implementing your own neurons)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "import numpy as np\n",
    "%matplotlib inline\n",
    "from matplotlib import pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To simplify your work, we provide a first set of functions that provides simple plotting functionnalities (from the `helper_plot.py` file)\n",
    "\n",
    "  |**File**|*Explanation*|\n",
    "  |-------:|:---------|\n",
    "  |`plot_boundary`|Plots the decision boundary of a single neuron with 2-dimensional inputs|\n",
    "  |`plot_patterns`|Plots (bi-dimensionnal) input patterns|"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from helper_plot import plot_boundary\n",
    "from helper_plot import plot_patterns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember that a single neuron is only able to learn _linearly separable_ problems. To produce such classes of problems, we provide a script that draw a set of random 2-dimensional points, then choose a random line in this space that will act as the linear frontier between 2 classes (hence defining a linear 2-class problem). The variables that will be used by your code are the following.  \n",
    "\n",
    "```Matlab\n",
    "desired       % classes of the patterns \n",
    "inputs        % 2 x n final matrix of random input patterns\n",
    "weights       % 2 x 1 vector of neuron weights\n",
    "bias          % 1 x 1 vector of bias\n",
    "```\n",
    "\n",
    "You can execute the code below to see our simple classification problem. (Note that running the same cell multiple times produces a different starting dataset)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAoAAAAH8CAYAAABBxR4EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAABFu0lEQVR4nO3de3RU5b3/8c+Q2QG5dMdjmMhRBMQL1jqFUirjJZSKl7hET8tqaW09klQ5LqtHq/UXRls5YGu2Bz22Vk+t2lD1WIvWilXxgogE5WIESmgBQUtQVGYSLlMUNJPh+f2RZmpMAiGZyZ49+/1a61kJe2b2fGc/GeYzz748AUlGAAAA8I0+bhcAAACA3kUABAAA8BkCIAAAgM8QAAEAAHyGAAgAAOAzBEAAAACfIQAC8JVBgwapvr5eu3bt0q5du/Tuu++6XVLGXHrppdqyZcshPaZv37669dZb9be//U2NjY3aunWrVq1apcrKyow+Ty7Lt9cDdAUBEMgB1113XTqQ7Nq1S6lUKv37zp079a1vfcvtEg/Z6NGj9fOf/zzrzzNgwAD9+Mc/1l//+lft2rVL77//vh566CGNGjVKt956q8aMGdPm/nv27NHw4cN1+OGH6/DDD896fbnurrvu0qBBgzR69GgVFxdrxIgRuuiii/TQQw+5Xdoh6a2/NyCfGBqNllvt3Xffdb2GnrYJEyaYhx9+OKvPEQqFzOrVq81tt91mhg0bZiSZf/mXfzEVFRXmrbfeMm+++aY566yz8n5bt7bPfe5z5thjjz2kx7zzzjsmEAhk/Xmy3Xry95aLr4dGy3YLCgA86oEHHtBvf/tb3XXXXellO3fuVHV1tRYuXKg1a9a4WF3v+/vf/66///3vh/SYQCAgY0zWnyeX5dvrAbrK9RRKo9HatoONSl1xxRVmz5495uGHHzaRSMS88MILZufOnWbHjh3m8ccfN0OGDGlz/+9+97tm586dJplMmpEjR5qpU6ea1157zezYscNs377dLFiwwJx55pntnqeuri79PJ9e7jiO2bVrl9m3b5/5/ve/3+a2iy66yOzcudPs2bPHfPLJJ2bXrl1t2rp16zKyjUaPHm1Wr159wPvcfffd5ktf+lKPtrUkY9u2+cEPfmCefPJJs23bNrNjxw6zdu1aU1FRccDHff7znzcPPfSQeffdd82OHTvM+vXrza233mruvvtu8+Uvfzkjj+nTp4/ZuHFjevumUqmDvp5vfetbZufOnen7f7aPrrnmmow8jyTTv39/M3PmTLNhwwazc+dO09DQYP70pz+Z008/3bz44os93tbd/Xvr7uspKSkxd999t9m6davZtWuXeeutt0xVVZUZNGhQj9+nNFovN9cLoNFon2ldCSUTJkww27ZtM9u2bTMXXnih6dOnj+nXr5+55pprzKpVqzrcrbd48WLz1FNPmU2bNplzzjnH9OnTxwQCATNp0iSzfv16U15e3uHzdLZrbebMme0CYFcel4n24x//2PzgBz/olW195plnmqqqKjNhwgRz2GGHGUlmxIgRZsGCBeaKK67o8DHHHHOM2bZtm7nkkktM//79jdSyq3H69Olm79695txzz83IY7rzenpy/0N53KBBg8zrr79ubr75ZhMKhYwkEwwGzRlnnGEWLlxompqaMrKtM/H31pXXc/zxx5vNmzeba6+91hQVFRmp5TCEW265xdTV1Zni4uIO6zrU9ymN1kvN9QJoNNpnWlcD4AcffGBOOOGEdrc9+eSTZtKkSe2WL1682Lz33nvpD+NPt5KSEvPWW2+lj6X79PPkYgC87777uhSIMrGtO2uhUMisWrWqw9suvvhic9ddd3V42y233GK+8IUvZOQxPX092QyA9913n7nuuus6vK2wsPCQRoMPtK2l7AfAQCBgVq1a1eH7SpIpLy83Tz/9dId1Her7lEbrjcZZwICHvfTSS9q0aVO75XV1dTrmmGM6fExVVZXi8Xi75bFYTL/4xS80bdq0TJeZFZZlqbm52dUa4vG4QqFQh7ctW7ZM559/vs4++2wFg20Pt/7JT36iv/zlLxl5TK7q37+/zjnnnDbHZ35aU1OTTjnllC6v70DbujdMnDhRW7du1UsvvdTh7XPnztWRRx6p448/vt1t3XmfAtlGAATy0P79+xUIBDq8raamptPHLVmyRKNHj85SVZmVTCbVt2/fXnmufv36KRqNqra2Vjt27FBjY6M2bdqkp59+utPH1NfXa+rUqaqoqFBdXZ3WrVunp59+WjNmzNARRxyRscfkqpNOOklr1qw55JDenW3dG8aMGaMlS5Yc8D6vvPKKvvSlL3V5nQd6nwLZxlnAgM8c6AMnEAi0G3nKVfX19Tr55JO1YMGCrD5PYWGhXnnlFW3evFlXXnmlNmzYoA8//FCf+9znFA6H9eijj3b62FWrVuk73/mOpJYRy1GjRmny5MlauXKlJk+erA0bNmTkMbmoT58+2r9//yE9pifbOlcc6hnVgFsYAQR8prS09IC3rV27ts2yXP1AmzdvnqZNm3bQwNrTUcKpU6dq06ZNuuSSS1RbW6sPP/xQUsulQ1599dUuryeZTGrdunW69dZb9aMf/Ug33nhjVh6TKzZt2qQxY8Yc0ghXprZ1NqxZs0Zf/epXD3ifiRMnavXq1b1TENBDBEDAZ2644QYNHjy43fLBgwfr2muv1QMPPNBmeSwW05FHHtnu/p/73Oc0adKkTp9n586dWd1t+fbbb2vevHm69957Ow0Z999/v6688soePc/hhx/e6bF3Z555ZqePq6mp6XSmkf3796tfv34ZeUyuSiQSev311/Wf//mfnd7nlltu0TXXXJP+d3e3tZT9v7fFixfrmGOO6fRvvqKiQu+//77eeuutrNUAZBIBEPCZX/3qV3rllVc0adIkBQIBBQIBnXXWWXrllVc0a9Ys1dfXt7n/pk2bVFxcrClTpkhqCX6XXHKJ/vznP2vYsGGdPs/69et1zDHH6NJLL02P0h199NEqLy9XbW2tJk6c2OPXMnv2bFmWpRdeeEHjx4+X1LIb+ytf+YoWLVokST2eHmzBggW6/PLLFYlE0usfPXq0fv3rX+uRRx7p9HFHH320HnnkEY0bN059+rT8Vztw4EBdfPHFuueee3TnnXdm5DG57KqrrtJ//Md/yHEc/eu//quklt28Z599tl5++WWNGzdOc+fOTd+/u9tayv7fmzFG3/72t3Xvvffq2muvlW3bkqRQKKRbbrlF1157rSoqKnr0HEBvc/1UZBrN7+36669vc/HaT1+cd+fOnWbq1Klt7v/II4+kL3z73nvvmaOPPrrNbfv27TMfffSRmTNnTpvHLV682IwcOdKcdtpp5vnnnzcNDQ1mx44dZuHChWbChAmd1jdq1CizdOlSs2vXLrN9+3bz2GOPmdNPP93MnDmzw+dpbcOHDzePPfaYef/9983OnTvNxo0bzb333mtOPfXUjG6/qVOnmkWLFpkdO3aYxsZGs2jRInPRRRe1u1+fPn3Mpk2bOt3Wu3btMkuXLm33uDPPPNPU1NSYhoYGs2vXLrNy5Upzww03mEGDBqUf/+Mf/7jNY1588UVz+umnm/nz55vt27ebRCJhGhoazFNPPWXGjRvX4evozmNa+6Wz17N582YTDAbbPGbOnDmd3r+hocGMHz8+I88jtVzY2XEc89Zbb6XX/9xzz5lvfOMbHb6e7mzr7vy9dff1tF4I+p133jG7du0yf/vb34zjOB1eCLq771MarTda4B+/APCBxYsX67LLLtPbb7/tdikAABexCxgAAMBnCIAAAAA+QwAEfOB73/uedu7cqTPOOEOrV6/Wrl27NH/+fLfLAgC4hGMAAQAAfIYRQAAAAJ/xxpxPOSIej2vr1q1ul9Ftxx9/vDZv3ux2Gegi+ss76Ctvob+8hf7qvmHDhikUCnV6u+vXovFKq62tdb0GP9fvt0Z/eafRV95q9Je3Gv2VnW3HLmAAAACfIQACAAD4DAEQAADAZwiAAAAAPkMABAAA8BkCIAAAgM8QAAEAAHyGAAgAAOAzBEAAAACfIQACAAD4DAEQAADAZwiAAAAAPkMABAAA8BkCIAAAgM8QAAEAAHyGAAgAAOAzBEAAAACfIQACAAD4DAEQAADAZwiAAAAAPkMABAAA8BkCIAAAgM8QAAEAAHyGAAgAAOAzBEAAAACfIQACAAD4DAEQAADAZwiAAAAAPkMABAAA8BkCIAAAgM8QAAEAAHyGAAgAAOAzBEAAAACfIQACAAD4DAEQAADAZwiAAAAAPkMABAAA8BkCIAAAgM8QAAEAAHyGAAgAAOAzBEAAAACfIQACAAD4DAEQAADAZwiAAAAAPkMABAAA8BkCIAAAgM8QAAEAAHyGAAgAAOAzBEAAAACfIQACAAD4DAEQAADAZwiAAAAAPkMABAAA8BlPBEDLsvSFL3zB7TIAAADyQtDtAg5k8uTJmjp1qkaMGKG9e/fq7LPP7vE6Bw4cqHvuuUcjR45UIBDQH//4R91xxx0ZqBYAAMAbcnoE8KijjtJPf/pTnX766Rlb59y5c/Xkk0/qjDPO0Omnn64jjzxSV111VcbWDwB+ZlmW2yUA6IKcDoD33nuvNm7cmLH1jRo1Sv3799f8+fPTy2666SZdeumlCgQCGXseAPAT27blOI5isZiampoUi8XkOI5s23a7NACdyOkAmGmlpaV64okn2ixramrS+vXrdcIJJ7hUFQB4l23bqqmpUWVlpUKhkFImpVAopMrKStXU1BACgRzlqwA4ePBgbd26td3yLVu2aPDgwS5UBADeFo1GFQ6HtbG5XpHGcgW3n6pIY7k2NtcrHA5rxowZbpcIoAM5fRJIpgWDQX388cftlieTSRUUFHT4mMsvv1zTp0+XJJ188smqra3Nao3ZdNJJJ3m6fr+hv7zDz331xS9+UZJkjNEvP3dDerkxLT+vv/56TZo0yY3SOuXn/vIi+is7fBUAU6lUhwco9+vXT83NzR0+5v7779f9998vSaqtrdW4ceOyWmM2eb1+v6G/vMOvfWVZlpqampQyKX2+8Zvtbm8+cqUsy9Jpp52mZDLpQoUd82t/eRX91X0HCs6+2gUcj8c1fPjwdsuHDRumeDze+wUBgIclk0nF43EVBAo03jqlzW0RK6yCQIHi8XhOhT8ALfIqANq2rSFDhnR6+8KFC3XRRRe1Wda3b1+NGTNGmzdvznZ5AJB35s6d2/KzaKYiVlhSS/irLrpZklRdXe1abQA6lzcBcOjQodq2bZu2bdumKVOmdHifLVu2qLm5WZMnT04v+9nPfqZf//rXvVUmAOSVqqoq1dXVaVRwuJYVV6v5yJVaVlytUcHhqqurk+M4bpcIoAM5HQDnzZunxYsXa/HixRo7dmz6947OKtu3b5+2b9+uDz/8UDt37ux0nRUVFbrwwgu1dOlSLV++XDt37tTdd9+dzZcBAHkrkUiotLRUjuOkdwfH43E5jqPS0lIlEgm3SwTQgZw+CWTq1Kldvm9jY6OOP/74g95vz549uvzyy3tSFgDgUxKJhKLRqKLRqCzL4pg/wANyegQQAOAthD/AGwiAAAAAPkMABAAA8BkCIAAAgM8QAAEA6EBHM0cB+YIACHgMH0pA9ti2LcdxFIvF1NTUpFgsJsdxZNu226UBGUUABDyADyUg+2zbVk1NjSorKxUKhZQyKYVCIVVWVqqmpob3G/IKARDIcXwoAb0jGo0qHA5rY3O9Io3lCm4/VZHGcm1srlc4HO5wEgLAqwiAQI7jQwnoHeXl5S0/d8/SiuQ6SdKK5DpV7J4tqWUmKSBfEACBHMeHEpB9lmWlR9hb32etlifr0iPvHIOLfEEABHIYH0pA70gmk+m5jMdbp7S5LWKF03McM9MJ8gUBEMhhfCgBvWfu3LktP4tmKmKFJbW8z6qLbpYkVVdXu1YbkGkEQCDH8aEE9I6qqirV1dVpVHC4lhVXq/nIlVpWXK1RweGqq6uT4zhulwhkDAEQyHF8KAG9I5FIqLS0VI7jpEfe4/G4HMdRaWmpEomE2yUCGUMABHIcH0pA70kkEopGoyopKVFhYaFKSkoUjUZ5nyHvBN0uAMDBtX4oRaNRWZbFMX9AL+B9hnzGCCDgMXwoAQB6igAIAADgMwRAAAAAnyEAAgAA+AwBEMAhY+YRAPA2AiCALrFtW47jKBaLqampSbFYTI7jyLZtt0sDABwiLgMD4KBs21ZNTY3C4ZaZSFrnIK6srFRZWRnXIwQAj2EEEMBBRaNRhcNhbWyuV6SxXMHtpyrSWK6NzfUKh8OaMWOG2yUCAA4BARDAQZWXl7f83D1LK5LrJEkrkutUsXu2JKmiosK12vyO4zEBdAcBEMABWZalUCiklEmlw1+r5cm69O5ggkjv4XhMAD1FAARwQMlkMj0H8XjrlDa3Raxwem5iZijpHa3HY1ZWVqaDeevxmDU1NYRAAF1CAARwUHPnzm35WTRTEavlRJCIFVZ10c2SpOrqatdq8xuOxwSQCQRAAAdVVVWluro6jQoO17LiajUfuVLLiqs1KjhcdXV1chzH7RJ9g+MxAWQCARDAQSUSCZWWlspxnPTu4Hg8LsdxuARML+J4TACZwnUAAXRJIpFQNBpVNBqVZVkc8+eC1uMxQ6GQxluntAmBHI8J4FAwAgjgkBEw3OOF4zEZgQRyHwEQADwkV4/H5NI0gLcQAAHAQ3LxeEwuTQN4DwEQADym9XjMkpISFRYWqqSkRNFo1LWTcbg0DeA9BEAA8LBcOB6TS9MA3kMABAB0G5emAbyJAAgA6DamCgS8iQAIAOgRL1yaBkBbXAgaANAjVVVVKisrUzgc1rLiaqVMSgWBAkliqkAgRzECCADokVy8NA2AA2MEEAA8Ipen4GOqQMBbGAEEgBzmxRk2CH9A7mMEEAByVOsMG+Fwy4kVn55ho6ysjN2rALqNEUAAyFHMsAEgWwiAAJCjmGEDQLYQAAEgBzHDBoBsIgACQA5ihg0A2UQABIAcxQwbALKFs4ABIEcxwwaAbGEEEAByFDNsAMgWRgABIIcxwwaAbGAEEAA8gvAHIFMIgAAAoNu4FJE3EQABAMAh8eIc1WiLYwABAECXtZujOmWYo9qDGAEEAABdlp6jur5ZkfJGBU/drkh5ozbWNzNHtYcQAAEAQJel56ietVsr1rWcmLRiXVIVs3dLYo5qryAAAgCALknPUZ0y6fDXanldMr07mBNDch8BEAAAdEl6juqCgMaf0jbkRcKWCgoCzFHtEQRAAADQZek5qmcWKRJuCYGRsKXqm4skMUe1V3AWMAAA6LI2c1RXFyuVMiooCEhijmovYQQQAAB0Wbs5qv+x25c5qr2FEUAAAHBImKPa+xgBBAAA3Ub48yYCIJAlXAahe9huAJB9BEAgg5gfs3vYbgDQuzgGEMiQdvNjmhTzY3YB2w0Ael9OjwBecMEFeu211/TKK6/omWee0XHHHdfjda5fv16LFy9u02688cYMVAu/S8+P2VyvSGO5gttPVaSxXBub65kf8wDYbgDgDpOLbcyYMebll182AwYMMJLMyJEjTW1trenXr1+P1rtw4cJuP7a2ttb17dKT5vX6c73FYjFjjDHjG6YZvT823SIN5cYYY2KxGP3VC9vNjeaXvsqXRn95q9Ff2dl2OTsCWFlZqcrKSn300UeSpLffflu/+93vdPHFF7tcGdBeen5Mk9KK5Lo2ty1P1qV3a3KCQ1tst97FdgTQKmcD4IgRI1RbW9tm2fz583X22Wf3aL2HH364HnnkES1evFgrVqzQU089paFDh/ZonUB6fsxAgcZbp7S5LWKFVRAoYH7MDrDdso8TbAB0JGcD4N69e9stq6+v1xFHHNGj9d577736yU9+ookTJ2r8+PG65ZZb9OCDDyoQCPRovUB6fsyimYpYLSc0RKywqotulsT8mJ1hu2VP6wk2lZWV6ZHW1hNsampqCIGAjwXUsi845yxcuLDD0b7OlvfETTfdpFWrVun5559vd9vll1+u6dOnS5JOPvlk/fWvf83oc/emk046SRs2bHC7jLxVUFCgE088Uf3795ckGWPSXyz27t2rN998U6lUqsvr80t/ZXq7uSFX++qoo47SkCFDtM98ovrm9/WR2acBgcM0PPivOizQVx988IHee+89t8vsdbnaX+gY/dUz48aN6/Q21w9S7KgtWrSo3bJgMGhefPHFjD/Xueeea6677roeHUzpheb1+r3QbNs2VVVV6RMbYrGYqaqqMrZt01+9tN3caLnaV/lwgo2f+otGf/XmtsvZ6wAOGDCg3bJjjjlGDQ0NGX+uww8/XM3NzRlfL/yH+TG7h+2WeYdygg3bG/CfnD0GcOPGjRozZkybZVOmTNHSpUs7fYxt2xoyZEintx911FEdLi8rK9Orr77avUKBTvCh2j1st8zgBBsAB5KzAfDnP/+5br/99vRxQSNHjtQ3v/lNPfzwwx3ef+jQodq2bZu2bdumKVOmdHifyy67THfddZcGDRqUXlZRUaG+fftq9erVmX8RAOAiTrAB0Jmc3QX85z//WXPmzNGLL76oVCqljz/+WJdcckn6uoCftW/fPm3fvl2hUEg7d+7s8D6zZs3SBRdcoD/84Q+yLEtFRUV6+eWX9e///u/ZfCkA4IqqqiqVlZUpHA5rWXG1UialgkCBJKmurk6O47hcIQA3uX6Qolea1w9E9Xr9fmv0l3daLveV10+w8Vt/0eiv3tp2OTsCCADoOU6wAdCRnD0GEACQWYQ/AK0IgAAAAD5DAAQAAPAZAiAAAIDPEAABAAB8hgAIAABykmVZbpeQtwiAAAAgZ9i2LcdxFIvF1NTUpC9+8YtyHEe2bbtdWl7hOoAAACAn2LatmpoahcMtUxemUkaWZamyslJlZWUqLS1VIpFwucr8wAggAADICdFoVOFwWBvrmxUpb1Tw1O3asCWpjfXNCofDmjFjhtsl5g0CIAAAyAnl5eUtP2ft1op1LRcu/2ifUcXs3ZKkiooKt0rLOwRAAADgOsuyFAqFlEqZdPhrtbwuqVTKKBQKcWJIhhAAAQCA65LJpOLxuAoKAhp/StuQFwlbKigIKB6PM6VhhhAAAfgCowZA7ps7d27Lz5lFioRb3rMD+gdUfXORJKm6utqt0vIOARBA3vrs5SRisRiXkwByWFVVlerq6jRqeFDLqovVvPJInTTc0qjhQdXV1clxHLdLzBsEQAB5qfVyEpWVlS3HFZmUQqGQKisrVVNTQwgEclAikVBpaakcx0nvDk4mk3Ich0vAZBgBEEBeSl9OorlekcZyBbefqkhjuTY213M5CSCHJRIJRaNRlZSUqLCwUGvXrlU0GiX8ZRgBEEBeSl9OYvcsrUiukyStSK5Txe7ZkricBOAFnPCRPQRAAHknfTkJk0qHv1bLk3Xp3cGcGALArwiAAPJO+nISgQKNt05pc1vECqsgUMDlJAD4GgEQQF5KX06iaKYiVsu8ohErrOqimyVxOQkA/hZ0uwAAyIaqqiqVlZUpHA5rWXG1UialgkCBJHE5CQC+xwgggLzU7nIS/9jty+UkAIARQAB5rPVyEtFoVJZlccwfAPwDI4AAfIHwBwD/RAAEAADwGQIgAACAzxAAAQAAfIYACAAA4DMEQAAAAJ8hACKvMLcrAAAHRwCE59m2LcdxFIvF1NTUpFgsJsdxZNu226UBAJCTuBA0PM22bdXU1CgcbpnrNWVSCoVCqqysVFlZGTM+AL2Ii20D3sEIIDwtGo0qHA5rY3O9Io3lCm4/VZHGcm1srlc4HNaMGTPcLhHIa4zAA95EAISnlZeXt/zcPUsrkuskSSuS61Sxe7YkqaKiwrXagHzXOgJfWVmpUCjUZgS+pqaGEAjkMAIgPMuyrPSHTmv4a7U8WZf+MOLEECA7GIEHvIsACM9KJpOKx+MqCBRovHVKm9siVlgFgQLF43GOSQKyhBF4wLsIgPC0uXPntvwsmqmI1XIiSMQKq7roZklSdXW1a7UB+YwReMDbOAsYnlZVVaWysjKFw2EtK65WyqRUECiQJNXV1clxHJcrBPJT6wh8KBTSeOuUNiGQEXgg9zECCE9LJBIqLS2V4zjp3cHxeFyO43AJGCDLGIEHvM3QutZqa2tdr8HP9XelWZbleg30l/+aX/vKtm2zdu1a06p5f3P697Vr1xrbtl2vkf7yfqO/srPtGAFEXmF3E9B7GIEHvItjAAEA3ZZIJBSNRhWNRpkJBPAQRgABABlB+AO8gwAIAADgMwRAAAAAnyEAAgAA+AwBEAAAwGcIgAAAAD5DAAQAAPAZAiAAAIDPEAABAAB8hgCIvGdZltslAACQUwiAyEu2bctxHMViMTU1NSkWi8lxHNm27XZpAAC4jrmAkXds21ZNTY3C4bAkKWVSCoVCqqysVFlZGZPUA93APL9AfmEEEHknGo0qHA5rY3O9Io3lCm4/VZHGcm1srlc4HNaMGTPcLhHwBD+NpHOoCPyGAIi8U15e3vJz9yytSK6TJK1IrlPF7tmSpIqKCtdqA7yidSS9srJSoVCozUh6TU1NXoRAPwVc4LMIgMgrlmWlP6xaw1+r5cm69IcY3/aBA8v3kfR2ATdl8i7gAgdCAEReSSaTisfjKggUaLx1SpvbIlZYBYECxeNxjmUCDiLfR9LTAbe+WZHyRgVP3a5IeaM21jfnRcAFDoYAiLwzd+7clp9FMxWxWk4EiVhhVRfdLEmqrq52rTbAC/wwkp4OuLN2a8W6li+EK9YlVTF7tyTvB1zgYAiAyDtVVVWqq6vTqOBwLSuuVvORK7WsuFqjgsNVV1cnx3HcLhHIafk+kh4IBNK7fVvDX6vldcn07mAvB1zgYAiAyDuJREKlpaVyHCf9IRaPx+U4DpeAAboon0fSjTEt/zcUBDT+lLYhLxK2VFAQ8HTABbrKHKgFAgFz2mmnma997WumoKCg3e0jR4403/nOdw64jnxptbW1rtfg5/q72yzLcr0G+iu/Wz72lW3bZu3ataZV8/7m9O9r1641tm27XmNP+stxHGOMMRu2JE2kvMFo7PsmUt5gNmxJGmOMqaqqcr1O2j/7y+0avNoOtO0OOAJYVFSk5557Tuecc46+/OUv6/nnn9ewYcPa3KdPnz76r//6rwOtBnAV3+KBQ5fvI+npQ0WGB7WsuljNK4/UsupijRoe5FAR+Ean6fBXv/qVOf3009P/Puqoo8xLL71kjj322Db3W7Zsmesptzea17+FeL1+vzX6yzvND33l1ZH0A/WXbdumqqrKxGIxY4wxsVjMVFVVeXp0Mx+bH95fbmy7A44AHnfccXrttdfS/37vvfd0ySWX6IEHHtDRRx+dXm6MOdBqAAAel48j6YlEQtFoVCUlJSosLFRJSYmi0ajnRzeBrjhgAOzoDKgPPvhA06dP16OPPqri4uKsFQYAQG/Jx4ALHMgBA+CWLVs0evTodsvfeustXXPNNfrDH/6gAQMGKBAIZKW4Cy64QK+99ppeeeUVPfPMMzruuON6vM6BAwfqwQcf1KuvvqrXXntN119/fQYqBYDexSVKAPTEAQNgNBrVL37xC337299ud9vq1as1e/ZszZs3T8FgMOOFjRkzRtddd53OOeccffWrX9U111yjRx99VP369evReufOnasnn3xSZ5xxhk4//XQdeeSRuuqqqzJUNQBkD3PXIpv4UuE/BzyAsF+/fua8887r9PZvfvObZtGiRRk/cPH3v/+9GTduXJtlP/zhD01FRUW31zlq1Cjz7LPPtllWWFhoamtrTSAQ6NHBlF5oXq/fb43+8k7rjb7K58uy5GN/eaXZtm0cx2lzIozjODn190R/ZW3buV9gR23lypXtlo0YMcI8+uij3V7n9OnTOwyQDz74oDnxxBN7uiFzvnm9fr81+ss7rTf6Kn3duuQWM75hmtH7Y834hmlmQ3IL163Lwf7yQmv3paJ5f05+qaC/srPtujwTyOTJkzu97f/+7/+6upou27t3b7tl9fX1OuKII7q9zsGDB2vr1q3tlm/ZskWDBw/u9noBINvSc9funpWen3dFcp0qds+WxNy1OHTRaFThcFgb65sVKW9U8NTtipQ3amN9s8LhsGbMmOF2iciiLgfAhx56SF/5ylfaLb/99tt14YUXZrQoSWpubm63zBjToxNOgsGgPv7443bLk8mkCgoKur1eAMgmy7Ja5q41qXT4a7U8WaeUSTF3LQ5Z+kvFrN3pOZFXrEuqYvZuSXypyHddPnvj7bff1qxZszR79mwtX75cknTnnXdq27Ztqq2tzXhhffq0z6bBYLBH1xxMpVId/gfZr1+/DgOnJF1++eWaPn26JOnkk0/OymvtLSeddJKn6/cb+ss7eqOvksmkLMvS+uLH9ZHZl14+INBfBYECJZNJLVu2LKs15AveW1IgEFAoFJIxRr+84XPtbjfGKBQK6Y033nD9Wr/0V/Z0aT/yhAkTzIABA8yCBQvMGWecYX75y1+aa665xkgykUgk4/utV6xY0W7Zscceax555JFur3P69Olm2rRp7ZY/9NBD5vjjj+/RvnQvNK/X77dGf3mn9fYxgJGGcqP3x5pIQznHAOZof3mhtZ74MX5ay1zIrS1S3pA+IcTtGumvrG67Q1vZgAEDzHvvvWfuvvvurBb929/+1owZM6bNshtuuMFcccUVnT7Gtm0zZMiQTm8fMWKEefLJJ9ss69u3r1m3bl0mNmTON6/X77dGf3mncRawtxrvrZaW/lKxJWki5Q3p8LdhSzKnvlTQX1nbdl1byfDhw40k8+tf/9pcffXV5k9/+pM5++yzjdQyR3Cmix49erRZtGiR6d+/v5FkRo4caV5//XUzYMCADu8/dOhQs2fPHpNKpcyUKVM6Xe/jjz9uJk+enP737bffbq666qpMbMicb16v32+N/vJO662+Yu5ab/VXrjfOAs7/lpEAuGjRIlNdXW3Ky8uN1HJ9wPnz55vzzz/fvPDCC1kp/LzzzjOvvvqqWbJkiXnhhRcOeKmW4uJis3nzZpNIJMzEiRM7vd+gQYPM/fffb5YuXWqWL19ubrzxxkxtyJxvXq/fb43+8k5zo68sy3L9dXu18d76Z/PClwr6K2vbrmsraWpqMt/97nfbLCssLDRPPvmk+eCDD1x/kTmwIXO+eb1+vzX6yzuNvvJWo786brn6pYL+ys626/JlYKLRqB555JE2y5qamjR16lS99957XV0NAADIQclk0u0S0Iu6HADvuOOODpc3NTXptNNOy1hBAAAAyK4uB8ADaWpqysRqAAAA0AsyEgABAADgHQRAAAAAnyEAAgAA+AwBEAAAwGcIgMABWJbldgkAAGQcARD4DNu25TiOYrGYmpqaFIvF5DiObNt2uzQAADIi6HYBQC6xbVs1NTUKh8OSpJRJKRQKqbKyUmVlZSotLVUikXC5SgAAeoYRQOBTotGowuGwNjbXK9JYruD2UxVpLNfG5nqFw2HNmDHD7RIBAOgxAiDwKeXl5S0/d8/SiuQ6SdKK5DpV7J4tSaqoqHCtNgAAMoUACPyDZVkKhUJKmVQ6/LVanqxL7w7mxBAAgNcRAIF/SCaTisfjKggUaLx1SpvbIlZYBYECxeNxJkwHAHgeARD4lLlz57b8LJqpiNVyIkjECqu66GZJUnV1tWu1AQCQKZwFDHxKVVWVysrKFA6Htay4WimTUkGgQJJUV1cnx3FcrhAAgJ5jBBD4lEQiodLSUjmOk94dHI/H5TgOl4ABAOQNRgCBz0gkEopGo4pGo7Isi2P+0CP8DQHIRYwAAgfABze6g9lkAOQ6RgABIIOYTQaAFzACCF/iWn7IFmaTAeAFBED4Brvl0BuYTQaAF7ALGL7Abjn0hkOZTYbjSwG4iRFA+AK75dAbmE0GgFcQAOEL7JZDb2E2GQBewC5g5D12y6E3MZsMAC9gBBB5j91y6E3MJgPACwiA8AV2y6E3tc4mU1JSosLCQpWUlCgajRL+AOQMdgHDF9gtB7cwsgwgFzECCF9gtxwAAP/ECCB8o3W3XDQa5YQPAICvMQIIXyL8AQD8jAAIAADgMwRAAAAAnyEAAgCAnGNZltsl5DUCIAAAyAm2bctxHMViMTU1NSkWi+moo46Sbdtul5Z3OAsYAAC4zrZt1dTUKBxuuVh/KmUUCoUkSTU1NVyyK8MYAQQAAK6LRqMKh8PaWN+sSHmjgqduV6S8Ufs+MQqHw5oxY4bbJeYVAiAAAHBdeXl5y89Zu7ViXcululasS6r+g2ZJUkVFhWu15SMCIAAAcJVlWQqFQkqlTDr8tfpor0nvDubEkMwhAAIAAFclk8mWaToLAhp/StuQN6B/QAUFAcXjcS7in0EEQAAA4Lq5c+e2/JxZpEi4JQRGwpaGD2k5X7W6utq12vIRZwEDAADXVVVVqaysTOFwWMuqi5VKGRUUBCRJdXV1chzH5QrzCyOAAADAdYlEQqWlpXIcJ707OB6P64MPPuASMFlAAAQAADkhkUgoGo2qpKREhYWFKikp0XvvvUf4ywICIAAAyDmc8JFdBEAA8CkuqQH4FwEQAHyko7lWHcdhrlXAZzgLGAB8orO5VisrK1VWVsaB9oCPMAIIAD7R2VyrG+ubmWsV8BkCIAD4RGdzrVbM3i2JuVYBPyEAAoAPHGiu1eV1SeZaBXyGAAjP4QMKOHQHmms1EraYaxXwGQIgPIEzF4Ge62yu1eqbiyQx12om8UUVXmBoXWu1tbWu1+DH+m3bNmvXrjWtmvc3p39fu3atsW3b9RrpL383r/RVu/dS835fvJd6q79s2zaO45hYLGaMMSYWixnHcXyzXb3WX35oB9p2jAAi56XPXGyuV6SxXMHtpyrSWK6NzfWcuQgcgs7mWnUch0vA9FDrJXYqKyvTx1q2XmKnpqaGvRXISa4nVK80r38L8Wr9rd+mxzdMM3p/bLpFGsrT37LdrpH+8nfzal9ZluV6DfnSX47jGGOM2bAlacZPazAa+74ZP63BbNiSNMYYU1VV5frr9mrz6vsrFxojgPCs9JmLJqUVyXVtbluerFPKpDhzEegmTvjIHC6xA68hACKnpc9cDBRovHVKm9siVlgFgQLOXATgKi6xAy8iACLnpc9cLJqpiNUyhVXECqu66GZJnLkIwF1cYgdeRABEzquqqlJdXZ1GBYdrWXG1mo9cqWXF1RoVHK66ujo5juN2iQB8jkvswGsIgMh57c5c/MduX85cBJAr0l9Uhwe1rLpYzSuP1LLqYo0aHuSLKnISARCekEgkFI1GVVJSosLCQpWUlCgajRL+AOQELrEDrwm6XQBwqDiOBkAuav2iGo1GZVkW/1chpzECCABAhhH+kOsIgAAAAD5DAAQAAPAZAiAAAIDP5OxJIDfeeKMmT56sZDKpDRs26Prrr9eHH37Y7fUNGzZMy5cv15tvvtlm+Zw5c7RgwYKelguf6uxAbw4ABwDkspwcAbzqqqtUXFysSCSi0tJSPfPMM+mLbPbEs88+q4kTJ7ZphL/eky/TINm2LcdxFIvF1NTUpFgsJsdxdMwxx3S43LZtt0sGAKAdk2vtL3/5i+nbt2+bZfPnzzcjR47s9jqHDRtm7r///h7VVVtb6/q28Vr9tm0bx3FMLBYzxhgTi8WM4zjGtm3Xt0d3X8/atWtNq+b9zenf9+7d2+HytWvXduv1ev3vLZvNsizXa6CvvNvoL281+is72y7nRgBPOOEErV69Wp988kmb5U899ZQmTZrUo3WPHTtWTz/9tF5++WWtWrVKd999twoLC3u0TnTOtm3V1NSosrKyZaJ0k1IoFFJlZaVqampycmTsYKOU0WhU4XBYG5vrFWksV3D7qYo0lmtjc70OO+ww7difaLc8HA5rxowZvfQK8ldnI6+5+HcEALku5wLg4MGDtXXr1nbLt2zZosGDB3d7vQ0NDXrggQf0ve99T1/72tc0duxYrVmzRrfffntPysUBHCgs5VIoOpRgUV5e3vJz9yytSK6TJK1IrlPF7tmSpIACHS6vqKjojZeSt7z4ZQIAcllALUOBWTdr1iyVlZUpEAi0u80Yo9tuu01PPPGEJkyYoDPOOEM/+9nP2tzn9NNP16RJkzRr1qyM1vXcc8/pm9/8ZqcnmFx++eWaPn26JOnkk0/WX//614w+f2866aSTtGHDhl57vi9+8YuyLEsbklv0kdmXXj4g0F8nWcOVTCa1du3aXqunIwUFBTrxxBPVv39/SS1/i61/o3v37tWbb76pVColSQoEAho7dqyMMVqVbL8dx1onKRAIaFXTBplPva3Sy1etkjFdf7t11F+BQOCQ1pEvjjrqKA0ZMkT7zCeqb35fH5l9GhA4TMOD/6rDAn31wQcf6L333nOtvt5+b6Fn6C9vob96Zty4cZ3e5vo+6k+3M844w/zXf/1Xu+VnnXWWuemmmzL+fFVVVeZLX/pSj/ele6H1Zv2WZaWPhdP7Y9u11mPk3D6Wy3EcY4wxG5JbzPiGaUbvjzXjG6aZDcktxhhjqqqq2ty/9VjG1vu2tkhDuTHGmB2pRIfLY7FYt/sr346j7E472HbvzvbNZPP6/w1+a/SXtxr9lZ1tl3O7gOPxuIYPH95u+bBhwxSPxzP+fIcffriam5szvl6/SyaTLROiBwo03jqlzW0RK6yCQIHi8bjrl0o52C7dz+66bT0bfW7RTEWssKSW11NddLMkych0uLy6urpb9bHrs+W4zNbX3tpHrZYn69LbJF/OMgeA3uJ6Qv1sW7lypSksLGyz7JlnnjGjRo3q9DFDhgw54IjIUUcd1W5ZMBg0K1euNMFgsMdJ2gutt+v/9OhapKE8PWLT2ehab7fujFL29lnAhzpCma+NEUAa/eXfRn9lbdu5X+Bn29VXX23mzJmT/veFF15oHnvssU7vP2XKFJNKpcyePXvM0KFDO7zPggULzPTp002fPn2M1PLhf99995krr7wyUxsy51tv13+gsNTdUJTp1p1gYdu2qaqqarNLtqqqyhxzzDEdLu/u66ytrc354NNbLde/THj9/wa/NfrLW43+ytq2c7/AjtoNN9xgli1bZpYuXWoefPBBM3DgwE7vO2HCBLN7927z9ttvm+Li4g7v06dPH3PdddeZV155xSxZssTU1taayy67LJMbMuebW9cBzGQoynTrabDo7BjGTBzb+MYbbxzyCGW+tlz/MuH1/xv81ugvbzX6K2vbzv0CvdK8/kfodv25GFRyOVgwAti+r3L1y4Tb7y0a/ZXPjf7KzrbLuZNAkL/cPuGjI4lEQqWlpXIcJ33SSjwel+M4Ki0tVSKRcLW+g5100t2TS7wokUgoGo2qpKREhYWFKikpUTQadb2PAMCrXE+oXmle/xbi9fp7o+XSKGVtbW1Oj1DS2vaV2zXQ6K98bfRXdrYdI4DAp+TaKGWuj1ACALwp6HYBAA6sdddnNBqVZVk5F1IBAN7DCCDgIYQ/AEAmEAABAAB8hgAIAADgMwRAAAAAnyEAAgAA+AwBEAAAwGcIgAAAAD5DAAQAAPAZAiAAAIDPEAABAAB8hgAIAADgMwRAAAAAnyEAAgAA+AwBEAAAwGcIgAAAAD5DAAQAAPAZAiAAAIDPEAABAAB8hgAIAADgMwRAAAAAnyEAAgAA+AwBEAAAwGcIgAAAAD5DAITnWZbldgmHxGv1AgDyDwEQnmTbthzHUSwWU1NTk2KxmBzHkW3bbpfWIa/VCwDIb0G3CwAOlW3bqqmpUTgcliSlTEqhUEiVlZUqKytTaWmpEomEy1X+k9fqBTLNsiwlk0m3ywDwKYwAwnOi0ajC4bA2Ntcr0liu4PZTFWks18bmeoXDYc2YMcPtEtvwWr1AJjDqDeQ+Q+taq62tdb0GP9ff2mKxmDHGmPEN04zeH5tukYZyY4wxsVjM9RozUe/B+suyLNdfG61rfeW3Ztu2Wbt2rWnV3Lw//fvatWuNbdv0F43+cnnbMQIIT2g9ccKyLIVCIaVMSiuS69rcZ3myLr17NVdOtMh0vYyqwAvSo971zYqUNyp46nZFyhu1sb6ZUW8gRxAAkbM6Cju33HKLGhoaVBAo0HjrlDb3j1hhFQQKFI/Hc+Z4o2QyqXg8npF6W48lrKysTIfK1mMJa2pqCIHIGeXl5S0/Z+3WinUtf9sr1iVVMXu3JKmiosKt0gD8AwEQOelAYccYI0maWzRTEavlxIqIFVZ10c2SpOrqatfq7sjcuXNbfvawXo4lhBekR71TJh3+Wi2vSyqVMjk1Sg/4mev7qL3SvH4cgpfqdxzHGGPMhuSW9LFz4xummQ3JLenj5tLHF+1vzqnjiz7b2h0P1cV6P9tfXjv20U/NS++t3mjpv9VpDUZj30+3SHlDTvyt0l/eavRXdrYdI4DISeldSLtnpY+dW5Fcp4rdsyVJgUBAjuOkd6/G43E5jpOTl1RJJBIqLS3tUb1eO/Yxl7GNsi896j2zSJFwy/aOhC1V31wkKfdG6QG/cj2heqV5/VuIV+q3LCs9Uvbpka7W1jqC1noWrNfOhu1qvYwAZq7Ztm3++7//O70NY7GYcRwnY6PFXnlv9eb25ixgGv3lfmMEEJ5yqCdO5MoJH13V3XozdSyhn9i2rTvvvFONjY264YYbFAqFtH+/4eSZLGs36l0QyOlResCvXE+oXmle/xbipfo/fQxgpKE8PdLVegxgVVWV6zX2dn9191hCv7YDjUJtfidp3tyazNjfkpfeW260XBulp7+81eivrG079wv0SvP6H6GX6ifsdNxftm2bqqqqNrsyq6qqfLE9DrWlv0RsSaZPRhg/rcFs2NIS/B569qOM7Tr30nuLRn95rdFfWdt27hfoleb1P0Kv1e/3sMNMID1rBz0TdUdzelSwp9vSa+8tvzf6y1uN/srOtgsKyFGJRELRaFTRaJTJ5DvA9uhcl65F9y8FkpRTFw4HgN7CSSDwBD6gcSjSJxIVBDT+lLaXfYmELRUUBNScMpI4eQaAPxEAAeSlg12LLlgQUF1dnRzHcatEAHANARBAXqqqqlJdXZ1GDQ9qWXWxmlceqWXVxRo1PKjm5mbdeeedXJIEgG8RAAHkpc6uRTdnzhwVFxfruuuuI/wB8C1OAgGQtziRCAA6xgggAF8g/AHAPxEAAQAAfIYACAAA4DMEQAAAAJ8hAAIAAPgMARAAAMBnCIAAAAA+QwAEAADwGQIgAACAzxAAAQAAfIYACAAA4DMEQAAAAJ8hAAIAAPgMARAAkFGWZbldAoCDIAACAHrMtm05jqNYLKampibFYjE5jiPbtt0uDUAHgm4XAADwNtu2VVNTo3A4LElKpYxCoZAqKytVVlam0tJSJRIJl6sE8GmMAAIAeiQajSocDmtjfbMi5Y0KnrpdkfJGbaxvVjgc1owZM9wuEcBnEAABAD1SXl7e8nPWbq1Yl5QkrViXVMXs3ZKkiooKt0oD0ImcDoC2bWvEiBFulwEA6IRlWQqFQkqlTDr8tVpel0zvDubEECC35FwA7Nu3r6ZPn65nn31WS5cu1cUXX5yxdZ944ol68cUXtWTJEi1dulTnnntuxtYNAH6UTCYVj8dVUBDQ+FPahrxI2FJBQUDxeFzJZLKTNQBwQ84FQNu2lUqlNG3aNF199dUZW++AAQP08MMP68orr9SECRN07rnn6v/9v/+n0aNHZ+w5AMCP5s6d2/JzZpEi4ZYQGAlbqr65SJJUXV3tVmkAOpFzATAej+s3v/mNGhoaMrreSy65RI8//rjeeustSdLevXv1ox/9SNddd11GnwcA/Kaqqkp1dXUaNTyoZdXFal55pJZVF2vU8KDq6urkOI7bJQL4jJwLgNlSWlqqJ554os2yNWvW6MQTT3SpIgDID4lEQqWlpXIcJ707OB6Py3EcLgED5CjfXAewuLhY77zzTrvlH374oQvVAEB+SSQSikajikajsiyLY/6AHNdrAXDWrFkqKytTIBBod5sxRrfddlu7EbpMCgQCam5ubrd8//79WXtOAPAjwh+Q+3otAM6cOVMzZ87srafrUCAQkDGmzbKDXZrg8ssv1/Tp0yVJJ598smpra7NWX7addNJJnq7fb+gv76CvvIX+8hb6K3tMrrYJEyaYm266KSPr+t3vfmeGDx/ebvnKlSu7vI7a2lrXt0lPmtfr91ujv7zT6CtvNfrLW43+ys62y5uTQIYMGXLAScdffPFFff3rX2+zbNy4cdqyZUu2SwMAAMgpeREAp0yZom3btmnbtm0aOnRoh/eZN2+eLr74Yh177LGSpP79+6uqqkq33XZbb5YKAADgupw7C3jo0KGqrq5WMBhUUVGRbNvWpEmTJEm33Xabnn/++XaPaWxs1J49e7Rjxw7t27evw/Xu27dPF198se6++24NGDBAwWBQVVVVWrNmTVZfDwAAQK7JuQD47rvv6uyzzz6kxyxZskRFRUUHvd/mzZt1/vnnd7MyAACA/JAXu4ABAADQdQRAAAAAnyEAAgAA+AwBEAAAwGcIgAAAAD5DAAQAAPAZAiAAAIDPEAABAAB8hgAIAADgMwRAAAAAnyEAAgAA+AwBEAAAwGcIgAAAAD5DAAQAAPAZAiAAAIDPEAABAAB8hgAIAADgMwRAAAAAnyEAAgAA+AwBEAAAwGcIgAAAAD5DAAQAAPAZAiAAAIDPEAABAAB8hgAIAADgMwRAAAAAnyEAAgAA+AwBEAAAwGcIgAAAAD5DAAQAAPAZAiAAAIDPEAABAAB8hgAIAADgMwRAAAAAnyEAAgAA+AwBEAAAwGcIgAAAAD5DAAQAAPAZAiAAAIDPEAABAAB8hgAIAADgMwRAAAAAnyEAAgAA+AwBEAAAwGcIgAAAAD5DAAQAAPAZAiAAAIDPEAABAAB8hgAIAADgMwFJxu0ivCIej2vr1q1ul9FtxcXFamxsdLsMdBH95R30lbfQX95Cf3XfsGHDFAqFOr3d0PzRamtrXa+BRn/lY6OvvNXoL281+is7jV3AAAAAPkMABAAA8BkCoI/cd999bpeAQ0B/eQd95S30l7fQX9nBSSAAAAA+wwhgnrFtWyNGjHC7DAAAOmVZlr7whS+4XYavBd0uAD3Xt29fXXrppbrooos0dOhQzZs3Tz/72c8ysu4TTzxRv/zlL9W3b1/16dNHP/3pT/XCCy9kZN1+duONN2ry5MlKJpPasGGDrr/+en344YfdXt+wYcO0fPlyvfnmm22Wz5kzRwsWLOhpub5ywQUXKBqNKplM6sMPP9S1116rt956q0frHDhwoO655x6NHDlSgUBAf/zjH3XHHXdkqGJ/y0Z/rV+/XrFYrM2yhQsX6tZbb+3ReiFNnjxZU6dO1YgRI7R3716dffbZPV4n76/uc/1UZFrPWigUMt///vfN4MGDzYQJE8xNN92UkfUOGDDAvP766+a4444zkkz//v3NokWLzOjRo11/zV5uV111lfmf//mf9L8nT55sHn/88R6tc9iwYeb+++93/bV5vY0ZM8a8/PLLZsCAAUaSGTlypKmtrTX9+vXr0Xoff/xx82//9m/pf8+ZM8dcddVVrr9er7ds9dfChQtdf2352q644gozatSojG5n3l/dbq4XQMtgy2QAvOKKK8wNN9zQZtmYMWPMQw895Prr9HL7y1/+Yvr27dtm2fz5883IkSO7vU4CYGba73//ezNu3Lg2y374wx+aioqKbq9z1KhR5tlnn22zrLCw0NTW1ppAIOD6a/Zyy0Z/SQTA3mqZ2M68v7rfOAYQnSotLdUTTzzRZtmaNWt04oknulSR951wwglavXq1PvnkkzbLn3rqKU2aNKlH6x47dqyefvppvfzyy1q1apXuvvtuFRYW9midfjNixAjV1ta2WTZ//vwe7abq6H3U1NSk9evX64QTTuj2epGd/pKkww8/XI888ogWL16sFStW6KmnntLQoUN7tE5kB++v7iMAolPFxcV655132i3vybFqfjd48OAOpxPcsmWLBg8e3O31NjQ06IEHHtD3vvc9fe1rX9PYsWO1Zs0a3X777T0p13f27t3bbll9fb2OOOKIbq8zW32O7PSXJN177736yU9+ookTJ2r8+PG65ZZb9OCDDyoQCPRovcg83l/dx0kgOWzWrFkqKyvr8D8dY4xuu+22dt98MikQCKi5ubnd8v3792ftOb2qq30VDAb18ccft7tPMplUQUFBt59/7969+t///d82y37zm9/oueee08CBAwntXdTR37sxpkcf/Nnqc2SnvyTpgQceaPPvN954Q4sWLdK5556r559/vkfrRmbx/uo+AmAOmzlzpmbOnOlqDYFAQMaYNsssy3KpmtzV1b5KpVIdbr9+/fp1+GHWU3/+85/Tu51xcH36tN8pEgwG270HDkVv97mfZKO/OvPGG2/o5JNPJgDmGN5f3ccuYHSqoaFBw4YNa7f8sMMOc6Ga/BCPxzV8+PB2y4cNG6Z4PJ7x5zv88MP5T/AQDBgwoN2yY445Rg0NDd1eZ2/3uZ9ko786w3spN/H+6j4CoI8NGTJEtm13evuLL76or3/9622WjRs3Tlu2bMl2aXlr06ZNOumkk9qdnPGNb3xDS5cu7fRxB+uro446qt2yYDCoMWPGaP369d0v2Gc2btyoMWPGtFk2ZcqUA/aNbdsaMmRIp7cvXLhQF110UZtlffv21ZgxY7R58+aeFexz2eivjt5LklRWVqZXX321e4Wi23h/ZZfrpyLTMte6ehmYKVOmmFQqZfbs2WOGDh3a4X0OO+wwU1tba4499lgjtVwH8KWXXjJjxoxx/XV6uV199dVmzpw56X9feOGF5rHHHutRXy1YsMBMnz7d9OnTx0gylmWZ++67z1x55ZWuv14vtdGjR5tFixaZ/v37G6nlunKvv/56+jpzn21Dhw41e/bsMalUykyZMqXT9T7++ONm8uTJ6X/ffvvtXKcsR/tr5syZ5q677jKDBg1KL6uoqDC///3vXX+9+dYOdhkY3l9Zb64XQOthGzp0qFm4cKFZvHixWbNmjfnb3/5mFi9ebBYvXmzOO++8Dh8zYcIEs3v3bvP222+b4uLiTtd9/PHHmwULFpglS5aY1157zVxwwQWuv958aDfccINZtmyZWbp0qXnwwQfNwIEDO71vV/qqT58+5rrrrjOvvPKKWbJkiamtrTWXXXaZ66/Ti+28884zr776qlmyZIl54YUXzIknntjpfYuLi83mzZtNIpEwEydO7PR+gwYNMvfff79ZunSpWb58ubnxxhtdf5350rLRXxdccIF54YUXzMsvv2xWr15tbr/9dlNYWOj6a82HNm/evPTn086dO9O/z5gxo9v9xfurey3wj18AAADgExwDCAAA4DMEQAAAAJ8hAAIAAPgMARAAAMBnCIAAAAA+QwAEgBxj27ZGjBjhdhkA8hgBEAByQN++fTV9+nQ9++yzWrp0qS6++GK3SwKQx4JuFwAAaBn1S6VSmjZtmj7/+c/rjDPOcLskAHmMAAgAOSAej+s3v/mN22UA8Al2AQNAFv3oRz9SMpnU4sWLdeyxx0qSvvrVr+qNN95QIpFoN5E9APQGRgABIItuv/12xWIxnXXWWdqyZYsk6Y033tCOHTt09dVXa/ny5S5XCMCPGAEEgCx7+OGHVVNTo/vuu08DBw7U448/rtmzZxP+ALiGEUAA6AXV1dUaOHCgNm3apG9/+9t67bXX3C4JgI8xAggAvaBfv346++yzNX/+fE2dOtXtcgD4HAEQALKsb9++mjdvnu655x5deeWV2rx5s37xi1+4XRYAHyMAAkAWFRYWat68ebr//vv1/PPPS5J+/vOf691339Udd9zhcnUA/IpjAAEgS/r27atly5bp6KOP1oABA7R27Vq9++67Ou2003T++efrK1/5ioYMGaKLL75YQ4cOVXV1tYLBoIqKimTbtiZNmiRJuu2229LhEQAyISDJuF0EAAAAeg+7gAEAAHyGAAgAAOAzBEAAAACfIQACAAD4DAEQAADAZwiAAAAAPkMABAAA8BkCIAAAgM8QAAEAAHzm/wMqIDxeMaKU9QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x576 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Number of points to generate\n",
    "nPats = 30 + np.floor(np.random.rand() * 30);\n",
    "# Generate 2-dimensional random points\n",
    "patterns = np.random.rand(2, int(nPats)) * 2 - 1;\n",
    "# Slope of separating line\n",
    "slope = np.log(np.random.rand() * 10);\n",
    "yint = np.random.rand() * 2 - 1;\n",
    "# Create the indexes for a two-class problem\n",
    "desired = (patterns[1,:] - patterns[0,:] * slope - yint > 0) * 1;\n",
    "# Plot the corresponding pattern\n",
    "fig = plot_patterns(patterns,desired);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Single neuron\n",
    "\n",
    "For the first parts of the tutorial, we will perform the simplest classification model possible in a neural network setting, a single neuron. We briefly recall here that; given an input vector $ \\mathbf{x} \\in \\mathbb{R}^{n} $, a single neuron computes the function  \n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "\\hat{y}=\\phi\\left(\\sum_{i = 1}^{n}w_{i}.x_{i} + b\\right)\n",
    "\\label{eq1}\n",
    "\\end{equation}\n",
    "$$\n",
    "\n",
    "with $ \\mathbf{w} \\in \\mathbb{R}^{n} $ a weight vector, $ b $ a bias and $ \\phi\\left(\\cdot\\right) $ an *activation function*. Therefore, if we consider the *threshold* activation function ($ \\phi_{0}\\left(x\\right)=1 $ if $ x \\geq 0$), a single neuron simply performs an *affine transform* and then a *linear* discrimination of the space. A network will be composed of _layers_ of these neurons, which produce successive computations\n",
    "\n",
    "<img src=\"images/02_feedforward_nn.png\" align=\"center\"/>\n",
    "\n",
    "Geometrically, a single neuron computes an hyperplane that separates the space. In order to learn, we have to adjust the weights and know \"how much wrong we are\". To do so, we consider that we know the desired output $ d $ of a system for a given example $ \\mathbf{x} $ (eg. a predicted value for a regression system, a class value for a classification system). Therefore, we define the loss function $ \\mathcal{L}_{\\mathcal{D}} $ over a whole dataset as\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "\\mathcal{L}_{\\mathcal{D}}=\\sum_{j=1}^{k_{\\mathcal{D}}}\\left\\Vert y_{j}-\\hat{y}_{j}\\right\\Vert ^{2}\n",
    "\\label{eq2}\n",
    "\\end{equation}\n",
    "$$\n",
    "\n",
    "In order to know how to change the weights based on the value of the errors, we need to now \"how to change it to make it better\". Therefore, we should compute the sets of derivatives of the error given each parameter\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "\\Delta\\bar{\\mathbf{w}}=\\left(\\frac{\\delta\\mathcal{L}}}{\\delta w_{1}},\\ldots,\\frac{\\delta\\mathcal{L}}}{\\delta w_{n}}\\right)\n",
    "\\label{eq3}\n",
    "\\end{equation}\n",
    "$$ \n",
    "\n",
    "***\n",
    "\n",
    "**Exercise**  \n",
    "\n",
    "  1. Perform the derivatives of the output given a single neuron\n",
    "  2. Perform the derivatives for the bias as well\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training your own neuron\n",
    "\n",
    "We will start by training a single neuron to learn how to perform this discrimination with a linear problem (so that a single neuron is enough to solve it). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Inputs to use\n",
    "inputs = patterns;\n",
    "# Initialize the weights\n",
    "weights = np.random.randn(1, 2);\n",
    "bias = np.random.randn(1, 1)\n",
    "# Learning rate\n",
    "eta = 0.05;\n",
    "# Weight decay\n",
    "lambdaW = 0.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you need to update the following code loop to ensure that your neuron learns to separate between the classes \n",
    "***\n",
    "\n",
    "**Exercise**\n",
    "  1. Update the loop so that it computes the forward propagation error\n",
    "  2. Update the loop to perform learning (based on back-propagation)\n",
    "  3. Run the learning procedure, which should produce a result similar to that displayed on the website\n",
    "  4. Perform multiple re-runs by **tweaking the hyperparameters** (learning rate, weight decay)\n",
    "  5. What observations can you make on the learning process?\n",
    "  6. (Optional) Change the input patterns, and confirm your observations.\n",
    "  6. (Optional) Incorporate the bias in the weights to obtain a **vectorized** code.\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot the corresponding pattern\n",
    "fig = plot_patterns(patterns,desired);\n",
    "plt.draw()\n",
    "# Update loop\n",
    "for i in range(50):\n",
    "    \n",
    "    ######################\n",
    "    # YOUR CODE GOES HERE\n",
    "    ######################\n",
    "    \n",
    "    print('%2d.  weights = %f, %f, %f' % (i, bias[0, 0], weights[0, 0], weights[0, 1]))\n",
    "    plot_boundary(np.concatenate((bias, weights), axis=1), i, '--', fig)\n",
    "    plt.draw()\n",
    "    time.sleep(0.2)\n",
    "plot_boundary(np.concatenate((bias, weights), axis=1), i, '-', fig);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2-layer XOR problem\n",
    "\n",
    "In most cases, classification problems are far from being linear. Therefore, we need more advanced methods to be able to compute non-linear class boundaries. The advantage of neural networks is that the same principle can be applied in a *layer-wise* fashion. This allows to further discriminate the space in sub-regions (as seen in the course). We will try to implement the 2-layer *perceptron* that can provide a solution to the infamous XOR problem. The idea is now to have the output of the first neurons to be connected to a set of other neurons. Therefore, if we take back our previous formulation, we have the same output for the first neuron(s) $\\hat{y}$, that we will now term as $y^{(1)}$. Then, we feed these outputs to a second layer of neurons, which gives\n",
    "\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "y^{(2)}=\\phi\\left(\\sum_{i = 1}^{n}w_{i}.y^{(1)}_{i} + b\\right)\n",
    "\\label{eq6}\n",
    "\\end{equation}\n",
    "$$  \n",
    "\n",
    "Finally, we will rely on the same loss $\\mathcal{L_{D}}$ as in the previous exercise, but the outputs used are $y_2$ instead of $y$. As in the previous case, we now need to compute the derivatives of the weights and biases for several layers . However, you should see that some form of generalization might be possible for any number of layer.  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "\n",
    "**Exercise**  \n",
    "  1. Perform the derivatives for the last layer specifically\n",
    "  2. Define a generalized derivative for any previous layer\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can construct the prototypical set of XOR values by using the following code (note that this is the most simple case, but still this is typically a problem that cannot be solved by a _linear classifier_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "patterns = np.array([[-1, -1],[-1,  1],[1, -1],[1,  1]]).transpose() # Input patterns\n",
    "desired = np.array([0, 1, 1, 0])                       # Corresponding classes\n",
    "# Initialize based on their sizes\n",
    "nInputs = patterns.shape[0]\n",
    "nOutputs = 1\n",
    "nPat = patterns.shape[1]\n",
    "# First plot the patterns\n",
    "fig = plot_patterns(patterns, desired);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The variables that will be used by your code are the following.\n",
    "\n",
    "```Matlab\n",
    "patterns          % 2 x n matrix of random points\n",
    "desired           % classes of the patterns \n",
    "inputs1           % 3 x n final matrix of inputs (accounting for bias)\n",
    "nHiddens          % Number of hidden units\n",
    "learnRate         % Learning rate parameter\n",
    "momentum          % Momentum parameter (bonus)\n",
    "weights1          % 1st layer weights\n",
    "weights2          % 2nd layer weights\n",
    "TSS_Limit         % Sum-squared error limit\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nHiddens = 2           # Number of hidden units\n",
    "learnRate = 0.001       # Learning rate parameter\n",
    "momentum = 0.1         # Momentum parameter\n",
    "# Overall input patterns\n",
    "inputs = patterns\n",
    "TSS_Limit = 0.02                                           # Sum-squared error limit\n",
    "# Learning rate\n",
    "eta = 0.001"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "\n",
    "**Exercise**  \n",
    "\n",
    "  1. Update the forward propagation and error computation (compared to desired).\n",
    "  2. Update the back-propagation part to learn the weights of both layers.\n",
    "  3. Run the learning, which should produce a result similar to that displayed below.\n",
    "  4. Perform multiple re-runs of the learning procedure (re-launching with different initializations)\n",
    "  5. What observations can you make on the learning process?\n",
    "  6. What happens if you initialize all weights to zeros?\n",
    "  7. (Optional) Implement the *sparsity* constraint in your neural network.\n",
    "  7. (Optional) Implement the *weight decay* constraint in your network.\n",
    "  7. (Optional) Add the *momentum* to the learning procedure.\n",
    "  \n",
    "*For optional questions, please look after the first code box for more information*\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Weights of first and second layer\n",
    "weights1 = (np.random.randn(nHiddens, nInputs) - 0.5)      # 1st layer weights\n",
    "weights2 = (np.random.randn(nOutputs, nHiddens) - 0.5)     # 2nd layer weights\n",
    "bias1 = (np.random.randn(nHiddens, 1) - 0.5)               # 1st layer biases\n",
    "bias2 = (np.random.randn(nOutputs, 1) - 0.5)               # 2nd layer biases\n",
    "# First plot the patterns\n",
    "fig = plot_patterns(patterns, desired);\n",
    "# Iterate for a fixed number of iterations\n",
    "for epoch in range(200):\n",
    "  \n",
    "    ######################\n",
    "    # YOUR CODE GOES HERE\n",
    "    ######################\n",
    "    \n",
    "    #print('Epoch %3d:  Error = %f' % (epoch, TSS));\n",
    "    #if TSS < TSS_Limit:\n",
    "    #    break\n",
    "    if (epoch - 1 % 20)==0:\n",
    "        plot_boundary(np.concatenate((bias2, weights2), axis=1), epoch, '--', fig)\n",
    "\n",
    "plot_boundary(np.concatenate((bias1, weights1), axis=1), epoch, '-', fig)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Optional questions**\n",
    "\n",
    "2. *Weight decay* constraint\n",
    "\n",
    "As nothing constrains the weights in the network, we can note that usually all weights vector given a multiplicative factor might be equivalent, which can stall the learning (and lead to exploding weights). The *weight decay* allows to regularize the learning by penalizing weights with a too wide amplitude. The idea is to add this constraint as a term to the final loss (which leads to an indirect \"pressure\" on the learning process. Therefore, the final loss will be defined as\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "\\mathcal{L}_{final}=\\mathcal{L_D} + \\lambda \\sum_{l} \\sum_{i} \\sum_{j} \\left( W_{ij}^{l} \\right)^{2}\n",
    "\\end{equation}\n",
    "$$\n",
    "\n",
    "where the parameter $\\lambda$ controls the relative importance of the two terms.\n",
    "\n",
    "3. *Momentum* in learning\n",
    "\n",
    "Usually, in complex problems, the gradient can be very noisy and, therefore, the learning might oscillate widely. In order to reduce this problem, we can *smooth* the different gradient updates by retaining the values of the gradient at each iteration and then performing an update based on the latest gradient $\\delta_{i}^{t}$ and the gradient at the previous iteration $\\delta_{i}^{t-1}$. Therefore, a gradient update is applied as\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "\\delta_{final}^{t} = \\delta_{i}^{t} + m.\\delta_{i}^{t-1}\n",
    "\\end{equation}\n",
    "$$\n",
    "\n",
    "with $m$ the momentum parameter, which control the amount of gradient smoothing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3-layer audio classification\n",
    "\n",
    "Finally, we will attack a complete audio classification problem and try to perform neural network learning on a set of audio files. The data structure will be the same as the one used for parts 1 and 2. As discussed during the courses, even though a 2-layer neural network can provide non-linear boundaries, it can not perform \"holes\" inside those regions. In order to obtain an improved classification, we will now rely on a 3-layer neural network. The modification to the code of section 3.2 should be minimal, as the back-propagation will be similar for the new layer as one of the two others. We do not develop the math here as it is simply a re-application of the previous rules with an additional layer (which derivatives you should have generalized in the previous exercise).  \n",
    "\n",
    "However, up until now, we only performed *binary classification* problems, but this time we need to obtain a decision rule for multiple classes. Therefore, we cannot rely on simply computing the distance between desired patterns and the obtained binary value. The idea here is to rely on the *softmax regression*, by considering classes as a vector of probabilities. The desired answers will therefore be considered as a set of *probabilities*, where the desired class is $1$ and the others are $0$ (called *one-hot* representation). Then, the cost function will rely on the softmax formulation\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "\\mathcal{L_D}(\\theta) = - \\frac{1}{m} \\left[ \\sum_{i=1}^{m} \\sum_{j=1}^{k} 1\\left\\{y^{(i)} = j\\right\\} log \\frac{e^{\\theta_{j}^{T} x^{(i)}}}{\\sum_{l=1}^{k} e^{ \\theta_{l}^{T} x^{(i)} }}  \\right]\n",
    "\\end{equation}\n",
    "$$\n",
    "\n",
    "Therefore, we compute the output of the softmax by taking \n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "p(y^{(i)} = j | x^{(i)}; \\theta) = \\frac{e^{\\theta_{j}^{T} x^{(i)}}}{\\sum_{l=1}^{k} e^{ \\theta_{l}^{T} x^{(i)}} }\n",
    "\\end{equation}\n",
    "$$\n",
    "\n",
    "By taking derivatives, we can show that the gradient of the softmax layer is\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "\\nabla_{\\theta_{j}} \\mathcal{L_D}(\\theta) = - \\frac{1}{m} \\sum_{i=1}^{m}{ \\left[ x^{(i)} \\left( 1\\{ y^{(i)} = j\\}  - p(y^{(i)} = j \\mid x^{(i)}, \\theta) \\right) \\right]}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sweet activation functions\n",
    "\n",
    "As discussed in the course, the interest of stacking layers is that there is an _activation function_, which allows non-linear interactions between the dimensions (and avoids to only compute a single huge affine transform). Although the `sigmoid` function has been historically the most used, there has been some large developments since. Notably the `ReLU` (Rectified Linear Unit) is one of the major difference in modern networks (we will see more about that in a later course)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function for computing the Sigmoid activation\n",
    "def sigmoid(x):\n",
    "    return 1 / (1 + np.exp(-x))\n",
    "def dsigmoid(a):\n",
    "    return a * (1.0 - a)\n",
    "# Function for computing the ReLU activation\n",
    "def relu(x):\n",
    "    return np.maximum(0, x)\n",
    "def drelu(x):\n",
    "    return 1 / (1 + np.exp(-x))\n",
    "# Function for computing the Tanh activation\n",
    "def tanh(x):\n",
    "    return np.tanh(x);\n",
    "def dtanh(x): \n",
    "    return np.cosh(x) ^ -2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we plot some simple examples of what these activation functions look like. You can try to rely on these functions in your previous training code and witness the differences in training.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from helper_plot import prep_plots, finalize_plots\n",
    "funcs = [('sigmoid',sigmoid,'red'), ('tanh',tanh,'orange'), ('relu',relu,'yellow')]\n",
    "axes = prep_plots([funcs[0][0], funcs[1][0], funcs[2][0]], fig_size=(18,5), fig_num=1)\n",
    "x_plot = np.linspace(-5,5,100)\n",
    "np.maximum(0, x_plot)\n",
    "for f in range(3):\n",
    "    axes[f].plot(x_plot, funcs[f][1](x_plot), color=funcs[f][2], linewidth=4, label=funcs[f][0])\n",
    "finalize_plots(axes, fig_title=\"Activation functions\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Coding the whole network from scratch\n",
    "\n",
    "You should now have all the tools necessary to apply neural networks from scratch to a more complex problem. In the following exercise, we simply removed any guideline code, and you need to code all the procedure for training a NN and **apply it to audio data**. You will use the spectral features discussed in the previous exercise as an input.\n",
    "\n",
    "***\n",
    "\n",
    "**Exercise**\n",
    "\n",
    "  1. Based on the previous neural network, upgrade the code to a 3-layer neural network\n",
    "  2. Implement the *softmax regression* on top of your 3-layer network\n",
    "  3. Use the provided code to perform classification on a pre-defined set of features\n",
    "  4. As previously, change the set of features to assess their different accuracies\n",
    "  5. Evaluate the neural network accuracy for all features combinations\n",
    "  6. What happens if the learning rate is too large ? What is this phenomenon ?\n",
    "  7. (Optional) Perform a more advanced visualization of the learning process.\n",
    "  \n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Iterate for a fixed number of iterations\n",
    "for epoch in range(500):\n",
    "    \n",
    "    ######################\n",
    "    # YOUR CODE GOES HERE\n",
    "    ######################\n",
    "    \n",
    "    print('Epoch %3d:  Error = %f\\n' % epoch, TSS);\n",
    "    if TSS < TSS_Limit:\n",
    "        break\n",
    "plo_boundary(np.concatenate(bias1, weights1), epoch, '-');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using Pytorch to enjoy life\n",
    "\n",
    "Up to now, we have been writing every operations by ourselves (in order to better understand the mathematics behind NN). However, there exists of course some simplifying libraries that provide large simplifications to this question.\n",
    "\n",
    "One of the most powerful and complete library of this sort is `Pytorch`, which has been developed for several years (even prior to the recent boom of deep learning). `Pytorch` provides a large set of pre-coded layers, but also **computational graphs** and **autograd**, which are very powerful paradigms allowing to define complex operators and automatically taking derivatives.\n",
    "\n",
    "### Defining our network\n",
    "\n",
    "When building neural networks we frequently think of arranging the computation into layers, some of which have learnable parameters which will be optimized during learning. In `PyTorch`, the `nn` package provides higher-level abstractions over raw computational graphs that are useful for building neural networks. The `nn` package defines a set of `Modules`, which are roughly equivalent to neural network layers. A `Module` receives input `Tensors` and computes output `Tensors`, but may also hold internal state such as `Tensors` containing learnable parameters. The nn package also defines a set of useful loss functions that are commonly used when training neural networks.\n",
    "\n",
    "In the following example, we use the `nn` package to show how easy it is to instantiate our previous three-layers network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "# Define the input dimensions\n",
    "in_size = 1000\n",
    "# Number of neurons in a layer\n",
    "hidden_size = 100\n",
    "# Output (target) dimension\n",
    "output_size = 10\n",
    "# Use the nn package to define our model and loss function.\n",
    "model = torch.nn.Sequential(\n",
    "    torch.nn.Linear(in_size, hidden_size),\n",
    "    torch.nn.ReLU(),\n",
    "    torch.nn.Linear(hidden_size, hidden_size),\n",
    "    torch.nn.ReLU(),\n",
    "    torch.nn.Linear(hidden_size, output_size),\n",
    "    torch.nn.Softmax(dim=1)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Optimizing the network\n",
    "\n",
    "Up to this point we have updated the weights of our models by manually performing the gradient descent algorithm (changing the parameters vectors). Although this is not a huge burden for simple optimization algorithms like stochastic gradient descent, in practice we often train neural networks using more sophisticated optimizers like AdaGrad, RMSProp or Adam (that we will see later in this course)\n",
    "\n",
    "The `optim` package in PyTorch abstracts the idea of an optimization algorithm and provides implementations of commonly used optimization algorithms, and greatly simplfies the training loop associated with training a neural network.\n",
    "\n",
    "For the sake of presentation we will use random inputs $\\mathbf{x}$ that should be matched with random outputs $\\mathbf{y}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 64\n",
    "# Create random Tensors to hold inputs and outputs\n",
    "x = torch.randn(N, D_in)\n",
    "y = torch.randn(N, D_out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the following example we optimize the model using the Adam algorithm provided by the `optim` package, based on a `MSE` loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Learning rate\n",
    "learning_rate = 1e-4\n",
    "# Loss function that we will use\n",
    "loss_fn = torch.nn.MSELoss(reduction='sum')\n",
    "# Optimizer to fit the weights of the network\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)\n",
    "for t in range(500):\n",
    "    # Forward pass: compute predicted y by passing x to the model.\n",
    "    y_pred = model(x)\n",
    "    # Compute the loss.\n",
    "    loss = loss_fn(y_pred, y)\n",
    "    # Before the backward pass, zero all of the network gradients\n",
    "    optimizer.zero_grad()\n",
    "    # Backward pass: compute gradient of the loss with respect to parameters\n",
    "    loss.backward()\n",
    "    # Calling the step function to update the parameters\n",
    "    optimizer.step()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using Pytorch to classify audio\n",
    "\n",
    "Now that we know the main components of `Pytorch` to define and optimize networks, your assignement is to define a complete classification problem from audio data, by relying on this toolbox\n",
    "\n",
    "***\n",
    "\n",
    "**Exercise**\n",
    "\n",
    "  1. Use `Pytorch` to define a model for audio classification\n",
    "  2. Import the audio features dataset and check that your model produces an output\n",
    "  3. Write the optimization loop (think carefully about the _loss function_\n",
    "  4. As previously, change the set of features to assess their different accuracies\n",
    "  5. Think of how you could use more complex features (time series, audio, STFT) to classify your data\n",
    "  \n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "######################\n",
    "# YOUR CODE GOES HERE\n",
    "######################\n",
    "    "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}