Batch Gradient Descent for Softmax Regression (with early stopping).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training Models\n",
    "\n",
    "## Implementing Batch Gradient Descent with early stopping for Softmax Regression\n",
    "\n",
    "We will be implementing Batch Gradient Descent with early stopping for Softmax Regression, without usig Scikit-Learn. We will import the iris dataset from scikit-learn to test the algorithm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import scipy as sp\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from sklearn.datasets import load_iris"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "iris=load_iris()\n",
    "X=iris['data']\n",
    "y=iris['target']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now split the data into training, validation and test sets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_with_bias = np.c_[np.ones([len(X), 1]), X]\n",
    "np.random.seed(1234)\n",
    "\n",
    "test_ratio = 0.2\n",
    "validation_ratio = 0.2\n",
    "total_size = len(X_with_bias)\n",
    "\n",
    "test_size = int(total_size * test_ratio)\n",
    "validation_size = int(total_size * validation_ratio)\n",
    "train_size = total_size - test_size - validation_size\n",
    "\n",
    "rnd_indices = np.random.permutation(total_size)\n",
    "\n",
    "X_train = X_with_bias[rnd_indices[:train_size]]\n",
    "y_train = y[rnd_indices[:train_size]]\n",
    "X_valid = X_with_bias[rnd_indices[train_size:-test_size]]\n",
    "y_valid = y[rnd_indices[train_size:-test_size]]\n",
    "X_test = X_with_bias[rnd_indices[-test_size:]]\n",
    "y_test = y[rnd_indices[-test_size:]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have to convert the classes into probability matrix vector (in this case, one-hot vector) in order to use Softmax"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def one_hot(Y):\n",
    "    nclasses=Y.max()+1\n",
    "    m = len(Y)\n",
    "    Y_one_hot=np.zeros((m,nclasses))\n",
    "    Y_one_hot[np.arange(m),Y]=1\n",
    "    return Y_one_hot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 0, 1, 2, 1, 1, 1, 0, 0, 0])"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_valid[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.,  1.,  0.],\n",
       "       [ 1.,  0.,  0.],\n",
       "       [ 0.,  1.,  0.],\n",
       "       [ 0.,  0.,  1.],\n",
       "       [ 0.,  1.,  0.],\n",
       "       [ 0.,  1.,  0.],\n",
       "       [ 0.,  1.,  0.],\n",
       "       [ 1.,  0.,  0.],\n",
       "       [ 1.,  0.,  0.],\n",
       "       [ 1.,  0.,  0.]])"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "one_hot(y_valid[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Testing it worked. So we can now convert all our target values into the probability matrix vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y_train_prob = one_hot(y_train)\n",
    "y_valid_prob = one_hot(y_valid)\n",
    "y_test_prob = one_hot(y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can not write the Softmax Regression function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def softmax(sk_X):\n",
    "    top = np.exp(sk_X)\n",
    "    bottom = np.sum(top,axis=1,keepdim=True)\n",
    "    return top/bottom"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "n_inputs = X_train.shape[1]\n",
    "n_outputs = len(np.unique(y_train))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5 3\n"
     ]
    }
   ],
   "source": [
    "print (n_inputs, n_outputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can train. The equations we need to look at are the cost function and the gradients.\n",
    "\n",
    "* The cost function:\n",
    "\n",
    "$$J(\\mathbf{\\Theta}) =\n",
    "-\\dfrac{1}{m}\\sum_{i=1}^{m}\\sum_{k=1}^{K}{y_k^{(i)}\\log\\left(\\hat{p}_k^{(i)}\\right)}$$\n",
    "\n",
    "* And the equation for the gradients:\n",
    "\n",
    "$$\\nabla_{\\mathbf{\\theta}^{(k)}} \\, J(\\mathbf{\\Theta}) = \\dfrac{1}{m} \\sum_{i=1}^{m}{ \\left ( \\hat{p}^{(i)}_k - y_k^{(i)} \\right ) \\mathbf{x}^{(i)}}$$\n",
    "\n",
    "Note that $\\log\\left(\\hat{p}_k^{(i)}\\right)$ may not be computable if $\\hat{p}_k^{(i)} = 0$. So we will add a tiny value $\\epsilon$ to $\\log\\left(\\hat{p}_k^{(i)}\\right)$ to avoid getting nan values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}