3.1.1_mighty_mlp.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3.1.1 Mighty MLP, the most basic form of neural network\n",
    "### On tabular data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### requirements\n",
    "```\n",
    "python 3.6\n",
    "\n",
    "numpy == '1.14.3'\n",
    "pandas == '0.23.0'\n",
    "tensorflow == '1.8.0'\n",
    "keras == '2.2.0'\n",
    "```\n",
    "Other versions of above library will probably work"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are going to start with this dataset:[Simplified Human Activity Recognition w/Smartphone](https://www.kaggle.com/mboaglio/simplifiedhuarus)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "Content\n",
    "Abstract: Human Activity Recognition database built from the recordings of 30 subjects performing activities of daily living (ADL) while carrying a waist-mounted smartphone with embedded inertial sensors.\n",
    "\n",
    "Data Set Characteristics: Multivariate, Time-Series\n",
    "\n",
    "Data Set Information:\n",
    "\n",
    "The experiments have been carried out with a group of 30 volunteers within an age bracket of 19-48 years. Each person performed six activities (WALKING, WALKING_UPSTAIRS, WALKING_DOWNSTAIRS, SITTING, STANDING, LAYING) wearing a smartphone (Samsung Galaxy S II) on the waist. Using its embedded accelerometer and gyroscope, we captured 3-axial linear acceleration and 3-axial angular velocity at a constant rate of 50Hz. The experiments have been video-recorded to label the data manually. The obtained dataset has been randomly partitioned into two sets, where 70% of the volunteers was selected for generating the training data and 30% the test data.\n",
    "\n",
    "The sensor signals (accelerometer and gyroscope) were pre-processed by applying noise filters and then sampled in fixed-width sliding windows of 2.56 sec and 50% overlap (128 readings/window). The sensor acceleration signal, which has gravitational and body motion components, was separated using a Butterworth low-pass filter into body acceleration and gravity. The gravitational force is assumed to have only low frequency components, therefore a filter with 0.3 Hz cutoff frequency was used. From each window, a vector of features was obtained by calculating variables from the time and frequency domain.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Actually I didn't finish reading the part I have posted above..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can download the train and test datasets also from here, and place it in /data\n",
    "\n",
    "http://45.76.223.58/human_phone_train.csv\n",
    "\n",
    "http://45.76.223.58/human_phone_test.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "train = pd.read_csv(\"/data/human_phone_train.csv\")\n",
    "test = pd.read_csv(\"/data/human_phone_test.csv\")\n",
    "# or run this:\n",
    "# train = pd.read_csv(\"http://45.76.223.58/human_phone_train.csv\")\n",
    "# test = pd.read_csv(\"http://45.76.223.58/human_phone_test.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3609, 563)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.values.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_y = train.values[...,1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['STANDING', 'STANDING', 'STANDING', ..., 'WALKING_UPSTAIRS',\n",
       "       'WALKING_UPSTAIRS', 'WALKING_DOWNSTAIRS'], dtype=object)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use set to remove the duplicates and see the categories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "cates = list(set(train_y.tolist()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['LAYING',\n",
       " 'SITTING',\n",
       " 'WALKING_UPSTAIRS',\n",
       " 'WALKING_DOWNSTAIRS',\n",
       " 'WALKING',\n",
       " 'STANDING']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "cates2idx = dict((v,k) for k,v in enumerate(cates))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A mapping dictionary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'LAYING': 0,\n",
       " 'SITTING': 1,\n",
       " 'WALKING_UPSTAIRS': 2,\n",
       " 'WALKING_DOWNSTAIRS': 3,\n",
       " 'WALKING': 4,\n",
       " 'STANDING': 5}"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cates2idx"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([5, 5, 5, ..., 2, 2, 3])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_y_arr = np.vectorize(lambda x:cates2idx[x])(train_y)\n",
    "train_y_arr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Transform to onehot encoding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1., 0., 0., 0., 0., 0.],\n",
       "       [0., 1., 0., 0., 0., 0.],\n",
       "       [0., 0., 1., 0., 0., 0.],\n",
       "       [0., 0., 0., 1., 0., 0.],\n",
       "       [0., 0., 0., 0., 1., 0.],\n",
       "       [0., 0., 0., 0., 0., 1.]])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.eye(len(cates))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_y_oh = np.eye(len(cates))[train_y_arr]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0., 0., 0., 0., 0., 1.],\n",
       "       [0., 0., 0., 0., 0., 1.],\n",
       "       [0., 0., 0., 0., 0., 1.],\n",
       "       ...,\n",
       "       [0., 0., 1., 0., 0., 0.],\n",
       "       [0., 0., 1., 0., 0., 0.],\n",
       "       [0., 0., 0., 1., 0., 0.]])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_y_oh"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.27899999999999997, -0.0196, -0.11, ..., -0.852, 0.182, -0.043],\n",
       "       [0.27699999999999997, -0.0127, -0.10300000000000001, ..., -0.852,\n",
       "        0.188, -0.0347],\n",
       "       [0.27699999999999997, -0.0147, -0.107, ..., -0.85,\n",
       "        0.18899999999999997, -0.0351],\n",
       "       ...,\n",
       "       [0.284, -0.00796, -0.11900000000000001, ..., -0.657, 0.272, 0.183],\n",
       "       [0.207, 0.0246, -0.10400000000000001, ..., -0.657, 0.267, 0.188],\n",
       "       [0.39299999999999996, -0.0178, -0.0902, ..., -0.807, 0.19,\n",
       "        0.11800000000000001]], dtype=object)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_x_arr  = train.values[...,2:]\n",
    "train_x_arr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((3609, 561), (3609, 6))"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_x_arr.shape, train_y_oh.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/salvor/anaconda3/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n",
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "from keras.layers import *\n",
    "from keras.models import *\n",
    "from keras.optimizers import Adam"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pure Linear"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "ipt = Input((561,))\n",
    "x = Dense(len(cates),activation=\"softmax\")(ipt)\n",
    "model = Model(ipt,x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(optimizer=Adam(),loss=\"categorical_crossentropy\",metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 2526 samples, validate on 1083 samples\n",
      "Epoch 1/15\n",
      "2526/2526 [==============================] - 1s 340us/step - loss: 1.6585 - acc: 0.2629 - val_loss: 1.4660 - val_acc: 0.3232\n",
      "Epoch 2/15\n",
      "2526/2526 [==============================] - 1s 220us/step - loss: 1.2705 - acc: 0.4640 - val_loss: 1.1537 - val_acc: 0.5549\n",
      "Epoch 3/15\n",
      "2526/2526 [==============================] - 1s 214us/step - loss: 1.0586 - acc: 0.6049 - val_loss: 1.0198 - val_acc: 0.6602\n",
      "Epoch 4/15\n",
      "2526/2526 [==============================] - 1s 209us/step - loss: 0.9361 - acc: 0.7110 - val_loss: 0.9082 - val_acc: 0.7424\n",
      "Epoch 5/15\n",
      "2526/2526 [==============================] - 1s 218us/step - loss: 0.8397 - acc: 0.7724 - val_loss: 0.8339 - val_acc: 0.7747\n",
      "Epoch 6/15\n",
      "2526/2526 [==============================] - 1s 227us/step - loss: 0.7677 - acc: 0.8076 - val_loss: 0.7705 - val_acc: 0.8172\n",
      "Epoch 7/15\n",
      "2526/2526 [==============================] - 1s 242us/step - loss: 0.7065 - acc: 0.8373 - val_loss: 0.7162 - val_acc: 0.8403\n",
      "Epoch 8/15\n",
      "2526/2526 [==============================] - 1s 236us/step - loss: 0.6570 - acc: 0.8468 - val_loss: 0.6654 - val_acc: 0.8661\n",
      "Epoch 9/15\n",
      "2526/2526 [==============================] - 1s 206us/step - loss: 0.6141 - acc: 0.8567 - val_loss: 0.6394 - val_acc: 0.8587\n",
      "Epoch 10/15\n",
      "2526/2526 [==============================] - 1s 198us/step - loss: 0.5783 - acc: 0.8650 - val_loss: 0.5947 - val_acc: 0.8938\n",
      "Epoch 11/15\n",
      "2526/2526 [==============================] - 1s 206us/step - loss: 0.5480 - acc: 0.8654 - val_loss: 0.5743 - val_acc: 0.8846\n",
      "Epoch 12/15\n",
      "2526/2526 [==============================] - 1s 201us/step - loss: 0.5191 - acc: 0.8808 - val_loss: 0.5465 - val_acc: 0.8873\n",
      "Epoch 13/15\n",
      "2526/2526 [==============================] - 1s 208us/step - loss: 0.4943 - acc: 0.8812 - val_loss: 0.5209 - val_acc: 0.8984\n",
      "Epoch 14/15\n",
      "2526/2526 [==============================] - 0s 196us/step - loss: 0.4746 - acc: 0.8895 - val_loss: 0.5083 - val_acc: 0.8910\n",
      "Epoch 15/15\n",
      "2526/2526 [==============================] - 1s 207us/step - loss: 0.4529 - acc: 0.8947 - val_loss: 0.4835 - val_acc: 0.8957\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0xb22358f28>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(train_x_arr,train_y_oh,batch_size=256,epochs=15,validation_split=.3,shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Neural Network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "ipt = Input((561,))\n",
    "x = Dense(1000,activation=\"relu\")(ipt)\n",
    "x = Dense(len(cates),activation=\"softmax\")(x)\n",
    "model = Model(ipt,x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### About \"relu\"\n",
    "\n",
    "\"relu\" is a frequently used activation layer for neural network\n",
    "\n",
    "relu helps the model to [learn the xor gate](https://www.beyondthelines.net/machine-learning/neural-network/).  With this neural network can figure out automatically, by learning, a full spectrum of [logic gates](https://en.wikipedia.org/wiki/Logic_gate).\n",
    "\n",
    "And mlp is like hundreds/thousands of logic gates assembled together. It's some sort of powerful organ to learn,summarize and figure out reasons from the historical pattern between input feature and output results. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_2 (InputLayer)         (None, 561)               0         \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 1000)              562000    \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 6)                 6006      \n",
      "=================================================================\n",
      "Total params: 568,006\n",
      "Trainable params: 568,006\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(optimizer=Adam(),loss=\"categorical_crossentropy\",metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 2526 samples, validate on 1083 samples\n",
      "Epoch 1/15\n",
      "2526/2526 [==============================] - 1s 419us/step - loss: 1.2296 - acc: 0.5344 - val_loss: 0.6912 - val_acc: 0.7747\n",
      "Epoch 2/15\n",
      "2526/2526 [==============================] - 1s 293us/step - loss: 0.4883 - acc: 0.8298 - val_loss: 0.4162 - val_acc: 0.8550\n",
      "Epoch 3/15\n",
      "2526/2526 [==============================] - 1s 305us/step - loss: 0.3147 - acc: 0.8975 - val_loss: 0.3628 - val_acc: 0.8569\n",
      "Epoch 4/15\n",
      "2526/2526 [==============================] - 1s 298us/step - loss: 0.2424 - acc: 0.9192 - val_loss: 0.2658 - val_acc: 0.9141\n",
      "Epoch 5/15\n",
      "2526/2526 [==============================] - 1s 297us/step - loss: 0.2034 - acc: 0.9303 - val_loss: 0.2356 - val_acc: 0.9298\n",
      "Epoch 6/15\n",
      "2526/2526 [==============================] - 1s 288us/step - loss: 0.1720 - acc: 0.9466 - val_loss: 0.2352 - val_acc: 0.9141\n",
      "Epoch 7/15\n",
      "2526/2526 [==============================] - 1s 301us/step - loss: 0.1536 - acc: 0.9525 - val_loss: 0.2136 - val_acc: 0.9271\n",
      "Epoch 8/15\n",
      "2526/2526 [==============================] - 1s 298us/step - loss: 0.1382 - acc: 0.9565 - val_loss: 0.1994 - val_acc: 0.9354\n",
      "Epoch 9/15\n",
      "2526/2526 [==============================] - 1s 287us/step - loss: 0.1299 - acc: 0.9584 - val_loss: 0.2072 - val_acc: 0.9234\n",
      "Epoch 10/15\n",
      "2526/2526 [==============================] - 1s 295us/step - loss: 0.1277 - acc: 0.9557 - val_loss: 0.1946 - val_acc: 0.9261\n",
      "Epoch 11/15\n",
      "2526/2526 [==============================] - 1s 295us/step - loss: 0.1074 - acc: 0.9679 - val_loss: 0.1883 - val_acc: 0.9354\n",
      "Epoch 12/15\n",
      "2526/2526 [==============================] - 1s 290us/step - loss: 0.0991 - acc: 0.9711 - val_loss: 0.1715 - val_acc: 0.9483\n",
      "Epoch 13/15\n",
      "2526/2526 [==============================] - 1s 321us/step - loss: 0.0991 - acc: 0.9683 - val_loss: 0.1787 - val_acc: 0.9455\n",
      "Epoch 14/15\n",
      "2526/2526 [==============================] - 1s 303us/step - loss: 0.0833 - acc: 0.9782 - val_loss: 0.1781 - val_acc: 0.9363\n",
      "Epoch 15/15\n",
      "2526/2526 [==============================] - 1s 299us/step - loss: 0.0829 - acc: 0.9762 - val_loss: 0.1700 - val_acc: 0.9418\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0xb22648dd8>"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(train_x_arr,train_y_oh,batch_size=256,epochs=15,validation_split=.3,shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deeper Neural Network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "ipt = Input((561,))\n",
    "x = Dense(1000,activation=\"relu\")(ipt)\n",
    "x = Dense(1000,activation=\"relu\")(x)\n",
    "x = Dense(1000,activation=\"relu\")(x)\n",
    "x = Dense(len(cates),activation=\"softmax\")(x)\n",
    "model = Model(ipt,x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_3 (InputLayer)         (None, 561)               0         \n",
      "_________________________________________________________________\n",
      "dense_4 (Dense)              (None, 1000)              562000    \n",
      "_________________________________________________________________\n",
      "dense_5 (Dense)              (None, 1000)              1001000   \n",
      "_________________________________________________________________\n",
      "dense_6 (Dense)              (None, 1000)              1001000   \n",
      "_________________________________________________________________\n",
      "dense_7 (Dense)              (None, 6)                 6006      \n",
      "=================================================================\n",
      "Total params: 2,570,006\n",
      "Trainable params: 2,570,006\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(optimizer=Adam(),loss=\"categorical_crossentropy\",metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.fit(train_x_arr,train_y_oh,batch_size=256,epochs=15,validation_split=.3,shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Overfiting is happending, we can try a mechanism to handle over fitting\n",
    "\n",
    "### Add dropouts\n",
    "\n",
    "Dropouts are not very complex mechanism but sure is very useful, to help understand dropout, I found google's deepmind's explain on [nueron deletion](https://deepmind.com/blog/understanding-deep-learning-through-neuron-deletion/) very helpful.\n",
    "\n",
    "Dropout is just a mechanism(an nn layer actually) to do neuron deletion ramdomly during training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "do_ratio = .3 # dropout ratio\n",
    "ipt = Input((561,))\n",
    "x = Dense(1000,activation=\"relu\")(ipt)\n",
    "x = Dropout(do_ratio)(x)\n",
    "x = Dense(1000,activation=\"relu\")(x)\n",
    "x = Dropout(do_ratio)(x)\n",
    "x = Dense(1000,activation=\"relu\")(x)\n",
    "x = Dropout(do_ratio)(x)\n",
    "x = Dense(len(cates),activation=\"softmax\")(x)\n",
    "model = Model(ipt,x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(optimizer=Adam(),loss=\"categorical_crossentropy\",metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.fit(train_x_arr,train_y_oh,batch_size=256,epochs=15,validation_split=.3,shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sequetial Format "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Aside from the \"functional API\" of keras you've seen above, you can also do \"Sequential\" model if your model is a long one way pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "do_ratio = .3\n",
    "model = Sequential([\n",
    "    Dense(1000, input_shape=(561,)),\n",
    "    Activation('relu'),\n",
    "    Dropout(do_ratio),\n",
    "    Dense(1000),\n",
    "    Activation('relu'),\n",
    "    Dropout(do_ratio),\n",
    "    Dense(1000),\n",
    "    Activation('relu'),\n",
    "    Dropout(do_ratio),\n",
    "    Dense(len(cates)),\n",
    "    Activation('softmax'),\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(optimizer=Adam(),loss=\"categorical_crossentropy\",metrics=[\"accuracy\"])\n",
    "model.fit(train_x_arr,train_y_oh,batch_size=256,epochs=15,validation_split=.3,shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Saving and loading keras model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can save the model to a h5 file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.save_weights(\"/data/mlp.0.0.1.h5\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can load the model from a h5 file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "do_ratio = .3\n",
    "model_2 = Sequential([\n",
    "    Dense(1000, input_shape=(561,)),\n",
    "    Activation('relu'),\n",
    "    Dropout(do_ratio),\n",
    "    Dense(1000),\n",
    "    Activation('relu'),\n",
    "    Dropout(do_ratio),\n",
    "    Dense(1000),\n",
    "    Activation('relu'),\n",
    "    Dropout(do_ratio),\n",
    "    Dense(len(cates)),\n",
    "    Activation('softmax'),\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_2.load_weights(\"/data/mlp.0.0.1.h5\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prediction with keras model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_array = test.values[...,1:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result_array = model_2.predict(test_array)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result_array, result_array.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result_idx = result_array.argmax(axis=-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result_idx[:20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Reverse mapping dictionary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "idx2cate = dict(enumerate(cates))\n",
    "idx2cate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_predict = np.vectorize(lambda x:idx2cate[x])(result_idx).tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_predict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Understanding Multi-Layer Percetron (simplist deep neural network)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I found hands on entertainment a lot better than I draw overwhelming lines and dots on the board. So we can try this [google neural network playground](https://playground.tensorflow.org/)\n",
    "\n",
    "Also the [explain from 3blue 1brown](https://www.youtube.com/watch?v=aircAruvnKk) is very intuitive and brief. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extras: The pytorch version of the code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch import nn\n",
    "from torch.nn import functional as F"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What extra libs you need for requirements\n",
    "\n",
    "```\n",
    "pytorch == 0.4.0\n",
    "ray\n",
    "```\n",
    "\n",
    "\"ray\" is my [own ml/dl tool github repo](https://github.com/raynardj/ray)\n",
    "\n",
    "You can gitclone it to your ```lib/python3.6/site-packages/``` folder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build up model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "class mlp(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(mlp,self).__init__()\n",
    "        self.fc1 = nn.Linear(561,1000)\n",
    "        self.fc2 = nn.Linear(1000,1000)\n",
    "        self.fc3 = nn.Linear(1000,1000)\n",
    "        self.fc4 = nn.Linear(1000,len(cates))\n",
    "    \n",
    "    def forward(self,x):\n",
    "        x = self.fc1(x)\n",
    "        x = F.relu(x)\n",
    "        x = F.dropout(x,p=0.3)\n",
    "        x = self.fc2(x)\n",
    "        x = F.relu(x)\n",
    "        x = F.dropout(x,p=0.3)\n",
    "        x = self.fc3(x)\n",
    "        x = F.relu(x)\n",
    "        x = F.dropout(x,p=0.3)\n",
    "        x = self.fc4(x)\n",
    "        x = F.softmax(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "mlp(\n",
       "  (fc1): Linear(in_features=561, out_features=1000, bias=True)\n",
       "  (fc2): Linear(in_features=1000, out_features=1000, bias=True)\n",
       "  (fc3): Linear(in_features=1000, out_features=1000, bias=True)\n",
       "  (fc4): Linear(in_features=1000, out_features=6, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = mlp()\n",
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prepare the data generator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.utils.data.dataset import TensorDataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "trn_slice = np.random.rand(len(train_x_arr))>.7"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "ds_train = TensorDataset(torch.from_numpy(train_x_arr[trn_slice].astype(np.float)),\n",
    "                        torch.from_numpy(train_y_arr[trn_slice].astype(np.float)),)\n",
    "\n",
    "ds_valid = TensorDataset(torch.from_numpy(train_x_arr[~trn_slice].astype(np.float)),\n",
    "                        torch.from_numpy(train_y_arr[~trn_slice].astype(np.float)),)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.utils.data import DataLoader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_ld = DataLoader(ds_train,batch_size = 256, shuffle = True)\n",
    "valid_ld = DataLoader(ds_valid,batch_size = 256, shuffle = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Test generator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[tensor([[ 0.2820, -0.0153, -0.1100,  ...,  0.3620, -0.3660, -0.6090],\n",
       "         [ 0.2680, -0.0281, -0.1190,  ..., -0.6900,  0.3080,  0.0615],\n",
       "         [ 0.3580,  0.0543, -0.0047,  ..., -0.7110,  0.2920,  0.0657],\n",
       "         ...,\n",
       "         [ 0.3460, -0.0371, -0.1010,  ..., -0.6970,  0.1240, -0.1990],\n",
       "         [ 0.2790, -0.0180, -0.1090,  ..., -0.6080, -0.1610, -0.0717],\n",
       "         [ 0.2760, -0.0170, -0.1090,  ..., -0.8120,  0.1650, -0.0987]], dtype=torch.float64),\n",
       " tensor([ 0.,  5.,  5.,  0.,  4.,  4.,  4.,  3.,  5.,  5.,  3.,  1.,\n",
       "          5.,  3.,  3.,  3.,  2.,  2.,  3.,  2.,  2.,  2.,  5.,  3.,\n",
       "          3.,  1.,  5.,  5.,  0.,  0.,  1.,  5.,  2.,  0.,  0.,  0.,\n",
       "          0.,  2.,  1.,  0.,  2.,  4.,  1.,  2.,  5.,  5.,  0.,  4.,\n",
       "          2.,  2.,  5.,  0.,  0.,  0.,  4.,  5.,  3.,  1.,  2.,  1.,\n",
       "          2.,  5.,  5.,  2.,  1.,  1.,  4.,  0.,  1.,  0.,  0.,  1.,\n",
       "          4.,  1.,  4.,  5.,  5.,  3.,  3.,  5.,  5.,  1.,  5.,  2.,\n",
       "          5.,  4.,  1.,  5.,  4.,  0.,  5.,  0.,  4.,  4.,  4.,  0.,\n",
       "          3.,  2.,  1.,  1.,  2.,  0.,  1.,  0.,  2.,  2.,  4.,  0.,\n",
       "          3.,  5.,  5.,  4.,  1.,  1.,  2.,  1.,  5.,  4.,  1.,  5.,\n",
       "          0.,  3.,  0.,  4.,  2.,  3.,  1.,  1.,  5.,  4.,  2.,  3.,\n",
       "          1.,  1.,  1.,  5.,  5.,  0.,  2.,  5.,  1.,  2.,  5.,  0.,\n",
       "          1.,  1.,  0.,  1.,  5.,  5.,  5.,  4.,  4.,  3.,  5.,  2.,\n",
       "          3.,  0.,  2.,  2.,  1.,  2.,  4.,  0.,  1.,  0.,  1.,  5.,\n",
       "          5.,  4.,  3.,  3.,  2.,  0.,  1.,  2.,  0.,  2.,  0.,  2.,\n",
       "          0.,  0.,  1.,  2.,  3.,  4.,  5.,  2.,  1.,  0.,  1.,  0.,\n",
       "          5.,  2.,  2.,  4.,  2.,  3.,  0.,  5.,  3.,  1.,  4.,  4.,\n",
       "          5.,  4.,  2.,  0.,  3.,  5.,  4.,  2.,  3.,  4.,  2.,  4.,\n",
       "          0.,  3.,  3.,  2.,  3.,  3.,  1.,  4.,  1.,  2.,  5.,  5.,\n",
       "          1.,  5.,  3.,  4.,  2.,  0.,  2.,  3.,  0.,  2.,  4.,  3.,\n",
       "          3.,  5.,  1.,  5.,  0.,  0.,  4.,  4.,  5.,  2.,  1.,  4.,\n",
       "          1.,  3.,  1.,  1.], dtype=torch.float64)]"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "next(iter(train_ld))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Wrap up trainer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ray.matchbox import Trainer\n",
    "from torch.optim import Adam"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "trainer = Trainer(ds_train,val_dataset=ds_valid,batch_size=256)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = Adam(model.parameters(),lr = 1e-3)\n",
    "loss_func = nn.CrossEntropyLoss()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define a training step\n",
    "def action(*args,**kwargs):\n",
    "    x,y = args[0]\n",
    "    model.zero_grad()\n",
    "    \n",
    "    # forward pass calculation\n",
    "    y_ = model(x.float())\n",
    "    loss = loss_func(y_,y.long())\n",
    "    \n",
    "    # backward propagation (calculate the gradients)\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "    return {\"loss\":loss.item()}\n",
    "\n",
    "def val_action(*args,**kwargs):\n",
    "    x,y = args[0]\n",
    "    \n",
    "    # forward pass calculation\n",
    "    y_ = model(x.float())\n",
    "    loss = loss_func(y_,y.long())\n",
    "    return {\"loss\":loss.item()}\n",
    "trainer.action = action\n",
    "trainer.val_action = val_action"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  0%|          | 0/5 [00:00<?, ?it/s]/Users/salvor/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:20: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.\n",
      "100%|██████████| 5/5 [00:00<00:00,  9.70it/s]\n",
      "😎[val_ep_0_i_9]\tloss\t1.646: 100%|██████████| 10/10 [00:00<00:00, 27.93it/s]\n",
      "100%|██████████| 5/5 [00:00<00:00,  9.75it/s]\n",
      "😎[val_ep_1_i_9]\tloss\t1.535: 100%|██████████| 10/10 [00:00<00:00, 32.45it/s]\n",
      "100%|██████████| 5/5 [00:00<00:00, 10.69it/s]\n",
      "😎[val_ep_2_i_9]\tloss\t1.401: 100%|██████████| 10/10 [00:00<00:00, 28.59it/s]\n",
      "100%|██████████| 5/5 [00:00<00:00,  8.89it/s]\n",
      "😎[val_ep_3_i_9]\tloss\t1.325: 100%|██████████| 10/10 [00:00<00:00, 29.50it/s]\n",
      "100%|██████████| 5/5 [00:00<00:00, 10.23it/s]\n",
      "😎[val_ep_4_i_9]\tloss\t1.306: 100%|██████████| 10/10 [00:00<00:00, 27.62it/s]\n",
      "100%|██████████| 5/5 [00:00<00:00,  9.79it/s]\n",
      "😎[val_ep_5_i_9]\tloss\t1.275: 100%|██████████| 10/10 [00:00<00:00, 32.14it/s]\n",
      "100%|██████████| 5/5 [00:00<00:00, 10.74it/s]\n",
      "😎[val_ep_6_i_9]\tloss\t1.212: 100%|██████████| 10/10 [00:00<00:00, 28.64it/s]\n",
      "100%|██████████| 5/5 [00:00<00:00,  9.86it/s]\n",
      "😎[val_ep_7_i_9]\tloss\t1.177: 100%|██████████| 10/10 [00:00<00:00, 26.36it/s]\n",
      "100%|██████████| 5/5 [00:00<00:00,  9.34it/s]\n",
      "😎[val_ep_8_i_9]\tloss\t1.235: 100%|██████████| 10/10 [00:00<00:00, 26.39it/s]\n",
      "100%|██████████| 5/5 [00:00<00:00,  9.07it/s]\n",
      "😎[val_ep_9_i_9]\tloss\t1.185: 100%|██████████| 10/10 [00:00<00:00, 26.60it/s]\n",
      "100%|██████████| 5/5 [00:00<00:00,  9.62it/s]\n",
      "😎[val_ep_10_i_9]\tloss\t1.196: 100%|██████████| 10/10 [00:00<00:00, 32.23it/s]\n",
      "100%|██████████| 5/5 [00:00<00:00,  9.24it/s]\n",
      "😎[val_ep_11_i_9]\tloss\t1.153: 100%|██████████| 10/10 [00:00<00:00, 27.98it/s]\n",
      "100%|██████████| 5/5 [00:00<00:00,  9.33it/s]\n",
      "😎[val_ep_12_i_9]\tloss\t1.145: 100%|██████████| 10/10 [00:00<00:00, 27.85it/s]\n",
      "100%|██████████| 5/5 [00:00<00:00,  9.20it/s]\n",
      "😎[val_ep_13_i_9]\tloss\t1.132: 100%|██████████| 10/10 [00:00<00:00, 29.56it/s]\n",
      "100%|██████████| 5/5 [00:00<00:00,  9.39it/s]\n",
      "😎[val_ep_14_i_9]\tloss\t1.173: 100%|██████████| 10/10 [00:00<00:00, 29.88it/s]\n"
     ]
    }
   ],
   "source": [
    "trainer.train(15)"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}