Lab-5.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab 5- Deep Learning Model\n",
    "\n",
    "This lab is meant to get you started in using Keras to design Deep Neural Networks. The goal here is to simply repeat Lab 4, but with DNNs.\n",
    "\n",
    "Let's start with reading the data, like before:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "filename=\"/data/afarbin/DLClass/SUSY/SUSY.csv\"\n",
    "VarNames=[\"signal\", \"l_1_pT\", \"l_1_eta\",\"l_1_phi\", \"l_2_pT\", \"l_2_eta\", \"l_2_phi\", \"MET\", \"MET_phi\", \"MET_rel\", \"axial_MET\", \"M_R\", \"M_TR_2\", \"R\", \"MT2\", \"S_R\", \"M_Delta_R\", \"dPhi_r_b\", \"cos_theta_r1\"]\n",
    "RawNames=[\"l_1_pT\", \"l_1_eta\",\"l_1_phi\", \"l_2_pT\", \"l_2_eta\", \"l_2_phi\"]\n",
    "FeatureNames=[ \"MET\", \"MET_phi\", \"MET_rel\", \"axial_MET\", \"M_R\", \"M_TR_2\", \"R\", \"MT2\", \"S_R\", \"M_Delta_R\", \"dPhi_r_b\", \"cos_theta_r1\"]\n",
    "\n",
    "df = pd.read_csv(filename, dtype='float64', names=VarNames)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now lets define training and test samples. Note that DNNs take very long to train, so for testing purposes we will use only about 10% of the 5 million events in the training/validation sample. Once you get everything working, make the final version of your plots with the full sample. \n",
    "\n",
    "Also note that Keras had trouble with the Pandas tensors, so after doing all of the nice manipulation that Pandas enables, we convert the Tensor to a regular numpy tensor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "N_Max=550000\n",
    "N_Train=500000\n",
    "\n",
    "Train_Sample=df[:N_Train]\n",
    "Test_Sample=df[N_Train:N_Max]\n",
    "\n",
    "X_Train=np.array(Train_Sample[VarNames[1:]])\n",
    "y_Train=np.array(Train_Sample[\"signal\"])\n",
    "\n",
    "X_Test=np.array(Test_Sample[VarNames[1:]])\n",
    "y_Test=np.array(Test_Sample[\"signal\"])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now lets setup everything. Note that you will need to use a GPU to get reasonable training times. The Jupyter server will have up tp 4 GPUs in it. Your Jupyter session is now running through a batch queue system that effectively assigns a GPU to you. Here we tell Keras's backend, Theano, to use the GPU assigned to you."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os \n",
    "print \"Using Queue:\", os.environ[\"PBS_QUEUE\"]\n",
    "gpuid=int(os.environ[\"PBS_QUEUE\"][3:4])\n",
    "print \"Using GPU:\", gpuid\n",
    "os.environ['THEANO_FLAGS'] = \"mode=FAST_RUN,device=gpu%s,floatX=float32,force_device=True\" % (gpuid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There was some problem getting the profiler to work in this notebook, so we turn it off: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import theano\n",
    "theano.config.profile=False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will build a simple model, as described in class. Note that this is very small model, so things run fast. You should attempt more ambitious models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Dense\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Dense(12, input_dim=X_Train.shape[1], init='uniform', activation='relu'))\n",
    "model.add(Dense(8, init='uniform', activation='relu'))\n",
    "model.add(Dense(1, init='uniform', activation='sigmoid'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "import keras\n",
    "grapher = keras.utils.dot_utils.Grapher()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model has to be compiled. At this time we set the loss function and the optimizer too:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we train. We are running only 10 epochs in this example. Models may need hundreds of epochs before they stop improving."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "history=model.fit(X_Train, y_Train, validation_data=(X_Test,y_Test), nb_epoch=10, batch_size=2048)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model history keeps track of the loss and accuracy for each epoch. Note that the training above was setup to run on the validation sample at the end of each epoch:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print history.history"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 5.1\n",
    "\n",
    "You will need to create several models and make sure they are properly trained. Write a function that takes this history and plots the values versus epoch. For every model that you train in the remainder of this lab, assess:\n",
    "\n",
    "    * Has you model's performance plateaued? If not train for more epochs. \n",
    "\n",
    "    * Compare the performance on training versus test sample. Are you over training?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "## Your Solution Here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can evaluate how the trained model does on the test sample as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scores = model.evaluate(X_Test, y_Test)\n",
    "print scores"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we can make ROC curves as before:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import roc_curve, auc\n",
    "fpr, tpr, _ = roc_curve(y_Test, model.predict(X_Test))\n",
    "                        \n",
    "roc_auc = auc(fpr, tpr)\n",
    "\n",
    "plt.plot(fpr,tpr,color='darkorange',label='ROC curve (area = %0.2f)' % roc_auc)\n",
    "plt.legend(loc=\"lower right\")\n",
    "plt.xlabel('False Positive Rate')\n",
    "plt.ylabel('True Positive Rate')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 5.2\n",
    "\n",
    "Following lab 4, make a comparison of the performance between models trained with raw, features, and raw+features data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "## Your solution here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 5.3\n",
    "\n",
    "Again, following lab 4, design and implement at least 3 different DNN models. Train them and compare performance. You may try different architectures, loss functions, and optimizers to see if there is an effect."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "## Your solution here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Honors Problem- Exercise 5.4\n",
    "\n",
    "Write a function that evaluates the performance (AUC) as a function of a given input variable. You will need to bin the test data in the variable (i.e. make sub-samples for events which have the particular variable in a range), evaluate the performance in each bin, and plot the results.\n",
    "\n",
    "Apply your function to each input variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}