Ex3.Classifying-movie-reviews.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://www.th-koeln.de/img/logo.svg\" style=\"float: right;\" width=\"200\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3rd exercise: <font color=\"#C70039\">Binary sentiment classification with IMDB movie reviews</font>\n",
    "* Course: DIS21a.1\n",
    "* Lecturer: <a href=\"https://www.gernotheisenberg.de/\">Gernot Heisenberg</a>\n",
    "* Author of notebook modifications and adaptations: <a href=\"https://www.gernotheisenberg.de/\">Gernot Heisenberg</a>\n",
    "* Date:   09.09.2024\n",
    "\n",
    "<img src=\"https://brand24.com/blog/wp-content/uploads/2017/04/Screen-Shot-2017-04-12-at-16.24.20.png\" style=\"float: center;\" width=\"200\">\n",
    "\n",
    "**GENERAL NOTE 1**: \n",
    "Please make sure you are reading the entire notebook, since it contains a lot of information about your tasks (e.g. regarding the set of certain paramaters or specific computational tricks, etc.), and the written mark downs as well as comments contain a lot of information on how things work together as a whole. \n",
    "\n",
    "**GENERAL NOTE 2**: \n",
    "* Please, when commenting source code, just use English language only. \n",
    "* When describing an observation (for instance, after you have run through your test plan) you may use German language.\n",
    "This applies to all exercises in DIS 21a.1.  \n",
    "\n",
    "--------------------\n",
    "\n",
    "### <font color=\"ce33ff\">DESCRIPTION</font>:\n",
    "This notebook allows you for a binary classification (two classes only). You will classify movie reviews into \"positive\" reviews and \"negative\" reviews, just based on the text content of the reviews.\n",
    "\n",
    "The **IMDB** dataset is a set of 50.000 highly-polarized reviews from the Internet Movie Database. They are split into 25.000 reviews for training and 25.000 reviews for testing, each set consisting in 50% negative and 50% positive reviews.\n",
    "The IMDB dataset comes packaged with Keras. It has already been preprocessed: the reviews (sequences of words) \n",
    "have been turned into sequences of integers, where each integer stands for a specific word in a dictionary.\n",
    "\n",
    "-----------------------\n",
    "\n",
    "### <font color=\"FFC300\">TASKS</font>:\n",
    "Within this notebook, the tasks that you need to work on are always listed as bullet points below. \n",
    "If a task is more challenging and consists of several steps, this is indicated as well. \n",
    "Make sure you have worked down the task list and commented your doings. \n",
    "This should be done using markdown.<br> \n",
    "<font color=red>Make sure you don't forget to specify your name and your matriculation number in the notebook before submitting it.</font>\n",
    "\n",
    "**YOUR TASKS in this exercise are as follows**:\n",
    "1. import the notebook into Google Colab.\n",
    "2. make sure you specified you name and your matriculation number in the header below my name and date.\n",
    "    * set the date too and remove mine.\n",
    "3. read the entire notebook carefully.\n",
    "    * add comments whereever you feel it necessary for better understanding\n",
    "    * run the notebook for the first time and note the result in your markdown result table (your test plan). \n",
    "4. go into the section 'building the ANN'.\n",
    "    * add the missing code that does create a network as shown in the image in the lecture slides on page 166 (File: 'DIS21a.1-7.HANDS_ON.First.DLNetwork.Architectures.for.Solving.Three.Interesting.Problems.pdf')\n",
    "    * set the activation function to ReLu\n",
    "    * set the correct activation function in the last layer (the output layer). What is correct?\n",
    "5. stay in the 'building the ANN' section again.\n",
    "    * add the missing code for compiling the network by setting\n",
    "        * the loss function\n",
    "        * the optimizer\n",
    "        * the evaluation metric (little hint: is the label distribution balanced ?)\n",
    "6. take less `training` data and rerun the network.\n",
    "    * add the size of the training data as a column in the table and note the accuracy you achieve\n",
    "7. take less `testing` data and rerun the network.\n",
    "    * add the size of the testing data as a column in the table and note the accuracy you achieve\n",
    "8. increase/decrease the number of epochs and the batch size \n",
    "    * add those hyperparameters as columns in the table and note the accuracy you achieve\n",
    "9. make combinations of this according to your testplan. Make sure you combine with sense and reason and not just chaotically.\n",
    "10. comment your observations.\n",
    "    * when is the accuracy increasing / decreasing\n",
    "11. until now there were 2 hidden layers. Try to use 1 or 3 hidden layers and see how it affects the test accuracy. Use again a little table. Describe your observations.\n",
    "12. Try to use layers with more hidden units or less hidden units: 32 units, 64 units, 128 units ...! What effect can you observe?\n",
    "13. Try to use the `mse` loss function instead of `binary_crossentropy`. What effect can you observe?\n",
    "14. Try to use the `tanh` activation (an activation that was popular in the early days of neural networks) instead of `relu`. What effect can you observe?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## START OF THE NOTEBOOK CODE\n",
    "----------------------------------------------------------------------------------------------------------------------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#tensorflow and keras (implicit)\n",
    "import tensorflow\n",
    "tensorflow.keras.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### loading the IMDB movie review data set\n",
    "This code loads the data set (when you run it for the first time on a local machine, about approx. 80MB of data will be downloaded into the memory):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras.datasets import imdb\n",
    "\n",
    "(train_data, train_labels), (test_data, test_labels) = imdb.load_data(num_words=10000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "The argument `num_words=10000` means that we will only keep the top 10.000 most frequently occurring words in the training data. Rare words will be discarded. This allows us to work with vector data of manageable size.\n",
    "\n",
    "The variables `train_data` and `test_data` are lists of reviews, each review being a list of word indices (encoding a sequence of words). \n",
    "`train_labels` and `test_labels` are lists of 0s and 1s, where 0 stands for \"negative\" and 1 stands for \"positive\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "train_data[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "train_labels[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we restricted ourselves to the top 10.000 most frequent words, no word index will exceed 10.000:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "max([max(sequence) for sequence in train_data])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### data preparation\n",
    "\n",
    "Since lists of integers cannot be fed into a neural network, the lists need to be turned into tensors by one-hot-encoding ( vectors of 0s and 1s). This means for instance, turning the sequence `[3, 5]` into a 10000-dimensional vector, that would be all-zeros except for indices 3 and 5, which would be ones. Then use as first layer in the network a `Dense` layer, capable of handling floating point vector data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "# just take the first 10.000 most frequent words\n",
    "def vectorize_sequences(sequences, dimension=10000):\n",
    "    # Create an all-zero matrix of shape (len(sequences), dimension)\n",
    "    results = np.zeros((len(sequences), dimension))\n",
    "    for i, sequence in enumerate(sequences):\n",
    "        results[i, sequence] = 1.  # set specific indices of results[i] to 1s\n",
    "    return results\n",
    "\n",
    "# Our vectorized training data\n",
    "x_train = vectorize_sequences(train_data)\n",
    "# Our vectorized test data\n",
    "x_test = vectorize_sequences(test_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's what our samples look like now:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "x_train[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Vectorizing the labels too, which is straightforward:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Our vectorized labels\n",
    "y_train = np.asarray(train_labels).astype('float32')\n",
    "y_test = np.asarray(test_labels).astype('float32')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### building the ANN\n",
    "\n",
    "Now the data is ready to be fed into a neural network. The input data is simply vectors, and the labels are scalars (1s and 0s): this is the easiest setup you will ever encounter.\n",
    "A type of network that performs well on such a problem would be a simple stack of fully-connected (`Dense`) layers as you have learned in my lectures. \n",
    "The final output layer will use a special activation so as to output a probability (a score between 0 and 1, indicating how likely the sample is to have the target \"1\", i.e. how likely the review is to be positive).\n",
    "\n",
    "The implementation is very similar to what you have learned from the MNIST example from the earlier exercise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# necessary inputs\n",
    "from tensorflow.keras import models\n",
    "from tensorflow.keras import layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "model = models.Sequential()\n",
    "\n",
    "'''ADD THE MISSING CODE HERE'''\n",
    "'''LOOK AT THE TASK LIST TO SEE WHAT ARCHITECTURE THE NETWORK SHALL HAVE '''\n",
    "\n",
    "# your code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At last, a loss function and an optimizer needs to be specified.\n",
    "Since this is a binary classification problem and the output of our network is a probability it is best to use the `binary_crossentropy` loss. It is not the only viable choice: you could use, for instance, `mean_squared_error`. But crossentropy is usually the best choice when you are dealing with models that compute and output probabilities. \n",
    "Crossentropy is a quantity from the field of Information Theory, that measures the \"distance\" \n",
    "between probability distributions (or in our case, between the actual distribution (ground-truth) and the predictions).\n",
    "\n",
    "In addition, since the problem is class-balanced, what do you think can be used as an evaluation metric? Is there a special metric that is used when the classification task is binary?\n",
    "\n",
    "The network can be configured with the `rmsprop` optimizer, since it always does a good job. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras import optimizers\n",
    "from tensorflow.keras import losses\n",
    "from tensorflow.keras import metrics\n",
    "\n",
    "'''ADD THE MISSING CODE HERE'''\n",
    "'''LOOK AT THE TEXT ABOVE TO SEE WHAT PARAMETERS THE NETWORK SHALL HAVE'''\n",
    "\n",
    "# your code\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "--------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### training the ANN\n",
    "Let's train the model for 4 epochs (4 iterations over all samples in the `x_train` and `y_train` tensors), in mini-batches of 512 samples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "history = model.fit(x_train, y_train, epochs=4, batch_size=512) # compare to the one from MNIST !!!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On CPU of a local machine (not on Google Colab), this will take less than two seconds per epoch -- training is over in 9 seconds. If you are using GPU support it is much faster. <font color=red>Note:</font> Colab also offers GPU support. Please read through the Colab documentation for finding out how to set it up. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the call to `model.fit()` returns a `history` object. This object has a member `history`, which is a dictionary containing data about everything that happened during training. Let's use it to output some info on the metrics used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "history_dict = history.history\n",
    "history_dict.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### evaluate the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results = model.evaluate(x_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our fairly naive approach achieves an accuracy of higher than 80%. \n",
    "With state-of-the-art approaches, one should be able to get close to 95% (we will come to this in a later exercise.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <font color=\"#C70039\">Include your result table here and reflect a good test plan (see task list)</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### using a trained network to generate predictions on new data\n",
    "\n",
    "After having trained a network, you want to use it in a practical setting. You can generate the likelihood of reviews being positive by using the `predict` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.predict(x_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print out the first 10 and the last 10 entries.\n",
    "As you can see, the network is very confident for some samples (0.99, 0.05) but less confident for others (0.7, 0.2). \n",
    "\n",
    "----------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---------------------------------\n",
    "## <font color=\"ce33ff\">SIDE PROJECT FOR IMPROVING YOUR PYTHON SKILLS</font>\n",
    "\n",
    "**DESCRIPTION:**\n",
    "Here is a way implemented to quickly decode one of the above used reviews back to English words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# word_index is a dictionary mapping words to an integer index\n",
    "word_index = imdb.get_word_index()\n",
    "\n",
    "# reverse it, mapping integer indices to words\n",
    "reverse_word_index = dict([(value, key) for (key, value) in word_index.items()])\n",
    "\n",
    "# decode the review; note that the indices were offset by 3\n",
    "# because 0, 1 and 2 are reserved indices for \"padding\", \"start of sequence\", and \"unknown\".\n",
    "decoded_review = ' '.join([reverse_word_index.get(i - 3, '?') for i in train_data[0]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "decoded_review"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.17"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}