Week 7 - Class 7 LAB (optional).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Week 7 - Class 7\n",
    "\n",
    "This notebook have the most important exercises for Chapter 7.\n",
    "The lab 7 from https://github.com/fchollet/deep-learning-with-python-notebooks is a guide for this notebook.\n",
    "\n",
    "It will be covered:\n",
    "- 3 ways to build keras models: sequential, funtional api, Multi-input, multi-output models\n",
    "- Using middle layers\n",
    "- Writting your own metrics\n",
    "- Callbacks\n",
    "- TensorBoard\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#perform all your imports in the beginning \n",
    "from tensorflow import keras\n",
    "from tensorflow.keras import layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sequential \n",
    "The weights are defined when the input is defined"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# the sequential class\n",
    "model = keras.Sequential([\n",
    "    layers.Dense(64, activation=\"relu\"),\n",
    "    layers.Dense(10, activation=\"softmax\")\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# this will give an error because the NN is not yet initialized \n",
    "model.weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# this can be increamental, adding layer by layer\n",
    "model = keras.Sequential()\n",
    "model.add(layers.Dense(64, activation=\"relu\"))\n",
    "model.add(layers.Dense(10, activation=\"softmax\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# the build method calls and initializes the weights of the layers (randomly)\n",
    "model.build(input_shape=(None, 3))\n",
    "model.weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "input_shape(728,) is the same as batch_input=(batch_size,728) or input_shape=(None, 728)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1.1 Call the summary method to see the layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "<your-answer>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Another way to initialize the weights without the build method, instead you spoecify the input\n",
    "model2 = keras.Sequential()\n",
    "model2.add(keras.Input(shape=(3,)))\n",
    "model2.add(layers.Dense(64, activation=\"relu\"))\n",
    "model2.add(layers.Dense(10, activation=\"softmax\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1.2 Check the weights values of model2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "<your-answer>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Funtional "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs = keras.Input(shape=(3,), name=\"my_input\")\n",
    "features = layers.Dense(64, activation=\"relu\")(inputs)\n",
    "outputs = layers.Dense(10, activation=\"softmax\")(features)\n",
    "modelF = keras.Model(inputs=inputs, outputs=outputs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "modelF.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "see the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "keras.utils.plot_model(modelF, \"ticket_classifier.png\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "keras.utils.plot_model(modelF, \"ticket_classifier_with_shape_info.png\", show_shapes=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's draw in the class this model as a NN in the graphical perspective "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Multi "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let’s say you’re building a system to rank customer support tickets by priority and route them to the appropriate department. Your model has three inputs:\n",
    "- The title of the ticket (text input)\n",
    "- The text body of the ticket (text input)\n",
    "- Any tags added by the user (categorical input, assumed here to be one-hot encoded)\n",
    "\n",
    "We can encode the text inputs as arrays of ones and zeros of size vocabulary_size (see chapter 11 for detailed information about text encoding techniques). Your model also has two outputs:\n",
    "- The priority score of the ticket, a scalar between 0 and 1 (sigmoid output)\n",
    "- The department that should handle the ticket (a softmax over the set of departments)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a new model by reusing intermediate layer outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1.3 \n",
    "\n",
    "The total number of diferent letters in all text features is 1000, the possible categorical input has 100 classes and there are 4 departments in this customer support ticket system.\n",
    "\n",
    "What are the following values of these variables:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "vocabulary_size = <your-answer>\n",
    "num_tags = <your-answer>\n",
    "num_departments = <your-answer>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# defining the multi NN with the functional pattern in keras\n",
    "\n",
    "# inputs\n",
    "title = keras.Input(shape=(None,vocabulary_size), name=\"title\")\n",
    "text_body = keras.Input(shape=(None,vocabulary_size), name=\"text_body\")\n",
    "tags = keras.Input(shape=(None,num_tags), name=\"tags\")\n",
    "# hidden layers\n",
    "features = layers.Concatenate()([title, text_body, tags])\n",
    "features = layers.Dense(64, activation=\"relu\")(features)\n",
    "# output\n",
    "priority = layers.Dense(1, activation=\"sigmoid\", name=\"priority\")(features)\n",
    "department = layers.Dense(\n",
    "    num_departments, activation=\"softmax\", name=\"department\")(features)\n",
    "# model\n",
    "model = keras.Model(inputs=[title, text_body, tags], outputs=[priority, department])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create random data\n",
    "\n",
    "num_samples = 128\n",
    "\n",
    "title_data = np.random.randint(0, 2, size=(num_samples, vocabulary_size))\n",
    "text_body_data = np.random.randint(0, 2, size=(num_samples, vocabulary_size))\n",
    "tags_data = np.random.randint(0, 2, size=(num_samples, num_tags))\n",
    "\n",
    "priority_data = np.random.random(size=(num_samples, 1))\n",
    "department_data = np.random.randint(0, 2, size=(num_samples, num_departments))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#investigate the input data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1.4 \n",
    "\n",
    "Choose the right metric for the department label"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(optimizer=\"rmsprop\",\n",
    "              loss=[\"mean_squared_error\", \"categorical_crossentropy\"],\n",
    "              metrics=[[\"mean_absolute_error\"], [<your-answer>]])\n",
    "model.fit([title_data, text_body_data, tags_data],\n",
    "          [priority_data, department_data],\n",
    "          epochs=1)\n",
    "model.evaluate([title_data, text_body_data, tags_data],\n",
    "               [priority_data, department_data])\n",
    "priority_preds, department_preds = model.predict([title_data, text_body_data, tags_data])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "keras.utils.plot_model(model, \"ticket_classifier.png\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The “None” in the tensor shapes represents the batch size: this model allows batches of any size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "keras.utils.plot_model(model, \"ticket_classifier_with_shape_info.png\", show_shapes=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1.5\n",
    "\n",
    "See the input and output of the 4th layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "<your-answer>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using middle layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#using previous layers\n",
    "features = model.layers[4].output\n",
    "\n",
    "#adding another output\n",
    "difficulty = layers.Dense(3, activation=\"softmax\", name=\"difficulty\")(features)\n",
    "\n",
    "new_model = keras.Model(\n",
    "    inputs=[title, text_body, tags],\n",
    "    outputs=[priority, department, difficulty])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "keras.utils.plot_model(new_model, \"updated_ticket_classifier.png\", show_shapes=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Writing own metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The standard workflow: compile(), fit(), evaluate(), predict()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.datasets import mnist\n",
    "\n",
    "def get_mnist_model():\n",
    "    inputs = keras.Input(shape=(28 * 28,))\n",
    "    features = layers.Dense(512, activation=\"relu\")(inputs)\n",
    "    features = layers.Dropout(0.5)(features)\n",
    "    outputs = layers.Dense(10, activation=\"softmax\")(features)\n",
    "    model = keras.Model(inputs, outputs)\n",
    "    return model\n",
    "\n",
    "(images, labels), (test_images, test_labels) = mnist.load_data()\n",
    "images = images.reshape((60000, 28 * 28)).astype(\"float32\") / 255\n",
    "test_images = test_images.reshape((10000, 28 * 28)).astype(\"float32\") / 255\n",
    "train_images, val_images = images[10000:], images[:10000]\n",
    "train_labels, val_labels = labels[10000:], labels[:10000]\n",
    "\n",
    "model = get_mnist_model()\n",
    "model.compile(optimizer=\"rmsprop\",\n",
    "              loss=\"sparse_categorical_crossentropy\",\n",
    "              metrics=[\"accuracy\"])\n",
    "model.fit(train_images, train_labels,\n",
    "          epochs=3,\n",
    "          validation_data=(val_images, val_labels))\n",
    "test_metrics = model.evaluate(test_images, test_labels)\n",
    "predictions = model.predict(test_images)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "class RootMeanSquaredError(keras.metrics.Metric):\n",
    "\n",
    "    def __init__(self, name=\"rmse\", **kwargs):\n",
    "        super().__init__(name=name, **kwargs)\n",
    "        self.mse_sum = self.add_weight(name=\"mse_sum\", initializer=\"zeros\")\n",
    "        self.total_samples = self.add_weight(\n",
    "            name=\"total_samples\", initializer=\"zeros\", dtype=\"int32\")\n",
    "\n",
    "    def update_state(self, y_true, y_pred, sample_weight=None):\n",
    "        y_true = tf.one_hot(y_true, depth=tf.shape(y_pred)[1])\n",
    "        mse = tf.reduce_sum(tf.square(y_true - y_pred))\n",
    "        self.mse_sum.assign_add(mse)\n",
    "        num_samples = tf.shape(y_pred)[0]\n",
    "        self.total_samples.assign_add(num_samples)\n",
    "\n",
    "    def result(self):\n",
    "        return tf.sqrt(self.mse_sum / tf.cast(self.total_samples, tf.float32))\n",
    "\n",
    "    def reset_state(self):\n",
    "        self.mse_sum.assign(0.)\n",
    "        self.total_samples.assign(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = get_mnist_model()\n",
    "model.compile(optimizer=\"rmsprop\",\n",
    "              loss=\"sparse_categorical_crossentropy\",\n",
    "              metrics=[\"accuracy\", RootMeanSquaredError()])\n",
    "model.fit(train_images, train_labels,\n",
    "          epochs=3,\n",
    "          validation_data=(val_images, val_labels))\n",
    "test_metrics = model.evaluate(test_images, test_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Callbacks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "callbacks_list = [\n",
    "    keras.callbacks.EarlyStopping(\n",
    "        monitor=\"val_accuracy\",\n",
    "        patience=2,\n",
    "    ),\n",
    "    keras.callbacks.ModelCheckpoint(\n",
    "        filepath=\"checkpoint_path.keras\",\n",
    "        monitor=\"val_loss\",\n",
    "        save_best_only=True,\n",
    "    )\n",
    "]\n",
    "model = get_mnist_model()\n",
    "model.compile(optimizer=\"rmsprop\",\n",
    "              loss=\"sparse_categorical_crossentropy\",\n",
    "              metrics=[\"accuracy\"])\n",
    "model.fit(train_images, train_labels,\n",
    "          epochs=10,\n",
    "          callbacks=callbacks_list,\n",
    "          validation_data=(val_images, val_labels))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.load_model(\"checkpoint_path.keras\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "custom callbacks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from matplotlib import pyplot as plt\n",
    "\n",
    "class LossHistory(keras.callbacks.Callback):\n",
    "    def on_train_begin(self, logs):\n",
    "        self.per_batch_losses = []\n",
    "\n",
    "    def on_batch_end(self, batch, logs):\n",
    "        self.per_batch_losses.append(logs.get(\"loss\"))\n",
    "\n",
    "    def on_epoch_end(self, epoch, logs):\n",
    "        plt.clf()\n",
    "        plt.plot(range(len(self.per_batch_losses)), self.per_batch_losses,\n",
    "                 label=\"Training loss for each batch\")\n",
    "        plt.xlabel(f\"Batch (epoch {epoch})\")\n",
    "        plt.ylabel(\"Loss\")\n",
    "        plt.legend()\n",
    "        plt.savefig(f\"plot_at_epoch_{epoch}\")\n",
    "        self.per_batch_losses = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = get_mnist_model()\n",
    "model.compile(optimizer=\"rmsprop\",\n",
    "              loss=\"sparse_categorical_crossentropy\",\n",
    "              metrics=[\"accuracy\"])\n",
    "model.fit(train_images, train_labels,\n",
    "          epochs=10,\n",
    "          callbacks=[LossHistory()],\n",
    "          validation_data=(val_images, val_labels))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TensorBoard"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = get_mnist_model()\n",
    "model.compile(optimizer=\"rmsprop\",\n",
    "              loss=\"sparse_categorical_crossentropy\",\n",
    "              metrics=[\"accuracy\"])\n",
    "\n",
    "tensorboard = keras.callbacks.TensorBoard(\n",
    "    log_dir=\"/full_path_to_your_log_dir\",\n",
    ")\n",
    "model.fit(train_images, train_labels,\n",
    "          epochs=10,\n",
    "          validation_data=(val_images, val_labels),\n",
    "          callbacks=[tensorboard])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext tensorboard\n",
    "%tensorboard --logdir /full_path_to_your_log_dir"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9.13",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "d33a1388bb95f6f47390937b44de6829ea073fadb3ed2a95296fdc5e4fdfba80"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}