t81_558_class_04_1_kfold.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/app_deep_learning/blob/main/t81_558_class_04_1_kfold.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "\n",
    "**Module 4: Training for Tabular Data**\n",
    "\n",
    "- Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "- For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 4 Material\n",
    "\n",
    "- **Part 4.1: Using K-Fold Cross-validation with PyTorch** [[Video]](https://www.youtube.com/watch?v=Q8ZQNvZwsNE&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/app_deep_learning/blob/main/t81_558_class_04_1_kfold.ipynb)\n",
    "- Part 4.2: Training Schedules for PyTorch  [[Video]](https://www.youtube.com/watch?v=lMMlbmfvKDQ&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/app_deep_learning/blob/main/t81_558_class_04_2_schedule.ipynb)\n",
    "- Part 4.3: Dropout Regularization [[Video]](https://www.youtube.com/watch?v=4ixjgw6Q42U&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/app_deep_learning/blob/main/t81_558_class_04_3_dropout.ipynb)\n",
    "- Part 4.4: Batch Normalization [[Video]](https://www.youtube.com/watch?v=1U5nOKh9OLQ&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/app_deep_learning/blob/main/t81_558_class_04_4_batch_norm.ipynb)\n",
    "- Part 4.5: RAPIDS for Tabular Data [[Video]](https://www.youtube.com/watch?v=KgoXuhG_kfs&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/app_deep_learning/blob/main/t81_558_class_04_5_rapids.ipynb)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Google CoLab Instructions\n",
    "\n",
    "The following code ensures that Google CoLab is running and maps Google Drive if needed. We also initialize the PyTorch device to either GPU/MPS (if available) or CPU.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: not using Google CoLab\n",
      "Using device: mps\n"
     ]
    }
   ],
   "source": [
    "import copy\n",
    "import torch\n",
    "\n",
    "try:\n",
    "    import google.colab\n",
    "\n",
    "    COLAB = True\n",
    "    print(\"Note: using Google CoLab\")\n",
    "except:\n",
    "    print(\"Note: not using Google CoLab\")\n",
    "    COLAB = False\n",
    "\n",
    "# Make use of a GPU or MPS (Apple) if one is available.  (see module 3.2)\n",
    "import torch\n",
    "has_mps = torch.backends.mps.is_built()\n",
    "device = \"mps\" if has_mps else \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
    "print(f\"Using device: {device}\")\n",
    "\n",
    "\n",
    "# Early stopping (see module 3.4)\n",
    "class EarlyStopping:\n",
    "    def __init__(self, patience=5, min_delta=0, restore_best_weights=True):\n",
    "        self.patience = patience\n",
    "        self.min_delta = min_delta\n",
    "        self.restore_best_weights = restore_best_weights\n",
    "        self.best_model = None\n",
    "        self.best_loss = None\n",
    "        self.counter = 0\n",
    "        self.status = \"\"\n",
    "\n",
    "    def __call__(self, model, val_loss):\n",
    "        if self.best_loss is None:\n",
    "            self.best_loss = val_loss\n",
    "            self.best_model = copy.deepcopy(model.state_dict())\n",
    "        elif self.best_loss - val_loss >= self.min_delta:\n",
    "            self.best_model = copy.deepcopy(model.state_dict())\n",
    "            self.best_loss = val_loss\n",
    "            self.counter = 0\n",
    "            self.status = f\"Improvement found, counter reset to {self.counter}\"\n",
    "        else:\n",
    "            self.counter += 1\n",
    "            self.status = f\"No improvement in the last {self.counter} epochs\"\n",
    "            if self.counter >= self.patience:\n",
    "                self.status = f\"Early stopping triggered after {self.counter} epochs.\"\n",
    "                if self.restore_best_weights:\n",
    "                    model.load_state_dict(self.best_model)\n",
    "                return True\n",
    "        return False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 4.1: Using K-Fold Cross-validation with PyTorch\n",
    "\n",
    "You can use cross-validation for a variety of purposes in predictive modeling:\n",
    "\n",
    "- Generating out-of-sample predictions from a neural network\n",
    "- Estimate a good number of epochs to train a neural network for (early stopping)\n",
    "- Evaluate the effectiveness of certain hyperparameters, such as activation functions, neuron counts, and layer counts\n",
    "\n",
    "Cross-validation uses several folds and multiple models to provide each data segment a chance to serve as both the validation and training set. Figure 5.CROSS shows cross-validation.\n",
    "\n",
    "**Figure 5.CROSS: K-Fold Crossvalidation**\n",
    "![K-Fold Crossvalidation](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_1_kfold.png \"K-Fold Crossvalidation\")\n",
    "\n",
    "It is important to note that each fold will have one model (neural network). To generate predictions for new data (not present in the training set), predictions from the fold models can be handled in several ways:\n",
    "\n",
    "- Choose the model with the highest validation score as the final model.\n",
    "- Preset new data to the five models (one for each fold) and average the result (this is an [ensemble](https://en.wikipedia.org/wiki/Ensemble_learning)).\n",
    "- Retrain a new model (using the same settings as the cross-validation) on the entire dataset. Train for as many epochs and with the same hidden layer structure.\n",
    "\n",
    "Generally, I prefer the last approach and will retrain a model on the entire data set once I have selected hyper-parameters. Of course, I will always set aside a final holdout set for model validation that I do not use in any aspect of the training process.\n",
    "\n",
    "## Regression vs Classification K-Fold Cross-Validation\n",
    "\n",
    "Regression and classification are handled somewhat differently concerning cross-validation. Regression is the simpler case where you can break up the data set into K folds with little regard for where each item lands. For regression, the data items should fall into the folds as randomly as possible. It is also important to remember that not every fold will necessarily have the same number of data items. It is not always possible for the data set to be evenly divided into K folds. For regression cross-validation, we will use the Scikit-Learn class **KFold**.\n",
    "\n",
    "Cross-validation for classification could also use the **KFold** object; however, this technique would not ensure that the class balance remains the same in each fold as in the original. The balance of classes that a model was trained on must remain the same (or similar) to the training set. Drift in this distribution is one of the most important things to monitor after a trained model has been placed into actual use. Because of this, we want to make sure that the cross-validation itself does not introduce an unintended shift. This technique is called stratified sampling and is accomplished by using the Scikit-Learn object **StratifiedKFold** in place of **KFold** whenever you use classification. In summary, you should use the following two objects in Scikit-Learn:\n",
    "\n",
    "- **KFold** When dealing with a regression problem.\n",
    "- **StratifiedKFold** When dealing with a classification problem.\n",
    "\n",
    "The following two sections demonstrate cross-validation with classification and regression.\n",
    "\n",
    "## Out-of-Sample Regression Predictions with K-Fold Cross-Validation\n",
    "\n",
    "The following code trains the simple dataset using a 5-fold cross-validation. The expected performance of a neural network of the type trained here would be the score for the generated out-of-sample predictions. We begin by preparing a feature vector using the **jh-simple-dataset** to predict age. This model is set up as a regression problem.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from scipy.stats import zscore\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# Read the data set\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/jh-simple-dataset.csv\",\n",
    "    na_values=['NA','?'])\n",
    "\n",
    "# Generate dummies for job\n",
    "df = pd.concat([df,pd.get_dummies(df['job'],prefix=\"job\",dtype=int)],axis=1)\n",
    "df.drop('job', axis=1, inplace=True)\n",
    "\n",
    "# Generate dummies for area\n",
    "df = pd.concat([df,pd.get_dummies(df['area'],prefix=\"area\",dtype=int)],axis=1)\n",
    "df.drop('area', axis=1, inplace=True)\n",
    "\n",
    "# Generate dummies for product\n",
    "df = pd.concat([df,pd.get_dummies(df['product'],prefix=\"product\",dtype=int)],axis=1)\n",
    "df.drop('product', axis=1, inplace=True)\n",
    "\n",
    "# Missing values for income\n",
    "med = df['income'].median()\n",
    "df['income'] = df['income'].fillna(med)\n",
    "\n",
    "# Standardize ranges\n",
    "df['income'] = zscore(df['income'])\n",
    "df['aspect'] = zscore(df['aspect'])\n",
    "df['save_rate'] = zscore(df['save_rate'])\n",
    "df['subscriptions'] = zscore(df['subscriptions'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that the feature vector is created a 5-fold cross-validation can be performed to generate out-of-sample predictions. We will assume 500 epochs and not use early stopping. Later we will see how we can estimate a more optimal epoch count.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fold #1\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/jeff/miniconda3/envs/torch/lib/python3.9/site-packages/torch/autograd/__init__.py:394: UserWarning: Error detected in LinearBackward0. Traceback of forward call that caused the error:\n",
      "  File \"/Users/jeff/miniconda3/envs/torch/lib/python3.9/site-packages/torch/nn/modules/container.py\", line 215, in forward\n",
      "    input = module(input)\n",
      " (Triggered internally at /Users/runner/work/_temp/anaconda/conda-bld/pytorch_1702400227158/work/torch/csrc/autograd/python_anomaly_mode.cpp:119.)\n",
      "  result = Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass\n",
      "[2024-01-07 07:01:12,347] [0/2] torch._dynamo.exc: [WARNING] Backend compiler failed with a fake tensor exception at \n",
      "[2024-01-07 07:01:12,347] [0/2] torch._dynamo.exc: [WARNING]   File \"/Users/jeff/miniconda3/envs/torch/lib/python3.9/site-packages/torch/nn/modules/container.py\", line 216, in forward\n",
      "[2024-01-07 07:01:12,347] [0/2] torch._dynamo.exc: [WARNING]     return input\n",
      "[2024-01-07 07:01:12,347] [0/2] torch._dynamo.exc: [WARNING] Adding a graph break.\n",
      "/Users/jeff/miniconda3/envs/torch/lib/python3.9/site-packages/torch/autograd/__init__.py:394: UserWarning: Error detected in LinearBackward0. Traceback of forward call that caused the error:\n",
      "  File \"/Users/jeff/miniconda3/envs/torch/lib/python3.9/site-packages/torch/nn/modules/container.py\", line 215, in forward\n",
      "    input = module(input)\n",
      " (Triggered internally at /Users/runner/work/_temp/anaconda/conda-bld/pytorch_1702400227158/work/torch/csrc/autograd/python_anomaly_mode.cpp:119.)\n",
      "  result = Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass\n",
      "[2024-01-07 07:01:12,377] [0/2] torch._dynamo.exc: [WARNING] Backend compiler failed with a fake tensor exception at \n",
      "[2024-01-07 07:01:12,377] [0/2] torch._dynamo.exc: [WARNING]   File \"/Users/jeff/miniconda3/envs/torch/lib/python3.9/site-packages/torch/nn/modules/container.py\", line 216, in forward\n",
      "[2024-01-07 07:01:12,377] [0/2] torch._dynamo.exc: [WARNING]     return input\n",
      "[2024-01-07 07:01:12,377] [0/2] torch._dynamo.exc: [WARNING] Adding a graph break.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 157/500, Validation Loss: 0.7110840678215027, Early stopping triggered after 5 epochs.\n",
      "Fold #2\n",
      "Epoch 149/500, Validation Loss: 0.4980887770652771, Early stopping triggered after 5 epochs.\n",
      "Fold #3\n",
      "Epoch 151/500, Validation Loss: 0.7315194010734558, Early stopping triggered after 5 epochs.\n",
      "Fold #4\n",
      "Epoch 191/500, Validation Loss: 0.42675867676734924, Early stopping triggered after 5 epochs.\n",
      "Fold #5\n",
      "Epoch 139/500, Validation Loss: 1.247514009475708, Early stopping triggered after 5 epochs.\n",
      "Fold score (RMSE): 1.1104768514633179\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "from torch.utils.data import DataLoader, TensorDataset\n",
    "from sklearn.model_selection import KFold\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "# Convert to PyTorch Tensors\n",
    "x_columns = df.columns.drop(['age', 'id'])\n",
    "x = torch.tensor(df[x_columns].values, dtype=torch.float32, device=device)\n",
    "y = torch.tensor(df['age'].values, dtype=torch.float32, device=device).view(-1, 1)\n",
    "\n",
    "# Set random seed for reproducibility\n",
    "torch.manual_seed(42)\n",
    "\n",
    "# Cross-Validate\n",
    "kf = KFold(n_splits=5, shuffle=True, random_state=42)\n",
    "\n",
    "# Early stopping parameters\n",
    "patience = 10\n",
    "\n",
    "fold = 0\n",
    "for train_idx, test_idx in kf.split(x):\n",
    "    fold += 1\n",
    "    print(f\"Fold #{fold}\")\n",
    "\n",
    "    x_train, x_test = x[train_idx], x[test_idx]\n",
    "    y_train, y_test = y[train_idx], y[test_idx]\n",
    "\n",
    "    # PyTorch DataLoader\n",
    "    train_dataset = TensorDataset(x_train, y_train)\n",
    "    train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True)\n",
    "\n",
    "    # Create the model and optimizer\n",
    "    model = nn.Sequential(\n",
    "        nn.Linear(x.shape[1], 20),\n",
    "        nn.ReLU(),\n",
    "        nn.Linear(20, 10),\n",
    "        nn.ReLU(),\n",
    "        nn.Linear(10, 1)\n",
    "    )\n",
    "    model = torch.compile(model,backend=\"aot_eager\").to(device)\n",
    "\n",
    "\n",
    "    optimizer = optim.Adam(model.parameters(), lr=0.001)\n",
    "    loss_fn = nn.MSELoss()\n",
    "\n",
    "    # Early Stopping variables\n",
    "    best_loss = float('inf')\n",
    "    early_stopping_counter = 0\n",
    "\n",
    "    # Training loop\n",
    "    EPOCHS = 500\n",
    "    epoch = 0\n",
    "    done = False\n",
    "    es = EarlyStopping()\n",
    "\n",
    "    while not done and epoch<EPOCHS:\n",
    "        epoch += 1\n",
    "        model.train()\n",
    "        for x_batch, y_batch in train_loader:\n",
    "            optimizer.zero_grad()\n",
    "            output = model(x_batch)\n",
    "            loss = loss_fn(output, y_batch)\n",
    "            loss.backward()\n",
    "            optimizer.step()\n",
    "\n",
    "        # Validation\n",
    "        model.eval()\n",
    "        with torch.no_grad():\n",
    "            val_output = model(x_test)\n",
    "            val_loss = loss_fn(val_output, y_test)\n",
    "\n",
    "        if es(model, val_loss):\n",
    "            done = True\n",
    "\n",
    "    print(f\"Epoch {epoch}/{EPOCHS}, Validation Loss: \"\n",
    "      f\"{val_loss.item()}, {es.status}\")\n",
    "\n",
    "# Final evaluation\n",
    "model.eval()\n",
    "with torch.no_grad():\n",
    "    oos_pred = model(x_test)\n",
    "score = torch.sqrt(loss_fn(oos_pred, y_test)).item()\n",
    "print(f\"Fold score (RMSE): {score}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, the above code also reports the average number of epochs needed. A common technique is to then train on the entire dataset for the average number of epochs required.\n",
    "\n",
    "## Classification with Stratified K-Fold Cross-Validation\n",
    "\n",
    "The following code trains and fits the **jh**-simple-dataset dataset with cross-validation to generate out-of-sample. It also writes the out-of-sample (predictions on the test set) results.\n",
    "\n",
    "It is good to perform stratified k-fold cross-validation with classification data. This technique ensures that the percentages of each class remain the same across all folds. Use the **StratifiedKFold** object instead of the **KFold** object used in the regression.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read the data set\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/jh-simple-dataset.csv\",\n",
    "    na_values=[\"NA\", \"?\"],\n",
    ")\n",
    "\n",
    "# Generate dummies for job\n",
    "df = pd.concat([df, pd.get_dummies(df[\"job\"], prefix=\"job\",dtype=int)], axis=1)\n",
    "df.drop(\"job\", axis=1, inplace=True)\n",
    "\n",
    "# Generate dummies for area\n",
    "df = pd.concat([df, pd.get_dummies(df[\"area\"], prefix=\"area\",dtype=int)], axis=1)\n",
    "df.drop(\"area\", axis=1, inplace=True)\n",
    "\n",
    "# Missing values for income\n",
    "med = df[\"income\"].median()\n",
    "df[\"income\"] = df[\"income\"].fillna(med)\n",
    "\n",
    "# Standardize ranges\n",
    "df[\"income\"] = zscore(df[\"income\"])\n",
    "df[\"aspect\"] = zscore(df[\"aspect\"])\n",
    "df[\"save_rate\"] = zscore(df[\"save_rate\"])\n",
    "df[\"age\"] = zscore(df[\"age\"])\n",
    "df[\"subscriptions\"] = zscore(df[\"subscriptions\"])\n",
    "\n",
    "# Convert to numpy - Classification\n",
    "x_columns = df.columns.drop(\"product\").drop(\"id\")\n",
    "x = df[x_columns].values\n",
    "dummies = pd.get_dummies(df[\"product\"],dtype=int)  # Classification\n",
    "products = dummies.columns\n",
    "y = dummies.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now loop through the five folds and use the validation data in each fold for early stopping. We also keep the validated predictions to build a complete set of \"out of sample\" predictions. These \"out of sample\" predictions allow us to have predictions across the entire dataset that were not in the training data. It is important to note that this separation is not 100% pure in that the validation set was used for early stopping. This small crossover is a tradeoff that allows us to use a large amount of training data in each fold."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fold #1\n",
      "Epoch 395/500, Validation Loss: 1.4793496131896973, Early stopping triggered after 5 epochs.\n",
      "Fold score (accuracy): 0.69\n",
      "Fold #2\n",
      "Epoch 6/500, Validation Loss: 1.4524176120758057, Early stopping triggered after 5 epochs.\n",
      "Fold score (accuracy): 0.73\n",
      "Fold #3\n",
      "Epoch 6/500, Validation Loss: 1.4165765047073364, Early stopping triggered after 5 epochs.\n",
      "Fold score (accuracy): 0.765\n",
      "Fold #4\n",
      "Epoch 6/500, Validation Loss: 1.4107660055160522, Early stopping triggered after 5 epochs.\n",
      "Fold score (accuracy): 0.7625\n",
      "Fold #5\n",
      "Epoch 7/500, Validation Loss: 1.4483850002288818, Early stopping triggered after 5 epochs.\n",
      "Fold score (accuracy): 0.7275\n",
      "Final score (accuracy): 0.735\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "from sklearn import metrics\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "\n",
    "# Assuming your data is in Numpy Arrays. If not, convert them into Numpy Arrays\n",
    "x = np.array(x)\n",
    "y = np.array(y)\n",
    "\n",
    "# Use the nn.Sequential API\n",
    "model = nn.Sequential(\n",
    "    nn.Linear(x.shape[1], 50),\n",
    "    nn.ReLU(),\n",
    "    nn.Linear(50, 25),\n",
    "    nn.ReLU(),\n",
    "    nn.Linear(25, y.shape[1]),\n",
    "    nn.Softmax(dim=1),\n",
    ")\n",
    "model = torch.compile(model,backend=\"aot_eager\").to(device)\n",
    "\n",
    "# Cross-validate\n",
    "kf = StratifiedKFold(5, shuffle=True, random_state=42)\n",
    "\n",
    "# Defining Loss and optimizer\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "optimizer = optim.Adam(model.parameters())\n",
    "\n",
    "oos_y = []\n",
    "oos_pred = []\n",
    "fold = 0\n",
    "\n",
    "for train, test in kf.split(x, df[\"product\"]):\n",
    "    fold += 1\n",
    "    print(f\"Fold #{fold}\")\n",
    "\n",
    "    x_train = torch.tensor(x[train], device=device, dtype=torch.float32)\n",
    "    y_train = torch.tensor(np.argmax(y[train], axis=1),device=device, dtype=torch.long)  # Convert to class indices\n",
    "    x_test = torch.tensor(x[test],device=device, dtype=torch.float32)\n",
    "    y_test = torch.tensor(np.argmax(y[test], axis=1),device=device, dtype=torch.long)  # Convert to class indices\n",
    "\n",
    "    # Training loop\n",
    "    EPOCHS = 500\n",
    "    epoch = 0\n",
    "    done = False\n",
    "    es = EarlyStopping(restore_best_weights=True)\n",
    "\n",
    "    while not done and epoch < EPOCHS:\n",
    "        epoch += 1\n",
    "        model.train()\n",
    "        optimizer.zero_grad()\n",
    "        output = model(x_train)\n",
    "        loss = criterion(output, y_train)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "\n",
    "        # Evaluate validation loss\n",
    "        model.eval()\n",
    "        with torch.no_grad():\n",
    "            y_val = model(x_test)\n",
    "            val_loss = criterion(y_val, y_test)\n",
    "\n",
    "        if es(model, val_loss):\n",
    "            done = True\n",
    "\n",
    "    # Prediction\n",
    "    with torch.no_grad():\n",
    "        y_val = model(x_test)\n",
    "        _, pred = torch.max(y_val, 1)\n",
    "\n",
    "    oos_y.append(y_test.cpu().numpy())\n",
    "    oos_pred.append(pred.cpu().numpy())\n",
    "\n",
    "    print(\n",
    "        f\"Epoch {epoch}/{EPOCHS}, Validation Loss: \" f\"{val_loss.item()}, {es.status}\"\n",
    "    )\n",
    "\n",
    "    # Measure this fold's accuracy\n",
    "    score = metrics.accuracy_score(y_test.cpu().numpy(), pred.cpu().numpy())\n",
    "    print(f\"Fold score (accuracy): {score}\")\n",
    "\n",
    "# Build the oos prediction list and calculate the error.\n",
    "oos_y = np.concatenate(oos_y)\n",
    "oos_pred = np.concatenate(oos_pred)\n",
    "\n",
    "score = metrics.accuracy_score(oos_y, oos_pred)\n",
    "print(f\"Final score (accuracy): {score}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 4 Assignment\n",
    "\n",
    "You can find the third assignment here: [assignment 4](https://github.com/jeffheaton/app_deep_learning/blob/main/assignments/assignment_yourname_t81_558_class4.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.11 (torch)",
   "language": "python",
   "name": "pytorch"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}