NameWeave - Multi Layer Perceptron.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "908f6d13-f8ae-4cc6-b33c-ccc67424b11f",
   "metadata": {},
   "source": [
    "# Welcome to NameWeave - Multi Layer Perceptron Approach"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9abcd4f7-a35a-42e6-96e4-061a8ec1d627",
   "metadata": {},
   "source": [
    "Like our original <a href=\"https://github.com/AvishakeAdhikary/Neural-Networks-From-Scratch/blob/main/NameWeave.ipynb\">NameWeave</a>,\\\n",
    "We will try to create a **Multi Layer Perceptron** to build a character level language model and predict names based on it."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22300d7d-798b-4837-9c4b-4264461cdb89",
   "metadata": {},
   "source": [
    "To Approach this model, we will follow an approach based on the paper,\\\n",
    "<a href=\"https://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf\">A Neural Probabilistic Language Model</a>,\\\n",
    "Which is a **word level language model** but solves the similar problem of predicting words...\\\n",
    "This paper is 19 pages long, and we don't have time to read the entire paper,\\\n",
    "But I invite you to read it."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "275e0abc-ea74-4034-be72-2a30002f1368",
   "metadata": {},
   "source": [
    "In this paper,\\\n",
    "They used a word vocabulary of 17000."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55a68db0-b3b3-4403-ad33-14e359b7fdab",
   "metadata": {},
   "source": [
    "![Word Vocabulary](ExplanationMedia/Images/Vocabulary.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5d33363f-56fc-4dc4-833c-d3b5f00a0a0b",
   "metadata": {},
   "source": [
    "They then converted this vocabulary into a 30 dimensional feature space"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38b8913c-97a4-4a7c-90d7-c2955bfeab19",
   "metadata": {},
   "source": [
    "![Vocabulary to Feature Space](ExplanationMedia/Images/VocabularytoFeatureSpace.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9484bf79-78db-4c4b-9316-5a6cc0dcd172",
   "metadata": {},
   "source": [
    "This is a very small space for a very large dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "348c5d75-0090-491f-ae03-69248b1d7b01",
   "metadata": {},
   "source": [
    "The approach of this paper is also very similar because,\\\n",
    "They used a **multilayer neural network** to **predict the next word given the previous ones**,\\\n",
    "& they **maximize the log-likelihood** of the training data or a regularized criterion."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68c148c6-4349-4a99-834c-af86cf5bb319",
   "metadata": {},
   "source": [
    "#### Why does this approach work? Let's take a concrete example"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6fc5c121-0e85-45c2-9ed2-2097ec1d555b",
   "metadata": {},
   "source": [
    "We have a phrase: *A dog was running in a room*, *The cat is walking in the bedroom*\n",
    "\n",
    "During the training of the network, the words move around to a similar corner of the space based on their features\\\n",
    "So, even if the model goes *out of distribution* during test, making predictions,\\\n",
    "The similar words which have never occured before may occur here.\n",
    "\n",
    "Resulting in the phrase: *A dog is walking in a bedroom*, *The cat is running in a room*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d6dfa77-e029-4a49-a726-501a666a5f8f",
   "metadata": {},
   "source": [
    "If we knew that dog and cat played similar roles (semantically and syntactically), and similarly for (the,a), (bedroom,room), (is,was), we could naturally transfer probability mass."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fcad7ecb-f523-4645-a6ac-803520992a4b",
   "metadata": {},
   "source": [
    "Let's now look at the Neural Network for this Approach\n",
    "\n",
    "![A Neural Probabilistic Language Model - Neural Network](https://miro.medium.com/v2/resize:fit:1200/1*EqKiy4-6tuLSoPP_kub33Q.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f53d53d7-4ee8-4724-ace1-15d922c91ed3",
   "metadata": {},
   "source": [
    "In this network,\\\n",
    "They are taking *3 previous words* and are trying to *predict the 4-th word in a sequence*.\n",
    "\n",
    "Now because they had the vocabulary of 17000 words (**'w'**),\\\n",
    "These previous words are the indexes ranging from 0-16999.\n",
    "\n",
    "There is also a lookup-table which they call **'C'** \\\n",
    "This is their lookup-embedding-matrix, which is shared among all the words\\\n",
    "This C is a matrix of say 17000x30. (So, number of words in vocabulary by number of dimensions in the feature space).\n",
    "\n",
    "So what this is essentially doing is,\\\n",
    "They are trying to pick out the row based on the index of the word from vocabulary\\\n",
    "And the row represents the 1x30 vector of the word's embedding.\\\n",
    "So they are using the same matrix over and over to look for their own vector of embedding.\n",
    "\n",
    "So, because they are taking *3 previous words*, and each vector uses 1x30 dimensions,\\\n",
    "They have 3x30 dimensions making up 90 dimensions in total.\n",
    "\n",
    "Next up is the hidden layer (tanh non-linearity layer).\\\n",
    "This is layer has the *hyper-parameter* (*hyper-parameter* is a parameter of the neural network, that is the designer' choice of the neural network).\\\n",
    "So, this layer's size can be as large as we'd like or as we'd like.\\\n",
    "So, we are going to go over multiple choices, and we are going to evaluage how good they work.\\\n",
    "Note: This layer will be fully connected to all the vector embeddings of the previous layer (90 dimensions).\n",
    "\n",
    "Next, they have a output layer of logits (you can refer to the original <a href=\"https://github.com/AvishakeAdhikary/Neural-Networks-From-Scratch/blob/main/NameWeave.ipynb\">NameWeave</a>  for reference).\\\n",
    "Now, because they had a vocabulary of **17000 words**, this layer has **17000 neurons** which is **fully connected to the hidden layer**.\\\n",
    "Resulting in the *maximum computation between the hidden layer and output layer*.\n",
    "\n",
    "This output layer is then having a softmax activation layer, which exponentiates the logits and normalized to sum to 1.\\\n",
    "Which results in a nice probability distribution for the next *4-th word in a sequence*.\n",
    "\n",
    "<hr>\n",
    "\n",
    "During training we have the label (identity of the next word in a sequence).\n",
    "That word's index is used to choose the probability of that word,\\\n",
    "And then they maximize the probability of that word, with respect to the parameters of this neural network.\n",
    "\n",
    "<hr>\n",
    "\n",
    "Parameters:\n",
    "1. The *weights and biases* of the *output layer*\n",
    "2. The *weights and biases* of the *hidden layer*\n",
    "3. The *embedding look-up table 'C'*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17a9e483-f162-4ca7-8ee5-70aecc3b0dd3",
   "metadata": {},
   "source": [
    "So, Let's implement our own neural network, based on the above approach."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7dea5c56-a624-4206-8e00-c719c229ac73",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "# Installing Dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ca8ad30a-3f8e-4c74-babc-f77e59c79180",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: torch in c:\\users\\avhis\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (2.1.2)\n",
      "Requirement already satisfied: filelock in c:\\users\\avhis\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from torch) (3.13.1)\n",
      "Requirement already satisfied: typing-extensions in c:\\users\\avhis\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from torch) (4.9.0)\n",
      "Requirement already satisfied: sympy in c:\\users\\avhis\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from torch) (1.12)\n",
      "Requirement already satisfied: networkx in c:\\users\\avhis\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from torch) (3.2.1)\n",
      "Requirement already satisfied: jinja2 in c:\\users\\avhis\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from torch) (3.1.3)\n",
      "Requirement already satisfied: fsspec in c:\\users\\avhis\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from torch) (2023.12.2)\n",
      "Requirement already satisfied: MarkupSafe>=2.0 in c:\\users\\avhis\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from jinja2->torch) (2.1.4)\n",
      "Requirement already satisfied: mpmath>=0.19 in c:\\users\\avhis\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from sympy->torch) (1.3.0)\n",
      "Requirement already satisfied: numpy in c:\\users\\avhis\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (1.26.3)\n",
      "Requirement already satisfied: pandas in c:\\users\\avhis\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (2.2.0)\n",
      "Requirement already satisfied: numpy<2,>=1.23.2 in c:\\users\\avhis\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from pandas) (1.26.3)\n",
      "Requirement already satisfied: python-dateutil>=2.8.2 in c:\\users\\avhis\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from pandas) (2.8.2)\n",
      "Requirement already satisfied: pytz>=2020.1 in c:\\users\\avhis\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from pandas) (2023.3.post1)\n",
      "Requirement already satisfied: tzdata>=2022.7 in c:\\users\\avhis\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from pandas) (2023.4)\n",
      "Requirement already satisfied: six>=1.5 in c:\\users\\avhis\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from python-dateutil>=2.8.2->pandas) (1.16.0)\n",
      "Requirement already satisfied: matplotlib in c:\\users\\avhis\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (3.8.2)\n",
      "Requirement already satisfied: contourpy>=1.0.1 in c:\\users\\avhis\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from matplotlib) (1.2.0)\n",
      "Requirement already satisfied: cycler>=0.10 in c:\\users\\avhis\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from matplotlib) (0.12.1)\n",
      "Requirement already satisfied: fonttools>=4.22.0 in c:\\users\\avhis\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from matplotlib) (4.47.2)\n",
      "Requirement already satisfied: kiwisolver>=1.3.1 in c:\\users\\avhis\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from matplotlib) (1.4.5)\n",
      "Requirement already satisfied: numpy<2,>=1.21 in c:\\users\\avhis\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from matplotlib) (1.26.3)\n",
      "Requirement already satisfied: packaging>=20.0 in c:\\users\\avhis\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from matplotlib) (23.2)\n",
      "Requirement already satisfied: pillow>=8 in c:\\users\\avhis\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from matplotlib) (10.2.0)\n",
      "Requirement already satisfied: pyparsing>=2.3.1 in c:\\users\\avhis\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from matplotlib) (3.1.1)\n",
      "Requirement already satisfied: python-dateutil>=2.7 in c:\\users\\avhis\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from matplotlib) (2.8.2)\n",
      "Requirement already satisfied: six>=1.5 in c:\\users\\avhis\\appdata\\local\\programs\\python\\python311\\lib\\site-packages (from python-dateutil>=2.7->matplotlib) (1.16.0)\n"
     ]
    }
   ],
   "source": [
    "!pip install torch\n",
    "!pip install numpy\n",
    "!pip install pandas\n",
    "!pip install matplotlib"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "248bd709-ec2a-48b6-bbaf-5193824e1426",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "# Importing Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "efa7700f-aa4a-47af-8a12-23edb9a6083e",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\avhis\\AppData\\Local\\Temp\\ipykernel_11248\\3959944622.py:5: DeprecationWarning: \n",
      "Pyarrow will become a required dependency of pandas in the next major release of pandas (pandas 3.0),\n",
      "(to allow more performant data types, such as the Arrow string type, and better interoperability with other libraries)\n",
      "but was not found to be installed on your system.\n",
      "If this would cause problems for you,\n",
      "please provide us feedback at https://github.com/pandas-dev/pandas/issues/54466\n",
      "        \n",
      "  import pandas as pd\n"
     ]
    }
   ],
   "source": [
    "import random\n",
    "import torch\n",
    "import torch.nn.functional as F # This is required for one-hot encoding\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "205f5d1b-c5be-4871-aed3-d349329f287f",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "# Loading Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d8277572-79a6-4ac7-878e-6235589d4f19",
   "metadata": {},
   "source": [
    "Once again, you can refer to the original <a href=\"https://github.com/AvishakeAdhikary/Neural-Networks-From-Scratch/blob/main/NameWeave.ipynb\">NameWeave</a> for reference, as to why we chose to load the dataset in the following way..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "36a0ea9b-821b-4492-a37d-f6f7a4e774c9",
   "metadata": {},
   "outputs": [],
   "source": [
    "words = open(\"Datasets/Indian_Names.txt\").read().splitlines()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "4f181a3d-a0e0-43fc-9a9e-3014b0be05a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "words = [word.lower() for word in words]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "fc8ce030-e90f-4527-82c4-aefcebe96242",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "53982"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(words)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5931037f-8025-4be4-969f-5c72eecffdef",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "# Building Vocabulary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "a74bfacc-e51a-4448-9f12-772f4487f378",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Characters: ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z']\n",
      "STOI: {'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5, 'f': 6, 'g': 7, 'h': 8, 'i': 9, 'j': 10, 'k': 11, 'l': 12, 'm': 13, 'n': 14, 'o': 15, 'p': 16, 'q': 17, 'r': 18, 's': 19, 't': 20, 'u': 21, 'v': 22, 'w': 23, 'x': 24, 'y': 25, 'z': 26, '.': 0}\n",
      "ITOS {1: 'a', 2: 'b', 3: 'c', 4: 'd', 5: 'e', 6: 'f', 7: 'g', 8: 'h', 9: 'i', 10: 'j', 11: 'k', 12: 'l', 13: 'm', 14: 'n', 15: 'o', 16: 'p', 17: 'q', 18: 'r', 19: 's', 20: 't', 21: 'u', 22: 'v', 23: 'w', 24: 'x', 25: 'y', 26: 'z', 0: '.'}\n"
     ]
    }
   ],
   "source": [
    "# Remember we need our starting and ending tokens as well in these mappings,\n",
    "characters = sorted(list(set(''.join(words)))) # Gives us all the characters in the english alphabet, hopefully our dataset has all of them\n",
    "stoi = {s:i+1 for i,s in enumerate(characters)} # Enumerate returns the tuples of number and string, which can then be mapped to string:index\n",
    "# We manually add these tokens for convenience\n",
    "stoi['.'] = 0\n",
    "itos = {i:s for s,i in stoi.items()} # After we have the string:index mapping, we can easily iterate over their items to map index:string\n",
    "print(\"Characters:\",characters)\n",
    "print(\"STOI:\",stoi)\n",
    "print(\"ITOS\",itos)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4bdd7c8d-b5a0-4f3b-8b61-316578d5d1df",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "# Building Dataset for Neural Network"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dafa3a85-f68d-4e2b-8f53-0b3d94f2dd39",
   "metadata": {},
   "source": [
    "Now, we can't just feed in names to our Neural Network.\\\n",
    "Rather, we need to build a dataset which will be able to feed into our neural network.\n",
    "\n",
    "Let's visualize how we are going to feed in to the neural network first...\n",
    "\n",
    "![Multi Layer Perceptron Approach](ExplanationMedia/Images/NameWeaveMultiLayerPerceptronApproach.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b64248f4-8188-431d-a613-6a5a2b4fc9cd",
   "metadata": {},
   "source": [
    "Let's now try to make this dataset...\n",
    "\n",
    "Remeber, this is **not bigram anymore**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "6dcc320c-f396-4bfd-b70f-229a2f235c6a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name: aaban\n",
      "... ---> a\n",
      "..a ---> a\n",
      ".aa ---> b\n",
      "aab ---> a\n",
      "aba ---> n\n",
      "ban ---> .\n",
      "Name: aabharan\n",
      "... ---> a\n",
      "..a ---> a\n",
      ".aa ---> b\n",
      "aab ---> h\n",
      "abh ---> a\n",
      "bha ---> r\n",
      "har ---> a\n",
      "ara ---> n\n",
      "ran ---> .\n",
      "Name: aabhas\n",
      "... ---> a\n",
      "..a ---> a\n",
      ".aa ---> b\n",
      "aab ---> h\n",
      "abh ---> a\n",
      "bha ---> s\n",
      "has ---> .\n",
      "Name: aabhat\n",
      "... ---> a\n",
      "..a ---> a\n",
      ".aa ---> b\n",
      "aab ---> h\n",
      "abh ---> a\n",
      "bha ---> t\n",
      "hat ---> .\n",
      "Name: aabheer\n",
      "... ---> a\n",
      "..a ---> a\n",
      ".aa ---> b\n",
      "aab ---> h\n",
      "abh ---> e\n",
      "bhe ---> e\n",
      "hee ---> r\n",
      "eer ---> .\n"
     ]
    }
   ],
   "source": [
    "# We define a Block Size based on the number of characters we feed are going to feed to predict the next one\n",
    "inputBlockSize = 3\n",
    "\n",
    "# We define two lists, inputs & outputs, where inputs are our blocks of the block size mentioned above and outputs are the label indexes\n",
    "inputs , outputs = [], []\n",
    "\n",
    "# We run a loop for each word in the original dataset\n",
    "for word in words[:5]:\n",
    "    # We define the block for each iteration and fill it with 0 values -> [0, 0, 0]\n",
    "    block = [0] * inputBlockSize # This is also known as the context of the network\n",
    "    # We print each word\n",
    "    print(\"Name:\", word)\n",
    "    # We run another loop for each word's character, here word also needs the ending token '.'\n",
    "    for character in word + '.':\n",
    "        # We take out the index from our look-up table\n",
    "        index = stoi[character]\n",
    "        # We append the input with our block\n",
    "        inputs.append(block)\n",
    "        # We append the output label with out index of the character\n",
    "        outputs.append([index])\n",
    "        # We can check our inputs and thier corresponsing outputs\n",
    "        print(''.join(itos[i] for i in block), '--->', itos[index])\n",
    "        # We then take the block, crop it 1 size from the left and append the next index to it (sliding window of name)\n",
    "        block = block[1:] + [index]\n",
    "# We also convert these inputs and outputs to tensors for neural network processing\n",
    "inputs = torch.tensor(inputs)\n",
    "outputs = torch.tensor(outputs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "d6e2afd8-a1d4-4319-909b-3e1edcf4d3c1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Inputs Shape: torch.Size([37, 3]) , Datatype: torch.int64\n",
      "Outputs Shape: torch.Size([37, 1]) , Datatype: torch.int64\n"
     ]
    }
   ],
   "source": [
    "# We can now check the shape of inputs and outputs and their corresponding datatypes\n",
    "print(\"Inputs Shape:\",inputs.shape,\", Datatype:\",inputs.dtype)\n",
    "print(\"Outputs Shape:\",outputs.shape,\", Datatype:\",outputs.dtype)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "b99fb60d-ba43-4f72-acd2-6e68f1e3cdf2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[ 0,  0,  0],\n",
      "        [ 0,  0,  1],\n",
      "        [ 0,  1,  1],\n",
      "        [ 1,  1,  2],\n",
      "        [ 1,  2,  1],\n",
      "        [ 2,  1, 14],\n",
      "        [ 0,  0,  0],\n",
      "        [ 0,  0,  1],\n",
      "        [ 0,  1,  1],\n",
      "        [ 1,  1,  2],\n",
      "        [ 1,  2,  8],\n",
      "        [ 2,  8,  1],\n",
      "        [ 8,  1, 18],\n",
      "        [ 1, 18,  1],\n",
      "        [18,  1, 14],\n",
      "        [ 0,  0,  0],\n",
      "        [ 0,  0,  1],\n",
      "        [ 0,  1,  1],\n",
      "        [ 1,  1,  2],\n",
      "        [ 1,  2,  8],\n",
      "        [ 2,  8,  1],\n",
      "        [ 8,  1, 19],\n",
      "        [ 0,  0,  0],\n",
      "        [ 0,  0,  1],\n",
      "        [ 0,  1,  1],\n",
      "        [ 1,  1,  2],\n",
      "        [ 1,  2,  8],\n",
      "        [ 2,  8,  1],\n",
      "        [ 8,  1, 20],\n",
      "        [ 0,  0,  0],\n",
      "        [ 0,  0,  1],\n",
      "        [ 0,  1,  1],\n",
      "        [ 1,  1,  2],\n",
      "        [ 1,  2,  8],\n",
      "        [ 2,  8,  5],\n",
      "        [ 8,  5,  5],\n",
      "        [ 5,  5, 18]])\n"
     ]
    }
   ],
   "source": [
    "# We can also check how the inputs look like\n",
    "print(inputs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "039c646f-5531-4d21-a100-03d1261646c4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[ 1],\n",
      "        [ 1],\n",
      "        [ 2],\n",
      "        [ 1],\n",
      "        [14],\n",
      "        [ 0],\n",
      "        [ 1],\n",
      "        [ 1],\n",
      "        [ 2],\n",
      "        [ 8],\n",
      "        [ 1],\n",
      "        [18],\n",
      "        [ 1],\n",
      "        [14],\n",
      "        [ 0],\n",
      "        [ 1],\n",
      "        [ 1],\n",
      "        [ 2],\n",
      "        [ 8],\n",
      "        [ 1],\n",
      "        [19],\n",
      "        [ 0],\n",
      "        [ 1],\n",
      "        [ 1],\n",
      "        [ 2],\n",
      "        [ 8],\n",
      "        [ 1],\n",
      "        [20],\n",
      "        [ 0],\n",
      "        [ 1],\n",
      "        [ 1],\n",
      "        [ 2],\n",
      "        [ 8],\n",
      "        [ 5],\n",
      "        [ 5],\n",
      "        [18],\n",
      "        [ 0]])\n"
     ]
    }
   ],
   "source": [
    "# We can also check how the outputs look like\n",
    "print(outputs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "d94a863c-3a0d-40fa-88b4-94c473cb3776",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We want our outputs to be single elements in a list and not add another dimension to the list\n",
    "# So we use a flatten method available in PyTorch to flatten these outputs\n",
    "outputs = torch.flatten(outputs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "6013cf4b-0813-4747-8a99-a6ffaf44b5fa",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([ 1,  1,  2,  1, 14,  0,  1,  1,  2,  8,  1, 18,  1, 14,  0,  1,  1,  2,\n",
      "         8,  1, 19,  0,  1,  1,  2,  8,  1, 20,  0,  1,  1,  2,  8,  5,  5, 18,\n",
      "         0])\n"
     ]
    }
   ],
   "source": [
    "print(outputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f45600e7-ea13-4f48-b536-75198d7feaf0",
   "metadata": {},
   "source": [
    "Now that we have our inputs and outputs configured, let's build our **embeddingLookUpMatrix 'C'**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4bbd4a0-4539-4566-b2ee-f4faebc60296",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "# Building Embedding Look-up Matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17ba1800-88f7-4510-b73b-98adf215d80b",
   "metadata": {},
   "source": [
    "In the paper the researchers had a big vocabulary of 17000 words,\\\n",
    "They used a very small 30 dimensional feature space.\n",
    "\n",
    "Because we have a vocabulary of only 27 characters,\\\n",
    "Let's use a very small 2 dimensional feature space for our embedding look-up matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "45b25701-bc49-483b-8c10-93302fe5427e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[-0.3732, -0.2275],\n",
      "        [-1.7182, -0.8157],\n",
      "        [-1.1759, -1.5987],\n",
      "        [-0.3984,  0.0547],\n",
      "        [ 0.1003, -0.8822],\n",
      "        [-2.1286, -1.6474],\n",
      "        [-0.5959,  0.3235],\n",
      "        [ 0.5287, -0.4012],\n",
      "        [ 0.5799, -0.7293],\n",
      "        [-1.4172, -2.0226],\n",
      "        [-0.7001,  0.6551],\n",
      "        [ 0.3579, -0.7145],\n",
      "        [ 0.8300, -1.0691],\n",
      "        [ 1.6646, -0.9261],\n",
      "        [ 0.0049,  0.9844],\n",
      "        [ 0.8385, -1.9852],\n",
      "        [ 0.7729,  0.5822],\n",
      "        [ 0.8481,  0.7299],\n",
      "        [ 0.9389,  0.7735],\n",
      "        [ 0.2951, -1.8969],\n",
      "        [-0.6994,  0.2413],\n",
      "        [-0.7727, -0.5433],\n",
      "        [ 1.0702,  0.7694],\n",
      "        [-1.2376, -0.0611],\n",
      "        [ 0.2379, -0.9606],\n",
      "        [ 0.2195, -0.5485],\n",
      "        [-0.2583, -0.4831]])\n"
     ]
    }
   ],
   "source": [
    "# We decide to build a embeddingLookUpMatrix with 27x2 because we have a vocabulary of 27 characters and we want to fit them in a 2 dimensional space\n",
    "# In the beginning we initialize it randomly\n",
    "embeddingFeatureSpaceLength = 2\n",
    "embeddingLookUpMatrix = torch.randn((len(stoi),embeddingFeatureSpaceLength))\n",
    "# So each one of our 27 characters will have a 2 dimensional embedding\n",
    "print(embeddingLookUpMatrix)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13c15f30-c740-4123-b11b-454ee80a4b2b",
   "metadata": {},
   "source": [
    "Now that we have a embedding-look-up matrix,\n",
    "\n",
    "For example,\n",
    "\n",
    "We can easily do:\n",
    "```python\n",
    "embeddingLookUpMatrix[6]\n",
    "```\n",
    "\n",
    "To get the embedding:\n",
    "```python\n",
    "tensor([-0.2483, -0.3909])\n",
    "```\n",
    "\n",
    "But there is a more similar way to do the exact same thing based on one-hot encoding....\n",
    "\n",
    "We can do:\n",
    "```python\n",
    "F.one_hot(torch.tensor(6), num_classes=27)\n",
    "```\n",
    "\n",
    "To get the one-hot embedding\n",
    "```python\n",
    "tensor([0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])\n",
    "```\n",
    "\n",
    "Then convert this to float and multiply it with our original *embeddingLookUpMatrix*:\n",
    "```python\n",
    "F.one_hot(torch.tensor(6), num_classes=27).float() @ embeddingLookUpMatrix\n",
    "```\n",
    "\n",
    "Which results in:\n",
    "```python\n",
    "tensor([-0.2483, -0.3909])\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b373dae-4ec4-45cd-99b9-2d1b53f1e8c9",
   "metadata": {},
   "source": [
    "This works because of the property of matrix multiplication,\n",
    "\n",
    "The 0's in our one-hot encoded vector discards all the zeros,\\\n",
    "And only multiplies the 1 to the corresponding column of the embeddingLookUpMatrix.\n",
    "\n",
    "So we can consider this matrix multiplication to be the first layer of our neural network,\\\n",
    "Giving us our corresponding embedding for the index. *(1x2 embedding vector for our case)*\n",
    "\n",
    "But we will simply index into our look-up table and discard the way of one-hot encoding for the time being"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "413d96f2-29d6-4d21-91e6-7f93856cff24",
   "metadata": {},
   "source": [
    "But, now that we know that we want to index into our look-up embedding matrix,\\\n",
    "How do we do that simultaneously for all the inputs?\n",
    "\n",
    "PyTorch has got you covered.\n",
    "\n",
    "In PyTorch we can, very flexibly pick out rows...\n",
    "\n",
    "For example,\\\n",
    "We can do indexing with lists of indexes:\n",
    "```python\n",
    "embeddingLookUpMatrix[[1,2,3]]\n",
    "```\n",
    "\n",
    "Which gives out the rows of the corresponding indexes:\n",
    "```python\n",
    "tensor([[ 0.2118,  1.0454],\n",
    "        [ 0.1876,  0.8921],\n",
    "        [ 0.9759, -0.2606]])\n",
    "```\n",
    "\n",
    "We can also do:\n",
    "```python\n",
    "embeddingLookUpMatrix[torch.tensor([1,2,3])]\n",
    "```\n",
    "\n",
    "Which gives out the rows of the corresponding indexes:\n",
    "```python\n",
    "tensor([[ 0.2118,  1.0454],\n",
    "        [ 0.1876,  0.8921],\n",
    "        [ 0.9759, -0.2606]])\n",
    "```\n",
    "\n",
    "We can similarly pick out the same rows again and again:\n",
    "```python\n",
    "embeddingLookUpMatrix[[1,2,3,3,3,3]]\n",
    "```\n",
    "\n",
    "Which gives us the same row again and again:\n",
    "```python\n",
    "tensor([[ 0.2118,  1.0454],\n",
    "        [ 0.1876,  0.8921],\n",
    "        [ 0.9759, -0.2606],\n",
    "        [ 0.9759, -0.2606],\n",
    "        [ 0.9759, -0.266],\n",
    "        [ 0.9759, -0.2606]])\n",
    "```\n",
    "\n",
    "Lastly, the magic happens when we try to do the same with multi dimensional lists as well:\n",
    "```python\n",
    "embeddingLookUpMatrix[[1, 0], [1, 1]]\n",
    "```\n",
    "\n",
    "Which results in:\n",
    "```python\n",
    "tensor([ 1.0454, -0.9078])\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "6ec17e65-4e17-442c-a9c8-c8d0db40377e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[-0.3732, -0.2275],\n",
       "         [-0.3732, -0.2275],\n",
       "         [-0.3732, -0.2275]],\n",
       "\n",
       "        [[-0.3732, -0.2275],\n",
       "         [-0.3732, -0.2275],\n",
       "         [-1.7182, -0.8157]],\n",
       "\n",
       "        [[-0.3732, -0.2275],\n",
       "         [-1.7182, -0.8157],\n",
       "         [-1.7182, -0.8157]],\n",
       "\n",
       "        [[-1.7182, -0.8157],\n",
       "         [-1.7182, -0.8157],\n",
       "         [-1.1759, -1.5987]],\n",
       "\n",
       "        [[-1.7182, -0.8157],\n",
       "         [-1.1759, -1.5987],\n",
       "         [-1.7182, -0.8157]],\n",
       "\n",
       "        [[-1.1759, -1.5987],\n",
       "         [-1.7182, -0.8157],\n",
       "         [ 0.0049,  0.9844]],\n",
       "\n",
       "        [[-0.3732, -0.2275],\n",
       "         [-0.3732, -0.2275],\n",
       "         [-0.3732, -0.2275]],\n",
       "\n",
       "        [[-0.3732, -0.2275],\n",
       "         [-0.3732, -0.2275],\n",
       "         [-1.7182, -0.8157]],\n",
       "\n",
       "        [[-0.3732, -0.2275],\n",
       "         [-1.7182, -0.8157],\n",
       "         [-1.7182, -0.8157]],\n",
       "\n",
       "        [[-1.7182, -0.8157],\n",
       "         [-1.7182, -0.8157],\n",
       "         [-1.1759, -1.5987]],\n",
       "\n",
       "        [[-1.7182, -0.8157],\n",
       "         [-1.1759, -1.5987],\n",
       "         [ 0.5799, -0.7293]],\n",
       "\n",
       "        [[-1.1759, -1.5987],\n",
       "         [ 0.5799, -0.7293],\n",
       "         [-1.7182, -0.8157]],\n",
       "\n",
       "        [[ 0.5799, -0.7293],\n",
       "         [-1.7182, -0.8157],\n",
       "         [ 0.9389,  0.7735]],\n",
       "\n",
       "        [[-1.7182, -0.8157],\n",
       "         [ 0.9389,  0.7735],\n",
       "         [-1.7182, -0.8157]],\n",
       "\n",
       "        [[ 0.9389,  0.7735],\n",
       "         [-1.7182, -0.8157],\n",
       "         [ 0.0049,  0.9844]],\n",
       "\n",
       "        [[-0.3732, -0.2275],\n",
       "         [-0.3732, -0.2275],\n",
       "         [-0.3732, -0.2275]],\n",
       "\n",
       "        [[-0.3732, -0.2275],\n",
       "         [-0.3732, -0.2275],\n",
       "         [-1.7182, -0.8157]],\n",
       "\n",
       "        [[-0.3732, -0.2275],\n",
       "         [-1.7182, -0.8157],\n",
       "         [-1.7182, -0.8157]],\n",
       "\n",
       "        [[-1.7182, -0.8157],\n",
       "         [-1.7182, -0.8157],\n",
       "         [-1.1759, -1.5987]],\n",
       "\n",
       "        [[-1.7182, -0.8157],\n",
       "         [-1.1759, -1.5987],\n",
       "         [ 0.5799, -0.7293]],\n",
       "\n",
       "        [[-1.1759, -1.5987],\n",
       "         [ 0.5799, -0.7293],\n",
       "         [-1.7182, -0.8157]],\n",
       "\n",
       "        [[ 0.5799, -0.7293],\n",
       "         [-1.7182, -0.8157],\n",
       "         [ 0.2951, -1.8969]],\n",
       "\n",
       "        [[-0.3732, -0.2275],\n",
       "         [-0.3732, -0.2275],\n",
       "         [-0.3732, -0.2275]],\n",
       "\n",
       "        [[-0.3732, -0.2275],\n",
       "         [-0.3732, -0.2275],\n",
       "         [-1.7182, -0.8157]],\n",
       "\n",
       "        [[-0.3732, -0.2275],\n",
       "         [-1.7182, -0.8157],\n",
       "         [-1.7182, -0.8157]],\n",
       "\n",
       "        [[-1.7182, -0.8157],\n",
       "         [-1.7182, -0.8157],\n",
       "         [-1.1759, -1.5987]],\n",
       "\n",
       "        [[-1.7182, -0.8157],\n",
       "         [-1.1759, -1.5987],\n",
       "         [ 0.5799, -0.7293]],\n",
       "\n",
       "        [[-1.1759, -1.5987],\n",
       "         [ 0.5799, -0.7293],\n",
       "         [-1.7182, -0.8157]],\n",
       "\n",
       "        [[ 0.5799, -0.7293],\n",
       "         [-1.7182, -0.8157],\n",
       "         [-0.6994,  0.2413]],\n",
       "\n",
       "        [[-0.3732, -0.2275],\n",
       "         [-0.3732, -0.2275],\n",
       "         [-0.3732, -0.2275]],\n",
       "\n",
       "        [[-0.3732, -0.2275],\n",
       "         [-0.3732, -0.2275],\n",
       "         [-1.7182, -0.8157]],\n",
       "\n",
       "        [[-0.3732, -0.2275],\n",
       "         [-1.7182, -0.8157],\n",
       "         [-1.7182, -0.8157]],\n",
       "\n",
       "        [[-1.7182, -0.8157],\n",
       "         [-1.7182, -0.8157],\n",
       "         [-1.1759, -1.5987]],\n",
       "\n",
       "        [[-1.7182, -0.8157],\n",
       "         [-1.1759, -1.5987],\n",
       "         [ 0.5799, -0.7293]],\n",
       "\n",
       "        [[-1.1759, -1.5987],\n",
       "         [ 0.5799, -0.7293],\n",
       "         [-2.1286, -1.6474]],\n",
       "\n",
       "        [[ 0.5799, -0.7293],\n",
       "         [-2.1286, -1.6474],\n",
       "         [-2.1286, -1.6474]],\n",
       "\n",
       "        [[-2.1286, -1.6474],\n",
       "         [-2.1286, -1.6474],\n",
       "         [ 0.9389,  0.7735]]])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# So we can now easily do\n",
    "embeddingLookUpMatrix[inputs]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "3edb92e9-8f70-444a-8129-3764415a5519",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([37, 3, 2])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We can also check the shape of this\n",
    "embeddingLookUpMatrix[inputs].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a95d0c59-9e9f-40e2-a166-1e2dcd7a37dc",
   "metadata": {},
   "source": [
    "We see that the size of this index is the shape of the original size of the dataset with a 2-dimensional embedding vector space"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5cc906b2-34d9-485b-88a6-b97ec2493cd4",
   "metadata": {},
   "source": [
    "So if we do:\n",
    "```python\n",
    "# Input of 5th block and 3rd index of the block\n",
    "inputs[5,2]\n",
    "```\n",
    "\n",
    "It gives us:\n",
    "```python\n",
    "tensor(14)\n",
    "```\n",
    "\n",
    "We can look that vector up by doing:\n",
    "```python\n",
    "embeddingLookUpMatrix[inputs][5,2]\n",
    "```\n",
    "\n",
    "Which gives us the corresponding vector of the item specified:\n",
    "```python\n",
    "tensor([ 1.7724, -0.9331])\n",
    "```\n",
    "\n",
    "We can verify the same by doing:\n",
    "```python\n",
    "embeddingLookUpMatrix[14]\n",
    "```\n",
    "\n",
    "Gives the same output:\n",
    "```python\n",
    "tensor([ 1.7724, -0.9331])\r",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "5a963c5f-13f5-47b7-84d3-33eae2eb22c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# So we can now define our embedding into a variable\n",
    "embedding = embeddingLookUpMatrix[inputs]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12d716f8-7e83-4b80-a994-5e8781195eef",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "# Constructing Hidden Layer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7273ea35-ab3b-4dec-ab40-7c4883f7b323",
   "metadata": {},
   "source": [
    "Let's understand what we will initially have in the hidden layer.\n",
    "\n",
    "1. The hidden layer will have it's own weights & biases\n",
    "2. The hidden layer will have it's own neurons which will act as a hyper-parameter to set the number of neurons we want in this layer\n",
    "\n",
    "So let's initialize weights and biases for now...\n",
    "\n",
    "Note: The size of the weights will be based on the block size of the inputs and its corresponding vector embedding.\\\n",
    "Thus,\n",
    "\n",
    "$$\\text{Hidden Layer Size} = [(\\text{Block Size} * \\text{Vector Embedding Dimensions}), \\text{Number of Neurons(Hyperparameter)}]$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "f11e60b6-9b9b-40c0-b854-0d4028481fcc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We can initialize the number of neurons we want in the hidden layer\n",
    "numberOfHiddenLayerNeurons = 100\n",
    "# Then we can randomly initialize the weights of the hidden layer\n",
    "weightsOfHiddenLayer = torch.randn((inputBlockSize*embeddingFeatureSpaceLength), numberOfHiddenLayerNeurons)\n",
    "# Then we can initialize the corresponding biases as well\n",
    "biasesOfHiddenLayer = torch.randn(numberOfHiddenLayerNeurons)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "379233f4-d2dd-4cfe-b5d5-9723663d38dd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of Weights of Hidden Layer: torch.Size([6, 100])\n",
      "Shape of Biases of Hidden Layer: torch.Size([100])\n"
     ]
    }
   ],
   "source": [
    "# We can check the shapes of our hidden layer weights and hidden layer biases\n",
    "print(\"Shape of Weights of Hidden Layer:\", weightsOfHiddenLayer.shape)\n",
    "print(\"Shape of Biases of Hidden Layer:\", biasesOfHiddenLayer.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "57bb8329-506e-4690-aa0b-76626363adac",
   "metadata": {},
   "source": [
    "Now that we have the weights and biases initialized for our hidden layer.\n",
    "\n",
    "By convention we would like to do something like:\n",
    "$$\\text{Layer Computation} = \\text{Embeddings} * \\text{Weights} + \\text{Biases}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dde1dd49-7470-434c-bff9-747c1ff7017a",
   "metadata": {},
   "source": [
    "But for our case:\n",
    "\n",
    "Embeddings are in the shape of the [number of blocks in all the names, block size, vector dimension size]\\\n",
    "for example, [37, 3, 2]\\\n",
    "& Weights are in the shape of [(block size * vector dimension size), number of neurons (hyperparameter)]\\\n",
    "for example, [6, 100]\n",
    "\n",
    "And thus, we cannot just simply multiply these matrices"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66f37a10-b884-47a9-8725-ac8a2cc46cca",
   "metadata": {},
   "source": [
    "There are a numerous ways to do this,\\\n",
    "Either we can convert [37, 3, 2] ---> [37, 6]\\\n",
    "Or we can convert [6, 100] ---> [3, 2, 100]\n",
    "\n",
    "We will stick with the first one, because it is fairly simpler and would reduce the complexity to understand the problem."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f386218c-e002-4531-a422-314e2b0da870",
   "metadata": {},
   "source": [
    "I invite you to also look into the documentation of <a href=\"https://pytorch.org/docs/stable/index.html\">PyTorch</a>."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "210401e5-8d41-419c-8bde-48a891b28a22",
   "metadata": {},
   "source": [
    "According to the official documentation,\n",
    "\n",
    "**TORCH.CAT**\n",
    "Concatenates the given sequence of seq tensors in the given dimension. All tensors must either have the same shape (except in the concatenating dimension) or be empty.\n",
    "\n",
    "```python\n",
    "torch.cat(tensors, dim=0, *, out=None) → Tensor\n",
    "```\n",
    "\n",
    "Parameters:\n",
    "- tensors (sequence of Tensors) – any python sequence of tensors of the same type. Non-empty tensors provided must have the same shape, except in the cat dimension.\n",
    "- dim (int, optional) – the dimension over which the tensors are concatenated\n",
    "\n",
    "Keyword Arguments:\n",
    "- out (Tensor, optional) – the output tensor.\n",
    "\n",
    "Example:\n",
    "```python\n",
    ">>> x = torch.randn(2, 3)\n",
    ">>> x\n",
    "tensor([[ 0.6580, -1.0969, -0.4614],\n",
    "        [-0.1034, -0.5790,  0.1497]])\n",
    ">>> torch.cat((x, x, x), 0)\n",
    "tensor([[ 0.6580, -1.0969, -0.4614],\n",
    "        [-0.1034, -0.5790,  0.1497],\n",
    "        [ 0.6580, -1.0969, -0.4614],\n",
    "        [-0.1034, -0.5790,  0.1497],\n",
    "        [ 0.6580, -1.0969, -0.4614],\n",
    "        [-0.1034, -0.5790,  0.1497]])\n",
    ">>> torch.cat((x, x, x), 1)\n",
    "tensor([[ 0.6580, -1.0969, -0.4614,  0.6580, -1.0969, -0.4614,  0.6580,\n",
    "         -1.0969, -0.4614],\n",
    "        [-0.1034, -0.5790,  0.1497, -0.1034, -0.5790,  0.1497, -0.1034,\n",
    "         -0.5790,  0.1497]])\n",
    "```\n",
    "\n",
    "So we can do:\n",
    "```python\n",
    "# Pickout the Embedding along the dimension 0, 1 & 2 and concatenate them along dimension 1\n",
    "# Each embedding[:, n, :] gives us the 3x2 embeddings\n",
    "torch.cat([embedding[:, 0, :], embedding[:, 1, :], embedding[:, 2, :]], dim=1)\n",
    "```\n",
    "\n",
    "And its shape turns out to be:\n",
    "```python\n",
    "torch.Size([37, 6])\n",
    "```\n",
    "\n",
    "But this is kind of ugly and we have another method...\n",
    "\n",
    "**TORCH.UNBIND**\n",
    "Removes a tensor dimension.\n",
    "\n",
    "Returns a tuple of all slices along a given dimension, already without it.\n",
    "\n",
    "```python\n",
    "torch.unbind(input, dim=0) → seq\n",
    "```\n",
    "Parameters:\n",
    "- input (Tensor) – the tensor to unbind\n",
    "- dim (int) – dimension to remove\n",
    "\n",
    "Example:\n",
    "```python\n",
    ">>> torch.unbind(torch.tensor([[1, 2, 3],\n",
    ">>>                            [4, 5, 6],\n",
    ">>>                            [7, 8, 9]]))\n",
    "(tensor([1, 2, 3]), tensor([4, 5, 6]), tensor([7, 8, 9]))\n",
    "```\n",
    "\n",
    "So now we can do:\n",
    "```python\n",
    "torch.cat(torch.unbind(embedding, dim=1), dim=1)\n",
    "```\n",
    "\n",
    "Whose shape also turns out to be:\n",
    "```python\n",
    "torch.Size([37, 6])\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4a0fbba-7b41-4e8b-9570-3a14aeb51038",
   "metadata": {},
   "source": [
    "We have a third way of doing the same thing.\n",
    "\n",
    "Its called:\\\n",
    "**TORCH.TENSOR.VIEW**\n",
    "\n",
    "Which gives me the paternity to explain some of the features of the internals of the PyTorch Library.\n",
    "\n",
    "We have an interesting blog post <a href=\"http://blog.ezyang.com/2019/05/pytorch-internals/\">here</a> by Edward Z. Yang which you can go through to understand more about this.\n",
    "\n",
    "To explain *TORCH.TENSOR.VIEW*.\\\n",
    "Let's take an example and explain each step one by one...\n",
    "\n",
    "For example,\\\n",
    "If we do:\n",
    "```python\n",
    "torch.arange(0,18)\n",
    "```\n",
    "\n",
    "It gives us:\n",
    "```python\n",
    "tensor([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17])\n",
    "```\n",
    "\n",
    "This can also be viewed as:\n",
    "```python\n",
    "torch.arange(0,18).view(9,2)\n",
    "```\n",
    "\n",
    "Which gives us:\n",
    "```python\n",
    "tensor([[ 0,  1],\n",
    "        [ 2,  3],\n",
    "        [ 4,  5],\n",
    "        [ 6,  7],\n",
    "        [ 8,  9],\n",
    "        [10, 11],\n",
    "        [12, 13],\n",
    "        [14, 15],\n",
    "        [16, 17]])\n",
    "```\n",
    "\n",
    "This can also be written as:\n",
    "```python\n",
    "torch.arange(0,18).view(3,3,2)\n",
    "```\n",
    "\n",
    "Which gives us:\n",
    "```python\n",
    "tensor([[[ 0,  1],\n",
    "         [ 2,  3],\n",
    "         [ 4,  5]],\n",
    "\n",
    "        [[ 6,  7],\n",
    "         [ 8,  9],\n",
    "         [10, 11]],\n",
    "\n",
    "        [[12, 13],\n",
    "         [14, 15],\n",
    "         [16, 17]]])\n",
    "```\n",
    "\n",
    "So,\\\n",
    "If we have an embedding of size say: [37, 3, 2]\\\n",
    "We can essentially do:\n",
    "```python\n",
    "embedding.view(37,6)\n",
    "```\n",
    "\n",
    "We can also verify the result to be the same by doing\n",
    "```python\n",
    "embedding.view(37,6) == torch.cat(torch.unbind(embedding, dim=1), dim=1)\n",
    "```\n",
    "\n",
    "Resulting in all True values"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00327478-b784-4b4b-a801-66610be1a353",
   "metadata": {},
   "source": [
    "So, to get the *hidden-states*, we can simply run:\n",
    "\n",
    "```python\n",
    "embedding.view(37,6)\n",
    "```\n",
    "\n",
    "to get all the hidden layer states as:\n",
    "\n",
    "```python\n",
    "hiddenLayerStates = embedding.view(37,6) @ weightsOfHiddenLayer + biasesOfHiddenLayer\n",
    "```\n",
    "\n",
    "<hr>\n",
    "\n",
    "Before we use this, you see how we are using 37 and 6 as a number which is hard-coded and does not make our model very flexible?\\\n",
    "Let's fix it by using *-1* instead of *37* to specify that it should take all the inputs, and use *inputBlockSize\\*embeddingFeatureSpaceLength* instead of *6*.\n",
    "\n",
    "<hr>\n",
    "\n",
    "Also, remembering our original multi-layer perceptron approach, we had something called tanh() non-linearity.\n",
    "\n",
    "So, instead we would want to know what tanh() is?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d32b1f17-80b3-4c0d-b619-f672b0f8b991",
   "metadata": {},
   "source": [
    "So **what is tanh()**?\n",
    "\n",
    "In order to answer that we have to understand a few more things,\n",
    "\n",
    "In mathematics, the trigonometric functions are real functions which relate an angle of a right-angled triangle to ratios of two side lengths.\n",
    "![Trigonometry](https://upload.wikimedia.org/wikipedia/commons/thumb/7/72/Sinus_und_Kosinus_am_Einheitskreis_1.svg/250px-Sinus_und_Kosinus_am_Einheitskreis_1.svg.png)\n",
    "\n",
    "In mathematics, hyperbolic functions are analogues of the ordinary trigonometric functions, but defined using the hyperbola rather than the circle.\n",
    "![Hyperbola v/s Parabola](ExplanationMedia/Images/Hyperbola-vs-Parabola.png)\n",
    "\n",
    "Here are some of the most used hyperbolic functions:\n",
    "![sinhcoshtanh](https://upload.wikimedia.org/wikipedia/commons/thumb/7/76/Sinh_cosh_tanh.svg/300px-Sinh_cosh_tanh.svg.png)\n",
    "\n",
    "So to answer the question,\\\n",
    "**tanh() is a hyperbolic function**, or a non-linearity we use to get a number between **-1 and 1** and ranges between $$-\\infty\\text{ and }\\infty$$\n",
    "\n",
    "This is the formula for tanh():\\\n",
    "$$\\tanh x = \\frac{\\sinh x}{\\cosh x} = \\frac{e^{x} - e^{-x}}{e^{x} + e^{-x}} = \\frac{e^{2x} - 1}{e^{2x} + 1}$$\n",
    "\n",
    "![tanh](https://miro.medium.com/v2/resize:fit:443/1*WeuJzmlt3iNVWsUsvf24Eg.png)\n",
    "\n",
    "So, we can now do:\n",
    "```python\n",
    "hiddenLayerStates = torch.tanh(embedding.view(37,6) @ weightsOfHiddenLayer + biasesOfHiddenLayer)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "a3317bb5-8771-4051-ae04-e8656ad0e08c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[-0.9746,  0.5218,  0.7038,  ...,  0.5388,  0.7837,  0.3920],\n",
      "        [-0.8478,  0.8180,  0.7753,  ...,  0.7804,  0.9930,  0.9403],\n",
      "        [-1.0000,  0.9342,  0.9968,  ...,  0.9875,  0.9945, -0.2034],\n",
      "        ...,\n",
      "        [ 0.9890,  0.7714, -0.9353,  ..., -0.9566,  1.0000,  0.5880],\n",
      "        [-1.0000, -0.2557,  0.9998,  ...,  0.5930,  0.9989, -0.3323],\n",
      "        [-1.0000,  0.9857,  0.9425,  ...,  0.9050,  0.9418, -1.0000]])\n"
     ]
    }
   ],
   "source": [
    "# So putting all the above things we learnt together, we get\n",
    "hiddenLayerStates = torch.tanh(embedding.view(-1, inputBlockSize*embeddingFeatureSpaceLength) @ weightsOfHiddenLayer + biasesOfHiddenLayer)\n",
    "print(hiddenLayerStates)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "cec60d4e-79f7-426e-9d64-f1318f79b850",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([37, 100])\n"
     ]
    }
   ],
   "source": [
    "# Let's see what the shape of the hidden layer states look like\n",
    "print(hiddenLayerStates.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a5e9161-2cd8-40a3-8249-d927d9f8dbb5",
   "metadata": {},
   "source": [
    "Keep in mind that we need to be careful with the broadcasting rules of the '+' of:\n",
    "```python\n",
    "hiddenLayerStates = torch.tanh(embedding.view(37,6) @ weightsOfHiddenLayer + biasesOfHiddenLayer)\n",
    "```\n",
    "\n",
    "I will move on to the next section, but to check the broadcasting rules you can refer to <a href=\"https://pytorch.org/docs/stable/notes/broadcasting.html\">original documentation</a> or refer to my <a href=\"https://github.com/AvishakeAdhikary/Neural-Networks-From-Scratch/blob/main/NameWeave.ipynb\">NameWeave Notebook</a> for a more simpler explanation.\n",
    "\n",
    "So let's create our final layer next..."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c1d5f57-87be-43e9-a953-f51e8f62f80a",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "# Constructing Final Layer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "515161be-f757-4ba5-b9df-5595c180c243",
   "metadata": {},
   "source": [
    "Looking at the shape that we have right now :\n",
    "\n",
    "```python\n",
    "torch.Size([37, 100])\n",
    "```\n",
    "\n",
    "We see that we have a *100* neurons, taking *37* inputs...\n",
    "\n",
    "We understand that each of these neurons would be the inputs to our final layer,\\\n",
    "Thus, we have to take a layer where it takes *100* inputs and produces the output of *27*.\n",
    "\n",
    "Why **27**?\n",
    "\n",
    "Because, we would be interested in the index of the output now.\n",
    "\n",
    "So we will do:\n",
    "```python\n",
    "# We can initialize the number of neurons we have in the final layer\n",
    "numberOfFinalLayerOutputs = 27\n",
    "# Then we can randomly initialize the weights of the final layer\n",
    "weightsOfFinalLayer = torch.randn(numberOfHiddenLayerNeurons, numberOfFinalLayerOutputs)\n",
    "# Then we can initialize the corresponding biases as well\n",
    "biasesOfFinalLayer = torch.randn(numberOfFinalLayerOutputs)\n",
    "```\n",
    "\n",
    "Therefore,\\\n",
    "The **logits** our final layer will produce would be:\n",
    "$$\\text{Logits} = \\text{hiddenLayerStates} * \\text{weightsOfFinalLayer} + \\text{biasesOfFinalLayer}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "99c85ff5-f44f-4eb7-b977-3f9a942b9577",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's construct our final layer\n",
    "# We can initialize the number of neurons we have in the final layer\n",
    "numberOfFinalLayerOutputs = 27\n",
    "# Then we can randomly initialize the weights of the final layer\n",
    "weightsOfFinalLayer = torch.randn(numberOfHiddenLayerNeurons, numberOfFinalLayerOutputs)\n",
    "# Then we can initialize the corresponding biases as well\n",
    "biasesOfFinalLayer = torch.randn(numberOfFinalLayerOutputs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "e95707b2-dfc5-4589-ad73-0eeb68c81bc1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's compute logits now\n",
    "logits = hiddenLayerStates @ weightsOfFinalLayer + biasesOfFinalLayer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "baa299fb-d459-48dd-94d7-e15b585dd6c1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[-4.4356e+00, -8.8625e+00, -6.4110e+00,  3.1217e+00, -2.3895e-01,\n",
      "         -9.1472e+00,  3.5807e+00,  6.4967e+00, -1.9470e+01,  1.5936e+00,\n",
      "          1.2955e+00,  5.4860e+00,  6.1289e+00, -3.6768e+00, -8.2575e+00,\n",
      "         -6.8105e+00,  4.4014e+00, -6.6630e+00,  6.3047e+00,  4.2948e+00,\n",
      "          3.7352e+00,  8.3515e+00, -1.2195e+01,  4.6262e+00, -6.4790e+00,\n",
      "          7.4604e+00, -2.3536e+00],\n",
      "        [ 2.3476e-01, -8.9568e+00, -5.2255e+00,  6.9277e+00, -4.9331e+00,\n",
      "         -1.0707e+01, -3.5129e+00,  1.0341e+01, -1.0205e+01,  1.9994e+00,\n",
      "          2.7964e-01,  8.9195e+00,  1.9596e+00, -1.1375e+01, -5.5082e+00,\n",
      "          1.6570e-01,  7.1465e+00, -8.9007e+00, -2.8143e+00,  9.6585e+00,\n",
      "          3.7122e+00,  1.8932e+01, -2.9878e+01,  1.2045e+01, -5.9551e+00,\n",
      "          1.1613e+01, -9.0741e+00],\n",
      "        [-5.5437e+00, -6.9723e+00, -4.4161e-01,  5.3509e+00, -4.4173e+00,\n",
      "         -1.7688e+01,  6.3378e+00,  1.0275e+01, -2.4718e+01,  1.0289e+00,\n",
      "         -2.9554e+00,  9.5491e-01,  7.7627e+00, -7.4367e+00, -8.2750e+00,\n",
      "          7.7400e-01,  6.2495e+00,  2.9863e-01, -6.4854e+00,  6.0232e+00,\n",
      "          1.2515e+01,  1.7104e+01, -2.8975e+01,  1.2406e+01, -2.6916e+00,\n",
      "          2.0020e+00, -2.5385e+00],\n",
      "        [ 3.3066e+00, -1.0208e+01,  1.0351e+01,  1.7414e+00,  5.4813e+00,\n",
      "         -1.2236e+01, -1.8461e+00,  1.1064e+01, -1.8591e+01,  1.3269e+01,\n",
      "          6.6166e-01,  1.1120e+01,  4.4457e+00, -3.1987e+00, -1.1022e+01,\n",
      "          8.1249e-01,  1.2322e+00, -2.4404e+00, -6.4215e+00,  1.2246e+01,\n",
      "          1.3385e+01,  1.4645e+01, -1.6118e+01,  3.1399e+00, -1.0771e+01,\n",
      "          8.3958e-01, -6.1222e-01],\n",
      "        [ 6.7180e+00, -7.8900e+00,  1.2541e+01, -1.5069e+00,  4.2007e+00,\n",
      "         -8.7647e+00, -3.8941e+00,  1.8257e+00, -1.8121e+01,  1.3882e+01,\n",
      "         -1.9503e+00,  4.4935e+00,  7.4093e-01, -3.5765e+00, -7.8034e+00,\n",
      "         -1.1913e+00,  2.2751e+00, -2.8461e+00, -7.4895e+00,  7.7438e+00,\n",
      "          8.3409e+00,  6.9898e+00, -1.5974e+01,  5.0328e+00, -5.8420e+00,\n",
      "          3.3095e+00,  9.8745e-01],\n",
      "        [-4.1747e+00, -2.1214e+00,  3.4516e-01, -1.9260e+00, -5.5588e+00,\n",
      "         -7.3468e+00, -2.1426e+00, -8.4366e-01, -1.8959e+01,  6.2335e+00,\n",
      "         -9.2336e+00, -1.8854e+01,  5.0275e+00, -2.7895e+00,  4.9814e-01,\n",
      "         -7.8567e+00,  9.7467e+00, -1.3460e+01,  1.5745e+00, -1.3843e+00,\n",
      "          1.3731e+01,  1.7357e+00, -1.6846e+00, -1.3420e+00,  2.3723e+00,\n",
      "         -7.5971e-01,  7.6400e+00],\n",
      "        [-4.4356e+00, -8.8625e+00, -6.4110e+00,  3.1217e+00, -2.3895e-01,\n",
      "         -9.1472e+00,  3.5807e+00,  6.4967e+00, -1.9470e+01,  1.5936e+00,\n",
      "          1.2955e+00,  5.4860e+00,  6.1289e+00, -3.6768e+00, -8.2575e+00,\n",
      "         -6.8105e+00,  4.4014e+00, -6.6630e+00,  6.3047e+00,  4.2948e+00,\n",
      "          3.7352e+00,  8.3515e+00, -1.2195e+01,  4.6262e+00, -6.4790e+00,\n",
      "          7.4604e+00, -2.3536e+00],\n",
      "        [ 2.3476e-01, -8.9568e+00, -5.2255e+00,  6.9277e+00, -4.9331e+00,\n",
      "         -1.0707e+01, -3.5129e+00,  1.0341e+01, -1.0205e+01,  1.9994e+00,\n",
      "          2.7964e-01,  8.9195e+00,  1.9596e+00, -1.1375e+01, -5.5082e+00,\n",
      "          1.6570e-01,  7.1465e+00, -8.9007e+00, -2.8143e+00,  9.6585e+00,\n",
      "          3.7122e+00,  1.8932e+01, -2.9878e+01,  1.2045e+01, -5.9551e+00,\n",
      "          1.1613e+01, -9.0741e+00],\n",
      "        [-5.5437e+00, -6.9723e+00, -4.4161e-01,  5.3509e+00, -4.4173e+00,\n",
      "         -1.7688e+01,  6.3378e+00,  1.0275e+01, -2.4718e+01,  1.0289e+00,\n",
      "         -2.9554e+00,  9.5491e-01,  7.7627e+00, -7.4367e+00, -8.2750e+00,\n",
      "          7.7400e-01,  6.2495e+00,  2.9863e-01, -6.4854e+00,  6.0232e+00,\n",
      "          1.2515e+01,  1.7104e+01, -2.8975e+01,  1.2406e+01, -2.6916e+00,\n",
      "          2.0020e+00, -2.5385e+00],\n",
      "        [ 3.3066e+00, -1.0208e+01,  1.0351e+01,  1.7414e+00,  5.4813e+00,\n",
      "         -1.2236e+01, -1.8461e+00,  1.1064e+01, -1.8591e+01,  1.3269e+01,\n",
      "          6.6166e-01,  1.1120e+01,  4.4457e+00, -3.1987e+00, -1.1022e+01,\n",
      "          8.1249e-01,  1.2322e+00, -2.4404e+00, -6.4215e+00,  1.2246e+01,\n",
      "          1.3385e+01,  1.4645e+01, -1.6118e+01,  3.1399e+00, -1.0771e+01,\n",
      "          8.3958e-01, -6.1222e-01],\n",
      "        [-2.8421e+00,  8.7685e-01,  6.4604e+00, -2.7273e+00,  8.8758e+00,\n",
      "         -6.7878e+00, -2.3644e+00,  3.9931e+00, -1.8163e+01,  1.2923e+01,\n",
      "          7.8366e+00,  5.9661e-01, -6.3733e+00, -7.2474e+00, -1.7708e+00,\n",
      "         -7.7871e+00,  1.6492e+00, -7.0582e+00,  7.5263e-01,  7.1385e+00,\n",
      "          7.8308e+00,  3.0268e+00,  1.2647e+00,  6.8169e+00,  2.1770e+00,\n",
      "          2.3540e+00,  5.4462e+00],\n",
      "        [ 1.2261e+01, -1.5318e+01, -2.0415e+00,  4.2956e+00,  1.3433e+00,\n",
      "          2.7929e+00,  3.4509e+00, -8.1581e+00, -1.1541e+01,  4.6856e+00,\n",
      "         -1.4047e+00,  7.5359e+00,  2.0134e+00, -3.6041e+00,  1.8199e-01,\n",
      "          1.5634e+00,  6.0163e+00, -6.5428e+00, -5.4033e+00,  4.6587e+00,\n",
      "          1.1133e+01,  6.5822e+00, -1.9064e+01, -6.1070e+00, -6.4065e+00,\n",
      "          6.7919e+00, -4.8555e+00],\n",
      "        [-9.5698e+00, -4.5844e-02, -9.9431e+00, -2.9966e+00,  1.6630e+00,\n",
      "         -1.8037e+01,  1.4521e+01,  1.2584e+00, -1.8230e+01, -4.8909e+00,\n",
      "         -1.8814e+00, -1.3296e+01, -7.6939e+00, -4.6964e+00, -9.6126e-02,\n",
      "         -1.7179e+00,  5.6996e+00, -7.4373e+00,  6.4745e+00, -2.2479e+00,\n",
      "          4.4010e+00, -3.0722e+00,  1.0785e+01,  2.1722e-02,  6.4349e+00,\n",
      "         -1.9981e+00,  6.5684e+00],\n",
      "        [ 4.2076e+00, -1.5703e+01, -6.6788e+00,  8.7969e+00, -7.6446e-01,\n",
      "         -1.5758e-01,  1.9925e+00, -1.5242e+00, -6.3783e+00,  2.2721e-01,\n",
      "         -3.3305e+00,  6.3117e+00,  1.0361e+01, -2.8075e+00, -2.7127e+00,\n",
      "         -3.0984e+00,  4.9851e+00, -6.7348e+00, -1.0262e+01,  1.1056e+01,\n",
      "          1.0206e+01,  1.1613e+01, -1.5904e+01,  4.1697e+00, -8.9419e+00,\n",
      "          1.2978e+01, -3.2104e+00],\n",
      "        [-1.1965e+01, -2.2247e+00, -6.7328e+00, -1.9177e+00,  2.7786e+00,\n",
      "         -3.1500e+01,  3.1115e+00,  1.0942e+00, -1.4496e+01, -1.0024e+01,\n",
      "         -3.1611e+00, -8.6104e+00, -1.5120e+01, -3.6086e+00,  1.1904e+00,\n",
      "          2.9404e+00,  4.4425e+00, -1.5160e+00,  7.3564e+00, -1.6516e+00,\n",
      "         -8.7477e+00, -8.9075e+00, -4.1746e-01,  2.1856e+00,  4.0690e+00,\n",
      "          4.6998e+00,  4.6664e+00],\n",
      "        [-4.4356e+00, -8.8625e+00, -6.4110e+00,  3.1217e+00, -2.3895e-01,\n",
      "         -9.1472e+00,  3.5807e+00,  6.4967e+00, -1.9470e+01,  1.5936e+00,\n",
      "          1.2955e+00,  5.4860e+00,  6.1289e+00, -3.6768e+00, -8.2575e+00,\n",
      "         -6.8105e+00,  4.4014e+00, -6.6630e+00,  6.3047e+00,  4.2948e+00,\n",
      "          3.7352e+00,  8.3515e+00, -1.2195e+01,  4.6262e+00, -6.4790e+00,\n",
      "          7.4604e+00, -2.3536e+00],\n",
      "        [ 2.3476e-01, -8.9568e+00, -5.2255e+00,  6.9277e+00, -4.9331e+00,\n",
      "         -1.0707e+01, -3.5129e+00,  1.0341e+01, -1.0205e+01,  1.9994e+00,\n",
      "          2.7964e-01,  8.9195e+00,  1.9596e+00, -1.1375e+01, -5.5082e+00,\n",
      "          1.6570e-01,  7.1465e+00, -8.9007e+00, -2.8143e+00,  9.6585e+00,\n",
      "          3.7122e+00,  1.8932e+01, -2.9878e+01,  1.2045e+01, -5.9551e+00,\n",
      "          1.1613e+01, -9.0741e+00],\n",
      "        [-5.5437e+00, -6.9723e+00, -4.4161e-01,  5.3509e+00, -4.4173e+00,\n",
      "         -1.7688e+01,  6.3378e+00,  1.0275e+01, -2.4718e+01,  1.0289e+00,\n",
      "         -2.9554e+00,  9.5491e-01,  7.7627e+00, -7.4367e+00, -8.2750e+00,\n",
      "          7.7400e-01,  6.2495e+00,  2.9863e-01, -6.4854e+00,  6.0232e+00,\n",
      "          1.2515e+01,  1.7104e+01, -2.8975e+01,  1.2406e+01, -2.6916e+00,\n",
      "          2.0020e+00, -2.5385e+00],\n",
      "        [ 3.3066e+00, -1.0208e+01,  1.0351e+01,  1.7414e+00,  5.4813e+00,\n",
      "         -1.2236e+01, -1.8461e+00,  1.1064e+01, -1.8591e+01,  1.3269e+01,\n",
      "          6.6166e-01,  1.1120e+01,  4.4457e+00, -3.1987e+00, -1.1022e+01,\n",
      "          8.1249e-01,  1.2322e+00, -2.4404e+00, -6.4215e+00,  1.2246e+01,\n",
      "          1.3385e+01,  1.4645e+01, -1.6118e+01,  3.1399e+00, -1.0771e+01,\n",
      "          8.3958e-01, -6.1222e-01],\n",
      "        [-2.8421e+00,  8.7685e-01,  6.4604e+00, -2.7273e+00,  8.8758e+00,\n",
      "         -6.7878e+00, -2.3644e+00,  3.9931e+00, -1.8163e+01,  1.2923e+01,\n",
      "          7.8366e+00,  5.9661e-01, -6.3733e+00, -7.2474e+00, -1.7708e+00,\n",
      "         -7.7871e+00,  1.6492e+00, -7.0582e+00,  7.5263e-01,  7.1385e+00,\n",
      "          7.8308e+00,  3.0268e+00,  1.2647e+00,  6.8169e+00,  2.1770e+00,\n",
      "          2.3540e+00,  5.4462e+00],\n",
      "        [ 1.2261e+01, -1.5318e+01, -2.0415e+00,  4.2956e+00,  1.3433e+00,\n",
      "          2.7929e+00,  3.4509e+00, -8.1581e+00, -1.1541e+01,  4.6856e+00,\n",
      "         -1.4047e+00,  7.5359e+00,  2.0134e+00, -3.6041e+00,  1.8199e-01,\n",
      "          1.5634e+00,  6.0163e+00, -6.5428e+00, -5.4033e+00,  4.6587e+00,\n",
      "          1.1133e+01,  6.5822e+00, -1.9064e+01, -6.1070e+00, -6.4065e+00,\n",
      "          6.7919e+00, -4.8555e+00],\n",
      "        [-1.3436e+01, -2.5957e-01, -7.5884e+00,  2.7797e+00,  8.9042e+00,\n",
      "         -6.3175e+00,  1.9963e+01,  1.4876e+01, -1.6628e+01, -8.3246e-01,\n",
      "          1.2115e+01,  6.3622e+00,  4.3550e+00, -1.0658e+01, -7.6415e+00,\n",
      "          1.6107e+00, -3.3438e+00, -2.8657e+00,  1.6576e-01,  7.3985e+00,\n",
      "          1.4919e+01,  1.9000e+01, -1.7770e+01,  1.3784e+01,  8.3052e-01,\n",
      "         -1.0894e+01, -6.1207e-01],\n",
      "        [-4.4356e+00, -8.8625e+00, -6.4110e+00,  3.1217e+00, -2.3895e-01,\n",
      "         -9.1472e+00,  3.5807e+00,  6.4967e+00, -1.9470e+01,  1.5936e+00,\n",
      "          1.2955e+00,  5.4860e+00,  6.1289e+00, -3.6768e+00, -8.2575e+00,\n",
      "         -6.8105e+00,  4.4014e+00, -6.6630e+00,  6.3047e+00,  4.2948e+00,\n",
      "          3.7352e+00,  8.3515e+00, -1.2195e+01,  4.6262e+00, -6.4790e+00,\n",
      "          7.4604e+00, -2.3536e+00],\n",
      "        [ 2.3476e-01, -8.9568e+00, -5.2255e+00,  6.9277e+00, -4.9331e+00,\n",
      "         -1.0707e+01, -3.5129e+00,  1.0341e+01, -1.0205e+01,  1.9994e+00,\n",
      "          2.7964e-01,  8.9195e+00,  1.9596e+00, -1.1375e+01, -5.5082e+00,\n",
      "          1.6570e-01,  7.1465e+00, -8.9007e+00, -2.8143e+00,  9.6585e+00,\n",
      "          3.7122e+00,  1.8932e+01, -2.9878e+01,  1.2045e+01, -5.9551e+00,\n",
      "          1.1613e+01, -9.0741e+00],\n",
      "        [-5.5437e+00, -6.9723e+00, -4.4161e-01,  5.3509e+00, -4.4173e+00,\n",
      "         -1.7688e+01,  6.3378e+00,  1.0275e+01, -2.4718e+01,  1.0289e+00,\n",
      "         -2.9554e+00,  9.5491e-01,  7.7627e+00, -7.4367e+00, -8.2750e+00,\n",
      "          7.7400e-01,  6.2495e+00,  2.9863e-01, -6.4854e+00,  6.0232e+00,\n",
      "          1.2515e+01,  1.7104e+01, -2.8975e+01,  1.2406e+01, -2.6916e+00,\n",
      "          2.0020e+00, -2.5385e+00],\n",
      "        [ 3.3066e+00, -1.0208e+01,  1.0351e+01,  1.7414e+00,  5.4813e+00,\n",
      "         -1.2236e+01, -1.8461e+00,  1.1064e+01, -1.8591e+01,  1.3269e+01,\n",
      "          6.6166e-01,  1.1120e+01,  4.4457e+00, -3.1987e+00, -1.1022e+01,\n",
      "          8.1249e-01,  1.2322e+00, -2.4404e+00, -6.4215e+00,  1.2246e+01,\n",
      "          1.3385e+01,  1.4645e+01, -1.6118e+01,  3.1399e+00, -1.0771e+01,\n",
      "          8.3958e-01, -6.1222e-01],\n",
      "        [-2.8421e+00,  8.7685e-01,  6.4604e+00, -2.7273e+00,  8.8758e+00,\n",
      "         -6.7878e+00, -2.3644e+00,  3.9931e+00, -1.8163e+01,  1.2923e+01,\n",
      "          7.8366e+00,  5.9661e-01, -6.3733e+00, -7.2474e+00, -1.7708e+00,\n",
      "         -7.7871e+00,  1.6492e+00, -7.0582e+00,  7.5263e-01,  7.1385e+00,\n",
      "          7.8308e+00,  3.0268e+00,  1.2647e+00,  6.8169e+00,  2.1770e+00,\n",
      "          2.3540e+00,  5.4462e+00],\n",
      "        [ 1.2261e+01, -1.5318e+01, -2.0415e+00,  4.2956e+00,  1.3433e+00,\n",
      "          2.7929e+00,  3.4509e+00, -8.1581e+00, -1.1541e+01,  4.6856e+00,\n",
      "         -1.4047e+00,  7.5359e+00,  2.0134e+00, -3.6041e+00,  1.8199e-01,\n",
      "          1.5634e+00,  6.0163e+00, -6.5428e+00, -5.4033e+00,  4.6587e+00,\n",
      "          1.1133e+01,  6.5822e+00, -1.9064e+01, -6.1070e+00, -6.4065e+00,\n",
      "          6.7919e+00, -4.8555e+00],\n",
      "        [-5.8503e+00, -3.9859e+00, -1.8554e+00,  1.5229e+00, -1.9976e+00,\n",
      "         -1.4683e+01,  1.5529e+01,  1.8123e+00, -1.6826e+01, -3.8858e+00,\n",
      "         -3.7928e+00, -4.5226e+00,  9.5882e-01, -1.6703e+00, -7.3085e+00,\n",
      "         -5.1309e-01,  3.8166e+00, -8.5795e+00,  3.9782e+00, -8.0849e-02,\n",
      "          5.3522e+00,  1.0821e+01, -1.4752e+01,  7.7735e+00, -2.0883e+00,\n",
      "         -4.8702e+00,  5.7407e+00],\n",
      "        [-4.4356e+00, -8.8625e+00, -6.4110e+00,  3.1217e+00, -2.3895e-01,\n",
      "         -9.1472e+00,  3.5807e+00,  6.4967e+00, -1.9470e+01,  1.5936e+00,\n",
      "          1.2955e+00,  5.4860e+00,  6.1289e+00, -3.6768e+00, -8.2575e+00,\n",
      "         -6.8105e+00,  4.4014e+00, -6.6630e+00,  6.3047e+00,  4.2948e+00,\n",
      "          3.7352e+00,  8.3515e+00, -1.2195e+01,  4.6262e+00, -6.4790e+00,\n",
      "          7.4604e+00, -2.3536e+00],\n",
      "        [ 2.3476e-01, -8.9568e+00, -5.2255e+00,  6.9277e+00, -4.9331e+00,\n",
      "         -1.0707e+01, -3.5129e+00,  1.0341e+01, -1.0205e+01,  1.9994e+00,\n",
      "          2.7964e-01,  8.9195e+00,  1.9596e+00, -1.1375e+01, -5.5082e+00,\n",
      "          1.6570e-01,  7.1465e+00, -8.9007e+00, -2.8143e+00,  9.6585e+00,\n",
      "          3.7122e+00,  1.8932e+01, -2.9878e+01,  1.2045e+01, -5.9551e+00,\n",
      "          1.1613e+01, -9.0741e+00],\n",
      "        [-5.5437e+00, -6.9723e+00, -4.4161e-01,  5.3509e+00, -4.4173e+00,\n",
      "         -1.7688e+01,  6.3378e+00,  1.0275e+01, -2.4718e+01,  1.0289e+00,\n",
      "         -2.9554e+00,  9.5491e-01,  7.7627e+00, -7.4367e+00, -8.2750e+00,\n",
      "          7.7400e-01,  6.2495e+00,  2.9863e-01, -6.4854e+00,  6.0232e+00,\n",
      "          1.2515e+01,  1.7104e+01, -2.8975e+01,  1.2406e+01, -2.6916e+00,\n",
      "          2.0020e+00, -2.5385e+00],\n",
      "        [ 3.3066e+00, -1.0208e+01,  1.0351e+01,  1.7414e+00,  5.4813e+00,\n",
      "         -1.2236e+01, -1.8461e+00,  1.1064e+01, -1.8591e+01,  1.3269e+01,\n",
      "          6.6166e-01,  1.1120e+01,  4.4457e+00, -3.1987e+00, -1.1022e+01,\n",
      "          8.1249e-01,  1.2322e+00, -2.4404e+00, -6.4215e+00,  1.2246e+01,\n",
      "          1.3385e+01,  1.4645e+01, -1.6118e+01,  3.1399e+00, -1.0771e+01,\n",
      "          8.3958e-01, -6.1222e-01],\n",
      "        [-2.8421e+00,  8.7685e-01,  6.4604e+00, -2.7273e+00,  8.8758e+00,\n",
      "         -6.7878e+00, -2.3644e+00,  3.9931e+00, -1.8163e+01,  1.2923e+01,\n",
      "          7.8366e+00,  5.9661e-01, -6.3733e+00, -7.2474e+00, -1.7708e+00,\n",
      "         -7.7871e+00,  1.6492e+00, -7.0582e+00,  7.5263e-01,  7.1385e+00,\n",
      "          7.8308e+00,  3.0268e+00,  1.2647e+00,  6.8169e+00,  2.1770e+00,\n",
      "          2.3540e+00,  5.4462e+00],\n",
      "        [ 1.4069e+01, -1.3007e+01, -2.9610e+00,  9.8754e+00,  8.6512e-01,\n",
      "         -1.6986e+00, -2.5456e-01, -4.2012e+00, -9.5849e+00,  4.8708e+00,\n",
      "         -2.4824e-02,  1.2365e+01, -2.9975e+00, -7.3214e+00, -7.1760e-01,\n",
      "          3.6582e+00,  9.8511e+00, -4.9319e+00, -9.6200e+00,  8.7288e+00,\n",
      "          1.3958e+01,  1.4818e+01, -2.2845e+01, -3.1649e-01, -1.0868e+01,\n",
      "          1.0952e+01, -8.5911e+00],\n",
      "        [-8.4543e+00, -7.1699e+00, -1.5645e-01,  4.6459e+00, -4.9868e+00,\n",
      "         -1.6649e+01,  9.8381e+00,  9.1675e+00, -1.8095e+01, -7.3937e-01,\n",
      "         -3.7151e-01,  2.5782e-01,  8.5094e+00, -9.3854e+00, -7.1315e+00,\n",
      "          8.1574e+00, -6.2608e-01,  2.5184e-01, -1.5672e+00,  7.8542e+00,\n",
      "          7.8682e+00,  1.3713e+01, -3.1462e+01,  1.6804e+01, -3.8049e+00,\n",
      "         -3.6788e+00, -5.0772e+00],\n",
      "        [-5.8696e+00,  2.7939e-01,  6.1838e+00, -3.2773e+00, -2.7249e+00,\n",
      "         -9.2387e+00, -5.3334e+00, -3.0648e-01, -2.1194e+01,  1.0181e+01,\n",
      "          5.1341e-01, -1.3003e+01, -6.6029e+00, -8.6858e-01,  6.6507e-01,\n",
      "         -7.1930e+00,  4.5475e+00, -9.7432e+00,  1.8197e+00,  2.1985e+00,\n",
      "          1.3847e+01,  7.7612e-01,  8.2672e+00, -5.0245e-01,  9.5023e+00,\n",
      "         -6.3760e-01,  7.3681e+00]])\n"
     ]
    }
   ],
   "source": [
    "# Let's check the output\n",
    "print(logits)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "0e6d24c7-3ffb-4a64-85f1-67863ac426f8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([37, 27])\n"
     ]
    }
   ],
   "source": [
    "#Let's check the shape as well\n",
    "print(logits.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3244beae-4a1a-493f-ae51-e59ffc59162a",
   "metadata": {},
   "source": [
    "So, we want to do exactly now what we did in our previous <a href=\"https://github.com/AvishakeAdhikary/Neural-Networks-From-Scratch/blob/main/NameWeave.ipynb\">NameWeave Notebook</a>.\\\n",
    "We want to:\n",
    "1. Take the logits\n",
    "2. Exponentiate them\n",
    "3. Normalize them into a probability that sum to 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "54483c0b-b2d1-4820-9622-c1f6d884056e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# So let's get our probabilities back\n",
    "# Calculating counts from logits\n",
    "counts = logits.exp()\n",
    "# Normalizing counts to probabilities that sum to 1\n",
    "probabilities = counts / counts.sum(1, keepdims=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "cf21e938-b523-4920-9649-3a9252f9ab77",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[1.4358e-06, 1.7160e-08, 1.9915e-07, 2.7491e-03, 9.5430e-05, 1.2909e-08,\n",
      "         4.3505e-03, 8.0338e-02, 4.2430e-13, 5.9642e-04, 4.4268e-04, 2.9241e-02,\n",
      "         5.5615e-02, 3.0666e-06, 3.1424e-08, 1.3357e-07, 9.8850e-03, 1.5480e-07,\n",
      "         6.6309e-02, 8.8849e-03, 5.0772e-03, 5.1343e-01, 6.1277e-10, 1.2376e-02,\n",
      "         1.8606e-07, 2.1060e-01, 1.1516e-05],\n",
      "        [7.5674e-09, 7.7109e-13, 3.2179e-11, 6.1043e-06, 4.3111e-11, 1.3392e-13,\n",
      "         1.7839e-10, 1.8536e-04, 2.2136e-13, 4.4187e-08, 7.9148e-09, 4.4736e-05,\n",
      "         4.2465e-08, 6.8674e-14, 2.4256e-11, 7.0624e-09, 7.5976e-06, 8.1561e-13,\n",
      "         3.5870e-10, 9.3679e-05, 2.4501e-07, 9.9798e-01, 6.3286e-22, 1.0183e-03,\n",
      "         1.5513e-11, 6.6168e-04, 6.8573e-13],\n",
      "        [1.4303e-10, 3.4277e-11, 2.3510e-08, 7.7077e-06, 4.4119e-10, 7.6049e-16,\n",
      "         2.0679e-05, 1.0604e-03, 6.7308e-19, 1.0230e-07, 1.9033e-09, 9.5007e-08,\n",
      "         8.5972e-05, 2.1543e-11, 9.3168e-12, 7.9284e-08, 1.8930e-05, 4.9287e-08,\n",
      "         5.5779e-11, 1.5096e-05, 9.9570e-03, 9.7990e-01, 9.5359e-21, 8.9298e-03,\n",
      "         2.4781e-09, 2.7072e-07, 2.8879e-09],\n",
      "        [7.0101e-06, 9.4702e-12, 8.0358e-03, 1.4653e-06, 6.1683e-05, 1.2467e-12,\n",
      "         4.0543e-08, 1.6388e-02, 2.1657e-15, 1.4866e-01, 4.9776e-07, 1.7332e-02,\n",
      "         2.1897e-05, 1.0483e-08, 4.1978e-12, 5.7880e-07, 8.8065e-07, 2.2378e-08,\n",
      "         4.1769e-10, 5.3478e-02, 1.6699e-01, 5.8902e-01, 2.5693e-14, 5.9336e-06,\n",
      "         5.3910e-12, 5.9469e-07, 1.3925e-07],\n",
      "        [6.0934e-04, 2.7586e-10, 2.0590e-01, 1.6324e-07, 4.9159e-05, 1.1503e-10,\n",
      "         1.5000e-08, 4.5724e-06, 9.9365e-15, 7.8764e-01, 1.0477e-07, 6.5878e-05,\n",
      "         1.5454e-06, 2.0606e-08, 3.0080e-10, 2.2380e-07, 7.1668e-06, 4.2777e-08,\n",
      "         4.1171e-10, 1.6997e-03, 3.0880e-03, 7.9964e-04, 8.5110e-14, 1.1298e-04,\n",
      "         2.1385e-09, 2.0162e-05, 1.9774e-06],\n",
      "        [1.6378e-08, 1.2765e-07, 1.5040e-06, 1.5520e-07, 4.1036e-09, 6.8651e-10,\n",
      "         1.2497e-07, 4.5808e-07, 6.2187e-15, 5.4262e-04, 1.0405e-10, 6.9013e-15,\n",
      "         1.6246e-04, 6.5447e-08, 1.7526e-06, 4.1232e-10, 1.8209e-02, 1.5192e-12,\n",
      "         5.1422e-06, 2.6677e-07, 9.7884e-01, 6.0416e-06, 1.9757e-07, 2.7831e-07,\n",
      "         1.1419e-05, 4.9819e-07, 2.2149e-03],\n",
      "        [1.4358e-06, 1.7160e-08, 1.9915e-07, 2.7491e-03, 9.5430e-05, 1.2909e-08,\n",
      "         4.3505e-03, 8.0338e-02, 4.2430e-13, 5.9642e-04, 4.4268e-04, 2.9241e-02,\n",
      "         5.5615e-02, 3.0666e-06, 3.1424e-08, 1.3357e-07, 9.8850e-03, 1.5480e-07,\n",
      "         6.6309e-02, 8.8849e-03, 5.0772e-03, 5.1343e-01, 6.1277e-10, 1.2376e-02,\n",
      "         1.8606e-07, 2.1060e-01, 1.1516e-05],\n",
      "        [7.5674e-09, 7.7109e-13, 3.2179e-11, 6.1043e-06, 4.3111e-11, 1.3392e-13,\n",
      "         1.7839e-10, 1.8536e-04, 2.2136e-13, 4.4187e-08, 7.9148e-09, 4.4736e-05,\n",
      "         4.2465e-08, 6.8674e-14, 2.4256e-11, 7.0624e-09, 7.5976e-06, 8.1561e-13,\n",
      "         3.5870e-10, 9.3679e-05, 2.4501e-07, 9.9798e-01, 6.3286e-22, 1.0183e-03,\n",
      "         1.5513e-11, 6.6168e-04, 6.8573e-13],\n",
      "        [1.4303e-10, 3.4277e-11, 2.3510e-08, 7.7077e-06, 4.4119e-10, 7.6049e-16,\n",
      "         2.0679e-05, 1.0604e-03, 6.7308e-19, 1.0230e-07, 1.9033e-09, 9.5007e-08,\n",
      "         8.5972e-05, 2.1543e-11, 9.3168e-12, 7.9284e-08, 1.8930e-05, 4.9287e-08,\n",
      "         5.5779e-11, 1.5096e-05, 9.9570e-03, 9.7990e-01, 9.5359e-21, 8.9298e-03,\n",
      "         2.4781e-09, 2.7072e-07, 2.8879e-09],\n",
      "        [7.0101e-06, 9.4702e-12, 8.0358e-03, 1.4653e-06, 6.1683e-05, 1.2467e-12,\n",
      "         4.0543e-08, 1.6388e-02, 2.1657e-15, 1.4866e-01, 4.9776e-07, 1.7332e-02,\n",
      "         2.1897e-05, 1.0483e-08, 4.1978e-12, 5.7880e-07, 8.8065e-07, 2.2378e-08,\n",
      "         4.1769e-10, 5.3478e-02, 1.6699e-01, 5.8902e-01, 2.5693e-14, 5.9336e-06,\n",
      "         5.3910e-12, 5.9469e-07, 1.3925e-07],\n",
      "        [1.3725e-07, 5.6575e-06, 1.5050e-03, 1.5394e-07, 1.6846e-02, 2.6542e-09,\n",
      "         2.2128e-07, 1.2764e-04, 3.0446e-14, 9.6385e-01, 5.9596e-03, 4.2748e-06,\n",
      "         4.0173e-09, 1.6761e-09, 4.0064e-07, 9.7707e-10, 1.2248e-05, 2.0252e-09,\n",
      "         4.9966e-06, 2.9650e-03, 5.9249e-03, 4.8565e-05, 8.3378e-06, 2.1495e-03,\n",
      "         2.0763e-05, 2.4782e-05, 5.4583e-04],\n",
      "        [7.4421e-01, 7.8369e-13, 4.5720e-07, 2.5838e-04, 1.3493e-05, 5.7502e-05,\n",
      "         1.1103e-04, 1.0085e-09, 3.4225e-11, 3.8164e-04, 8.6432e-07, 6.5994e-03,\n",
      "         2.6372e-05, 9.5823e-08, 4.2243e-06, 1.6816e-05, 1.4439e-03, 5.0722e-09,\n",
      "         1.5853e-08, 3.7152e-04, 2.4083e-01, 2.5429e-03, 1.8516e-14, 7.8428e-09,\n",
      "         5.8131e-09, 3.1363e-03, 2.7415e-08],\n",
      "        [3.3647e-11, 4.6042e-07, 2.3166e-11, 2.4079e-08, 2.5426e-06, 7.0735e-15,\n",
      "         9.7557e-01, 1.6967e-06, 5.8305e-15, 3.6222e-09, 7.3446e-08, 8.1026e-13,\n",
      "         2.1960e-10, 4.3997e-09, 4.3784e-07, 8.6496e-08, 1.4400e-04, 2.8385e-10,\n",
      "         3.1254e-04, 5.0909e-08, 3.9299e-05, 2.2326e-08, 2.3283e-02, 4.9260e-07,\n",
      "         3.0039e-04, 6.5360e-08, 3.4332e-04],\n",
      "        [9.9905e-05, 2.2517e-13, 1.8694e-09, 9.8338e-03, 6.9224e-07, 1.2701e-06,\n",
      "         1.0904e-05, 3.2384e-07, 2.5247e-09, 1.8661e-06, 5.3193e-08, 8.1921e-04,\n",
      "         4.6982e-02, 8.9738e-08, 9.8661e-08, 6.7088e-08, 2.1740e-04, 1.7675e-09,\n",
      "         5.1918e-11, 9.4131e-02, 4.0252e-02, 1.6435e-01, 1.8421e-13, 9.6191e-05,\n",
      "         1.9446e-10, 6.4321e-01, 5.9979e-08],\n",
      "        [3.1804e-09, 5.4049e-05, 5.9561e-07, 7.3471e-05, 8.0479e-03, 1.0436e-17,\n",
      "         1.1227e-02, 1.4934e-03, 2.5316e-10, 2.2160e-08, 2.1189e-05, 9.1101e-08,\n",
      "         1.3567e-10, 1.3545e-05, 1.6442e-03, 9.4615e-03, 4.2494e-02, 1.0980e-04,\n",
      "         7.8312e-01, 9.5876e-05, 7.9414e-08, 6.7683e-08, 3.2936e-04, 4.4481e-03,\n",
      "         2.9249e-02, 5.4963e-02, 5.3157e-02],\n",
      "        [1.4358e-06, 1.7160e-08, 1.9915e-07, 2.7491e-03, 9.5430e-05, 1.2909e-08,\n",
      "         4.3505e-03, 8.0338e-02, 4.2430e-13, 5.9642e-04, 4.4268e-04, 2.9241e-02,\n",
      "         5.5615e-02, 3.0666e-06, 3.1424e-08, 1.3357e-07, 9.8850e-03, 1.5480e-07,\n",
      "         6.6309e-02, 8.8849e-03, 5.0772e-03, 5.1343e-01, 6.1277e-10, 1.2376e-02,\n",
      "         1.8606e-07, 2.1060e-01, 1.1516e-05],\n",
      "        [7.5674e-09, 7.7109e-13, 3.2179e-11, 6.1043e-06, 4.3111e-11, 1.3392e-13,\n",
      "         1.7839e-10, 1.8536e-04, 2.2136e-13, 4.4187e-08, 7.9148e-09, 4.4736e-05,\n",
      "         4.2465e-08, 6.8674e-14, 2.4256e-11, 7.0624e-09, 7.5976e-06, 8.1561e-13,\n",
      "         3.5870e-10, 9.3679e-05, 2.4501e-07, 9.9798e-01, 6.3286e-22, 1.0183e-03,\n",
      "         1.5513e-11, 6.6168e-04, 6.8573e-13],\n",
      "        [1.4303e-10, 3.4277e-11, 2.3510e-08, 7.7077e-06, 4.4119e-10, 7.6049e-16,\n",
      "         2.0679e-05, 1.0604e-03, 6.7308e-19, 1.0230e-07, 1.9033e-09, 9.5007e-08,\n",
      "         8.5972e-05, 2.1543e-11, 9.3168e-12, 7.9284e-08, 1.8930e-05, 4.9287e-08,\n",
      "         5.5779e-11, 1.5096e-05, 9.9570e-03, 9.7990e-01, 9.5359e-21, 8.9298e-03,\n",
      "         2.4781e-09, 2.7072e-07, 2.8879e-09],\n",
      "        [7.0101e-06, 9.4702e-12, 8.0358e-03, 1.4653e-06, 6.1683e-05, 1.2467e-12,\n",
      "         4.0543e-08, 1.6388e-02, 2.1657e-15, 1.4866e-01, 4.9776e-07, 1.7332e-02,\n",
      "         2.1897e-05, 1.0483e-08, 4.1978e-12, 5.7880e-07, 8.8065e-07, 2.2378e-08,\n",
      "         4.1769e-10, 5.3478e-02, 1.6699e-01, 5.8902e-01, 2.5693e-14, 5.9336e-06,\n",
      "         5.3910e-12, 5.9469e-07, 1.3925e-07],\n",
      "        [1.3725e-07, 5.6575e-06, 1.5050e-03, 1.5394e-07, 1.6846e-02, 2.6542e-09,\n",
      "         2.2128e-07, 1.2764e-04, 3.0446e-14, 9.6385e-01, 5.9596e-03, 4.2748e-06,\n",
      "         4.0173e-09, 1.6761e-09, 4.0064e-07, 9.7707e-10, 1.2248e-05, 2.0252e-09,\n",
      "         4.9966e-06, 2.9650e-03, 5.9249e-03, 4.8565e-05, 8.3378e-06, 2.1495e-03,\n",
      "         2.0763e-05, 2.4782e-05, 5.4583e-04],\n",
      "        [7.4421e-01, 7.8369e-13, 4.5720e-07, 2.5838e-04, 1.3493e-05, 5.7502e-05,\n",
      "         1.1103e-04, 1.0085e-09, 3.4225e-11, 3.8164e-04, 8.6432e-07, 6.5994e-03,\n",
      "         2.6372e-05, 9.5823e-08, 4.2243e-06, 1.6816e-05, 1.4439e-03, 5.0722e-09,\n",
      "         1.5853e-08, 3.7152e-04, 2.4083e-01, 2.5429e-03, 1.8516e-14, 7.8428e-09,\n",
      "         5.8131e-09, 3.1363e-03, 2.7415e-08],\n",
      "        [2.2387e-15, 1.1815e-09, 7.7545e-13, 2.4682e-08, 1.1277e-05, 2.7637e-12,\n",
      "         7.1588e-01, 4.4213e-03, 9.1996e-17, 6.6624e-10, 2.7957e-04, 8.8759e-07,\n",
      "         1.1927e-07, 3.6018e-14, 7.3535e-13, 7.6684e-09, 5.4073e-11, 8.7216e-11,\n",
      "         1.8078e-09, 2.5021e-06, 4.6197e-03, 2.7330e-01, 2.9361e-17, 1.4847e-03,\n",
      "         3.5144e-09, 2.8441e-14, 8.3052e-10],\n",
      "        [1.4358e-06, 1.7160e-08, 1.9915e-07, 2.7491e-03, 9.5430e-05, 1.2909e-08,\n",
      "         4.3505e-03, 8.0338e-02, 4.2430e-13, 5.9642e-04, 4.4268e-04, 2.9241e-02,\n",
      "         5.5615e-02, 3.0666e-06, 3.1424e-08, 1.3357e-07, 9.8850e-03, 1.5480e-07,\n",
      "         6.6309e-02, 8.8849e-03, 5.0772e-03, 5.1343e-01, 6.1277e-10, 1.2376e-02,\n",
      "         1.8606e-07, 2.1060e-01, 1.1516e-05],\n",
      "        [7.5674e-09, 7.7109e-13, 3.2179e-11, 6.1043e-06, 4.3111e-11, 1.3392e-13,\n",
      "         1.7839e-10, 1.8536e-04, 2.2136e-13, 4.4187e-08, 7.9148e-09, 4.4736e-05,\n",
      "         4.2465e-08, 6.8674e-14, 2.4256e-11, 7.0624e-09, 7.5976e-06, 8.1561e-13,\n",
      "         3.5870e-10, 9.3679e-05, 2.4501e-07, 9.9798e-01, 6.3286e-22, 1.0183e-03,\n",
      "         1.5513e-11, 6.6168e-04, 6.8573e-13],\n",
      "        [1.4303e-10, 3.4277e-11, 2.3510e-08, 7.7077e-06, 4.4119e-10, 7.6049e-16,\n",
      "         2.0679e-05, 1.0604e-03, 6.7308e-19, 1.0230e-07, 1.9033e-09, 9.5007e-08,\n",
      "         8.5972e-05, 2.1543e-11, 9.3168e-12, 7.9284e-08, 1.8930e-05, 4.9287e-08,\n",
      "         5.5779e-11, 1.5096e-05, 9.9570e-03, 9.7990e-01, 9.5359e-21, 8.9298e-03,\n",
      "         2.4781e-09, 2.7072e-07, 2.8879e-09],\n",
      "        [7.0101e-06, 9.4702e-12, 8.0358e-03, 1.4653e-06, 6.1683e-05, 1.2467e-12,\n",
      "         4.0543e-08, 1.6388e-02, 2.1657e-15, 1.4866e-01, 4.9776e-07, 1.7332e-02,\n",
      "         2.1897e-05, 1.0483e-08, 4.1978e-12, 5.7880e-07, 8.8065e-07, 2.2378e-08,\n",
      "         4.1769e-10, 5.3478e-02, 1.6699e-01, 5.8902e-01, 2.5693e-14, 5.9336e-06,\n",
      "         5.3910e-12, 5.9469e-07, 1.3925e-07],\n",
      "        [1.3725e-07, 5.6575e-06, 1.5050e-03, 1.5394e-07, 1.6846e-02, 2.6542e-09,\n",
      "         2.2128e-07, 1.2764e-04, 3.0446e-14, 9.6385e-01, 5.9596e-03, 4.2748e-06,\n",
      "         4.0173e-09, 1.6761e-09, 4.0064e-07, 9.7707e-10, 1.2248e-05, 2.0252e-09,\n",
      "         4.9966e-06, 2.9650e-03, 5.9249e-03, 4.8565e-05, 8.3378e-06, 2.1495e-03,\n",
      "         2.0763e-05, 2.4782e-05, 5.4583e-04],\n",
      "        [7.4421e-01, 7.8369e-13, 4.5720e-07, 2.5838e-04, 1.3493e-05, 5.7502e-05,\n",
      "         1.1103e-04, 1.0085e-09, 3.4225e-11, 3.8164e-04, 8.6432e-07, 6.5994e-03,\n",
      "         2.6372e-05, 9.5823e-08, 4.2243e-06, 1.6816e-05, 1.4439e-03, 5.0722e-09,\n",
      "         1.5853e-08, 3.7152e-04, 2.4083e-01, 2.5429e-03, 1.8516e-14, 7.8428e-09,\n",
      "         5.8131e-09, 3.1363e-03, 2.7415e-08],\n",
      "        [5.1420e-10, 3.3176e-09, 2.7930e-08, 8.1899e-07, 2.4230e-08, 7.5007e-14,\n",
      "         9.9052e-01, 1.0938e-06, 8.7985e-15, 3.6669e-09, 4.0244e-09, 1.9397e-09,\n",
      "         4.6590e-07, 3.3612e-08, 1.1964e-10, 1.0692e-07, 8.1176e-06, 3.3563e-11,\n",
      "         9.5412e-06, 1.6473e-07, 3.7696e-05, 8.9375e-03, 6.9983e-14, 4.2448e-04,\n",
      "         2.2127e-08, 1.3702e-09, 5.5592e-05],\n",
      "        [1.4358e-06, 1.7160e-08, 1.9915e-07, 2.7491e-03, 9.5430e-05, 1.2909e-08,\n",
      "         4.3505e-03, 8.0338e-02, 4.2430e-13, 5.9642e-04, 4.4268e-04, 2.9241e-02,\n",
      "         5.5615e-02, 3.0666e-06, 3.1424e-08, 1.3357e-07, 9.8850e-03, 1.5480e-07,\n",
      "         6.6309e-02, 8.8849e-03, 5.0772e-03, 5.1343e-01, 6.1277e-10, 1.2376e-02,\n",
      "         1.8606e-07, 2.1060e-01, 1.1516e-05],\n",
      "        [7.5674e-09, 7.7109e-13, 3.2179e-11, 6.1043e-06, 4.3111e-11, 1.3392e-13,\n",
      "         1.7839e-10, 1.8536e-04, 2.2136e-13, 4.4187e-08, 7.9148e-09, 4.4736e-05,\n",
      "         4.2465e-08, 6.8674e-14, 2.4256e-11, 7.0624e-09, 7.5976e-06, 8.1561e-13,\n",
      "         3.5870e-10, 9.3679e-05, 2.4501e-07, 9.9798e-01, 6.3286e-22, 1.0183e-03,\n",
      "         1.5513e-11, 6.6168e-04, 6.8573e-13],\n",
      "        [1.4303e-10, 3.4277e-11, 2.3510e-08, 7.7077e-06, 4.4119e-10, 7.6049e-16,\n",
      "         2.0679e-05, 1.0604e-03, 6.7308e-19, 1.0230e-07, 1.9033e-09, 9.5007e-08,\n",
      "         8.5972e-05, 2.1543e-11, 9.3168e-12, 7.9284e-08, 1.8930e-05, 4.9287e-08,\n",
      "         5.5779e-11, 1.5096e-05, 9.9570e-03, 9.7990e-01, 9.5359e-21, 8.9298e-03,\n",
      "         2.4781e-09, 2.7072e-07, 2.8879e-09],\n",
      "        [7.0101e-06, 9.4702e-12, 8.0358e-03, 1.4653e-06, 6.1683e-05, 1.2467e-12,\n",
      "         4.0543e-08, 1.6388e-02, 2.1657e-15, 1.4866e-01, 4.9776e-07, 1.7332e-02,\n",
      "         2.1897e-05, 1.0483e-08, 4.1978e-12, 5.7880e-07, 8.8065e-07, 2.2378e-08,\n",
      "         4.1769e-10, 5.3478e-02, 1.6699e-01, 5.8902e-01, 2.5693e-14, 5.9336e-06,\n",
      "         5.3910e-12, 5.9469e-07, 1.3925e-07],\n",
      "        [1.3725e-07, 5.6575e-06, 1.5050e-03, 1.5394e-07, 1.6846e-02, 2.6542e-09,\n",
      "         2.2128e-07, 1.2764e-04, 3.0446e-14, 9.6385e-01, 5.9596e-03, 4.2748e-06,\n",
      "         4.0173e-09, 1.6761e-09, 4.0064e-07, 9.7707e-10, 1.2248e-05, 2.0252e-09,\n",
      "         4.9966e-06, 2.9650e-03, 5.9249e-03, 4.8565e-05, 8.3378e-06, 2.1495e-03,\n",
      "         2.0763e-05, 2.4782e-05, 5.4583e-04],\n",
      "        [2.3415e-01, 4.0781e-13, 9.4099e-09, 3.5345e-03, 4.3175e-07, 3.3253e-08,\n",
      "         1.4092e-07, 2.7223e-09, 1.2498e-11, 2.3706e-05, 1.7731e-07, 4.2620e-02,\n",
      "         9.0720e-09, 1.2019e-10, 8.8687e-08, 7.0509e-06, 3.4496e-03, 1.3111e-09,\n",
      "         1.2067e-11, 1.1230e-03, 2.0960e-01, 4.9511e-01, 2.1777e-17, 1.3245e-07,\n",
      "         3.4633e-12, 1.0378e-02, 3.3765e-11],\n",
      "        [1.0242e-11, 3.7001e-11, 4.1124e-08, 5.0086e-06, 3.2832e-10, 2.8284e-15,\n",
      "         9.0091e-04, 4.6072e-04, 6.6622e-16, 2.2958e-08, 3.3166e-08, 6.2231e-08,\n",
      "         2.3858e-04, 4.0365e-12, 3.8447e-11, 1.6779e-04, 2.5712e-08, 6.1860e-08,\n",
      "         1.0032e-08, 1.2390e-04, 1.2565e-04, 4.3408e-02, 1.0433e-21, 9.5457e-01,\n",
      "         1.0705e-09, 1.2144e-09, 2.9994e-10],\n",
      "        [2.6190e-09, 1.2264e-06, 4.4965e-04, 3.4992e-08, 6.0794e-08, 9.0150e-11,\n",
      "         4.4773e-09, 6.8263e-07, 5.7907e-16, 2.4479e-02, 1.5497e-06, 2.0905e-12,\n",
      "         1.2581e-09, 3.8911e-07, 1.8035e-06, 6.9725e-10, 8.7543e-05, 5.4432e-11,\n",
      "         5.7226e-06, 8.3575e-06, 9.5746e-01, 2.0154e-06, 3.6115e-03, 5.6115e-07,\n",
      "         1.2419e-02, 4.9021e-07, 1.4697e-03]])\n"
     ]
    }
   ],
   "source": [
    "# Let's see the probabilities in action\n",
    "print(probabilities)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5cb74251-8863-49f7-b238-60ba73e65f39",
   "metadata": {},
   "source": [
    "Now that we have our probabilities,\\\n",
    "We also want to:\n",
    "1. Calculate Loss\n",
    "2. Tune the particular weights depending on the gradients\n",
    "\n",
    "So just like our older <a href=\"https://github.com/AvishakeAdhikary/Neural-Networks-From-Scratch/blob/main/NameWeave.ipynb\">NameWeave Notebook</a>, we will calculate loss such that,\n",
    "1. We will take the log likelihood of the probabilities\n",
    "2. Take the average of the log likelihood\n",
    "3. Convert it to negetive average log likelihood"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "839393e7-6ca3-4c89-aca1-ee6dd18c467d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(20.5695)\n"
     ]
    }
   ],
   "source": [
    "# Let's calculate our own average negetive log likelihood straight from this tensor\n",
    "# This is the vectorized form of that expression\n",
    "loss = -probabilities[torch.arange(len(inputs)), outputs].log().mean()\n",
    "print(loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e29a6fe-f1bf-483c-a4f0-5e1bebe4d44b",
   "metadata": {},
   "source": [
    "Let's make our code a little more respectable...\n",
    "\n",
    "What is **entropy**?\n",
    "\n",
    "Entropy is the measurement of disorder or impurities in the information processed in machine learning.\n",
    "\n",
    "$$ H(X) = -\\sum_{x \\in \\mathcal{X}} p(x) \\log p(x) $$\n",
    "\n",
    "**When entropy becomes 0, then the dataset has no impurity.**\n",
    "\n",
    "![LowHighEntropy](https://static.javatpoint.com/tutorial/machine-learning/images/entropy-in-machine-learning3.png)\n",
    "\n",
    "What is the **Information Gain in Entropy**?\n",
    "\n",
    "Information gain is defined as the pattern observed in the dataset by calculating the reduction in entropy or surprise by splitting a dataset according to a given value of a random variable.\n",
    "\n",
    "$$ \\text{Information Gain} = 1-\\text{Entropy} $$\n",
    "\n",
    "![LowHighInformationGain](https://miro.medium.com/v2/resize:fit:1400/1*DsjX_bHYWn21Z0VIPjxnbw.png)\n",
    "\n",
    "What is **Cross-Entropy**?\n",
    "\n",
    "Cross-entropy is a measure of the difference between two probability distributions for a given random variable or set of events.\n",
    "\n",
    "What is **Cross-Entropy Loss**?\n",
    "\n",
    "Cross-entropy loss refers to the contrast between two random variables. It measures the variables to extract the difference in the information they contain, showcasing the results."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf303df1-2d03-409e-b519-8e48a24174bc",
   "metadata": {},
   "source": [
    "So,\\\n",
    "We now understand that our lines of code:\n",
    "```python\n",
    "# Calculating counts from logits\n",
    "counts = logits.exp()\n",
    "# Normalizing counts to probabilities that sum to 1\n",
    "probabilities = counts / counts.sum(1, keepdims=True)\n",
    "# Calculating the negetive average log likelihood as loss\n",
    "loss = -probabilities[torch.arange(len(inputs)), outputs].log().mean()\n",
    "```\n",
    "\n",
    "Can now be replaced by a ready-made function:\n",
    "```python\n",
    "loss = F.cross_entropy(logits, outputs)\n",
    "```\n",
    "\n",
    "Since we are doing classification..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "874bd94c-9f96-4bbc-bcd9-589eef109b40",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(20.5695)\n"
     ]
    }
   ],
   "source": [
    "# Calculating the negetive average log likelihood as loss (cross entropy loss)\n",
    "loss = F.cross_entropy(logits, outputs)\n",
    "# We get the same loss now\n",
    "print(loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d1f25c4-b6c4-40f9-9184-ab452bba55bd",
   "metadata": {},
   "source": [
    "Now, why are we using *F.cross_entropy()* might be the next question coming to your mind...\n",
    "\n",
    "There are a number of reasons, as to why,\n",
    "1. Its efficient and does not require to use new memory\n",
    "2. We practically don't use the three lines of code\n",
    "\n",
    "Now you might be wondering why my last point is valid.\\\n",
    "Let me explain...\n",
    "\n",
    "Suppose we write the same code:\n",
    "```python\n",
    "#Defining logits (example)\n",
    "logits = torch.tensor([-100, -3, 0, 100])\n",
    "# Calculating counts from logits\n",
    "counts = logits.exp()\n",
    "# Normalizing counts to probabilities that sum to 1\n",
    "probabilities = counts / counts.sum(1, keepdims=True)\n",
    "# Calculating the negetive average log likelihood as loss\n",
    "loss = -probabilities[torch.arange(len(inputs)), outputs].log().mean()\n",
    "```\n",
    "\n",
    "Even if the code seems simple, we run into problems...\\\n",
    "Because when we represent counts by performing an exponential function,\\\n",
    "Any large negetive number works fine(it represents a very tiny number),\\\n",
    "But the moment we associate a very positive number along with it(it tries to represent a very large number) and it goes out of memory and count turns out to be inifinity and probabilities remain undefined as well.\n",
    "\n",
    "So summarising the reasons:\n",
    "1. The forward pass would be much more efficient\n",
    "2. The backward pass would be much more efficient\n",
    "3. The calculations would be mathematically well behaved"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75dbeb27-c45a-48eb-9ae4-bf5ae0dffbb4",
   "metadata": {},
   "source": [
    "##### Now that we have all the layers, let's put all the parameters in a single variable to access all of them way faster\n",
    "\n",
    "Let's recall what those were:\n",
    "1. The *weights and biases* of the *output layer*\n",
    "2. The *weights and biases* of the *hidden layer*\n",
    "3. The *embedding look-up table 'C'*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "84f04d1e-a826-454b-84a8-16388a4698c1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[tensor([[-0.3732, -0.2275],\n",
      "        [-1.7182, -0.8157],\n",
      "        [-1.1759, -1.5987],\n",
      "        [-0.3984,  0.0547],\n",
      "        [ 0.1003, -0.8822],\n",
      "        [-2.1286, -1.6474],\n",
      "        [-0.5959,  0.3235],\n",
      "        [ 0.5287, -0.4012],\n",
      "        [ 0.5799, -0.7293],\n",
      "        [-1.4172, -2.0226],\n",
      "        [-0.7001,  0.6551],\n",
      "        [ 0.3579, -0.7145],\n",
      "        [ 0.8300, -1.0691],\n",
      "        [ 1.6646, -0.9261],\n",
      "        [ 0.0049,  0.9844],\n",
      "        [ 0.8385, -1.9852],\n",
      "        [ 0.7729,  0.5822],\n",
      "        [ 0.8481,  0.7299],\n",
      "        [ 0.9389,  0.7735],\n",
      "        [ 0.2951, -1.8969],\n",
      "        [-0.6994,  0.2413],\n",
      "        [-0.7727, -0.5433],\n",
      "        [ 1.0702,  0.7694],\n",
      "        [-1.2376, -0.0611],\n",
      "        [ 0.2379, -0.9606],\n",
      "        [ 0.2195, -0.5485],\n",
      "        [-0.2583, -0.4831]]), tensor([[ 0.3253, -1.7963,  0.9002, -0.0506, -1.0483,  0.0677, -0.2041, -0.8192,\n",
      "          0.5607, -0.2761, -0.1341,  1.1116,  1.0652, -2.0130, -0.6983, -0.5113,\n",
      "         -1.4599, -0.4864, -1.1663, -0.7709,  0.7650,  0.2706, -0.0645, -0.1961,\n",
      "         -0.0088,  2.1184, -0.0639, -0.6691, -0.4061,  0.3014, -0.6655, -2.3377,\n",
      "         -0.6532, -1.5500, -0.3734, -0.4678, -0.3563,  1.0529, -0.5916,  0.9534,\n",
      "         -0.1902,  1.1558, -1.8075,  0.0784, -1.5069, -0.5043, -1.2630, -1.7298,\n",
      "          0.2158,  0.1342,  0.2774, -1.0065, -0.3800, -0.3990,  0.6736, -2.2711,\n",
      "          0.3253,  0.6117, -0.4390, -0.0386, -1.5478,  0.7908, -0.6540,  0.5204,\n",
      "          1.0704, -0.8689,  0.6116,  0.6260, -0.2554,  0.6619,  0.9044,  0.4440,\n",
      "         -1.7827,  0.3408,  0.1396, -1.0577,  0.3800,  0.4604, -0.7846,  0.3993,\n",
      "          0.3198, -1.1531,  0.0648, -0.6514, -1.0848, -0.6183,  0.3103, -0.2779,\n",
      "          0.8100, -0.0516,  0.9636,  0.9552,  0.6490, -2.3521,  0.4529,  0.4301,\n",
      "         -0.1761, -1.1897, -0.8651,  0.9745],\n",
      "        [-2.0270,  0.7764,  0.5025, -2.7880, -0.3531, -0.6706, -0.5859, -0.3866,\n",
      "          0.4326, -0.5093,  0.3768, -0.8109, -0.0320, -0.9543, -2.9476,  1.0877,\n",
      "         -0.8908,  0.9021,  1.2282,  1.6197, -0.0045,  0.4460, -0.5955,  0.5873,\n",
      "          1.9542,  0.9056,  0.5373,  0.0232,  0.3290, -0.3733,  0.7902, -0.3607,\n",
      "          0.8246, -2.2372,  0.1764, -0.0818, -2.1996, -0.4270,  0.1776, -0.7351,\n",
      "         -0.7968, -1.3857,  1.8651, -1.3105, -0.6720,  1.7917,  0.1947, -0.4448,\n",
      "          1.7149, -0.1409,  0.6268,  0.4266,  0.7166, -1.6857, -0.3260, -0.2270,\n",
      "         -0.9428, -0.2051, -0.9676, -0.1834,  0.4121, -0.5908, -0.1401, -2.2907,\n",
      "          0.8480, -0.0589, -0.4097,  0.1406, -0.0607, -0.2859,  0.6473,  0.2460,\n",
      "         -1.6245,  0.1829, -0.0983, -0.7482, -0.8846,  2.4782, -0.0678,  1.0859,\n",
      "         -0.3905,  1.0937,  0.2140, -0.9901, -0.1689,  1.8458,  0.2901,  2.5761,\n",
      "          0.9772,  0.7983, -0.6576,  0.2139,  0.1970, -1.5143, -0.1406,  0.1551,\n",
      "         -0.1901,  1.8309, -0.6698,  1.0180],\n",
      "        [ 2.2765, -0.5772, -1.4159,  0.5679,  2.4357,  0.1884, -2.0103,  0.3697,\n",
      "          0.0696,  0.9205, -0.1341,  0.0421, -2.4502, -0.1431,  0.2532,  1.0379,\n",
      "         -0.9107,  0.5580, -2.3553,  0.1760,  2.0263, -0.0893,  0.7702, -0.5056,\n",
      "          0.2018,  0.4922,  0.0623,  1.5610,  0.4613, -1.1995,  0.6817,  0.2415,\n",
      "         -0.9572, -0.6360,  0.3272,  0.5775, -0.8813, -0.0577, -0.4364,  0.4293,\n",
      "          0.0535, -0.3417,  1.4204,  0.4525, -0.5042, -0.5534,  0.1652, -1.3800,\n",
      "          1.5989,  0.1098, -0.6826,  1.3309, -0.2948,  0.5442, -0.0306,  0.2794,\n",
      "         -0.3480, -1.0727,  0.9713, -1.0002, -0.2194, -0.8349,  0.6753, -0.7797,\n",
      "          0.5094, -1.0917,  0.1854, -0.0839,  0.2670, -0.2998, -1.6465,  0.4957,\n",
      "          0.2984, -1.7949, -0.1690, -0.6914,  0.2138, -0.5378,  0.7550, -0.2136,\n",
      "         -2.2604,  1.2889,  1.3820,  0.0041, -0.7035, -1.2726, -0.6640, -1.2690,\n",
      "         -0.2346, -1.0908,  0.8679, -0.3419, -0.2512,  0.9939, -0.6441, -1.0428,\n",
      "          1.0767, -1.2498,  0.0209,  0.9478],\n",
      "        [ 2.5329,  0.4026, -0.4775, -1.8783,  0.6014,  1.7065, -0.3684, -2.2805,\n",
      "          1.9421, -0.2666,  0.0661, -0.0487, -1.4695,  0.2857,  0.1127,  1.5833,\n",
      "         -0.8428,  0.8070,  1.4438, -0.3507, -0.5184, -1.1354, -2.8418,  1.2830,\n",
      "          0.4503,  2.5735,  0.8603, -0.7889,  0.1155,  0.1430,  0.6110,  0.6947,\n",
      "          0.0913,  1.0173,  0.3578,  0.0662,  0.4010,  0.1967,  1.1017,  1.0792,\n",
      "         -1.8830, -2.2918,  0.5187, -1.2804, -0.2759,  1.0584, -0.6777,  0.0890,\n",
      "         -1.6241, -0.6414,  0.0668,  1.5748,  0.7296, -0.3524, -0.7367, -1.2800,\n",
      "         -1.6725, -0.4029,  0.8398, -0.2452, -1.5930,  1.6563,  0.3860,  0.3066,\n",
      "         -0.2326,  0.0761,  1.6718,  0.9396,  0.8653, -1.3714,  0.7183,  0.8673,\n",
      "         -0.1685,  0.8150, -0.0465, -0.0568, -0.2586,  0.2965,  0.7808, -0.6115,\n",
      "         -0.0549,  2.0605,  0.9227, -1.3012,  0.0135, -0.4343, -0.9293, -1.9553,\n",
      "          0.5833, -0.3209,  0.4992,  0.6933, -0.5454,  0.7065,  0.5401, -1.5465,\n",
      "          1.4081,  0.3288, -0.2466,  1.1426],\n",
      "        [-0.5931, -0.3759, -0.2924, -1.4976, -0.8700, -1.6314,  1.4267, -0.6473,\n",
      "         -0.7721,  0.9052,  0.3517, -0.9731, -1.8326,  0.9582, -0.9671, -0.8409,\n",
      "          0.6208, -0.0809, -0.5280,  0.0591, -0.7220,  1.8340,  1.1444,  0.6795,\n",
      "          0.7369,  0.3258,  0.1301, -0.7508,  1.5058, -0.3541,  0.3189, -0.3061,\n",
      "         -0.7118,  0.9601, -0.5787,  0.5127,  0.1928, -0.3776, -0.1846, -0.1118,\n",
      "          1.9117,  0.1675, -0.2321, -0.0883, -1.1887, -0.1167,  0.3543, -0.1874,\n",
      "         -1.1262, -1.3003,  2.0609, -1.2051, -1.0341,  2.0798,  0.2026,  0.6191,\n",
      "         -0.1576,  0.7311,  0.0796,  0.1705, -0.4311,  0.9881,  0.6063, -0.0463,\n",
      "         -0.9656, -1.5963,  0.4850,  0.2678, -0.3998, -1.0735, -0.1273, -1.1492,\n",
      "         -0.8071,  2.6509,  0.1632,  0.1967,  1.2197,  0.4254,  0.3054,  0.2250,\n",
      "         -0.1558, -1.1263,  0.0285,  0.6536,  2.3054, -0.4622,  1.8106,  1.2791,\n",
      "          0.1186, -0.5048, -0.7597,  0.6161, -0.2943, -0.0331,  1.5077,  1.5164,\n",
      "          1.5549, -0.4467, -0.9512, -0.7331],\n",
      "        [-0.2225, -0.1127,  0.3987, -0.5345, -1.5944, -1.4433,  0.6771, -0.1130,\n",
      "         -1.6564,  0.3247,  0.7978,  0.2449,  1.3633,  1.9971,  1.7443, -0.2060,\n",
      "         -2.1894, -1.3015, -0.3570, -0.9840,  0.4396,  0.3117,  2.9272, -0.1514,\n",
      "          0.2980,  0.0927,  1.4653,  0.4201,  0.8132, -0.0636,  1.2224,  0.0380,\n",
      "          0.8355, -0.0495, -0.3504,  0.8630,  0.9640, -0.1010,  1.9875, -0.6659,\n",
      "         -0.0603,  0.5217,  0.8011,  1.5359, -0.6541,  1.3274, -0.4473,  0.4164,\n",
      "          1.2883, -0.7897, -0.5686, -0.3483,  0.4220, -1.0567, -1.0715,  0.6182,\n",
      "         -0.2866,  0.1636,  0.9273,  0.5883, -0.6760, -0.4300, -0.1312,  0.3646,\n",
      "         -1.1655,  1.4313, -0.7845,  0.2171, -2.2251, -1.2033, -1.2807, -1.5196,\n",
      "         -1.4576,  0.2908,  1.2843,  1.1384,  0.6731,  0.0784, -0.0116,  1.5189,\n",
      "          0.7644, -0.0167,  0.3170, -0.2887,  1.3273, -0.7960,  1.7524,  0.5046,\n",
      "          0.9814,  0.5444, -0.4836,  0.6656, -1.2227,  0.8604, -0.0540,  0.3301,\n",
      "          0.0263,  0.2666, -0.8354, -0.5786]]), tensor([-1.3627e+00, -2.0471e-01,  6.6957e-01, -5.2312e-01,  1.3615e-01,\n",
      "         1.6473e+00,  1.9338e-01,  3.3181e-01,  9.6356e-01, -9.6966e-01,\n",
      "        -3.8450e-01,  1.2209e-01, -2.4604e-01, -7.2170e-01, -1.4737e+00,\n",
      "        -6.0932e-01, -1.3219e+00,  2.0788e-01, -8.7827e-01,  3.0460e-01,\n",
      "        -9.7417e-01, -4.4327e-01, -6.6013e-01, -1.6956e+00,  5.5653e-01,\n",
      "        -8.5778e-01, -4.7371e-01,  6.0429e-01,  7.8260e-01,  7.5766e-01,\n",
      "        -1.8869e+00, -1.0616e+00, -1.5197e+00,  4.3492e-02, -7.6052e-01,\n",
      "        -1.2244e+00,  1.5624e-01,  1.0968e-01, -2.2864e+00,  3.7539e-01,\n",
      "         1.1633e-01,  8.4283e-01, -3.9827e-02, -5.8075e-01,  6.9952e-01,\n",
      "         1.2940e+00, -1.7236e+00, -2.9627e-01, -9.5724e-01, -9.3180e-01,\n",
      "        -8.0067e-01, -9.7252e-02,  1.5420e+00, -1.1626e+00,  9.7466e-01,\n",
      "         6.1648e-01, -1.3809e+00, -1.2411e+00,  4.1926e-01,  1.1795e-03,\n",
      "         5.7692e-01, -1.5215e+00,  1.6249e+00, -1.5533e+00, -1.0004e+00,\n",
      "         4.9203e-01, -9.6523e-01,  2.0473e+00,  9.1949e-01,  4.6598e-01,\n",
      "         8.8969e-01, -2.0309e+00, -4.3946e-01,  6.1225e-01,  4.1889e-02,\n",
      "         1.8714e+00, -6.8883e-01, -1.2433e-01,  1.0643e+00, -1.5914e+00,\n",
      "        -8.4742e-01,  2.2137e+00,  2.4029e-01, -8.4179e-01, -4.0617e-01,\n",
      "         5.9072e-02,  3.6342e-02, -1.3165e+00,  7.2829e-02, -1.0186e+00,\n",
      "        -3.5708e-01,  1.1629e-01, -6.7578e-01,  1.7435e-01, -1.5200e+00,\n",
      "        -4.3552e-01,  9.3752e-01,  7.7250e-02, -1.3670e-02,  1.2180e+00]), tensor([[ 1.0500, -0.3421, -0.5875,  ..., -0.6107,  0.9515, -1.1065],\n",
      "        [ 0.5758, -0.9495, -0.6587,  ..., -0.0939,  0.9703, -0.6361],\n",
      "        [-1.1180,  0.5688,  0.9076,  ...,  1.3352,  1.0932,  0.7802],\n",
      "        ...,\n",
      "        [-0.9220, -0.4180,  1.3515,  ..., -0.3228, -0.2162,  0.9086],\n",
      "        [ 0.4249, -0.0322,  0.2384,  ..., -0.8189, -1.2185,  0.6995],\n",
      "        [-0.6781,  1.1895, -1.8605,  ..., -0.7457,  1.7607,  0.2912]]), tensor([-0.6622,  0.3047, -0.4674, -0.8913,  0.4272, -1.2311,  1.7426,  0.1682,\n",
      "        -0.8730,  0.4021,  0.7651,  0.1106, -1.6317, -1.2776,  0.4806,  0.5602,\n",
      "        -0.5036, -1.5232, -3.1612,  0.8269,  1.1389, -0.5887, -1.9254, -1.1958,\n",
      "         0.5439,  1.8284,  2.7294])]\n"
     ]
    }
   ],
   "source": [
    "# Let's define our parameters variable\n",
    "parameters = [embeddingLookUpMatrix, weightsOfHiddenLayer, biasesOfHiddenLayer, weightsOfFinalLayer, biasesOfFinalLayer]\n",
    "print(parameters)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "281d509a-08f3-4344-93ea-e5098df77d90",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We must set requires_grad to True in all the parameters to avoid any errors in the future\n",
    "for parameter in parameters:\n",
    "    parameter.requires_grad = True"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e37420ba-00f2-44f1-b1e4-03c84906af13",
   "metadata": {},
   "source": [
    "Let's put everything we have together for now with a respectable generator so that we all get the same output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "dc779344-bbd3-4efa-9cd4-89c64e012392",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We will define a generator to give the same result on your machine, as of my machine\n",
    "generator = torch.Generator().manual_seed(6942069420)\n",
    "# Embedding Matrix (Input Layer)\n",
    "embeddingFeatureSpaceLength = 2\n",
    "embeddingLookUpMatrix = torch.randn((len(stoi),embeddingFeatureSpaceLength), generator=generator)\n",
    "# Hidden Layer\n",
    "numberOfHiddenLayerNeurons = 100\n",
    "weightsOfHiddenLayer = torch.randn((inputBlockSize*embeddingFeatureSpaceLength), numberOfHiddenLayerNeurons, generator=generator)\n",
    "biasesOfHiddenLayer = torch.randn(numberOfHiddenLayerNeurons, generator=generator)\n",
    "# Output Layer / Final Layer\n",
    "numberOfFinalLayerOutputs = 27\n",
    "weightsOfFinalLayer = torch.randn(numberOfHiddenLayerNeurons, numberOfFinalLayerOutputs, generator=generator)\n",
    "biasesOfFinalLayer = torch.randn(numberOfFinalLayerOutputs, generator=generator)\n",
    "# Parameters\n",
    "parameters = [embeddingLookUpMatrix, weightsOfHiddenLayer, biasesOfHiddenLayer, weightsOfFinalLayer, biasesOfFinalLayer]\n",
    "for parameter in parameters:\n",
    "    parameter.requires_grad = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "9a007f79-ff1e-43ef-9179-71dd0fbb0ff0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3481"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's check how many parameters we have\n",
    "sum(parameter.nelement() for parameter in parameters)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca4e78cc-285d-490a-b711-81d14d94cbc0",
   "metadata": {},
   "source": [
    "Let's understand how neural network will train itself with forward pass, backward pass and updatation now...\n",
    "\n",
    "Now that we have trained two neural networks already...\n",
    "\n",
    "We can safely say that they work in the sequence:\n",
    "1. Forward Pass - Makes calculations and calculates loss\n",
    "2. Backward Pass - Resets all the gradients and back propagtes through the network\n",
    "3. Data Updation - Updates the data for all the parameters in the opposite direction of the gradient depending on the learning rate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "a78f1bd9-3e12-4536-9dd0-d229744cd2c0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loss: tensor(17.2358, grad_fn=<NllLossBackward0>)\n"
     ]
    }
   ],
   "source": [
    "# Forward Pass\n",
    "embedding = embeddingLookUpMatrix[inputs]\n",
    "hiddenLayerStates = torch.tanh(embedding.view(-1, inputBlockSize*embeddingFeatureSpaceLength) @ weightsOfHiddenLayer + biasesOfHiddenLayer)\n",
    "logits = hiddenLayerStates @ weightsOfFinalLayer + biasesOfFinalLayer\n",
    "loss = F.cross_entropy(logits, outputs)\n",
    "print(\"Loss:\", loss)\n",
    "\n",
    "# Backward Pass\n",
    "for parameter in parameters:\n",
    "    parameter.grad = None\n",
    "loss.backward()\n",
    "\n",
    "# Update Weights\n",
    "learning_rate = 0.1\n",
    "for parameter in parameters:\n",
    "    parameter.data += -learning_rate * parameter.grad"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7b2c4f2-e6bd-469b-8a65-1796825f0234",
   "metadata": {},
   "source": [
    "So we can take it in a loop to train the model over and over..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "f3b6943d-9fa1-48ae-96ac-27ed1702b02f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loss: tensor(11.6850, grad_fn=<NllLossBackward0>)\n",
      "Loss: tensor(7.4660, grad_fn=<NllLossBackward0>)\n",
      "Loss: tensor(5.0195, grad_fn=<NllLossBackward0>)\n",
      "Loss: tensor(4.3795, grad_fn=<NllLossBackward0>)\n",
      "Loss: tensor(3.9817, grad_fn=<NllLossBackward0>)\n",
      "Loss: tensor(3.6464, grad_fn=<NllLossBackward0>)\n",
      "Loss: tensor(3.4343, grad_fn=<NllLossBackward0>)\n",
      "Loss: tensor(3.2572, grad_fn=<NllLossBackward0>)\n",
      "Loss: tensor(2.7560, grad_fn=<NllLossBackward0>)\n",
      "Loss: tensor(2.4867, grad_fn=<NllLossBackward0>)\n"
     ]
    }
   ],
   "source": [
    "# We define the number of epochs\n",
    "epochs = 10\n",
    "for _ in range(epochs):\n",
    "    # Forward Pass\n",
    "    embedding = embeddingLookUpMatrix[inputs]\n",
    "    hiddenLayerStates = torch.tanh(embedding.view(-1, inputBlockSize*embeddingFeatureSpaceLength) @ weightsOfHiddenLayer + biasesOfHiddenLayer)\n",
    "    logits = hiddenLayerStates @ weightsOfFinalLayer + biasesOfFinalLayer\n",
    "    loss = F.cross_entropy(logits, outputs)\n",
    "    print(\"Loss:\", loss)\n",
    "    \n",
    "    # Backward Pass\n",
    "    for parameter in parameters:\n",
    "        parameter.grad = None\n",
    "    loss.backward()\n",
    "    \n",
    "    # Update Weights\n",
    "    learning_rate = 0.1\n",
    "    for parameter in parameters:\n",
    "        parameter.data += -learning_rate * parameter.grad"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ddc97c9c-91c3-4a42-bcff-d23dea52e0b9",
   "metadata": {},
   "source": [
    "See how our loss comes down so fast...\n",
    "\n",
    "That's because we are **overfitting** our model...\n",
    "\n",
    "How you ask?\n",
    "\n",
    "That's because we are considering only 5 names (37 input examples) at the moment for 3479 parameters.\\\n",
    "Thus, the result of fitting a very less amount of data for a very large number of parameters.\n",
    "\n",
    "How come our loss does not come to exactly 0 then?\n",
    "\n",
    "The answer lies within the first input example of each name...\\\n",
    "Remember how we are fitting:\n",
    "- ... -> a (first input example of a name starting with a)\n",
    "- ... -> c (first input example of a name starting with c)\n",
    "- ... -> y (first input example of a name starting with y)\n",
    "and so on...\n",
    "\n",
    "So we are not completely overfitting the model, but rather we are overfitting the unique inputs for unique outputs...\n",
    "\n",
    "So let's consider the entire dataset now and optimize the neural network..."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "076e0dbf-93a0-4fad-b222-c4465e83eba1",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "# Optimizing Neural Network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "2e0d5047-5e95-4244-9181-51280b34a495",
   "metadata": {},
   "outputs": [],
   "source": [
    "words = open(\"Datasets/Indian_Names.txt\").read().splitlines()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "256a12aa-bfd0-445c-be7c-250cc5d9ee8f",
   "metadata": {},
   "outputs": [],
   "source": [
    "words = [word.lower() for word in words]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "a62c3434-6336-497a-aa1f-0ada3cb3d2d5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "53982\n"
     ]
    }
   ],
   "source": [
    "print(len(words))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "a834d348-6c6b-48d9-b831-ed14c1a66b9e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Characters: ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z']\n",
      "STOI: {'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5, 'f': 6, 'g': 7, 'h': 8, 'i': 9, 'j': 10, 'k': 11, 'l': 12, 'm': 13, 'n': 14, 'o': 15, 'p': 16, 'q': 17, 'r': 18, 's': 19, 't': 20, 'u': 21, 'v': 22, 'w': 23, 'x': 24, 'y': 25, 'z': 26, '.': 0}\n",
      "ITOS {1: 'a', 2: 'b', 3: 'c', 4: 'd', 5: 'e', 6: 'f', 7: 'g', 8: 'h', 9: 'i', 10: 'j', 11: 'k', 12: 'l', 13: 'm', 14: 'n', 15: 'o', 16: 'p', 17: 'q', 18: 'r', 19: 's', 20: 't', 21: 'u', 22: 'v', 23: 'w', 24: 'x', 25: 'y', 26: 'z', 0: '.'}\n"
     ]
    }
   ],
   "source": [
    "# Remember we need our starting and ending tokens as well in these mappings,\n",
    "characters = sorted(list(set(''.join(words)))) # Gives us all the characters in the english alphabet, hopefully our dataset has all of them\n",
    "stoi = {s:i+1 for i,s in enumerate(characters)} # Enumerate returns the tuples of number and string, which can then be mapped to string:index\n",
    "# We manually add these tokens for convenience\n",
    "stoi['.'] = 0\n",
    "itos = {i:s for s,i in stoi.items()} # After we have the string:index mapping, we can easily iterate over their items to map index:string\n",
    "print(\"Characters:\",characters)\n",
    "print(\"STOI:\",stoi)\n",
    "print(\"ITOS\",itos)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "bb555f44-624d-4b80-b519-ffb26cb94936",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We define a Block Size based on the number of characters we feed are going to feed to predict the next one\n",
    "inputBlockSize = 3\n",
    "\n",
    "# We define two lists, inputs & outputs, where inputs are our blocks of the block size mentioned above and outputs are the label indexes\n",
    "inputs , outputs = [], []\n",
    "\n",
    "# We run a loop for each word in the original dataset\n",
    "for word in words:\n",
    "    # We define the block for each iteration and fill it with 0 values -> [0, 0, 0]\n",
    "    block = [0] * inputBlockSize # This is also known as the context of the network\n",
    "    # We run another loop for each word's character, here word also needs the ending token '.'\n",
    "    for character in word + '.':\n",
    "        # We take out the index from our look-up table\n",
    "        index = stoi[character]\n",
    "        # We append the input with our block\n",
    "        inputs.append(block)\n",
    "        # We append the output label with out index of the character\n",
    "        outputs.append([index])\n",
    "        # We then take the block, crop it 1 size from the left and append the next index to it (sliding window of name)\n",
    "        block = block[1:] + [index]\n",
    "# We also convert these inputs and outputs to tensors for neural network processing\n",
    "inputs = torch.tensor(inputs)\n",
    "outputs = torch.flatten(torch.tensor(outputs))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "855f1a3c-6e3c-4c76-9ac4-7ca2e39baa71",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Inputs Shape: torch.Size([488074, 3]) , Datatype: torch.int64\n",
      "Outputs Shape: torch.Size([488074]) , Datatype: torch.int64\n"
     ]
    }
   ],
   "source": [
    "# We can now check the shape of inputs and outputs and their corresponding datatypes\n",
    "print(\"Inputs Shape:\",inputs.shape,\", Datatype:\",inputs.dtype)\n",
    "print(\"Outputs Shape:\",outputs.shape,\", Datatype:\",outputs.dtype)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "2825d6da-5235-41e0-b3a1-6bc858b53a23",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We will define a generator to give the same result on your machine, as of my machine\n",
    "generator = torch.Generator().manual_seed(6942069420)\n",
    "# Embedding Matrix (Input Layer)\n",
    "embeddingFeatureSpaceLength = 2\n",
    "embeddingLookUpMatrix = torch.randn((len(stoi),embeddingFeatureSpaceLength), generator=generator)\n",
    "# Hidden Layer\n",
    "numberOfHiddenLayerNeurons = 100\n",
    "weightsOfHiddenLayer = torch.randn((inputBlockSize*embeddingFeatureSpaceLength), numberOfHiddenLayerNeurons, generator=generator)\n",
    "biasesOfHiddenLayer = torch.randn(numberOfHiddenLayerNeurons, generator=generator)\n",
    "# Output Layer / Final Layer\n",
    "numberOfFinalLayerOutputs = 27\n",
    "weightsOfFinalLayer = torch.randn(numberOfHiddenLayerNeurons, numberOfFinalLayerOutputs, generator=generator)\n",
    "biasesOfFinalLayer = torch.randn(numberOfFinalLayerOutputs, generator=generator)\n",
    "# Parameters\n",
    "parameters = [embeddingLookUpMatrix, weightsOfHiddenLayer, biasesOfHiddenLayer, weightsOfFinalLayer, biasesOfFinalLayer]\n",
    "# We set all the requires gradient to True\n",
    "for parameter in parameters:\n",
    "    parameter.requires_grad = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "79639d70-488b-4ea5-b4ef-2ffbcd815ea5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3481"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's check how many parameters we have\n",
    "sum(parameter.nelement() for parameter in parameters)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "8642c60d-f8b5-4d48-961c-5d9ae46dae31",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loss: tensor(19.4960, grad_fn=<NllLossBackward0>)\n",
      "Loss: tensor(17.4382, grad_fn=<NllLossBackward0>)\n",
      "Loss: tensor(15.7174, grad_fn=<NllLossBackward0>)\n",
      "Loss: tensor(14.5988, grad_fn=<NllLossBackward0>)\n",
      "Loss: tensor(13.6550, grad_fn=<NllLossBackward0>)\n",
      "Loss: tensor(12.8257, grad_fn=<NllLossBackward0>)\n",
      "Loss: tensor(12.1082, grad_fn=<NllLossBackward0>)\n",
      "Loss: tensor(11.4890, grad_fn=<NllLossBackward0>)\n",
      "Loss: tensor(10.9438, grad_fn=<NllLossBackward0>)\n",
      "Loss: tensor(10.4558, grad_fn=<NllLossBackward0>)\n"
     ]
    }
   ],
   "source": [
    "# We define the number of epochs\n",
    "epochs = 10\n",
    "for _ in range(epochs):\n",
    "    # Forward Pass\n",
    "    embedding = embeddingLookUpMatrix[inputs]\n",
    "    hiddenLayerStates = torch.tanh(embedding.view(-1, inputBlockSize*embeddingFeatureSpaceLength) @ weightsOfHiddenLayer + biasesOfHiddenLayer)\n",
    "    logits = hiddenLayerStates @ weightsOfFinalLayer + biasesOfFinalLayer\n",
    "    loss = F.cross_entropy(logits, outputs)\n",
    "    print(\"Loss:\", loss)\n",
    "    \n",
    "    # Backward Pass\n",
    "    for parameter in parameters:\n",
    "        parameter.grad = None\n",
    "    loss.backward()\n",
    "    \n",
    "    # Update Weights\n",
    "    learning_rate = 0.1\n",
    "    for parameter in parameters:\n",
    "        parameter.data += -learning_rate * parameter.grad"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9c7cfb4-13a8-4f1e-a4c8-438156f19527",
   "metadata": {},
   "source": [
    "See how this takes a lot more time now?\n",
    "\n",
    "In practice, we never use the entire dataset for training in one go..\n",
    "\n",
    "We consider small batches, what I like to call is *'mini-batches'*...\n",
    "\n",
    "![MiniBatchesTraining](https://www.baeldung.com/wp-content/uploads/sites/4/2022/06/minibatch_gd.png)\n",
    "\n",
    "Doing this, the quality of our gradient becomes lower, because now it is not the actual gradient direction...\\\n",
    "But even then,\\\n",
    "**It is much better to have an approximate gradient and have more steps, than it is to calculate the exact gradient and take fewer steps.**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4e4ba30-4671-462b-bfb1-f9fc504bb459",
   "metadata": {},
   "source": [
    "We can define a list of random integers to index starting from 0 to the size of 0-th dimension of the inputs\n",
    "We define the size to be of 5 names, which were roughly say, 37 examples for the first 5 words\n",
    "```python\n",
    "indexes = torch.randint(low=0, high=inputs.shape[0], size=(37,))\n",
    "```\n",
    "\n",
    "We can then index into the lookup matrix to get the embeddings of only those indexes using:\n",
    "```python\n",
    "embeddingLookUpMatrix[inputs[indexes]]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "1a1c5046-1e0f-45ee-b364-2a061ffe8697",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Minibatch Loss: tensor(2.2117, grad_fn=<NllLossBackward0>)\n"
     ]
    }
   ],
   "source": [
    "# We define the number of epochs\n",
    "epochs = 1000\n",
    "for _ in range(epochs):\n",
    "    # Mini-Batches\n",
    "    indexes = torch.randint(low=0, high=inputs.shape[0], size=(37,))\n",
    "    \n",
    "    # Forward Pass (Mini-Batch)\n",
    "    embedding = embeddingLookUpMatrix[inputs[indexes]] # Indexing into look-up table\n",
    "    hiddenLayerStates = torch.tanh(embedding.view(-1, inputBlockSize*embeddingFeatureSpaceLength) @ weightsOfHiddenLayer + biasesOfHiddenLayer)\n",
    "    logits = hiddenLayerStates @ weightsOfFinalLayer + biasesOfFinalLayer\n",
    "    loss = F.cross_entropy(logits, outputs[indexes]) # Indexing into labels\n",
    "    \n",
    "    # Backward Pass (Mini-Batch)\n",
    "    for parameter in parameters:\n",
    "        parameter.grad = None\n",
    "    loss.backward()\n",
    "    \n",
    "    # Update Weights (Mini-Batch)\n",
    "    learning_rate = 0.1\n",
    "    for parameter in parameters:\n",
    "        parameter.data += -learning_rate * parameter.grad\n",
    "\n",
    "print(\"Minibatch Loss:\", loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "468bf7ec-d37e-4f1b-9b78-c3915742d679",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loss: tensor(2.3259, grad_fn=<NllLossBackward0>)\n"
     ]
    }
   ],
   "source": [
    "# Gradient of the entire training set\n",
    "embedding = embeddingLookUpMatrix[inputs]\n",
    "hiddenLayerStates = torch.tanh(embedding.view(-1, inputBlockSize*embeddingFeatureSpaceLength) @ weightsOfHiddenLayer + biasesOfHiddenLayer)\n",
    "logits = hiddenLayerStates @ weightsOfFinalLayer + biasesOfFinalLayer\n",
    "loss = F.cross_entropy(logits, outputs)\n",
    "print(\"Loss:\", loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e7381b6-c700-445c-8eac-1b3574f83cca",
   "metadata": {},
   "source": [
    "The problem now is...\n",
    "\n",
    "We don't know if we are stepping too slow or too fast into the dataset...\n",
    "\n",
    "So now the question arises is...\n",
    "\n",
    "**How do we determine the learning rate?**\n",
    "\n",
    "Let's experiment manually first..."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc846272-ca23-4df9-8d0a-2e0250aa8d2e",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "# Manual Learning Rate Experimentation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dfcc2607-2656-49ee-86e7-8df23186d0d5",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "### Resetting the parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "5bb7c887-867e-4a15-bf5f-b0d215d0a2d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Resetting the parameters\n",
    "# We will define a generator to give the same result on your machine, as of my machine\n",
    "generator = torch.Generator().manual_seed(6942069420)\n",
    "# Embedding Matrix (Input Layer)\n",
    "embeddingFeatureSpaceLength = 2\n",
    "embeddingLookUpMatrix = torch.randn((len(stoi),embeddingFeatureSpaceLength), generator=generator)\n",
    "# Hidden Layer\n",
    "numberOfHiddenLayerNeurons = 100\n",
    "weightsOfHiddenLayer = torch.randn((inputBlockSize*embeddingFeatureSpaceLength), numberOfHiddenLayerNeurons, generator=generator)\n",
    "biasesOfHiddenLayer = torch.randn(numberOfHiddenLayerNeurons, generator=generator)\n",
    "# Output Layer / Final Layer\n",
    "numberOfFinalLayerOutputs = 27\n",
    "weightsOfFinalLayer = torch.randn(numberOfHiddenLayerNeurons, numberOfFinalLayerOutputs, generator=generator)\n",
    "biasesOfFinalLayer = torch.randn(numberOfFinalLayerOutputs, generator=generator)\n",
    "# Parameters\n",
    "parameters = [embeddingLookUpMatrix, weightsOfHiddenLayer, biasesOfHiddenLayer, weightsOfFinalLayer, biasesOfFinalLayer]\n",
    "# We set all the requires gradient to True\n",
    "for parameter in parameters:\n",
    "    parameter.requires_grad = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "fc8bde46-c89b-4333-b95b-db9ad9605d6c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3481"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's check how many parameters we have\n",
    "sum(parameter.nelement() for parameter in parameters)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "49f9c764-7820-4df4-a638-49b1fb72c532",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Minibatch Loss: tensor(18.6522, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(17.9291, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.4235, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(21.0288, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(20.5132, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(19.8916, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(17.6657, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(22.0373, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(16.7524, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(17.9729, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(20.8925, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(19.8515, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(19.3152, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(20.7028, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(20.5437, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(16.6896, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(20.4077, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(17.7527, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(17.8709, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(19.4708, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(19.0085, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.1264, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(19.7507, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(21.9908, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(21.5978, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.0307, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(21.1110, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(17.5907, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.5521, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(21.3672, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.4080, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.3267, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(20.0741, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(19.2535, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(19.2409, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(19.9490, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(17.8367, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.5768, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.6652, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(19.8382, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(17.8815, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(19.4656, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(15.7578, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.1784, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(16.1548, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(19.5018, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.1019, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(19.0181, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(16.2947, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.3617, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.0572, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.9248, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(19.5003, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.1818, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(17.6492, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(19.0441, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(19.8581, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.2394, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(17.3472, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(15.0136, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(19.1022, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.7017, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(17.8602, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(16.1745, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(17.2472, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(14.5557, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.1645, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(15.7541, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(20.6173, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(17.1311, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(16.6341, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.2248, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(17.0703, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(16.0822, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(21.4789, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(19.8450, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.9815, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(19.9123, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(17.5313, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(17.2863, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(19.0911, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(19.5295, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(16.0768, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(19.8486, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.6026, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(16.7560, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(16.8558, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(17.2098, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(19.8196, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(20.6300, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.5599, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.1254, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(17.2125, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(20.7826, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.2060, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.4138, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.1076, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(19.6956, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(20.6084, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(17.0835, grad_fn=<NllLossBackward0>)\n"
     ]
    }
   ],
   "source": [
    "# We define the number of epochs\n",
    "epochs = 100\n",
    "for _ in range(epochs):\n",
    "    # Mini-Batches\n",
    "    indexes = torch.randint(low=0, high=inputs.shape[0], size=(37,))\n",
    "    \n",
    "    # Forward Pass (Mini-Batch)\n",
    "    embedding = embeddingLookUpMatrix[inputs[indexes]] # Indexing into look-up table\n",
    "    hiddenLayerStates = torch.tanh(embedding.view(-1, inputBlockSize*embeddingFeatureSpaceLength) @ weightsOfHiddenLayer + biasesOfHiddenLayer)\n",
    "    logits = hiddenLayerStates @ weightsOfFinalLayer + biasesOfFinalLayer\n",
    "    loss = F.cross_entropy(logits, outputs[indexes]) # Indexing into labels\n",
    "\n",
    "    print(\"Minibatch Loss:\", loss)\n",
    "    \n",
    "    # Backward Pass (Mini-Batch)\n",
    "    for parameter in parameters:\n",
    "        parameter.grad = None\n",
    "    loss.backward()\n",
    "    \n",
    "    # Update Weights (Mini-Batch)\n",
    "    learning_rate = 0.001 # Assuming this is very low\n",
    "    for parameter in parameters:\n",
    "        parameter.data += -learning_rate * parameter.grad"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f692a2b1-e61a-499e-ac04-131f520fc5b0",
   "metadata": {},
   "source": [
    "**Right now the loss is barely decreasing...**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7029d7d6-15ee-4e2f-b335-d3730f6cdf3b",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "### We reset the parameters again..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "38682afe-2169-4a2e-83a6-0c0317fb8fd6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Resetting the parameters\n",
    "# We will define a generator to give the same result on your machine, as of my machine\n",
    "generator = torch.Generator().manual_seed(6942069420)\n",
    "# Embedding Matrix (Input Layer)\n",
    "embeddingFeatureSpaceLength = 2\n",
    "embeddingLookUpMatrix = torch.randn((len(stoi),embeddingFeatureSpaceLength), generator=generator)\n",
    "# Hidden Layer\n",
    "numberOfHiddenLayerNeurons = 100\n",
    "weightsOfHiddenLayer = torch.randn((inputBlockSize*embeddingFeatureSpaceLength), numberOfHiddenLayerNeurons, generator=generator)\n",
    "biasesOfHiddenLayer = torch.randn(numberOfHiddenLayerNeurons, generator=generator)\n",
    "# Output Layer / Final Layer\n",
    "numberOfFinalLayerOutputs = 27\n",
    "weightsOfFinalLayer = torch.randn(numberOfHiddenLayerNeurons, numberOfFinalLayerOutputs, generator=generator)\n",
    "biasesOfFinalLayer = torch.randn(numberOfFinalLayerOutputs, generator=generator)\n",
    "# Parameters\n",
    "parameters = [embeddingLookUpMatrix, weightsOfHiddenLayer, biasesOfHiddenLayer, weightsOfFinalLayer, biasesOfFinalLayer]\n",
    "# We set all the requires gradient to True\n",
    "for parameter in parameters:\n",
    "    parameter.requires_grad = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "9aee9566-545a-403c-babc-57b9e0224021",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Minibatch Loss: tensor(22.0868, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(17.9440, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(19.5703, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(19.1174, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.6400, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(22.7232, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(16.6106, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.7132, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(19.4819, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(15.3402, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(16.3563, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(20.6350, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(19.9277, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(14.9505, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(17.4924, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.6699, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(15.1450, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(15.9933, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(16.6489, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(15.8873, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(15.6218, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(16.1925, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(12.3978, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.3315, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(15.7890, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(13.7610, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(15.8773, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(12.8401, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(14.7520, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(16.3941, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(13.1662, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(14.1389, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(13.4056, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(15.4774, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(14.1141, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(14.6401, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(14.3862, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(14.2762, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(12.9967, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(13.4571, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(14.5631, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(13.2595, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(14.6945, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(15.8902, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(13.3156, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(15.0156, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(14.8719, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(12.5753, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(14.3787, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(13.3145, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(12.6271, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(12.1100, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(15.7220, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(14.2272, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(11.7557, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(12.2701, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(14.5239, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(11.1007, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(12.8619, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(13.6768, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(9.9869, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(12.9642, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(11.5901, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(10.9823, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(11.2532, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(13.1881, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(9.3533, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(10.4868, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(11.7866, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(10.1160, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(12.6899, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(9.4507, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(11.3496, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(10.5647, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(13.0669, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(9.8005, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(11.2214, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(7.6394, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(11.0093, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(13.6696, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(9.6989, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(11.3227, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(10.5938, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(10.3949, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(11.4580, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(9.5024, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(12.5666, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(10.7598, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(10.6136, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(12.5916, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(12.7892, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(11.1812, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(8.7958, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(10.5507, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(9.3845, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(9.5053, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(9.9972, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(10.2224, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(9.5720, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(9.9898, grad_fn=<NllLossBackward0>)\n"
     ]
    }
   ],
   "source": [
    "# We define the number of epochs\n",
    "epochs = 100\n",
    "for _ in range(epochs):\n",
    "    # Mini-Batches\n",
    "    indexes = torch.randint(low=0, high=inputs.shape[0], size=(37,))\n",
    "    \n",
    "    # Forward Pass (Mini-Batch)\n",
    "    embedding = embeddingLookUpMatrix[inputs[indexes]] # Indexing into look-up table\n",
    "    hiddenLayerStates = torch.tanh(embedding.view(-1, inputBlockSize*embeddingFeatureSpaceLength) @ weightsOfHiddenLayer + biasesOfHiddenLayer)\n",
    "    logits = hiddenLayerStates @ weightsOfFinalLayer + biasesOfFinalLayer\n",
    "    loss = F.cross_entropy(logits, outputs[indexes]) # Indexing into labels\n",
    "\n",
    "    print(\"Minibatch Loss:\", loss)\n",
    "    \n",
    "    # Backward Pass (Mini-Batch)\n",
    "    for parameter in parameters:\n",
    "        parameter.grad = None\n",
    "    loss.backward()\n",
    "    \n",
    "    # Update Weights (Mini-Batch)\n",
    "    learning_rate = 0.01 # Assuming this is low\n",
    "    for parameter in parameters:\n",
    "        parameter.data += -learning_rate * parameter.grad"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bbf694e8-2154-499c-b891-5f248c77dfee",
   "metadata": {},
   "source": [
    "**We are decreasing the loss now, but not very quickly...**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21dca875-e5d4-4be3-9e37-c098bc7eea76",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "### We reset the parameters again..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "26cdb173-8ba2-4ced-a071-38a1c325a76d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Resetting the parameters\n",
    "# We will define a generator to give the same result on your machine, as of my machine\n",
    "generator = torch.Generator().manual_seed(6942069420)\n",
    "# Embedding Matrix (Input Layer)\n",
    "embeddingFeatureSpaceLength = 2\n",
    "embeddingLookUpMatrix = torch.randn((len(stoi),embeddingFeatureSpaceLength), generator=generator)\n",
    "# Hidden Layer\n",
    "numberOfHiddenLayerNeurons = 100\n",
    "weightsOfHiddenLayer = torch.randn((inputBlockSize*embeddingFeatureSpaceLength), numberOfHiddenLayerNeurons, generator=generator)\n",
    "biasesOfHiddenLayer = torch.randn(numberOfHiddenLayerNeurons, generator=generator)\n",
    "# Output Layer / Final Layer\n",
    "numberOfFinalLayerOutputs = 27\n",
    "weightsOfFinalLayer = torch.randn(numberOfHiddenLayerNeurons, numberOfFinalLayerOutputs, generator=generator)\n",
    "biasesOfFinalLayer = torch.randn(numberOfFinalLayerOutputs, generator=generator)\n",
    "# Parameters\n",
    "parameters = [embeddingLookUpMatrix, weightsOfHiddenLayer, biasesOfHiddenLayer, weightsOfFinalLayer, biasesOfFinalLayer]\n",
    "# We set all the requires gradient to True\n",
    "for parameter in parameters:\n",
    "    parameter.requires_grad = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "c62f1c00-f2ce-4832-9859-7f5e53b4246d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Minibatch Loss: tensor(19.3488, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(16.5682, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(14.6362, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(11.3325, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(9.1051, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(10.0847, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(8.4405, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(7.0867, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(8.8115, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(11.1865, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(8.4249, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(8.2271, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(7.3415, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(9.0825, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(8.7848, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(9.3682, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(8.8127, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(8.9995, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.7556, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.6535, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.9124, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(8.7296, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.8510, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.8212, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(8.2737, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.5476, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(7.5671, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(7.6214, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(7.9222, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(8.9529, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.6711, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.4208, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(8.8959, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.8508, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.3595, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.9938, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.6582, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(7.3013, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.7051, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.4657, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.5685, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.5501, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(8.9802, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(7.7993, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.9671, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.8387, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.4097, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.6264, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.9000, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.3288, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(7.1322, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.9193, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(8.0471, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(8.1873, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.1744, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.2164, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.9543, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.3758, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(9.7070, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.9756, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.8996, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(8.6791, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(7.2384, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(8.3349, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.9943, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.8042, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.4787, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.4319, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.4762, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.9482, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.5839, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.9171, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(8.2186, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.5745, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(8.4389, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.9167, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.8900, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(8.1304, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.6522, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.9375, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.1979, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.1501, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.7063, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.1515, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.9851, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.9977, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.8397, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.3233, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.2450, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.5664, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.0199, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.7750, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.9127, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.7772, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.4428, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.5679, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(7.9133, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.9615, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.5223, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(7.3910, grad_fn=<NllLossBackward0>)\n"
     ]
    }
   ],
   "source": [
    "# We define the number of epochs\n",
    "epochs = 100\n",
    "for _ in range(epochs):\n",
    "    # Mini-Batches\n",
    "    indexes = torch.randint(low=0, high=inputs.shape[0], size=(37,))\n",
    "    \n",
    "    # Forward Pass (Mini-Batch)\n",
    "    embedding = embeddingLookUpMatrix[inputs[indexes]] # Indexing into look-up table\n",
    "    hiddenLayerStates = torch.tanh(embedding.view(-1, inputBlockSize*embeddingFeatureSpaceLength) @ weightsOfHiddenLayer + biasesOfHiddenLayer)\n",
    "    logits = hiddenLayerStates @ weightsOfFinalLayer + biasesOfFinalLayer\n",
    "    loss = F.cross_entropy(logits, outputs[indexes]) # Indexing into labels\n",
    "\n",
    "    print(\"Minibatch Loss:\", loss)\n",
    "    \n",
    "    # Backward Pass (Mini-Batch)\n",
    "    for parameter in parameters:\n",
    "        parameter.grad = None\n",
    "    loss.backward()\n",
    "    \n",
    "    # Update Weights (Mini-Batch)\n",
    "    learning_rate = 1 # Assuming this is high\n",
    "    for parameter in parameters:\n",
    "        parameter.data += -learning_rate * parameter.grad"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d01dfb1c-8391-477a-a4af-5c2dadc633b0",
   "metadata": {},
   "source": [
    "**We are minimizing the loss but it is kind of unstable right now...**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51ac22d7-09e3-469f-a581-b3e7ebca569e",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "### We reset the parameters again..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "0a2628cf-ff09-403d-a6b8-6d852b4c73a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Resetting the parameters\n",
    "# We will define a generator to give the same result on your machine, as of my machine\n",
    "generator = torch.Generator().manual_seed(6942069420)\n",
    "# Embedding Matrix (Input Layer)\n",
    "embeddingFeatureSpaceLength = 2\n",
    "embeddingLookUpMatrix = torch.randn((len(stoi),embeddingFeatureSpaceLength), generator=generator)\n",
    "# Hidden Layer\n",
    "numberOfHiddenLayerNeurons = 100\n",
    "weightsOfHiddenLayer = torch.randn((inputBlockSize*embeddingFeatureSpaceLength), numberOfHiddenLayerNeurons, generator=generator)\n",
    "biasesOfHiddenLayer = torch.randn(numberOfHiddenLayerNeurons, generator=generator)\n",
    "# Output Layer / Final Layer\n",
    "numberOfFinalLayerOutputs = 27\n",
    "weightsOfFinalLayer = torch.randn(numberOfHiddenLayerNeurons, numberOfFinalLayerOutputs, generator=generator)\n",
    "biasesOfFinalLayer = torch.randn(numberOfFinalLayerOutputs, generator=generator)\n",
    "# Parameters\n",
    "parameters = [embeddingLookUpMatrix, weightsOfHiddenLayer, biasesOfHiddenLayer, weightsOfFinalLayer, biasesOfFinalLayer]\n",
    "# We set all the requires gradient to True\n",
    "for parameter in parameters:\n",
    "    parameter.requires_grad = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "6937973c-aa8c-4e06-b68c-e80a1618048b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Minibatch Loss: tensor(17.5197, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(58.8351, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(63.9221, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(54.7494, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(42.2917, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(59.3939, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(74.2832, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(53.8748, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(64.6195, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(93.0884, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(78.4838, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(51.2402, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(55.2153, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(58.6816, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(50.9520, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(38.5916, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(51.7255, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(65.3559, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(48.1410, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(50.0691, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(50.2359, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(72.5157, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(58.6648, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(67.8387, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(54.0528, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(53.9229, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(51.1829, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(40.0586, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(59.5062, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(69.5364, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(72.9374, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(45.7941, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(47.5000, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(62.9154, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(64.3138, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(58.2208, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(60.7182, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(53.4142, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(55.3416, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(63.0006, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(53.3782, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(44.4213, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(36.7608, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(41.3522, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(49.3730, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(42.6989, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(54.0559, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(46.2158, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(84.2187, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(78.6773, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(75.4010, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(57.4212, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(53.8975, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(45.1179, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(61.7127, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(70.7301, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(47.6858, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(57.8243, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(53.7394, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(96.4369, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(86.3925, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(65.3039, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(57.4151, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(59.9862, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(56.5594, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(44.5091, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(47.0270, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(47.4973, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(55.2268, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(57.5228, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(39.9544, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(52.4219, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(59.6538, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(49.5291, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(69.5949, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(45.5518, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(54.0847, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(54.7858, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(59.5585, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(55.1709, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(50.4135, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(64.1978, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(57.0554, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(38.4957, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(48.9871, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(65.0200, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(48.1735, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(57.1958, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(31.4239, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(44.8020, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(44.8804, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(46.9160, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(45.4049, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(58.2038, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(48.3335, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(46.6224, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(49.1909, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(52.2237, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(49.0568, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(50.8669, grad_fn=<NllLossBackward0>)\n"
     ]
    }
   ],
   "source": [
    "# We define the number of epochs\n",
    "epochs = 100\n",
    "for _ in range(epochs):\n",
    "    # Mini-Batches\n",
    "    indexes = torch.randint(low=0, high=inputs.shape[0], size=(37,))\n",
    "    \n",
    "    # Forward Pass (Mini-Batch)\n",
    "    embedding = embeddingLookUpMatrix[inputs[indexes]] # Indexing into look-up table\n",
    "    hiddenLayerStates = torch.tanh(embedding.view(-1, inputBlockSize*embeddingFeatureSpaceLength) @ weightsOfHiddenLayer + biasesOfHiddenLayer)\n",
    "    logits = hiddenLayerStates @ weightsOfFinalLayer + biasesOfFinalLayer\n",
    "    loss = F.cross_entropy(logits, outputs[indexes]) # Indexing into labels\n",
    "\n",
    "    print(\"Minibatch Loss:\", loss)\n",
    "    \n",
    "    # Backward Pass (Mini-Batch)\n",
    "    for parameter in parameters:\n",
    "        parameter.grad = None\n",
    "    loss.backward()\n",
    "    \n",
    "    # Update Weights (Mini-Batch)\n",
    "    learning_rate = 10 # Assuming this is very high\n",
    "    for parameter in parameters:\n",
    "        parameter.data += -learning_rate * parameter.grad"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2a37f1c-fb33-410d-93c4-41b200f59fbc",
   "metadata": {},
   "source": [
    "**So, now we are not minimizing the loss at all...**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4fe7cb7e-0b63-477d-af35-ee47b861ecb0",
   "metadata": {},
   "source": [
    "### Tracking Stats"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6b177ff-3e3d-488a-8156-c3ed247eb6f6",
   "metadata": {},
   "source": [
    "Now we can safely assume that ***learning-rate*** is somewhere **between 0.001 and 1**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31a3174a-9513-4f0e-b609-1fe80c4ec338",
   "metadata": {},
   "source": [
    "We can now list out start and end with 1000 different points betweeen 0.001 and 1:\n",
    "```python\n",
    "torch.linspace(start=0.001, end=1, steps=1000)\n",
    "```\n",
    "\n",
    "But now that we see a small pattern that this same line can be replaced with a small mathematical expression after relating to it.\n",
    "\n",
    "We see that we are considering the points between 0.001 and 1,\\\n",
    "This also evaluates to *10^-3 to 10^0*\n",
    "\n",
    "That is because:\n",
    "$$ 10^0 = 1 $$\n",
    "$$ 10^{-1} = 0.1 $$\n",
    "$$ 10^{-2} = 0.01 $$\n",
    "and so on...\n",
    "\n",
    "So we can replace the above code with these lines:\n",
    "```python\n",
    "lossExpression = torch.linspace(start=-3, end=0, steps=1000)\n",
    "lossExponentExpression = 10**lossExpression\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "8ea06de4-bb17-4755-ac37-39a491d26c8c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([0.0010, 0.0010, 0.0010, 0.0010, 0.0010, 0.0010, 0.0010, 0.0010, 0.0011,\n",
      "        0.0011, 0.0011, 0.0011, 0.0011, 0.0011, 0.0011, 0.0011, 0.0011, 0.0011,\n",
      "        0.0011, 0.0011, 0.0011, 0.0012, 0.0012, 0.0012, 0.0012, 0.0012, 0.0012,\n",
      "        0.0012, 0.0012, 0.0012, 0.0012, 0.0012, 0.0012, 0.0013, 0.0013, 0.0013,\n",
      "        0.0013, 0.0013, 0.0013, 0.0013, 0.0013, 0.0013, 0.0013, 0.0013, 0.0014,\n",
      "        0.0014, 0.0014, 0.0014, 0.0014, 0.0014, 0.0014, 0.0014, 0.0014, 0.0014,\n",
      "        0.0015, 0.0015, 0.0015, 0.0015, 0.0015, 0.0015, 0.0015, 0.0015, 0.0015,\n",
      "        0.0015, 0.0016, 0.0016, 0.0016, 0.0016, 0.0016, 0.0016, 0.0016, 0.0016,\n",
      "        0.0016, 0.0017, 0.0017, 0.0017, 0.0017, 0.0017, 0.0017, 0.0017, 0.0017,\n",
      "        0.0018, 0.0018, 0.0018, 0.0018, 0.0018, 0.0018, 0.0018, 0.0018, 0.0019,\n",
      "        0.0019, 0.0019, 0.0019, 0.0019, 0.0019, 0.0019, 0.0019, 0.0020, 0.0020,\n",
      "        0.0020, 0.0020, 0.0020, 0.0020, 0.0020, 0.0021, 0.0021, 0.0021, 0.0021,\n",
      "        0.0021, 0.0021, 0.0021, 0.0022, 0.0022, 0.0022, 0.0022, 0.0022, 0.0022,\n",
      "        0.0022, 0.0023, 0.0023, 0.0023, 0.0023, 0.0023, 0.0023, 0.0024, 0.0024,\n",
      "        0.0024, 0.0024, 0.0024, 0.0024, 0.0025, 0.0025, 0.0025, 0.0025, 0.0025,\n",
      "        0.0025, 0.0026, 0.0026, 0.0026, 0.0026, 0.0026, 0.0027, 0.0027, 0.0027,\n",
      "        0.0027, 0.0027, 0.0027, 0.0028, 0.0028, 0.0028, 0.0028, 0.0028, 0.0029,\n",
      "        0.0029, 0.0029, 0.0029, 0.0029, 0.0030, 0.0030, 0.0030, 0.0030, 0.0030,\n",
      "        0.0031, 0.0031, 0.0031, 0.0031, 0.0032, 0.0032, 0.0032, 0.0032, 0.0032,\n",
      "        0.0033, 0.0033, 0.0033, 0.0033, 0.0034, 0.0034, 0.0034, 0.0034, 0.0034,\n",
      "        0.0035, 0.0035, 0.0035, 0.0035, 0.0036, 0.0036, 0.0036, 0.0036, 0.0037,\n",
      "        0.0037, 0.0037, 0.0037, 0.0038, 0.0038, 0.0038, 0.0039, 0.0039, 0.0039,\n",
      "        0.0039, 0.0040, 0.0040, 0.0040, 0.0040, 0.0041, 0.0041, 0.0041, 0.0042,\n",
      "        0.0042, 0.0042, 0.0042, 0.0043, 0.0043, 0.0043, 0.0044, 0.0044, 0.0044,\n",
      "        0.0045, 0.0045, 0.0045, 0.0045, 0.0046, 0.0046, 0.0046, 0.0047, 0.0047,\n",
      "        0.0047, 0.0048, 0.0048, 0.0048, 0.0049, 0.0049, 0.0049, 0.0050, 0.0050,\n",
      "        0.0050, 0.0051, 0.0051, 0.0051, 0.0052, 0.0052, 0.0053, 0.0053, 0.0053,\n",
      "        0.0054, 0.0054, 0.0054, 0.0055, 0.0055, 0.0056, 0.0056, 0.0056, 0.0057,\n",
      "        0.0057, 0.0058, 0.0058, 0.0058, 0.0059, 0.0059, 0.0060, 0.0060, 0.0060,\n",
      "        0.0061, 0.0061, 0.0062, 0.0062, 0.0062, 0.0063, 0.0063, 0.0064, 0.0064,\n",
      "        0.0065, 0.0065, 0.0066, 0.0066, 0.0067, 0.0067, 0.0067, 0.0068, 0.0068,\n",
      "        0.0069, 0.0069, 0.0070, 0.0070, 0.0071, 0.0071, 0.0072, 0.0072, 0.0073,\n",
      "        0.0073, 0.0074, 0.0074, 0.0075, 0.0075, 0.0076, 0.0076, 0.0077, 0.0077,\n",
      "        0.0078, 0.0079, 0.0079, 0.0080, 0.0080, 0.0081, 0.0081, 0.0082, 0.0082,\n",
      "        0.0083, 0.0084, 0.0084, 0.0085, 0.0085, 0.0086, 0.0086, 0.0087, 0.0088,\n",
      "        0.0088, 0.0089, 0.0090, 0.0090, 0.0091, 0.0091, 0.0092, 0.0093, 0.0093,\n",
      "        0.0094, 0.0095, 0.0095, 0.0096, 0.0097, 0.0097, 0.0098, 0.0099, 0.0099,\n",
      "        0.0100, 0.0101, 0.0101, 0.0102, 0.0103, 0.0104, 0.0104, 0.0105, 0.0106,\n",
      "        0.0106, 0.0107, 0.0108, 0.0109, 0.0109, 0.0110, 0.0111, 0.0112, 0.0112,\n",
      "        0.0113, 0.0114, 0.0115, 0.0116, 0.0116, 0.0117, 0.0118, 0.0119, 0.0120,\n",
      "        0.0121, 0.0121, 0.0122, 0.0123, 0.0124, 0.0125, 0.0126, 0.0127, 0.0127,\n",
      "        0.0128, 0.0129, 0.0130, 0.0131, 0.0132, 0.0133, 0.0134, 0.0135, 0.0136,\n",
      "        0.0137, 0.0137, 0.0138, 0.0139, 0.0140, 0.0141, 0.0142, 0.0143, 0.0144,\n",
      "        0.0145, 0.0146, 0.0147, 0.0148, 0.0149, 0.0150, 0.0151, 0.0152, 0.0154,\n",
      "        0.0155, 0.0156, 0.0157, 0.0158, 0.0159, 0.0160, 0.0161, 0.0162, 0.0163,\n",
      "        0.0165, 0.0166, 0.0167, 0.0168, 0.0169, 0.0170, 0.0171, 0.0173, 0.0174,\n",
      "        0.0175, 0.0176, 0.0178, 0.0179, 0.0180, 0.0181, 0.0182, 0.0184, 0.0185,\n",
      "        0.0186, 0.0188, 0.0189, 0.0190, 0.0192, 0.0193, 0.0194, 0.0196, 0.0197,\n",
      "        0.0198, 0.0200, 0.0201, 0.0202, 0.0204, 0.0205, 0.0207, 0.0208, 0.0210,\n",
      "        0.0211, 0.0212, 0.0214, 0.0215, 0.0217, 0.0218, 0.0220, 0.0221, 0.0223,\n",
      "        0.0225, 0.0226, 0.0228, 0.0229, 0.0231, 0.0232, 0.0234, 0.0236, 0.0237,\n",
      "        0.0239, 0.0241, 0.0242, 0.0244, 0.0246, 0.0247, 0.0249, 0.0251, 0.0253,\n",
      "        0.0254, 0.0256, 0.0258, 0.0260, 0.0261, 0.0263, 0.0265, 0.0267, 0.0269,\n",
      "        0.0271, 0.0273, 0.0274, 0.0276, 0.0278, 0.0280, 0.0282, 0.0284, 0.0286,\n",
      "        0.0288, 0.0290, 0.0292, 0.0294, 0.0296, 0.0298, 0.0300, 0.0302, 0.0304,\n",
      "        0.0307, 0.0309, 0.0311, 0.0313, 0.0315, 0.0317, 0.0320, 0.0322, 0.0324,\n",
      "        0.0326, 0.0328, 0.0331, 0.0333, 0.0335, 0.0338, 0.0340, 0.0342, 0.0345,\n",
      "        0.0347, 0.0350, 0.0352, 0.0354, 0.0357, 0.0359, 0.0362, 0.0364, 0.0367,\n",
      "        0.0369, 0.0372, 0.0375, 0.0377, 0.0380, 0.0382, 0.0385, 0.0388, 0.0390,\n",
      "        0.0393, 0.0396, 0.0399, 0.0401, 0.0404, 0.0407, 0.0410, 0.0413, 0.0416,\n",
      "        0.0418, 0.0421, 0.0424, 0.0427, 0.0430, 0.0433, 0.0436, 0.0439, 0.0442,\n",
      "        0.0445, 0.0448, 0.0451, 0.0455, 0.0458, 0.0461, 0.0464, 0.0467, 0.0471,\n",
      "        0.0474, 0.0477, 0.0480, 0.0484, 0.0487, 0.0491, 0.0494, 0.0497, 0.0501,\n",
      "        0.0504, 0.0508, 0.0511, 0.0515, 0.0518, 0.0522, 0.0526, 0.0529, 0.0533,\n",
      "        0.0537, 0.0540, 0.0544, 0.0548, 0.0552, 0.0556, 0.0559, 0.0563, 0.0567,\n",
      "        0.0571, 0.0575, 0.0579, 0.0583, 0.0587, 0.0591, 0.0595, 0.0599, 0.0604,\n",
      "        0.0608, 0.0612, 0.0616, 0.0621, 0.0625, 0.0629, 0.0634, 0.0638, 0.0642,\n",
      "        0.0647, 0.0651, 0.0656, 0.0660, 0.0665, 0.0670, 0.0674, 0.0679, 0.0684,\n",
      "        0.0688, 0.0693, 0.0698, 0.0703, 0.0708, 0.0713, 0.0718, 0.0723, 0.0728,\n",
      "        0.0733, 0.0738, 0.0743, 0.0748, 0.0753, 0.0758, 0.0764, 0.0769, 0.0774,\n",
      "        0.0780, 0.0785, 0.0790, 0.0796, 0.0802, 0.0807, 0.0813, 0.0818, 0.0824,\n",
      "        0.0830, 0.0835, 0.0841, 0.0847, 0.0853, 0.0859, 0.0865, 0.0871, 0.0877,\n",
      "        0.0883, 0.0889, 0.0895, 0.0901, 0.0908, 0.0914, 0.0920, 0.0927, 0.0933,\n",
      "        0.0940, 0.0946, 0.0953, 0.0959, 0.0966, 0.0973, 0.0979, 0.0986, 0.0993,\n",
      "        0.1000, 0.1007, 0.1014, 0.1021, 0.1028, 0.1035, 0.1042, 0.1050, 0.1057,\n",
      "        0.1064, 0.1072, 0.1079, 0.1087, 0.1094, 0.1102, 0.1109, 0.1117, 0.1125,\n",
      "        0.1133, 0.1140, 0.1148, 0.1156, 0.1164, 0.1172, 0.1181, 0.1189, 0.1197,\n",
      "        0.1205, 0.1214, 0.1222, 0.1231, 0.1239, 0.1248, 0.1256, 0.1265, 0.1274,\n",
      "        0.1283, 0.1292, 0.1301, 0.1310, 0.1319, 0.1328, 0.1337, 0.1346, 0.1356,\n",
      "        0.1365, 0.1374, 0.1384, 0.1394, 0.1403, 0.1413, 0.1423, 0.1433, 0.1443,\n",
      "        0.1453, 0.1463, 0.1473, 0.1483, 0.1493, 0.1504, 0.1514, 0.1525, 0.1535,\n",
      "        0.1546, 0.1557, 0.1567, 0.1578, 0.1589, 0.1600, 0.1611, 0.1623, 0.1634,\n",
      "        0.1645, 0.1657, 0.1668, 0.1680, 0.1691, 0.1703, 0.1715, 0.1727, 0.1739,\n",
      "        0.1751, 0.1763, 0.1775, 0.1788, 0.1800, 0.1812, 0.1825, 0.1838, 0.1850,\n",
      "        0.1863, 0.1876, 0.1889, 0.1902, 0.1916, 0.1929, 0.1942, 0.1956, 0.1969,\n",
      "        0.1983, 0.1997, 0.2010, 0.2024, 0.2038, 0.2053, 0.2067, 0.2081, 0.2096,\n",
      "        0.2110, 0.2125, 0.2140, 0.2154, 0.2169, 0.2184, 0.2200, 0.2215, 0.2230,\n",
      "        0.2246, 0.2261, 0.2277, 0.2293, 0.2309, 0.2325, 0.2341, 0.2357, 0.2373,\n",
      "        0.2390, 0.2406, 0.2423, 0.2440, 0.2457, 0.2474, 0.2491, 0.2508, 0.2526,\n",
      "        0.2543, 0.2561, 0.2579, 0.2597, 0.2615, 0.2633, 0.2651, 0.2669, 0.2688,\n",
      "        0.2707, 0.2725, 0.2744, 0.2763, 0.2783, 0.2802, 0.2821, 0.2841, 0.2861,\n",
      "        0.2880, 0.2900, 0.2921, 0.2941, 0.2961, 0.2982, 0.3002, 0.3023, 0.3044,\n",
      "        0.3065, 0.3087, 0.3108, 0.3130, 0.3151, 0.3173, 0.3195, 0.3217, 0.3240,\n",
      "        0.3262, 0.3285, 0.3308, 0.3331, 0.3354, 0.3377, 0.3400, 0.3424, 0.3448,\n",
      "        0.3472, 0.3496, 0.3520, 0.3544, 0.3569, 0.3594, 0.3619, 0.3644, 0.3669,\n",
      "        0.3695, 0.3720, 0.3746, 0.3772, 0.3798, 0.3825, 0.3851, 0.3878, 0.3905,\n",
      "        0.3932, 0.3959, 0.3987, 0.4014, 0.4042, 0.4070, 0.4098, 0.4127, 0.4155,\n",
      "        0.4184, 0.4213, 0.4243, 0.4272, 0.4302, 0.4331, 0.4362, 0.4392, 0.4422,\n",
      "        0.4453, 0.4484, 0.4515, 0.4546, 0.4578, 0.4610, 0.4642, 0.4674, 0.4706,\n",
      "        0.4739, 0.4772, 0.4805, 0.4838, 0.4872, 0.4906, 0.4940, 0.4974, 0.5008,\n",
      "        0.5043, 0.5078, 0.5113, 0.5149, 0.5185, 0.5221, 0.5257, 0.5293, 0.5330,\n",
      "        0.5367, 0.5404, 0.5442, 0.5479, 0.5517, 0.5556, 0.5594, 0.5633, 0.5672,\n",
      "        0.5712, 0.5751, 0.5791, 0.5831, 0.5872, 0.5913, 0.5954, 0.5995, 0.6036,\n",
      "        0.6078, 0.6120, 0.6163, 0.6206, 0.6249, 0.6292, 0.6336, 0.6380, 0.6424,\n",
      "        0.6469, 0.6513, 0.6559, 0.6604, 0.6650, 0.6696, 0.6743, 0.6789, 0.6837,\n",
      "        0.6884, 0.6932, 0.6980, 0.7028, 0.7077, 0.7126, 0.7176, 0.7225, 0.7275,\n",
      "        0.7326, 0.7377, 0.7428, 0.7480, 0.7531, 0.7584, 0.7636, 0.7689, 0.7743,\n",
      "        0.7796, 0.7850, 0.7905, 0.7960, 0.8015, 0.8071, 0.8127, 0.8183, 0.8240,\n",
      "        0.8297, 0.8355, 0.8412, 0.8471, 0.8530, 0.8589, 0.8648, 0.8708, 0.8769,\n",
      "        0.8830, 0.8891, 0.8953, 0.9015, 0.9077, 0.9140, 0.9204, 0.9268, 0.9332,\n",
      "        0.9397, 0.9462, 0.9528, 0.9594, 0.9660, 0.9727, 0.9795, 0.9863, 0.9931,\n",
      "        1.0000])\n"
     ]
    }
   ],
   "source": [
    "# We define a list that goes from -3 to 0 with 1000 steps\n",
    "lossExpression = torch.linspace(start=-3, end=0, steps=1000)\n",
    "# We do a 10^value expression on the value of the above list\n",
    "lossExponentExpression = 10**lossExpression\n",
    "print(lossExponentExpression)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e984d652-76b3-4fb8-a254-a1cd51cdcd3e",
   "metadata": {},
   "source": [
    "#### We reset the parameters again..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "dad12071-d899-45c9-aa48-852997c283b4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Resetting the parameters\n",
    "# We will define a generator to give the same result on your machine, as of my machine\n",
    "generator = torch.Generator().manual_seed(6942069420)\n",
    "# Embedding Matrix (Input Layer)\n",
    "embeddingFeatureSpaceLength = 2\n",
    "embeddingLookUpMatrix = torch.randn((len(stoi),embeddingFeatureSpaceLength), generator=generator)\n",
    "# Hidden Layer\n",
    "numberOfHiddenLayerNeurons = 100\n",
    "weightsOfHiddenLayer = torch.randn((inputBlockSize*embeddingFeatureSpaceLength), numberOfHiddenLayerNeurons, generator=generator)\n",
    "biasesOfHiddenLayer = torch.randn(numberOfHiddenLayerNeurons, generator=generator)\n",
    "# Output Layer / Final Layer\n",
    "numberOfFinalLayerOutputs = 27\n",
    "weightsOfFinalLayer = torch.randn(numberOfHiddenLayerNeurons, numberOfFinalLayerOutputs, generator=generator)\n",
    "biasesOfFinalLayer = torch.randn(numberOfFinalLayerOutputs, generator=generator)\n",
    "# Parameters\n",
    "parameters = [embeddingLookUpMatrix, weightsOfHiddenLayer, biasesOfHiddenLayer, weightsOfFinalLayer, biasesOfFinalLayer]\n",
    "# We set all the requires gradient to True\n",
    "for parameter in parameters:\n",
    "    parameter.requires_grad = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "b1e2ffdf-d9a0-446f-9b0f-9a2386e26fbd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We define a list that goes from -3 to 0 with 1000 steps\n",
    "lossExpression = torch.linspace(start=-3, end=0, steps=1000)\n",
    "# We do a 10^value expression on the value of the above list\n",
    "lossExponentExpression = 10**lossExpression"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "054f2aec-201d-4594-b41e-1acc1f5e6adf",
   "metadata": {},
   "source": [
    "We will keep the steps and epoch same, to math the number of checks..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "4d12d5be-892c-43db-8739-f945407e794f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Minibatch Loss: tensor(20.4912, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(17.4186, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(20.8297, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.9427, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(20.8288, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(19.8154, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.9068, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.8447, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(17.1165, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(21.9518, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(19.8166, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(19.4713, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(19.6362, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(20.2488, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(19.6756, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(17.4145, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.3921, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(17.7899, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.0282, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(19.2862, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(16.6994, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(19.9380, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(17.8906, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(16.8535, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(20.2291, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(19.4602, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(19.3942, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(17.3150, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(16.9723, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.0909, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(17.8714, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(19.3003, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.0059, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.1657, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(19.0020, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.1723, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.6499, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(20.6922, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.4962, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(17.4427, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(20.0298, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.5214, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(16.5244, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(20.6902, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(19.0853, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(22.6060, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(20.0410, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.7485, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(17.5248, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.7001, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(15.7614, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.1734, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.8555, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(16.1428, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(16.3573, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.7726, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.2010, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(19.3895, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(17.8117, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(17.4714, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(16.9424, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.2857, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(20.0329, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(16.5971, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(17.0118, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(17.5614, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(14.5226, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(19.7410, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(16.1596, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(14.3694, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.6775, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(16.3507, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(17.6601, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(16.2013, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(19.5013, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(19.9213, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(16.8504, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.1459, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(16.7638, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.2206, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(19.2426, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(17.8632, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(19.1835, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(16.4059, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(15.9377, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(17.2539, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(17.0977, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(17.3585, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(17.3815, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(15.8533, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(15.5711, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(16.7846, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(19.9322, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(19.0058, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(15.6691, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(17.6618, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(17.0695, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(16.6016, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(14.8068, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(15.1435, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.4321, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(16.8395, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.1555, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.4815, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(14.4432, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(16.0232, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(17.4891, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(16.2668, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(14.6523, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(16.1018, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(13.7893, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(16.8730, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(16.2315, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(15.7081, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(15.5245, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(16.3191, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.7459, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(16.2755, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(19.3403, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(15.1807, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(15.2600, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(17.3231, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(15.3303, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(15.7613, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(15.6253, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(15.4255, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(16.4036, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(17.6477, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(14.9193, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(11.9905, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(12.9993, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(15.2438, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(14.8276, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(15.3252, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(13.4130, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(15.5446, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(14.6410, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(16.7733, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(14.3540, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(13.6546, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(14.5861, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(15.0301, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(17.3632, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(15.9596, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(14.0117, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(13.7332, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(16.7970, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(16.9079, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(13.7508, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(16.5703, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(16.1306, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(14.0562, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(14.6766, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(17.5642, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(17.0197, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(13.6685, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(14.4749, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(16.2406, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(11.9002, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(13.1910, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(14.5569, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(12.9444, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(15.3357, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(14.9535, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(15.2503, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(11.8590, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(15.5899, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(18.6072, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(13.2304, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(16.5471, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(15.1719, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(16.4125, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(15.0359, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(14.3595, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(13.7893, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(14.8197, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(12.7571, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(17.8429, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(15.5250, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(11.6326, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(14.0599, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(12.6003, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(15.4788, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(13.3946, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(15.3095, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(16.0967, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(15.0045, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(12.5983, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(14.9008, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(14.5302, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(12.7388, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(14.5004, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(13.5904, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(13.1373, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(13.9978, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(13.4316, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(11.5654, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(13.2507, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(11.7604, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(12.0264, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(15.4757, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(12.7509, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(14.0376, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(12.6865, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(13.4726, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(13.9470, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(12.3492, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(10.8517, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(13.7454, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(13.2177, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(13.3970, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(14.1428, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(11.4850, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(10.2485, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(10.8399, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(12.8080, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(13.5316, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(12.9736, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(11.9629, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(12.1304, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(13.2114, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(14.3808, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(12.9012, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(11.8290, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(12.8093, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(13.8044, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(13.8439, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(15.1905, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(13.5518, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(15.1755, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(13.4574, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(16.0383, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(9.8931, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(12.8031, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(12.6703, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(11.0735, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(11.0370, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(12.2278, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(12.3781, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(12.1849, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(12.0085, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(11.2228, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(12.1810, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(10.7823, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(14.8053, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(11.8044, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(10.8542, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(11.1839, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(11.7818, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(9.8621, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(11.7249, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(13.8904, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(10.1689, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(9.6776, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(12.5365, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(10.7882, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(9.4905, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(9.8545, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(11.0174, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(12.4723, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(11.6088, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(13.6235, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(12.3976, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(13.5923, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(10.8110, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(13.8400, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(11.4732, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(8.6952, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(10.5240, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(10.8705, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(10.6922, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(12.1532, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(13.0562, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(11.0198, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(10.7559, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(11.6621, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(9.6515, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(8.9815, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(10.0166, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(10.8754, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(11.1015, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(10.0597, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(10.1828, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(12.1216, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(10.1199, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(8.2428, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(8.4995, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(11.2877, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(10.7374, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(11.7477, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(12.5374, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(9.7936, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(10.7637, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(9.0521, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(12.2178, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(9.7976, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(10.1258, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(8.6971, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(11.6873, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(9.2130, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(11.4977, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(9.9371, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(12.0132, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(11.5767, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(11.5453, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(9.2883, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(9.9654, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(10.4787, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(11.3985, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(9.3882, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(9.5932, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(10.8470, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(11.4367, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(10.3075, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(8.5666, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(8.4824, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(11.3337, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(10.8704, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(9.3970, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(9.9005, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(7.8623, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(7.1744, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(9.3159, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(7.9415, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(9.8310, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(8.7361, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(8.7933, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(9.4119, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(8.9783, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(12.5447, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(9.3935, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(8.8151, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(8.9710, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(7.6835, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(10.6254, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(7.5135, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(9.7464, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(8.8841, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(9.9691, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(9.3117, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(9.3461, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(8.7871, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.9664, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(8.2588, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(9.4437, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(8.4799, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(9.1718, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(9.2627, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(8.9234, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(8.7448, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(7.0827, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(7.7979, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(8.7187, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(7.9522, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(10.5851, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(7.6949, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(8.1608, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(7.0734, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.8116, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(9.3782, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(8.9185, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.8833, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.2579, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(9.2238, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(7.3516, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(8.5661, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(7.2529, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(8.6248, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(7.3675, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(7.7188, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(9.2133, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(8.8272, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(9.4415, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(8.2808, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(7.2058, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(8.5812, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.8569, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(7.7025, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(8.5764, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(9.2971, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(8.0955, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(8.0089, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.1281, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(8.9231, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(7.3153, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(8.2925, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(10.0021, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(7.7346, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(8.5447, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.7268, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(7.7912, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(9.1724, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(7.0787, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.3020, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.2492, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.3686, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.8408, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(7.2837, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(8.1081, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(9.1641, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(7.1782, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.3273, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.9475, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.6859, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.3164, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.3142, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(7.6005, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.5019, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.5032, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.3056, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.1561, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.6929, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.2567, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(7.8991, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(7.7745, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(8.5245, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.3666, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.0870, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.3304, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.8057, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.6272, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.1547, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.4347, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.9472, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(7.2264, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.2934, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.8402, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.6946, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(7.2830, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.4114, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.2043, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(7.0075, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(7.0391, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.1147, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.7270, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.0668, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.1977, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(7.7689, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.8711, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.7478, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.2203, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.6282, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.6494, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.7574, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.4036, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.2250, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.7760, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.0868, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.3005, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.0378, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.5140, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.0503, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.5376, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.4956, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.8792, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.1721, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.6276, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.7565, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.5887, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.6420, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.8186, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.8662, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.3585, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.3312, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.2079, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.7271, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.0850, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.0397, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.8361, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.7151, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.6118, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.1278, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.0840, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.7138, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.9715, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.9443, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.0344, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.4522, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.8366, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.0607, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.9240, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.9477, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.8554, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.5014, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.5526, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.5685, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.8825, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.4743, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.5544, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.9730, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.6575, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.6130, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.2999, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.1874, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.1927, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.3721, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.9157, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.7582, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.5735, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.9710, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.0027, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.6547, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.9528, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.6853, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.7708, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.3240, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.0639, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.9838, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.4860, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.1543, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.2474, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.3442, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.1744, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.1449, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.4260, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.9841, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.0951, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.2256, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.8859, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.6860, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.5996, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.0174, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.5378, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.4632, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.5860, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.2713, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.8284, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.6684, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.3685, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.3204, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.7575, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.0139, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.1160, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.6940, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.5882, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.9343, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.5427, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.6851, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.5542, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.6567, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.2806, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.0175, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.8892, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.0544, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.2230, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.3644, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.1381, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.6517, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.5406, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.4827, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.4675, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.2011, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.5130, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.1130, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.0822, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.3254, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.6119, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.0661, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.1785, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.7636, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.3402, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.4231, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.3142, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.2087, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.6181, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.6190, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.3055, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.6946, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.9956, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.0093, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.2342, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.0116, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.8860, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.7274, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.5424, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.6028, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.0562, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.6141, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.3638, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.0034, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.1715, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.0620, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.1485, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.2516, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.4663, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.4586, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.8329, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.0198, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.4509, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.4989, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.5177, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.5173, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.8041, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.0583, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.4396, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.0951, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.3275, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.2951, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.5264, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.7573, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.4302, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.1317, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.6570, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.2871, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.3021, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.7207, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.6836, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.2189, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.5531, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.8258, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.4334, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.5305, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.9256, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.9770, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.1233, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.4572, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.1348, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.8609, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.7629, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.3052, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.7948, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.2810, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.5863, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.2496, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.8428, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.7525, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.0256, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.3055, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.9883, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.0784, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.0469, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.1976, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.3348, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.8717, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.9508, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.1728, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.4123, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.2565, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.1945, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.6065, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.3036, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.5075, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.4240, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.8673, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.8651, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.1075, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.9760, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.3214, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.8847, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.9154, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.3279, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.8672, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.4736, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.8214, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.7351, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.9844, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.1677, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.2246, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.0464, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.8237, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.6917, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.5612, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.6939, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.7218, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.3447, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.1385, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.3904, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.6493, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.1889, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.2298, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.3976, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.2549, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.7113, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.7164, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.0965, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.8778, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.1583, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.4735, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.8247, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.4040, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.4804, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.4767, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.0096, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.2775, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.8461, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.4689, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.9950, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.1587, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.1268, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.1065, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.6236, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.0683, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.4881, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.4859, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.4505, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.9125, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.9214, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.7874, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.6983, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.9684, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.1251, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.3038, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.0094, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.9245, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.7084, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.8558, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.0824, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.7279, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.8996, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.4505, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.1420, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.6486, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.7235, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.6513, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.1239, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.9665, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.0433, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.0704, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.5763, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.9479, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.8320, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.4647, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.9162, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.2407, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.1861, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.4920, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.6506, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.7008, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.1234, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.8841, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.8408, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.8432, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.5041, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.2696, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.0803, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.4093, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.7958, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.3550, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.1940, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.2778, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.8456, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.4902, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.0235, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.9795, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.3032, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.9420, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.7222, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.9345, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.4599, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.2153, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.2703, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.1361, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.6507, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.8314, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.5378, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.2147, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.6815, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.2494, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.8211, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.8165, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.0173, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.6428, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.4418, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.5791, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.2666, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.1890, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.0579, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.0375, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.2791, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.5206, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.7534, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.3896, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.9079, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.4202, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.4884, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.7763, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.1403, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.1091, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.1154, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.9103, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.2125, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.8847, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.3325, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.7768, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.5079, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.5834, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.8362, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.1704, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.4223, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.4083, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.4408, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.3903, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.8208, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.4035, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.9448, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.2326, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.2156, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.2275, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.2844, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.3825, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.3915, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.4635, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.3783, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.2519, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.6898, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.4711, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.1525, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.8035, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.8755, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.7925, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.0081, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.3395, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.7853, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.7368, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.4579, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.2999, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.8562, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.2875, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.3914, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.2373, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.1608, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.4813, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.9507, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.5454, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.1820, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.4034, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.3209, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.7503, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.7673, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.7485, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.7652, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.1654, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.4907, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.8491, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.8532, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.8825, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.8717, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.5258, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.2314, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.6301, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.8719, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.4109, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.7072, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.4025, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.5492, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.2140, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.0223, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.8044, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.7858, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.5065, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.2313, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.2064, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.0450, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.0559, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.5359, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.3184, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.8273, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.3789, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.8334, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.3433, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.4879, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.8402, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.9261, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.7438, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.0146, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.2149, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.0184, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.4692, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.0319, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.2623, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.7278, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.4890, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.3659, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.0255, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.0085, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.7435, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.7767, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.7773, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.7239, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(2.7940, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.5682, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.4581, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.3109, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.0837, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.7160, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.5682, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.5221, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.1561, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.0659, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.8451, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.1776, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.4453, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.7183, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.9349, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.9723, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.2942, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.1405, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.5436, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.6515, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.0576, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.4429, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.0539, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.5508, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.6898, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.6649, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.7990, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.4768, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.4402, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.8094, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.5616, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.2707, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.1196, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.2676, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.2018, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.2564, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.2850, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.0567, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.0821, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.8544, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.0748, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.3260, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.3925, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.1981, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.1856, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.2029, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.7878, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.1230, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.7503, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.4902, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.3714, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(7.7509, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(7.7169, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.3538, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(7.0388, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.5350, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.0623, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.3691, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.0915, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.9271, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.1847, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.4632, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.5479, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.3754, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.1154, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.0814, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.1597, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.9694, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.3178, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.1472, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.6873, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.6088, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.7085, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.5715, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.1246, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.9629, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.1239, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.4517, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.4937, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.3849, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.7596, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.9566, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.5601, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.5245, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.8115, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.7990, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.8567, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(8.6162, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.0928, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.0589, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.6343, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(7.6344, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.8231, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.8390, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.2491, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.2697, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.2776, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.9418, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.2260, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.5369, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.9716, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.1544, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.2904, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.0499, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.7655, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.4295, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.8551, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.2901, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.9725, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(7.0154, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.0674, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.6992, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(4.9498, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.7657, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.0973, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.1496, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.4973, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(5.6832, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.5158, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(3.8223, grad_fn=<NllLossBackward0>)\n",
      "Minibatch Loss: tensor(6.0186, grad_fn=<NllLossBackward0>)\n"
     ]
    }
   ],
   "source": [
    "# We want to track the learning rates that we use in the training\n",
    "learningRates = []\n",
    "# We want to track the losses that we use in the training\n",
    "losses = []\n",
    "\n",
    "# We define the number of epochs\n",
    "epochs = 1000\n",
    "for i in range(epochs):\n",
    "    # Mini-Batches\n",
    "    indexes = torch.randint(low=0, high=inputs.shape[0], size=(37,))\n",
    "    \n",
    "    # Forward Pass (Mini-Batch)\n",
    "    embedding = embeddingLookUpMatrix[inputs[indexes]] # Indexing into look-up table\n",
    "    hiddenLayerStates = torch.tanh(embedding.view(-1, inputBlockSize*embeddingFeatureSpaceLength) @ weightsOfHiddenLayer + biasesOfHiddenLayer)\n",
    "    logits = hiddenLayerStates @ weightsOfFinalLayer + biasesOfFinalLayer\n",
    "    loss = F.cross_entropy(logits, outputs[indexes]) # Indexing into labels\n",
    "\n",
    "    print(\"Minibatch Loss:\", loss)\n",
    "    \n",
    "    # Backward Pass (Mini-Batch)\n",
    "    for parameter in parameters:\n",
    "        parameter.grad = None\n",
    "    loss.backward()\n",
    "    \n",
    "    # Update Weights (Mini-Batch)\n",
    "    learning_rate = lossExponentExpression[i]\n",
    "    for parameter in parameters:\n",
    "        parameter.data += -learning_rate * parameter.grad\n",
    "\n",
    "    # Tracking the stats\n",
    "    # We append both the learning rates and losses that we use per iteration\n",
    "    learningRates.append(lossExponentExpression[i])\n",
    "    losses.append(loss.item())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "619465cb-e1ee-4cc1-b170-9b767516d7dd",
   "metadata": {},
   "source": [
    "Now we can plot and check the learning rates and the losses..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "e77e35f2-59d8-4b44-bb0b-aa384d7348cf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x20f30f610d0>]"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiwAAAGdCAYAAAAxCSikAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAABxR0lEQVR4nO3deXhU1fkH8O+dSWay7zuEfd8VAUFQFMqiVbF1qbWK1qW1aLVUW6lt1WqLVX9ttVK1VkXrgktFrVoQkUUURFZFIEIIS4AkBEgm62Qyc39/zJw7995ZMpPMZG6S7+d58mhm7sycuQm577znPe+RZFmWQURERGRgplgPgIiIiKgtDFiIiIjI8BiwEBERkeExYCEiIiLDY8BCREREhseAhYiIiAyPAQsREREZHgMWIiIiMry4WA8gElwuF44dO4bU1FRIkhTr4RAREVEIZFlGXV0dioqKYDIFz6F0i4Dl2LFjKC4ujvUwiIiIqB2OHDmC3r17Bz2mWwQsqampANxvOC0tLcajISIiolDYbDYUFxcr1/FgukXAIqaB0tLSGLAQERF1MaGUc7DoloiIiAyPAQsREREZHgMWIiIiMjwGLERERGR4DFiIiIjI8BiwEBERkeExYCEiIiLDY8BCREREhseAhYiIiAyPAQsREREZHgMWIiIiMrywApbFixdjwoQJSE1NRV5eHubNm4eSkhLl/lOnTuH222/H0KFDkZiYiD59+uDnP/85amtrgz7v9ddfD0mSNF9z5sxp3zsiIiKibiesgGXdunVYsGABNm3ahFWrVsHhcGDWrFloaGgAABw7dgzHjh3DY489hl27dmHp0qVYsWIFbrzxxjafe86cOTh+/Ljy9dprr7XvHUXY1kOn8e+NByHLcqyHQkRE1GOFtVvzihUrNN8vXboUeXl52Lp1K84991yMGjUK//nPf5T7Bw4ciD/+8Y/40Y9+hNbWVsTFBX45q9WKgoKCMIcffd9/6nMAQK/MRFwwLD/GoyEiIuqZOlTDIqZ6srKygh6TlpYWNFgBgLVr1yIvLw9Dhw7FrbfeipMnTwY81m63w2azab6iray6MeqvQURERP61O2BxuVy48847cc4552DUqFF+j6mursaDDz6IW265JehzzZkzBy+99BJWr16NP//5z1i3bh3mzp0Lp9Pp9/jFixcjPT1d+SouLm7v2yAiIqIuQJLbWZxx66234n//+x82bNiA3r17+9xvs9nwne98B1lZWXjvvfcQHx8f8nMfOHAAAwcOxMcff4wZM2b43G+322G32zWvVVxcrGRzIqnfPR8AAH733RG4cWr/iD43ERFRT2az2ZCenh7S9btdGZbbbrsN77//PtasWeM3WKmrq8OcOXOQmpqK5cuXhxWsAMCAAQOQk5OD/fv3+73farUiLS1N80VERETdV1gBiyzLuO2227B8+XJ88skn6N/fN+Ngs9kwa9YsWCwWvPfee0hISAh7UOXl5Th58iQKCwvDfiwRERF1P2EFLAsWLMDLL7+MV199FampqaioqEBFRQWampoAeIOVhoYGPPfcc7DZbMox6nqUYcOGYfny5QCA+vp63H333di0aRMOHjyI1atX49JLL8WgQYMwe/bsCL5VIiIi6qrCWtb81FNPAQCmT5+uuf2FF17A9ddfj23btuGLL74AAAwaNEhzTFlZGfr16wcAKCkpUVYYmc1mfPXVV3jxxRdRU1ODoqIizJo1Cw8++CCsVmt73lNUSLEeABERUQ8WVsDSVn3u9OnTQ2qwpj4mMTERK1euDGcYRERE1MNwLyEiIiIyPAYsREREZHgMWEJUVt0Q6yEQERH1WAxYQmRv9d91l4iIiKKPAUuIJK4TIiIiihkGLCGS0a4dDIiIiCgCGLAQERGR4TFgISIiIsNjwEJERESGx4CFiIiIDI8BCxERERkeA5YQhbBFEhEREUUJA5YQMV4hIiKKHQYsREREZHgMWIiIiMjwGLAQERGR4TFgISIiIsNjwBIirhIiIiKKHQYsREREZHgMWIiIiMjwGLAQERGR4TFgCZHM1nFEREQxw4CFiIiIDI8BS6iYYCEiIooZBixERERkeAxYgnA4Xcr/N7Y4YzgSIiKino0BSxDqZnFOdo4jIiKKGQYsREREZHgMWELEBAsREVHsMGAJQpJiPQIiIiICwgxYFi9ejAkTJiA1NRV5eXmYN28eSkpKNMc0NzdjwYIFyM7ORkpKCr7//e+jsrIy6PPKsozf//73KCwsRGJiImbOnIl9+/aF/26iiikWIiKiWAkrYFm3bh0WLFiATZs2YdWqVXA4HJg1axYaGhqUY37xi1/gv//9L958802sW7cOx44dw/e+972gz/vII4/giSeewNNPP40vvvgCycnJmD17Npqbm9v3rqKAU0JERESxI8ly+y/FJ06cQF5eHtatW4dzzz0XtbW1yM3NxauvvorLL78cALB3714MHz4cGzduxNlnn+3zHLIso6ioCL/85S9x1113AQBqa2uRn5+PpUuX4gc/+EGb47DZbEhPT0dtbS3S0tLa+3Z8OJwuDL73fwCAGcPy8Nz1EyL23ERERD1dONfvDtWw1NbWAgCysrIAAFu3boXD4cDMmTOVY4YNG4Y+ffpg48aNfp+jrKwMFRUVmsekp6dj0qRJAR9jt9ths9k0X9GgLmH5ouxUVF6DiIiI2tbugMXlcuHOO+/EOeecg1GjRgEAKioqYLFYkJGRoTk2Pz8fFRUVfp9H3J6fnx/yYxYvXoz09HTlq7i4uL1vI2T19taovwYRERH51+6AZcGCBdi1axeWLVsWyfGEZNGiRaitrVW+jhw50uljICIios7TroDltttuw/vvv481a9agd+/eyu0FBQVoaWlBTU2N5vjKykoUFBT4fS5xu34lUbDHWK1WpKWlab6IiIio+worYJFlGbfddhuWL1+OTz75BP3799fcP378eMTHx2P16tXKbSUlJTh8+DAmT57s9zn79++PgoICzWNsNhu++OKLgI/pLBIbsRARERlCWAHLggUL8PLLL+PVV19FamoqKioqUFFRgaamJgDuYtkbb7wRCxcuxJo1a7B161bccMMNmDx5smaF0LBhw7B8+XIA7qDgzjvvxEMPPYT33nsPX3/9Na677joUFRVh3rx5kXunRERE1GXFhXPwU089BQCYPn265vYXXngB119/PQDgr3/9K0wmE77//e/Dbrdj9uzZ+Mc//qE5vqSkRFlhBAC/+tWv0NDQgFtuuQU1NTWYOnUqVqxYgYSEhHa8JSIiIupuOtSHxSii1YfF6ZIx8DcfKt8ffPiiiD03ERFRT9dpfVi6O1awEBERGQMDFiIiIjI8BixERERkeAxYguCqZiIiImNgwEJERESGx4CFiIiIDI8BCxERERkeA5Yg2JqfiIjIGBiwEBERkeExYCEiIiLDY8BCREREhseAhYiIiAyPAQsREREZHgMWIiIiMjwGLERERGR4DFiIiIjI8BiwhKG2yRHrIRAREfVIDFjC4HLJsR4CERFRj8SAhYiIiAyPAQsREREZHgMWIiIiMjwGLERERGR4DFiIiIjI8BiwhIFrhIiIiGKDAQsREREZHgMWIiIiMjwGLERERGR4DFiIiIjI8BiwhKG63h7rIRAREfVIDFjC8M2x2lgPgYiIqEcKO2BZv349Lr74YhQVFUGSJLzzzjua+yVJ8vv16KOPBnzO+++/3+f4YcOGhf1mok3mumYiIqKYCDtgaWhowNixY7FkyRK/9x8/flzz9fzzz0OSJHz/+98P+rwjR47UPG7Dhg3hDo2IiIi6qbhwHzB37lzMnTs34P0FBQWa7999912cf/75GDBgQPCBxMX5PNZoJCnWIyAiIuqZolrDUllZiQ8++AA33nhjm8fu27cPRUVFGDBgAK655hocPnw44LF2ux02m03zRURERN1XVAOWF198Eampqfje974X9LhJkyZh6dKlWLFiBZ566imUlZVh2rRpqKur83v84sWLkZ6ernwVFxdHY/g+JDDFQkREFAtRDVief/55XHPNNUhISAh63Ny5c3HFFVdgzJgxmD17Nj788EPU1NTgjTfe8Hv8okWLUFtbq3wdOXIkGsMnIiIigwi7hiVUn376KUpKSvD666+H/diMjAwMGTIE+/fv93u/1WqF1Wrt6BDDxhoWIiKi2IhahuW5557D+PHjMXbs2LAfW19fj9LSUhQWFkZhZO3HZc1ERESxEXbAUl9fjx07dmDHjh0AgLKyMuzYsUNTJGuz2fDmm2/ipptu8vscM2bMwJNPPql8f9ddd2HdunU4ePAgPv/8c1x22WUwm824+uqrwx1eVDHDQkREFBthTwlt2bIF559/vvL9woULAQDz58/H0qVLAQDLli2DLMsBA47S0lJUV1cr35eXl+Pqq6/GyZMnkZubi6lTp2LTpk3Izc0Nd3hERETUDUmy3PUnOmw2G9LT01FbW4u0tLSIPne/ez5Q/v/xH4zDpeN6RfT5iYiIeqpwrt/cSygMEueEiIiIYoIBCxERERkeA5YwML9CREQUGwxYiIiIyPAYsBAREZHhMWAhIiIiw2PAQkRERIbHgCUMXNVMREQUGwxYiIiIyPAYsIRB4sJmIiKimGDAQkRERIbHgIWIiIgMjwELERERGR4DFiIiIjI8Bixh4LJmIiKi2GDAQkRERIbHgCUMj64sifUQiIiIeiQGLGEoq26I9RCIiIh6JAYsREREZHgMWIiIiMjwGLAQERGR4TFgISIiIsNjwEJERESGx4CFiIiIDI8BCxERERkeAxYiIiIyPAYsREREZHgMWIiIiMjwGLAQERGR4TFgISIiIsMLO2BZv349Lr74YhQVFUGSJLzzzjua+6+//npIkqT5mjNnTpvPu2TJEvTr1w8JCQmYNGkSNm/eHO7QiIiIqJsKO2BpaGjA2LFjsWTJkoDHzJkzB8ePH1e+XnvttaDP+frrr2PhwoW47777sG3bNowdOxazZ89GVVVVuMMjIiKibigu3AfMnTsXc+fODXqM1WpFQUFByM/5l7/8BTfffDNuuOEGAMDTTz+NDz74AM8//zzuueeecIdIRERE3UxUaljWrl2LvLw8DB06FLfeeitOnjwZ8NiWlhZs3boVM2fO9A7KZMLMmTOxcePGaAyPiIiIupiwMyxtmTNnDr73ve+hf//+KC0txW9+8xvMnTsXGzduhNls9jm+uroaTqcT+fn5mtvz8/Oxd+9ev69ht9tht9uV7202W2TfBBERERlKxAOWH/zgB8r/jx49GmPGjMHAgQOxdu1azJgxIyKvsXjxYjzwwAMReS4iIiIyvqgvax4wYABycnKwf/9+v/fn5OTAbDajsrJSc3tlZWXAOphFixahtrZW+Tpy5EjEx01ERETGEfWApby8HCdPnkRhYaHf+y0WC8aPH4/Vq1crt7lcLqxevRqTJ0/2+xir1Yq0tDTNV2dpdjg77bWIiIjILeyApb6+Hjt27MCOHTsAAGVlZdixYwcOHz6M+vp63H333di0aRMOHjyI1atX49JLL8WgQYMwe/Zs5TlmzJiBJ598Uvl+4cKFePbZZ/Hiiy9iz549uPXWW9HQ0KCsGjKSuubWWA+BiIioxwm7hmXLli04//zzle8XLlwIAJg/fz6eeuopfPXVV3jxxRdRU1ODoqIizJo1Cw8++CCsVqvymNLSUlRXVyvfX3XVVThx4gR+//vfo6KiAuPGjcOKFSt8CnGJiIioZ5JkWZZjPYiOstlsSE9PR21tbcSnh/rd84Hm+y/vnYncVGuAo4mIiChU4Vy/uZdQmGR0+fiOiIioy2HAQkRERIbHgIWIiIgMjwFLmCRIsR4CERFRj8OAJUysYSEiIup8DFiIiIjI8BiwEBERkeExYCEiIiLDY8BCREREhseAhYiIiAyPAQsREREZHgOWMB2sboz1EIiIiHocBixhuu75L2I9BCIioh6HAUuYmh2uWA+BiIiox2HAQkRERIbHgIWIiIgMjwELERERGR4DFiIiIjI8BixERERkeAxYiIiIyPAYsBAREZHhMWAhIiIiw2PAQkRERIbHgIWIiIgMjwELERERGR4DFiIiIjI8BixERERkeAxYiIiIyPAYsBAREZHhMWAhIiIiwws7YFm/fj0uvvhiFBUVQZIkvPPOO8p9DocDv/71rzF69GgkJyejqKgI1113HY4dOxb0Oe+//35IkqT5GjZsWNhvhoiIiLqnsAOWhoYGjB07FkuWLPG5r7GxEdu2bcPvfvc7bNu2DW+//TZKSkpwySWXtPm8I0eOxPHjx5WvDRs2hDs0IiIi6qbiwn3A3LlzMXfuXL/3paenY9WqVZrbnnzySUycOBGHDx9Gnz59Ag8kLg4FBQXhDicm7K1OWOPMsR4GERFRjxH1Gpba2lpIkoSMjIygx+3btw9FRUUYMGAArrnmGhw+fDjaQ2u3CQ99HOshEBER9ShhZ1jC0dzcjF//+te4+uqrkZaWFvC4SZMmYenSpRg6dCiOHz+OBx54ANOmTcOuXbuQmprqc7zdbofdble+t9lsURl/ILbm1k59PSIiop4uagGLw+HAlVdeCVmW8dRTTwU9Vj3FNGbMGEyaNAl9+/bFG2+8gRtvvNHn+MWLF+OBBx6I+JiJiIjImKIyJSSClUOHDmHVqlVBsyv+ZGRkYMiQIdi/f7/f+xctWoTa2lrl68iRI5EYNhERERlUxAMWEazs27cPH3/8MbKzs8N+jvr6epSWlqKwsNDv/VarFWlpaZqvzibLcqe/JhERUU8VdsBSX1+PHTt2YMeOHQCAsrIy7NixA4cPH4bD4cDll1+OLVu24JVXXoHT6URFRQUqKirQ0tKiPMeMGTPw5JNPKt/fddddWLduHQ4ePIjPP/8cl112GcxmM66++uqOv0MiIiLq8sKuYdmyZQvOP/985fuFCxcCAObPn4/7778f7733HgBg3LhxmsetWbMG06dPBwCUlpaiurpaua+8vBxXX301Tp48idzcXEydOhWbNm1Cbm5uuMMjIiKibijsgGX69OlBp0NCmSo5ePCg5vtly5aFOwwiIiLqQbiXEBERERkeA5Z2Ys0tERFR52HA0oYbp/aP9RCIiIh6PAYsbThvCAt/iYiIYo0BSxsmDciK9RCIiIh6PAYsbQi0KzNLWIiIiDoPAxYiIiIyPAYsREREZHgMWIiIiMjwGLC0Ezc/JCIi6jwMWIiIiMjwGLAQERGR4TFgISIiIsNjwNJOrGAhIiLqPAxYiIiIyPAYsBAREZHhMWAhIiIiw2PA0k5sw0JERNR5GLAQERGR4TFgISIiIsNjwEJERESGx4ClnWR2YiEiIuo0DFiIiIjI8BiwEBERkeExYGmnW1/eFushEBER9RgMWNrpk71VsR4CERFRj8GAhYiIiAyPAQsREREZHgMWIiIiMjwGLERERGR4DFiIiIjI8MIOWNavX4+LL74YRUVFkCQJ77zzjuZ+WZbx+9//HoWFhUhMTMTMmTOxb9++Np93yZIl6NevHxISEjBp0iRs3rw53KFFzbVn9431EIiIiHq0sAOWhoYGjB07FkuWLPF7/yOPPIInnngCTz/9NL744gskJydj9uzZaG5uDvicr7/+OhYuXIj77rsP27Ztw9ixYzF79mxUVRlj6XBhRkKsh0BERNSjhR2wzJ07Fw899BAuu+wyn/tkWcbf/vY3/Pa3v8Wll16KMWPG4KWXXsKxY8d8MjFqf/nLX3DzzTfjhhtuwIgRI/D0008jKSkJzz//fLjD61R7jttiPQQiIqIeIaI1LGVlZaioqMDMmTOV29LT0zFp0iRs3LjR72NaWlqwdetWzWNMJhNmzpwZ8DF2ux02m03zFQtzH/8UVXWBM0dEREQUGRENWCoqKgAA+fn5mtvz8/OV+/Sqq6vhdDrDeszixYuRnp6ufBUXF0dg9O1zsLoxZq9NRETUU3TJVUKLFi1CbW2t8nXkyJGovt7wwrSoPj8REREFF9GApaCgAABQWVmpub2yslK5Ty8nJwdmszmsx1itVqSlpWm+omn6kNyoPj8REREFF9GApX///igoKMDq1auV22w2G7744gtMnjzZ72MsFgvGjx+veYzL5cLq1asDPqazSZIU6yEQERH1aHHhPqC+vh779+9Xvi8rK8OOHTuQlZWFPn364M4778RDDz2EwYMHo3///vjd736HoqIizJs3T3nMjBkzcNlll+G2224DACxcuBDz58/HWWedhYkTJ+Jvf/sbGhoacMMNN3T8HRIREVGXF3bAsmXLFpx//vnK9wsXLgQAzJ8/H0uXLsWvfvUrNDQ04JZbbkFNTQ2mTp2KFStWICHB28uktLQU1dXVyvdXXXUVTpw4gd///veoqKjAuHHjsGLFCp9CXCNi8oWIiCj6JFmW5VgPoqNsNhvS09NRW1sbtXqWfvd84Pf2N386GRP6ZUXlNYmIiLqzcK7fXXKVEBEREfUsDFiIiIjI8BiwEBERkeExYCEiIiLDY8BCREREhseApYO2HTod6yEQERF1ewxYOmjx//bGeghERETdHgMWIiIiMjwGLERERGR4DFiIiIjI8BiwEBERkeExYCEiIiLDY8BCREREhseAhYiIiAyPAQsREXUpsizjpY0HsenAyVgPhToRA5YQpSXExXoIREQE4NvKevz+3W/wm+Vfx3oo1IkYsIRIkqSA972742gnjoSIqGerrrcDAGobHTEeCXUmBiwhGlecEfC+O5bt6LRxEBH1dHXNrQCAZoczxiOhzsSAJURn9smM9RCIiAhAvd0dsNhbXTEeCXUmBiwhCjIjREREnai+2T0V1OqS0epk0NJTMGAJEeMVIiJjEBkWgFmWnoQBS4iKs5JiPQQiIgJQpwpYWMfSczBgCdElY4tiPQQiIgJQ36wKWJhh6TEYsITIZJJwxfjesR4GEVGPp5kSYoalx2DAEgYW3hIRxZ4mw+JghqWnYMBCRERdSp2m6JYZlp6CAQsREXUpzLD0TAxYiIioS6lnhqVHYsASBlkOfN+avVWdNxAi6vJkWcbWQ6ewv6oecrA/LuSj3s4MS0/EgCUMwf6k3LD0y04bBxF1fRtLT+L7T23EzL+sw3mPrsVn+6tjPaQuQz0lxAxLzxHxgKVfv36QJMnna8GCBX6PX7p0qc+xCQkJkR5WRPBDEBFFysGTjcr/Hz7ViNc2H47haLoOe6sTLap2/PY2Miyfl1Zj9zFbtIdFnSAu0k/45Zdfwun0Rry7du3Cd77zHVxxxRUBH5OWloaSkhLle4nrh4mom2vRZQaaWpgpCIU6uwIAzUEyLJW2Zlz73GZkJ1uw+d6Z0R4aRVnEA5bc3FzN9w8//DAGDhyI8847L+BjJElCQUFBpIcScXLQSSEiotA5nO6/J2aTBKdLDnrhJS91/QoQPMOyt6IOTpeMqjo7mlqcSLSYoz08iqKo1rC0tLTg5Zdfxo9//OOgWZP6+nr07dsXxcXFuPTSS/HNN98EfV673Q6bzab5IiLqSsS0RlqC+3Mji0dDU6fPsATpdFtaVa/8/8kGe9TGRJ0jqgHLO++8g5qaGlx//fUBjxk6dCief/55vPvuu3j55ZfhcrkwZcoUlJeXB3zM4sWLkZ6ernwVFxdHYfR+MMFCRBEidhlOTYgHwE38QqXPsATLTJWeUAUs9S1RGxN1jqgGLM899xzmzp2LoqLAGwdOnjwZ1113HcaNG4fzzjsPb7/9NnJzc/HMM88EfMyiRYtQW1urfB05ciQaw/fBeIWIIqXFE7CkJYoMCwOWUOhrWIJNCR040aD8PzMsXV/Ea1iEQ4cO4eOPP8bbb78d1uPi4+NxxhlnYP/+/QGPsVqtsFqtHR1i2NgrgYgiRQlYlAwLp4RC0d4MSzUzLF1e1DIsL7zwAvLy8nDRRReF9Tin04mvv/4ahYWFURoZEVHstXhWU4qAhf1EQlMXYtFtXbMDVXXerAqnhLq+qAQsLpcLL7zwAubPn4+4OG0S57rrrsOiRYuU7//whz/go48+woEDB7Bt2zb86Ec/wqFDh3DTTTdFY2gdwvwKEUWKyLCkJzLDEg7fZc3+z5t6OggATtZzSqiri8qU0Mcff4zDhw/jxz/+sc99hw8fhsnkjZNOnz6Nm2++GRUVFcjMzMT48ePx+eefY8SIEdEYWof86Oy+eHfHsVgPg4i6AbGsmTUs4am3OwAAljgTWlpdAc+bejoIAE42MMPS1UUlYJk1a1bAeo+1a9dqvv/rX/+Kv/71r9EYRsRN6JcV6yEQUTehr2FpdclodbpQXd+CZz89gOsm90Xf7ORYDtGQRIYlJ9mCY7XNymorPRGwJFvMaGhxoroLZ1iO1jQhLSFOWVHWU3EvISKiGPAua/Z+bmxudeHNLUfw3IYyPL56X6yGZmiihiU7xb3wIlCGRUwJjfd80OyqNSzV9Xac/+haXPOvL2I9lJhjwEJEFAOicVyK6lNzs8OJmib3lMfmslMxGZfRKRmWFAsAtJlhmdTfE7B00WXNB040oMXpwr7K+rYP7uYYsIRp8fdGx3oIRNQNiL2ErHEmWOPcf4qbHU40evYUKj/dhGM1TTEbn1GJZc05ngyL3U+GxemScbDavbnkBFWGpSu2phDFwk0OJxzOnl2YzYAlTEPyUwPe53R1vX8MRBQboobFEmdCQrx7j5tmhwuNLd5VMF8eZJZFlmU8trIEb3zpbhBar5sS8pdhKT/diBanC9Y4E8b0TgfgrhGyNbX6HNuZapscYQdN1apiYf0KqZ6GAUuYhhUEDlie/CRwszsiIjUxJeQOWHwzLACnhQBgX1U9nlyzH799dxdanS6fKSF/NSxiOqh/TjIS4s1KnVB1DKeFth46hTP+8BEe+6gkrMepl2Pbmh2RHlaXwoAlTMnWwAurnttwoBNHQkRdmciwWM3eDIu91anJsDBg8W5g2NLqQvnpJqXoNidI0a0ouB2Yl6I5NpaFt1sOnoZLBrYdqgnrceox6zd+7GkYsBARxYDow2KJMyEhTj0l5L0A76uqx+ke3j/kQLW3Adz+qnpVhiXwlJDIsAzMcS8Lz052Z2Ni2TxOdN0VRdWhUhcLM8NCRGQwtU0OvLzpULe+WGtrWNx/iptanGi0azMGPb2ORd2xtqSyDk2ejEq2akpIXxdSftpdrFyclaQ5Npa9WJSApTG832n1HkidUYPjMnAtJgMWohBV2Zrx5pYj7EjaCf698SB++84uPP9ZWayHEjUiMxBvNsEqim5bnWh0uC9KwwvTAABbD5+OzQAN4kC1dznvziM1yv+LDItLdhfUqlXZ3MFBfloCAG+Bbiw3QKy0NQMATocZsKizQnVRzrAcrG7AGQ+uwmMrw6uz6SwMWIhCdMUzG3H3W1/hr6u+jfVQur1jte4/7t25nbpY1uyzSsiTYRle6C7wFxffnkiWZU2GZYcnYLHEmbQN93QfIk54LvJ5ae5AJUdMCcWw6PaEJ8PS7Ai8nYA/6n8D0a5hWflNBWqbHFhTUhXV12kvBixEITp00t3XYeU3FTEeSfdX65nnD7QTb3egrBIym5Dgpw9L70z3dEZ3DtracrrRofwuAN5plVRrnNK7BtBuHNnS6sIpzznLS9VmWGJZdFvlybAAQE1jaJkSh9OlOTbaNSzbPNm8UMfX2RiwRJCtuRX/3ngw1sOgKJMkKdZD6PZsnotUc2v3nX5TVgmpMixNLU6lRqM4MxEAcKoTsgLHappw68tbscVg9TIHPMWzOSlWmFT/7FIS4iBJkhK02FW/J6JOJd4sITPJ3UVY1LDEKmCpt7eiQVVMHWhaqKK2GbuP2bzH6YLVaGZYZFnGtsM1AKAJEo2EAUuE/e7db2I9BIoyxivR190zLK1OF0TZhbroVn0hExmWU51wkX1i9T78b1cFnv3UWK0ZxAqhYQWp6OMpoAWAFE97CW+HYO/vicjC5KZYlQ8X2cmeGpYYTQmpsytA4IDlR899gUue3KDUu+hrbqJZw1J+ukmZtqq3txqyqy4DFqIwmRixhKzB3r5PhErA0k0zLGJJM+AOWBI9GRZxIZMkoFeGO8NysiG6LeUbW1rx/lfHAQDHaprbOLpzifqV/jnJGJibotwuAhZ1/xpBBAe5noJbwNtkLlYZFhFECbV+plyaHU7sr6pHq0vG3oo6AL41N9FcJbRNV9xtM2CWhQELUZgYroRm04GTGHnfSvxfmJ09ge6fYWlR9Q6JVzWOE7UXifFm5KR6N/dTTycAkd0GZMWuCqXd/fFaY+1dVOZZITQgNxmD8rwBiyi4VRcrCyI4yEu1KreJGpbaJofm3HcWfcBy2k/AcrzWGywePuWul9MHWHX28IOIkoo6LFmzH1c+vRGj7lupbHGgt+2QNmAJt19MZ2DA0g5pCYG73QrHa5vw2MoSVNQa6xMLdRwTLKF54L+7AQB/D3PLCpdLVj7dxTLDsq+yDnMf/zQqRdZ2p/t9SRIQZ5KUZc0iYEmyxCHJEqdMFamnhTbsq8ao+1biqbWlERnLm1vKlf+vrm8xVFZLk2HJ882wKDUsqlU3/gKWjMR4pQYm3GXFkRDKlNDR095g8YgnYBH1OOL9hlvDsnx7OWb/bT0eXVmCzQdPod7eihcD1Fnql88bsfCWAUs7FKvmUgO5/vkv8eSa/bhh6ZedMCLqTJwSCk2yxdyux9W3tCr1Hf66mHaW/351HHuO2/D7d3dF/CKuNI0zmyBJkjcwUQIW97kTtRfqqYHPSqvR5HDikZV7sf7bEx0ax5FTjdh44CQkyV2kCsAwH7KcLllZmTcwN0U7JZSgnxLy/p6cqHOPX6wQAgCTSUKWqGOJQfM4fYbFX/O4ozWNyv8f9rxvsUKsX477mhNOwNJgb8WfPtwLAJgyMBu/vWg4JAn45pgNVXXan3FjSyv2HHdPQ+Uo2SjvGGsbHRj/4CrM+uu6mNa2MGCJkpJK9w9/z3FbG0eSP0brtqiuIeAqodAkBdl3Kxj1/H4sm/RVei7clTY73t52NKLPre5yC0Bpza8PWLI8/UNOqVaLiE++sgz84vUdSoFmeyzf7n5fUwZmo9hT5GuUOpajp5vQ4nTBEmdCUUYiBmlqWNyrf6yq5eCC6FsjerAIsaxjERmWDM+qJX/ZC02G5bSYEnK/l/457vceTl3Jvz4tw4k6O/pmJ2HpDRNx07QBGN3LvXP1+m+rNcfuPFILp0tGYXqC0v9HPcbqBjtONrTgeE0z4s2xCxsYsLQDr1fRtWp3JcY88JGh+p2oP8Hxxx+aFGv7MizqJZWxzLBUqj6FPrOuNKJ1I6IHi7jgikyBqBvQByzqXizik69Jct/++Op97R6HaMQ2Z2QBCjPcGQmj1LGIDrf9spNgNklIT4pXPv3ra1jUvyf+poQA1dLmGKwUEmMaku8OBvzVsJTXeM/74ZONkGVZCa76Z4eXYTlRZ8cz691Thr+aPUwJjM8bkgsAWKfLzImC2zP7ZCI90TeoEuMQ5zBWGLC0Q2ZSbH9o3d3NL21Bvb0VP/n31lgPRdHUYpx5/a4iyeLNsIRz/mwGCVjUUyMHTzZi9Z7KiD23ekoIgDIlJBJ5Ylf4bD8ZltMN7vMza0QBAGDX0dp2j2NflTsTPDg/FYXp7lVJx2oMErB46lcG5HgzK4Py3JsZ+i5rVtew+E4JAarptRhkWEQWbGi+yF74jkF93uvsrahtcqBamRJyv+8WZ2hdch9f/S0aW5wYW5yBC0cXKLeLgOXTfSc0Afh2EbD0zfRmgZrUAYs74BLFy7HCgKUd/nTZ6FgPgTpZk+qPRKure65c8WflNxW45MkN2F9V3/bBOnGqTl/hfKrVZFhiOSXkuciMLc4A4J77jxRRB2DRZVgEsczZ75SQ5/xM7J8FANhXWd+uKdSmFqeySeDgvBQUpbsv8McMUsNy8KQ7YBEXawC4fko/jO+bifOH5gHwzbA4XbLSu0Q/JZTrybgcj8H7UzIsBZ6Axc/UzlFdoHj4VKMSKPTNTlIy+211uy09UY/XNrtXAv1m7jDNFPa44gykJsShptGBr8prAGgbxp3ZJwMZie7fOfUHBxE4iQA6VhiwtEMoRbfUvTSqMgTN3XSprT8/+fdWfFVei98s/zrsx6qzI6fCaC+vDliaY5RhaXY4lbT92QPcgcGhkw3BHhIW9caHgDfDIihTQn7qLmo9n87HFmfAYjahyeH0udiFovREPWQZyEyKR3aKFYWevi/HPc/1zbFabDl4Kqo9YIIRGYfeno6/ADBnVCH+c+sU9PFMkVjjtRmWUw0tcLpkSJLvxbW/J/ApPRF+8N0RzQ6nMpUTKMPidMk47qkdEv133AGL+7jclISQVwq9u/0onC4Z5w/NxaQB2Zr74swmTBucAwBYW+KeFjp4shGnGlpgiTNhZFG6akrIO0ZmWIi6EPWURlMP3K25vh0twdUra9obsDhdMlpjsCpBdPy0xJkwrncGAPcf9kgJVHQriILlHM80hro9vwikclIsGJDrvgh/6ynyD4e4cA/Oc19EC9NFDUszapscuPLpjbj86Y24/oUvO/0iD3iLf8UF3B9rnLYPi5gOyk62Ik5XHCr6uHT2exFFwAnxJhRnud9LTaNDEwhW1TWj1SUjziRhfN9MAMC3FXXK35rsFAvSEtyBRFsBi8iWzBie7/d+dR2L0yXj754aqNG90mGJMyHd75SQ+99vDmtYup8/fbgn1kOgCFMHKc0OZ0yaT8WSPgMQCnUmqr0BCxCbOpYKz3RQQVoC+ma7g4JIZlj0AYtVNyWUFO+/6LbZ4d1rKCPRohRxlugClmaHs80i4X2V7gu36G9SlOGtYfmy7JTSrG7dtydw4eOfKkttHU4X/r3pEMqqI3c+/DnmKf4VxcD+iN9LERwHKrgFvAFL+emmTl19VqmqqRH1j60uGXWqLtBihVBBeoIyBbbdUxCdEG9CksWsFBoHWynkdMlKIfUZfTL8HnOuJ2DZWV6Dny/bjre3H4XZJOFn0wcCcPesAXRFt56AmVNC3dA/1xtrPw7quMYW7x+XuuZWjH9wlWH6VXSGxHb0VIlEhgWIzdJm8bN1Byzu6Qf9zsEdod6pGfAzJeTJsOinhMTFyiS5V8oMyXdfhEXw0ep0Ycma/Rhz/0f40b++CNozQ9QlDfZcyEWGxdbcik9KqgAAM4bloSg9AfZWl1Kgu3pPFX73zi489P7ukN5rg7017CmrphancsEUxcD+6DMsJwIsaQbcF9uMpHjIcudmWZRl1qlWJMSblZ+1evm+OD9FGYnKnkk7PJmS7GT3nkihZFj2V9Wj3t6KJItZmX7SK0xPxLCCVMgy8MFXx2E2SXj8B+OUjEyGJ6hS/65XK6uEOCVEpKEu1jQK/SqXOnsrln15OEaj6RzqT+iJ8eEHLF05wyIKbvPSrEi2xikFm4cjNC3kMyWkz7AojeO0RbdiOig9MR4mk4TBnovSt5V1qG104PtPb8SjK0vQ4nRh44GTeCzItggiABGZh9SEeKR6AqUPPHsLXTKuSMnAiABCLHsWvULaMv/5zZj650/wxOp9IRcHi+xKijUuaGdx3wyLyGb4XlglSVJ6ubSniLy9lDF5giiRZVF3uxXFz71VAYvIwIhpGJFhCbYBoliePKZ3us+UmJqYFjKbJPztqnH47pgi5T5vrxh/NSzMsBBpmI0YsPj5lJ/Uzk6uXYU6cLDGxS7DEpMpIVWGBQD6ei4iByM0LSQCFn0fFiFZ14elyeH0ZB3c51F8ChZTQvur6vHM+lLsPFKDtIQ4XD+lHwDgmXUHsNaTLdG/vugiOzjfu2xYTL+In8Gk/tnKa4maBhE0nahre+WX0yVjZ3kNZBn4y6pvcf3SL9tc5QJ4C24L0xOCNmr0rWER2Qz/00hKHUsnBiyVNu2YRFHraT8Zll6ZiZpdqQFvViPN87hg52+7qp9KMNdN6YeZw/Pwj2vOxMVjizT3iSmh2iaHEmCKKckcZliItIyYYWn000ckuZ2dXLsKdZDR0o7C1/ZmWPRz9LGYEqr0XPgKPNMk6jqWz0urfXa2DVeLfllznPZPcaKnh02KNU6ZNjrZYFeCBnHR65OVBGucCfZWF/61oQwA8MjlY3D/JSNx3eS+AIDfv/uNz+sfOtmAVpeMFGucEpQB3joWwN2wrSA9AZm6T9xildLpxrY3EjxW0wSHU4bZ5N5+YP23J7D0s4NBHwNAWTFTGKTgFvBmWJpFhiXIlBDgDVj2d+aUUIAMizqDIQK0XhmJyEu1Kr8XgDfL5s2wBJ4SEgW3Z7QRsPTKSMS/5k/A7JEFPveJwMglu7fJcDhdSnaNNSw9wN9X74tYKrknMGSGxU/AYrTtAyJNnbJuT+O89mZY9D0qOiPDUl1vx6YDJ5WVG6Itf77nYt7PU8fy8Z4qXPOvLzD/+c0dWr3U4rOs2f+UkCRJml4s4iInggizSVL22GlpdaFvdhK+42kod+fMIQDcy2P1eyHtq/IW3KozGOp6kUn93Uti9UWY6sxAW/vyiF2H+2Yl4fYLBgMILUslpoR6BSm4BVR9WHSrhPxNCQHeAuPSKu8YSirq8N+dx7BkzX48u/5Ahzsa25odWPj6Dlz/wmbUNTuUTJTIsGQm+xa1iqLbXpmJMJkkzVLubF1330ABS22TQ5nqClRwGwp9nc3pBm9n5YwYN03t3h8RDeL/Vn2Lf20ow877ZsV6KF1CsLnXWPE3JdSVljdvOXgKa0tO4M6Zg0M+v+reH+15rx2tYZEkd+fXaDaPs7c68fjH+/DCZwfR5HDiuflnYcbwfGWVkAhYRN8PsQKjrrkVVXV2TUYiHL5Ft/4DFsBdN1Bha8bJhhblIqe+cAzJT8Fuz55lN07trwT8mUnxsMSZ0NLqQpXNrukfJS5s6v15ACjN4wDg7IHu/jPpPlNC3p/liTbOgQhY+mQnKUFEdQidZr1TQsHPr7Jbs26VUG6gKSHP+y2rbkCr04U1JSdw80tbNMf0zU7CLD+Zh5c2HsS6khP42w/GIdVTAKt34EQ9bnppi9Kl9/73ditZn3xPhiU9UVvDIsuyd0rIcy77ZCUpzyFqWETRbaBVQuJ3s09WUoenbjISLahwNKOm0aH8PmUlW2L+YTLiV4b7778fkiRpvoYNGxb0MW+++SaGDRuGhIQEjB49Gh9++GGkhxVzkVpd0BPE+h+FP/6mhPzdZkROl4zLn96IJ9fsxwdfHw/5ceoLU3veqzrQOOWnFbk/Lpes/EEW6edIN4+TZVnJcLy1tRz/WFuqBGTfVtZDlmXNsmYA6Jed7PM8HWlhry+6NZskJXgBtNsaKBmW+hafKSEASuFtRlI8Lh/fW7ldkiTlIqnenbel1YUvD57yPFYbsKinYHwzLC2e/3r/lul3IVYfB0Cpk+mblYQcEbCEUPsiutEWpoeeYZFlOeiyZsAdECTEm9DidOHI6Sa8tPEgAPdUkVgNJnYtVttfVYcH/rsbq/dWYcUu/3ucbTt8Gpcu+QwHTjQgL9UKkwT8Z1u5UtysZFh0GyDWNDqUf19FqoBFyFaKbkUNi/8Mi7d+JcPv/eHwtudvUS1pjm39ChClKaGRI0fi+PHjyteGDRsCHvv555/j6quvxo033ojt27dj3rx5mDdvHnbt2hWNoVEXYMwaFt8/El1lf6HPS707swb6Y+ePOivSnjoSdaBR0+gIaQqlzt4KkZEXn5IjnWG5Y9kOTPjjx6iut+OgrpdIRW0Tapu8tRmi7sBfwNKe7rKCPmABvF1bAV2Gxe+UkDfDcsnYIowoTMPvLhqhCXQAIN9zDkXh544jNZj7+Hp8us/9OzGhn7bWQTSiG5CTrFw89VMY+gyL2v99VIJxf1iFf286BAA4fMp9fvtkJytN8ELZpuGYLuMQiLKXUKsTtqZW5bzmBghYTCZJ2Zto/bcnsGG/+zw8P38CfjixDwD/9S0Pvr9HmSractB//dKST/a7Wx70zcQHP5+GWz19TcTvswii9DUs4vcoJ8WiBGDqgCUrWT8l5P/Dr9Jev2/w+pVQqDdANMrGh0CUApa4uDgUFBQoXzk5OQGPffzxxzFnzhzcfffdGD58OB588EGceeaZePLJJ6MxNOoCTAbcDttfGrarTAn9Z2u58v/hXPxPN7S/hkWdxVCez88OtXpVnsxGWkKc8sk+0jUsa0qqUNvkwI7DNcq+OaJt+/HaZiW7kpkUr1xA0pPilU/GYpltpAMW9bSQOmARF6zqertqSsibYSnOSsKHd0zD91XZFUFMaYll2ncu247SEw3ITrbgL1eOxfi+WZrjz+yTib9eNRZLrjlTuU1MYdQ0iaJb789RHbC8tvkw/v7JfgDuHdcBfYbF21MmWP2XLMtKl9u2i27FKiGnkkVKS4jzmWJTE4W3f/9kH2QZmDwgG32yk7wFuboVRGtKqjS7G4vslN4BT/B716yhyE214o4ZQzCqVxoA99Sf+JmJTrLi34NY0qwOztTTdyJgFcWw/mpYXC5ZybCcUdzxgEW9AWK1QdryA1EKWPbt24eioiIMGDAA11xzDQ4fDtyvYuPGjZg5c6bmttmzZ2Pjxo0BH2O322Gz2TRf1H3EmQ0YsPj5I9EVpoRcLhkrv/HuMhxWhkUzJRRea351kCEyZv52qBVjFLxTAYk++8REgq3ZofzBP1rTpCxfPsOzwWGFrRnlp7xNvNQeuHQUfnLuAPxwknv1TYemhDzZJqtZHbB4/1+9Ak20cy+rbvAbsAQjMkSVNjsaW1qV7QU+vGMavnemb4ADAJed0RvDC9OU75WLV4MDDqdL06FVBAnrvz2B377jzYp/VV4DWZaVxQZ9spOUqa1Wl6wszT1e26TJAALu6XPxYaCtKSFvDYt3mXbvzOB7vYnARNTSXDWhWHN76Yl6JZvicLqUBnlXnVUMSXIHJvrMUqvThSOeep1+Oe7Xt8SZ8LerxiEzKR6TBmQpxc2BMiy9VIW2xar3kKMruvW3rPlAdT3qmluREG/CsEL/DePCITZArG1sUZY0x3qFEBCFgGXSpElYunQpVqxYgaeeegplZWWYNm0a6ur873VRUVGB/Hztngf5+fmoqPA/TwgAixcvRnp6uvJVXFwc0fcQLbuP2dBgD+8Pf09kxCkhf2nYrpBhqbO3asYZrK233mnNlFB4WQ676njxKd/fDrVvbS3HqPtXYoNnikLpf5KeoLkYRYpYjQG4LxRio78zPGn047XNyioWUdMgXDK2CIsuHK7cLrIA7aFfJQRo9xNSdxYe6tnhd29FnTIdE+pqjQJVhuWIJxBLT4xXfiahEBfYOnurphAb8GZY/rxiL5wuGZeOK4LFbEJNowM7y2uV4Ma9/NqsZKfEp/a73tyJHz77hSaDIc5rdrIlaKYE0GZYxFSOCDwCUd+fmhCHOaPcBba9M91LxFtaXSj3NMVbvu2okpG697vDle6xW3RZlmM17r2ArHEmZRrO/Vqp+PyeGXjxhonKbZm6vXqO+smw9M1OQmK8GanWOCXQSwuySmi7ZzpoTK8Mze9Ue2UkqaeE3D+rWO8jBEQhYJk7dy6uuOIKjBkzBrNnz8aHH36ImpoavPHGGxF7jUWLFqG2tlb5OnLkSMSeO5oufOJTXPJk4HoecoszGW+VkK3J/Ufipqn9ldu6Qg2LPtAKpWmXcEqV+m9xusJaxiv6YpgkKMWWp/2sFFr/7Qk0tjjx6X73BUtdbKkUVEYpYDl8slHptyIyLNX1dpSeEAGLb90KoN1zp730fVgA3ZSQ6v+HFbizHYdPNSrnJyMxtAyLekpIWbET5m7z6k6z+iXJJ+rtaHW6lM0X75o1FMOL3OP9785jnjFYlfcmsgUiu7H7mDs7/saX3r/hygqhNpY0A+pOty6lGdzA3NADlkvHFSljM5skDMjVbnWw9lt3073rJvdDWkI8JvRzT6Ft1gUs6iDXpPvAlWgxa24TwYD49+CvXifZGofXf3I2XrvlbOV3xNua3+Gzg7aYjhLBbUeJaavaJodq48NuOiWklpGRgSFDhmD//v1+7y8oKEBlZaXmtsrKShQU+C4rE6xWK9LS0jRfXUXpiQY02Fs1VfukpV4l9Pa2ckNkpcSF/tJxvfDE1WcAiF7A8v5XxzDnb+tx95s72+xz0Rb9pzEReIVCH2CEk1ESGZaEeLPPJ0rNa3gyBuKTeoXNuwmcUlAZ4uu+8sUh3PLSlqDHq+tOdhypgdOzQ+6wglRYzCbIsrdGoW+AC7u4sKiDn3D5r2ExKbepl55nJVuUIlKx2jD8KaFmZfPGcAOWOLNJmY7QFylX2ew4eLIRDqeMxHgzemUkYlzvdADu32MA6JvlDfy8AYt7ikrUcazaU6nUxojW/20taQbUnW5Dz7D0y05WfreuOquP5j51YzlZlrG5zP27MHmge8XUWZ4iZX3h7aGTwYNcNZEdszW3otnhxJZD7tforwu0xvTOwKhe6cr3YpWQS4ayMaUgfi5i48SOUopumxyoFlNCPSFgqa+vR2lpKQoLC/3eP3nyZKxevVpz26pVqzB58uRoDy1mxjzwESb+cTVW76nEn1fsjUknTyNTBywL39iJe5d/HcPRuImplLTEOOXTb2OUfm6vf3kEeyvq8ObWcry5pbztBwThE7CEkWHpUMDiybBY40w+y2LVanRt3tUZFnExCjXD8vS6Uny0uzLgKg5AG7Coe63EmU3IT3f/QRZFl32y/V/Yizyf/OvsrWGdT7VgRbf+tnwYpvvkHOqUkMiwVNnsSo1FcZgBi/v13D/DMs+FWQRQJ+rt2OfJrgzOT4HJJGFM7wwA3pVJ6tcThbfVdXZNhqql1aUsuT/qmRJqa4UQAFWdk0v5uQ3MC37RtsSZsOSHZ+L/rhiL0b3TNfcNVhXeHqhuQHV9CyxxJowtdh83sb87w/LNsVrUqz5IidqgfgF+Z9TU2bE3t5ajur4FualWTPEERYEkxJuU6XJ95lTsnN0/J/yfrf8xihoWh2H2EQKiELDcddddWLduHQ4ePIjPP/8cl112GcxmM66++moAwHXXXYdFixYpx99xxx1YsWIF/u///g979+7F/fffjy1btuC2226L9NAMQxR03fjiFjy1thT/WOM/+9RT6fuwvLPjWIxG4tbqdCmfaNIS4pULSlOYhaihUjfWqrd3rH+Pz5RQiDUszQ6nprgSCC+j1KzKsCh70fhZJeSTYVFqWBJ9NrZrS3Wd+7mC9XzxlxUR7fcLdHUd/pYyA+4eKSJr1N5pIX3jOMCbLUi2+PbzVO+8a5KgbFLYFhGw1NlbsafCHVjoa3NCIepYxCd5cWFvaXUp2xSI7MRYz/SaoH69bGVpc4uyOkZ4e5s7OPdmWNqeElLvcVXX3AqTFPjnpjZzRL7fVVXiPeyrqleyK2cUZyivU5ieiN6ZiXDJwLZD3sA4nAxLnNmk/PyeWVcKwF3Q21btiSRJfrvdyrLsXY0VwuuHQtOHRUwJdcc+LOXl5bj66qsxdOhQXHnllcjOzsamTZuQm+veHfLw4cM4ftzbvGrKlCl49dVX8c9//hNjx47FW2+9hXfeeQejRo2K9NAiat3d0yP2XPs6cSOursBoNbfqPw6pCXFIEAFLlDIsp1R9Ktraq6UtYuziD2SoAYsIHBLjzcrqgFDfryzL2gyLbhmnmsjiBM2whFDw29jiLS4OtBoJAMr9BBjiwligmoKwxJl8Ahi1jtaxBJsSSvSXYVGt2hE7NYcixRqnbKS4U9UJNVxiiuBgtfvCmJ+WoNS2fLb/JABgcJ47qBqQk6wJqNQBi3pKSGS7RvdKh0kCthw6jYPVDSHvIwRoV1YB7mxOW4W6wag3R/zigPt9TeqvXfo90VPHol7e7M2whBYwZHh625SfboIkAT+YGNrCEWUDRNW/46o6O5ocTpgk7eqijhA/7+M1zcq/KyNkWCLemn/ZsmVB71+7dq3PbVdccQWuuOKKSA8lqiIVyQLu9uPkZbQtesRFP8liRpzZpMqwRC5geWPLERw40YBfzxmqadjW0YJTkWHplZmIvRV1IS9rVgcOLU4X0BDaMu4qWzO++/cNypSBu4bFk15u0gYSLa3ezNWpxhbYmh1KjYZ2lVDbryuyKwBwukEbGDW2tOLGpVsweWC2kmGJM0lo9fyiieBD/Ym+2LOnSyC9MhLxzTFbu+tY9Ls1A94poeQ2poQyw9zPJT8tAQeqG5TfpfYELCJLJopLM5LikZtqha25VdkWYIina67JJGF073R8XnrS5/XERa+6vkU5d2f0yUBmsgXrvz2Bny/brmRx2tpHCHBnqMQWDoDvVgPh6pedDLNJQr29FR/vcRfcTuyvnao5q18W3t5+VMnAuFyyd8+kELNXmUkWZdXW9CG5bS7FFvxlWJTzlZmoCYA7QnzIEFnWhHiTIXanN95yjB5IhsGu0DHmMlgEJ+oURJV+YnzkA5ZfvfUVnl5Xik/2VsHh9L7/ULILwYgARWymVm9vxcbSk0qDtkDUxa/iD1Wz7v2qm3UJXx48jao6O77xrP7QZFhUgYQsy5pMiCwDu47WAnBnBdIS4n02tgumWpWVOq3LsGwsPYmNB07iidX7lCJmdTGjyKSoMyptfVIWQc7RMJY2tzpdWPT2V3h7W7kyJRTvpw+LvwzLoLwUJfOYHmLBraDeuTjOJIU01aInpsBE0JOZZPHpJisyLACUOhZA++HOX4alKCMR10xyF79+VV6r/M6Gsk+TJEmaoG9gGwW3bbHEmZSgo97eijiThDP7ZmiOEXUs24/UoKnFiQpbM1paXYg3h35u1VsrXOPp6xOKVKtoz+/9tySCyFCzO6HQ10hlJ1s1m2TGCgOWDhBtrT/8+bROe81DJxvw/lfHfJa1dScd3S010tQFt4D3gtLocEb851BSqe1X1NKBHYEB7ycxdQHj1c9uwi/f3Bn0ccdV/VCUAE03JfTjpV/inIc/UaaPAG9HVcGqrmHxnMf/fX0cE/74Md7/Sruv0Vfltcpruh+r7cPy8P/24jt/Wed3ebS6P4g+YBGffkVGJcliVjqQAt4iWvXFJlDBrdCrHVNCX5Sdwmubj+DRlSX+p4TiRNGtb+I7Id6srAAJdUmzoO650iszsV2bi+pfMzMpXtkbxz0+k2aH4XGeItUUq7feBwByVd1u1f1HZo8swMo7z8V9F4/AhaMLcPsFg0JaJeR+bW+ANzC34xdtdZZmVK90n5/HwNxkFKUnoKXVhU0HTioBQ3FmUsjnVmTJCtMTMH1obshjE3+D1JnSMs80Xf8IrRAC3Fk+dT8sI/RgAbhbc4e8fstktDhdHZozBcKbEjrv0bXux1wNXDy2qEOva1T6gEUd2LtcMhpaWgPulhoN+gyL+AMmy+6Lacd//t73K3Z2FdqaDimrbsCtL2/FT84boLTkVi9tFFNCGUkWJFnMyrTOloOn4XLJmmmP/2wtx76qevx6zlAlCClMT1AuLPopoa/La+FwythfVa8EGfrN8PytEvpkbxWq61vw5tZyn+cTrykeC3iXNT/tKVD8x9r9uPeiEZrHnqxXZ1i0U0KiIFHolZGIXhnegERcGAtUAUugJc3Kc3guzvuq3MtfQ/n0KVaxVNXZlfdmMYe2SghwTwsdONHQrikhoT3TQYB3x2YhQ5dhGZSXovldmjo4F+P7ZmJCvyzNuclWbTPg8ATj4lwOLUjF0IJU3HCOt9dRKNQZlraWNIdicH4KPvJsLTBpQJbP/ZIkYfqwPLz6xWF8srcKIzx9Z8IpZh5akArsBOZP6RdWAJmq6sUihFPwGypJkpCeGO/tcmuAJc0AMywdYjJJHb5YAQg6ISTLMr4ur/VZ+qzvtNid6KeE1HsL3f7adox54COl5Xe0rfv2BHZ7dm8VBW+Jqp95JKaF1FkUfYYiUNHtm1uOYM7f1uOFz8qwt6IOL208hOmPrcX0x9Zqghyl6DYhTgm4AHe25Mhp7Tn85Zs78fS6UqzaXanKsCR6a3ZUv4NNLd5VROqMhn6qSV3DouxO68m07NNlk3aW17hf03OBDdQ4bueRWp/zoe5Xoy+6Fct5hV6ZiZpsgAiQ1J/o+7bxafXMPpmwxJmw57hN2TenLaWePiFOl6ycX3WGRUydZQVogS6aloX7SVq9c3F7ljQDvhkWUcMiqKeDAHdm5T+3TsE9c4dpbhdNBBs9UykA0DuEqZ9gtBmWjgcs6qBHX3ArXDA0D4B7nyFRQxJOwHDj1P74z61T8JNzB4Q1Nn81LJFe0iyopx6N0JYfYMBiCMEyLK98cRgXP7kBP176pfYxUR5TtPx35zEseGVb0L1pfDIsqv//4OvjkGXglc2Hgr6O2MekI1M2n5dWY/7zm/HE6n0AvH8szCZJudBEoheL+oKsD1gCFd3e/dZXSqACeKdTAO30SJ0qOyTSycLPl+3A9EfXoKK2WRMQbztc482wpCUoU2Dq4CxQgFCpq2mxxpmUP3xNDieaHU7l+Fbdz1kscxWrQ9RFtw5VULenwnfvMPVS8FO6KaNDnoBFBF69MxOVT/VxJkmpq8hNtSLes49V/zYuPgXpCbh5mjsT8McP92iCxEC/c6WqXYDtfqaELh/fG3fNGoJbAlzEfnR2X7z508n4yXkDg47N31iF9mZYxI7NyvdJFk0gFGpmI9liVn6usuzOMHW0g6p4vpwUS8j9aYIRwZckwWdzSGHKoGxY4kwoP92E1Xvdxbmh9GAREuLNGN83M+y6EPGhQ0xTq5c0R7KGBdAGqUbJsHBKyAA+3lOJ2iaHphBLeGnjQQBQKu67uttf2w4AGJKfijtmDvZ7jP7vvciwqC8Ebe039MjKEjy1thS/vWg4bpoW3qcYYauuAZk6Q5EYb0ZLqysiGRZ1UakIFJItZjS0OENeJaQO8k41tCgFi+oMS4qud4dY5vra5sO4coJ3WeW+yjpdDYv7ceoMi3rqR91fpVI3pZUQ794/xmyS4HTJqG1y+O3HoiY+1Xq7mLqU1UPiPR2radIUZZ5UBSnq53e5ZCXD8vvvjsCznx7AvHG9MKIwDcMKUjGiME2ZyjCbJNx38UhU1dlD6hh66/RBeGNLOQ6dbMSM/1uH7BSr8lpv3zrF5zlKqxp8nsOqybBYcNsF/v9NAO4CXZFlCYd6Sqitqa5AxI7Ngj7DMiQ/Vf8QvyTJHSB6C24TQl6iHYjIsAyIQHYFAEYUpuHqiX3QKyPB799kwD0tfPaAbKz/9oQy1ddWVi4S9BkW9ZLmUFcahUod/BmlhoUZFoO45z9fxXoInepoTeApHafsv4ZF3Vbe3MZ+Q0+tddc7PPTBnnaOED5zy+oMhVh6GoltA9RTQiJNLj4Vt2dZszr7IbpxpibEazpzqjW2tKJaFYBsOnBSeY7C9AQkWjzZJFVwpt6t9rQmYPHNsIj5cPexLT4t+tW9NJIsZqX9uVXVOE4f5HxRpg3g1eOvt7cqU2lVdXbYW10wmyR8f3xvrP7ldJzVLwsJ8WasuPNc/OWqcZrn+dHZfbHwO0MQihRrHH570XAA7uzQziM1ONXQglMNLXhX1+ywrtmh/GzVLOboLxVVb8bX7imhJN8Mi3ZKKPRgQX3xU+9Q3F6iWDkS9SuAe6p/8fdGBw0eAeACXbFspDMc/qTpaljEdFDvzKSILWkWtBkWBiyk8r9dgXen7o7US3f1AhXdnqj3/sFv7IT9hcT0gKDOsKTrVr50hF2VuRBvXWQP2tM4Tjsl5M2wqFfzqB082agJckRvFIvZhKxki1Kzo542OuFnSqipxemzFYD49Kte2lyrCz7U9Q/nDMpRMivq1vz6Hi6bSrU1XCcbtJmdGs/xoiCxV0ZiRHax1bt0XC9sXHQBlt1yNp7+0Zm4/YJBAIBP953QHHfghG92BQDi46K/VDQvzQprnAkWs6ldXW4B7cXL4ulF1DszCQnxJuSkWMIKhNRTQKG032+LCGwjUb8SjvOH5Sn/bzZJEXkvbdGvEhK/35HaQ0hNW8PCKSEK05tbvDuadsVVzS5VIOIIslzX5fJfdKuehgjWfj1ctmYHfrFsBy4ZV4RLx/VSbtdf4NJUf7SVDf0iMA5/WZRCJcPif8pJ3SxLT13DIVY4pSbEBWz8drC6we8miwXpCZAkCYmeVVHq6a8TqmyBKLr1t6GnmPIQF7zjtU0+S7UH56fga08PFvUST/UqoVpdYCg2jBPUQRrgnhbKS00Iu6FXexSmJyoFuyOL0vH3T/Zj+5Ea2JodSpCrrl9Rs0QhiNJLiDfjufkT4JLldq+uU0+NpCfFQ5IkpFjj8O6CqUiIN/lspxGMNmDp+M/lqgnFaGpx4sLRgTfMjYa+2ckYkJuMAyca0Csjck3bghEfZL45VosTdXZlSXM49TOhylBNAzLDQmG7+y3ttJEsy9hz3Ba1XYMBd+p//vObsfKbjmeA1HvTBAtY9FNCImA5EaBuoqOeXX8Aq/dW4Y5lOzS364tC1RkWsfLFX0+QcPkPWIJnWILtJSOaqLlcsmZK6PEfnIF4s4THfzAO8yf3xTmD3B08D51qVJZTx5sl5eIzzrMnjMiwaKaE1BkWTzChr18BvBkWZS8aP6u71PUP04d6P7WqVwmJn7fo+HqgukGZjnO6ZCWAFe3iRdB2uAMb/rVHcVYSBuQkw+mSsVFVdyYCFn0X2864yAHA1ME5OHdI6P0+9NQ7Nqv7qgwtSA17OW12hKeEvjumCG/dOiXkvi2RdL7n9zWaAbHa6F7pGFucgWaHC89tKPNmWKIwHZWumgLvaGF0pDBg6aJkyFi1uxJzH/8UVz6zMWqvs/h/e7Du2xP4yb+3dvi51PtfBKqnAAB9LCM+u6kDFv1KkI4IFPzol5KLP9iAN10aiSkhf0FJYRs1LM1BpopO1rfA6elXI2K/1IQ4XDSmEN88MAeXjuuFBy4dhRdvmIh4s4SWVpeS4bhx6gDsun82/nfHNDxy+RgAQGK8N9Mh+Ase9fUrgDdLIs6XWAKqNrZ3Bn50dh/8bPpATVpdWSXk8AYsg/JSkJdqhSwDez2rhU41tECW3Vmn/p5pAZH5UjIsnRSwAMC0wTkAgPXfeqeFRMHtpAHaNu+dFbBEgpjW6+hKHPXFryiE9vtGdsM5/XDOoGz8OMzeMe0lSRJuP9897fjvjQex65j7322/CC9pBrQ/53B7/0RL1/nXYnBjdNuUt8cv39ippNUffH83/vjB7qBTP295Gm+Ji000VNdHdupF0Kfw1fR9WEQNi3YlSAtufmkLLvvHZx3ujKsu+lTXV+gDFv9TQhGoYfEz7VMYpIbF6ZI1t2fqCiLf2lqO0fevxJoS9wXTYjYp2Qr1BTLObFIyD1s8O8/mpFiQaDFjeGGaqpGZO1BTL0XXFt26fy4iYLH42R8nU7cXjVpmcjwemjcav5qj7dkhahOaW51KYJieGI+RnkZdov2/qF/JSrIgx9MvQhQCiyWf7V3O2x7TBrszGZ/uq1ZuExmWKQN1AUsnTAlFivgZ6n/fwqXOsPSOwJRQLPXOTMIrN52tqWeJthnD8zCsIBUNLU5lP6KoZFg8P+f0xHjDBNbGGEU38K/rzsKsEfkdeo7/bCvHlU9vRG2jA89tKMOzn5b53eFWiEYRoV4kSwLVq3z0NQlqPo3jPFMU6gzNwZONWLW7EtsP1yhp0fZSN54a+4ePlOkvn4BFlWFRpoQiUcPiZ6+coiA1LPpxzZ/SD8kWs2aX68YWJ+5/7xsA2syQnug3IjJW+v1hACi7UwdaJVTb5IDTJSu3iU3wAG+WJN+zn01JhbZZHBD405v4uciyd+VTRlI8Rha5Pxx8c9QTsNSLbpwWZCZ7fy6tTpcSKESyC2hbzh6YjTiThMOnGnHkVCNanS4lUJsyMEc5zr1xX+z3ZwmVqGPp6KftXE+GRZK0PWIoNJIk4TZPcTfgLviN9JJmwL3rNqD99xxrDFgiJC8tAb8IcTlkMPo5fv3FSS2cQjd/ZFnGHz/YjTdUxbx6kfx7qs6wBHtf+oyJKMINtNNwKLsIB6M/j4ve/hqA79456gxLemLHMyyyLOOuN3di4Rs7NLcnxpuV529pdfk0IlOP65Nfnoc7ZgzG5ntn4s2fTtEcJ85jsIBFv7rA31y1eLyYxpNlWVPDIsvuZZYiwzK8wLtPjwg6RIZDTHGp5/z1S2YFdY8S0UE3I9HizbAcr8UbXx7Bvz3N87KTrcqn/9MNLfjqaC3qmluRlhDnboXeSVKscRhe6M0CHTrVCIdTRkK8CcMKUpWaIP0qNKMTUwQdnRLqn5sMkwQMyUs1zCf3rmbuqEIM8OybFK2C377Zyfjw59PwzLVnRfy524u/LREUqYu7evfmQJvf2Zpa22ye1pZNB07h2U/L8CtdMa+aKYIRizqrog8G1PSrhMQSaFuArIz+9nC72+rHIoKFphbtuU/1k2HpyCqhkso6vLW13CcQy0q2KEt6XbJv8a8osk6IN2FAbgokSUKyNU755CqIaaNgXSrH983UfO8vYMnSFRjXNjmUn4kIKk43OpRmYOqdkEV2rE+WNjA6ozgDI4vScN6QXOW96rkzEO7/FwW97ikh9/PvOmrDr/7zFVZ4MmLZqk6npxsd+PRb95TM1ME5HQ7uwyUKib+trMOe4+5M0NACd5M6kVXoahfrMZ6f6+heHZv+LkxPxLsLpmLpjydEYlg9ktkk4Y4Z7j4xZ/TJiNrrjChKC7hVRCxwWXOUFKQl+G0UFQr19SnQapr3dh7DvHHazQ9LKupQVl2POaMKQ3odfW8LfyL5Z149jdDscAXcNE6/SkicA3WGRk0/vdTsZ4olGP0qK5FJadZNx6gvrKJVebApu7bsOurbYh5wX3jVFzP31vXe70V2KlG3j5V+6aEIxIJ1qZw+NBfxZkkJQPwdK/5giZU44ueYnhiPFGscjtY04XRji1Ivoq7navLUvRRnaVdw5KRY8Vdd0zY9SZKQkRiP040OZWonPSkexVmJSE2IU3q+jC3OgN3hxOXje+NYjfvfXE1jizJVKGpKOtPQAncavaSyTpnWG1HoDmLyUq0oq27ocgHLTdP6Y94ZvfxOG4ZrdARq/nq6S8f1Qr/s5Kj0YDGqrvUvpgv53x3TAu4J0pbnNhxQ/j9YsmDXMe0Fb/bf1uOnL2/Dl342RtQHPrWNDk2Tr0BZiUAJFlmW8drmw8oOu23539fH8ejKEs1tgVbAuHQ3t7pkuFyykknR/8HUBzL6FUitQZZQA75TSiLD0hxkqkm0Kg+UYdl9zKZZ8tzY0oqrntmIJWv2K7eJ1vh62cnagEV/npoCBCyBdvkNtiQxyRKn+cTsrz5B1IU0O9xbEYh+OLmpViVwO17TrNyu3pyv3u4ea2pCvOaTWoanl0dbNRyiGZj4GWUkuh8npoXG9k7HWz+djBV3novpQ/OQ5RnPkVNN2Ha4BgAwdVCO7xNHmciwlFTUYa9n80wxTdRVMyySJEUkWKHIGVucEXD7gO6oa/2L6ULMZqndU0RL1pSGdJzYwwIAPvjquPL/IgUt7DhSg5H3rUS/ez7Adc9vhr3ViRl/Wafp6yI+YX9VXoM/frBbuegHuqBs2F+NRW9/jYuf3BBwfM9tKMOdy7aj1enCra9s87nfX7Ep4JthAdxTY2Lq5KLR2gzSvsp6/O/r4/i81D0FoN9YsdHhxH93HsOVT2/E0ZombDt8GpP+9DHe2X4UgO+UkJg90GdY1ESthK251Scg+rq8Fhc+8SmuUC03X7GrAl+UncKjK0uUKa8dAQKWrGQrzCZJqXHQrxRSpoR0AUqgn1VbPRRmjfQ23PK3r0uyxaysZjnV2KJ0zC1IS1ACHLFSLTUhDumJ8cpuyOpGcOpeKOkh1kHo262LlQs/v2AwLhxdgCd/eKYm+ySmhEoq6+B0yeifk9xpPVjURM1MWXUDvvKcm2Ge2h6xG3VXWiFEZAScEoog9XXWrLt4PHbFWNz15s6ovfaCV70BwUPv78E7249i2uBc3DlzMOYt+Uy5b/23J7D10GmfzqbNrU5Y4ky45En3sQ6njPsvGRlwSuioZ2ddADjn4U/wyk2TfFKTD76/GwAwc0S+svmd/jXT4fvpQBw3rjhDuag7nC4lwzJ/Sj+MKEzDun0n8MFXx/GvDWX414YySBIwoV+WJpAD3NkOsenif3ceg93hQqXN7p5WO6OXz5RQoycrEKwhn/pTTW2TQ1Mn8p9t7uXm6nG0qrYiOFDdgN6ZiT6BpSCmdixmExxOp89KoUAZFgB4aN4ovPrFYexWPXdOG5+KbzinHw6dbMAZxZl+75ckCZnJ8ai02XG6oUWz35HI/nxVXgPAXUwrSRI++sW5ONXQolm90CcrSckqZYT4qVAfsIjum1MG5WCKn8zJ8MI09MpIVOppzh3c+dkVwB2UiGkrMYU2TEwJiYAlQO0OEfnHED9KTJIESXW5D7cQtCNanC5sO1yDx1fvw9ZDp33u97dCp/xUk+b77YfdjwuUJUpRFaAerWnC9MfWKhfgUw0tmn4mR041aZYEBxuH+jz9a763Or3B7t25OCvZgisnFGOgLkCSZWBz2SmfpnKiX404xqbbOEyfkRHZJXUtjL6PR5zZpHSb1dfQqINBkR1RbyWw80iNe7lrgP4xYurEqur0qhaohgVwb973+k/O1tyW20ZbbWucGYu/N0azY7OeyKScatBmWETgIQKRvp7i2iRLnM9Syz6qOpZAK4P0BuoDljYel54YjzV3Tcc/rx2P2y8YhDtmdnzlXntIkoShqg6+vTMTlU7JYsl6Yjz//BKFg/9iokR/oY/V1j/fVvruYaJf/QIAFz7xqWbnYX+reKpURcT+sg9zH/8Ux2ubcOaDqzD2Dx8pt/95xV5NrxOh2eHCkVONeHTlXpz0XOTVWZg4k6SshBJBgCR529Krlxlfe3Zfn+cX1AHLlwdPKU3SDp9qREury+e9NLSIgMV9+63TB2L5z7RLhgEg2TOOBrv28cdVmwyKpmbqguMdR2qUDIA/2cneDAvgZ0pIBCwBalZSrHGa6YZItNXOUvU3Ee+vID1BqW8RGyYGm37pq1oppN6nJJhBqg3tLHEmv79HepY4E2aNLMAvZw2N6QqHIaql1KJ+BXBvP3DlWb01vTSIqG0MWKLEJLW/hiWS/DVVC7TapkzVNl1kF9TLmif+abVSNBqo98kD7+32e/txPzsFNzmc+POKvViyplSZilLXr5hMklKYKLrcpljjlDoLq+ri9Z0Qm/Z9srdKyQY4XTKOnG5UAoAfTuoDwBuAiNsvHFXod2lwiq4/CeBekq1ujiYCFXXAsrO8RlnN4o+YEhKdXr/79w2Y+/in+Hy/u0ZHBJyBLt5iCkeIRKGkCExONbQo/VYK0hJw9oAszXHB9lRRBzOhZlh6ZSQqnYhDnUYyCnWGRR2wJFrMeOTysbhgWMcaTRL1NAxYIkg962OSdEuCY5RieWb9AZ/bHv94n99j1RfexpZW7Dpa67NK6dGVJTh0ssFvAAL437FXTxSsNjucyjTS0ZomVNmaNSuEzJKkFFR+7amRUK9iUWeE1BvohaPsRIMSfE3qn6V5XpFhSQiQuhcZFvV5Kz/dpPneX8By6GQjjtb4bgIIuDNIYmWMOkuy57gNt722Hc0OZ9AaFiFLtR18RDIsqv4m6gzL2f2zNfv/BNuzp0+2uug2tODDZJIwIMd9PkINcoxC/TspljQTUfsxYIkSfaMqOWaTQr4C9Yd540tvx9vq+hZ89+8blKZcauc9uhZPr/O/ksnfbrxqj1w+RtlVtdnh1NSGrCmp0mRYzCZJSek/9tG3ALS1JN87oxdyUqy4cWp/TUZB+PkFg4J2eQWAA9X1ypSQyEQ0OZxwumQlMAiUyUhVpoS8Acr+E9rW80rAUq9tZX/ghG/m66fnDcQHt09T2shbdYHSqYYWvPDZQdR66mGCByzxyjHJQXZ2DpUIMqtszco0V0F6AkwmCfPO8PYDCjYlVJSegItGF2LeuCLNztdtEYW3XW355tAAU0JE1D5cJRRB6r4Kkm5KqDP2/emotz3LfDtCX/CaGG/W1MOMKkpXMhbv7TymyUb84b+7MbY4Q/neJEm4/5KRmP/8ZuU29af0vLQEfHnvjIDLeQfmpSAzyaLpN6NXVu3NsKg7xdbbW5VpsUC1IslW9+11qvegD0REwFKlCxK3eYqa1Q0GC9KsGFHkvbCpMyxixcmfV+xVbgs0LsCbYclJjUwNh5gS2ltRB1l2j01kXS47o7eyFL8oIzHgc0iShCXXnBn2a3sDFuN03AxFVrIFd84cjGaHq1M3XyTqrox/Fe1CBuYm46qzivHT8wb63Hfh6EKc0ScDN0/rjx8EWY3R3ej/UKdY45ROvm9vO6pZHt3Q4sQG1Q63Jgk4b0guXr5xknKbfsohWOOxQXkpSGkju1B6okEJqDKSLEpmTN30LVCGJcXq/sSvzrCo64AAd2al2eFUesiITIVoNT9YvVGg7nXUnXV/fsFg5OlqUYIGLJ7XicR0EOAtuhVTeHlpVqWWaFBeCt786WS8f/vUqLTA/+6YQozpnY7Lx/eK+HNH250zh+CeucO61CaHREbFDEsESZKEP18+xvu9qoolId6M5T87R/l+mWr65bn5Z6Gqzo5eGYm4TpVN6ApyUiyorg/c4r84Kwklld5pkmSrGUdOeaeNjqmWyFbYmjXZEHHxU3diDWdaYEBO2wGLuitwksWMZIsZtuZWZdoDABICdCRN8WRY6lVjFrvyju2djp3ltaiy2fHxnkoA7qzE4PxUbC7zvuaQ/FR86gnSrLrXUWfsBuen4PN7LsDcxz/FPk9/l1BqWCIVsIjaIbHEWjQ/Eyb0y/J5TKQMyE3Be7dNjdrzE1HXwAxLFIW6dXqKNQ5XT+zj0ySrK2hrmWk/3aqRZGucsuJHrTDDfa7ECiZJ8mZP0pPilezC2QOyQxrX7747AokWs6ZfjD/qouKEeLMS4Jyoc48x3iwhLsB0nr9VQmWeKaGJngLeFd9U4LZX3U3rclOtKFT9TsSZJE0be/0mgOoAJifFijizSZORCRawTBmUjRRrHM4fmhfwmHDolweH+rtNRBQpzLBE0VUTirGvsg5T29h8TVzcQtl9uX9OMh6aNwrPbyjD6r1VERmnJAXfsyiYgrQElJ8O3FNEH4RZ40yazfbUzwN4a2D0nYI/vGMaquvtIW30NaZ3Om6c2h8Awio4NZsk5fiKWvd7CpahEccer23Cil3H0eKUlYzR7JEFePbTMgDuoCc72YofTCjW1LsMykvRFAXr95aJM3vPgQjY1EWt+tb8ahP6ZeGr+2b5bbXfHpn6gCWNAQsRdS4GLFEUbzbhgUtHBT3mzD4ZSiYhlPn/NXdNBwBMGZgNl+y+wE/448chjScnxYKfnjcQD32wR3P77ecPwhOf7A/wKP8G5iZj4XeGYkRRGs5/bC2GFaRir6f/SGF6grL0tb8uwJAkCS/eMBE//NcXmtvzPRfAd3ccAwCfLrA5KdY2pzeW/2wKnlpbit9cOFy5LVjAMX9yX7y48RAA7w7DSZ7jRS1KsB4mYpXQym8qsfKbSuX2zKR4nNUvC+/fPhWnGlowtneGsoz3uQ1lynFTBuYg2eIdn35KSN2QTmQ41M3XgmVYAP/7ArVXlm7vH3VxNBFRZ4j4lNDixYsxYcIEpKamIi8vD/PmzUNJSUnQxyxdulTZuVV8JST0jE9w+apPqvqiSwAYkOu9QC29YYLy/5IkwWwKb/fUswdk+yzzfeMnk3H7jMHhDBkAcPXEPrhoTCH65yRj2+++g//e7q0xyFO9J3WGRWQQpgzKwU9UO1lLkruIs6PO6JOJf153liYLkxwkC/HApaNQ+qcLsfk3M/Dqze529mKlkNjDKC818O+hPnvTKyMRg/JScO9FIwAAo3ql49whuZqeI+rMxDmDsjXPoQ9Y1A3+xLSUuoi5rYAlkhItZuR7fkY/v2AQvjumsI1HEBFFVsQzLOvWrcOCBQswYcIEtLa24je/+Q1mzZqF3bt3Izk5cDo/LS1NE9h096r6f1xzJl7aeBD3XzJSuc1fcecdMwbjjmU7ALhbevsztjhD6d6qdsu5A/BPXeM4k+68iumot346GR/vqcIFw/JwpWqX4UAuGOYdi76+QZZl/O+OaWhyOJGdYlU2o/v+md5VHuqgJsUSp2QrIi1Q1urXc4Yp96vHMqIwFR/vqcTOcvcOu/qVOWrq7M2IwjR8eMe0NsejDhgn9s/CwWpvAbJ+SsjW5NuRWB2w6AOcaHvt5rNRb2/FmN4Znfq6RERAFAKWFStWaL5funQp8vLysHXrVpx77rkBHydJEgoKCgLe391cOLoQF47Wfkr1V9wZ7BO+oA50Dj58Ed7beQytThe+d2ZvTcAie1737re+8nmOs/pl4SzPSo/Xbzkbz6w/gCvPKsZPX97q9zUH5AYuEHY4ZU2jrH/fOBEffHUcP/bUlbgf7w1eUxLiItLczC8/8cpvLxqOm6YN8L0D0PRBAYJPCakDlmAt6dXOHpCN84fmYmRROlIT4pFk9WZJ9EW3tU2+/WNEcTIA1Nn9b7EQLcF+5kRE0Rb1GpbaWvcn1ays4Mse6+vr0bdvX7hcLpx55pn405/+hJEjR/o91m63w273Lju12WyRG7CBnNknA2cPyMKv5wzDkPzAFwt9MuqSsUX+D4R7GiMr2eLT4E1t0oBsTPKsxvnrVWPxi9d3hjXuVqd2s74BuSk+004jVAFN/5zkqAUs+owSELxWaERhuub7oAGLKlsSamMwS5wJL9ww0fsc1sBFt/4yKOoGhKIjLhFRTxDVnLLL5cKdd96Jc845B6NGBS4+HTp0KJ5//nm8++67ePnll+FyuTBlyhSUl5f7PX7x4sVIT09XvoqLu18jtl/NGYo3fzoFkiTh1ukDMWN44I3S/F2U/Znl2SBw4XeGAACuGN+7zcdcdkZvzBmpzXw9ouo1o3b1xD6a5w9GPdUyOIQGb+3l78wEC1iKsxI10zbBAhZ1kBWsJX0wSaoaG/2P8Ymrz0D/nGQ8N/8sze2rfnEunv7RmTizT2a7XpOIqCuKaoZlwYIF2LVrFzZs2BD0uMmTJ2Py5MnK91OmTMHw4cPxzDPP4MEHH/Q5ftGiRVi4cKHyvc1m6zZBS/+cZJRVN2DeuF4hdw0NFq+svWs6Vu2uxLDCVEwdlAMAuGZSH0wZmB3yJ/Q/fW80+uYk4YrxxUhLjAs4TfXHeaNw2wWDNJvhBR6zhAXnD8T/vq7AgvMHKcuBI00/zQIED1gkScI5A3OUPZSCTcmp627a23pdvUpIX0Q7vm+msipMbXB+Kga3c7NHIqKuKmoBy2233Yb3338f69evR+/ebX+SV4uPj8cZZ5yB/fv9L7W1Wq2wWiPTwdNoVt55LuqaHcgOo0Op5DeP4NYvJxk3n6ut15AkKax6hKxkCxbNHd7mcSaTFFKwItw9exjunu0ufq1VFZgWpifgmkl9Qn6eYOZP6Yt3dhzFhaMLcLy2GZ/vP4mLg0yZAcBFYwqVgCXUDEt7VzmZTBIeu2IsapscQffhISLq6SIesMiyjNtvvx3Lly/H2rVr0b9//7YfpON0OvH111/jwgsvjPTwDM8SZworWAGCZ1i6iiTVxX/NXdPb7KAbqowkiyZL4XTJbWauxAqohHgTijICZ1iSLGacNyQXTS1ODM5rf8bj8hCm5oiIerqIBywLFizAq6++infffRepqamoqHB/Uk1PT0diovsT5HXXXYdevXph8eLFAIA//OEPOPvsszFo0CDU1NTg0UcfxaFDh3DTTTdFenhkUL0yEnH37KHITrZELFjxJ5RptmRrHD791flwumQkWQL/E5EkCS/+eGLA+4mIKHIiHrA89dRTAIDp06drbn/hhRdw/fXXAwAOHz4Mk8lb73v69GncfPPNqKioQGZmJsaPH4/PP/8cI0aMiPTwuqXJA7OVDfS6sgXnD4r1EBTtLaIlIqLokGS5vbvIGIfNZkN6ejpqa2uRlpbW9gO6mZZWF97YcgRTB+WEtNcOERGREYRz/eZeQt2AJc6EH53dN9bDICIiiprO7e1NRERE1A4MWIiIiMjwGLAQERGR4TFgISIiIsNjwEJERESGx4CFiIiIDI8BCxERERkeAxYiIiIyPAYsREREZHgMWIiIiMjwGLAQERGR4TFgISIiIsNjwEJERESG1y12a5ZlGYB7m2oiIiLqGsR1W1zHg+kWAUtdXR0AoLi4OMYjISIionDV1dUhPT096DGSHEpYY3AulwvHjh1DamoqJEmK6HPbbDYUFxfjyJEjSEtLi+hzkxfPc+fgee4cPM+dh+e6c0TrPMuyjLq6OhQVFcFkCl6l0i0yLCaTCb17947qa6SlpfEfQyfgee4cPM+dg+e58/Bcd45onOe2MisCi26JiIjI8BiwEBERkeExYGmD1WrFfffdB6vVGuuhdGs8z52D57lz8Dx3Hp7rzmGE89wtim6JiIioe2OGhYiIiAyPAQsREREZHgMWIiIiMjwGLERERGR4DFgALFmyBP369UNCQgImTZqEzZs3Bz3+zTffxLBhw5CQkIDRo0fjww8/7KSRdm3hnOdnn30W06ZNQ2ZmJjIzMzFz5sw2fy7kFu7vs7Bs2TJIkoR58+ZFd4DdRLjnuaamBgsWLEBhYSGsViuGDBnCvx0hCvdc/+1vf8PQoUORmJiI4uJi/OIXv0Bzc3MnjbbrWb9+PS6++GIUFRVBkiS88847bT5m7dq1OPPMM2G1WjFo0CAsXbo06uOE3MMtW7ZMtlgs8vPPPy9/88038s033yxnZGTIlZWVfo//7LPPZLPZLD/yyCPy7t275d/+9rdyfHy8/PXXX3fyyLuWcM/zD3/4Q3nJkiXy9u3b5T179sjXX3+9nJ6eLpeXl3fyyLuWcM+zUFZWJvfq1UueNm2afOmll3bOYLuwcM+z3W6XzzrrLPnCCy+UN2zYIJeVlclr166Vd+zY0ckj73rCPdevvPKKbLVa5VdeeUUuKyuTV65cKRcWFsq/+MUvOnnkXceHH34o33vvvfLbb78tA5CXL18e9PgDBw7ISUlJ8sKFC+Xdu3fLf//732Wz2SyvWLEiquPs8QHLxIkT5QULFijfO51OuaioSF68eLHf46+88kr5oosu0tw2adIk+Sc/+UlUx9nVhXue9VpbW+XU1FT5xRdfjNYQu4X2nOfW1lZ5ypQp8r/+9S95/vz5DFhCEO55fuqpp+QBAwbILS0tnTXEbiPcc71gwQL5ggsu0Ny2cOFC+ZxzzonqOLuLUAKWX/3qV/LIkSM1t1111VXy7NmzozgyWe7RU0ItLS3YunUrZs6cqdxmMpkwc+ZMbNy40e9jNm7cqDkeAGbPnh3weGrfedZrbGyEw+FAVlZWtIbZ5bX3PP/hD39AXl4ebrzxxs4YZpfXnvP83nvvYfLkyViwYAHy8/MxatQo/OlPf4LT6eysYXdJ7TnXU6ZMwdatW5VpowMHDuDDDz/EhRde2Clj7glidR3sFpsftld1dTWcTify8/M1t+fn52Pv3r1+H1NRUeH3+IqKiqiNs6trz3nW+/Wvf42ioiKffyTk1Z7zvGHDBjz33HPYsWNHJ4ywe2jPeT5w4AA++eQTXHPNNfjwww+xf/9+/OxnP4PD4cB9993XGcPuktpzrn/4wx+iuroaU6dOhSzLaG1txU9/+lP85je/6Ywh9wiBroM2mw1NTU1ITEyMyuv26AwLdQ0PP/wwli1bhuXLlyMhISHWw+k26urqcO211+LZZ59FTk5OrIfTrblcLuTl5eGf//wnxo8fj6uuugr33nsvnn766VgPrdtZu3Yt/vSnP+Ef//gHtm3bhrfffhsffPABHnzwwVgPjTqoR2dYcnJyYDabUVlZqbm9srISBQUFfh9TUFAQ1vHUvvMsPPbYY3j44Yfx8ccfY8yYMdEcZpcX7nkuLS3FwYMHcfHFFyu3uVwuAEBcXBxKSkowcODA6A66C2rP73NhYSHi4+NhNpuV24YPH46Kigq0tLTAYrFEdcxdVXvO9e9+9ztce+21uOmmmwAAo0ePRkNDA2655Rbce++9MJn4Ob2jAl0H09LSopZdAXp4hsVisWD8+PFYvXq1cpvL5cLq1asxefJkv4+ZPHmy5ngAWLVqVcDjqX3nGQAeeeQRPPjgg1ixYgXOOuuszhhqlxbueR42bBi+/vpr7NixQ/m65JJLcP7552PHjh0oLi7uzOF3Ge35fT7nnHOwf/9+JSAEgG+//RaFhYUMVoJoz7lubGz0CUpEoChz67yIiNl1MKolvV3AsmXLZKvVKi9dulTevXu3fMstt8gZGRlyRUWFLMuyfO2118r33HOPcvxnn30mx8XFyY899pi8Z88e+b777uOy5hCEe54ffvhh2WKxyG+99ZZ8/Phx5auuri5Wb6FLCPc863GVUGjCPc+HDx+WU1NT5dtuu00uKSmR33//fTkvL09+6KGHYvUWuoxwz/V9990np6amyq+99pp84MAB+aOPPpIHDhwoX3nllbF6C4ZXV1cnb9++Xd6+fbsMQP7LX/4ib9++XT506JAsy7J8zz33yNdee61yvFjWfPfdd8t79uyRlyxZwmXNneXvf/+73KdPH9liscgTJ06UN23apNx33nnnyfPnz9cc/8Ybb8hDhgyRLRaLPHLkSPmDDz7o5BF3TeGc5759+8oAfL7uu+++zh94FxPu77MaA5bQhXueP//8c3nSpEmy1WqVBwwYIP/xj3+UW1tbO3nUXVM459rhcMj333+/PHDgQDkhIUEuLi6Wf/azn8mnT5/u/IF3EWvWrPH791ac1/nz58vnnXeez2PGjRsnWywWecCAAfILL7wQ9XFKsswcGRERERlbj65hISIioq6BAQsREREZHgMWIiIiMjwGLERERGR4DFiIiIjI8BiwEBERkeExYCEiIiLDY8BCREREhseAhYiIiAyPAQsREREZHgMWIiIiMjwGLERERGR4/w9+MlF/HywEqwAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(learningRates, losses)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90f492f3-24b9-4c95-b990-346df42efa9a",
   "metadata": {},
   "source": [
    "### Deciding The Learning Rate"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f49bb7e0-5f72-4484-8f06-a7eb4aa3b4e6",
   "metadata": {},
   "source": [
    "We see that we get a nice descent when the learning rates start at 0.001 and go till approximately 0.1...\\\n",
    "But generally as we decide to go up... The losses start becoming unstable..."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9da68b26-51f0-4204-8c1e-c0f3f2b44317",
   "metadata": {},
   "source": [
    "So we can now go back to 0.01, which seems to be a fairly good learning rate..."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f098ec3-576d-473d-9813-7817a7358903",
   "metadata": {},
   "source": [
    "### Understanding Learning Rate Decay"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe885d4c-fa12-4919-be11-5242d1a77c36",
   "metadata": {},
   "source": [
    "What is **learning-rate decay**?\n",
    "\n",
    "Learning rate decay is a technique for training modern neural networks. It starts training the network with a large learning rate and then slowly reducing/decaying it until *local minima* is obtained.\n",
    "\n",
    "![LossesonLearningRates](https://miro.medium.com/v2/resize:fit:918/0*xH_N3_XR83b9LwzI)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8ea98e0-122f-446c-9157-7ed9794c4474",
   "metadata": {},
   "source": [
    "# Retraining Model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8238259d-5038-4900-9584-cc4541750f34",
   "metadata": {},
   "source": [
    "Now that we know what our learning rate should be...\n",
    "We can go and check the model's loss and if it beats our original bigram loss from <a href=\"https://github.com/AvishakeAdhikary/Neural-Networks-From-Scratch/blob/main/NameWeave.ipynb\">NameWeave</a>.\n",
    "\n",
    "Original <a href=\"https://github.com/AvishakeAdhikary/Neural-Networks-From-Scratch/blob/main/NameWeave.ipynb\">NameWeave</a> Loss:\n",
    "```python\n",
    "Loss: 2.2942557334899902\n",
    "```\n",
    "\n",
    "Keep in mind, that we will take *Learning Rate Decay* in mind and try to see the dent it makes..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "5342ae06-fbd2-4cf6-b643-a7c9fe247f88",
   "metadata": {},
   "outputs": [],
   "source": [
    "words = open(\"Datasets/Indian_Names.txt\").read().splitlines()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "c78416dd-e871-467f-8fae-d64086396c0a",
   "metadata": {},
   "outputs": [],
   "source": [
    "words = [word.lower() for word in words]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "cc75a130-2f53-40d1-ba49-3a064d9b5ede",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "53982\n"
     ]
    }
   ],
   "source": [
    "print(len(words))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "692fb529-6df6-4028-8519-fa3b324e5781",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Characters: ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z']\n",
      "STOI: {'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5, 'f': 6, 'g': 7, 'h': 8, 'i': 9, 'j': 10, 'k': 11, 'l': 12, 'm': 13, 'n': 14, 'o': 15, 'p': 16, 'q': 17, 'r': 18, 's': 19, 't': 20, 'u': 21, 'v': 22, 'w': 23, 'x': 24, 'y': 25, 'z': 26, '.': 0}\n",
      "ITOS {1: 'a', 2: 'b', 3: 'c', 4: 'd', 5: 'e', 6: 'f', 7: 'g', 8: 'h', 9: 'i', 10: 'j', 11: 'k', 12: 'l', 13: 'm', 14: 'n', 15: 'o', 16: 'p', 17: 'q', 18: 'r', 19: 's', 20: 't', 21: 'u', 22: 'v', 23: 'w', 24: 'x', 25: 'y', 26: 'z', 0: '.'}\n"
     ]
    }
   ],
   "source": [
    "# Remember we need our starting and ending tokens as well in these mappings,\n",
    "characters = sorted(list(set(''.join(words)))) # Gives us all the characters in the english alphabet, hopefully our dataset has all of them\n",
    "stoi = {s:i+1 for i,s in enumerate(characters)} # Enumerate returns the tuples of number and string, which can then be mapped to string:index\n",
    "# We manually add these tokens for convenience\n",
    "stoi['.'] = 0\n",
    "itos = {i:s for s,i in stoi.items()} # After we have the string:index mapping, we can easily iterate over their items to map index:string\n",
    "print(\"Characters:\",characters)\n",
    "print(\"STOI:\",stoi)\n",
    "print(\"ITOS\",itos)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "62ba8e87-cda1-4117-9d92-d0a4d8b8224d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Inputs Shape: torch.Size([488074, 3]) , Datatype: torch.int64\n",
      "Outputs Shape: torch.Size([488074]) , Datatype: torch.int64\n"
     ]
    }
   ],
   "source": [
    "# We can now check the shape of inputs and outputs and their corresponding datatypes\n",
    "print(\"Inputs Shape:\",inputs.shape,\", Datatype:\",inputs.dtype)\n",
    "print(\"Outputs Shape:\",outputs.shape,\", Datatype:\",outputs.dtype)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "62b07d88-a7fd-48ce-9ff4-0fa383217def",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We will define a generator to give the same result on your machine, as of my machine\n",
    "generator = torch.Generator().manual_seed(6942069420)\n",
    "# Embedding Matrix (Input Layer)\n",
    "embeddingFeatureSpaceLength = 2\n",
    "embeddingLookUpMatrix = torch.randn((len(stoi),embeddingFeatureSpaceLength), generator=generator)\n",
    "# Hidden Layer\n",
    "numberOfHiddenLayerNeurons = 100\n",
    "weightsOfHiddenLayer = torch.randn((inputBlockSize*embeddingFeatureSpaceLength), numberOfHiddenLayerNeurons, generator=generator)\n",
    "biasesOfHiddenLayer = torch.randn(numberOfHiddenLayerNeurons, generator=generator)\n",
    "# Output Layer / Final Layer\n",
    "numberOfFinalLayerOutputs = 27\n",
    "weightsOfFinalLayer = torch.randn(numberOfHiddenLayerNeurons, numberOfFinalLayerOutputs, generator=generator)\n",
    "biasesOfFinalLayer = torch.randn(numberOfFinalLayerOutputs, generator=generator)\n",
    "# Parameters\n",
    "parameters = [embeddingLookUpMatrix, weightsOfHiddenLayer, biasesOfHiddenLayer, weightsOfFinalLayer, biasesOfFinalLayer]\n",
    "# We set all the requires gradient to True\n",
    "for parameter in parameters:\n",
    "    parameter.requires_grad = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "2d1907c1-e517-4934-bc4d-3ce7eaa5f143",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Minibatch Loss: tensor(2.1826, grad_fn=<NllLossBackward0>)\n"
     ]
    }
   ],
   "source": [
    "# We define the number of epochs\n",
    "epochs = 10000\n",
    "for _ in range(epochs):\n",
    "    # Mini-Batches\n",
    "    indexes = torch.randint(low=0, high=inputs.shape[0], size=(37,))\n",
    "    \n",
    "    # Forward Pass (Mini-Batch)\n",
    "    embedding = embeddingLookUpMatrix[inputs[indexes]] # Indexing into look-up table\n",
    "    hiddenLayerStates = torch.tanh(embedding.view(-1, inputBlockSize*embeddingFeatureSpaceLength) @ weightsOfHiddenLayer + biasesOfHiddenLayer)\n",
    "    logits = hiddenLayerStates @ weightsOfFinalLayer + biasesOfFinalLayer\n",
    "    loss = F.cross_entropy(logits, outputs[indexes]) # Indexing into labels\n",
    "    \n",
    "    # Backward Pass (Mini-Batch)\n",
    "    for parameter in parameters:\n",
    "        parameter.grad = None\n",
    "    loss.backward()\n",
    "    \n",
    "    # Update Weights (Mini-Batch)\n",
    "    learning_rate = 0.1\n",
    "    for parameter in parameters:\n",
    "        parameter.data += -learning_rate * parameter.grad\n",
    "\n",
    "print(\"Minibatch Loss:\", loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab3b9bcb-f019-4f20-ba3c-27d9b82a6319",
   "metadata": {},
   "source": [
    "See how we have already beaten the bigram loss from <a href=\"https://github.com/AvishakeAdhikary/Neural-Networks-From-Scratch/blob/main/NameWeave.ipynb\">NameWeave</a>\n",
    "\n",
    "Which was:\n",
    "```python\n",
    "Loss: 2.2942557334899902\n",
    "```\n",
    "\n",
    "But now we have:\n",
    "```python\n",
    "Minibatch Loss: 2.1826\n",
    "```\n",
    "\n",
    "**Well, even if it seems like we have beaten the model...** \\\n",
    "**This is a fairly small model, and models can get larger and complex if we keep adding *Neurons* and *Parameters*** \\\n",
    "**And so we can have millions and billions of parameters,** \\\n",
    "**And as the capacity of the neural network grows, the model becomes more capable of overfitting the training set...**\n",
    "\n",
    "That means if we try to sample from our model, we will only get examples based on only the training data the model was trained on...\\\n",
    "We won't be getting any newer data, Or rather it would be **bad** at predictions.\\\n",
    "In addition to that, the loss on the predicted data would be very high."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50f3df9b-2dbb-4ebd-912a-081363c004a0",
   "metadata": {},
   "source": [
    "To overcome this, **we split our data into three sets**:\n",
    "1. **Training Split (Roughly 80%) - Used to Optimize all the Parameters of the Model**\n",
    "2. **Validation/Dev Split (Roughly 10%) - Used to Optimize all the Hyper-Parameters of the Model**\n",
    "3. **Testing Split (Roughly 10%) - Used to evaluate the performance of the model at the end**\n",
    "\n",
    "The hyper-parameters could be:-\n",
    "1. The size of the hidden layer neurons\n",
    "2. The size of the embeddings\n",
    "3. The strength of the regularization (we are not using yet)\n",
    "and so on...\n",
    "\n",
    "We check over many different *variations* of them to test which works the best for us...\n",
    "\n",
    "**We also tend to calculate the loss on a *test-split* very cautiously and very few times,** \\\n",
    "**Because, every single time we evaluate our test loss, the model learns something from the split..** \\\n",
    "**So, we basically end up overfitting the test-split as well** \\\n",
    "**Therefore, we are only allowed to test the loss on test-split very less times**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9fe43ba8-14ff-4f43-8587-858674affc2a",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "# Splitting Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "75a0e1f8-6f40-421a-8c10-c2274860355b",
   "metadata": {},
   "outputs": [],
   "source": [
    "words = open(\"Datasets/Indian_Names.txt\").read().splitlines()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "10ea37a0-653d-4613-aa81-a04951b61722",
   "metadata": {},
   "outputs": [],
   "source": [
    "words = [word.lower() for word in words]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "id": "3022888b-138f-430f-81d7-3657802ed0a9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "53982\n"
     ]
    }
   ],
   "source": [
    "print(len(words))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "2465e0af-3ce6-422d-8836-35b0adc4e3e8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Characters: ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z']\n",
      "STOI: {'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5, 'f': 6, 'g': 7, 'h': 8, 'i': 9, 'j': 10, 'k': 11, 'l': 12, 'm': 13, 'n': 14, 'o': 15, 'p': 16, 'q': 17, 'r': 18, 's': 19, 't': 20, 'u': 21, 'v': 22, 'w': 23, 'x': 24, 'y': 25, 'z': 26, '.': 0}\n",
      "ITOS {1: 'a', 2: 'b', 3: 'c', 4: 'd', 5: 'e', 6: 'f', 7: 'g', 8: 'h', 9: 'i', 10: 'j', 11: 'k', 12: 'l', 13: 'm', 14: 'n', 15: 'o', 16: 'p', 17: 'q', 18: 'r', 19: 's', 20: 't', 21: 'u', 22: 'v', 23: 'w', 24: 'x', 25: 'y', 26: 'z', 0: '.'}\n"
     ]
    }
   ],
   "source": [
    "# Remember we need our starting and ending tokens as well in these mappings,\n",
    "characters = sorted(list(set(''.join(words)))) # Gives us all the characters in the english alphabet, hopefully our dataset has all of them\n",
    "stoi = {s:i+1 for i,s in enumerate(characters)} # Enumerate returns the tuples of number and string, which can then be mapped to string:index\n",
    "# We manually add these tokens for convenience\n",
    "stoi['.'] = 0\n",
    "itos = {i:s for s,i in stoi.items()} # After we have the string:index mapping, we can easily iterate over their items to map index:string\n",
    "print(\"Characters:\",characters)\n",
    "print(\"STOI:\",stoi)\n",
    "print(\"ITOS\",itos)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ae5208c-acd0-4524-bd2a-ed366daf5f73",
   "metadata": {},
   "source": [
    "We will convert our code:\n",
    "```python\n",
    "# We define a Block Size based on the number of characters we feed are going to feed to predict the next one\n",
    "inputBlockSize = 3\n",
    "\n",
    "# We define two lists, inputs & outputs, where inputs are our blocks of the block size mentioned above and outputs are the label indexes\n",
    "inputs , outputs = [], []\n",
    "\n",
    "# We run a loop for each word in the original dataset\n",
    "for word in words[:5]:\n",
    "    # We define the block for each iteration and fill it with 0 values -> [0, 0, 0]\n",
    "    block = [0] * inputBlockSize # This is also known as the context of the network\n",
    "    # We print each word\n",
    "    print(\"Name:\", word)\n",
    "    # We run another loop for each word's character, here word also needs the ending token '.'\n",
    "    for character in word + '.':\n",
    "        # We take out the index from our look-up table\n",
    "        index = stoi[character]\n",
    "        # We append the input with our block\n",
    "        inputs.append(block)\n",
    "        # We append the output label with out index of the character\n",
    "        outputs.append([index])\n",
    "        # We can check our inputs and thier corresponsing outputs\n",
    "        print(''.join(itos[i] for i in block), '--->', itos[index])\n",
    "        # We then take the block, crop it 1 size from the left and append the next index to it (sliding window of name)\n",
    "        block = block[1:] + [index]\n",
    "# We also convert these inputs and outputs to tensors for neural network processing\n",
    "inputs = torch.tensor(inputs)\n",
    "outputs = torch.flatten(torch.tensor(outputs))\n",
    "```\n",
    "\n",
    "Into a function now..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "ee411a89-ffc1-40d7-8f74-4718681731be",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Will build the dataset on only the words we take as input\n",
    "def buildDataset(words):\n",
    "    # We define a Block Size based on the number of characters we feed are going to feed to predict the next one\n",
    "    inputBlockSize = 3\n",
    "    # We define two lists, inputs & outputs, where inputs are our blocks of the block size mentioned above and outputs are the label indexes\n",
    "    inputs , outputs = [], []\n",
    "    # We iterate over each word\n",
    "    for word in words:\n",
    "        # We define the block for each iteration and fill it with 0 values -> [0, 0, 0]\n",
    "        block = [0] * inputBlockSize # This is also known as the context of the network\n",
    "        # We run another loop for each word's character, here word also needs the ending token '.'\n",
    "        for character in word + '.':\n",
    "            # We take out the index from our look-up table\n",
    "            index = stoi[character]\n",
    "            # We append the input with our block\n",
    "            inputs.append(block)\n",
    "            # We append the output label with out index of the character\n",
    "            outputs.append([index])\n",
    "            # We then take the block, crop it 1 size from the left and append the next index to it (sliding window of name)\n",
    "            block = block[1:] + [index]\n",
    "    # We also convert these inputs and outputs to tensors for neural network processing\n",
    "    inputs = torch.tensor(inputs)\n",
    "    outputs = torch.flatten(torch.tensor(outputs))\n",
    "    # We return the inputs and outputs\n",
    "    return inputs, outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "0e35e198-d35c-4efa-b472-7ed7294079d4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We define a manual seed to random\n",
    "random.seed(69)\n",
    "# We shuffle all the words, so that the model receives all kinds of data\n",
    "random.shuffle(words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "38329636-a307-4e5d-b37a-ff130ed9f4df",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We define two number of inputs\n",
    "# We take the number of examples to 80% in the first variable\n",
    "numberOfInputs1 = int(0.8*len(words))\n",
    "# We take the number of examples to 90% in the first variable\n",
    "numberOfInputs2 = int(0.9*len(words))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "ff6473db-96c9-4df6-8388-b8bae7881580",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Inputs and outputs that go till 80% of the examples\n",
    "trainingInputs, trainingOutputs = buildDataset(words[:numberOfInputs1])\n",
    "# Inputs and outputs that start at 80% of the examples and go till 90% of the examples\n",
    "validationInputs, validationOutputs = buildDataset(words[numberOfInputs1:numberOfInputs2])\n",
    "# Inputs and outputs that start at 90% of the examples\n",
    "testInputs, testOutputs = buildDataset(words[numberOfInputs2:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "id": "83379bcd-18aa-48b5-af52-8b0faad03764",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total Examples: 53982\n",
      "Training Examples: 43185\n",
      "Validation Examples: 5398\n",
      "Test Examples: 5399\n"
     ]
    }
   ],
   "source": [
    "# We can check the numbers\n",
    "print(\"Total Examples:\",len(words))\n",
    "print(\"Training Examples:\",len(words[:numberOfInputs1]))\n",
    "print(\"Validation Examples:\",len(words[numberOfInputs1:numberOfInputs2]))\n",
    "print(\"Test Examples:\",len(words[numberOfInputs2:]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "id": "464a8d73-f735-4c6c-9d68-35c3b15d2978",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We will define a generator to give the same result on your machine, as of my machine\n",
    "generator = torch.Generator().manual_seed(6942069420)\n",
    "# Embedding Matrix (Input Layer)\n",
    "embeddingFeatureSpaceLength = 2\n",
    "embeddingLookUpMatrix = torch.randn((len(stoi),embeddingFeatureSpaceLength), generator=generator)\n",
    "# Hidden Layer\n",
    "numberOfHiddenLayerNeurons = 100\n",
    "weightsOfHiddenLayer = torch.randn((inputBlockSize*embeddingFeatureSpaceLength), numberOfHiddenLayerNeurons, generator=generator)\n",
    "biasesOfHiddenLayer = torch.randn(numberOfHiddenLayerNeurons, generator=generator)\n",
    "# Output Layer / Final Layer\n",
    "numberOfFinalLayerOutputs = 27\n",
    "weightsOfFinalLayer = torch.randn(numberOfHiddenLayerNeurons, numberOfFinalLayerOutputs, generator=generator)\n",
    "biasesOfFinalLayer = torch.randn(numberOfFinalLayerOutputs, generator=generator)\n",
    "# Parameters\n",
    "parameters = [embeddingLookUpMatrix, weightsOfHiddenLayer, biasesOfHiddenLayer, weightsOfFinalLayer, biasesOfFinalLayer]\n",
    "# We set all the requires gradient to True\n",
    "for parameter in parameters:\n",
    "    parameter.requires_grad = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "id": "107ee356-6e46-4453-8107-d410bb478a28",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Minibatch Loss: tensor(2.4522, grad_fn=<NllLossBackward0>)\n"
     ]
    }
   ],
   "source": [
    "# We define the number of epochs\n",
    "epochs = 10000\n",
    "for _ in range(epochs):\n",
    "    # Mini-Batches\n",
    "    indexes = torch.randint(low=0, high=trainingInputs.shape[0], size=(37,)) # Changing to Training Inputs\n",
    "    \n",
    "    # Forward Pass (Mini-Batch)\n",
    "    embedding = embeddingLookUpMatrix[inputs[indexes]]\n",
    "    hiddenLayerStates = torch.tanh(embedding.view(-1, inputBlockSize*embeddingFeatureSpaceLength) @ weightsOfHiddenLayer + biasesOfHiddenLayer)\n",
    "    logits = hiddenLayerStates @ weightsOfFinalLayer + biasesOfFinalLayer\n",
    "    loss = F.cross_entropy(logits, trainingOutputs[indexes]) # Changing to Training Outputs\n",
    "    \n",
    "    # Backward Pass (Mini-Batch)\n",
    "    for parameter in parameters:\n",
    "        parameter.grad = None\n",
    "    loss.backward()\n",
    "    \n",
    "    # Update Weights (Mini-Batch)\n",
    "    learning_rate = 0.01\n",
    "    for parameter in parameters:\n",
    "        parameter.data += -learning_rate * parameter.grad\n",
    "\n",
    "print(\"Minibatch Loss:\", loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96550a21-9ca7-40d6-a09a-4697194d768c",
   "metadata": {},
   "source": [
    "Let's evaluate the loss over the *Training-Split*..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "861daaaf-c515-43c1-a668-0c41585a5237",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loss: tensor(2.7447, grad_fn=<NllLossBackward0>)\n"
     ]
    }
   ],
   "source": [
    "# Forward Pass (Training-Split)\n",
    "embedding = embeddingLookUpMatrix[trainingInputs]\n",
    "hiddenLayerStates = torch.tanh(embedding.view(-1, inputBlockSize*embeddingFeatureSpaceLength) @ weightsOfHiddenLayer + biasesOfHiddenLayer)\n",
    "logits = hiddenLayerStates @ weightsOfFinalLayer + biasesOfFinalLayer\n",
    "loss = F.cross_entropy(logits, trainingOutputs)\n",
    "print(\"Loss:\", loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24c0300c-ece5-4c71-9255-f8a7280620fe",
   "metadata": {},
   "source": [
    "Now we will evaluate over the *Dev-Split* or the *Validation-Split*..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "id": "6cd0ed83-a9b7-4614-9336-0bc55985b8f2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loss: tensor(2.7460, grad_fn=<NllLossBackward0>)\n"
     ]
    }
   ],
   "source": [
    "# Forward Pass (Validation-Split)\n",
    "embedding = embeddingLookUpMatrix[validationInputs]\n",
    "hiddenLayerStates = torch.tanh(embedding.view(-1, inputBlockSize*embeddingFeatureSpaceLength) @ weightsOfHiddenLayer + biasesOfHiddenLayer)\n",
    "logits = hiddenLayerStates @ weightsOfFinalLayer + biasesOfFinalLayer\n",
    "loss = F.cross_entropy(logits, validationOutputs)\n",
    "print(\"Loss:\", loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e0323bf-9be4-4a28-8a5b-60a473477e9a",
   "metadata": {},
   "source": [
    "We see that the loss of the *Training-Split* and the *Validation-Split* are almost equal,\\\n",
    "That means that we are not **overfitting**, and that our model is not powerful enough to purely memorize the data.\n",
    "\n",
    "Rather, so far we are **underfitting** the data, because, the loss of the *Training-Split* and the *Validation-Split* are roughly equal...\\\n",
    "That means our model is very small and we can make performance improvements by scaling up our model..."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5893507f-0918-4374-84bb-7f59f4cc5e08",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "# Scaling Up Model (Increasing Complexity)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "id": "f33b0daa-4183-4a88-bce4-b088862f063a",
   "metadata": {},
   "outputs": [],
   "source": [
    "words = open(\"Datasets/Indian_Names.txt\").read().splitlines()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "id": "cf30e513-a764-4568-866e-7111ed61710d",
   "metadata": {},
   "outputs": [],
   "source": [
    "words = [word.lower() for word in words]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "id": "dd891667-ff88-4eb4-8cad-f78da41c8d32",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "53982\n"
     ]
    }
   ],
   "source": [
    "print(len(words))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "id": "fccbea2a-7b69-4ac7-8809-4c61eb5b6b89",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Characters: ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z']\n",
      "STOI: {'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5, 'f': 6, 'g': 7, 'h': 8, 'i': 9, 'j': 10, 'k': 11, 'l': 12, 'm': 13, 'n': 14, 'o': 15, 'p': 16, 'q': 17, 'r': 18, 's': 19, 't': 20, 'u': 21, 'v': 22, 'w': 23, 'x': 24, 'y': 25, 'z': 26, '.': 0}\n",
      "ITOS {1: 'a', 2: 'b', 3: 'c', 4: 'd', 5: 'e', 6: 'f', 7: 'g', 8: 'h', 9: 'i', 10: 'j', 11: 'k', 12: 'l', 13: 'm', 14: 'n', 15: 'o', 16: 'p', 17: 'q', 18: 'r', 19: 's', 20: 't', 21: 'u', 22: 'v', 23: 'w', 24: 'x', 25: 'y', 26: 'z', 0: '.'}\n"
     ]
    }
   ],
   "source": [
    "# Remember we need our starting and ending tokens as well in these mappings,\n",
    "characters = sorted(list(set(''.join(words)))) # Gives us all the characters in the english alphabet, hopefully our dataset has all of them\n",
    "stoi = {s:i+1 for i,s in enumerate(characters)} # Enumerate returns the tuples of number and string, which can then be mapped to string:index\n",
    "# We manually add these tokens for convenience\n",
    "stoi['.'] = 0\n",
    "itos = {i:s for s,i in stoi.items()} # After we have the string:index mapping, we can easily iterate over their items to map index:string\n",
    "print(\"Characters:\",characters)\n",
    "print(\"STOI:\",stoi)\n",
    "print(\"ITOS\",itos)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "id": "0db22304-a878-4a1b-9210-ce8bc52f749b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Will build the dataset on only the words we take as input\n",
    "def buildDataset(words):\n",
    "    # We define a Block Size based on the number of characters we feed are going to feed to predict the next one\n",
    "    inputBlockSize = 3\n",
    "    # We define two lists, inputs & outputs, where inputs are our blocks of the block size mentioned above and outputs are the label indexes\n",
    "    inputs , outputs = [], []\n",
    "    # We iterate over each word\n",
    "    for word in words:\n",
    "        # We define the block for each iteration and fill it with 0 values -> [0, 0, 0]\n",
    "        block = [0] * inputBlockSize # This is also known as the context of the network\n",
    "        # We run another loop for each word's character, here word also needs the ending token '.'\n",
    "        for character in word + '.':\n",
    "            # We take out the index from our look-up table\n",
    "            index = stoi[character]\n",
    "            # We append the input with our block\n",
    "            inputs.append(block)\n",
    "            # We append the output label with out index of the character\n",
    "            outputs.append([index])\n",
    "            # We then take the block, crop it 1 size from the left and append the next index to it (sliding window of name)\n",
    "            block = block[1:] + [index]\n",
    "    # We also convert these inputs and outputs to tensors for neural network processing\n",
    "    inputs = torch.tensor(inputs)\n",
    "    outputs = torch.flatten(torch.tensor(outputs))\n",
    "    # We return the inputs and outputs\n",
    "    return inputs, outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "id": "66e6aac1-86a4-468b-9a5f-44d9056b4b6b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We define a manual seed to random\n",
    "random.seed(69)\n",
    "# We shuffle all the words, so that the model receives all kinds of data\n",
    "random.shuffle(words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "id": "081d4c8a-b238-495c-8d1f-59c680790206",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We define two number of inputs\n",
    "# We take the number of examples to 80% in the first variable\n",
    "numberOfInputs1 = int(0.8*len(words))\n",
    "# We take the number of examples to 90% in the first variable\n",
    "numberOfInputs2 = int(0.9*len(words))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "id": "adf9da10-b4c9-49b5-88b9-ec730c70d5ae",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Inputs and outputs that go till 80% of the examples\n",
    "trainingInputs, trainingOutputs = buildDataset(words[:numberOfInputs1])\n",
    "# Inputs and outputs that start at 80% of the examples and go till 90% of the examples\n",
    "validationInputs, validationOutputs = buildDataset(words[numberOfInputs1:numberOfInputs2])\n",
    "# Inputs and outputs that start at 90% of the examples\n",
    "testInputs, testOutputs = buildDataset(words[numberOfInputs2:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "id": "cecaad8f-e46f-4709-8b55-9386f5a5479e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total Examples: 53982\n",
      "Training Examples: 43185\n",
      "Validation Examples: 5398\n",
      "Test Examples: 5399\n"
     ]
    }
   ],
   "source": [
    "# We can check the numbers\n",
    "print(\"Total Examples:\",len(words))\n",
    "print(\"Training Examples:\",len(words[:numberOfInputs1]))\n",
    "print(\"Validation Examples:\",len(words[numberOfInputs1:numberOfInputs2]))\n",
    "print(\"Test Examples:\",len(words[numberOfInputs2:]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "id": "eac7a89a-81ec-4c2c-9079-f17e2e027c59",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We will define a generator to give the same result on your machine, as of my machine\n",
    "generator = torch.Generator().manual_seed(6942069420)\n",
    "# Embedding Matrix (Input Layer)\n",
    "embeddingFeatureSpaceLength = 2\n",
    "embeddingLookUpMatrix = torch.randn((len(stoi),embeddingFeatureSpaceLength), generator=generator)\n",
    "# Hidden Layer\n",
    "numberOfHiddenLayerNeurons = 300 # We increase the number of neurons here\n",
    "weightsOfHiddenLayer = torch.randn((inputBlockSize*embeddingFeatureSpaceLength), numberOfHiddenLayerNeurons, generator=generator)\n",
    "biasesOfHiddenLayer = torch.randn(numberOfHiddenLayerNeurons, generator=generator)\n",
    "# Output Layer / Final Layer\n",
    "numberOfFinalLayerOutputs = 27\n",
    "weightsOfFinalLayer = torch.randn(numberOfHiddenLayerNeurons, numberOfFinalLayerOutputs, generator=generator)\n",
    "biasesOfFinalLayer = torch.randn(numberOfFinalLayerOutputs, generator=generator)\n",
    "# Parameters\n",
    "parameters = [embeddingLookUpMatrix, weightsOfHiddenLayer, biasesOfHiddenLayer, weightsOfFinalLayer, biasesOfFinalLayer]\n",
    "# We set all the requires gradient to True\n",
    "for parameter in parameters:\n",
    "    parameter.requires_grad = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "id": "73bf29c9-7114-479b-883e-b2f161765328",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Minibatch Loss: tensor(3.0100, grad_fn=<NllLossBackward0>)\n"
     ]
    }
   ],
   "source": [
    "# We define the number of epochs\n",
    "epochs = 20000\n",
    "for _ in range(epochs):\n",
    "    # Mini-Batches\n",
    "    indexes = torch.randint(low=0, high=trainingInputs.shape[0], size=(37,)) # Changing to Training Inputs\n",
    "    \n",
    "    # Forward Pass (Mini-Batch)\n",
    "    embedding = embeddingLookUpMatrix[inputs[indexes]]\n",
    "    hiddenLayerStates = torch.tanh(embedding.view(-1, inputBlockSize*embeddingFeatureSpaceLength) @ weightsOfHiddenLayer + biasesOfHiddenLayer)\n",
    "    logits = hiddenLayerStates @ weightsOfFinalLayer + biasesOfFinalLayer\n",
    "    loss = F.cross_entropy(logits, trainingOutputs[indexes]) # Changing to Training Outputs\n",
    "    \n",
    "    # Backward Pass (Mini-Batch)\n",
    "    for parameter in parameters:\n",
    "        parameter.grad = None\n",
    "    loss.backward()\n",
    "    \n",
    "    # Update Weights (Mini-Batch)\n",
    "    learning_rate = 0.01\n",
    "    for parameter in parameters:\n",
    "        parameter.data += -learning_rate * parameter.grad\n",
    "\n",
    "print(\"Minibatch Loss:\", loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "id": "0e578746-42f6-4113-b41f-2c10a5f513af",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loss: tensor(2.7526, grad_fn=<NllLossBackward0>)\n"
     ]
    }
   ],
   "source": [
    "# Forward Pass (Training-Split)\n",
    "embedding = embeddingLookUpMatrix[trainingInputs]\n",
    "hiddenLayerStates = torch.tanh(embedding.view(-1, inputBlockSize*embeddingFeatureSpaceLength) @ weightsOfHiddenLayer + biasesOfHiddenLayer)\n",
    "logits = hiddenLayerStates @ weightsOfFinalLayer + biasesOfFinalLayer\n",
    "loss = F.cross_entropy(logits, trainingOutputs)\n",
    "print(\"Loss:\", loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "id": "af25da61-f905-4698-ac3a-2a7a49dc95b6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loss: tensor(2.7547, grad_fn=<NllLossBackward0>)\n"
     ]
    }
   ],
   "source": [
    "# Forward Pass (Validation-Split)\n",
    "embedding = embeddingLookUpMatrix[validationInputs]\n",
    "hiddenLayerStates = torch.tanh(embedding.view(-1, inputBlockSize*embeddingFeatureSpaceLength) @ weightsOfHiddenLayer + biasesOfHiddenLayer)\n",
    "logits = hiddenLayerStates @ weightsOfFinalLayer + biasesOfFinalLayer\n",
    "loss = F.cross_entropy(logits, validationOutputs)\n",
    "print(\"Loss:\", loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a64416a7-6228-421e-9be6-1c50b11308bb",
   "metadata": {},
   "source": [
    "We see that we get a slightly lower loss that what we had before...\\\n",
    "But even then it does not happen too well...\n",
    "\n",
    "One of the reasons might be is because,\\\n",
    "Even though we made our hidden layer much bigger, the input or the embedding layer could be a bottleneck.\\\n",
    "That is because we could be pushing too many characters into just 2-dimensions and the neural network could be not able to use the space effectively.."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb391d35-f10b-4a7b-bb24-abaa6c799660",
   "metadata": {},
   "source": [
    "If we try to visualize our embeddings now, we would not see a pattern like we would on the *names that use vowels* because the Indian names are based more on the *syllables*...\n",
    "\n",
    "But roughly speaking, if these were the names which used *vowels*, we would be seeing that the vowels would stack up almost forming a cluster, the token '.' treating itself separately, the character 'q' treating itself differently.\\\n",
    "Thus the embeddings would make sense...\n",
    "\n",
    "Ours make sense too,\\\n",
    "In a different way."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac3b5073-5a55-4c92-a038-2fb6550a7921",
   "metadata": {},
   "source": [
    "### Making the embeddings bigger"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "id": "a3a047e5-d83c-4b21-b432-b74135bbe9bd",
   "metadata": {},
   "outputs": [],
   "source": [
    "words = open(\"Datasets/Indian_Names.txt\").read().splitlines()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "id": "11f852f1-92c7-4b41-ba47-2e2a0d96566b",
   "metadata": {},
   "outputs": [],
   "source": [
    "words = [word.lower() for word in words]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "id": "55e1b428-3251-43f9-8c0d-f71abceb81e8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "53982\n"
     ]
    }
   ],
   "source": [
    "print(len(words))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "id": "26d14918-807b-4175-bebc-ad5902a91dbb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Characters: ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z']\n",
      "STOI: {'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5, 'f': 6, 'g': 7, 'h': 8, 'i': 9, 'j': 10, 'k': 11, 'l': 12, 'm': 13, 'n': 14, 'o': 15, 'p': 16, 'q': 17, 'r': 18, 's': 19, 't': 20, 'u': 21, 'v': 22, 'w': 23, 'x': 24, 'y': 25, 'z': 26, '.': 0}\n",
      "ITOS {1: 'a', 2: 'b', 3: 'c', 4: 'd', 5: 'e', 6: 'f', 7: 'g', 8: 'h', 9: 'i', 10: 'j', 11: 'k', 12: 'l', 13: 'm', 14: 'n', 15: 'o', 16: 'p', 17: 'q', 18: 'r', 19: 's', 20: 't', 21: 'u', 22: 'v', 23: 'w', 24: 'x', 25: 'y', 26: 'z', 0: '.'}\n"
     ]
    }
   ],
   "source": [
    "# Remember we need our starting and ending tokens as well in these mappings,\n",
    "characters = sorted(list(set(''.join(words)))) # Gives us all the characters in the english alphabet, hopefully our dataset has all of them\n",
    "stoi = {s:i+1 for i,s in enumerate(characters)} # Enumerate returns the tuples of number and string, which can then be mapped to string:index\n",
    "# We manually add these tokens for convenience\n",
    "stoi['.'] = 0\n",
    "itos = {i:s for s,i in stoi.items()} # After we have the string:index mapping, we can easily iterate over their items to map index:string\n",
    "print(\"Characters:\",characters)\n",
    "print(\"STOI:\",stoi)\n",
    "print(\"ITOS\",itos)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "id": "865c81f9-bce0-4239-9759-22f29f5f2e26",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Will build the dataset on only the words we take as input\n",
    "def buildDataset(words):\n",
    "    # We define a Block Size based on the number of characters we feed are going to feed to predict the next one\n",
    "    inputBlockSize = 3\n",
    "    # We define two lists, inputs & outputs, where inputs are our blocks of the block size mentioned above and outputs are the label indexes\n",
    "    inputs , outputs = [], []\n",
    "    # We iterate over each word\n",
    "    for word in words:\n",
    "        # We define the block for each iteration and fill it with 0 values -> [0, 0, 0]\n",
    "        block = [0] * inputBlockSize # This is also known as the context of the network\n",
    "        # We run another loop for each word's character, here word also needs the ending token '.'\n",
    "        for character in word + '.':\n",
    "            # We take out the index from our look-up table\n",
    "            index = stoi[character]\n",
    "            # We append the input with our block\n",
    "            inputs.append(block)\n",
    "            # We append the output label with out index of the character\n",
    "            outputs.append([index])\n",
    "            # We then take the block, crop it 1 size from the left and append the next index to it (sliding window of name)\n",
    "            block = block[1:] + [index]\n",
    "    # We also convert these inputs and outputs to tensors for neural network processing\n",
    "    inputs = torch.tensor(inputs)\n",
    "    outputs = torch.flatten(torch.tensor(outputs))\n",
    "    # We return the inputs and outputs\n",
    "    return inputs, outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "id": "e5ce82b6-f298-46cd-9074-202496ebb140",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We define a manual seed to random\n",
    "random.seed(69)\n",
    "# We shuffle all the words, so that the model receives all kinds of data\n",
    "random.shuffle(words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "id": "a808ebcc-fd83-4703-8242-70112854e8ce",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We define two number of inputs\n",
    "# We take the number of examples to 80% in the first variable\n",
    "numberOfInputs1 = int(0.8*len(words))\n",
    "# We take the number of examples to 90% in the first variable\n",
    "numberOfInputs2 = int(0.9*len(words))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "id": "d616edc9-22c9-4b7d-b88f-3687f80488ce",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Inputs and outputs that go till 80% of the examples\n",
    "trainingInputs, trainingOutputs = buildDataset(words[:numberOfInputs1])\n",
    "# Inputs and outputs that start at 80% of the examples and go till 90% of the examples\n",
    "validationInputs, validationOutputs = buildDataset(words[numberOfInputs1:numberOfInputs2])\n",
    "# Inputs and outputs that start at 90% of the examples\n",
    "testInputs, testOutputs = buildDataset(words[numberOfInputs2:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "id": "268da7d2-9c7c-455b-a321-149359f715a9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total Examples: 53982\n",
      "Training Examples: 43185\n",
      "Validation Examples: 5398\n",
      "Test Examples: 5399\n"
     ]
    }
   ],
   "source": [
    "# We can check the numbers\n",
    "print(\"Total Examples:\",len(words))\n",
    "print(\"Training Examples:\",len(words[:numberOfInputs1]))\n",
    "print(\"Validation Examples:\",len(words[numberOfInputs1:numberOfInputs2]))\n",
    "print(\"Test Examples:\",len(words[numberOfInputs2:]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "id": "b8e41855-2e8c-4993-8c2c-d02d39278b98",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We will define a generator to give the same result on your machine, as of my machine\n",
    "generator = torch.Generator().manual_seed(6942069420)\n",
    "# Embedding Matrix (Input Layer)\n",
    "embeddingFeatureSpaceLength = 10 \n",
    "embeddingLookUpMatrix = torch.randn((len(stoi),embeddingFeatureSpaceLength), generator=generator)\n",
    "# Hidden Layer\n",
    "numberOfHiddenLayerNeurons = 200 # We increase the number of neurons here\n",
    "weightsOfHiddenLayer = torch.randn((inputBlockSize*embeddingFeatureSpaceLength), numberOfHiddenLayerNeurons, generator=generator)\n",
    "biasesOfHiddenLayer = torch.randn(numberOfHiddenLayerNeurons, generator=generator)\n",
    "# Output Layer / Final Layer\n",
    "numberOfFinalLayerOutputs = 27\n",
    "weightsOfFinalLayer = torch.randn(numberOfHiddenLayerNeurons, numberOfFinalLayerOutputs, generator=generator)\n",
    "biasesOfFinalLayer = torch.randn(numberOfFinalLayerOutputs, generator=generator)\n",
    "# Parameters\n",
    "parameters = [embeddingLookUpMatrix, weightsOfHiddenLayer, biasesOfHiddenLayer, weightsOfFinalLayer, biasesOfFinalLayer]\n",
    "# We set all the requires gradient to True\n",
    "for parameter in parameters:\n",
    "    parameter.requires_grad = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "id": "236b6c5d-6a5d-42f5-a476-f96b74cb498f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We define a list that goes from -3 to 0 with 1000 steps\n",
    "lossExpression = torch.linspace(start=-3, end=0, steps=1000)\n",
    "# We do a 10^value expression on the value of the above list\n",
    "lossExponentExpression = 10**lossExpression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "id": "5937f500-4326-47e8-a4df-b4b567eccc5d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We want to track the learning rates that we use in the training\n",
    "learningRates = []\n",
    "# We want to track the losses that we use in the training\n",
    "losses = []\n",
    "# We want to track the number of iterations we use in the training\n",
    "numberOfIterations = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "id": "5963ee81-3d72-4f0f-9593-d940edd19503",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Minibatch Loss: tensor(2.7674, grad_fn=<NllLossBackward0>)\n"
     ]
    }
   ],
   "source": [
    "# We define the number of epochs\n",
    "epochs = 50000\n",
    "for i in range(epochs):\n",
    "    # Mini-Batches\n",
    "    indexes = torch.randint(low=0, high=trainingInputs.shape[0], size=(37,)) # Changing to Training Inputs\n",
    "    \n",
    "    # Forward Pass (Mini-Batch)\n",
    "    embedding = embeddingLookUpMatrix[inputs[indexes]]\n",
    "    hiddenLayerStates = torch.tanh(embedding.view(-1, inputBlockSize*embeddingFeatureSpaceLength) @ weightsOfHiddenLayer + biasesOfHiddenLayer)\n",
    "    logits = hiddenLayerStates @ weightsOfFinalLayer + biasesOfFinalLayer\n",
    "    loss = F.cross_entropy(logits, trainingOutputs[indexes]) # Changing to Training Outputs\n",
    "    \n",
    "    # Backward Pass (Mini-Batch)\n",
    "    for parameter in parameters:\n",
    "        parameter.grad = None\n",
    "    loss.backward()\n",
    "    \n",
    "    # Update Weights (Mini-Batch)\n",
    "    learning_rate = 0.01\n",
    "    for parameter in parameters:\n",
    "        parameter.data += -learning_rate * parameter.grad\n",
    "\n",
    "    # Tracking the stats\n",
    "    # We append both the learning rates and losses that we use per iteration\n",
    "    numberOfIterations.append(i)\n",
    "    losses.append(loss.log10().item()) # We do that to squash the steep curve to a nicer graph\n",
    "print(\"Minibatch Loss:\", loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "id": "93930055-e517-4631-8886-e4780f98bb5f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Minibatch Loss: tensor(2.5523, grad_fn=<NllLossBackward0>)\n"
     ]
    }
   ],
   "source": [
    "# Learning Rate Decay\n",
    "# We define the number of epochs\n",
    "epochs = 1000\n",
    "for i in range(epochs):\n",
    "    # Mini-Batches\n",
    "    indexes = torch.randint(low=0, high=trainingInputs.shape[0], size=(37,)) # Changing to Training Inputs\n",
    "    \n",
    "    # Forward Pass (Mini-Batch)\n",
    "    embedding = embeddingLookUpMatrix[inputs[indexes]]\n",
    "    hiddenLayerStates = torch.tanh(embedding.view(-1, inputBlockSize*embeddingFeatureSpaceLength) @ weightsOfHiddenLayer + biasesOfHiddenLayer)\n",
    "    logits = hiddenLayerStates @ weightsOfFinalLayer + biasesOfFinalLayer\n",
    "    loss = F.cross_entropy(logits, trainingOutputs[indexes]) # Changing to Training Outputs\n",
    "    \n",
    "    # Backward Pass (Mini-Batch)\n",
    "    for parameter in parameters:\n",
    "        parameter.grad = None\n",
    "    loss.backward()\n",
    "    \n",
    "    # Update Weights (Mini-Batch)\n",
    "    learning_rate = 0.001\n",
    "    for parameter in parameters:\n",
    "        parameter.data += -learning_rate * parameter.grad\n",
    "\n",
    "    # Tracking the stats\n",
    "    # We append both the learning rates and losses that we use per iteration\n",
    "    numberOfIterations.append(i)\n",
    "    losses.append(loss.log10().item()) # We do that to squash the steep curve to a nicer graph\n",
    "print(\"Minibatch Loss:\", loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "id": "db10422c-7d72-446a-b536-39965b11a998",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x20f30f9ea10>]"
      ]
     },
     "execution_count": 107,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiMAAAGdCAYAAADAAnMpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAABOxklEQVR4nO3dd1hT9+IG8DdhBFCGgoIDBeteuCrirIriqB2/3tbWtrZWbW31Xlu7pO4O8XZYO7S21tH21tXW2uGkKOIAERQnigoIKhvZm5zfH0hIICEJJDmEvJ/n4VHOOTnny2GcN98pEQRBABEREZFIpGIXgIiIiCwbwwgRERGJimGEiIiIRMUwQkRERKJiGCEiIiJRMYwQERGRqBhGiIiISFQMI0RERCQqa7ELoAu5XI67d+/C0dEREolE7OIQERGRDgRBQH5+Ptq3bw+pVHP9h1mEkbt378LT01PsYhAREVEDJCcno2PHjhr3m0UYcXR0BFD1xTg5OYlcGiIiItJFXl4ePD09Fc9xTcwijFQ3zTg5OTGMEBERmRltXSzYgZWIiIhExTBCREREomIYISIiIlExjBAREZGoGEaIiIhIVAwjREREJCqGESIiIhIVwwgRERGJimGEiIiIRMUwQkRERKJiGCEiIiJRMYwQERGRqCw6jMSm5OH74/Eor5SLXRQiIiKLZRar9hrL5C+OA6haTXD2SG+RS0NERGSZLLpmpNrlu7liF4GIiMhiMYwA8HJtIXYRiIiILBbDCIA794rFLgIREZHFYhgBsCsqWewiEBERWSyGESIiIhIVwwgRERGJimGEiIiIRMUwQkRERKJiGCEiIiJRMYwQERGRqPQOI2FhYZg2bRrat28PiUSCvXv36vzakydPwtraGgMGDND3skRERNRM6R1GCgsL4ePjg/Xr1+v1upycHMycORPjx4/X95JERETUjOm9UN7kyZMxefJkvS80b948zJgxA1ZWVnrVphAREVHzZpI+I1u3bkV8fDxWrFih0/GlpaXIy8tT+SAiIqLmyehh5Pr161i8eDH+97//wdpat4qYoKAgODs7Kz48PT2NXEoiIiISi1HDSGVlJWbMmIFVq1ahe/fuOr8uMDAQubm5io/kZK4dQ0RE1Fzp3WdEH/n5+YiKisK5c+ewYMECAIBcLocgCLC2tsbhw4cxbty4Oq+TyWSQyWTGLBoRERE1EUYNI05OTrh48aLKtg0bNuDIkSP49ddf4e3tbczLExERkRnQO4wUFBTgxo0bis8TEhIQExOD1q1bo1OnTggMDMSdO3fw448/QiqVom/fviqvb9u2Lezs7OpsJyIiIsukdxiJiorC2LFjFZ8vWrQIAPDCCy9g27ZtSElJQVJSkuFKSERERM2aRBAEQexCaJOXlwdnZ2fk5ubCycnJYOf1WrxP8f/ENVMNdl4iIiLS/fnNtWmIiIhIVAwjREREJCqGESIiIhIVw8h99wrLxC4CERGRRWIYue/roze0H0REREQGxzByX3F5pdhFICIiskgMI/dJxC4AERGRhWIYuU/CNEJERCQKhpH7Dl9OE7sIREREFolh5L70/FKxi0BERGSRGEaIiIhIVAwjREREJCqGESIiIhIVwwgRERGJimGEiIiIRMUwQkRERKJiGCEiIiJRMYwQERGRqBhGiIiISFQMI0RERCQqhhEiIiISFcMIERERiYphhIiIiETFMEJERESiYhghIiIiUTGMEBERkagYRoiIiEhUDCNKCkorxC4CERGRxWEYUbLyz8tiF4GIiMjiMIwoCb2WLnYRiIiILA7DiBKpRCJ2EYiIiCwOw4iS9PxSsYtARERkcRhGiIiISFQMI0RERCQqhhEiIiISFcNILYIgiF0EIiIii8IwUsuOyGSxi0BERGRRGEZq2XjspthFICIisigMI7VkcHgvERGRSTGM1CKAfUaIiIhMiWGkFvZfJSIiMi2GESIiIhIVw0gtrBghIiIyLYsOIy+N8Ba7CERERBbPosMIF+klIiISn2WHETXbyirkJi8HERGRJbPsMMKaESIiItFZeBhhGiEiIhKbRYcRaynDCBERkdj0DiNhYWGYNm0a2rdvD4lEgr1799Z7/J49ezBhwgS0adMGTk5O8PPzw6FDhxpaXoN6YnBHsYtARERk8fQOI4WFhfDx8cH69et1Oj4sLAwTJkzA/v37ER0djbFjx2LatGk4d+6c3oU1tBa21mIXgYiIyOLp/TSePHkyJk+erPPx69atU/l89erV+OOPP/DXX39h4MCB+l7eoDR1GUnMLISXWwvTFoaIiMhCmbzPiFwuR35+Plq3bq3xmNLSUuTl5al8GIOmdWje/e2CUa5HREREdZk8jHz66acoKCjAU089pfGYoKAgODs7Kz48PT1NWEKgsKzCpNcjIiKyZCYNI9u3b8eqVauwe/dutG3bVuNxgYGByM3NVXwkJycbpTwc2UtERCQ+k/Xg3LlzJ+bMmYNffvkF/v7+9R4rk8kgk8mMXiZNzTRERERkOiapGdmxYwdmzZqFHTt2YOrUqaa4ZKOk5JRg++kklJRXil0UIiKiZk/vmpGCggLcuHFD8XlCQgJiYmLQunVrdOrUCYGBgbhz5w5+/PFHAFVNMy+88AK++OIL+Pr6IjU1FQBgb28PZ2dnA30ZhpVVWIb3fr+I2/eK8M6knmIXh4iIqFnTu2YkKioKAwcOVAzLXbRoEQYOHIjly5cDAFJSUpCUlKQ4/rvvvkNFRQXmz5+Pdu3aKT4WLlxooC+h4bT1GTlxI9M0BSEiIrJgeteMPPTQQxDq6Wyxbds2lc9DQ0P1vYTJsM8IERGR+Cx6bRoiIiISH8MIERERiYphhIiIiETFMFIP9ikhIiIyPoYRIiIiEpVFhxEne5NNQEtEREQaWHQYcbBlGCEiIhKbRYcRbeLS8lHEFXyJiIiMimGkHqUVckz54rjYxSAiImrWGEa0SMwqErsIREREzRrDCBEREYmKYYSIiIhExTBCREREomIYISIiIlExjBAREZGoGEaIiIhIVAwjREREJCqGESIiIhIVwwgRERGJimGEiIiIRMUwogO5XMDfF+5i47GbYheFiIio2bEWuwDm4O+LKfjPjnMAgBEPuKFfR2eRS0RERNR8sGZEB4mZhYr/ZxWWilgSIiKi5odhhIiIiERl8WEkoI+7XscLRioHERGRpbL4MOJgy24zREREYrL4MKILucD6ECIiImNhGNGBypBe5hIiIiKDsvgwIuhQ61FSLq85nmmEiIjIoCw+jOirvFLQKcAQERGRbhhG9PTKT9GY/m2E2MUgIiJqNhhGGiAyMVvsIhARETUbDCNEREQkKoYRIiIiEhXDCBEREYmKYYSIiIhEZfFhpKGDdIvKKgxaDiIiIktl8WGkoaZ+eQJlFXLtBxIREVG9GEYaKCGzENtP3xK7GERERGaPYaQRMgvKxC4CERGR2WMYISIiIlExjBAREZGoGEYaoaisUuwiEBERmT2GkUbILCgVuwhERERmj2GkESQSsUtARERk/iw+jPTr4Nzg10Yl3kNpRSUi4rNw6U6uAUtFRERkOazFLoDYXhjuhbDrmQiLy9D7tXdyivH85khEJmQDABLXTDV08YiIiJo9i68ZsbGSYs5I7wa/vjqIEBERUcNYfBgBGr4+DRERETWe3mEkLCwM06ZNQ/v27SGRSLB3716trwkNDcWgQYMgk8nQtWtXbNu2rQFFNR5BYBwhIiISi95hpLCwED4+Pli/fr1OxyckJGDq1KkYO3YsYmJi8Prrr2POnDk4dOiQ3oVt6h79+gSyONyXiIhIL3p3YJ08eTImT56s8/EbN26Et7c3PvvsMwBAr169cOLECXz++ecICAjQ9/JN2vnbuVj3z3V88FhfsYtCRERkNozeZyQ8PBz+/v4q2wICAhAeHq7xNaWlpcjLy1P5MBfF5ZyVlYiISB9GDyOpqalwd3dX2ebu7o68vDwUFxerfU1QUBCcnZ0VH56enkYt4xCv1kY9PxEREWnWJEfTBAYGIjc3V/GRnJxs1Ou1lBluupXc4nL8FHEL9wrLDHZOIiKi5szok555eHggLS1NZVtaWhqcnJxgb2+v9jUymQwymczYRTOK4CtpCL6Shr9i7mL3PD+xi0NERNTkGb1mxM/PDyEhISrbgoOD4efXvB/UkYmcDI2IiEgXeoeRgoICxMTEICYmBkDV0N2YmBgkJSUBqGpimTlzpuL4efPmIT4+Hu+88w6uXr2KDRs2YPfu3XjjjTcM8xUQERGRWdM7jERFRWHgwIEYOHAgAGDRokUYOHAgli9fDgBISUlRBBMA8Pb2xr59+xAcHAwfHx989tln+P7775vdsF4iIiJqGIlgBtOP5uXlwdnZGbm5uXBycjLKNfyCQpCSW2LQc3LhPCIismS6Pr+b5GgaMfw0e6jBz7nunzhcTTWfOVKIiIjEwDByX8dWDgY/57p/rmPSuuMGPy8REVFzwjBCREREomIYuU8iEbsERERElolhhIiIiETFMHKfBKwaISIiEgPDyH1spiEiIhIHwwgRERGJimHkPimrRoiIiETBMHKflZRhhIiISAwMI0RERCQqhhEl88c+YLJrmcGSQERERCbBMKLE2d7GKOedtC4MmQWlis/n/hiFxzacQqVcQFpeCVb+eRk3MwqMcm0iIqKmjmFEiaOdccLI1dR8rPsnTvF58JU0nE/OQWxKHub/fBbbTiXisa9PGuXaRERETZ212AVoSozZhbWkXI5XfopClzYta64nAc7fzgEA5JdWGPHqRERETRfDiBJjju6NSsxGYlYRgDTFttX7YyFX6jpyNukeBnVqZbxCEBERNUFspjGRsgp5nW0nb2ShUimN3EhnvxEiIrI8DCNEREQkKoYRJTJrK6OdOy2/VPtB94XFZeCRr08gNiXPaOUhIiJqKhhGlIzp3sZo51ZujtFm5pZIXLidi9nbzhitPERERE0Fw4gSFwfjDO1tqJzicrGLQEREZHQMI0okXCyPiIjI5BhGzMS9wjK89nM0jl5LF7soREREBsUwYiaCDsRi/8VUzNrKfiRERNS8MIyYiZTcEsX/7xWWiVgSIiIiw2IYMRPK/Vk+PnRNxJIQEREZFsNIEyYojQZW7lq7IzIJH+27YvLyEBERGQPDSBNyNSUfgqDbfCSbjidwUjQiImoWuFBeE7LlZAKGeqtfKE/dqOOiMq70S0RE5o81I03Mb2fvKP5fqVRLousMKIIgICW32MClIiIiMh6GkSasrEKOwD0X9XrNij8vwy/oCHZEJhmpVERERIbFMNLEVYcKXWeH/TH8FgDgvwevGq1MREREhsQwUouzvbjr0wRfSVO7Xd+J6nXsB0tERCQ6hpFa/lowUuwiqKWuYqS+wKHrqBwiIiKxMYzU0snVARdXThS7GI3GKEJEROaCYUQNRztxm2rU44rCRETUPDGMNANxafmIiM9S3ciqESIiMhMMI2ZCbZ+R+/9O/DwMT38XgeTsIpV9WQWleOuX84hKzDZJGYmIiBqCYcQMJGcXaRxloywxq1Dx/4LSCrz720X8Gn0b/9oYDgAoKa80WhmJiIgaimHEDIz6+Kja7doGzPwTWxNgjsVloOeyg1h/9IYhi0ZERNRoDCNm7IfwRKz667JOxy7+7QIA4JND14xZJCIiIr1xoTwztu9CithFICIiajTWjGjwyuguYheBiIjIIjCMaBA4pZfYRdCbhHOREBGRGWIYISIiIlExjDQjQj0znbHOhIiImiqGESIiIhIVw0gz8vd5jq4hIiLzwzDSjOyKSha7CERERHprUBhZv349vLy8YGdnB19fX0RGRtZ7/Lp169CjRw/Y29vD09MTb7zxBkpKShpUYGqY3OJysYtARESklt5hZNeuXVi0aBFWrFiBs2fPwsfHBwEBAUhPT1d7/Pbt27F48WKsWLECsbGx2Lx5M3bt2oX33nuv0YUn3RWWcV0aIiJqmvQOI2vXrsXcuXMxa9Ys9O7dGxs3boSDgwO2bNmi9vhTp05hxIgRmDFjBry8vDBx4kQ888wzWmtTiIiIyDLoFUbKysoQHR0Nf3//mhNIpfD390d4eLja1wwfPhzR0dGK8BEfH4/9+/djypQpGq9TWlqKvLw8lQ8iIiJqnvRamyYzMxOVlZVwd3dX2e7u7o6rV6+qfc2MGTOQmZmJkSNHQhAEVFRUYN68efU20wQFBWHVqlX6FI2IiIjMlNFH04SGhmL16tXYsGEDzp49iz179mDfvn344IMPNL4mMDAQubm5io/kZI4SISIiaq70qhlxc3ODlZUV0tLSVLanpaXBw8ND7WuWLVuG559/HnPmzAEA9OvXD4WFhXj55ZexZMkSSKV185BMJoNMJtOnaEbRu50TrqSwiYiIiMiY9KoZsbW1xeDBgxESEqLYJpfLERISAj8/P7WvKSoqqhM4rKysAACCoHn68qbgr3+PFLsIRvHi1kik5nJoNRERNQ16N9MsWrQImzZtwg8//IDY2Fi8+uqrKCwsxKxZswAAM2fORGBgoOL4adOm4ZtvvsHOnTuRkJCA4OBgLFu2DNOmTVOEkqbKSto8V3QJvZaB+dvP6vWaS3dykVNUZqQSERGRJdOrmQYApk+fjoyMDCxfvhypqakYMGAADh48qOjUmpSUpFITsnTpUkgkEixduhR37txBmzZtMG3aNHz00UeG+ypIb9G37in+L5cLOHI1HT6eLmjjWLd5LDIhG099G44Wtla4/P4kUxaTiIgsgERo6m0lAPLy8uDs7Izc3Fw4OTmZ9Npei/eZ9HqmlLhmKgBgR2QSAvdchJOdNS6sDKhzXNCBWHx7LF7lNURERNro+vzm2jSEkNiqDsl5JRUil4SIiCwRw4gWTnZ6t2QRERGRHhhGtDjy1kN4+kFPsYthNHkl5Wj6DXVERNScMYxo4dZShpHd3MQuhlFE38pG/5WHEXJV/SKHREREpsAwYsE2HL1ZZ9vfF+6KUBIiIrJkDCM6sKRmjO/C4nH4cipe3BqJjPxSAIAEzXO+FSIiahoYRkjFhdu5ePmnaIRey8Dq/bFiF4eIiCwAw4gOBnVuJXYRRJFVyBlXiYjI+BhGdNDBxR77/tP81qkJjcuod39xGecdISIi42MY0VFn1xZiF8HgKuX1d4Y5k3gPHx+8aqLSEBGRpWIYoXptCK074oaIiMiQGEZ01EwX8NXJzYwCsYtARETNGMOIjhxsLXda+OAraYr/y+UC9l1IwZ2cYhFLREREzYnlPmGpQX6JTsa7v10EwBV8iYjIMFgzQno5cSNL7CIQEVEzwzBCREREomIYISIiIlExjBAREZGoGEZIL3+d56q+RERkWAwjZHYi4rOQnl8idjGIiMhAOLSXzMqxuAy8sCUSUgkQH8ShxUREzQFrRsio7uYUQxDqXwNHH8fvL+6nZVkdIiIyIwwj1GA30vMREZ+Fikq52v3fhN7E8DVH8NnhOBOXjIiIzAnDCDXYi1vP4OnvIvBlyHW1+/97f8Xfr4/eMGWxiIjIzDCMUIPdvle1Ps2PEbdMdk2JBS9YSETUXDGMEBERkagYRvTg09FZ7CI0STlF5Sa7lkSpaiQkNo1DfImImgGGET389upwRC/1F7sYTdKpm5mK/++7kIJ/fXPK6Nec/UMUxn4SavTrEBGRcTGM6MHaSgrXljKxi9Ekzdh0GmH3h93O334WUbfuqezPyC81ynULyyqNcl4iIjIdhpFGspayR2W1k0q1I7U9+NE/mLEpArnFqk0619Py8VPELVTqOHEI7zYRUfPDMNJIW158UOwiNB1a8sSpm1nYEKo6zHfC52FYtvcStkcmGbFgRETUlDGMNBInAq2RkFmo9ZiCkgq1288n5xi4NOK6m1OMUzc01xQREVENhhEymMNX0lBYqj5sVGt0eDOTdprha45gxvencTo+S+yiEBE1eQwjjWTIdVeag8wC3Tuqnkuq6eTaXG9j7Y68RERUF8MIGZS2UHGvsEzx/8c36D/8V2IuVSNERKQzhpEG6O7eEgDQwcVe5JI0Pf/aWH/AOHApFcnZRQ0+/5YTCQ1+LRERNU0MIw2wddZQvDjcCzvmDmMH1loyC8q0HnPocmqdbYIOdzI5uwhlGlYIJiIi88Uw0gAdXOyx8pE+6OTqIHZRLErtOUqIiKh5YBghIiIiUTGMNBbbaageEva3JSLSimGETO5sUt3hriXllSgpr0REfBa+Px5v8CHTYg3B5ugfIiLtrMUuAFme/RfrdmDdfzEV/1w5rOig+tH+WIQvHg8PZ7tGXy/oQCz+Pp+Cv/89Eq1a2Db6fEREZFisGaEmQ3mkjCAAS/debNB54tLyFSsIA8C3x+JxJ6cY204lNraIRERkBAwjjdS3g7PYRWi20vNrZnMtKa+sd1hvQmYhtp5MQEl5JSZ+HoaZWyJxNTVP5RgxGmrYZ4SISDs20zRSG0cZwgPHwcHGGv/aeApWUgmupuaLXawmb8/Z21qPuXA7F4Ig4KeIW1j+x+V6jx37aSgAIFtphtcb6QXo6eFUc1BznXOeiMjMsWbEANo528PZwQaHXh+N/f8ZJXZxzMKi3ed1Ou7vCylag4iyM4nZGvcdVDPZmrGxYoSISDuGEQOSSiWQSvn4MSR1s7XWR7nyo/ZIlri0AkMUSS9iNNP8EXMHk9aFISGz0PQXJyJqAIYRE3FrKRO7CGYpv6RC6zH/i7hlgpKYj4U7Y3A1NR+Lf7sgdlGIiHTCMGIiZ5aMF7sIZumY0qgYTZbuvaT4/+mEmmYaY9RK3M0pRn6JeUxLX1RWKXYRiIh00qAwsn79enh5ecHOzg6+vr6IjIys9/icnBzMnz8f7dq1g0wmQ/fu3bF///4GFdhcSTiswuSyCsuQU6R94T5dpeQWY/iaIxjwfrDOr+GkZ0RE2ukdRnbt2oVFixZhxYoVOHv2LHx8fBAQEID09HS1x5eVlWHChAlITEzEr7/+imvXrmHTpk3o0KFDowtPVJ9ley/VCQ6X7+aiooEr/0YlVs0cWyk331E5pRWVOJd0D3Iz/hqIqPnRO4ysXbsWc+fOxaxZs9C7d29s3LgRDg4O2LJli9rjt2zZguzsbOzduxcjRoyAl5cXxowZAx8fn0YXnkhfU788ofNIHnMnqJlZZcH2c3h8wylsCL0hQomIiNTTK4yUlZUhOjoa/v7+NSeQSuHv74/w8HC1r/nzzz/h5+eH+fPnw93dHX379sXq1atRWam5Pbu0tBR5eXkqH+bElVOON2l/nr+r03G117NpSF2CRALI5QLS80sa8GrDC76SBgDYcjJR3IIQESnRK4xkZmaisrIS7u7uKtvd3d2Rmqp+CGZ8fDx+/fVXVFZWYv/+/Vi2bBk+++wzfPjhhxqvExQUBGdnZ8WHp6enPsUU3f/m+GJUNzexi0GNEBGfhUEfBGsMLlkFpUjOLtLpXAt3xWDoRyE4elV9U2ZTcC01H3/pGNIsmVgLLhI1d0YfTSOXy9G2bVt89913GDx4MKZPn44lS5Zg48aNGl8TGBiI3NxcxUdycrKxi2lQvdo54afZvmIXgxph1tYzuFdUjv/sOKd2/+AP/8Goj4/iXqFqB9nYlDxcuJ2jsq36If9N6M0GlcUYD8Da5wxYF4Z/7zinsqYPqcrIL4Xv6hAEHYgVuyhEzY5eYcTNzQ1WVlZIS0tT2Z6WlgYPDw+1r2nXrh26d+8OKysrxbZevXohNTUVZWXqRzrIZDI4OTmpfDQn9jZW+M+4rmIXw6Kl55Ug/GaW4vOyCjlmbIrAp4euAQDkOgaAhKxC5JWUQxAEyOUCJn9xHI98fVLtsZGJ2biRrt/EayeuZ8Jn1WH8fcE0tRZXUsyrSdSUNh2PR3p+Kb49Fi92UYiaHb3CiK2tLQYPHoyQkBDFNrlcjpCQEPj5+al9zYgRI3Djxg3I5TUjGOLi4tCuXTvY2lpW3wpnexsAwN75I7BoYg/senkYdswdJnKpLNPQ1SF4ZlOEoibgwKUUnLqZha+P3sC9wjKd5yj5/ng8+q88jNd3xaBSTYCpPaT7+c2n9Srnc5tPI6+kAgu2q6+hqY+pWhSyC8twJ6fYNBcTEUcgERmP3s00ixYtwqZNm/DDDz8gNjYWr776KgoLCzFr1iwAwMyZMxEYGKg4/tVXX0V2djYWLlyIuLg47Nu3D6tXr8b8+fMN91WYiVOLx+HY2w+hh4cjAMC3iyv8HnAVuVSW7fj1DATuuYAdkUmKbb+dva12fpALyTl1tu2/WNVX6o8Y3WouUnJLsDY4DrujjNP0GBGfpf0gAxv0QTBGrDli0DldiEyJQVN8eq/aO336dGRkZGD58uVITU3FgAEDcPDgQUWn1qSkJEilNRnH09MThw4dwhtvvIH+/fujQ4cOWLhwId59913DfRVmooXMGi1kXCi5KfnrfApS87SPdLl0Jxffn0jQ+/yfHb5WZ9uXIdcBAE8NqeqY/cmhq8jML8OaJ/o1enI8XWasBfQfGXTpTi7+On8XC8Z1haOdjWJ7Sm5NjcjNjEIM7my42s70vBJsj0zC0w92goezncHO21BN+XGVU1SGkzeyML5XW9jZWGl/gYGFxWVg55kkLHu4N9o525v8+o2RkV+KgHVheGxAByyf1lvs4lisBj0ZFyxYgAULFqjdFxoaWmebn58fIiIiGnIpIqNSF0Q+OxyH4nLVoeeRCZpXA66mLkboMiX7+qNVHVtnj/JGd3dHrceL4eGvTgAACkor8NHj/RTbl+3VvKLyz6dv4dDlNGx8bhAcbOv+qfkq5DpcHGzwvJ+X2tfP/TEK52/n4uClVBx8fbTWMqbnl2Dln5fxnG9nDO+q/2i2Y3EZSM4uwnPDOqvd35QH0jy3+TQu3cnDrBFeWDGtj8mvP3NL1Szc+y+mInHNVJNfvzE2n0hAdmEZtpxMYBgREdemaYJm+HaCl6uD2MWwWLWDiCAIyC7U3gTRkJoTZWUV9c8MGxaXoXb2WEEQUFJumnVorqXmq3yeXViq8dglv19CWFwGtp1KrLMvKasInwXHYdkfmsPM+du5AICrta6pyfK9l7H/YipmfK9fv5xqL2yJxNK9l3Dx/nVr07VTszYR8VmY+2MU7uQUo7isEo+uP6m2Bk0fl+5UdTzm8GwyVwwjTZCtlRSDOrcSuxh037u/XcDXR7XPWLrmwNVGXUfbNPMzt0Si65IDdYLHu79dQM9lB3E9LV/l3bsuz871R29gbXCcymvKK+V4adsZrL//Nes+fb76C4bFZdRpky8o1b4as77u5hqmE6262rILt3PUhqqGePq7CARfScObu2Ow59xtnE/OwVdHOCOuWLhsWNPAMNIESSRo2g3UFmZ31G2jnDcuTfUd/5wfo3Ra9+ZMomqTUXX5Fu6MwaHL6icfVOdc0j18cuiaog8LAMRnFGD/xRQcuZqOTw5dQ2JmIXqvOKTYL5FUhZUT1zOx9WQCzibl1Dlv9TDnahHx2fhC6RrqNPXJxAL3XDT4OVNyS1CupTZMf+b3ZP37wl1saWStIpk/hhEj6uDS8I5cMht+a5q7iZ+HqYxAycgvxQPv7cf200n4KTxR7xE3V1LykJBZqHG/ctApLqvE4xtO1Tnml+jbKjUvXx65Xqf5KGj/VTy3+TRW/XWl1qslEAQBT24Mx7SvT6jsqS+MRCVmY+jqEOy7kKLxmNryS8rxR8wdo9SwNIXHeU5RGQ5dTtXadCeWi7dz60z411ALtp/D+39f0XsOHkNpCt9vamAHVtLNuJ5t8VPELb1fJwiAlHWHFqG6rV/Ze78b/l24XC7Af+0xxeeluj7kalVYnEm8hzP3Vy9Wd3BxeSWibmnaX0P5x/ulbWeQV1KB+dvPYmp/3To/vr4zBiFX0zGhtzv++0R//GfHOVzQ0NdDX7EpeRjfqy0kEglupOfDtYXMIOfVxc2MAhyPy8BPEbdwM6MQC8Z2xVsBPXR+vSn+bJxJzMaTG8Nhay1F3IeTkV9SbpDzijU0nH9qmwaGEQB7z91BcnYR/j2+m9hFUWjaldYkpmup+fDr4gprq/prz5RnU80oKK231sQQ/jqfgie+Ub9gZm3KrTJ5JfrXboTcX+cn+Eoa2jhew4kbmXqfQ5PPguPgZG+DEV1d4b82DADQp71pZoEe/9kxlc//PH9XvzBy/9/U3BI42VurHcXUWKHXqu59WYUcKbnF8As6YvBrfHzwKtLySvHpk/0bPdxdG+U5hQRBwKHLqejVzgmdXVsY9bqkim0BAF7fFYPPguPqrCkipibehE4i+nBfLN7+9YJOxz70yVH8cyWtwT9P+rxMnw6ef8Tc0bssQFUNz7K9l1S2aXpHXVpRiaQs3RYzrG3bqUSEx9f0zTHU76Nys4uxHrHJ2UUYFhQC34+qZso+cjUN/95xDrnFhqnBUL4XBy7q3kdJHxtCb+K3s7cRm6LbSCpd5BSVoVxNZ2zlrHP4Shrm/e8sxnwSarDr6iKvpNxgNUzmimFEyb0iw/4wVE9qNZgjY0iDnyISG/S638/p9jBPzCrCnB+jkKTjCsO1xWcYrh0/+EqaoqPqOTUdX7W5cjcPXd7br3PT5/9tOIXRnxzFqZv615oYqkNtWYVcMVX+jsgkdF96QK/XK3/f5HJBawdniQSKrzf/fn+al7ZF4a/zd/G50qipxojPqKlhM3bTSpnOI7mA8JtZ8F39D4KvpNXZdyenGAPeD8bkL47X2accCqN1aGJsjPJKOcLiMlT6OpVXytF/5WH0W3lYj5Fr+ikorUBkQnaTnmmWYcSI+nV0RuSS8dj1MtefIfUOXa77h1NXGfml+L8N6hflq+2pb3VrPqktLs1wYWTuj1H4+0IKohKzcemufv07lvx+EVO+rPsgqc/lu1XNVL9FVwU3fQKGPn+yb6QX4LCGUUxPfHMKI9YcwZnE7EaNyBEEAVO+PI6HPj1a54F1+15NYCkqrURZpfrSp+fXDFk+eCkFj3x9Aol6Nt1FxGfhoNLX+qUBhyQ3tjXm2e8jkJZXirk/RtXZF3y/zGo7yWq4cKVcQOCei/g12nCj6b745zpmbonE7G1nFNuUa6z0bbIsrajElC+OI3BP/TWl078Nx1PfhuNnpWUvmhqGESXGqDZt62intW1fvaabYKlpePCjf9QOrTWk2hPANda+Cyn418ZwnWamVfbz6Yb/ERUgIKeoDKM+PoqgA7E6vab2BGeaVjMuKquA/9pjePmnaLXrAl28UxW6ftPwQNO1P0RphRxXU/ORnF2Muzk1oSK3uBxX7taULb+0ok4zlmKf0oNu3v/O4sLtXLz1y3nFNl3CmqlWj9amoLQCm08kqCzQqOub/qwCzRP1Kdt/MQU7IpNU7lFDrA2Ow0/hiQCAnWeqfo5Pa5jRWd8auSOx6biSkocdkTUj727fK8L6ozeQq1TTXx3M95w1zjQFhsAwooSPf6Ka6emNQfnduaEUlNYfbPacvYOPD13D7XvF+PZYvGJ7SXmlxj/+ujwT0vNK0Ht5zRwsl+7kIqugFEEHYuu0//8Tm17n9YlZRVi9X7dwpE58RgF8Vh3Gyz9F63T88euZdfqN5N0v54nrmXjwoxD8o6aJoyn64K8r+ODvK5j21QntB0M19NVuIpRo+L8uTVAXb+fW+zN9Oj4LX4Zcr3emYeVr6vsMUhfAHt9wCp8cuobA3+vWljTlvogMI01QG8f6hxLaWHEsGhlPQ/uX6EKXDomn47NUFuDTJkzN4oB/1poWfXutmpWU3GL0XHYQr/18Vu05dfmjvWDHOZXPJRIJBn/4D749Fo9+Kw+r7MvU8G5c5yHWKtep+veXBjQfTFoXhv8erDtT8HObTyOzoBRz1DRxqFzbxLNyqAuL9wrLsOv+HDzVyzR8WWseG3UdVTXRVDml7Ufg8t1cTPv6BIZ+FILraep/rjX9fAFASGwafFf/g4h47eteKcstLscHf1/BU9+Go0Je83VWB8mM/KqftVM369bUNeEsYtlh5MrdPGwIrWnzbAqP+McGtMfskd7w6eii8ZieHqYZZkhkaNqafaJvZWP6dxGNHi76n1pBobad96u1D1yq6ksQq6EZpj4x9bzDBgzXCTY9v0QlHG07lYjvj8c3aNKxlNwSfBNaU/OlbxENPcpW00O82uMbTuHxDSdxK6umb4vv6pA6x62t1Tm325ID+PZYzddZX7lVApaG407dyKzT3+S0UoiY8HmYSt+daln1fI9m/xCFtLxSzN9eE1i0fT++Px4Pn1WHsflEAiITshF6rSaIz/kxCueStHTAFQQUlRl+okBDsOgwMuXL4/j4YOMWqDK0dU8PhJ2NFZ4c4omg/+uHfxaprlb62ID22PDsILw0wlukEhIZT2SCcUczAMCcH6Lq9AmpPcriTk4xSvTs15Jc62Gk78gZTUNvn/42AoLSe9rNJxLw4b5Y7DyjfYbevVpGXV1PL8CNdN2HzzY0i/wv4pZiNE9uUdUw1hvp+ZjweZjKcb9F38blWp2bzyXl4IlvwpGUVYScojKdR9gEaVgrqvbzXlNQqVDqCDzj+9MqkwaqM/K/RwFUNX3VXupB6Wr1nkNbgP1wn2qzXu1RdepmVVZ2/nYuei8/hF+jbxtl9uLG4KRnTZSVVIJnhnZS2WZrLcW6pwcCAJZM7YWJfdzRt4Mz+q081KTbAol0JTVB9eQ/sWkqqw3fzVHfJPTxIc0LH67883Kdh+LWk4kqn5drGNWiic+qw2q3x2cWNniq9Nd3xWg9pnpit9rkcgHSWt8QXTrcZheWwcXeBmWVctjZWAEAlt7vVDuhtzse1tDPI/RahsYFAzMLSjH6k6Nar11bcVkl7G2tNO4vKa+sU6tS7f2/ay93UCOvpBw/n1Y/xHzEmiPIL6nA3vkj9CssVINSel4JziTeg98Drvg54hYeHdBBr3PV951665fz+OzwNYQHjse11HycSczGjKGd6ny/Tcmia0Zqi0vLx+s7zxl9pkpNtPUFUd5rJZVgWBdXtJQxT1LzYaqpuZVHIQ1fo75JqL4wYagVfHX1yNe6DeE2lG0nE9B35SGcT87R+7XX0/Lx8Fcn0HPZQfxQ6z7VF6oaOsGZ1+J9Gvf1Wn6w3hqAzbUW6NOlT8ytrEL0X3kYNzPUPyeqRy0duVq307I2C7afxaX7I7AmrgvD/O1nMeiDYHwWHIepeg5tv1dUXu/XnpJb1fE2YF0Ylu69hC7v7TfaPCe6YBhR8uG+WOyNuYsXtkSKcv3nh3nVu59rKFBzt3q/5toIMp2Vf11BUVkl3lGa6TeroFQxVLk+r++KUQyFXvHnZY2dd2uLuqVfR05dXUjOUYkYy/+4pGia0veN55u7z+O3s7pNOKhc+1ZNWzPMmcR7ipqjnFqTcOY3oFllt5amvIe/Ug046kZ9mQrfVqthzNEEDdHWUYb0/FL4dXFVu1+Cpt1LmojMR+0HZkxyDj4+eFXt6Ax1qt9xVytSGnpdX9NH7YevoZRVylXeyaXklsB/bRgS10yFQz1NOOr8psc8Hf+LUB3BFZuSV2+HVmWn1cxZ0xC1+0bVVnuhTn1GIRkaw4gZ+O3V4fj93B08P6yz2v0SiaRpDyAnIrPhHbhf8f9rafl4bH3jmoiOXqt5t53dgBFAjbXlZCIm9HZXu89aatjGgfpmtFU3Fb0m07+LMERxcDU1H8V6dsQWC5tp6nHociqGB4WIPhTKs7UD/jO+G1q1sBW1HERE+todpX3UjzHFZxQgKlF9E1Bqnmrn5V8aWdaHPg1t1OsN7dfo29iuxxTwYnYFYBjRoLxSjld+isbdXNVZFpsidiUhoqbq8l3953AxpNv3ivFHTN1p7O/mFGN/rVWHdW1GMSfr/jHMAonGxjCiQbcl+s0R0BidWjsAAKb29zDZNYmILNlz358Wuwgmka/n4ntiYZ+RJuDwG6ORmlsCL7cWDXo9R9kQEeknXqQpHEg91ow0AXY2Vg0OIrXVniiNiIioqWMYaWYWT+4pdhGIiIj0wjDSDCjPGuhsbyNiSYiIyFyZelVmZQwjREREJCqGkeaAHViJiMiMMYzo6OClVO0HERERkd4YRnQUEpsmdhE0+n7mEFhLJfjvE/3ELgoREZHeOM9IMzC6extc/WASrK2YLYmIqGEq5OItlMenVzOhbxBp6yjTuO/GR5MbWxwiIjIz19MKRLs2w4iOrqeL900yBuU1fif3VZ2GnjUsRERkSnzq6CgmOUfsIhjNV88MxODOrcQuBhERWSiGEQszunsbAMBzvp0V26ytpHBraStWkYiIyMKxA6uF2fbig8gpLkdM8j2V7e9N6YWY5BzMGdkFAPDUkI7YHXVbjCISEZEIKgVB+0FGwpqRZu675wdjVDc3xedSqQStW9StBens2gIRgeMxd3RVGFn6cG+TlZGIiCwba0aauYl9PDCxjweOXk2Hg61VvcdKJDVTubaw5Y8GERGZBp84FmJsz7YNfu1fC0Yi5nYOlu29ZMASERERVWEY0cN5PUfUSIy4ZoyuqytevJ2rYXue2mOqy1wpr2k7TMwqRHJ2UQNKSUREpJ1EEETssaKjvLw8ODs7Izc3F05OTgY7r9fifQY7FxERkTl7Zqgngv6vv0HPqevzmzUjeujgYi92ERTqy5B3c0sU/2/vbKfXMZrOmqJ0PBERNT+JmeLVgDOM6OHk4nFiF0Ens7edQcjVdAT0cce3zw9Re0xIbBpm/xAFADgVOF7rOQVBwOr9sbh8Nw+nbmYZtLxERGTZGEaaoXVPD8CRq+kY38td4zH6Ns5JJBIsmdob55Nz8Oj6k40sIRERUQ3OM9IMOdrZ4NEBHdBSZvisqS3DLJ7c0+DXJCKi5o1hhAxq3pgHxC4CERE1gKOdeI0lDCMWyhg/dAF91DcLffnMQINfi4iIDMvLrYVo12YYsVBDvVvjBb/OeP/RPnq9rr5RPBufG6x2+yM+7fW6BhERmZ7MWrxIwA6sFkoikWDVo30Nfk4iIjJPYv4Nb1AMWr9+Pby8vGBnZwdfX19ERkbq9LqdO3dCIpHgsccea8hliYiIyEjEfDupdxjZtWsXFi1ahBUrVuDs2bPw8fFBQEAA0tPT631dYmIi3nrrLYwaNarBhSXxaWqkGdeItW+IiMiy6R1G1q5di7lz52LWrFno3bs3Nm7cCAcHB2zZskXjayorK/Hss89i1apV6NKlS6MKTE3T5hfUT662bdaDarevfcrHmMUhIiI9BfTxEO3aeoWRsrIyREdHw9/fv+YEUin8/f0RHh6u8XXvv/8+2rZti9mzZze8pCJ7w7+72EVo0jS1NT7UQ32NyfAH3IxZHCIi0lPrFraiXVuvMJKZmYnKykq4u6sO4XR3d0dqaqra15w4cQKbN2/Gpk2bdL5OaWkp8vLyVD7E1ru94RboM2ddlIZ+dXZ1AAD0aqf/vRE0NPjY21jhv0/0a1jhiIiowcwmjOgrPz8fzz//PDZt2gQ3N93fCQcFBcHZ2Vnx4enpacRS6objRKq4ONji+DtjcWaJP36e44tXRnfBlhfVN9Fo8qBXK3g4qV/ADwDcWsrqbPtKz7lK7G2s9DqeNJskYtUtEZmOrYhDe/W6spubG6ysrJCWlqayPS0tDR4edf9g3bx5E4mJiZg2bRqsra1hbW2NH3/8EX/++Sesra1x8+ZNtdcJDAxEbm6u4iM5OVmfYhoFR63W8GztgDaOMnRs5YDAKb3Qzlm/1Yx3v+IHiUSCn2YPhZerA3a+PEyxT4CAUd3aoKeHI54Y1BH7/jMSN1dPwTSf9pjQW/NaO7XJ65kPZViX1nhlTBeM6Oqq9TyJa6bWu/+BNuJNEmQKS6f24qR1RGR0eoURW1tbDB48GCEhIYptcrkcISEh8PPzq3N8z549cfHiRcTExCg+HnnkEYwdOxYxMTEaazxkMhmcnJxUPsQ2qFMrsYvQbFT3LxnVrQ1C3x6LYV1UQ4GttRQHFo7CZ0/5oE97Z1hJq45/YlBHxTHPDK362RnWpbXaa6gLI2O6t0HimqnY+bIfAif3ws9zhql5pX683Vo2+hxN2awR3gZ5t6SutksMc0Z6AwDGc/SXiu1zfcUuAlk4vf/KLFq0CJs2bcIPP/yA2NhYvPrqqygsLMSsWbMAADNnzkRgYCAAwM7ODn379lX5cHFxgaOjI/r27QtbW/Hap/RxfvlEtBKxLc2SVGcIbZPvBP1ffyQETcHOl+uGYACQq6kYqW+Rv74dGhZ4DV1jNqpbTXPmnwtG4OyyCTi3bAJcHGy0vva3V4cbtCxje7RRBMHGGt29aXRYHnU/kG5+8UEsHN+tUedaMLYreno4GqhkujPGjMaaOpRXhzdL1b+jMy6vChC7GBZB7zAyffp0fPrpp1i+fDkGDBiAmJgYHDx4UNGpNSkpCSkpKQYvqJicdXgQUONUP2zrq4GqvQqxpsAyw7cT7HR8N794ck/4eLqoDTW+3lW1Lm9OqBpJNdOvc51jWtXzs/HWxO6IXz1FZdt7U+pf1fi/T/TH2B5t8NPsoejf0QWtW9iiVQtbRC+dgI//1R9DvdTXBAHA4M6t8Nurfnj6wcb1sXpiUEfELJ+AzS+oH5atr+UP98b7Bp7ttyE6tXbAaKWw5+Gsud8SAPz2qh/emdRD4/63Anrg4Ouj62yfN+YB7H7FD7/OUx+U1VEOocp6q+kc/vigDjqftzG8XB0MFrY1fX26+vRJH3jrsG5KQ2vx1P1uA8CjAzqghY6rn9f3u2lsgzq5qN0+1Fu8MumrQd+5BQsW4NatWygtLcXp06fh61tTxRcaGopt27ZpfO22bduwd+/ehlyWzMT3M4dAZi3F1zNq+ho83L9dva/5/bUReGV0F3zxzACNx4zo6oqnhnTEsod7q93fqbUDEtdMxerH++HH2UPRqbWD1s6188Y8gD/mj6gTdADgP/ffOS8Y1xX/LBqDldPqruMzf2xXPDpA/TtViUQCqVSi0jfl5dH1r2rc3sUeW2cNxahubVS2W0kleGqIJ3bP88OV9zW/UxvcuTXWPNG/3mvUtnRqL6yfMQjBb4zGymm98dHjfeHiYAupmlqRsT3aIPb9SZjh20nj+WaP9MaKaTXfo5dGequ9v9V8Ojqr3V6703I7pfCg6Z5rcmrxOIS9M1YlwNb3nL20KgCDO7fGaw91VWyrDqXaPDmkI4Z6t8aQeh5OtlZS/DzHF4lrpiJxzVRM1NBJWKy+alFL/RG8aIzOx3/4WF8krpmqsXZu8wsP4uDruk94+epDNb8nnq3t8a/BVf3HantljOq8VV3cWiA8cBxeGqF7jc7WWQ9iSj/Vv0/LH+6NWSO88OJwL53P8/WzA3Hs7Yfw0eOGC9493LXXvK2bPgB7XhtRp3+bT0dnrJs+AO2c7fB2QA988fQAAMCa/+uHyX3r/rw9N0zz77QpcG0aMjj/3u648v4klSr+5dN6o7C0As/6qn8H4u3WAoFTetV7XolEgo//VXeytJ0vD8PnwXH44LGaPwKDO7dG2DtjVY6rb5E/ADj8xmhEJmRj0/F43Moqgo+ni+K6XdtW9Q2ZP/YBrD9a0/G6s2sLfP7UAPTv6IIHvVrhVlYR/r3jHADAwbZqRM/qx/vh5R+j6/zhVPbd84PR6f5QaW0cbGt+bft1cMbTQz3Rr4P6BzpQ9e66ZztH7Dl7R2X7Dy8NhbVUAr8urorg0U3DH7/PnvTBj+GJWP/sINjbWmH14/2w/XSS2mNfHt0Fri1sERaXobiH6myaOQSZBaWYPsQTXd7br9h+dtkEZOSXooeHo+Je9u/oDJm1FCm5JRrPVx91Qxbre8erLjwN7+qKf4+filM3M9HBpabTtmsLW2QVlmFkVze8O6knHmhT04/oQa9WOJN4r865lkzthRFdlWoLNPxsqlu4rLsODyhdjezqhieHdKyzvbqPT7e29V9r9eP9EBaXoTiHpnfottZS9PTQ3hTq1tIWb0zojorKmvshuR8bHWytEffhZHRfekCxz6ej6vUEAWjnbI//G9QBW04moLOrA3a/4oc2LWUqP2PVnhrSEWN7tEVEfJbK9pc0NE/17eCES3c0TDUhVP096NTaATsik1SOc2spg09HZ4RcVT9T+T+LxsB/7TEAwPF3xmLUx0fVX0NP7V3scWrxOEUIn9TXAzJrKxyLy1Ac8/6jfeDl2gK+GvrfmQrDyH1DvVojMjEbzwzthB2R6v/Iku5q9zVo62iHrbOGGuVaw7q4YtcruleJa9Ld3RHd3R3x9IOeqJALsFMzPPjtgJ4oKZdj84kExZwrUqkEs+//8erf0QUZ+aU4cjUdzwyteqfR2bUFDr1RtzpfmaZ3xrrQFPCq7V84CvsupCjCSDtnO6Tnl8LXu7Xar1GdJwZ3xBOD6z60AGDFtN5Y9dcVxefu94dt1/5++/dyx82MAiRkFgIAuru3VIyQ8u/VFv/EVv2hbt3CVhEeQt4cg91RyXh9fHcs3XtJ7YP9m2cHIeRqOuIzCnA2KQdSSdXPX3ll/eGzIfPjAHX7V4S+/RDu5pSgh5r+I51dWyjK/MqYLvj2WDyAur8f6ko6/AFXlFbIFZ+fXDwOuUXlKkFIHV/v1jidkK1236BOLjiblKP4/H9zamq1L68KwJwforBoYk0N0BODOyKrsAyuLWzxzm8X6pxvhm8nlVoybX29DiwchclfHNe4/+TicZBZWyG/pBwr/rwMACpB29Zaij8XjMD0byPw5sTuGNezLbzdWih+pqrnL+rbwRkn3h0Lt5ayen/GJQactKG6OV8ikSDo8f6Y9vUJxb7T742HVALczCjAJ4eu4dDlmhGpTnbW6Niq5nta/SYGALp7OOKTJ/vjka9Pqlwr9K2H8NCnoXXKcGHlRPRfeRj3C6IoTzWZdd17MamPB9rWM9WCqYg3qLiJ2T3PDzdXT0HQ/3HCLUtnbSWt9w/Y2wE9sG76APyioU/ASyO98b85vhrPceTNMXi4f7sGPwyVaZo87v8GVvUrGNK5bh+csHfG4vKqAJ2DiDZd27bEK6Oran3qq6HZNHMwDixUX1X/3yf6Y85IbwTXCm0PtGmJwMm9YG9rhXcn9UDfDk5Y/bjq7+jkfu3w6ZM+2PjcYDw/rDMOLByt00OmurarsRztbNQGEUC1D8PbE2v6n9T+rsnV9bhGTWfVB9q0QAcX+zqTL7ZQenAdf2cs4j6cjJ0vD8PVDyYptru1rAp2zw3rhD2vjdDYf6OFzBo7Xh6GB5Wal6ykErz60AOY3M8wc830aueEo289BJm1FLNGeKk8eIGah6WjnQ0Ovj4Kc0d548PHVJs9+nd0waVVAZgzqgvsbKxw5M2a5iTlCqaOrRxUfsaXTOmFcT3bqsxxpGszWHWz0ZIpqk3Eri1s8f3MIYhcMl7lQS+zUX20Wkkl92tYHTG6u2oT7NuTesLOxgofPNYXyx/uDdeWMvz975F4cbgX3n+kD/or1f5IJcAHj/WFl4b+M052+vdvbApBBLDwmpEOLva4k1Os+NxQIweoaamuWv2Xhnf2+rKzscJjAxveibBLm5b4esYgrN4fi9gU48wu/OHjfTG6exuMvT8dv/LPto2VFIaeE+7NiT0wsFMr+HXRPHeLRCKBrVXNH2l3pT+Cri1lWKqhL1C1tk52+PvfVWEm6lbdd/5tnewUTXUP9WiDw1dq3n1aa/jd7uLWAvH331Vro6WVT63Xx3fDieuZeHqoJ6yt9Hvv19m1BZ4b1hld27ZEXw0hr7uHI87dr+lo4yhThB87Gyt88fQAbD2ZiA3PDoKzvY2iWaohX4ejnQ12vTwMv529jciEbCRmFWnsUPrJv/pjb8wdXEjORX5pRZ393m4tcHlVAKytpLhXWIa9MXfVnqenhxOWTFX/M6H886zrsvdzR3fB3NFdMOrjI0qvrfpX+Wfx99fq9nt5d1JP/GdcN9grhScHWytELfVXe/1u9QRdF/uaJsMT745Fx1ZVzbPPD6up4ezbwVnt93zbrKF1wkxtY7q3wbG4DLw4XHONaQ8PRxy4pH7WdLFYdBjR1oeAmodf5w3HzYwCtSMTxPTvcV2RWVCKaUYYqulga60SmMb1bIvBnVvVaWM3FFtrKSap6RRXm1QqwYWVEyHX0Aymq7cm9sDp+Gw8N0z9H9xP/uWDIVFJ6NbWER1b2WsMAr3aOekcRhqirZNdnb5LQN2+IMp/iX6Z54c9Z+9g8eSesJJKVPuW6OHRAR3w6ADDjbzx7eIK3y6uyCwoxc8RSWr7mgDAk0M88eQQTwwPClEbRgAovh/KD3JdJiHURpe/6ON7umPbqUQAQO/2VQ98b7cWWDd9AFxb2mKghhF99rVqcQRBcxCSSCRYN30AXt8VU2ffpL4eeGpIRwzu3EoRRAzp+xeG4FZWUb01f/PGPAC5UNU82lRYdBixs9X+x3C2hY+zbw7sbKzQp73m5gOxONrZYO1TAxr02up39A/31y3I2FpLDT4PSUM1pCq5tvYu9ji5eJzG/c4ONlpHLgFVo0Aq5HKVNnxjejugB8JvZtUZDaTcSvOgV2uVphJDm+bTDiduZMJLxw7Ttbm1lGGhv/Y5WjS0PKlQHiG19cXG9ynT5Q3mu5N6QhAE2Nta4xmlYfCNqe1U5xGf9igorcDAWp16raTqO+LrQt3kgbWb7myspFqbIO1srLBIx9FhpmLRYWT9jEGYv/0s3pygfi6BTq0dsETLCA8iMex5bTjOJeU0ev4GS9eqhS2+fX4ISsor8fzm02o7yFbTNi+JLuaP7Yr5Y7tqP9CInhzsic6uLQzSZ6k+jw3sgI3Hbtbbj2j+2K5Izy/F1H7tDDLTry41I/a2VlhlgnlvpFKJxpo7fX0/cwju5harBI9jbz+EtLxSg46uEpNFh5Fe7Zxw5M2HNO7v6eGodq4FIrG5ONhiLKc0N5j6moz2vDYcOUVlRqlSr/ZQjzb44G/9V01tSEuzVCqpswSDMSya0B2DOrnAt55rtZBZ49MnG1ZLIJYXh3th26lELJ5c/wSGhuSvZl2uzq4t0Nm1+ayNZdFhhIhIG1OsS/VAm5Y4uXhcvTP6qjPUuzViknOMU6hGsrWWNmrIuj6m9muHfRdTMG+M9qa5xloxrTfmjPI2aji1RAwj9agv0RMRGZK2+UOUHXv7IZxJvIchnVvhu7CquUssuT/+l88MxDuTepikpkAikTCIGAHDiBrH3xmL0wnZeEzPKaeJLFE7Z90fok3Zc8M640ziPcWaRE1ZdRV9cnaR2EVpEqykkmbVZGGJGEbU8GztAM/WTL5E9dn18jCk55cabPIwsT3i0x692zmZ7UNN0wR4ROaAYYSIGqS5NWNKJBKNa/M0VWItpEdkaJwOnoiIiETFMEJEZKYclSaQs5byzzmZLzbTEBGZKWd7G2x5cQhsrKQGmTSMSCwMI0REZmxcz7oTYhGZG0ZpIiIiEhXDCBEREYmKYYSIiIhExTBCREREomIYISIiIlExjBAREZGoGEaIiIhIVAwjREREJCqGESIiIhIVwwgRERGJimGEiIiIRMUwQkRERKJiGCEiIiJRmcWqvYIgAADy8vJELgkRERHpqvq5Xf0c18Qswkh+fj4AwNPTU+SSEBERkb7y8/Ph7Oyscb9E0BZXmgC5XI67d+/C0dEREonEYOfNy8uDp6cnkpOT4eTkZLDzkireZ9PhvTYN3mfT4H02DWPeZ0EQkJ+fj/bt20Mq1dwzxCxqRqRSKTp27Gi08zs5OfEH3QR4n02H99o0eJ9Ng/fZNIx1n+urEanGDqxEREQkKoYRIiIiEpVFhxGZTIYVK1ZAJpOJXZRmjffZdHivTYP32TR4n02jKdxns+jASkRERM2XRdeMEBERkfgYRoiIiEhUDCNEREQkKoYRIiIiEpVFh5H169fDy8sLdnZ28PX1RWRkpNhFajLCwsIwbdo0tG/fHhKJBHv37lXZLwgCli9fjnbt2sHe3h7+/v64fv26yjHZ2dl49tln4eTkBBcXF8yePRsFBQUqx1y4cAGjRo2CnZ0dPD098fHHH9cpyy+//IKePXvCzs4O/fr1w/79+w3+9YolKCgIDz74IBwdHdG2bVs89thjuHbtmsoxJSUlmD9/PlxdXdGyZUs88cQTSEtLUzkmKSkJU6dOhYODA9q2bYu3334bFRUVKseEhoZi0KBBkMlk6Nq1K7Zt21anPM31d+Kbb75B//79FZM6+fn54cCBA4r9vMfGsWbNGkgkErz++uuKbbzXjbdy5UpIJBKVj549eyr2m+U9FizUzp07BVtbW2HLli3C5cuXhblz5wouLi5CWlqa2EVrEvbv3y8sWbJE2LNnjwBA+P3331X2r1mzRnB2dhb27t0rnD9/XnjkkUcEb29vobi4WHHMpEmTBB8fHyEiIkI4fvy40LVrV+GZZ55R7M/NzRXc3d2FZ599Vrh06ZKwY8cOwd7eXvj2228Vx5w8eVKwsrISPv74Y+HKlSvC0qVLBRsbG+HixYtGvwemEBAQIGzdulW4dOmSEBMTI0yZMkXo1KmTUFBQoDhm3rx5gqenpxASEiJERUUJw4YNE4YPH67YX1FRIfTt21fw9/cXzp07J+zfv19wc3MTAgMDFcfEx8cLDg4OwqJFi4QrV64IX331lWBlZSUcPHhQcUxz/p34888/hX379glxcXHCtWvXhPfee0+wsbERLl26JAgC77ExREZGCl5eXkL//v2FhQsXKrbzXjfeihUrhD59+ggpKSmKj4yMDMV+c7zHFhtGhg4dKsyfP1/xeWVlpdC+fXshKChIxFI1TbXDiFwuFzw8PIRPPvlEsS0nJ0eQyWTCjh07BEEQhCtXrggAhDNnziiOOXDggCCRSIQ7d+4IgiAIGzZsEFq1aiWUlpYqjnn33XeFHj16KD5/6qmnhKlTp6qUx9fXV3jllVcM+jU2Fenp6QIA4dixY4IgVN1XGxsb4ZdfflEcExsbKwAQwsPDBUGoCo5SqVRITU1VHPPNN98ITk5Oinv7zjvvCH369FG51vTp04WAgADF55b2O9GqVSvh+++/5z02gvz8fKFbt25CcHCwMGbMGEUY4b02jBUrVgg+Pj5q95nrPbbIZpqysjJER0fD399fsU0qlcLf3x/h4eEilsw8JCQkIDU1VeX+OTs7w9fXV3H/wsPD4eLigiFDhiiO8ff3h1QqxenTpxXHjB49Gra2topjAgICcO3aNdy7d09xjPJ1qo9prt+n3NxcAEDr1q0BANHR0SgvL1e5Bz179kSnTp1U7nW/fv3g7u6uOCYgIAB5eXm4fPmy4pj67qMl/U5UVlZi586dKCwshJ+fH++xEcyfPx9Tp06tcz94rw3n+vXraN++Pbp06YJnn30WSUlJAMz3HltkGMnMzERlZaXKNwIA3N3dkZqaKlKpzEf1Parv/qWmpqJt27Yq+62trdG6dWuVY9SdQ/kamo5pjt8nuVyO119/HSNGjEDfvn0BVH39tra2cHFxUTm29r1u6H3My8tDcXGxRfxOXLx4ES1btoRMJsO8efPw+++/o3fv3rzHBrZz506cPXsWQUFBdfbxXhuGr68vtm3bhoMHD+Kbb75BQkICRo0ahfz8fLO9x2axai+RJZg/fz4uXbqEEydOiF2UZqlHjx6IiYlBbm4ufv31V7zwwgs4duyY2MVqVpKTk7Fw4UIEBwfDzs5O7OI0W5MnT1b8v3///vD19UXnzp2xe/du2Nvbi1iyhrPImhE3NzdYWVnV6V2clpYGDw8PkUplPqrvUX33z8PDA+np6Sr7KyoqkJ2drXKMunMoX0PTMc3t+7RgwQL8/fffOHr0KDp27KjY7uHhgbKyMuTk5KgcX/teN/Q+Ojk5wd7e3iJ+J2xtbdG1a1cMHjwYQUFB8PHxwRdffMF7bEDR0dFIT0/HoEGDYG1tDWtraxw7dgxffvklrK2t4e7uznttBC4uLujevTtu3Lhhtj/PFhlGbG1tMXjwYISEhCi2yeVyhISEwM/PT8SSmQdvb294eHio3L+8vDycPn1acf/8/PyQk5OD6OhoxTFHjhyBXC6Hr6+v4piwsDCUl5crjgkODkaPHj3QqlUrxTHK16k+prl8nwRBwIIFC/D777/jyJEj8Pb2Vtk/ePBg2NjYqNyDa9euISkpSeVeX7x4USX8BQcHw8nJCb1791YcU999tMTfCblcjtLSUt5jAxo/fjwuXryImJgYxceQIUPw7LPPKv7Pe214BQUFuHnzJtq1a2e+P896d3ltJnbu3CnIZDJh27ZtwpUrV4SXX35ZcHFxUeldbMny8/OFc+fOCefOnRMACGvXrhXOnTsn3Lp1SxCEqqG9Li4uwh9//CFcuHBBePTRR9UO7R04cKBw+vRp4cSJE0K3bt1Uhvbm5OQI7u7uwvPPPy9cunRJ2Llzp+Dg4FBnaK+1tbXw6aefCrGxscKKFSua1dDeV199VXB2dhZCQ0NVhukVFRUpjpk3b57QqVMn4ciRI0JUVJTg5+cn+Pn5KfZXD9ObOHGiEBMTIxw8eFBo06aN2mF6b7/9thAbGyusX79e7TC95vo7sXjxYuHYsWNCQkKCcOHCBWHx4sWCRCIRDh8+LAgC77ExKY+mEQTea0N48803hdDQUCEhIUE4efKk4O/vL7i5uQnp6emCIJjnPbbYMCIIgvDVV18JnTp1EmxtbYWhQ4cKERERYhepyTh69KgAoM7HCy+8IAhC1fDeZcuWCe7u7oJMJhPGjx8vXLt2TeUcWVlZwjPPPCO0bNlScHJyEmbNmiXk5+erHHP+/Hlh5MiRgkwmEzp06CCsWbOmTll2794tdO/eXbC1tRX69Okj7Nu3z2hft6mpu8cAhK1btyqOKS4uFl577TWhVatWgoODg/D4448LKSkpKudJTEwUJk+eLNjb2wtubm7Cm2++KZSXl6scc/ToUWHAgAGCra2t0KVLF5VrVGuuvxMvvfSS0LlzZ8HW1lZo06aNMH78eEUQEQTeY2OqHUZ4rxtv+vTpQrt27QRbW1uhQ4cOwvTp04UbN24o9pvjPZYIgiDoX59CREREZBgW2WeEiIiImg6GESIiIhIVwwgRERGJimGEiIiIRMUwQkRERKJiGCEiIiJRMYwQERGRqBhGiIiISFQMI0RERCQqhhEiIiISFcMIERERiYphhIiIiET1/1fFprYpRMbSAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(numberOfIterations, losses)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "id": "605a03bd-8f5f-4db7-9d6a-1a2f901c3584",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loss: tensor(2.7506, grad_fn=<NllLossBackward0>)\n"
     ]
    }
   ],
   "source": [
    "# Forward Pass (Training-Split)\n",
    "embedding = embeddingLookUpMatrix[trainingInputs]\n",
    "hiddenLayerStates = torch.tanh(embedding.view(-1, inputBlockSize*embeddingFeatureSpaceLength) @ weightsOfHiddenLayer + biasesOfHiddenLayer)\n",
    "logits = hiddenLayerStates @ weightsOfFinalLayer + biasesOfFinalLayer\n",
    "loss = F.cross_entropy(logits, trainingOutputs)\n",
    "print(\"Loss:\", loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "id": "e6de4711-a262-4eb0-8fe7-b55ef1fc7d9d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loss: tensor(2.7535, grad_fn=<NllLossBackward0>)\n"
     ]
    }
   ],
   "source": [
    "# Forward Pass (Validation-Split)\n",
    "embedding = embeddingLookUpMatrix[validationInputs]\n",
    "hiddenLayerStates = torch.tanh(embedding.view(-1, inputBlockSize*embeddingFeatureSpaceLength) @ weightsOfHiddenLayer + biasesOfHiddenLayer)\n",
    "logits = hiddenLayerStates @ weightsOfFinalLayer + biasesOfFinalLayer\n",
    "loss = F.cross_entropy(logits, validationOutputs)\n",
    "print(\"Loss:\", loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a2218a6-e7ab-4464-bfeb-505440508dda",
   "metadata": {},
   "source": [
    "**As we see the Training and Validation Splits to slowly depart from each other, that means that our model is finally good enough to slightly overfit the data.**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b9e76ed-7608-4f6a-910a-7100371b0905",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "# Final Multi-Layer Perceptron Approach Code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "id": "1e917ad2-543c-4ac7-86a1-d2a19b05b218",
   "metadata": {},
   "outputs": [],
   "source": [
    "words = open(\"Datasets/Indian_Names.txt\").read().splitlines()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "id": "c2f3a6c9-2046-4b88-b79b-5d59faf1f68d",
   "metadata": {},
   "outputs": [],
   "source": [
    "words = [word.lower() for word in words]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "id": "ca448ea8-9988-4a9a-a039-5c422364491d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "53982\n"
     ]
    }
   ],
   "source": [
    "print(len(words))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "id": "6e2e1c19-f4e4-4d1c-8b68-b8f07b300894",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Characters: ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z']\n",
      "STOI: {'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5, 'f': 6, 'g': 7, 'h': 8, 'i': 9, 'j': 10, 'k': 11, 'l': 12, 'm': 13, 'n': 14, 'o': 15, 'p': 16, 'q': 17, 'r': 18, 's': 19, 't': 20, 'u': 21, 'v': 22, 'w': 23, 'x': 24, 'y': 25, 'z': 26, '.': 0}\n",
      "ITOS {1: 'a', 2: 'b', 3: 'c', 4: 'd', 5: 'e', 6: 'f', 7: 'g', 8: 'h', 9: 'i', 10: 'j', 11: 'k', 12: 'l', 13: 'm', 14: 'n', 15: 'o', 16: 'p', 17: 'q', 18: 'r', 19: 's', 20: 't', 21: 'u', 22: 'v', 23: 'w', 24: 'x', 25: 'y', 26: 'z', 0: '.'}\n"
     ]
    }
   ],
   "source": [
    "# Remember we need our starting and ending tokens as well in these mappings,\n",
    "characters = sorted(list(set(''.join(words)))) # Gives us all the characters in the english alphabet, hopefully our dataset has all of them\n",
    "stoi = {s:i+1 for i,s in enumerate(characters)} # Enumerate returns the tuples of number and string, which can then be mapped to string:index\n",
    "# We manually add these tokens for convenience\n",
    "stoi['.'] = 0\n",
    "itos = {i:s for s,i in stoi.items()} # After we have the string:index mapping, we can easily iterate over their items to map index:string\n",
    "print(\"Characters:\",characters)\n",
    "print(\"STOI:\",stoi)\n",
    "print(\"ITOS\",itos)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "id": "32467263-ae8b-4e08-8a4a-b0d35305cb85",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Will build the dataset on only the words we take as input\n",
    "def buildDataset(words):\n",
    "    # We define a Block Size based on the number of characters we feed are going to feed to predict the next one\n",
    "    inputBlockSize = 3\n",
    "    # We define two lists, inputs & outputs, where inputs are our blocks of the block size mentioned above and outputs are the label indexes\n",
    "    inputs , outputs = [], []\n",
    "    # We iterate over each word\n",
    "    for word in words:\n",
    "        # We define the block for each iteration and fill it with 0 values -> [0, 0, 0]\n",
    "        block = [0] * inputBlockSize # This is also known as the context of the network\n",
    "        # We run another loop for each word's character, here word also needs the ending token '.'\n",
    "        for character in word + '.':\n",
    "            # We take out the index from our look-up table\n",
    "            index = stoi[character]\n",
    "            # We append the input with our block\n",
    "            inputs.append(block)\n",
    "            # We append the output label with out index of the character\n",
    "            outputs.append([index])\n",
    "            # We then take the block, crop it 1 size from the left and append the next index to it (sliding window of name)\n",
    "            block = block[1:] + [index]\n",
    "    # We also convert these inputs and outputs to tensors for neural network processing\n",
    "    inputs = torch.tensor(inputs)\n",
    "    outputs = torch.flatten(torch.tensor(outputs))\n",
    "    # We return the inputs and outputs\n",
    "    return inputs, outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "id": "ef9ce2d9-0eb8-4168-9d74-b03cd4378aa1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We define a manual seed to random\n",
    "random.seed(69)\n",
    "# We shuffle all the words, so that the model receives all kinds of data\n",
    "random.shuffle(words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "id": "c571fca5-300f-4ab8-93c9-64fb762211e6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We define two number of inputs\n",
    "# We take the number of examples to 80% in the first variable\n",
    "numberOfInputs1 = int(0.8*len(words))\n",
    "# We take the number of examples to 90% in the first variable\n",
    "numberOfInputs2 = int(0.9*len(words))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "id": "412a005f-28a5-454b-92fc-36c641a78537",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Inputs and outputs that go till 80% of the examples\n",
    "trainingInputs, trainingOutputs = buildDataset(words[:numberOfInputs1])\n",
    "# Inputs and outputs that start at 80% of the examples and go till 90% of the examples\n",
    "validationInputs, validationOutputs = buildDataset(words[numberOfInputs1:numberOfInputs2])\n",
    "# Inputs and outputs that start at 90% of the examples\n",
    "testInputs, testOutputs = buildDataset(words[numberOfInputs2:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "id": "4fba1121-dc1d-443c-85f6-9dc6e4bd0418",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total Examples: 53982\n",
      "Training Examples: 43185\n",
      "Validation Examples: 5398\n",
      "Test Examples: 5399\n"
     ]
    }
   ],
   "source": [
    "# We can check the numbers\n",
    "print(\"Total Examples:\",len(words))\n",
    "print(\"Training Examples:\",len(words[:numberOfInputs1]))\n",
    "print(\"Validation Examples:\",len(words[numberOfInputs1:numberOfInputs2]))\n",
    "print(\"Test Examples:\",len(words[numberOfInputs2:]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "id": "e50624e4-2a31-4f76-9b6c-bafaad0567d5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We will define a generator to give the same result on your machine, as of my machine\n",
    "generator = torch.Generator().manual_seed(6942069420)\n",
    "# Embedding Matrix (Input Layer)\n",
    "embeddingFeatureSpaceLength = 10 \n",
    "embeddingLookUpMatrix = torch.randn((len(stoi),embeddingFeatureSpaceLength), generator=generator)\n",
    "# Hidden Layer\n",
    "numberOfHiddenLayerNeurons = 200 # We increase the number of neurons here\n",
    "weightsOfHiddenLayer = torch.randn((inputBlockSize*embeddingFeatureSpaceLength), numberOfHiddenLayerNeurons, generator=generator)\n",
    "biasesOfHiddenLayer = torch.randn(numberOfHiddenLayerNeurons, generator=generator)\n",
    "# Output Layer / Final Layer\n",
    "numberOfFinalLayerOutputs = 27\n",
    "weightsOfFinalLayer = torch.randn(numberOfHiddenLayerNeurons, numberOfFinalLayerOutputs, generator=generator)\n",
    "biasesOfFinalLayer = torch.randn(numberOfFinalLayerOutputs, generator=generator)\n",
    "# Parameters\n",
    "parameters = [embeddingLookUpMatrix, weightsOfHiddenLayer, biasesOfHiddenLayer, weightsOfFinalLayer, biasesOfFinalLayer]\n",
    "# We set all the requires gradient to True\n",
    "for parameter in parameters:\n",
    "    parameter.requires_grad = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "id": "81139374-fff2-4972-ae68-b6574c74e6d5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We define a list that goes from -3 to 0 with 1000 steps\n",
    "lossExpression = torch.linspace(start=-3, end=0, steps=1000)\n",
    "# We do a 10^value expression on the value of the above list\n",
    "lossExponentExpression = 10**lossExpression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "id": "31d08af5-0b5e-4aea-bb43-f1a816e12e87",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We want to track the learning rates that we use in the training\n",
    "learningRates = []\n",
    "# We want to track the losses that we use in the training\n",
    "losses = []\n",
    "# We want to track the number of iterations we use in the training\n",
    "numberOfIterations = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "id": "a7f402cf-e288-4014-9505-0b43c3b889ae",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Minibatch Loss: tensor(2.7366, grad_fn=<NllLossBackward0>)\n"
     ]
    }
   ],
   "source": [
    "# We define the number of epochs\n",
    "epochs = 200000\n",
    "for i in range(epochs):\n",
    "    # Mini-Batches\n",
    "    indexes = torch.randint(low=0, high=trainingInputs.shape[0], size=(37,)) # Changing to Training Inputs\n",
    "    \n",
    "    # Forward Pass (Mini-Batch)\n",
    "    embedding = embeddingLookUpMatrix[inputs[indexes]]\n",
    "    hiddenLayerStates = torch.tanh(embedding.view(-1, inputBlockSize*embeddingFeatureSpaceLength) @ weightsOfHiddenLayer + biasesOfHiddenLayer)\n",
    "    logits = hiddenLayerStates @ weightsOfFinalLayer + biasesOfFinalLayer\n",
    "    loss = F.cross_entropy(logits, trainingOutputs[indexes]) # Changing to Training Outputs\n",
    "    \n",
    "    # Backward Pass (Mini-Batch)\n",
    "    for parameter in parameters:\n",
    "        parameter.grad = None\n",
    "    loss.backward()\n",
    "    \n",
    "    # Update Weights (Mini-Batch)\n",
    "    learning_rate = 0.1 if i < 100000 else 0.01\n",
    "    for parameter in parameters:\n",
    "        parameter.data += -learning_rate * parameter.grad\n",
    "\n",
    "    # Tracking the stats\n",
    "    # We append both the learning rates and losses that we use per iteration\n",
    "    numberOfIterations.append(i)\n",
    "    losses.append(loss.log10().item()) # We do that to squash the steep curve to a nicer graph\n",
    "print(\"Minibatch Loss:\", loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "id": "bc3be16f-3015-47bb-82ef-4e834c8d736d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x20f3ab86510>]"
      ]
     },
     "execution_count": 123,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAicAAAGdCAYAAADJ6dNTAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAABMeUlEQVR4nO3deVhU5eIH8O8AMoCyiMimKO67iKiI5pa4oNl6b141NStL018WrWTqzUq9Wdata1qWWrdMs0y7uaUorriA4i6KsqksKrLLOu/vD5hhhtkRmAPz/TwPz8OcOct75sCc73nP+75HJoQQICIiIpIIG0sXgIiIiEgdwwkRERFJCsMJERERSQrDCREREUkKwwkRERFJCsMJERERSQrDCREREUkKwwkRERFJip2lC2AKhUKBW7duwdnZGTKZzNLFISIiIhMIIZCXlwdfX1/Y2JheH9IgwsmtW7fg5+dn6WIQERFRDaSmpqJ169Ymz98gwomzszOAip1zcXGxcGmIiIjIFLm5ufDz81Odx03VIMKJ8laOi4sLwwkREVEDY26TDDaIJSIiIklhOCEiIiJJYTghIiIiSWE4ISIiIklhOCEiIiJJYTghIiIiSWE4ISIiIklhOCEiIiJJYTghIiIiSWE4ISIiIklhOCEiIiJJYTghIiIiSWkQD/6rK98dTkRqViH+McAPXb35QEEiIiIpsOqak+1nb2H90SSk3C20dFGIiIioklWHEyIiIpIehhMiIiKSFIYTIiIikhSGEwDC0gUgIiIiFasOJzKZzNJFICIiomqsOpwQERGR9DCcEBERkaQwnBAREZGkMJwAEGwRS0REJBlWHU7YHJaIiEh6rDqcEBERkfQwnBAREZGkMJwQERGRpDCcAOAYsURERNJh1eGEA8QSERFJj1WHEyIiIpIehhMiIiKSFIYTIiIikhSzw8nBgwcxYcIE+Pr6QiaTYevWrSYve+TIEdjZ2aFPnz7mbpaIiIishNnhpKCgAAEBAVi5cqVZy2VnZ2PatGkYOXKkuZuscxy+noiISDrszF0gLCwMYWFhZm9o1qxZmDx5Mmxtbc2qbalLMg5gT0REJDn10uZk3bp1uH79OhYtWlQfmyMiIqIGzOyaE3NdvXoV77zzDg4dOgQ7O9M2V1xcjOLiYtXr3NzcuioeERERSUyd1pyUl5dj8uTJeP/999G5c2eTl1u6dClcXV1VP35+fnVYSiIiIpKSOg0neXl5iImJwdy5c2FnZwc7OzssXrwYZ86cgZ2dHfbt26dzuYiICOTk5Kh+UlNT67KYHLyeiIhIQur0to6LiwvOnTunMe2rr77Cvn378Ouvv6Jdu3Y6l5PL5ZDL5XVZtApsD0tERCQ5ZoeT/Px8JCQkqF4nJiYiLi4O7u7uaNOmDSIiInDz5k388MMPsLGxQc+ePTWW9/T0hIODg9Z0IiIiIqAG4SQmJgYjRoxQvQ4PDwcATJ8+HevXr0daWhpSUlJqr4RERERkVWRCSH8IstzcXLi6uiInJwcuLi61tt6nv47GicQsfDWlL8b18qm19RIREVHNz998tg44QiwREZGUWHU4YXtYIiIi6bHqcEJERETSw3BCREREksJwQkRERJLCcAJAcIxYIiIiybDqcCJji1giIiLJsepwQkRERNLDcEJERESSwnBCREREksJwAo4QS0REJCVWHU5kHCOWiIhIcqw6nBAREZH0MJwQERGRpDCcEBERkaQwnAAcH5aIiEhCrDqccIRYIiIi6bHqcEJERETSw3BCREREksJwQkRERJLCcAJAcIhYIiIiybDqcMIGsURERNJj1eGEiIiIpIfhhIiIiCSF4YSIiIgkheGEiIiIJMWqw4kMbBFLREQkNVYdToiIiEh6GE6IiIhIUhhOiIiISFIYToiIiEhSGE4AcPR6IiIi6bDqcMLh64mIiKTHqsMJERERSQ/DCREREUkKwwkRERFJCsMJAAG2iCUiIpIKhhMiIiKSFIYTIiIikhSGEyIiIpIUhhMiIiKSFIYTcIRYIiIiKbHqcCLjELFERESSY9XhhIiIiKSH4YSIiIgkheGEiIiIJIXhBGwQS0REJCVWHU7YHJaIiEh6zA4nBw8exIQJE+Dr6wuZTIatW7canH/Lli0YNWoUWrZsCRcXF4SEhGD37t01LS8RERE1cmaHk4KCAgQEBGDlypUmzX/w4EGMGjUKO3bsQGxsLEaMGIEJEybg9OnTZheWiIiIGj87cxcICwtDWFiYyfN//vnnGq+XLFmCbdu24X//+x8CAwPN3TwRERE1cmaHkwelUCiQl5cHd3d3vfMUFxejuLhY9To3N7dOy8T2sERERNJR7w1iP/nkE+Tn5+Ppp5/WO8/SpUvh6uqq+vHz86uTsnCAWCIiIump13CyYcMGvP/++/jll1/g6empd76IiAjk5OSoflJTU+uxlERERGRJ9XZbZ+PGjXjhhRewefNmhIaGGpxXLpdDLpfXU8mIiIhISuql5uTnn3/GjBkz8PPPP2P8+PH1sUkiIiJqoMyuOcnPz0dCQoLqdWJiIuLi4uDu7o42bdogIiICN2/exA8//ACg4lbO9OnT8e9//xvBwcFIT08HADg6OsLV1bWWduPBCA4RS0REJBlm15zExMQgMDBQ1Q04PDwcgYGBWLhwIQAgLS0NKSkpqvm/+eYblJWVYc6cOfDx8VH9zJs3r5Z2oebYHpaIiEh6zK45GT58uMGahvXr12u8joqKMncTREREZMWs+tk6REREJD0MJ0RERCQpDCfgCLFERERSYtXhRMYhYomIiCTHqsPJnfyK5/cUl5ZbuCRERESkZNXh5OyNHADAv3bFW7gkREREpGTV4UQpv7jM0kUgIiKiSgwnREREJCkMJ0RERCQpDCdEREQkKQwnREREJCkMJ0RERCQpDCdEREQkKQwnREREJCkMJ0RERCQpDCdEREQkKQwnREREJCkMJ0RERCQpDCdEREQkKQwnREREJCkMJ0RERCQpDCdEREQkKQwnREREJCkMJ0RERCQpDCdEREQkKQwnREREJCkMJ0RERCQpDCdEREQkKQwnREREJCkMJ0RERCQpDCdEREQkKQwnREREJCkMJ0RERCQpDCdEREQkKQwnREREJCkMJ0RERCQpDCdEREQkKQwnREREJCkMJ0RERCQpDCdEREQkKQwnREREJCkMJ0RERCQpDCdEREQkKQwnREREJCkMJ0RERCQpDCdEREQkKQwnREREJClmh5ODBw9iwoQJ8PX1hUwmw9atW40uExUVhb59+0Iul6Njx45Yv359DYpKRERE1sDscFJQUICAgACsXLnSpPkTExMxfvx4jBgxAnFxcXj11VfxwgsvYPfu3WYXloiIiBo/O3MXCAsLQ1hYmMnzr169Gu3atcOnn34KAOjWrRsOHz6Mzz77DGPGjDF380RERNTI1Xmbk+joaISGhmpMGzNmDKKjo+t600RERNQAmV1zYq709HR4eXlpTPPy8kJubi7u378PR0dHrWWKi4tRXFysep2bm1vXxSQiIiKJkGRvnaVLl8LV1VX14+fnZ+kiERERUT2p83Di7e2NjIwMjWkZGRlwcXHRWWsCABEREcjJyVH9pKam1nUxiYiISCLq/LZOSEgIduzYoTFtz549CAkJ0buMXC6HXC6v66IRERGRBJldc5Kfn4+4uDjExcUBqOgqHBcXh5SUFAAVtR7Tpk1TzT9r1ixcv34db731Fi5fvoyvvvoKv/zyC1577bXa2QMiIiJqVMwOJzExMQgMDERgYCAAIDw8HIGBgVi4cCEAIC0tTRVUAKBdu3bYvn079uzZg4CAAHz66af49ttv2Y2YiIiIdJIJIYSlC2FMbm4uXF1dkZOTAxcXl1pbr/8721W/Jy0bX2vrJSIiopqfvyXZW4eIiIisF8MJERERSQrDCREREUkKwwkRERFJCsMJERERSQrDCREREUkKwwkRERFJCsMJERERSQrDCREREUkKwwkRERFJCsMJERERSQrDCREREUkKwwkRERFJCsMJERERSQrDCREREUkKwwkRERFJCsMJERERSQrDSSUhhKWLQERERGA4ISIiIolhOKnEihMiIiJpYDghIiIiSWE4ISIiIklhOKnEuzpERETSwHBSib11iIiIpIHhpNLy3fGWLgIRERGB4UTl+p0CSxeBiIiIwHCiwrs6RERE0sBwosJ0QkREJAUMJ0RERCQpDCeVeFuHiIhIGhhOiIiISFIYToiIiEhSGE4q8a4OERGRNDCcVOIIsURERNLAcEJERESSwnBSifUmRERE0sBwUknBdEJERCQJDCdEREQkKQwnlQ5euW3pIhAREREYToiIiEhiGE6IiIhIUhhOiIiISFIYToiIiEhSGE6IiIhIUhhOiIiISFIYToiIiEhSGE6IiIhIUhhOiIiISFJqFE5WrlwJf39/ODg4IDg4GCdOnDA4/+eff44uXbrA0dERfn5+eO2111BUVFSjAhMREVHjZnY42bRpE8LDw7Fo0SKcOnUKAQEBGDNmDDIzM3XOv2HDBrzzzjtYtGgRLl26hO+++w6bNm3Cu++++8CFJyIiosbH7HCyYsUKzJw5EzNmzED37t2xevVqODk5Ye3atTrnP3r0KAYPHozJkyfD398fo0ePxqRJk4zWthAREZF1MiuclJSUIDY2FqGhoVUrsLFBaGgooqOjdS4zaNAgxMbGqsLI9evXsWPHDowbN07vdoqLi5Gbm6vxQ0RERNbBzpyZ79y5g/Lycnh5eWlM9/LywuXLl3UuM3nyZNy5cwcPPfQQhBAoKyvDrFmzDN7WWbp0Kd5//31zikZERESNRJ331omKisKSJUvw1Vdf4dSpU9iyZQu2b9+ODz74QO8yERERyMnJUf2kpqbWdTGJiIhIIsyqOfHw8ICtrS0yMjI0pmdkZMDb21vnMgsWLMDUqVPxwgsvAAB69eqFgoICvPjii5g/fz5sbLTzkVwuh1wuN6doRERE1EiYVXNib2+PoKAgREZGqqYpFApERkYiJCRE5zKFhYVaAcTW1hYAIIQwt7xERETUyJlVcwIA4eHhmD59Ovr164cBAwbg888/R0FBAWbMmAEAmDZtGlq1aoWlS5cCACZMmIAVK1YgMDAQwcHBSEhIwIIFCzBhwgRVSCEiIiJSMjucTJw4Ebdv38bChQuRnp6OPn36YNeuXapGsikpKRo1Je+99x5kMhnee+893Lx5Ey1btsSECRPw0Ucf1d5e1JK7+cVo0Yy3k4iIiCxJJhrAvZXc3Fy4uroiJycHLi4utbZe/3e2a7yOjngYPq6OtbZ+IiIia1bT8zefrUNERESSwnBCREREksJwQkRERJLCcKJmy6mbli4CERGR1WM4UbN8dzzHXiEiIrIwqw4nMpn2tEtpefVfECIiIlKx7nCiY9r90vJ6LwcRERFVse5woqvqhIiIiCzKqsNJi6b2li4CERERVWPV4eTZwf6WLgIRERFVY9XhxLEJHzxIREQkNVYdToiIiEh6rDqcsDksERGR9Fh3ONHRW4cdeIiIiCzLqsOJLln5JZYuAhERkVVjOKnmhR9iLF0EIiIiq2bV4YS3cIiIiKTHqsNJHz83SxeBiIiIqrHqcNLciSPEEhERSY1VhxPe1iEiIpIeKw8nutPJ9LUnkJpVWM+lISIiIsDKw4k+B67cxqub4ixdDCIiIqvEcKJHek6RpYtARERklaw6nLDJCRERkfRYdTghIiIi6bHqcMLeOkRERNJj1eHEECGEpYtARERklaw6nMjY6oSIiEhyrDqcEBERkfQwnOjBmzpERESWYdXhxFCDWDY5ISIisgzrDicG3kvP5SBsRERElmDV4cQUey5m4HJ6rqWLQUREZDXsLF0AKTuTmo2ZP8QAAJKWjbdwaYiIiKyDddecGOlJHJ+RVz/lICIiIhXrDidEREQkOVYdTpraG7mrxR47RERE9c66w4nccDgpUzCdEBER1TerDifGrD2SaOkiEBERWR2GEwMSMvMtXQQiIiKrw3BCREREksJwUo+yC0tQVFpu6WKguKwcO8+lIaew1NJFISIi0sJwUk+yC0vQZ/Ee9P9or6WLgmU7L2P2T6cwbd0JSxeFiIhIC8NJPYlLzQYA5BWVWbYgALaevgmgYgRcIiIiqWE4MVFqViEEH1VMRERU5xhOTDTk4/145rvjAIBfTqbirV/PoJzjoBAREdU6PvjPDEcS7kIIgbd+OwsAyMwrxvoZA0xaViYz8iAfIiIiAlDDmpOVK1fC398fDg4OCA4OxokThhtWZmdnY86cOfDx8YFcLkfnzp2xY8eOGhXY0qatrdrXqPjbFiwJERFR42R2ONm0aRPCw8OxaNEinDp1CgEBARgzZgwyMzN1zl9SUoJRo0YhKSkJv/76K+Lj47FmzRq0atXqgQtvCYeu3tF4HZucBYUJt3dqu94kM7cIK/cn4HZecS2vmYiIyLLMDicrVqzAzJkzMWPGDHTv3h2rV6+Gk5MT1q5dq3P+tWvXIisrC1u3bsXgwYPh7++PYcOGISAg4IELLwVPrYqul2HuVx+4hiU7LqleP/99DJbvjsfMH2LMXhdvMRERkZSZFU5KSkoQGxuL0NDQqhXY2CA0NBTR0dE6l/njjz8QEhKCOXPmwMvLCz179sSSJUtQXm75wchqy0/HU4zO86B5YNnOy/jm4HVcu10xpP65mzkAqrooG5KQmYdhy/djc0zqgxWCiIioHpgVTu7cuYPy8nJ4eXlpTPfy8kJ6errOZa5fv45ff/0V5eXl2LFjBxYsWIBPP/0UH374od7tFBcXIzc3V+OnoShXCFxOzzXY7TgzrwjJdwtqtP6ajDD75q9nkXy3EG/+erZG2yQiIqpPdd6VWKFQwNPTE9988w2CgoIwceJEzJ8/H6tXr9a7zNKlS+Hq6qr68fPzq+tiPpDEOwXYFlcxsNmCbecx9vND+HzvVY15ZGqtTgZ8FIlhy6OQVVCimna/pBzpOUVGtyUz0HpFCIG9FzOQcrdQY3pxqaLaOoiIiKTLrHDi4eEBW1tbZGRkaEzPyMiAt7e3zmV8fHzQuXNn2NraqqZ169YN6enpKCkp0blMREQEcnJyVD+pqdK/HTFvYxxu3CvEhspbPP+OvAohBI4k3EFmru7QobxFAwAP/WsfBi6N1AoW1Rm6PRR15TZe+CEGQ5fvN38HqmmsA84VFJfh33uv4kpGnqWL0iisPZyIA1fYa42IapdZ4cTe3h5BQUGIjIxUTVMoFIiMjERISIjOZQYPHoyEhAQoFFVX71euXIGPjw/s7e11LiOXy+Hi4qLx0xA89C/NULD3UiamfHscIcv26QwVyvO/QiFwt7IW5cAV3b2eTBGTlKVzuqFAc/ZGtta0Kxl5CPpwL747bLyhb1m5AltP38St7PumFhMlZQp8c/AaLqXV/+26T/6Kx2d7r2D0ZwfrfduNzYnELCz+8yKmr+Uzmoiodpl9Wyc8PBxr1qzB999/j0uXLmH27NkoKCjAjBkzAADTpk1DRESEav7Zs2cjKysL8+bNw5UrV7B9+3YsWbIEc+bMqb29kKiDlVeU5Qph8JZNklr7kxV7ruCmgRN9bXS0UV/Hgq3ncTuvGKXlVeFx/u/nkFVQgg/+vGh0XeuOJOHVTXF4+NMok7e/7kgiluy4jLB/HzKn2AblFZUi8lIGSsoUBuc7dyOn1rYpBddu51vs6dLmBNLGWhNHRHXD7HAyceJEfPLJJ1i4cCH69OmDuLg47Nq1S9VINiUlBWlpaar5/fz8sHv3bpw8eRK9e/fGK6+8gnnz5uGdd96pvb2QqIKSqof8vb75jNb7QghcvJWLmOR7qmn3Cksx8WvdPZ8A/W1O9l/OxJ9n03S+ZyjQXE7PQ/+P9uKJr46oppkzKv/BqxUBrKjUcChQp+xpVJueW38Sz38fg0/+iq/1dUtVQmYeRn56AIEf/GXpohh0MikLQR/uxR9nblm6KETUQNRo+Pq5c+di7ty5Ot+LiorSmhYSEoJjx47VZFMN2pZTNw2+LwCM+0K79uDGPfNrTmasP2lO0VSKK2sazt+susVi6Cq3oLgMaw8nIqyXNzp6Opu0jdjkLCz+8xL+OaE7Ats0r1E5qzuRmIWzN7Lx/EPtIJPJcDKpIuBtjknFu+O61co2pC762l0A5oVJS3hu/UnkFZXhlZ9P49EAX0sXh4gaAD74z4ImrTEtsKmHBV3ZxFiVuaEePjq3Z+C9j3ddxqd7riB0heE2G6lZhRjxSRT+G52Ep1ZF40xqNv6+uqJGyJRB4LIKShB97a7WvpUrBP44cwtPfx2ND7dfwldR14yuiyzLlBGUiYjUMZxYkKFMMeXbY9h4oqLnT0FJ1dgmus7r//fz6Tor19Oro/GvXZfx14V05BaV4vvoZI15qw/nr7T4z4tIvFOABdsuqKaVmXCSyswrQkZuEYYt349Ja45h13nN8XM2HE/GK2r7u3x3PJLuVLXZMWf026dWHcW+yxlG5zuScAfHrt/VmJZTWIrULMM9q4z5aPtFfH3gGsoVAlHxmbhXoLv3ml61NNKvrqdrl5UrMGz5fszdcErvcsJgjCUiqjk+lViijiTcxZGEuwjr5YOA9w23KdDX1qS4rByZudrP3rmTr/8kWFau0Bh19kRSFk5U9gIa20N3d3FdDDVM1XdKLStXYMBHkRrT9l7KRFgvH9VrXWHIlFFyASD5bgEy1Z5FFJt8D8+tj0HSsvF6l8ktKsWUb48DAOI/HAu5XUWX+IDFFcfk6DsPw9fNEeUKgZd/ikVTezu0dJHjpaEd4N5Ud280AEjIzMeaQxW9odJzi7DuSBIAGC3Lir+u4NE+vujbpnmNxqtZ8Vc84m7kYO30frCztcFfF9Ixd8NpfDaxD8b3rvqcX/vlDJLvFiL5biEWP1ZicF+AijZPHT2bwc/dqQalIiLSxHAicdu1gofpp6THVx7V6q57N9/wgwI/3XNF73u7LugeBVgXfdfUQgi9DSOLdQQac6/Os6rVPnx76Dpiku5h8WM9MGx5lM5lXvn5NOaFdkKHls0AVISk/OIyuDnZI/d+VU+YsnIBebX/mLjUbPi6OSLyUgZ2X6iqhUm6U4Cvp/bTW877arVhymCij7J2Z9/lTPx4LAXrjyZh96tD8UXkVYPL6fLFvgQAQOTlTIzp4Y0X/xsLAJiz4RTG964KRv9TO0a6albO3cjBqeRs1WtlmydD4QqoOP58thMRGcNwInHlCs0TdmxyFjp6NjNpWV3jiHyk9vDA6oQQ+P5oklnlM9eyXZfNmr9cIbAq6hpCOrRAHz83k5bJyC2Cl4sDjl+/iw+3V+yvp4tc7/x/nLmFmKQsHI0YCQB4/KsjOH8zFwfeHA5bG9NOpPerPVbgrI4uy7lFpdh9Ph2jjdRAlZUrYGsjQ0FJOWxkwHPrKx7u2N6jqWqeMZ+bP06Lem2WriCoT/WAWK4QmPCfw2ZvHwB+P30TT/ZtXaNlich6MJxInHqbDQB4+7dz6OxlvJdMQma+zumGehCtOXQdhSV1+0DGrw9c13j9wvcx+HZ6RQ2DrgvqbXG3sA0VV/FJy8ab1MwiLjUbOYWleOu3qmcJFRQb3q9bauPQKHsu/X76Jv4WZPhEas7wHa/8fBpR8bfx59k0vDG6i855Ssoq2nqkVZZndPeq51hdv2P685jKyhWYs+EUerd2w5wRHXEq5R6e/OqoRlmqD9q3/Wwa/NwdEXlJcyDA3RcyMHVg26p1K4wHm7JyBWxkMthUC3ebY24wnBCRUQwnDdC2OOPjRYSuOGD2ev+1y/AYITKZ4ZOxEAJXM/Ox4XgK7uQZvn2ktPdSxS2LotJyk9uOGLNs52UkVjuRn6/B2Cqf772KLytvg6hLMiEkyABsi7uJ6Gt38eHjPWFna4Oo+IoxYQ5cuY3XR3fWudzFtFxVMAGAvy4ab7BbnRACHefvBFARLOaM6Ij3/6c9oN4P1Ro3z9HT+HXB1vOY1N8PClHxN2Cs91dFY9ooNJPbYderQzTei67WsLihKlcIk2vViMh8DCcN0Po6uvWiq22BuurBJDZZ88q734d7VcPwm2vGupNGT1y5RaUmjelRPZgAQLwJz9LZFncTj/VppTFN12cy/JMo1e/Z90uw5dQNnSFm3sY4AICjvS1OpWRrvPfof45ozQ8YbkhsSFR8JgL9msPVqQlSdPQiOvOAwa/bwl0oLRdo6SzH4bdHGJw36W6hapRjXZ+fQiG0alQakoTMfEz48jBmDm2P8FG6QybRg8gvLsPK/QkY38sHPVu5Wro4FsGuxFRjF9M0T/g1DSZCCJOuqHv/8y/s0VGTcPSa7u7M5lKGCXPM//08wn85oxWI1G8TrTuSZHI4eNrA6MCGPLvuJP7+dcVtm+oh8tDVB38wX2l5xUpv5xWj0MAtshV7rqCguGpk5Nd+OaPVAPb576sGDCyo/BJWfwim1C3beQn3S8tr1CCZzHO/pFyj8bi1+NfOy1gVdQ2PfFmztl2m+G90EsL+fQiZefofrWJJDCdUY7U1uNZ7W88/0PK/xNyolXIAumtdlNYeToT/O9trbVu17UpGxQm+erucqd/V7oP57hjo8fVF5FUs21nV6Pl/Onpm7Y+vCktLd17C8t3xGPmp+bchqXErVwj0WLSrstauZjWKDdWFW3X/DLAF2y7gUlouPjPQQ9OSGE6oxjaeTK2V9fx0PKVW1lMbRqjdsqnOUDdrKTF3RGBzjTLyROfqtWCGRjCOSbqn9z1j4tPzJHvVRw8uv7hMdRu3+hABjdW9gpJ6f0imOc9Fq08MJ1Rjuroqk+U1hGFE/rXrMmasO4HL6cbbAulyJSMPYz4/qDVoX2Mhpac4x6VmY+p3x2v8/34lIw+z/huLy+kP/n1xv6T8gUdmtrRyhUDKXe192HkuDYEf7MFiHU+DL1cIFJWW43ZeMV78IQZR8Zla89SUlP7W1DGcEDUiU749hiEf77d0MTQU6GgzsCrqmsbtHWPu5Bdr1JKMNlJ7oySEwJqD13EiMcv4zBLx0n9j8NjKI0YbqBuSlnMf//fzacRWPvE8PacIXx+4hpzCUiNLant85REcunoHz1SOlGyuSd8cw64L6apnaz2I4Z/sx5CP9zfoC6M5P53C0OX7sfW05rAOS3ZWjMmka1DG8V8cQtcFu/Dmr2fw18UMPLuuZg961UWa0YThBEFta+cpuURScCShcXTVVVeuEOj34V4M+CgSRaXmNY7cdT4dH+24VOOGxrXNlLYTuy9k4OyNnBp1fxdCIPraXfxtVTT+d+YWnlpV0Uh64jfRWLrzMl7ffEbvsoeu3kbElrMaDZrVVW/w/tmeK5i+9gTKjOyTcrm8It3rNYXy4j6j8nEce2vQxd6YotJynL+ZU+c1CcqRtr85eF3vPNVLoKxhjDIj0FeXmlWIjSdStHoExibfw7Y4/eNfWYrVh5NnB/lbughEZIB6IDG3R9ivsVWNpZPuFOBm9n3svpD+QCcgfYtuOJ6CeRtPo6xcgdjkLByv1vZm2c7L6DR/p8lX/ebenitXCAxbHoVJa46punIDwOGrd5BceRvhwJWq2wFnb2Tj33uvIiO3CKXlCkz97gR+PpGKeRtNe5DovyOv4sCV29h7qfZuMdQXIYRWe6Vn153AI18errW2dOaq67YfQz7ej3e2nMM3BzWf5H7j3n3M2xhXozBcl6w+nAzt3NLSRSCiSqlZhdh1Xn940Dc9NasQy3dfxm21wf+OJtxB5OWqE+eozw5g8LJ9eOm/sXqf77RyfwIWbD1fo/Dy7u/nsC3uFracvomnVkVj4jfHkFdUdRtl9YGKk8Inu+NRUqbAmdRsgz3elA2bU+4W6q3NUPo19gZ6LNqlc4ybZ76ruh2j7BIOVIy189neKwheEolHvqjqsqoMGyVlCqTlVIUcfUrM6Elj6HMtK1fg6LU7qq7DhsLZxbRcTF97AhdvmRb0cu6XYtPJFNVtrXd+O4cBH0XC/53tqvYfx65X3PrbcDwFRaXlOHfD9FoUfccn+tpdTPrmGBIyjbetuq1n4Mqajn2kTn1MKn3DNuj627Ekqw8nDaHxIJG1GPLxfsz6MRa7zlc9ZNJG7Z9U17ni20PXMeTj/Vi5/xpe+bniqv/Ps7cwuVobCfUT87HKL+jqY2gs3x2P/x5LNrmh7pIdlxCTlIW3fq26XaLe1Vp5K6O4THM7r28+g8dWHsG/q42VEr4pTvW7TAZcTs/F0OX78dC/9mlt+59/XFB1A31j85kHuvLWNUjhI18eQshSze3+dDwZQgiNk7aNrKI3zSs/n8ZhtaeGF5eVY/pazW7s6sf1bn4xwjfF4fj1u/j20HV0nL8Tk9ccx+yfYo2Wd+f5dBy4chvjvjiE5LtV3f93nU/D7B9jNUKhQiEw8etovP3bOQQs/gshSyOxKaaqduSdLWc11n3uZg6eW38SE/5zGD+q9SQUQuD8zRzkVwsiX0ReRY9Fu/H8+pO4fjtfo0Zm0ppjiL5+F6ErDmLOT6c0ynUnv1jVJqg69b9z9a751VX/uxJC4HZeMe6XlCMjt6oc6iGurnvz1RarHyG2YRwmosZPvRZh9k+n0NGzGRIy8/Fk36pRe3WFE+XDHYGKq8KycgWW7jD8gMmd59PRoWUzfLj9Er6a0hfjevlovF9cplCdgL/clwB/j6Z4NMAXADRqY745eF2r7UD1Mu65mIGZP8RoTFOO/7L6wDW8pjbK7Ba1RpLlCoEDVyraGNyrvOIXQiDpbiGEEKqRol+ro1FqlePmqJv/+3ks2HoeXbxdVNNkkOGj7Zfwx5lb+OPMLdWTqX+Lvakqv1LqvYqr8+u38/Fw5dg2W6o1DI2Kv40Lt3Lg5+6kmmbo6eShKw7g6kfjAACzfqx4BIOfuxPeHdcNscn3VO1ulNQfDwEAR69p1yQop607nKh6rtTeS5mY+UMM2nk0xU8vBOPQ1dvo5OWMFZUBMfJypupvI3HpOK3BB7efS9PYp8y8Yjy16ig2zwpBf393jXnVH+Wx9kiizv1Wjrm0N3yY6mGwb2w+i99OVd3KPPTWCI1tAhWhV1fXbKl12mHNCatOiCSh/bs7NF4rH16p/rDKvy6mw5g1hxI12lzokl1Yqgo1r6nVVij9FnsDA5ZEYsG281ix54qqRsbcBrkAtIKJ0PN7dUt3aj9B/OcTqRjxSZTqxF5X0nP0jx+jEJrDCNjIgFs6Pu/CEu1bHcqrdmPlH//FYWw2cXDF0nKB0nKFRo8sZe1V9WBirut3ClS3bJS3AhPvFGDQsn14+7dzGg/TVHc4Qfeo1bpGsz6iZ15TqbchUQ8mQMVzvBTKh2JVOnT1Dvp+sOeBtlkfGE4sXQAiMtmH2y9h4wnDg/b9a5fhWhNT/PdYMm7nFePHY1Xb2nU+HV0X7DK6rPoYFKY+OqBcIbTaFijbQCgVlZbj3d/PaS1rarsLpZzCUrSLMDzS8cClpo8fsz8+U+eTqnVdiQsI9Fn8l0nr/UDHeB/6dJq/U6NHVraZXaYNhU5lLYqu0Y71WfTHBZ3Tz97QbnSae78M3x7S33PnQby39TzGfXGoTtZd12RCqiOwqMnNzYWrqytycnLg4uJifAEzFJaUofvC3bW6TiJqOOxtbbD7taF4fOUR5Nw3fxwQc/XwdcEFtUDR3785TuoZKffhrp7Yd7l2e8P8o79fnfZISVo2Hpm5RRiwpPYGyGvbwgnfTe+H0BWmjW8DAON7+WD7ubRa2f6mFwdi4jfHamVdtc3J3hYXF4/F8et39Zbxw8d7Gn1MyH8mB+KR3r61Xr6anr+tPpzcLylHt4XGr4aIiMi46SFt8ceZW6p2Mo1B9UApNVc+DEPQB3uQp6fXUHuPprhu4LlhAPBaaGfMC+1U62Wr6fnb6hvEyu2s/s4WEVGt+T462dJFqHVSDiYA0Pm9nQbfNxZMAMMP9LQEqz8z29iw1QkREVm3/x5LltRzdqw+nBARERFw8OqD9RyqTQwnREREpHdQOEtgOCEiIiJJjcTGcEJEREQGBwWsbwwnREREJCkMJ0RERCSluzoMJ0RERGT4AYv1jeGEiIiIJIXhhIiIiHhbh4iIiEgfhhNUPKiKiIiIpIHhhIiIiCTUHJbhBACgkNIRISIisgC2OZEYKXWfIiIisgQ+lVhiWHNCRETWTkqnQoYTSCstEhERWTuGEwAKhaVLQEREZFlxqdmWLoIKwwkABWtOiIjIyt3JL7Z0EVQYTiCt+2xERESW4ObYxNJFUGE4AWtOiIiIgto2t3QRVBhOIK2+3URERJYgk8ksXQQVhhOw5oSIiEg60YThBABrToiIiKSE4QSsOSEiIpJS1UmNwsnKlSvh7+8PBwcHBAcH48SJEyYtt3HjRshkMjz++OM12WydYTYhIiKSDrPDyaZNmxAeHo5Fixbh1KlTCAgIwJgxY5CZmWlwuaSkJLzxxhsYMmRIjQtbV/TVnHw2MaCeS0JERERmh5MVK1Zg5syZmDFjBrp3747Vq1fDyckJa9eu1btMeXk5pkyZgvfffx/t27d/oALXhZ6tXHVOH9HFs55LQkREZBkyCd3XMSuclJSUIDY2FqGhoVUrsLFBaGgooqOj9S63ePFieHp64vnnnzdpO8XFxcjNzdX4qUsvDGmHd8K64rvp/ep0O0RERFIlJDQkqVnh5M6dOygvL4eXl5fGdC8vL6Snp+tc5vDhw/juu++wZs0ak7ezdOlSuLq6qn78/PzMKabZ5Ha2mDWsA0Z288KCR7rX6baIiIgkSTrZpG576+Tl5WHq1KlYs2YNPDw8TF4uIiICOTk5qp/U1NQ6LKWmCQE+9bYtIiIi0mZnzsweHh6wtbVFRkaGxvSMjAx4e3trzX/t2jUkJSVhwoQJqmmKykcA29nZIT4+Hh06dNBaTi6XQy6Xm1O0OlGbvXj2hg9D6IoDtbdCIiKiRsqsmhN7e3sEBQUhMjJSNU2hUCAyMhIhISFa83ft2hXnzp1DXFyc6ufRRx/FiBEjEBcXV+e3a2riQRsEXfkwTOf0jp7NHmi9REREdam1u5Oli6Bi9m2d8PBwrFmzBt9//z0uXbqE2bNno6CgADNmzAAATJs2DREREQAABwcH9OzZU+PHzc0Nzs7O6NmzJ+zt7Wt3byzgrbFdNF43sZXBtdqTHVs6W64WaG/4MIttu6Z6+LpYuggNwpBOHjj+7kit6c3kZlWI1jtbGxlaNG34//tEjU0zua2li6BidjiZOHEiPvnkEyxcuBB9+vRBXFwcdu3apWokm5KSgrS0tFovqFS9PLwjJge3Ub0WAvjv8wM05vl6apDBdbwT1lXn9FdGdoL6c5j83B3NLl9Hz2Ya5fNyqd2g9ERgq1pd39WPwrD9lSH45SXtmjhTPdzVE+08mmLpk7203hvfS9ptil4c2h6rnzH896LUoWUzeLk4aE0XJt6PrB6iTfHmmC5Y8oT252qORwN8EfNeqPEZiahe2TT0B//NnTsXycnJKC4uxvHjxxEcHKx6LyoqCuvXr9e77Pr167F169aabLZeNLGtOjg2NqYdqCVP9MLrozpj0YTusLGRoXdrN7wxurPq/b5tKh5D/dMLwVrL/nNCd8wa1gG6NhU+qjPeG1/Ve0iIinV8NaUvvpgUiJlD2pm0L3K7qsP8x9yHTNonUy18pLtG+DHV3BEdtaZtenEgmthWlNXWxM9el7XP9sf+N4Zj0oCqcoV288R747thiY7AIiURYV3h66YdOHTRF0LatGhq0vLjahDU3JvaY2xPzfZlPzw3AO09TNsmACx4pDtkMhkufzDW7O0TUd3hU4klzM3JHi8Na49ZwzqYdWX5fyM7YcZgw2FBPSQo2VWejH+bPQj92jbHlGonevWajpD2LTC4owfG9fLBowG+GNFV9yBx43p54/OJfRD15git99SvtPv7Nzf7FsDWOYM1Xjdvam/wStrJXrOa8IPHeyJp2Xi8MaYLPp/YR+O94PYtVL+rh0RjXhraHn3buBmcx9vVAS8MaV+j2gJzPd7Ht8bLymQyk9snKXRkk+4+Llj8WA+Tll/wSDdziqZiV+3YDO3cEl9MCjRp2cA2bnCvvKXj0MS21mveiKjmpBNNGE50igjrpvdWi6naeZh2glEG1cA2zfHr7EEI8HPTeH9cTx+M7+WDoLbNsXBCtTFY9NTe29va4PHAVmjlVnEbqL+/u+5tm/CnWH0I/z7VymeMDMDJ+aF4pLcPfn95EKYObKt67/HAVhjd3Uvncr1auep9T2mAvzu+nhqEiHHd6qQ6cmB7dwxs747j747EpcWmX+V/NrEPRuoJjkrbX9GswQrwc8OYHhX762Rvh/Pvj9EKb9VVHzBp0YTu2DFvCDyamXbrzsneDutm9EegnmB3ZuFovDy8A/a/MVw1TQbAxaGJ6ti8/2hFEOrZyhVfmhBQqlf2fDaxD2YM9jepvGSaVVP61tm618/oX2frJt2eHeRfb9sK6dDC+Ez1hOHECPU/jG+MtB1RN66XN94d1xWbZ1W1nahJlZmNjQwrp/TFb7MHwdmhZlf9YT29sWpKXxysVpPSvmVTjO6hGQC+nhqEnq2qGqQ+EdgaD3XUPUaNKXdehnZuiZbOcvxncl8EVt7eMoVMJsM306pG7B3fW/sWxC+zQjCmR8UthjfHVDRM1vePrF5jsv2Vh9DZqxnmjeyEk/P1t314bnA7bHwxBF4uDnC0N62h2B9zB0Mmk+Hb6f2w7ln9X+Q9fDUfmbD15UH4emrV/jaT28HZoapWa8crQ/BaaGcYUpOANqKLJ35/eTDa6bgt4+rUBG+N7arzvW+m9UPi0nGYrvZ5TwjwRRcvZ9Xrr3ScJHXl6Ud6V9Q0PcitPACYHtLW6DzGAq9UPP9QO3RoqftWma7joS7MwO26p/u1Vv3+96DWRm8NX/1Is/dh37bG/4efrOfasC8mBWp9tzWT2+ERHd8Z5hrXyxsLHumOveFDzV52RJeWqt9NvWBQOvRW1f5MCDCvJlbfd2BXb2f86yn9tdxnFo42u5x1ieHEDKN7eCNp2XjVl+574/VXi8tkMrw4tINGrYUp545AM2smdGnfUrPWRiaTIayXD9q0qOgm9tvsEDwzsA0ixnXDB4/1xJIneqF9y6Zo5eaIEV08EdBaswytm5veEHdCgK+qBuCR3j5Y9lRvg/Obej5t4+6ELS8Pwqxh2uPiABW3hC68Pwb/fFTzlsbyv/XG0M4tNZbr4euKv14bhtdGdTbYkyq0m/ETmXe1Bqm9Kz87mUymsW/GrmZ1BdeuPlUhsbuvC+aFdtJ4v3qNmK7PUr2m6uO/6T8Wxg6DshZuaOeqL1xjYdvUNi1BbZtj/xvDcf6fY9Dfv+Lkp6tNkjGeOhoHj+vljT//7yHEfzgW8R+O1Qi8ALDl5UE6/49XP9MXPq4OsNMTmOztbODm1AS99DyXS90Hj/UwuyH2gke6I0hPEJgQ4It9rw+De1N7rbB1+G3tW7nqIWdifz8kLRuPhI/CsPzvhh9semrBKFUbMCUXhyZo28Jwd9OOXs20bsvOH2f4FuLaZ/shOuJhg/PoM6qbF9q0cNLoOakQAv+Z3Bcn3h2JBLWANXVgW63tfPRET43/E3VfTQnC8w+1Q0dPZ633Dr01AknLxutcbverQ7FuxgAkLRuPxKXjcOLdkVj7bD+cqNa7bt2M/lq3vgGgia0NPn6qN14e3sHoLWsAGKb2f1n9OxCouJ2669WhGN/bFy10/N0AFRcjUiLtPocSNa6XDy4tHmvy1bQuLZra425BCYZ2aqkxvZOXM7bNGQzPGvSq2TwrBHsvZuDFoYYfrhjU1h1BbatObJOD22DSAD8oRMXVq2MTzf16uKsnNp5MVbVPeWZgG/x4LAWvj9bsRq20+pkg5N4vq5U/9r8HtcbWuJuYFtIWPq6OuJKep3fepjraz/y9nx/+3s+88XQcmtggbuFokxpE6/piUVKvJQjr5YPvnxuAt389qwoJXb2dcTk9D76uuhvAtnJzxO5Xh8JNz+f4qAlXVAsndMfUkLY4lXwPf+vbGhuOpyAuNRsBrTVPqvPHd8Pz38foXc++N4Yhv6gMLeroykpZG/DTCwNxK/s+/D2aYtbwDnBsYosO7+4wunxnr2Y6g8RXU7RrO9t5NEXinQIcfHME2rRwQncfF3y4/ZLq/c8mBmBsTx+M7OaFcoVA1wW7tNaxZfYg9Gzlig3HU3Du93MAKq7W84vLtOadGuKPpnI7bD9X1YtRbmeD4jKFwX2qXhMW+fow7L+ciWcGtoVDE1ucWjAKAOD/znYAgI+rA1o3rwgOb47pguW74wEAu14dipOJWbh+p0D1f29na/y6VNk2aP2M/ojYcg6fVIaZl4Z2wLuV+6xLQGs37HhlCEZ9dlA1bWxPb3y0o+ozXv633vjxWDLO3MgBADzcVf+FwKQBbfDziRTV6+khbfF9dDKAimEalN/DLw/viI93Veyzsm2dMrAqP+8Zg/3h46p5sTUluCKY3C8tx6+xNwAAPz4fjEF6bnH092+O98Z3h5+OMUFauTliw8xgtFVrlK68UFHuo72tDUrKK479iC6eOLtoNDrO36mxHgGBp/trf28N7dwSsUlZeGVkJ3T0bIbjiVmITb6Hp/v54cCV26r5WjrLcTuvGKHdPPF/D3dC58oazWZyO5yYHwobGdAuwvj/lSUxnBjx8vAO+PPsLfwtSPMPpSbBRP2r5sg7DyPnfqnOrqDV253o09FLs4akv7+73vYlRssmk0F5sTNnREdEX7+LJ/tWVAGP6u6FTS8ORKfKP/DFj/bEs4P0VzvLZDKTg8kjvX2x+0KG6sq8uuV/D8CSJ3upruDqozF59Dsj4dDEtOPb398d1+8UANC8egGgdQ9jWOeWOKZ25bT22f5Yc+g6ZgzSX7XexVv7ig2o6CFTveZC30fT2ctZ9eW0Zlo/bI5Nxd+CWmvMM7KbF87+czTm/HQKh67e0VqH3M4W8mbGP5Pqx8fJ3haFJeVVE4x0c7a3s4F/ZVBRhmE/d0ekZt3H+F4+uHGvUHVCU/f9cwNwr6AUS3deBlBRY+LprDv07X51KPKKSlVBq/qxbiav+NttYmuDJrbAysl9MWfDKQAVJ6bXQjurnmQ+sb8f7hWWYGD7FnBxsNM4IatTDxoJH4UhZNk+3M4rVk37bfYgtPNoiuPX76JXZXCs/ll2aNkMHVpqt2XbMDMYv5xMxcIJVVfMs4d1wKAOLdDNxwVNbG0wqKMHBum5PavO1bEJcu6Xakwb3sUT0RFVf7dP92uNxDv5GNzRA39dzMCG4yl4fVRnTAjwxdXMiukA0K9tc8Qk3wMA+Lk74cm+rbDl1E2snNwX43v7wEYmw+ubz+gsx7SQtvghOhnje/todCT47/MDMKRTS4SP7oJNJ1MwvrdmQP/lpRD8O/KKqi2UUsx7obhXUKqqPQ4f1Rkr9lzRmCcirKsqnHT1cdZ7cTItxF/nd3SLpvY48o7x2p+3xnbRCMN2tjY4s3A0cotKMeTj/QaXHdXdC+uf7a8q28jK2t3IS5qjtm+ZPQi/n76JqQPbonm1MYWUt0/VA6wUMZwY4enigBPvhprcrdgQ9ZOJQxNbk0+A+ng6O2Bv+FBsOpmKRwNq7z5v86b22P7KENVrmUym0ZPGxsb0HiXGPNLbB75ujujkpX996lXLXbzrfoC26v/MSifmj0RadhEeW3lENe29R7qhTQsndGjZDA8baQRbna+bIxZNMK1njdKpBaNwv7RcZ5hTP0b6tHSW4+Xhum+ZuDg0waIJ3fHUqmi8NMxw7ZupIl8fhqMJd/WehEyx+aVB+PPsLfy9nx9+OZmqM5z4uDrCx9URe14bCk9nB4Ph2N7OxmANUPX/9PG9fTBnQ8Xv7T2aaZzkbW1kmGPCLSj1mlBdtRbKWzia7UVM+84Z1MEDgzpoBg8bG5lZbbyUvFzkWuGkOjtbG8yvHOJgaKeWmDHIHx09m0Emk6mCJQCM6OqJmOR7qpq/FU/3wYqn+6jefyKwFTLzijGgnXY5XxzaHq+M7IQWTe2RVVCCY9fv4m9BrTGksqbZ1bEJXhyqfYt3QDt3/PTCQK3pzg5NNNrsPRXUWiucPKjuJg4e6a2jptTVqQls1P4sqmf49TP64/DVO/hHfz+d56I21Wpx/Nyd8MrITlrzqVP+3Uo1oDCcmKA2gglQN920Ono6q74oLMnZwQ55RWVGe6lUJ5PJ9N5b16WPnxu+ndZPZ5VqXfN0doCnswOef6gdvjuciGkhbeHs0ETvyal35RWwvnYLNeGuIzidXjAKt/OLVbUjD6KjpzNOLxhV47/510d3wcwfYvCPyippH1dHPBXU+oHCibIbOGC85qxTDT+Dt8Z2Ud0S0LWNt8d2xYYTyQgfbbhRsj4h7VvgtdDO6OJdEcI//XsApq09AaCifYsu6uWoq0Hrqp8EP/5bAL45eA2P9THtYsfGRqb3M39xaHu0bu6IgXpCs42NDLOHawaM4++ORHZhqer2FAC0aCbHrlfNb5BqSCs3R3wxKRAuDqafAlu5OeJm9n2tHi3b5gzGD9HJWqOF6zO2hzdCu3lpfe+pX7xWr18c3sUTw7vo/27t5OWM1c/01VkTb4i9Cbf3LIXhhGrF/jeG4+KtXAzpZPrTp2sq1MI9Lt4d1w1PBLZCNx/DV0otmslxcn6owXYptaF5U3uN2h71XFGTh1c+SBgf1d0LpxaMQnOJNa4zZvawDqpwovP94R20TqTmkMlkGg2ah3ZuiSsfhqGkXKF3rCH1o1AfvShWP9MXffzcdLbTqYkmtjYmhxwlLxcHs0+wNVW9zZb6v4qu/4CoN4fjfmk5XKr1mgzwc8OnZnRksLO1wbfT+2lNf9BLmLE9pT36tbkYTupR9ZbvjYlHM7lGT46G6nRlI0NDbG1kqjYHxljiuUpt3J0wvEtLODs0gb2Ogf/qmq7anYe7emLf5UyjAxVaivpVq66G1XXB3s7G4PGpq8bH+jS2k1ttq2iDVD//T6Y+guJBNTOj5qi+SbdkjVA3H2eE9fSutysDMp++9iYNiUwmw/oZA4zPWI++mRqElKxCrW7u5qrLEX4XPNIdCZl5CG5Xs0bluvyz+sCJZnhpaHvEp+fW6DED1PBYYuT4p/q2xv7LmfVS420uhpN6JJPJsMrEh7oRNSZ2tjYPHEyAilGFo67cxsD2LXA1Iw8/RCdjnpGGf6Z6/qHaq9Vp59EUu14dArldzW/pNZXbaQzMR42bg50tmtrborhMUW8XsPZ2Nlpj/0gFwwkRNRhNbG2wcnJFA1KFQmDqwLa11nOsNqx+pi+W747Hl5P6PlAwqS+10YiaaoeNjQynFo6CEI27CYCpGE7I6v0tqDV+jb3xQA/so/pnqKeIpYzt6dOg2m48FdQa2fdLMKCddJ6pYinqg08+yACbD6IhBNr6IhP11fLmAeTm5sLV1RU5OTlwcan7cS7IuhSVluNEYhYGtHN/4LFniKjhiorPBACD3XbJPDU9f7PmhKyeQxPbRtHTiIgeDEOJdPDGFhEREUkKwwkRERFJCsMJERERSQrDCREREUkKwwkRERFJCsMJERERSQrDCREREUkKwwkRERFJCsMJERERSQrDCREREUkKwwkRERFJCsMJERERSQrDCREREUlKg3gqsRACQMWjl4mIiKhhUJ63ledxUzWIcJKXlwcA8PPzs3BJiIiIyFx5eXlwdXU1eX6ZMDfOWIBCocCtW7fg7OwMmUxWa+vNzc2Fn58fUlNT4eLiUmvrlZLGvo/cv4avse8j96/ha+z7WJf7J4RAXl4efH19YWNjekuSBlFzYmNjg9atW9fZ+l1cXBrlH5y6xr6P3L+Gr7HvI/ev4Wvs+1hX+2dOjYkSG8QSERGRpDCcEBERkaRYdTiRy+VYtGgR5HK5pYtSZxr7PnL/Gr7Gvo/cv4avse+jFPevQTSIJSIiIuth1TUnREREJD0MJ0RERCQpDCdEREQkKQwnREREJClWHU5WrlwJf39/ODg4IDg4GCdOnLB0kbB06VL0798fzs7O8PT0xOOPP474+HiNeYYPHw6ZTKbxM2vWLI15UlJSMH78eDg5OcHT0xNvvvkmysrKNOaJiopC3759IZfL0bFjR6xfv16rPLX9Gf3zn//UKnvXrl1V7xcVFWHOnDlo0aIFmjVrhqeeegoZGRkNYt+U/P39tfZRJpNhzpw5ABre8Tt48CAmTJgAX19fyGQybN26VeN9IQQWLlwIHx8fODo6IjQ0FFevXtWYJysrC1OmTIGLiwvc3Nzw/PPPIz8/X2Oes2fPYsiQIXBwcICfnx8+/vhjrbJs3rwZXbt2hYODA3r16oUdO3aYXRZz9q+0tBRvv/02evXqhaZNm8LX1xfTpk3DrVu3NNah65gvW7ZMEvtnbB8B4Nlnn9Uq/9ixYzXmaajHEIDO/0eZTIbly5er5pHyMTTlvCCl705TymKUsFIbN24U9vb2Yu3ateLChQti5syZws3NTWRkZFi0XGPGjBHr1q0T58+fF3FxcWLcuHGiTZs2Ij8/XzXPsGHDxMyZM0VaWprqJycnR/V+WVmZ6NmzpwgNDRWnT58WO3bsEB4eHiIiIkI1z/Xr14WTk5MIDw8XFy9eFF9++aWwtbUVu3btUs1TF5/RokWLRI8ePTTKfvv2bdX7s2bNEn5+fiIyMlLExMSIgQMHikGDBjWIfVPKzMzU2L89e/YIAGL//v1CiIZ3/Hbs2CHmz58vtmzZIgCI33//XeP9ZcuWCVdXV7F161Zx5swZ8eijj4p27dqJ+/fvq+YZO3asCAgIEMeOHROHDh0SHTt2FJMmTVK9n5OTI7y8vMSUKVPE+fPnxc8//ywcHR3F119/rZrnyJEjwtbWVnz88cfi4sWL4r333hNNmjQR586dM6ss5uxfdna2CA0NFZs2bRKXL18W0dHRYsCAASIoKEhjHW3bthWLFy/WOKbq/7OW3D9j+yiEENOnTxdjx47VKH9WVpbGPA31GAohNPYrLS1NrF27VshkMnHt2jXVPFI+hqacF6T03WmsLKaw2nAyYMAAMWfOHNXr8vJy4evrK5YuXWrBUmnLzMwUAMSBAwdU04YNGybmzZund5kdO3YIGxsbkZ6erpq2atUq4eLiIoqLi4UQQrz11luiR48eGstNnDhRjBkzRvW6Lj6jRYsWiYCAAJ3vZWdniyZNmojNmzerpl26dEkAENHR0ZLfN33mzZsnOnToIBQKhRCiYR+/6l/8CoVCeHt7i+XLl6umZWdnC7lcLn7++WchhBAXL14UAMTJkydV8+zcuVPIZDJx8+ZNIYQQX331lWjevLlq/4QQ4u233xZdunRRvX766afF+PHjNcoTHBwsXnrpJZPLYu7+6XLixAkBQCQnJ6umtW3bVnz22Wd6l5HK/gmhex+nT58uHnvsMb3LNLZj+Nhjj4mHH35YY1pDOobVzwtS+u40pSymsMrbOiUlJYiNjUVoaKhqmo2NDUJDQxEdHW3BkmnLyckBALi7u2tM/+mnn+Dh4YGePXsiIiIChYWFqveio6PRq1cveHl5qaaNGTMGubm5uHDhgmoe9f1XzqPc/7r8jK5evQpfX1+0b98eU6ZMQUpKCgAgNjYWpaWlGtvs2rUr2rRpo9qm1PetupKSEvz444947rnnNB5a2ZCPn7rExESkp6drbMfV1RXBwcEax8zNzQ39+vVTzRMaGgobGxscP35cNc/QoUNhb2+vsT/x8fG4d++eSftsSllqQ05ODmQyGdzc3DSmL1u2DC1atEBgYCCWL1+uUV3eEPYvKioKnp6e6NKlC2bPno27d+9qlL+xHMOMjAxs374dzz//vNZ7DeUYVj8vSOm705SymKJBPPivtt25cwfl5eUaBwkAvLy8cPnyZQuVSptCocCrr76KwYMHo2fPnqrpkydPRtu2beHr64uzZ8/i7bffRnx8PLZs2QIASE9P17lvyvcMzZObm4v79+/j3r17dfIZBQcHY/369ejSpQvS0tLw/vvvY8iQITh//jzS09Nhb2+v9aXv5eVltNxS2Dddtm7diuzsbDz77LOqaQ35+FWnLI+u7aiX1dPTU+N9Ozs7uLu7a8zTrl07rXUo32vevLnefVZfh7GyPKiioiK8/fbbmDRpksYD0l555RX07dsX7u7uOHr0KCIiIpCWloYVK1Y0iP0bO3YsnnzySbRr1w7Xrl3Du+++i7CwMERHR8PW1rZRHcPvv/8ezs7OePLJJzWmN5RjqOu8IKXvTlPKYgqrDCcNxZw5c3D+/HkcPnxYY/qLL76o+r1Xr17w8fHByJEjce3aNXTo0KG+i2mWsLAw1e+9e/dGcHAw2rZti19++QWOjo4WLFnd+O677xAWFgZfX1/VtIZ8/KxZaWkpnn76aQghsGrVKo33wsPDVb/37t0b9vb2eOmll7B06VJJDQmuzz/+8Q/V77169ULv3r3RoUMHREVFYeTIkRYsWe1bu3YtpkyZAgcHB43pDeUY6jsvNDZWeVvHw8MDtra2Wq2HMzIy4O3tbaFSaZo7dy7+/PNP7N+/H61btzY4b3BwMAAgISEBAODt7a1z35TvGZrHxcUFjo6O9fYZubm5oXPnzkhISIC3tzdKSkqQnZ2td5sNad+Sk5Oxd+9evPDCCwbna8jHT7kuQ9vx9vZGZmamxvtlZWXIysqqleOq/r6xstSUMpgkJydjz549Rh8rHxwcjLKyMiQlJRksu3q5Lbl/1bVv3x4eHh4af5MN/RgCwKFDhxAfH2/0fxKQ5jHUd16Q0nenKWUxhVWGE3t7ewQFBSEyMlI1TaFQIDIyEiEhIRYsWUU3s7lz5+L333/Hvn37tKoRdYmLiwMA+Pj4AABCQkJw7tw5jS8T5Rdq9+7dVfOo779yHuX+19dnlJ+fj2vXrsHHxwdBQUFo0qSJxjbj4+ORkpKi2mZD2rd169bB09MT48ePNzhfQz5+7dq1g7e3t8Z2cnNzcfz4cY1jlp2djdjYWNU8+/btg0KhUAWzkJAQHDx4EKWlpRr706VLFzRv3tykfTalLDWhDCZXr17F3r170aJFC6PLxMXFwcbGRnUrRMr7p8uNGzdw9+5djb/JhnwMlb777jsEBQUhICDA6LxSOobGzgtS+u40pSwmMbnpbCOzceNGIZfLxfr168XFixfFiy++KNzc3DRaMlvC7Nmzhaurq4iKitLo0lZYWCiEECIhIUEsXrxYxMTEiMTERLFt2zbRvn17MXToUNU6lF3GRo8eLeLi4sSuXbtEy5YtdXYZe/PNN8WlS5fEypUrdXYZq+3P6PXXXxdRUVEiMTFRHDlyRISGhgoPDw+RmZkphKjogtamTRuxb98+ERMTI0JCQkRISEiD2Dd15eXlok2bNuLtt9/WmN4Qj19eXp44ffq0OH36tAAgVqxYIU6fPq3qrbJs2TLh5uYmtm3bJs6ePSsee+wxnV2JAwMDxfHjx8Xhw4dFp06dNLqhZmdnCy8vLzF16lRx/vx5sXHjRuHk5KTVTdPOzk588skn4tKlS2LRokU6u2kaK4s5+1dSUiIeffRR0bp1axEXF6fxP6ns4XD06FHx2Wefibi4OHHt2jXx448/ipYtW4pp06ZJYv+M7WNeXp544403RHR0tEhMTBR79+4Vffv2FZ06dRJFRUUN/hgq5eTkCCcnJ7Fq1Sqt5aV+DI2dF4SQ1nensbKYwmrDiRBCfPnll6JNmzbC3t5eDBgwQBw7dszSRRIAdP6sW7dOCCFESkqKGDp0qHB3dxdyuVx07NhRvPnmmxrjZAghRFJSkggLCxOOjo7Cw8NDvP7666K0tFRjnv3794s+ffoIe3t70b59e9U21NX2ZzRx4kTh4+Mj7O3tRatWrcTEiRNFQkKC6v379++Ll19+WTRv3lw4OTmJJ554QqSlpTWIfVO3e/duAUDEx8drTG+Ix2///v06/yanT58uhKjoHrlgwQLh5eUl5HK5GDlypNZ+3717V0yaNEk0a9ZMuLi4iBkzZoi8vDyNec6cOSMeeughIZfLRatWrcSyZcu0yvLLL7+Izp07C3t7e9GjRw+xfft2jfdNKYs5+5eYmKj3f1I5bk1sbKwIDg4Wrq6uwsHBQXTr1k0sWbJE48Ruyf0zto+FhYVi9OjRomXLlqJJkyaibdu2YubMmVohtqEeQ6Wvv/5aODo6iuzsbK3lpX4MjZ0XhJDWd6cpZTFGVrnjRERERJJglW1OiIiISLoYToiIiEhSGE6IiIhIUhhOiIiISFIYToiIiEhSGE6IiIhIUhhOiIiISFIYToiIiEhSGE6IiIhIUhhOiIiISFIYToiIiEhSGE6IiIhIUv4fnZyLpoNCDP8AAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(numberOfIterations, losses)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f56b1387-e041-4c3a-9a28-be75ce7ce43f",
   "metadata": {},
   "source": [
    "I now also invite you to beat this loss and make this model more optimized...\\\n",
    "Its way more fun, to learn when we take on a challenge on ourselves..."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77e30334-6ee3-4f4d-b3d3-98ec4b024bc6",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "# Sampling from our model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "id": "330589b7-85b0-4b7d-978c-f004c50fbaff",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "k.\n",
      "orunyrehihaanjhiaitnth.\n",
      "noh.\n",
      ".\n",
      ".\n",
      "kahiaeineihhdmirisennaaaknaagaak.\n",
      "narhrhsehieksa.\n",
      "agaai.\n",
      "n.\n",
      "jti.\n",
      "aaeavnav.\n",
      "han.\n",
      "kianhhonte.\n",
      "h.\n",
      "baktvavaa.\n",
      "naj.\n",
      "animevmeketonynaninhe.\n",
      "aio.\n",
      "aia.\n",
      "yi.\n"
     ]
    }
   ],
   "source": [
    "# We will define a generator to give the same result on your machine, as of my machine\n",
    "generator = torch.Generator().manual_seed(6942069420)\n",
    "\n",
    "# We define a Block Size based on the number of characters we feed are going to feed to predict the next one\n",
    "inputBlockSize = 3\n",
    "\n",
    "# We will define a generator to give the same result on your machine, as of my machine\n",
    "numberOfWordsToSample = 20\n",
    "\n",
    "# We iterate over the number of words we want to predict\n",
    "for _ in range(numberOfWordsToSample):\n",
    "    # We define a output list to append the next character and print it at the end\n",
    "    output = []\n",
    "    # We define the block for each iteration and fill it with 0 values -> [0, 0, 0]\n",
    "    block = [0] * inputBlockSize\n",
    "\n",
    "    # We will now iterate over each word's characters\n",
    "    while True:\n",
    "        # We would create an output embedding that would be based on the block\n",
    "        embedding = embeddingLookUpMatrix[torch.tensor([block])]\n",
    "        # We would be doing a foward pass on the above embedding vector \n",
    "        hiddenLayerStates = torch.tanh(embedding.view(1, -1) @ weightsOfHiddenLayer + biasesOfHiddenLayer)\n",
    "        logits = hiddenLayerStates @ weightsOfFinalLayer + biasesOfFinalLayer\n",
    "        probabilities = F.softmax(logits, dim=1)\n",
    "        # We can now sample the next character\n",
    "        index = torch.multinomial(probabilities, num_samples=1, generator=generator).item()\n",
    "        # We then take the block, crop it 1 size from the left and append the next index to it (sliding window of name)\n",
    "        block = block[1:] + [index]\n",
    "        # We then append the sampled character to the output\n",
    "        output.append(index)\n",
    "        # If we hit '.' end token, we will break out from the loop\n",
    "        if index == 0:\n",
    "            break\n",
    "    # We print generated name out\n",
    "    print(''.join(itos[index] for index in output))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9fd27095-55d7-403d-bda4-62ebdc0315fa",
   "metadata": {},
   "source": [
    "As we can see that our names have started to sound a lot more name like, other than before...\\\n",
    "So we are definately making progress but we can improve our model by quite a bit in the next notebooks..."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}