week_03_2_attention_basics_and_tensorboard.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Practice: Attention Basics\n",
    "Original notebook is provided by Udacity at [github](https://github.com/udacity/deep-learning-v2-pytorch/blob/master/attention/Attention_Basics.ipynb).\n",
    "\n",
    "In this notebook, we look at how attention is implemented. We will focus on implementing attention in isolation from a larger model. That's because when implementing attention in a real-world model, a lot of the focus goes into piping the data and juggling the various vectors rather than the concepts of attention themselves.\n",
    "\n",
    "We will implement attention scoring as well as calculating an attention context vector.\n",
    "\n",
    "## Attention Scoring\n",
    "### Inputs to the scoring function\n",
    "Let's start by looking at the inputs we'll give to the scoring function. We will assume we're in the first step in the decoding phase. The first input to the scoring function is the hidden state of decoder (assuming a toy RNN with three hidden nodes -- not usable in real life, but easier to illustrate):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "dec_hidden_state = np.array([5,1,20])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's visualize this vector:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:>"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAJAAAAEYCAYAAACz0n+5AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAR2UlEQVR4nO2de7BV1X3HP19Qx8gVMFrgCviYSkjVFjRI46AT3wWDYjuUQmciWqbYTGzjNNORUo0ybQwzTsxQiRFaqDhjMUaDMgkiDGPH0IkGJaDiBSWIesMrJuEVQ+iFX//YGzxcz+Pes84+6+59f5+ZPffstfc+a234zG+t/Vi/IzPDceqlT+wGOPnGBXKCcIGcIFwgJwgXyAnCBXKCcIEKhKThkl6U1CZpk6SvpuWflrRa0jvp3zMqHD9e0hZJWyXN6lKdfh+oOEhqBVrNbL2k04HXgFuA24Bfm9ncVIwzzOzuTsf2Bd4GrgfagXXANDN7q1qdHoEKhJntNLP16ecDQBswFJgELEl3W0IiVWfGAlvNbJuZHQaeTI+rigtUUCSdB1wCvAIMNrOdkEgGDCpzyFDgg5L19rSsKicFt7Q23kd+jEpX5mhOt/9t7uf+O4CZJUULzWzhCZVILcAzwF1mtl86odoutS2lZvuaIRBbX9zajGp6PBdcfcGJBV36fz0RO2oLgYWVtks6mUSeJ8zsB2nxbkmtZrYzHSftKXNoOzC8ZH0YsKNWe7wLi4ikbi81vk/AIqDNzB4q2bQcmJ5+ng48V+bwdcAISedLOgWYmh5XFRcoJqpjqc444EvANZI2pMuNwFzgeknvkFxlzQWQdLakFQBm1gHcCbxAMvh+ysw21aqwKV2YU54ujk26jJmtpbJm15bZfwdwY8n6CmBFd+p0gWLSWH+i4AJFpNERKAYuUEzy748LFJMiRCC/CnOC8AgUkSJEIBcoJvn3xwWKigvkhOBdmBNG/v1xgWKiPvk3yAWKiHdhThj598cFiolHICeM/PvjAkXFBXJC8C7MCSP//rhAMfEI5ISRf39coJioAAa5QDHJvz8uUEx8DOSEkYE/khYDE4E9ZnZxWvY9YGS6y0Bgr5mNLnPsduAAcAToMLMxtepzgSKSUQR6DJgPPH6swMz+qqTObwH7qhx/tZl92NXKXKCYZOCPmb2Upnb5ZHWJsVOAaxpVn8/KiEnj58bX4kpgt5m9U2G7AaskvSZpZoV9TsAjUETq6cLS/9iq+YGqMA1YWmX7ODPbIWkQsFrSZjN7qdoXukAxqSc/kFXPD1SxKukk4C+Az1X57h3p3z2SlpGkvasqkHdhEWl0fqAaXAdsNrP2Cm3plybmRFI/4AbgzVpf6gLFJIMxkKSlwE+AkZLaJc1IN02lU/dVmh8IGAyslbQR+CnwIzNbWau+wndht8++nU+d+in69OlD3z59mTd7XuwmHSeLl+rNbFqF8tvKlB3PD2Rm24BR3a2v8AIBfPMfv8mAlgGxm/EJ/E60E0b+/aktkKTPkiScHkpyn2AHsNzM2jJuW0OQxL3z7gXBhCsnMOHKCbGb9DFFF0jS3ST3Dp4kGVhBkv51qaQnzWxuxu0L5sF/epAzB57J3v17uWfePQwfMpyLR1wcu1lA73idYwZwkZn9X2mhpIeATaTZPjtTerNrwYIFXDOiYXfOu82ZA88EYGD/gVw++nK2vLulxwhUAH9qCnQUOBt4r1N5a7qtLJ1udlmsROOHfn+Io3aU0049jUO/P8T6tvVM+2LZi5Qo9IZB9F3AmjS/8LHfUTgHuIAkp3CP5jf7f8M3Hv0GAEeOHuELl32BMRfVfEOheeTfn+oCmdlKSZ8huaU9lOSU24F1ZnakCe0LovUPWpl/7/zYzahIb4hAmNlR4OUmtKX3kX9//D5QTHpFBHIyJP/+uEAx8QjkhJF/f1ygqLhATgjehTlh5N8fFygmHoGcMPLvjwsUE49AThj598cFiklveKHMyZICTKpygSLiYyAnjPz7U4Qgml+ymNosabGkPZLeLCm7X9IvJG1IlxsrHDte0hZJWyXN6so5uEAxySa9y2PA+DLl3zaz0emyovNGSX2B7wATgAuBaZIurFWZCxSRLCJQmo7l13U0Zyyw1cy2mdlhkqlck2od5ALFpLkJpu6U9HraxZ1RZvtQPp44Acm770NrfakLFJF6IpCkmZJeLVm6kknsu8AfAqOBncC3yjWnTJnV+mK/CotJkxJMmdnu41VK/wH8sMxu7cDwkvVhJNPYq+IRKCLNSjAlqbVk9c8pnzhqHTBC0vmSTiHJJ7S81nd7BIpJNnmilwJXAWdJagfuA66SNJqkS9oO3JHuezbwn2Z2o5l1SLoTeAHoCyw2s0216nOBYpJNmt9yc7cXVdj3eIKpdH0F8IlL/Gq4QBHxRxlOGPn3xwWKSREikF+FOUF4BIpIESKQCxST/PvjAsXEI5ATRv79cYGi4gI5IWTxUwfNxgWKiI+BnDDy709zBLrg6guaUU3u8AjURQ7uO9iMano8LQNaTizIvz/ehcXEI5ATRv79cYFi4hHICSP//vjrHE4YHoEi4l2YE0b+/XGBYuIRyAkj//64QDHJIgJJWgxMBPaY2cVp2YPATcBh4OfA7Wa2t8yx24EDwBGgw8xq/ryjX4XFpHn5gVYDF5vZnwBvA/9c5fir0xxCXfptUBcoIs3KD2Rmq8ysI119mSRxQkNwgWLS3PxAx/gb4PkK2wxYJem1LqaN8TFQVOoQIv2PLf3PXZimfOnKsf8CdABPVNhlnJntkDQIWC1pcxrRKuICRaSeQXQ9+YHSuqaTDK6vNbOyiaPSZAuY2R5Jy0jS3lUVyLuwmDSpC5M0HrgbuNnMPqqwTz9Jpx/7DNxA+TxCJ+ACRSSjNL9LgZ8AIyW1S5oBzAdOJ+mWNkh6NN33bEnH0rkMBtZK2gj8FPiRma2sVZ93YRHJYlZGvfmBzGwbMKq79blAMfE70U4IRXgW5mMgJwiPQBEpQgRygWKSf39coKi4QE4I3oU5YeTfHxcoJh6BnDDy748LFBOPQE4Y+ffHBYqJRyAnjPz74wJFxQVyQihCF1b4p/Fz/nUO1/3ZdUyZOiV2Uz5JnFkZDaXwAt30xZt4eN7DsZtRlmb9ZmqWFF6gSy+9lAH9B8RuRnkKEIF8DBQR9UQjukndEUjS7VW2zZT0qqRXFy7s9hSmXoP6qNtLTyMkAs0B/qvchk6T38zzRFeg5/nQbaoKJOn1SptI5hE5AfTEQXF3qdWFDQZuJckt03n5VbZNawyz75nNbTNuY/t725kwcQLPPvds7CZ9TAaDaEmLJe2R9GZJ2aclrZb0Tvr3jArHjpe0RdJWSbO6cgq1urAfAi1mtqFMZf/TlQpi88C/PRC7CZXJJgA9RjIT9fGSslnAGjObm4oxi2Sq88dNkfoC3wGuB9qBdZKWm9lb1SqrGoHMbIaZra2w7a9rnIhTg2blBwImAUvSz0uAW8ocOhbYambbzOww8GR6XFUKfx+oR9O8+0CDzWwnQPp3UJl9hgIflKy3p2VVcYEiUk8EKr1Fki5dSgTVleaUKSubBqYUv5EYkzoiSp35gXZLajWznZJagT1l9mkHhpesDwN21Ppij0ARaeKzsOXA9PTzdOC5MvusA0ZIOl/SKcDU9LiquEAxyeYyvlx+oLnA9ZLeIbnKmpvuezw/UJqE807gBaANeMrMNtWqz7uwiGRxI7FCfiCAa8vsezw/ULq+AljReb9quEAxyf+NaBcoKi6QE0IRXudwgWKSf39coJgU4Wm8CxST/PvjAsXEI5ATRv79cYFi4hHICSP//rhAMemJsyy6iwsUk/z74wLFxMdAThj598cFiolHICeM/PvjAsXEI5ATRv79cYFi4u8DOWHk3x8XKCoukBOCD6KdMPLvj08sjEmjZ6ZKGilpQ8myX9Jdnfa5StK+kn2+HnIOHoFi0uAIZGZbgNFwPN/PL4BlZXb9sZlNbESdLlBEMh4DXQv83Mzey7IS78Jikm1+oKnA0grbLpe0UdLzki6qq+0pLlBEssoPlGbXuBn4fplq1wPnmtko4GHg2ZBz8C4sJtnlB5oArDez3WWO31/yeYWkRySdZWYfdr81TRKoZUBLM6rJHRmOgaZRofuSNATYbWYmaSxJL1R3xt2mCDRHc5pRTY/nPrvvxIIM/JF0GkkOoDtKyv4OwMweBSYDX5bUAfwOmGpmNVPZVcK7sIhk8VK9mX0EnNmp7NGSz/NJ0gA3BBcoJgW4E+0CRaQIz8L8Mt4JwiNQRIoQgVygmOTfHxcoJh6BnDDy748LFBUXyAnBuzAnjPz74wLFxCOQE0b+/XGBYuIRyAkj//64QDHxCOSEkX9/XKCYeARywsi/P/4+kBOGR6CIeBfmhJF/f1ygmPhPHThBeBfmhJHNxMLtwAHgCNBhZmM6bRcwj+T34j8CbjOz9fXW5wJFJMMIdHWVue4TgBHp8qfAd9O/deGX8THJNr1LJSYBj1vCy8BASa31fpkLFJNsBDJglaTXyqV+AYYCH5Sst6dldeFdWETq6cJSKUrFWJimfDnGODPbIWkQsFrSZjN7qfQrynytJ1fIJRnkBzKzHenfPZKWAWOBUoHageEl68OAHd1vSYJ3YRHJIEtrP0mnH/sM3AC82Wm35cCtSvg8sM/MdtZ7Dh6BYtL4i7DBwLJUtJOA/zazlZ3yA60guYTfSnIZf3tIhS5QRBp9GW9m24BRZcpL8wMZ8JVG1ekCxST/N6KLJ1D/Yf255fFbaBnSgh011i9czyv//gqnnnEqk783mYHnDWTv9r08PeVpDu09FLWtRXiUUbhB9NGOo6z62ioeufARFn1+EZd95TLO+qOzuGLWFby75l3mf2Y+7655lytmXRG7qYWgcAId3HWQXT/bBcDhg4f5Zdsv6T+0PyMnjWTjko0AbFyykZG3jIzZTKDxV2ExqCmQpM9KulZSS6fy8dk1qzEMOHcArZe00v5KOy2DWzi46yCQSNZvUL/IrSPWo4yGUlUgSf8APAf8PfCmpEklmx/IsmGhnNzvZKY8M4WVd63k8IHDsZtTngIIVGsQ/bfA58zsoKTzgKclnWdm86hyOqW32xcsWNCotnaZPif1YcozU3jjiTfYvGwzAAd3H6RlSBKFWoa08Ns9v216uzrTE7uk7lKrC+trZgcBzGw7cBUwQdJDVBHIzBaa2RgzGzNzZrnnedly86Kb+bDtQ17+9svHy95e/jajpie3SEZNH8WW57Y0vV2foBdEoF2SRpvZBoA0Ek0EFgN/nHXj6mH4uOGMunUUu1/fzR0/S5K1r5m9hrVz1zL5qclcMuMS9r2/j+//ZbnfIWkuRYhAqpblXtIwkrfadpXZNs7M/rcLdZj/1EHCfXbfCca8/+r73X4Kfs6Yc3qUdVUjkJm1V9nWFXmcKvhL9U4QRejCXKCY5N8fFygqLpATgndhThj598cFiolHICeM/PvjAsVEBTDIBYpJ/v1xgWLiYyAnjPz74wLFxCOQE0b+/SneS/W5osEvlEkaLulFSW2SNkn6apl9rpK0T9KGdPl6yCl4BIpIBl1YB/A1M1ufzpF/TdJqM3ur034/NrOJjajQI1BMGhyBzGznsXR1ZnYAaCMg909XcIEiUs+8MEkzJb1aspR96TydBHEJ8EqZzZdL2ijpeUkXhZyDd2ExySA/EEA6h+8Z4C4z299p83rg3PT99huBZ0nyJdaFR6CIZDEzVdLJJPI8YWY/6LzdzPaXzLRZAZws6ax6z8EFiknjr8IELALazOyhCvsMSfdD0lgSB35V7yl4FxaRDK7CxgFfAt6QtCEtmw2cA8fzBE0GviypA/gdMNWqTc2pgQsUkwbHfzNbS404ZWbzgfmNqtMFioi/zuGEkX9/XKCY+MNUJ4z8++MCxcQjkBNG/v1xgWLiEcgJI//+uEAx8QjkhJF/f1ygqLhATgjehTlh5N8fFygmRYhAVbO0FglJMzv9tqjTAHrTG4nNz3jeC+hNAjkZ4AI5QfQmgXz8kwG9ZhDtZENvikBOBhReIEnjJW2RtFXSrNjtKRqF7sIk9QXeBq4H2oF1wLQy2SqcOil6BBoLbDWzbWZ2GHgSmFTjGKcbFF2gocAHJevtZJzupLdRdIHKPWwqbp8dgaIL1A4ML1kfBuyI1JZCUnSB1gEjJJ0v6RRgKrA8cpsKRaFf5zCzDkl3Ai8AfYHFZrYpcrMKRaEv453sKXoX5mSMC+QE4QI5QbhAThAukBOEC+QE4QI5QbhAThD/D2eU0PkuFWVTAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 108x324 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "# Let's visualize our decoder hidden state\n",
    "plt.figure(figsize=(1.5, 4.5))\n",
    "sns.heatmap(np.transpose(np.matrix(dec_hidden_state)), annot=True, cmap=sns.light_palette(\"purple\", as_cmap=True), linewidths=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our first scoring function will score a single annotation (encoder hidden state), which looks like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "annotation = [3,12,45] #e.g. Encoder hidden state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:>"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAIYAAAEYCAYAAACZYo4WAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAOUUlEQVR4nO3df5BV5X3H8fd3F6iuuxFpgCJEcFJmTAYaqobB0jY2kMQkpGl+TqQS0zCimdqY1rQ1jp3EOmNpa2o6aZuwGhJrRENTUx2mRtCEoYkRqcYqilGTaqSAFIvEberiLt/+cQ94d3nuPZzl3PM89+7nNXOHu+fce/bh8uF5zo/7nK+5OyKjdcVugKRJwZAgBUOCFAwJUjAkSMGQIAWjA5lZt5n90Mw2ZD9/zsz+y8wezh7vytvGhNY3UyK4DNgBvKZu2fXuft2xbkA9Rocxs1nAu4Ebj2c7Ckbn+QLwJ8ChUcsvNbNHzGytmZ2St5EqhhKdc3+VjfhpnRX+bOx3uRhYVbeo3937AcxsGbDX3R80s3PrXvMl4Bpq/xbXAJ8HPt7091RwrcQHDuxv9e9oC70nnzIqGF3FP/zlh6zRKjP7C2AFMAScQG0f43Z3v6DuNXOADe4+r9mv0VASk1nxRxPu/hl3n+Xuc4CPAN9x9wvMbEbdy94HbM9rmo5Komr+D12ivzKzBdSGkmeAi/PeoGBE1bpguPtmYHP2fEXR9ysYUVXWYxSmYMSUs88Qk4IRVbrB0FGJBKnHiElDiYQpGBLgYwhGVVFSMGLSUCJhCoYEKRgSYumeLVAwolKPIUEKhoToqETCFAwJUjAkSMGQkIT3MdI9kJao1GNEpR5DQkqePvDqZo+a1DzFzDaZ2VPZn7kz0RSMqGwMj2NyeFLzYVcA97r7XODe7OemFIyoyg9Gg0nN7wVuyp7fBPxO3na0jxFTa45KvkBtUnNf3bLp7r4bwN13m9m0vI2ox4iqeI9hZqvM7N/rHkcmONdPaj7elqnHiKp4j5HNbO9vsHox8NvZHXNOAF5jZl8HnjezGVlvMQPYm/d71GNE5FjhR9PtNZjUDNwJXJi97ELgjry2qceIqbozn6uB9Wa2Evgp8KG8NygYUVU2qfkFYEmR9ysYUaV75lPBiCnhi2gdHYzBwUEuuvgTHDx4kOHhYZYseSuXrLoodrPqpLvv39HBmDRpEl/+h7+jp6eHV4aGWHnRKhafcw7z5ze9/ZTQ4cEwM3p6egAYGhpiaGgorWG9nYcSMzuD2rn2mdTu4bQLuNPddzR9YyKGh4e54KMf47mdO/nwBz/A/Hkp9RbpBqPpIGdmfwrcRu1v8ACwLXt+q5nlXqFLQXd3N7fecjN3bbiT7Y8/ztM//nHsJtVp2dXV45a397MSeLO7r3b3r2eP1cDCbF1Q/fn8/v5GZ2+r1dfXx9lnnsl9P7g/dlNe1aLvY5Qhbyg5BJwKPDtq+QyOviXxEaPO50e7Aez+/fuZMGECfX19vPzyy2x9YBsXfrTwDexaKN2hJC8YnwLuNbOngOeyZacBvwxc2sJ2lWLfvn189uprGD40jB9yli5dwm/+xq/HbladdIORe8toM+uiNnTMpPY32Qlsc/fhY/wdumV0ZvQtow9tmFf4ltFdy7ZXkqbcoxJ3PwQkNDB3knR7jI4+j5E+BUOCFAwJaeczn9JKCoYEKRgS4AkPJel+IUCiUo8RlXoMCSr36qqZnWBmD5jZf5jZY2Z2dbZclZrbSvn7GIPAW919wMwmAt8zs7uydYUqNSsYUZUbDK9d+BrIfpyYPcZUP1VDSVQtme3ebWYPU5uGuMndt2arClVqVjBisq7Cj2aTmgHcfdjdFwCzgIVmNo9apebXAwuA3dQqNTeloSSq0ic117/uRTPbDJxXv29hZjcAG/Lerx4jppK/2mdmU81scvb8RGAp8IQqNbed0o9KZgA3mVk3tf/06919g5ndrErNbaX0o5JHgF8NLFel5vaS7plPBSOmhC+iKRhRKRgSpGBISLq5UDDiSjcZCkZUCoaE6KhEQsZS270qCkZUCoYEKRgSon0MCVMwJEjBkBANJRKmYEhQut+sVDBiSngoSTeyElUlPUbvybnzW8apdHuMSoLx82fvreLXJK9n9qgiQxpKJKyy2e4q4d1eSp+7eni2+5uoTUc8z8wWoRLebabkmWheE5rtXriEt4IRVWWz3UeU8AZUwjttxYMxxtnuhekEV1TVzHZHJbzbTEWz3VEJ7/bSgu98Nprt/gNUwrudVDbbXSW820rCZz4VjKgUDAlSMCRIwZCQdHOhYMSVbjIUjKgUDAnR4aqEpXtFQsGIST2GhCkYEqRgSJCCISHax5AwBUOCFAwJ0VAiYQqGBIyp7mVFFIyo0u0x0j1ZPx6UP33gdWb2XTPbkU1qvixbrhLe7aX0HmMIuNzdHzKzPuBBM9uUrVMJ7/ZR+vSB3dQK7uLuL5nZDmDmWLaloSSmkoeSkZu2OdTmmKiEd/spf1IzgJn1Av8MfMrdf4ZKeMPnPn8zW+5/lCmT+/jmDX8GwPX9t7Pl/keZOLGbWTOmcvWnV9DX2xO5pdCKSc1mNpFaKG5x99uz9zxft358lvB+z9sW8ffXXjpi2aIzz+CfbriK9WuuYvasaay97e5IrRul/KMSA74C7HD3v6lbrhLeZ/3KXHbteWHEsnPOfuOR5/PPOJ17/u2HVTergdKPShYDK4BHs5unAFwJnK8S3jnuuPs+3v6Ws2I3I1P6Ucn3Gmz0X4tua8xDiZn9XpN1R3aQ+vtz7/FRmRvX3UV3dzfvWrIwdlNqrKv4oyLH02NcDXw1tGLUDpKncJ/POzfez5at21nzl5dhyVzVTKUdR2saDDN7pNEqYHr5zWmN7297jK+t38iN1/0hJ54wKXZz6rRpMKj9478D2D9quQH3taRFx+mKa9fy4CNP8uKBAd6x/EouWfFuvvqNjRw8+AqfuOKLAMx/wxyuumx55JaSci5yg7EB6HX3h0evyG78lZzVV378qGXve+fiCC05Fukmo2kw3H1lk3UJ/Jdrd20aDGk1BUNCkjk6OpqCEZWCIQGq7S5hGkokTMGQIAVDQjSUSJiCIUEKhgQpGBKifQwJSzcYHfctcSmHghFTdZOaVam5vRSfiZbj8KTmNwCLgN83szeiSs3tpmsMj8bcfbe7P5Q9fwk4PKm5cKVm7XzG1MKjklGTmkdUajYzVWpOW2WTmgtTjxFVNZOaUaXmNlPRpGZUqbndVDapeTWq1NxOKpvUDKrU3D5c10okTMGQIAVDghQMCUk3FwpGXOkmQ8GISsGQEB2uSpiCIUEKhgQpGBIy3vcxemYXun4zjozzYLAu3Q+gUstHl8dL93PRUBJThbeALkrBiEo9hoSM951PaSTdYKQ7yElU6jGiUo8hAW5W+JEnK5+518y21y0rXKlZwYiq9EnNAF8Dzgssv97dF2SP3FJYCkZU5QfD3bcA/3O8LVMwYmphpeYAVWpuH62Z1BygSs3tpfxJzQ3eo0rNbaWioUSVmttO+ecxzOxW4FzgtWa2E/gscK4qNbeV8oPh7ucHFn+l6HYUjKjSPfOpYMSkq6sSpmBIkIIhIQkPJTqPIUHqMaJKt8dQMGLSt8QlTD2GBCkYEpLwUYmCEZFqu0sDCoaEaCiRMAVDghQMCVIwJET7GBKmYFRu+BB84IbTmN43xJrlu/ji5l9k/UMnM6VnCIA/WvICb5n7v5Fbma6ODcY/bp3M6197kIHBVy9UfWzRflb+2v6IrRqlBUOJma0FlgF73X1etmwK8A1gDrVviX/Y3Zt+EOle3jsOe342gc1P9fLBMw/EbkqOyiY1l1+p2czOMLMlWS3P+uWhGdVJuPbbU/njpf9N16jP8ZYHJvOeL83mM3dM58D/pfB/orJJzYUrNTf9dMzsk9RKMf4BsN3M3lu3+trcVkbw3SdPYspJw8w7dXDE8vPPfpFNn/xP7rjkWab1DrF649RILazXkh4jZESlZiC3UnPePsZFwFnuPpCVhP6mmc1x979t1spsou0qgDVr1rCqt9Ery/fQT0/kOz86iS1Pnc7gkDEw2MWnb/8lrnv/niOv+dBZB7hk3czqGtXIGPYx6j/bTH82n7VUecHodvcBAHd/xszOpRaO2TQJxqiJt8663Blxpbl86T4uX7oPgK3PnMja+07huvfvYe9L3UzrGwbgnh29zJ022GwzFalmUjNjqNScF4w9ZrbA3R/OGjVgZsuAtcD8go2L6q/vmcoTe34BgJmTX+HPlz2f844qVHYe43Cl5tUcY6Vmcx99G+O6lWazgCF33xNYt9jdv38MjXLdMjqz3Ed8ED/fta3xh99Az6lvbvph1k9qBp6nNqn5X4D1wGlklZrdveldd5r2GO6+s8m6YwmFNFX+kVGDSc2gSs1tRNdKJEzBkAB951MaUDAkJOF9jBQuGEiC1GNElW6PoWDElPBQomBEpWBIkIIhQQqGhGgfQ8IUDAlSMCREQ4mEKRgSpGBIiIYSCVMwJEjBkIqY2TPAS8AwtW/4nz2W7SgYMbXultG/5e77jmcDCkZEKX/nU9/giqk15TUd2GhmDx5jsd4g9RhRtWRS82J332Vm04BNZvZEdmuEQhSMqMqf1Ozuu7I/95rZt4CFQOFgaCiJqtz7Y5jZSWbWd/g58HaOoSpziHqMmMo/8zkd+JbVtjsBWOfu3x7LhhSMqMoNhrv/BHhTGdtSMKJK93BVwYhJF9EkTMGQIAVDQhIeSnQeQ4LUY0SVbo/R9K59ncTMVrXiRqmdajwNJWO+0jgejadgSAEKhgSNp2Bo/6KAcbPzKcWMpx5DCuj4YJjZeWb2IzN72sxySz5JTUcPJWbWDTwJvA3YCWwDznf3x6M2rA10eo+xEHja3X/i7geB26jVB5McnR6MmcBzdT/vzJZJjk4PRuhiROeOnSXq9GDsBF5X9/MsYFektrSVTg/GNmCumZ1uZpOAj1CrDyY5Ovqyu7sPmdmlwN1AN7DW3R+L3Ky20NGHqzJ2nT6UyBgpGBKkYEiQgiFBCoYEKRgSpGBIkIIhQf8PdhBVQRuJyswAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 108x324 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Let's visualize the single annotation\n",
    "plt.figure(figsize=(1.5, 4.5))\n",
    "sns.heatmap(np.transpose(np.matrix(annotation)), annot=True, cmap=sns.light_palette(\"orange\", as_cmap=True), linewidths=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### IMPLEMENT: Scoring a Single Annotation\n",
    "Let's calculate the dot product of a single annotation. NumPy's [dot()](https://docs.scipy.org/doc/numpy/reference/generated/numpy.dot.html) is a good candidate for this operation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "927"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def single_dot_attention_score(dec_hidden_state, enc_hidden_state):\n",
    "    # TODO: return the dot product of the two vectors\n",
    "    return np.dot(dec_hidden_state, enc_hidden_state)\n",
    "    \n",
    "single_dot_attention_score(dec_hidden_state, annotation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Annotations Matrix\n",
    "Let's now look at scoring all the annotations at once. To do that, here's our annotation matrix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "annotations = np.transpose([[0.3,1,4], [5,2,9], [0.1,3,5], [4,3,5.3]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And it can be visualized like this (each column is a hidden state of an encoder time step):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVQAAAD8CAYAAAAoqlyCAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAXXElEQVR4nO3deXRV5bnH8e+TAUJAZkIpiIJVUKkiIMULioh6KbVqrS4Vhzqm2opYZ9Zta7W1eNVaEZaupuqqWoc63tvSLq+0itQiKmBaEwPVWhUEDUYGmTM8948cmSQnJ+ZN3pOd32etvZpkv3vvx70Ov757v+fd29wdERFpvpzYBYiIJIUCVUQkEAWqiEggClQRkUAUqCIigShQRUQCUaCKiDTAzKaZWZmZlZvZFY21V6CKiOyBmQ0DLgZGA4cCJ5jZ/um2UaCKiOzZgcBCd9/k7jXAi8C30m2Q1wpFaSqWiGTKmr2HRyzjzLGz+C5QvNOfSty9JPVzGXCzmfUCNgOTgUXp9tcagcqGtatb4zBZr0v3PlQvnBq7jKyQP2YWG9atiV1GVujSrQdbS2fELiMrdBw+PcBeMs9k97oSoGTP67zCzP4bmAtsAP4O1KTbny75RSRZzDJfGuHu97n7CHc/CvgEeCtd+1bpoYqItJ7m3zXYviezInevNLOBwCnAEenaK1BFJGHCBSrwVOoeajXwfXdPe59KgSoiyWLh7mS6+5FNaa9AFZGECdpDbRINSomIBKIeqogkSwaj9y1FgSoiCaNAFREJwpsQqKGjV4EqIskScJS/qRSoIpIwuuQXEQlEgSoiEoZG+UVEQlGgioiEoUEpEZFQ1EMVEQlEgSoiEogCVUQkjIij/HralIhIIApUEUkWy8l8aWxXZj8ws3IzKzOzR82sIF17BaqIJIw1YUmzF7P+wOXAKHcfBuQCZ6TbRvdQRSRhgt5DzQM6mVk1UAisTNdYPVQRSZYmvEbazIrNbNFOS/Fnu3H3D4DbgfeBVcA6d38u3aHVQxWRhMm8h+ruJUDJHvdi1gM4CRgErAWeMLOz3f23De1PPVQRSZgw91CBY4F/u/tqd68Gngb+I90G6qGKSKJ4E+byNxKp7wNjzKwQ2AxMBBal20CBKiIJE2ZQyt1fMbMngSVADfA6Ddwe+IwCVUQSJtwov7vfANyQaXsFqogkix4wHcaClxdy+x0zqa2r4+QTT+D875yzy/p5L/6Ve0ruJceM3NxcrvrB5Rw2/NBI1ba+46+aT+eCPHJyjNwc4/Ebx8QuqVUsePllbv/FL+s/FyedyPnfOXeX9f9+911uvOlnLF22jO9degnnnn1WpErjqa2r48zpcyjqWcjs646NXU4zKVCbrba2lltuu4O7Z/2SvkVFnHPeRYw/chyDBw/a3mb04SMZf9Q4zIy33nqb6/7rxzz9+CMRq259918/ih57dYhdRqupra3llltv5+7Zd9V/Lr5zPuOPPHKXz0W3rl255uormTfvxYiVxvXwnyoY1L8bGzdXxy4lAD0cpdnK36xg7wEDGNC/P/n5+Rx/3LHMm//SLm0KCwux1OXA5i1btv8syVVe/uaun4vjj2Pe/Pm7tOnZsycHH3QQeXmJ6V80yYdVG5n/+gpOOeaA2KWE0YQv9oeWmE9QZeVq+vYt2v5736I+lJW/+bl2z897kdl3/4o1a9Yw847bWrPE6Awovm0xBpw2YW9OmzAgdkktrnL17p+LIsrKyyNWlH1ufeBVrjxrZEJ6p5DVl/xmNpT62QL9Aad+Luvv3b2ihWtrEsc/97c99UCPOXo8xxw9niWvl3LPr37NPbNntkZ5WeGhH46mqEcBVeu3cvGtixnUr5BRQ3vGLqtFue/hcxGhjmz14uLl9OxawEGDe/Na+arY5QSSpZf8ZnYd8Bj1Fb4KvJb6+VEzuz7Ndtvnx5aUpP3aVjB9i4r46KPK7b9/VLma3r17N9h+xGHDWbFiJWvWrm2F6rJDUY/6J4/16tqRiSOLeOOd9ZEranmf/1xU0rtPn4gVZZfSZZXMW7ycSZc9wbUzX+TVslVMnzW/8Q2zWRZf8l8IHJyadrVTvXYHUA7csqeNdpsf6xvWrm5unY066MChLF++nA9WrqSoTx+em/tnbv7prl8fW758BQMG9MfMqFi6jOqaarp369bitWWDTVtr8Dro3CmPTVtrWFBWxaUn7Re7rBZ30EEH1n8uPlhJUVEfnntuLjf/9KbYZWWNaVNGMm3KSABeK1/FA3PKmTH1qMhVNVf2XvLXAV8G3tvt7/1S67JGXl4e1159JZddfiW1dXWc9M1vsN/gwTz59P8AcOopJ/OXF+bxxz89S15eHh07dmTGz25sNwNTVeu2Me2uUgBqa53JR/Rj3CEN9+CTIi8vj2uvuZrLLp+W+lycwH77DebJp54G4NRvn8LHH1dxznnnsXHjRsxyePSxx3jiscfo0qVz5OrlC4n4Gmnb0z2m7SvNJgGzgbeA5ak/DwS+Alzm7s9mcIxW6aG2BV2696F64dTYZWSF/DGz2LBuTewyskKXbj3YWjojdhlZoePw6c3u4dTN+WrDobabnBPeCP7w1Aa5+7NmdgAwmvpBKQNWAK+5e23IQkREwsjeS37cvQ5Y2Aq1iIgEkMWBKiLSpmguv4hIKApUEZEgmvKA6dASM5dfRCQkMxtiZqU7LevN7Ip026iHKiIJE+yJ/cuA4QBmlgt8ADyTbhsFqogkTIvcQ50I/Mvdd5/ktAtd8otIsjRhLv/Ozx1JLcUN7PUM4NHGDq0eqogkTOb9xN2eO7JHZtYBOBGY3tj+FKgikizhv4f6dWCJu3/UWEMFqogkTPBAPZMMLvdB91BFJHGsCUsjezIrBI4Dns7kyOqhikiyBLzkd/dNQK9M2ytQRSRhNPVURCSQeHcyFagikix62pSISCgKVBGRQBSoIiJhRHzvpgJVRBJGPVQRkTAiPmBagSoiieLqoYqIhKJAFREJRIEqIhKGvtgvIhKKBqVERMJQD1VEJJR4gaoHTItIwgR9wHR3M3vSzJaaWYWZHZGuvXqoIpIwQXuoM4Fn3f3U1Mv6CtM1VqCKSLIEuodqZl2Bo4DzANx9G7At7TbuHuTgabT4AUQkMZqdhjUvfCvjzMmb8EyDxzOz4dS/YvpN4FBgMTDN3Tc2uL/My/ziNlYubY3DZL3ORUPZ9N5fYpeRFQr3mciWZQ/GLiMrFAw5V+cipWDIuc3fSRN6qGZWDBTv9KcSdy9J/ZwHjACmuvsrZjYTuB74UUP70yW/iCRM5oGaCs+SBlavAFa4+yup35+kPlAbpFF+EUmYMKP87v4hsNzMhqT+NJH6y/8GqYcqIskS9ov9U4GHUyP87wDnp2usQBWRhAkXqO5eCozKtL0CVUQSRnP5RUTC0Fx+EZEw9MR+EZFgFKgiImHokl9EJBQNSomIBKIeqohIGPHyVIEqIkmjHqqISCAKVBGRMDTKLyISikb5RUTCUA9VRCQUBaqISCAKVBGRQBSoIiJhWLhBKTN7F/gUqAVq3D3tw6YVqCKSMMF7qBPc/eNMGipQRSRZIo7y662nIpIwmb/11MyKzWzRTkvxbjtz4DkzW7yHdZ+jHqqIJIo3pa17CVCSpslYd19pZkXAXDNb6u7zG2qsHqqIJEzmPdTGuPvK1P9WAs8Ao9O1V6CKSLJYTuZLut2YdTazvT77GTgeKEu3jS75RSRhgg1K9QWesfpBrjzgEXd/Nt0GClQRSZZAo/zu/g5waFO2UaCKSMJoplSL+cmMu/jrgkX07NGNJx6cFbucaD6s/IQf3fYAVZ+sx3Jy+PbksUz51jGxy4pi67Yazp/+INXVtdTU1nHc2KF8b8r42GVFkcxzoUBtMd/8+kROP+Ub/PjmO2OXElVubi5XFn+bA/cfyMZNW5jy/Vv42ogD2W+ffrFLa3Ud8nO592dnU9ipA9U1tZx3/YOMG/EVDhnaP3ZprS6R50Jf7G85I4cfTLeuXWKXEV2fXt04cP+BAHQuLGDQwC+x+uO1cYuKxMwo7NQBgJraOmpqaqO+2C2mZJ6LnCYsYSW+hyqft/LDKpa9vZxhQ/eNXUo0tbV1nHnlfby/ag2nTx7FIUPacI+smZJ3LtpgD9XMzk+zbvt0rpKSdJMQpLVt2ryFq28q4epLT6VL506xy4kmNzeHx2dezHP3X07ZWyt5673K2CVFk7hzYZb5Elhz+rw3NrTC3UvcfZS7jyoubnT6q7SS6pparr7p13z9mNFMHHdY7HKyQtcuBRw+bCALlrwTu5ToknMuws2Uaqq0gWpm/2hgeYP6L71KG+Hu3HjHQwwa+CXOOXVi7HKi+mTdRtZv2ALAlq3VLPz7u+w7oFfkquJI5rmIF6iN3UPtC/wnsGa3vxuwIHg1LWD6T25n8etlrF23nkmnXMAlF5zJySccF7usVlda/i/++OdX2X/Qlzn9kp8DcNkFJ3Lk6GGRK2t9H3+ygR/e+Qfq6pw6d44fdyDjD98/dllRJPJcZPFL+uYAXdy9dPcVZjavJQoKbcZPro5dQlY4bNhXeP25u2OXkRUOGNSXx2deFLuMrJDMc5GlgeruF6ZZNyV8OSIizZWlgSoi0uZk8SW/iEgbo0AVEQnCFagiIoEEfI10UyV+Lr+ItDdhv4dqZrlm9rqZzWmsrXqoIpIs4QelpgEVQNfGGqqHKiIJE66HamYDgG8A92ZyZAWqiCRM0Ev+O4FrgbpMGitQRSRhMg/UnZ+Ml1q2P83JzE4AKt19caZH1j1UEUmWJozyu3sJ0NAzRscCJ5rZZKAA6Gpmv3X3sxvan3qoIiJ74O7T3X2Au+8LnAE8ny5MQT1UEUkaTT0VEQklfKC6+zxgXmPtFKgikjDqoYqIhBFx6qkCVUQSRj1UEZFAFKgiImFolF9EJBQFqohIIApUEZEgXKP8IiKhqIcqIhKIAlVEJIx4eapAFZGkUQ9VRCQQBaqISBga5RcRCUU9VBGRQBSoIiJhBJrLb2YFwHygI/VZ+aS735B2G3cPcvA0WvwAIpIYzU7DLUt/k3HmFAw9r8HjmZkBnd19g5nlAy8B09x9YUPbtEoPdWvpjNY4TNbrOHw6PBLxS3LZZIpTvXBq7CqyQv6YWVT/7aLYZWSF/LH3Nn8ngQalvL63uSH1a35qSRvWeuupiCSMZbyYWbGZLdppKd5lT2a5ZlYKVAJz3f2VdEfWPVQRSZjMrwLdvQQoSbO+FhhuZt2BZ8xsmLuXNdRePVQRSRazzJcMufta6t96OildOwWqiCRM5pf8afdi1ifVM8XMOgHHAkvTbaNLfhGRPesHPGBmudR3Ph939znpNlCgikiihHrAtLv/AzisKdsoUEUkYTRTSkQkEAWqiEgYeo20iEgoClQRkTD0PFQRkVDUQxURCUSBKiISiAJVRCQMjfKLiISiQBURCSPiKL+eNiUiEoh6qCKSMLrkFxEJRIEqIhJGxFF+3UMVkYQJ9sT+vc3sBTOrMLNyM5vW2JHVQxWRZAk3yl8DXOXuS8xsL2Cxmc119zcb2kCBKiKJ4oHuobr7KmBV6udPzawC6A80GKi65BeRhMn8kt/Mis1s0U5L8R73aLYv9a9DeSXdkdVDFZFkadrroUuAkvS7sy7AU8AV7r4+XVsFqogkTLhRfjPLpz5MH3b3pxtrr0AVkYQJcyfTzAy4D6hw9zta78giItnCLPMlvbHAOcAxZlaaWian20A9VBFJmGCj/C81dWftIlBr6+o4c/ocinoWMvu6Y2OXE80DC7vzxJJuOHDaiHWcN2Zt7JKiOf6q+XQuyCMnx8jNMR6/cUzsklrF8de8SueC3B3/3Tcctsv651+vYtYz75Jj9euvP3MwIw7oFqnatqddBOrDf6pgUP9ubNxcHbuUaP5Z2YEnlnTjiYvfJz/Xuei3/Tl6/43s26v9npP7rx9Fj706xC6j1d1/7SH02Ct/j+vGHNidCcNHYGYsW76Rq++p4A8/H9XKFTaTpp62nA+rNjL/9RWccswBsUuJ6l+rO3DogC10ynfycuDwfTYzd2mX2GVJliksyMVSgbR5a23M54w0Q5ipp19E4nuotz7wKleeNbJd904BDijaxp3P92bNphwK8p35b3dmWL8tscuKxoDi2xZjwGkT9ua0CQNil9QqzKD4F29gZpw2/kucdnS/z7X58+KPmfnUu1R9Ws3d0w6OUGVzZfFrpM1sKPXTrV5x9w07/X2Suz/bksU114uLl9OzawEHDe7Na+WrYpcT1X59tnHR2E+44KEBFHaoY0jfreQm/vqkYQ/9cDRFPQqoWr+Vi29dzKB+hYwa2jN2WS3uoemHUtSjI1Xrt3Hx7WX1/91Ddr1HeuzI3hw7sjeLlq1j9jPvce81X41U7ReUrZf8ZnY58L/AVKDMzE7aafXP02y3fTpXSUnaSQgtqnRZJfMWL2fSZU9w7cwXebVsFdNnzY9WT2ynjVjPM999n4fPX0H3TrXs02tb7JKiKepRAECvrh2ZOLKIN95JOwEmMYp6dASgV9cOTBzRizf+/WmDbUcN6cby1ZtZ82lbu7rL3kv+i4GR7r4hNZf1STPb191npqtmt+lcvrV0RpBim2ralJFMmzISgNfKV/HAnHJmTD0qSi3ZoGpjLr0617JyXR7PVezF7y58P3ZJUWzaWoPXQedOeWzaWsOCsiouPWm/2GW1uE1ba/E6T/1317KgfA2Xnjhwlzbvf7SZvYsKMDPefG8D1TVO9y5t7c5g9j5gOvezy3x3f9fMjqY+VPehjd6ubs+mPt6PtZtyycuFGyZ/RLdOdbFLiqJq3Tam3VUKQG2tM/mIfow7pHfcolpB1bptTJtdAUBtnTP5a30Y99We/O6F+tthp0/ox9zFH/P7BZXk5RoFHXK4/ZKh2wep2oyI9Zq7N7zS7HngSncv3elvecD9wFnunpvBMaL1ULNNx+HT4ZE29uFsKVOc6oVTY1eRFfLHzKL6bxfFLiMr5I+9t9n/QDatXNRwqO2m8Mujgv6DbKyHei71D1ndzt1rgHPN7FchCxERCSLia6TTBqq7r0iz7m/hyxERaa7svYcqItKmhHpi/xehQBWRhFGgioiEEXGUX4EqIgmjQBURCSNbp56KiLQ94aaemtn9ZlZpZmWZHFmBKiIJE3Qu/2+ASZkeWYEqIgkTLlDdfT7wSaZHVqCKSLI04SV9Oz8ZL7UUN+fQGpQSkYTJvJ+425Pxmk2BKiLJou+hioiEoq9NiYgEEvRrU48CLwNDzGyFmV2Yrr16qCKSLAEv+d39zKa0V6CKSMLoHqqISCAKVBGRMDTKLyISigJVRCQIPbFfRCQUXfKLiISSpW89FRFpe9RDFREJQ5f8IiKhKFBFRAJRoIqIhKFLfhGRUDTKLyIShl4jLSLS9qmHKiIJE6+Hau4e7eCtycyKUy/kavd0LnbQudhB56L52tMlf7NeD5swOhc76FzsoHPRTO0pUEVEWpQCVUQkkPYUqLo3tIPOxQ46FzvoXDRTuxmUEhFpae2phyoi0qIUqCIigSQ+UM1skpktM7O3zez62PXEZGb3m1mlmZXFriUmM9vbzF4wswozKzezabFrisXMCszsVTP7e+pc3Bi7prYs0fdQzSwX+CdwHLACeA04093fjFpYJGZ2FLABeNDdh8WuJxYz6wf0c/clZrYXsBg4uT1+LszMgM7uvsHM8oGXgGnuvjByaW1S0nuoo4G33f0dd98GPAacFLmmaNx9PvBJ7Dpic/dV7r4k9fOnQAXQP25VcXi9Dalf81NLcntZLSzpgdofWL7T7ytop/9wZM/MbF/gMOCVyKVEY2a5ZlYKVAJz3b3dnovmSnqg7ukpCfp/XwHAzLoATwFXuPv62PXE4u617j4cGACMNrN2ezuouZIeqCuAvXf6fQCwMlItkkVS9wufAh5296dj15MN3H0tMA+YFLeStivpgfoasL+ZDTKzDsAZwO8j1ySRpQZi7gMq3P2O2PXEZGZ9zKx76udOwLHA0qhFtWGJDlR3rwEuA/6P+oGHx929PG5V8ZjZo8DLwBAzW2FmF8auKZKxwDnAMWZWmlomxy4qkn7AC2b2D+o7IHPdfU7kmtqsRH9tSkSkNSW6hyoi0poUqCIigShQRUQCUaCKiASiQBURCUSBKiISiAJVRCSQ/wdsxVEAwVtE4QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Let's visualize our annotation (each column is an annotation)\n",
    "ax = sns.heatmap(annotations, annot=True, cmap=sns.light_palette(\"orange\", as_cmap=True), linewidths=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### IMPLEMENT: Scoring All Annotations at Once\n",
    "Let's calculate the scores of all the annotations in one step using matrix multiplication. Let's continue to us the dot scoring method\n",
    "\n",
    "<img src=\"img/scoring_functions.png\" />\n",
    "\n",
    "To do that, we'll have to transpose `dec_hidden_state` and [matrix multiply](https://docs.scipy.org/doc/numpy/reference/generated/numpy.matmul.html) it with `annotations`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((3,), (3, 4))"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dec_hidden_state.shape, annotations.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 82.5, 207. , 103.5, 129. ])"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def dot_attention_score(dec_hidden_state, annotations):\n",
    "    # TODO: return the product of dec_hidden_state transpose and enc_hidden_states\n",
    "    return np.array(dec_hidden_state).T @ annotations\n",
    "    \n",
    "attention_weights_raw = dot_attention_score(dec_hidden_state, annotations)\n",
    "attention_weights_raw"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking at these scores, can you guess which of the four vectors will get the most attention from the decoder at this time step?\n",
    "\n",
    "## Softmax\n",
    "Now that we have our scores, let's apply softmax:\n",
    "<img src=\"img/softmax.png\" />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([8.51798759e-55, 1.00000000e+00, 1.12336561e-45, 1.33361482e-34])"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def softmax(x):\n",
    "    x = np.array(x, dtype=np.longdouble)\n",
    "    e_x = np.exp(x)\n",
    "    return e_x / e_x.sum(axis=0) \n",
    "\n",
    "attention_weights = softmax(attention_weights_raw)\n",
    "attention_weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Even when knowing which annotation will get the most focus, it's interesting to see how drastic softmax makes the end score become. The first and last annotation had the respective scores of 927 and 929. But after softmax, the attention they'll get is 0.12 and 0.88 respectively.\n",
    "\n",
    "# Applying the scores back on the annotations\n",
    "Now that we have our scores, let's multiply each annotation by its score to proceed closer to the attention context vector. This is the multiplication part of this formula (we'll tackle the summation part in the latter cells)\n",
    "\n",
    "<img src=\"img/Context_vector.png\" />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[2.55539628e-55, 5.00000000e+00, 1.12336561e-46, 5.33445926e-34],\n",
       "       [8.51798759e-55, 2.00000000e+00, 3.37009682e-45, 4.00084445e-34],\n",
       "       [3.40719504e-54, 9.00000000e+00, 5.61682803e-45, 7.06815852e-34]])"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def apply_attention_scores(attention_weights, annotations):\n",
    "    # TODO: Multiple the annotations by their weights\n",
    "    return attention_weights * annotations\n",
    "\n",
    "applied_attention = apply_attention_scores(attention_weights, annotations)\n",
    "applied_attention"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's visualize how the context vector looks now that we've applied the attention scores back on it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVQAAAD8CAYAAAAoqlyCAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAl0klEQVR4nO3deXwU9f3H8dcnG4EckEOIYqgY0AqIvyoVKiqWwxP5eSOEAF4QQeQoyFktoli1th5VfkIQrSKCd7VVVCqgxQPk8ACC9UA5olJOQxJCdvP9/ZEYCSTLJkyyy/p+Ph7zcNn5zsx3xtl3vt/57syacw4RETl0MeGugIhItFCgioh4RIEqIuIRBaqIiEcUqCIiHlGgioh4RIEqIlINMxtpZqvNbI2ZjTpYeQWqiEgVzKw9MBjoBPwK6GVmJwRbRoEqIlK1tsAHzrlC55wfeBu4LNgCsfVQKd2KJSKhskNew9MWcuZYFjcA2fu8leOcyyl/vRq408yOBIqAnsDyYOurj0Bl964d9bGZiJeYlELJ+8PCXY2IcETnaTovyiUmpehYlEtMSvFgLaFnsnOlOUBO1fNcrpndAywAdgMfA/5g61OXX0Sii1no00E452Y55zo4584GtgOfBytfLy1UEZH6c+hXDSrWZJbmnNtiZscClwOdg5VXoIpIlPEuUIEXyq+hlgDDnHNBr80oUEUkuph3VzKdc11qUl6BKiJRxtMWao1oUEpExCNqoYpIdAlh9L6uKFBFJMooUEVEPOFqEKheR68CVUSii4ej/DWlQBWRKKMuv4iIRxSoIiLe0Ci/iIhXFKgiIt7QoJSIiFfUQhUR8YgCVUTEIwpUERFvhHGUX0+bEhGphpn9zszWmNlqM5trZo2ClVegikh0sZjQp2CrMUsHRgCnOefaAz6gb7Bl1OUXkSjjaZc/FogzsxIgHsgLVlgtVBGJMlaDqXrOuc3An4ENwLfALufcm8GWUaCKSHSpwc9Im1m2mS3fZ8r+aTWWAlwCZADHAAlm1j/YptXlF5EoE3qX3zmXA+RUM/scYL1z7r8AZvYicAbwVHXrUwtVRKKMN11+yrr6p5tZvJkZ0APIDbaAWqgiElVcDe7lDxapzrmlZvY8sBLwA6uovjULKFBFJOp4N8rvnJsMTA61vAJVRKKMbj0VEfGGHjBdte++/54/3DaFbdu2EWMxXHbZpfTr2+eAcstXrOAv9z2A3+8nOTmZmTMeCXkbeXl5XNknk5bHHgvAye3bM2nieACyhwxl69ZtNGzYEIBpDz1IamqqB3sWHueNWUJCnI8YM3w+49nbfhPuKtXKlDum8u8l75KaksKz854+YP76r79myu1TWffZZ9w4dAgD+2fVajvffvcdvftkkj14UMU6SkpKuOfeP7NixUosJoZhQ2+gR/fuh7Q/h6rXJZcSH5+ALyYGn8/HU0/+rdL8xW+/wyMzZhBjZfPHjB7FqaecEvL6V69Zw51/vBsA5xzZgwfRvVvXSmV+N+ZmNm/Oq/L/R/1ToFbJ5/Pxu5EjaNumDQUFBfQfeA2nd+pEq1YZFWXy8/O5+0/38tCDD9D86KPZvn17jbfTIj2duXNmVzlv6u1TaNeuba33IdI8Nv7XpDRuEO5qHJL/vegirup9JZNvu73K+UlNmjD25tEsXvz2IW3nvvsf4IzOnSu9N+vxv5GaksJLLzxHaWkpu3744ZC24ZUZj0wjJTm5ynmdOp7Gb8/ugpnx+eefM37SLbz43DMhr7t169bMfuJxYmNj+e/WrWRmDeDsLmcRG1sWHwsXLSIuLt6L3fCIArVKzZo2pVnTpgAkJCSQkXEcW/67pVKgzn/jDbp37Urzo48GqNSCfG3+fOY98xwlJSW0b38SE8aNxefz1e9OiOc6dDiVvLzq7wBMTU0lNTWVJUvePWBeqOfEosVvk56eTlyjys/CeOWVf/BCeRjFxMRUG2KRJD7+p7ArKtpTqUccyvHY9xjsLd5bafnCwkKeenout0ycyIRJv6+zfagRPW3q4PLy8lj32X9of1L7Su9v2LCRH/LzyR4ylKyBV/PPV18DYP369by54F/MejSHuXNmExMTw/zX36hy3Zvz8ujXfyCDbxjKqlUfVZp32x1TycwawMxZj+Gcq5N9qy9mkP3nVVw1eSnPLd4U7urUu1DPiaKiIp54cjbZg66v9H5+fj4Aj0yfQb8BAxk3YRLbtm2rl7oHYxjDho8ga+DVvPjS36sss3DRYi7v3YeRo8cw+ZZbgJp9Rj5dvZrefTLp0y+LiePHV7ROH5meQ/9+/WjUqGGd7FvtePY91Bo7aAvVzNpQdvtVOuAoezjAK865oF9w9VJhYSFjJ0zk5tGjSExMqDQvEAiQu24d06c9zJ7iYq69fhAnt2/Psg+Xk7vuMwZefS0AxcXFpKakHLDupk2b8uorL5OcnERu7jrGjB3Hs/PmkpiYwNTbp5CWlkZBQQFjJ0zk1dfm0+uinvWyz3Vh9u87kpbSkG0/7GXwvSvJaJ7AaSceeEyiVajnxPScmfTL7FupZQfgDwT4fssWfvWr/2H070bx1JyneeCvD3HHlNvqo/rVeuzRHJo1a8b27du58aYRHNeyJR06nFqpTPduXenerSsrV67ikRkzeGTawyEfDygbW3jumbmsX7+eyVPu4MwzOvP1N9+wcdNGxoweFbTHUP8itMtvZuOBTGAesKz87RbAXDOb55y7u5rlsoFsgBkzZtCvT+9aV7DE72fs+IlceP75dO/W7YD5aWlpJCclERcXR1xcHB1OOZX/fP45zjl6XdST4cNurFR+4aLFzHx0FgC3/n4S7dq1pUGDsmuKbdu2oUWLdDZs2EC7dm1JS0sDyi43XHD+eaxZu/awDtS0lLJWxJFNGtCjQzM+/eqHn1WghnpOrF69hrcWLuSvDz9Mfv5uYmJiaNigAVf1vpJGjRrRrWtXAM45pwcvv/KP+t6NAzRr1gwou9TRretvWb127QGB+qMOHU5l05TN7Ni5s0afkR9lZGTQKK4RX375FWvWriV33Wf0uuRSAoEA27fvIHvIUHKmhz4oXCcieJT/euAk51zJvm+a2X3AGqDKQN3v/li3e9eOWlXOOccdd9xJRsZx9M/qV2WZrmd34Z57/4Lf76fE72f1mjX069eX1q1aMXrsWLIy+5KamsquXbsoLCys+Ev9ox07dtCkSRN8Ph+bNm9mw8ZNpKcfg9/vJ3/3blKSkynx+1my5F06dexYq/2IBIXFAVypIyEulsLiAO+t2c7QizMOvmAU6dSxY0jnxKyZMypez8iZSVx8PH2uKmsUnN3lLJavWEmnjqex7MMPycgI7zEsKiqitLSUhIQEioqK+GDpMgYPuq5SmY0bN9KiRQvMjNx16yjx+0lOSgr5eGzenMdRR6URGxvLt99+yzffbKD5Mc1p164tva+8Aii7JDdq9M3hD1MgYluoQCllT1n5Zr/3m5fPq1Mfffwxr86fz/HHtyYzawAAw24cynfffQfAlVdcTkZGBmd0Pp2+Wf2JsRguveRijm/dGoAbh9zAsOEjKXWlxMbGMmHsWJo3b15pGytXrWL6jJn4fD5ifDFMmjCOpKQkioqKuGnESPx+P6WBUjp16shll15S17tcZ7btKmbkQ58AEAg4ep5+NGf9T9Mw16p2Jt1yK8tXrGTnzp1c2Ot/uWHwYPx+P1B2Tmzduo0B11xDQUEBZjHMnTeP5+bNo1WrjJDOiWBG3DSMWydP4S/3309KcgqT/3BLXe1mSLZt387NY8u+5hcIBLjg/PM4o3Nnnn/hRaDseLy1cBGvvjaf2NhYGjZsyF133oGZhXw8Pvr4Y/72xJPExsZiMcaEcWMjezAujD8jbcEGWszsAuBh4HNgY/nbxwLHAzc5514PYRu1bqFGm8SkFEreHxbuakSEIzpPQ+dFmcSkFB2LcolJKYfcvCz958khjx7H9PrU86dRV8s597qZ/RLoRNmglAGbgA+dcwEvKyIi4o3I7fLjnCsFPqiHuoiIeCCCA1VE5LASwaP8IiKHGQWqiIgnavKAaa8dNreeiojUJzM70cw+2mf6wcxGBVtGLVQRiTLedPmdc58BpwCYmQ/YDLwUbBkFqohEmTq5htoD+NI5t/9NTpWoyy8i0cUs5MnMss1s+T5TdjVr7QvMPdim1UIVkSgTejtxv+eOVMnMGgAXAxMPtj4FqohEF++/h3ohsNI59/3BCipQRSTKeB6omYTQ3QddQxWRqOPdE/vNLB44F3gxlC2rhSoi0cXDLr9zrhA4MtTyClQRiTK69VRExCPhu5KpQBWR6KKnTYmIeEWBKiLiEQWqiIg3wpenClQRiTZqoYqIeCOMD5hWoIpIVHFqoYqIeEWBKiLiEQWqiIg39MV+ERGvaFBKRMQbaqGKiHglfIGqB0yLSJTx9AHTyWb2vJmtM7NcM+scrLxaqCISZTxtoT4IvO6cu7L8x/rigxVWoIpIdPHoGqqZNQHOBq4BcM7tBfYGW6ZeAjUxKaU+NnNYOKLztHBXIWLovPiJjoWXQg9UM8sGsvd5K6f8p6UBWgH/BR43s18BK4CRzrmC6tZXL4G6e9eO+thMxEtMSqHomwXhrkZEiGt5rs6LcolJKToW5Tz5w1KDFmp5eOZUMzsW6AAMd84tNbMHgQnArdWtT4NSIhJlPBuU2gRscs4tLf/385QFbLUUqCISZbwJVOfcd8BGMzux/K0ewNpgy2hQSkSii7df7B8OzCkf4f8KuDZYYQWqiEQZ7wLVOfcRcFqo5RWoIhJldC+/iIg3dC+/iIg39MR+ERHPKFBFRLyhLr+IiFc0KCUi4hG1UEVEvBG+PFWgiki0UQtVRMQjClQREW9olF9ExCsa5RcR8YZaqCIiXlGgioh4RIEqIuIR7wLVzL4G8oEA4HfOBX02qgJVRKKLeT4o1c05tzWUggpUEYky4evy60f6RCS6mIU8mVm2mS3fZ8reb20OeNPMVlQx7wBqoYpIlAm9heqcywFyghQ50zmXZ2ZpwAIzW+ece6e6wmqhikhUcTWYDrou5/LK/7sFeAnoFKy8AlVEoozVYAqyFrMEM2v842vgPGB1sGXU5ReR6OLdKP9RwEtWdudVLPC0c+71YAsoUEUkyngzyu+c+wr4VU2WUaCKSHTRvfwiIl5RoFZrztNz+fvLr2BmHH98aybfegsNGzasmL98xQpG3zyO9GOOAaBbt65kD7o+5PXn5eVxZZ9MWh57LAAnt2/PpInjAcgeMpStW7dVbG/aQw+SmprqzY7Vs++27OCWe59k2/YfsBjjip5nknVZt3BXq1aKi4sZfMNQ9u7dSyAQoEeP7gzJHlxl2TVr13LNdYO4686pnNOje4239e1339G7TybZgwcxsH8WcPicF4FAgAFXX0uzZs148P6/hLzcTSNGsXXrVgKBAKeecgrjx92Mz+ermP+vtxYyfuIkZv/tcdq1a1sXVT9ECtQqbdmyhXnPPMtzz8ylUaNGjJ/4e95YsICLe/WqVO7UU06p0Qmzvxbp6cydM7vKeVNvnxKhJ03N+HwxjMm+nLYn/IKCwj1kDruH0zu0oXXL5uGuWo01aNCA6f/3MPHx8ZT4/Vw/OJszO3fm5JPbVyoXCAT460PT6Hz6b2q9rfvuf4AzOnc+4P3D4byYO+8ZjjvuOAoKCmq03N1/vJPExAScc4ybMJF/vbWQ8887F4CCggLmPfMs7dufVBdV9kYYu/wR/7WpQCBAcXExfr+fPXv20Kxps5CXfW3+fAZecx2ZWQO48667CQQCdVjTyNbsyCTanvALABLiG9Hq2KPZsnVneCtVS2ZGfHw8AH6/H7/fX2Wj5Jlnn6NH926kpKRUej/U82LR4rdJT0+ndasMz/ehrn3//RaWvPsel15yccV7Gzdt4qYRo8gaeDXXD76B9V9/XeWyiYkJAPgDAUpKSirl0yMzchg4oD8NGzSoy+ofopgaTN5vOWKlpaXRv38WF118Kef37EViYkKVrY1PP/2Uvv36M3zkKL788isA1q9fz5sL/sWsR3OYO2c2MTExzH/9jSq3szkvj379BzL4hqGsWvVRpXm33TGVzKwBzJz1GM6F8lXgyLf5u22s+2ITJ7c5LtxVqbVAIEBm1gDOPf9CTu/UiZPbV26dbtmyhUWL3+aKyy+r9H6o50VRURFPPDm72stHkX5e/OX++xk5/CZiYn5Kwzv/eDfjbh7NnCefYNTI4dx9z73VLj9s+EjOPf9C4uMT6NG97FLJus8+4/vvv+fsLmfVef0PjTffQ62NWnf5zexa59zj1czLBrIBZsyYQb8+vWu1jR9++IG3336Hf/z9RRIbN2b8hEm8Nn8+PS+8sKJMmxPb8M9X/k58fDxL3n2PMePG8fcXnmfZh8vJXfcZA6++Fii77pa6X0sFoGnTprz6ysskJyeRm7uOMWPH8ey8uSQmJjD19imkpaVRUFDA2AkTefW1+fS6qGet9iVSFBYVc/PtjzJ26BUkJsSFuzq15vP5mDtnNvn5+YwZN54vvvyS41u3rpj/5/seYMRNwypd+wNCPi+m58ykX2bfipbwviL9vHjn30tISUmhbds2LF+xAoDCwkI++fRTxk/8fUW5vSV7q13HtIcepLi4mFv+MJkPly+nU8eO3Hf/A9z2h1vrvP6H7DAd5Z8CVBmo+90f63bv2lGrDSxd9iHpxxxT0WXr3q0rH3/yaaVA/bF7AnDWmWdw95/+xI6dO3HO0euingwfdmOldS5ctJiZj84C4NbfT6Jdu7Y0KO++tG3bhhYt0tmwYQPt2rUlLS0NgISEBC44/zzWrF0bUR+cmirxBxhz+0x6dj+NHmedEu7qeKJx48ac1qED773/QaVAzc3NZeIttwCwc+cu3n3vfXw+X8jnxerVa3hr4UL++vDD5OfvJiYmhoYNGtDnqt4Rf158/MknvPPvf/Pue++xt3gvuwsK+MNtU0hMTDxgrCAQCNB/4DUAnH12F4be8NPzPxo2bMjZXbrw9jv/5qSTTuKLL78ie2jZcdu2bTu/u3ks9//53gi8lhyhgWpmn1Q3i7K7COrU0UcfxaerV1O0Zw+NGjZk2YfLade2TaUyW7du48gjUzEzVq9ZQ2mpIzkpiU4dOzJ67FiyMvuSmprKrl27KCwspHu3rnTv1rVi+R07dtCkSRN8Ph+bNm9mw8ZNpKcfg9/vJ3/3blKSkynx+1my5F06dexY17tcZ5xzTLlvDhnHHs2AK3uEuzqHZMeOHcTGxtK4cWP27NnD0mUfcvXAAZXK/OPllypeT55yO13OOotuXX/LV1+tD+m8mDVzRsXrGTkziYuPp89VvQ+L82L4sBsr/mAsX7GC2U89zZ//dA/XXj+YBf96i3PP6YFzjs8//4Jf/vKESiFbWFhIQWEhzZo2xe/38+5773HqKafQODGRhQt+ujSSPWQoo0aMiMAwhYgNVMpC83xg/yamAe/VSY32cXL79vTo0Z2sAVcT6/Nx4om/5PLLLuX5F14E4MorLuethQt5/oUX8fl8NGzUkLvuvAMzo1WrDG4ccgPDho+k1JUSGxvLhLFjad688qj2ylWrmD5jJj6fjxhfDJMmjCMpKYmioiJuGjESv99PaaCUTp06ctmll9T1LteZj9Z8xT//tYwTMo7hqiF3ATD8uovp0imCR2ursXXrViZPuYNAaQBX6jjnnB6c3eWsSudFdUI9L6pTUlJy2J4XU2+fwl33/IlZjz2OP+DnvHPP5Ze/PKFSmaKiIkaPGcvekr2UBkrpeNqvD7gOHfHC2OW3YBfUzWwW8LhzbkkV8552zvULYRu17vJHm8SkFIq+WRDuakSEuJbnovOiTGJSio5FucSklENOw70rbg15lLDBr+/wNH2DtlCdc9V+Qz7EMBURqWeR2+UXETm8HKaj/CIiEUiBKiLiCadAFRHxiMc/I21mPmA5sNk51ytY2Yi+9VREpOY8v/V0JJAbSkEFqohElxr8jPTBV2UtgIuAR0PZtAJVRKJM6C1UM8s2s+X7TNn7rewBYBxQGsqWdQ1VRKJM6INS+z13pPJazHoBW5xzK8ysayjrU6CKSJTxbJT/TOBiM+sJNAKamNlTzrn+1S2gLr+IRBeLCX0Kwjk30TnXwjl3HNAXWBgsTEGBKiLiGXX5RSS61MGtp865xcDig5VToIpIlNGdUiIiHlGgioh4w+NbT2tCgSoiUUYtVBERjyhQRUS8oQdMi4h4RYEqIuIRBaqIiCecRvlFRLyiFqqIiEcUqCIi3ghfnipQRSTaqIUqIuIRBaqIiDc0yi8i4hVvWqhm1gh4B2hIWVY+75ybHGwZBaqIRBnPuvzFQHfn3G4zOwJYYmbznXMfVLeAAlVEootH9/I75xywu/yfR5RPLtgy9RKoiUkp9bGZw0Jcy3PDXYWIofPiJzoWXgo9UM0sG8je562c8p+W/nG+D1gBHA9Mc84tDba+egnU3bt21MdmIl5iUgo8HcYvyUWSfk7nRbnEpBQdi3Ke/GGpwaBUeXjmBJkfAE4xs2TgJTNr75xbXV15/eqpiEQZq8EUGufcTsp+pO+CYOUUqCISZbwJVDNrVt4yxczigHOAdcGW0aCUiEQX7x4w3Rx4ovw6agzwrHPun8EWUKCKSJTxbJT/E+DUmiyjLr+IiEfUQhWRqKIHTIuIeEYPRxER8YgCVUTEG/oZaRERryhQRUS8oUEpERGvqIUqIuIRBaqIiEcUqCIi3tAov4iIVxSoIiLeCOMovx6OIiLiEbVQRSTKhK/LrxaqiEQZz57Y/wszW2RmuWa2xsxGHmzLaqGKSHTxbpTfD4xxzq00s8bACjNb4JxbW90CaqGKSJTxpoXqnPvWObey/HU+kAukB1tGgSoi0cViQp7MLNvMlu8zZVe5SrPjKPs5lKXBNq0uv4hEFVezn4fOAXKClTGzROAFYJRz7odgZRWoIhJlvBvlN7MjKAvTOc65Fw9WXoEqItHFo0EpMzNgFpDrnLsvlGV0DVVEoow3g1LAmcAAoLuZfVQ+9Qy2gFqoIhJlvGknOueWUMPrBwpUEYkuetqUiIhXFKhVKi4uZvANQ9m7dy+BQIAePbozJHtwlWXXrF3LNdcN4q47p3JOj+4hbyMvL48r+2TS8thjATi5fXsmTRxfqczvxtzM5s15PDvv6drvTAR44oNknluZhAN6d9jFNafvDHeVaq3XJZcSH5+ALyYGn8/HU0/+7YAyy1es4C/3PYDf7yc5OZmZMx6p8Xa+/e47evfJJHvwIAb2zwIge8hQtm7dRsOGDQGY9tCDpKamHtL+1NbX33zDxEm3VPx7c95mhmRn0y+zb8V767/+mim3T2XdZ59x49AhFfsRqtVr1nDnH+8GwDlH9uBBdO/WtVKZaPmMHKqIDtQGDRow/f8eJj4+nhK/n+sHZ3Nm586cfHL7SuUCgQB/fWganU//Ta220yI9nblzZlc5b+GiRcTFxddqvZHkP1sa8NzKJJ4bvIEjfI5BT6XT9YQCjjuyJNxVq7UZj0wjJTm5ynn5+fnc/ad7eejBB2h+9NFs3769Vtu47/4HOKNz5wPen3r7FNq1a1urdXrpuJYtK87dQCDAhRf9L926/rZSmaQmTRh782gWL367Vtto3bo1s594nNjYWP67dSuZWQM4u8tZxMaWxUfEfUbC2OWP6FF+MyM+vux/lN/vx+/3V9maf+bZ5+jRvRspKSmV3n9t/nwGXnMdmVkDuPOuuwkEAjXafmFhIU89PZdB111b632IFF/+twG/arGHuCMcsTHQsWURC9YlhrtadWb+G2/QvWtXmh99NEClFmSo58WixW+Tnp5O61YZ9VLnQ7Xsw+W0aJFO8+bNK72fmprKSe3aVQTgvkI5FnGNGlUsu7d4b6W8iszPiGej/DUW0YEKZX91M7MGcO75F3J6p06c3L5y63TLli0sWvw2V1x+WaX3169fz5sL/sWsR3OYO2c2MTExzH/9jSq3sTkvj379BzL4hqGsWvVRxfuPTM+hf79+NGrU0PP9qm+/TNvL8m/i2FEYQ1GJ8c4XCXy3K6I7KEEZxrDhI8gaeDUvvvT3A+Zv2LCRH/LzyR4ylKyBV/PPV18DQj8vioqKeOLJ2WQPur7K7d92x1QyswYwc9ZjOOc83bfaenPBAs4/77yQy9fkM/Lp6tX07pNJn35ZTBw/viJgI/MzElODyVsH/USZWRvKHgiw1Dm3e5/3L3DOve55jfbj8/mYO2c2+fn5jBk3ni++/JLjW7eumP/n+x5gxE3D8Pl8lZZb9uFyctd9xsCry/5yFhcXk7pfCxagadOmvPrKyyQnJ5Gbu44xY8fx7Ly5bM7bzMZNGxkzehR5eXl1u5P1oHWzvQw6czvXzW5BfINSTjyqGF/E/zmt3mOP5tCsWTO2b9/OjTeN4LiWLenQ4dSK+YFAgNx165g+7WH2FBdz7fWDOLl9+5DPi+k5M+mX2beih7SvqbdPIS0tjYKCAsZOmMirr82n10VBv55Y50pKSnj7nX9z041DQ14m1GMBZWMLzz0zl/Xr1zN5yh2ceUZnvv7mm8j8jETqKL+ZjQCGUfaUlVlmNtI593L57D8CVQZq+QMGsgFmzJhBvz69D7mijRs35rQOHXjv/Q8qBWpubi4Tbym7KL9z5y7efe99fD4fzjl6XdST4cNurLSehYsWM/PRWQDc+vtJtGvXlgYNGgDQtm0bWrRIZ8OGDaxZu5bcdZ/R65JLCQQCbN++g+whQ8mZXvOBjUjRu8MP9O5QdivyfW8dyVFN/GGuUe01a9YMKOvOduv6W1avXVspUNPS0khOSiIuLo64uDg6nHIq//n885DPi9Wr1/DWwoX89eGHyc/fTUxMDA0bNKDPVb1JS0sDICEhgQvOP481a9eGPVDffe992rQ5kSOPPDLkZWryGflRRkYGjeIa8eWXX0XwZyRCAxUYDPzaObe7/Gkrz5vZcc65BwlS6/0eOOB279pRq8rt2LGD2NhYGjduzJ49e1i67EOuHjigUpl/vPxSxevJU26ny1ln0a3rb/nqq/WMHjuWrMy+pKamsmvXLgoLC+nerWulEcodO3bQpEkTfD4fmzZvZsPGTaSnH0O7dm3pfeUVQNk3AUaNvjkCTpRDs63Ax5EJAfJ2xfJmbmOeuX5DuKtUK0VFRZSWlpKQkEBRUREfLF3G4EHXVSrT9ewu3HPvX/D7/ZT4/axes4Z+/frSulWrkM6LWTNnVLyekTOTuPh4+lzVG7/fT/7u3aQkJ1Pi97Nkybt06tixfnY8iDfefJMLatDdB+jUsWNIx2Lz5jyOOiqN2NhYvv32W775ZgPNj2kewZ+RyA1U34/dfOfc12bWlbJQbUk91Hrr1q1MnnIHgdIArtRxzjk9OLvLWTz/QtkzCq684vJql23VKoMbh9zAsOEjKXWlxMbGMmHs2AMu2K9ctYrpM2bi8/mI8cUwacI4kpKS6nS/wmX4s83ZWegj1geTe35PUlxpuKtUK9u2b+fmsWVfbQsEAlxw/nmc0blzpfMiIyODMzqfTt+s/sRYDJdecnFFzyaU86I6JSUl3DRiJH6/n9JAKZ06deSySy+pmx0NUdGePSxduoxJEydUvLfvsdi6dRsDrrmGgoICzGKYO28ez82bF/Jn5KOPP+ZvTzxJbGwsFmNMGDe22m9XRIQwdvkt2AV1M1sIjHbOfbTPe7HAY0CWc85X3bL7qHULNdokJqXA0+H7nx1R+jl0XpRJTErRsSiXmJRyyB+QwrzlIY8Sxh9zmqcfyIO1UAdS9jMAFZxzfmCgmc2oehERkTAK489IBw1U59ymIPPe9b46IiKHKnKvoYqIHFZq8sR+rylQRSTK6NZTERFvmIU+HXRV9piZbTGz1aFsWoEqIlHG03v5/wZcEOqW1eUXkeji4fdQnXPvlN/UFBIFqohEGV1DFRHxSOhdfjPLNrPl+0zZh7JltVBFJMqE3kLd77kjh0yBKiLRRU/sFxHxincPmDazucD7wIlmtsnMqn7ieDm1UEUkung7yp9Zk/IKVBGJMrr1VETEIwpUERFvROpvSomIHH4UqCIiHlGgioh4Q11+ERGvKFBFRDyhJ/aLiHhFXX4REa9E6K+eiogcftRCFRHxhrr8IiJeUaCKiHhEgSoi4g11+UVEvBK+UX49sV9EootZ6NNBV2UXmNlnZvaFmU04WHkFqohIFczMB0wDLgTaAZlm1i7YMgpUEYkyof+M9EF0Ar5wzn3lnNsLzAMuCbZAfVxDtcSklHrYzEEqYZZd/pOx4dXPhbsGEXMsdF78RMfCO4lJKSGPSplZNpC9z1s5+xyDdGDjPvM2Ab8Jtr6fUws1++BFfjZ0LH6iY/GTn92xcM7lOOdO22fa9w9KVcEctEX0cwpUEZGa2AT8Yp9/twDygi2gQBURqdqHwAlmlmFmDYC+wCvBFvg5fQ/1sL825CEdi5/oWPxEx2Ifzjm/md0EvAH4gMecc2uCLWPOhX+QREQkGqjLLyLiEQWqiIhHoj5Qa3rrWDQzs8fMbIuZrQ53XcLJzH5hZovMLNfM1pjZyHDXKVzMrJGZLTOzj8uPxZRw1+lwFtXXUMtvHfsPcC5lX4H4EMh0zq0Na8XCxMzOBnYDTzrn2oe7PuFiZs2B5s65lWbWGFgBXPpzPC/MzIAE59xuMzsCWAKMdM59EOaqHZaivYVa41vHoplz7h1ge7jrEW7OuW+dcyvLX+cDuZTdFfOz48rsLv/nEeVT9Lay6li0B2pVt479LD84UjUzOw44FVga5qqEjZn5zOwjYAuwwDn3sz0WhyraA7XGt47Jz4eZJQIvAKOccz+Euz7h4pwLOOdOoexOoE5m9rO9HHSooj1Qa3zrmPw8lF8vfAGY45x7Mdz1iQTOuZ3AYuCC8Nbk8BXtgVrjW8ck+pUPxMwCcp1z94W7PuFkZs3MLLn8dRxwDrAurJU6jEV1oDrn/MCPt47lAs8e7NaxaGZmc4H3gRPNbJOZXR/uOoXJmcAAoLuZfVQ+9Qx3pcKkObDIzD6hrAGywDn3zzDX6bAV1V+bEhGpT1HdQhURqU8KVBERjyhQRUQ8okAVEfGIAlVExCMKVBERjyhQRUQ88v8pMtt6NV1OMgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Let's visualize our annotations after applying attention to them\n",
    "ax = sns.heatmap(applied_attention, annot=True, cmap=sns.light_palette(\"orange\", as_cmap=True), linewidths=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Contrast this with the raw annotations visualized earlier in the notebook, and we can see that the second and third annotations (columns) have been nearly wiped out. The first annotation maintains some of its value, and the fourth annotation is the most pronounced.\n",
    "\n",
    "# Calculating the Attention Context Vector\n",
    "All that remains to produce our attention context vector now is to sum up the four columns to produce a single attention context vector\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([5., 2., 9.])"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def calculate_attention_vector(applied_attention):\n",
    "    return np.sum(applied_attention, axis=1)\n",
    "\n",
    "attention_vector = calculate_attention_vector(applied_attention)\n",
    "attention_vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:>"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAIAAAAEYCAYAAACUfP5RAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAANCElEQVR4nO2dfYxcVRmHn9+WAv2AqhUMthhKqrQW02JrI1ZBQAVrI9RoUgS/YlgTgbagFfxDwZAYiYboPxo2iiEqoBaJBkNto4JRUyiFFrfdYhURlgKtfBTZbWy3+/rHTNkPdu7sLHP3nJ33fZKbnY9z7r6795n33HvmzDsyMwK/tKUOIEhLCOCcEMA5IYBzQgDnhADOCQFaDElrJHVK2iFpbb32IUALIel04DJgKbAQWCHprUV9QoDWYj6w2cx6zawPuA9YWdQhBGgtOoGzJM2UNBVYDpxc1OGocQgq5poH0JA7Gsv/Rl8A2gc90GFmHQBm1iXpRmAT8DKwHegr3Ns4vBdg3/72/8r+HROCdeuOGSJAW1vjAvT3D5WoCEnfBLrN7Pu12oxHBghqoFEfykb2qRPNbK+ktwAfA84sah8CJKQMAYA7Jc0EDgGXm9kLRY1DgISUIYCZva+R9iFAQkrKAA0RAiQkBHBOCOCcHASImUDnRAZISA4ZIARIiDSWWdjmWhMCJCQygHNCAOeEAM5py+AaLARISGQA54QAzgkBnBMCOCcEcE4I4JwQwDkhgHNCAOfkIEAGk5FBSiIDJCSHDBACJCQEcE4I4JwcBIiTwIRI1vBWf5+6qloeplPS7ZKOLWofAiREanwr3p9mAauBJWZ2OjAJWFXUJ4aAhJQ0BBwFTJF0CJgK7ClqHBkgIc3OAGb2FPAd4AngaWC/mW0s6hMCJGQsAkhql/TgoK19YH96PXAhMAd4MzBN0qVFMbT8ENDefjQHDxpm0N8PP/nJodQhvcJYFoVW6wF11Hj6A8C/zGwfgKRfAe8Bflprfy0vAMDPf36IAwdSR/FqSjgHeAJ4d7VC2AHgPODBog4uBMiVZgtgZvdLWg88RKU62MPUzhbAKASQNI/KuDKLSsm3PcBvzKzrNUc8DpjBJz4xGTPYvv0wjzzSnzqkVyipRMx1wHWjbV8ogKRrgIuBO4AHqg/PBm6XdIeZfWusgY4Xt912kJ4emDq1IsLzzxvd3XmULsxhJrBeBvg8sMDMhpw5SboJ2AGMKED1zLQd4OabbwY+89ojHSM9PZWfvb2we3c/J53URnf34WTxDCYHAeqdh/ZTuZwYzknV50bEzDrMbImZLWlvb6/VrHQmT65sR26fckob+/blMwTkQL0MsBb4vaTdwJPVx94CzAWuKDGupjB1Klx0UcWAtjbo6urn8cfzSP+QRwYoFMDMNkh6G5Xy47OoVCfoBraYWR55tID9++HWW/O57h9O9gIAmFk/sHkcYnHHhBAgKI8QwDkhgHNCAOeEAM4JAZwztjqBzSUESEhkAOeEAM4JAZwTAjgnBHBOCOCcHErFZhBCkJLIAAmJIcA5IYBzQgDnhADOCQGck4MAcRmYkBIqhJwmadug7SVJa4v6RAZISAkfDn0UWFTZtyYBTwF3FfUJARJS8hBwHvBPM/t3UaMQICElC7AKuL1eozgHSMhYysQVlYgZ2K+OBj4K/LJeDJEBEjKWDFCnRMwRPgw8ZGbP1ttfCJCQEoeAixlF+ocQICllCFCtD/RB4AujaR8CJKSkEjG9wMzRtg8BEhIzgUFyIgMkJIcMEAIkJARwTg6LQkOAhEQGcI4bAdatO2Y8fs2Ew40A+/e/PB6/JntmzJg+5L4bAYKRCQGcEwI4JwRwTgjgnBDAOSGAc6JMnHMiAzgnBwEyeD8qSElkgITkkAFCgISEAM4JAZwTAjgnBHBOCOCcHASIeYCEtLU1vtVD0uskrZe0S1KXpDOL2kcGSEhJGeB7wAYz+3i1TsDUosYhQEKaLYCk44GzgM8CmNlB4GBRnxgCEtLsKmHAqcA+4MeSHpb0Q0nTijqEAAkZiwB1SsQcBbwT+IGZnQH0ANcWxRBDQEJKKBHTDXSb2f3V++upI0BkgIQ0ewgws2eAJyWdVn3oPGBnUZ/IAAkp6SrgSuBn1SuAx4DPFTUOAVoMM9sGLBlt+xAgIbEm0Dk5TAWHAAkJAZwTAjgnBHBOCOCcEMA5IUDJPPvsM1x//dd57rnnkNpYuXIlq1Z9MnVYrxAClMykSZNYs+Yq5s2bT09PD5/+9KUsXfpuTj311NShAXkI0NJvBr3xjScwb958AKZNm8acOXPYt29v4qgGKGE9QMO0dAYYzJ49e3j00V0sWHB66lBeYUJnAEk132UavGiho6Pet5uUT29vL9deu46rr/4y06dPr99hnChjUWijvJYM8A3gxyM9MWzRgqWsE9jXd4hrrlnH+ed/mHPOOTdZHCORQwYoFEDSI7WeAt7U/HCai5lxww03MGfOHC655NLU4byK7AWgcpDPB14Y9riAv5YSURPZvn0b99zzW+bOncsll1wMwBe/eDnLlr03cWQVJoIAdwPTq4sMhiDp3jICaiaLFp3BAw9sTR1G1hQKYGafL3gunxmVCcpEyABBiYQAzgkBnBNrAp0TGcA5IYBzQgDnhADOCQGcU9LXxz8O/Bc4DPSZWeHHxEKAhJSYAc4xs/+MpmEIkJAchoCWXhKWOyUtCTNgo6Stw6qHjEhkgISMJQNUD+rgA9tRXYBzhGVmtkfSicAmSbvM7E+19hcCJKSEEjGY2Z7qz72S7gKWAjUFiCEgIc0eAiRNk3TckdvAh4DOoj6RAVqLNwF3qWLKUcBtZrahqEMIkJBmr/I1s8eAhY30CQESksNlYAiQkBDAOSGAc0IA54QAzok1gc6JDOCcEMA5IYBzQgDnhADOCQGcEwI4JwRwTg4CxIog50QGSEgOGSAESIgbAWbMyKc4Y064ESCHPzQHbNibfzn8X2IISEgZpV8bJQRISGQA54QAzgkBnBMCOCeHNYEZnIf6payvjJE0SdLDku6u1zYyQEJKHALWAF3A8fUaRgZISBkZQNJs4CPAD0cTQ2SAhJSUAb4LfAU4bjSNIwMkZCwZYPAXclW39oH9aQWw18xG/S0ZkQEmGHVKxCwDPippOXAscLykn5pZzS9MigyQkGafA5jZV81stpmdAqwC/lB08CEyQFJiIsg5ZQpgZvcC99ZrFwIkJDKAc0IA54QAzgkBnBMCOCcEcE4I4JxYFeycyADOyWFJWAiQkMgAzgkBnJODABmchwYpiQyQkBwyQAiQkBDAOSGAc0IA5+QgQMtfBaxeDX/7G3R2wpo1qaMZSlmfDWyElhZgwQK47DJYuhQWLoQVK2Du3NRRDRAClMz8+bB5Mxw4AIcPw333wcqVqaMaYEIIIGmepPMkTR/2+AXND6e5dHbCWWfBG94AU6bA8uVw8smpoxogewEkrQZ+DVwJdEq6cNDT32x+OM1l1y648UbYtAk2bIDt26GvL3VUA2QvAHAZsNjMLgLeD3xN0pFTqZrhDP4AY0dHzW86HxduuQUWL4azz4bnn4fdu5OGM4QcBJANr144JEDtNLO3D7o/HVgP7ATONbNFo/gdlvJy54QTYN++SurfuBHOPBNefDFNLGZDXzRbtvQ2vCDgXe+a2tT/Zr15gGckLTKzbQBm9nL1I8i3AO9oZiBlceedMHMmHDoEl1+e7uCPRLNfGJKOBf4EHEPl2K43s+sK+9TJALOBPjN7ZoTnlpnZX0YRV9IMkBPDM8DWrY1ngMWLa2cASQKmVV+ok4E/A2vMbHOtPoUZwMy6C54bzcEPCmj2olCrvJpfrt6dXN0KJWvpeYD8sTFsxVQrhG0D9gKbzOz+ovYhQEKaXSIGwMwOV0/OZwNLJZ1eFEO8GZSQsZwb1SkRM7jdi5LuBS4AOmu1iwyQkGbPA0g6QdLrqrenAB8AdhX1iQyQkBKujk4CbpU0icqL+xdmVlgtNARISLMFMLNHgDMa6RMCJCSH+ZEQICEhgHNCAOeEAM4JAZwTAjgnBHBOCOCcEMA5IYBzQgDvhAC+yeD4hwApiSHAOSGAc9raolCkayIDOCcEcE4I4JwQwDkhgHNCAOeEAM4JAZwTAjgnBwHiw6EJKeHDoSdL+qOkLkk7BhX0qklkgISUkAH6gC+Z2UOSjgO2StpkZjtrdYgM0EKY2dNm9lD19n+BLmBWUZ/IAAkp8xxA0ilUPilcWCJmPARQQSGycUNSe7W6RjbMmDG9YQWqJWEGl4XpGP53Ves53gmsNbOXCvdXVCaulZD0oJktSR1H2VTLw90N/M7MbqrXPs4BWohqncAfAV2jOfgQArQay4BPAedK2lbdlhd18HQSmNX4XwZm9mcaXGzs5hwgGJkYApzT8gJIukDSo5L+Iena1PHkRksPAdV6eX8HPgh0A1uAi4umRr3R6hlgKfAPM3vMzA4CdwAX1unjilYXYBbw5KD73dSZG/dGqwsw0iVR6455Y6DVBegGBn9R3GxgT6JYsqTVBdgCvFXSHElHA6uA3ySOKStaeibQzPokXQH8DpgE3GJmOxKHlRUtfRkY1KfVh4CgDiGAc0IA54QAzgkBnBMCOCcEcE4I4Jz/A6SkP8dQcO9FAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 108x324 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Let's visualize the attention context vector\n",
    "plt.figure(figsize=(1.5, 4.5))\n",
    "sns.heatmap(np.transpose(np.matrix(attention_vector)), annot=True, cmap=sns.light_palette(\"Blue\", as_cmap=True), linewidths=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have the context vector, we can concatenate it with the hidden state and pass it through a hidden layer to produce the the result of this decoding time step."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Your turn:\n",
    "Now implement the _general_ and _concat_ attention scores and check it below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Yeah, you need to initialize the matrix first (just use random, the main idea is the dimentionality)\n",
    "Wa = np.random.random((3, 6))\n",
    "def general_attention_score(dec_hidden_state, annotations, Wa):\n",
    "    # TODO: return the product of dec_hidden_state transpose and enc_hidden_states\n",
    "    return \n",
    "    \n",
    "attention_weights_raw = general_attention_score(dec_hidden_state, annotations, Wa)\n",
    "attention_weights_raw"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Some post-processing like above"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# And here you need to initialize both the vector v and the matrix Wa (\n",
    "# (again, random is fine)\n",
    "\n",
    "Wa = # <YOUR CODE HERE> \n",
    "va = # <YOUR CODE HERE> \n",
    "def concat_attention_score(dec_hidden_state, annotations, Wa, va):\n",
    "    # TODO: return the product of dec_hidden_state transpose and enc_hidden_states\n",
    "    return \n",
    "    \n",
    "attention_weights_raw = concat_attention_score(dec_hidden_state, annotations, Wa, va)\n",
    "attention_weights_raw"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# And again some post-processing"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}