week_11_rl_for_seq2sec.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Myashka/MIPT_Applied_ML/blob/main/week_11_rl_for_seq2sec.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R7cvdqY-9rxi"
      },
      "source": [
        "# Practice: Reinforcement Learning for seq2seq\n",
        "_Reference: based on Practical RL_ [week07](https://github.com/yandexdataschool/Practical_RL/blob/master/week07_seq2seq)\n",
        "\n",
        "This time we'll solve a problem of transсribing Hebrew words in English, also known as g2p (grapheme2phoneme)\n",
        "\n",
        " * word (sequence of letters in source language) -> translation (sequence of letters in target language)\n",
        "\n",
        "Unlike what most deep learning practitioners do, we won't only train it to maximize likelihood of correct translation, but also employ reinforcement learning to actually teach it to translate with as few errors as possible."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1wb0Dy9J-ukg"
      },
      "source": [
        "## About the task\n",
        "\n",
        "One notable property of Hebrew is that it's a consonant language. That is, there are no vowels in the written language. One could represent vowels with diacritics above consonants, but you don't expect people to do that in everyday life.\n",
        "\n",
        "Therefore, some Hebrew characters will correspond to several English letters and others - to none, so we should use encoder-decoder architecture to figure that out.\n",
        "\n",
        "![img](https://esciencegroup.files.wordpress.com/2016/03/seq2seq.jpg)\n",
        "_(img: esciencegroup.files.wordpress.com)_\n",
        "\n",
        "Encoder-decoder architectures are about converting anything to anything, including\n",
        " * Machine translation and spoken dialogue systems\n",
        " * [Image captioning](https://cocodataset.org/#captions-2015) and [image2latex](https://htmlpreview.github.io/?https://github.com/openai/requests-for-research/blob/master/_requests_for_research/im2latex.html) (convolutional encoder, recurrent decoder)\n",
        " * Generating [images by captions](https://arxiv.org/abs/1511.02793) (recurrent encoder, convolutional decoder)\n",
        " * Grapheme2phoneme - convert words to transcripts\n",
        "\n",
        "We chose simplified __Hebrew->English__ machine translation for words and short phrases (character-level), as it is relatively quick to train even without a gpu cluster."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hKzrzfKL-x4K"
      },
      "source": [
        "## Setup"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "tY-2UIDJ9rxl",
        "outputId": "660073e4-10e0-4dcd-93b4-3a34ac4ee11b",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2022-08-24 08:05:27--  https://raw.githubusercontent.com/girafe-ai/ml-mipt/21f_advanced/week1_11_rl_outside_games/basic_model_torch.py\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 7322 (7.2K) [text/plain]\n",
            "Saving to: ‘basic_model_torch.py’\n",
            "\n",
            "basic_model_torch.p 100%[===================>]   7.15K  --.-KB/s    in 0s      \n",
            "\n",
            "2022-08-24 08:05:27 (80.8 MB/s) - ‘basic_model_torch.py’ saved [7322/7322]\n",
            "\n",
            "--2022-08-24 08:05:27--  https://raw.githubusercontent.com/girafe-ai/ml-mipt/21f_advanced/week1_11_rl_outside_games/main_dataset.txt\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 8305565 (7.9M) [text/plain]\n",
            "Saving to: ‘main_dataset.txt’\n",
            "\n",
            "main_dataset.txt    100%[===================>]   7.92M  --.-KB/s    in 0.03s   \n",
            "\n",
            "2022-08-24 08:05:30 (273 MB/s) - ‘main_dataset.txt’ saved [8305565/8305565]\n",
            "\n",
            "--2022-08-24 08:05:30--  https://raw.githubusercontent.com/girafe-ai/ml-mipt/21f_advanced/week1_11_rl_outside_games/voc.py\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.111.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 2581 (2.5K) [text/plain]\n",
            "Saving to: ‘voc.py’\n",
            "\n",
            "voc.py              100%[===================>]   2.52K  --.-KB/s    in 0s      \n",
            "\n",
            "2022-08-24 08:05:31 (41.7 MB/s) - ‘voc.py’ saved [2581/2581]\n",
            "\n"
          ]
        }
      ],
      "source": [
        "!wget -nc https://raw.githubusercontent.com/girafe-ai/ml-mipt/21f_advanced/week1_11_rl_outside_games/basic_model_torch.py -O basic_model_torch.py\n",
        "!wget -nc https://raw.githubusercontent.com/girafe-ai/ml-mipt/21f_advanced/week1_11_rl_outside_games/main_dataset.txt -O main_dataset.txt\n",
        "!wget -nc https://raw.githubusercontent.com/girafe-ai/ml-mipt/21f_advanced/week1_11_rl_outside_games/voc.py -O voc.py"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "mD7MHJWb9rxm"
      },
      "outputs": [],
      "source": [
        "# If True, only translates phrases shorter than 20 characters (way easier).\n",
        "EASY_MODE = True\n",
        "# Useful for initial coding.\n",
        "# If false, works with all phrases (please switch to this mode for homework assignment)\n",
        "\n",
        "# way we translate. Either \"he-to-en\" or \"en-to-he\"\n",
        "MODE = \"he-to-en\"\n",
        "# maximal length of _generated_ output, does not affect training\n",
        "MAX_OUTPUT_LENGTH = 50 if not EASY_MODE else 20\n",
        "REPORT_FREQ = 100  # how often to evaluate validation score"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dAKuALyH9rxn"
      },
      "source": [
        "## Preprocessing\n",
        "\n",
        "We shall store dataset as a dictionary\n",
        "`{ word1:[translation1,translation2,...], word2:[...],...}`.\n",
        "\n",
        "This is mostly due to the fact that many words have several correct translations.\n",
        "\n",
        "We have implemented this thing for you so that you can focus on more interesting parts."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "bp-nSrAO9rxn",
        "outputId": "10d6bcfc-4e36-4625-9a0a-170ceb9bda10",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "dataset size = 130114\n"
          ]
        }
      ],
      "source": [
        "from collections import defaultdict\n",
        "\n",
        "import numpy as np\n",
        "\n",
        "\n",
        "word_to_translation = defaultdict(list)\n",
        "bos, eos = '_', ';'\n",
        "with open(\"main_dataset.txt\", encoding=\"utf-8\") as fin:\n",
        "    for line in fin:\n",
        "        en, he = line[:-1].lower().replace(bos, ' ').replace(eos, ' ').split('\\t')\n",
        "        word, trans = (he, en) if MODE == 'he-to-en' else (en, he)\n",
        "\n",
        "        if len(word) < 3:\n",
        "            continue\n",
        "\n",
        "        if EASY_MODE and max(len(word), len(trans)) > 20:\n",
        "            continue\n",
        "\n",
        "        word_to_translation[word].append(trans)\n",
        "\n",
        "# get all unique lines in source language\n",
        "all_words = np.array(list(word_to_translation.keys()))\n",
        "# get all unique lines in translation language\n",
        "all_translations = np.array([ts for all_ts in word_to_translation.values() for ts in all_ts])\n",
        "\n",
        "print(f\"dataset size = {len(word_to_translation)}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rUIc-pwH9rxo"
      },
      "source": [
        "We shall hold out 10% of all words to be used for validation.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "8ULcOaYN9rxo"
      },
      "outputs": [],
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "\n",
        "train_words, test_words = train_test_split(all_words, test_size=0.1, random_state=42)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CP9Eu_2k9rxp"
      },
      "source": [
        "## Building vocabularies\n",
        "\n",
        "We now need to build vocabularies that map strings to token ids and vice versa. We're gonna need these fellas when we feed training data into model or convert output matrices into english words."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "m0kSks8-2DF-",
        "outputId": "b3ca66a9-54e6-473e-c569-a78270df1fbc",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array(['משתמש:צלף/!', 'סימן קריאה', 'תבנית:!!', ..., '星野仙一', 'user:榎',\n",
              "       'מומנט מגנט'], dtype='<U20')"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ],
      "source": [
        "all_words"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "t6pONzuq9rxq"
      },
      "outputs": [],
      "source": [
        "from voc import Vocab\n",
        "\n",
        "\n",
        "input_vocab = Vocab.from_lines(''.join(all_words), bos=bos, eos=eos, sep='')\n",
        "output_vocab = Vocab.from_lines(''.join(all_translations), bos=bos, eos=eos, sep='')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "id": "zgzsRiWo9rxq",
        "outputId": "6822985d-ad51-4603-b10b-bb47e5cecda2",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Lines:\n",
            "['משתמש:צלף/!' 'סימן קריאה' 'תבנית:!!' '$9.99' \"תבנית:'\"]\n",
            "\n",
            "Words to ids (0 = bos, 1 = eos):\n",
            "[[  0 127 138 139 127 138  27 135 125 132  16   3   1]\n",
            " [  0 130 122 127 128   2 136 137 122 113 117   1   1]\n",
            " [  0 139 114 129 122 139  27   3   3   1   1   1   1]\n",
            " [  0   6  26  15  26  26   1   1   1   1   1   1   1]\n",
            " [  0 139 114 129 122 139  27   8   1   1   1   1   1]]\n",
            "\n",
            "Back to lines:\n",
            "['משתמש:צלף/!', 'סימן קריאה', 'תבנית:!!', '$9.99', \"תבנית:'\"]\n"
          ]
        }
      ],
      "source": [
        "# Here's how you cast lines into ids and backwards.\n",
        "batch_lines = all_words[:5]\n",
        "batch_ids = input_vocab.to_matrix(batch_lines)\n",
        "batch_lines_restored = input_vocab.to_lines(batch_ids)\n",
        "\n",
        "print(\"Lines:\")\n",
        "print(batch_lines)\n",
        "print()\n",
        "print(\"Words to ids (0 = bos, 1 = eos):\")\n",
        "print(batch_ids)\n",
        "print()\n",
        "print(\"Back to lines:\")\n",
        "print(batch_lines_restored)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z_EUPt909rxr"
      },
      "source": [
        "Draw word/translation length distributions to estimate the scope of the task."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "id": "D_0Kt-4E9rxr",
        "outputId": "83b05083-041f-4833-c598-2d214409fcf0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 281
        }
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 576x288 with 2 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfMAAAEICAYAAABLWh2RAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAb1UlEQVR4nO3dfbRfVX3n8fdHEOsDSpAMgwkYrGkdZE2t3gKtfaClQgDbMLMUabskWtp0pljtwxoLrWvhqMzg9IFKVWZiSQmOFSi1Q1poaRaKD10FCeoSASkphJI0QCQBVKo2+J0/zr7443pvcnPv7z6ce9+vtbLuOfvsc3775N79+56zzz57p6qQJEn99Yy5LoAkSZoeg7kkST1nMJckqecM5pIk9ZzBXJKknjOYS5LUcwZzzaokNyX5pbkuh6TvSHJ5kvdMY/+vJXnJMMuk/WMwl6R5IsnWJD891+XYm/EuyKvqeVV171yVSQZzzZB0/PuShiTJgXNdBs1fftkKgCRvTvJXA+v3JPnzgfUHkrwiyY8kuTXJY+3njwzkuSnJhUn+HngCeEmS1yT5csv/fiAD+V+a5JNt21eSXDVLpyvNO0k+DBwF/FVrtn57kkpyTpJ/Bj7e8v15kgdbvflUkpcPHOPyJB9Icl2Srya5Jcn3tm1JcnGSh5M8nuT2JMeOU44lSf46yc4ku9vy8rbtQuDHgPe3Mr6/pVeSl7blFyS5ou1/f5J3jF7YJ3lTks8k+f127PuSnDrw2W9Kcm8r+31JfmGG/rsXHIO5Rn0S+LEkz0jyIuAg4IcB2rOw5wH/DFwHXAK8EPhD4LokLxw4zhuBtcDBwGPAx4B3AIcB/wS8eiDvu4G/A5YAy4E/nqmTk+a7qnojXR37map6HnB12/QTwH8ATmnrfwOsBP4d8DngI2MOdRbw3+nq1RbgwpZ+MvDjwPcBLwDOBB4ZpyjPAP4UeDHdxcW/Au9vZfxd4NPAW1rT+lvG2f+P2/Ff0sp+NvDmge3HA3fTfSf8L+CydqHxXLrvllOr6mDgR4AvjHN8jcNgLgDa866vAq+gq/A3AP+S5GV0FfLTwOnAPVX14araU1UfBb4M/MzAoS6vqjuqag9wKnBHVV1TVf8G/BHw4EDef6P7wnhRVX2jqj4zw6cp9dE7q+rrVfWvAFW1vqq+WlXfBN4J/ECSFwzk/8uq+myrgx+hq9PQ1beDgZcBqaq7qmrH2A+rqkeq6i+q6omq+irdxcBPTKagSQ6gu5g4v5VxK/AHdBf5o+6vqg9V1ZPABuAI4PC27dvAsUmeXVU7quqOyXyuDOZ6uk8CJ9IF808CN9FV4p9o6y8C7h+zz/3AsoH1BwaWXzS4Xt2sPoPb307X7P7ZJHck+cVhnIS0wDxVZ5IckOSiJP+U5HFga9t02ED+wQvmJ+ha1aiqj9PdYX8AeDjJuiTPH/thSZ6T5P+0JvLHgU8Bh7RAvS+HAc/k6d8TY78jnipfVT3RFp9XVV8H3gD8F2BHe1Twskl8pjCY6+lGg/mPteVP8vRg/i90d9KDjgK2D6wPTsO3AzhydCVJBter6sGq+uWqehHwK8AHR5+7SYvUeNNYDqb9PLAa+Gm6puwVLT1MQlVdUlWvAo6ha27/b+Nk+y3g+4Hjq+r5dBf3g5+xt6k2v8J3WtxGjf2O2Fv5bqiq19DdrX8Z+NBk9pPBXE/3SeAngWdX1Ta6pvVVdM/HPw9cD3xfkp9PcmCSN9B9Kfz1BMe7Dnh5kv/ceuK+Ffj3oxuTvH60Yw2wm+5L4tszcF5SXzxE96x5IgcD36R71v0c4H9M9sBJfijJ8UmeCXwd+Abj17eD6Z6TP5rkUOCCyZaxNZ1fDVyY5OAkLwZ+E/i/kyjf4UlWt2fn3wS+NkH5NA6DuZ5SVf9IV4E+3dYfB+4F/r6qnqyqR4DX0l25P0LXTP7aqvrKBMf7CvB64KKWfyXw9wNZfgi4JcnXgI3A23xXVYvc/wTekeRR4HXjbL+Crtl6O3AncPN+HPv5dHe6u9sxHgF+b5x8fwQ8m+4u+2bgb8dsfx/wutYb/ZJx9v81uouFe4HPAH8GrJ9E+Z5BF/j/BdhF1yL4Xyexn+g6Qcx1GSRJ0jR4Zy5JUs8ZzCVJ6jmDuSRJPWcwlySp53o7cP9hhx1WK1asmOtiSPPebbfd9pWqWjrX5ZiIdVmanL3V5d4G8xUrVrB58+a5LoY07yUZO2rfvGJdliZnb3XZZnZJknrOYC5JUs8ZzCVJ6jmDuSRJPWcwlySp5wzmkiT1nMFckqSeM5hLktRzBnNJknqutyPAaXhWnHfdUI6z9aLTh3IcSQvHZL5f/O6YPoO5JGlKhnUjoOmzmV2SpJ7bZzBPsj7Jw0m+NJD2e0m+nOSLSf4yySED285PsiXJ3UlOGUhf1dK2JDlvIP3oJLe09KuSHDTME5QkaaGbzJ355cCqMWmbgGOr6j8C/wicD5DkGOAs4OVtnw8mOSDJAcAHgFOBY4Cfa3kB3gtcXFUvBXYD50zrjCRJWmT2Gcyr6lPArjFpf1dVe9rqzcDytrwauLKqvllV9wFbgOPavy1VdW9VfQu4ElidJMBPAde0/TcAZ0zznCRJWlSG0QHuF4Gr2vIyuuA+altLA3hgTPrxwAuBRwcuDAbzf5cka4G1AEcdddS0Cy5JmnuT7Uhnr/eJTasDXJLfBfYAHxlOcfauqtZV1UhVjSxdunQ2PlKSpHlvynfmSd4EvBY4qaqqJW8HjhzItrylMUH6I8AhSQ5sd+eD+SVJ0iRM6c48ySrg7cDPVtUTA5s2AmcleVaSo4GVwGeBW4GVref6QXSd5Da2i4BPAK9r+68Brp3aqUiStDhN5tW0jwL/AHx/km1JzgHeDxwMbEryhST/G6Cq7gCuBu4E/hY4t6qebHfdbwFuAO4Crm55AX4b+M0kW+ieoV821DOUJGmB22cze1X93DjJEwbcqroQuHCc9OuB68dJv5eut7skSZoCh3NdwBxqUZIWB4dzlSSp5wzmkiT1nMFckqSeM5hLi8gEEycdmmRTknvazyUtPUkuaZMgfTHJKwf2WdPy35NkzUD6q5Lc3va5pA3ZLGmGGcylxeVyvnvipPOAG6tqJXBjW4duYqSV7d9a4FLogj9wAd2QzMcBF4xeALQ8vzyw39jPkjQD7M0uLSJV9akkK8YkrwZObMsbgJvoxn9YDVzRBne6OckhSY5oeTdV1S6AJJuAVUluAp5fVTe39CvoJk76m5k7I80U34bpF+/MJR1eVTva8oPA4W15Gd89QdKyfaRvGyf9uyRZm2Rzks07d+6c/hlIi5zBXNJT2l147TPj9D/HSZOkIbKZXUMzmWY5pzCclx5KckRV7WjN6A+39IkmTtrOd5rlR9NvaunLx8kvaYZ5Zy5pI90kR/D0yY42Ame3Xu0nAI+15vgbgJOTLGkd304GbmjbHk9yQuvFfjZOnCTNCu/MpUWkTZx0InBYkm10vdIvAq5ukyjdD5zZsl8PnAZsAZ4A3gxQVbuSvJtuNkSAd412hgN+la7H/LPpOr7Z+U2aBQZzaRGZYOIkgJPGyVvAuRMcZz2wfpz0zcCx0ymjpP1nM7skST1nMJckqecM5pIk9ZzBXJKknrMDnCTNIMdf0GwwmEvSHJvNgO+Y6wuTzeySJPWcwVySpJ4zmEuS1HM+M5ck9YKdCSfmnbkkST23z2CeZH2Sh5N8aSDt0CSbktzTfi5p6UlySZItSb6Y5JUD+6xp+e9JsmYg/VVJbm/7XNJmW5IkSZM0mTvzy4FVY9LOA26sqpXAjW0d4FRgZfu3FrgUuuBPNzvT8cBxwAWjFwAtzy8P7Df2syRJ0l7sM5hX1aeAXWOSVwMb2vIG4IyB9CuqczNwSJIjgFOATVW1q6p2A5uAVW3b86vq5jZD0xUDx5IkSZMw1Wfmh1fVjrb8IHB4W14GPDCQb1tL21v6tnHSx5VkbZLNSTbv3LlzikWXJGlhmXYHuHZHXUMoy2Q+a11VjVTVyNKlS2fjIyVJmvemGswfak3ktJ8Pt/TtwJED+Za3tL2lLx8nXZIkTdJUg/lGYLRH+hrg2oH0s1uv9hOAx1pz/A3AyUmWtI5vJwM3tG2PJzmh9WI/e+BYkiRpEvY5aEySjwInAocl2UbXK/0i4Ook5wD3A2e27NcDpwFbgCeANwNU1a4k7wZubfneVVWjnep+la7H/LOBv2n/JEnSJO0zmFfVz02w6aRx8hZw7gTHWQ+sHyd9M3DsvsohSZLG53CukrRAOL3p4mUwl6QeMFBrbxybXZKknjOYS5LUcwZzSZJ6zmAuSVLPGcwlSeo5g7kkST3nq2mTMNlXQrZedPoMl0SSpO9mMJ+HJnPx4IWDhi3JbwC/RDcL4u10wzEfAVwJvBC4DXhjVX0rybOAK4BXAY8Ab6iqre045wPnAE8Cb62qG2b5VKRFx2Z2SSRZBrwVGKmqY4EDgLOA9wIXV9VLgd10QZr2c3dLv7jlI8kxbb+XA6uADyY5YDbPRVqMDOaSRh0IPDvJgcBzgB3ATwHXtO0bgDPa8uq2Ttt+Upv5cDVwZVV9s6ruo5t06bhZKr+0aBnMJVFV24HfB/6ZLog/Rtes/mhV7WnZtgHL2vIy4IG2756W/4WD6ePs85Qka5NsTrJ5586dwz8haZHxmblmlZ0J56ckS+juqo8GHgX+nK6ZfEZU1TpgHcDIyEjN1OdIi4V35pIAfhq4r6p2VtW/AR8DXg0c0prdAZYD29vyduBIgLb9BXQd4Z5KH2cfSTPEYC4Juub1E5I8pz37Pgm4E/gE8LqWZw1wbVve2NZp2z9eVdXSz0ryrCRHAyuBz87SOUiLls3skqiqW5JcA3wO2AN8nq4Z/DrgyiTvaWmXtV0uAz6cZAuwi64HO1V1R5Kr6S4E9gDnVtWTs3oys8QpSeenxfpqr8FcEgBVdQFwwZjkexmnN3pVfQN4/QTHuRC4cOgFlDQhm9klSeo5g7kkST1nMJckqecM5pIk9Zwd4IZosfailCTNLe/MJUnqOYO5JEk9N61gnuQ3ktyR5EtJPprke5IcneSWJFuSXJXkoJb3WW19S9u+YuA457f0u5OcMr1TkiRpcZlyMHf+Y0mS5ofpNrM7/7EkSXNsysF8tuc/BudAliRpPNNpZh+c//hFwHOZwfmPoZsDuapGqmpk6dKlM/lRkiT1xnTeM39q/mOAJE+b/7jdfY83//G2xTz/sTMtSZKGbTrPzJ3/WJKkeWDKd+bOfyxJ0vwwreFcnf9YkqS55whwkiT1nMFckqSeM5hLktRzBnNJknrOYC5JUs8ZzCVJ6rlpvZq2EDgimySp7xZ9MJckLS6TuYnbetHps1CS4TGYa15aiJVNkmaKz8wlSeo5g7kkST1nMJckqecM5pIASHJIkmuSfDnJXUl+OMmhSTYluaf9XNLyJsklSbYk+WKSVw4cZ03Lf0+SNRN/oqRhMZhLGvU+4G+r6mXADwB3AecBN1bVSuDGtg5wKrCy/VsLXAqQ5FC6mRSPp5s98YLRCwBJM8dgLokkLwB+HLgMoKq+VVWPAquBDS3bBuCMtrwauKI6NwOHJDkCOAXYVFW7qmo3sAlYNYunIi1KBnNJAEcDO4E/TfL5JH+S5LnA4VW1o+V5EDi8LS8DHhjYf1tLmyj9aZKsTbI5yeadO3cO+VSkxcdgLgm6MSdeCVxaVT8IfJ3vNKkDUFUF1DA+rKrWVdVIVY0sXbp0GIeUFjWDuSTo7qC3VdUtbf0auuD+UGs+p/18uG3fDhw5sP/yljZRuqQZZDCXRFU9CDyQ5Ptb0knAncBGYLRH+hrg2ra8ETi79Wo/AXisNcffAJycZEnr+HZyS5M0gxzOVdKoXwM+kuQg4F7gzXQX/FcnOQe4Hziz5b0eOA3YAjzR8lJVu5K8G7i15XtXVe2avVOQFieDuSQAquoLwMg4m04aJ28B505wnPXA+uGWTtLe2MwuSVLPGcwlSeo5g7kkST03rWfmSQ4B/gQ4lu79018E7gauAlYAW4Ezq2p3ktANF3kaXYeZN1XV59px1gDvaId9T1VtQJLmyIrzrpvrIkj7Zbod4EbHcn5d6wH7HOB36MZyvijJeXQDT/w2Tx/L+Xi6sZyPHxjLeYTuguC2JBvbUJCSJM26yVzQbb3o9FkoyeRMuZndsZwlSZofpvPMfFbHcgbHc5YkaTzTCeazOpZzO57jOUuSNMZ0grljOUuSNA9MOZg7lrMkSfPDdHuzO5azJElzbFrB3LGcJUmae44AJ0lSzzlrmha0vg38IElT4Z25JEk9ZzCXJKnnDOaSJPWcwVySpJ4zmEuS1HMGc0mSes5gLklSzxnMJUnqOYO5JEk95whwkiRNwWRGmITZGWXSO3NJknrOYC5JUs8ZzCVJ6jmDuSRJPWcwlySp5wzmkp6S5IAkn0/y12396CS3JNmS5KokB7X0Z7X1LW37ioFjnN/S705yytycibS4GMwlDXobcNfA+nuBi6vqpcBu4JyWfg6wu6Vf3PKR5BjgLODlwCrgg0kOmKWyS4uWwVwSAEmWA6cDf9LWA/wUcE3LsgE4oy2vbuu07Se1/KuBK6vqm1V1H7AFOG52zkBavAzmkkb9EfB24Ntt/YXAo1W1p61vA5a15WXAAwBt+2Mt/1Pp4+zzlCRrk2xOsnnnzp3DPg9p0TGYSyLJa4GHq+q22fi8qlpXVSNVNbJ06dLZ+EhpQXM4V0kArwZ+NslpwPcAzwfeBxyS5MB2970c2N7ybweOBLYlORB4AfDIQPqowX0kzRDvzCVRVedX1fKqWkHXge3jVfULwCeA17Vsa4Br2/LGtk7b/vGqqpZ+VuvtfjSwEvjsLJ2GtGh5Z65FbzKTJczGRAnz1G8DVyZ5D/B54LKWfhnw4SRbgF10FwBU1R1JrgbuBPYA51bVk7NfbGlxmXYwb6+dbAa2V9Vr29X4lXSdYW4D3lhV30ryLOAK4FV0zXFvqKqt7Rjn073q8iTw1qq6YbrlkjQ1VXUTcFNbvpdxeqNX1TeA10+w/4XAhTNXQkljDaOZ3fdSJUmaQ9O6Mx94L/VC4DcH3kv9+ZZlA/BO4FK690/f2dKvAd4/9r1U4L7WbHcc8A/TKZskSfPBbDzKm+6d+ay9lwq+mypJ0nimfGc++F5qkhOHV6SJVdU6YB3AyMhIzcZnSjC5K2tY1B3lJM2h6TSz+16qJEnzwJSb2X0vVZKk+WEm3jP3vVRJkmbRUIK576VKkjR3HM5VkqSeM5hLktRzBnNJknrOYC5JUs8ZzCVJ6jmDuSRJPWcwlySp5wzmkiT13EyMADdvTHZyDEmS+sw7c0mSes5gLklSzxnMJUnqOYO5JEk9ZzCXJKnnDOaSJPXcgn41TZqPJvPK5NaLTp+FkkhaKLwzlySp57wzl4bIgYokzQXvzCVJ6jnvzKV5yOfqkvaHwVwSSY4ErgAOBwpYV1XvS3IocBWwAtgKnFlVu5MEeB9wGvAE8Kaq+lw71hrgHe3Q76mqDbN5LvvioxAtRDazSwLYA/xWVR0DnACcm+QY4DzgxqpaCdzY1gFOBVa2f2uBSwFa8L8AOB44DrggyZLZPBFpMTKYS6KqdozeWVfVV4G7gGXAamD0znoDcEZbXg1cUZ2bgUOSHAGcAmyqql1VtRvYBKyaxVORFiWDuaSnSbIC+EHgFuDwqtrRNj1I1wwPXaB/YGC3bS1tovSxn7E2yeYkm3fu3DnU8kuL0ZSDeZIjk3wiyZ1J7kjytpZ+aJJNSe5pP5e09CS5JMmWJF9M8sqBY61p+e9pz9skzYEkzwP+Avj1qnp8cFtVFd3z9GmrqnVVNVJVI0uXLh3GIaVFbTp35j5jkxaQJM+kC+QfqaqPteSHWvM57efDLX07cOTA7stb2kTpkmbQlIO5z9ikhaP1Tr8MuKuq/nBg00ZgtLVsDXDtQPrZrcXtBOCx1hx/A3BykiXtovzkliZpBg3l1bTZeMbWPmct3V09Rx111DCKLqnzauCNwO1JvtDSfge4CLg6yTnA/cCZbdv1dK+lbaF7Ne3NAFW1K8m7gVtbvndV1a7ZOQVp8Zp2MB/7jK27wO9UVSUZyjO2drx1wDqAkZGRoR1XWuyq6jNAJth80jj5Czh3gmOtB9YPr3SS9mVavdl9xiZJ0tybTm92n7FJkjQPTKeZ3WdskiTNA1MO5j5jkyRpfnAEOEmSes5gLklSzxnMJUnqOYO5JEk9ZzCXJKnnDOaSJPWcwVySpJ4zmEuS1HMGc0mSes5gLklSzxnMJUnqOYO5JEk9ZzCXJKnnDOaSJPWcwVySpJ4zmEuS1HMGc0mSes5gLklSzxnMJUnqOYO5JEk9ZzCXJKnnDOaSJPWcwVySpJ4zmEuS1HMGc0mSem7eBPMkq5LcnWRLkvPmujySps76LM2uA+e6AABJDgA+ALwG2AbcmmRjVd05tyWTtL/msj6vOO+6mf4IaV6aF8EcOA7YUlX3AiS5ElgNGMyl/hl6fTZIS3s3X4L5MuCBgfVtwPFjMyVZC6xtq19LcvcslG08hwFfmaPPBiDvnZHDzvl57Y/9+D/o1XlNVt476fN68UyXZYx91ud5VJeHYSH8ffX9HPpe/snW5wnr8nwJ5pNSVeuAdXNdjiSbq2pkrssxbJ5Xv/T5vOZLXR6GPv8eRvX9HPpefpj+OcyXDnDbgSMH1pe3NEn9Y32WZtl8Cea3AiuTHJ3kIOAsYOMcl0nS1FifpVk2L5rZq2pPkrcANwAHAOur6o45LtbeLIjmwXF4Xv0yL8+rh/V5uubl72E/9f0c+l5+mOY5pKqGVRBJkjQH5kszuyRJmiKDuSRJPWcw3w9Jtia5PckXkmye6/JMR5L1SR5O8qWBtEOTbEpyT/u5ZC7LOBUTnNc7k2xvv7cvJDltLss4FUmOTPKJJHcmuSPJ21p6739nfdbH74S+1/2+1/GZqssG8/33k1X1ir6/0whcDqwak3YecGNVrQRubOt9cznffV4AF7ff2yuq6vpZLtMw7AF+q6qOAU4Azk1yDAvjd9Z3fftOuJx+1/3L6Xcdn5G6bDBfpKrqU8CuMcmrgQ1teQNwxqwWaggmOK/eq6odVfW5tvxV4C66kdZ6/zvT7Op73e97HZ+pumww3z8F/F2S29pwlAvN4VW1oy0/CBw+l4UZsrck+WJropu3TYiTkWQF8IPALSzs31kfLJTvhIXwd9S7Oj7Mumww3z8/WlWvBE6laxr58bku0Eyp7p3FhfLe4qXA9wKvAHYAfzC3xZm6JM8D/gL49ap6fHDbAvud9cWC+07o6d9R7+r4sOuywXw/VNX29vNh4C/pZodaSB5KcgRA+/nwHJdnKKrqoap6sqq+DXyInv7ekjyTrvJ/pKo+1pIX5O+sLxbQd0Kv/476Vsdnoi4bzCcpyXOTHDy6DJwMfGnve/XORmBNW14DXDuHZRma0QrS/Cd6+HtLEuAy4K6q+sOBTQvyd9YHC+w7odd/R32q4zNVlx0BbpKSvITuyhu6YXD/rKounMMiTUuSjwIn0k0d+BBwAfD/gKuBo4D7gTOrqlcdTSY4rxPpmt8K2Ar8ysCzqV5I8qPAp4HbgW+35N+he9bW699ZX/X1O6Hvdb/vdXym6rLBXJKknrOZXZKknjOYS5LUcwZzSZJ6zmAuSVLPGcwlSeo5g7kkST1nMJckqef+P+MnuyNTD8AUAAAAAElFTkSuQmCC\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "\n",
        "%matplotlib inline\n",
        "\n",
        "plt.figure(figsize=[8, 4])\n",
        "plt.subplot(1, 2, 1)\n",
        "plt.title(\"words\")\n",
        "plt.hist(list(map(len, all_words)), bins=20)\n",
        "\n",
        "plt.subplot(1, 2, 2)\n",
        "plt.title('translations')\n",
        "plt.hist(list(map(len, all_translations)), bins=20);"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "juAIXtNu9rxr"
      },
      "source": [
        "## Our model\n",
        "\n",
        "Our architecture consists of two main blocks:\n",
        "* Encoder reads words character by character and outputs code vector (usually a function of last RNN state)\n",
        "* Decoder takes that code vector and produces translations character by character\n",
        "\n",
        "Than it gets fed into a model that follows this simple interface:\n",
        "* __`model(inp, out, **flags) -> logp`__ - takes symbolic int32 matrices of hebrew words and their english translations. Computes the log-probabilities of all possible english characters given english prefices and hebrew word.\n",
        "* __`model.translate(inp, **flags) -> out, logp`__ - takes symbolic int32 matrix of hebrew words, produces output tokens sampled from the model and output log-probabilities for all possible tokens at each tick.\n",
        "  * if given flag __`greedy=True`__, takes most likely next token at each iteration. Otherwise samples with next token probabilities predicted by model.\n",
        "\n",
        "That's all! It's as hard as it gets. With those two methods alone you can implement all kinds of prediction and training."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "id": "vq9SVPb69rxr"
      },
      "outputs": [],
      "source": [
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.nn.functional as F\n",
        "from basic_model_torch import BasicTranslationModel\n",
        "\n",
        "\n",
        "device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')\n",
        "model = BasicTranslationModel(input_vocab, output_vocab, emb_size=64, hid_size=256).to(device)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "id": "feq2JWHs9rxs",
        "outputId": "7b5e2149-b679-4b60-c0e5-198b027d6ef1",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Sample translations:\n",
            "tensor([[  0, 122,  30,  ..., 270, 212,   1],\n",
            "        [  0, 150, 139,  ..., 225,  32, 224],\n",
            "        [  0, 159,  45,  ..., 136,   1, 125]], device='cuda:0')\n",
            "\n",
            "Log-probabilities at each step:\n",
            "tensor([[[  0.0000, -69.0776, -69.0776,  ..., -69.0776, -69.0776, -69.0776],\n",
            "         [ -5.5854,  -5.6326,  -5.7735,  ...,  -5.5296,  -5.6970,  -5.6840],\n",
            "         [ -5.5634,  -5.6384,  -5.7932,  ...,  -5.5752,  -5.6907,  -5.7203],\n",
            "         ...,\n",
            "         [ -5.5291,  -5.6520,  -5.6538,  ...,  -5.5247,  -5.6420,  -5.5730],\n",
            "         [ -5.5715,  -5.6454,  -5.5863,  ...,  -5.5610,  -5.5574,  -5.4377],\n",
            "         [ -5.5761,  -5.7098,  -5.6060,  ...,  -5.6117,  -5.6760,  -5.4474]],\n",
            "\n",
            "        [[  0.0000, -69.0776, -69.0776,  ..., -69.0776, -69.0776, -69.0776],\n",
            "         [ -5.5510,  -5.6639,  -5.7261,  ...,  -5.5163,  -5.6778,  -5.6573],\n",
            "         [ -5.6437,  -5.8234,  -5.6987,  ...,  -5.6174,  -5.6019,  -5.5890],\n",
            "         ...,\n",
            "         [ -5.6192,  -5.7709,  -5.4589,  ...,  -5.6373,  -5.6740,  -5.6264],\n",
            "         [ -5.6535,  -5.7887,  -5.6011,  ...,  -5.5949,  -5.6281,  -5.6774],\n",
            "         [ -5.5892,  -5.8536,  -5.5112,  ...,  -5.5883,  -5.7554,  -5.7631]],\n",
            "\n",
            "        [[  0.0000, -69.0776, -69.0776,  ..., -69.0776, -69.0776, -69.0776],\n",
            "         [ -5.6051,  -5.6183,  -5.7267,  ...,  -5.5345,  -5.7061,  -5.6563],\n",
            "         [ -5.5667,  -5.7669,  -5.6432,  ...,  -5.4648,  -5.8188,  -5.6339],\n",
            "         ...,\n",
            "         [ -5.5863,  -5.6385,  -5.5870,  ...,  -5.5573,  -5.6783,  -5.4920],\n",
            "         [ -5.6023,  -5.6239,  -5.4564,  ...,  -5.6628,  -5.7353,  -5.6061],\n",
            "         [ -5.6781,  -5.6400,  -5.6071,  ...,  -5.8014,  -5.6774,  -5.5275]]],\n",
            "       device='cuda:0', grad_fn=<LogSoftmaxBackward0>)\n"
          ]
        }
      ],
      "source": [
        "# Play around with symbolic translate and symbolic score.\n",
        "input = torch.tensor(np.random.randint(0, 10, [3, 5]), dtype=torch.int64, device=device)\n",
        "target = torch.tensor(np.random.randint(0, 10, [3, 5]), dtype=torch.int64, device=device)\n",
        "\n",
        "# translate inp (with untrained model)\n",
        "output, logp = model.translate(input, greedy=False)\n",
        "\n",
        "print(\"Sample translations:\")\n",
        "print(output)\n",
        "print()\n",
        "print(\"Log-probabilities at each step:\")\n",
        "print(logp)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "id": "qOPM8RDI9rxs",
        "outputId": "f3b63008-ba46-41a1-a169-584562ddf202",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Log-probabilities output:\n",
            "tensor([[[  0.0000, -69.0776, -69.0776,  ..., -69.0776, -69.0776, -69.0776],\n",
            "         [ -5.6822,  -5.6381,  -5.7399,  ...,  -5.7408,  -5.6395,  -5.5474],\n",
            "         [ -5.7123,  -5.6283,  -5.7587,  ...,  -5.8246,  -5.6295,  -5.4864],\n",
            "         [ -5.6419,  -5.5944,  -5.7865,  ...,  -5.6576,  -5.6963,  -5.5851],\n",
            "         [ -5.7414,  -5.6135,  -5.7280,  ...,  -5.6086,  -5.5671,  -5.4998]],\n",
            "\n",
            "        [[  0.0000, -69.0776, -69.0776,  ..., -69.0776, -69.0776, -69.0776],\n",
            "         [ -5.6595,  -5.7631,  -5.6167,  ...,  -5.6197,  -5.5743,  -5.5185],\n",
            "         [ -5.7200,  -5.5235,  -5.8046,  ...,  -5.5992,  -5.5308,  -5.4692],\n",
            "         [ -5.7305,  -5.5877,  -5.8077,  ...,  -5.7579,  -5.5810,  -5.4840],\n",
            "         [ -5.7376,  -5.6107,  -5.7921,  ...,  -5.8447,  -5.5964,  -5.4652]],\n",
            "\n",
            "        [[  0.0000, -69.0776, -69.0776,  ..., -69.0776, -69.0776, -69.0776],\n",
            "         [ -5.7607,  -5.6278,  -5.6483,  ...,  -5.5846,  -5.5660,  -5.5145],\n",
            "         [ -5.6592,  -5.5910,  -5.6959,  ...,  -5.5071,  -5.6486,  -5.5526],\n",
            "         [ -5.7366,  -5.6459,  -5.6875,  ...,  -5.5471,  -5.5492,  -5.5148],\n",
            "         [ -5.6648,  -5.6469,  -5.5922,  ...,  -5.5487,  -5.5124,  -5.5336]]],\n",
            "       device='cuda:0', grad_fn=<LogSoftmaxBackward0>)\n",
            "\n",
            "Log-probabilities of output tokens:\n",
            "tensor([[[-69.0776],\n",
            "         [ -5.6381],\n",
            "         [ -5.7123],\n",
            "         [ -5.6680],\n",
            "         [ -5.7280]],\n",
            "\n",
            "        [[-69.0776],\n",
            "         [ -5.6167],\n",
            "         [ -5.5235],\n",
            "         [ -5.5877],\n",
            "         [ -5.6646]],\n",
            "\n",
            "        [[-69.0776],\n",
            "         [ -5.6698],\n",
            "         [ -5.8125],\n",
            "         [ -5.7585],\n",
            "         [ -5.6928]]], device='cuda:0', grad_fn=<GatherBackward0>)\n"
          ]
        }
      ],
      "source": [
        "# score logp(target | input) with untrained model\n",
        "logp = model(input, target)\n",
        "print(\"Log-probabilities output:\")\n",
        "print(logp)\n",
        "print()\n",
        "print(\"Log-probabilities of output tokens:\")\n",
        "print(torch.gather(logp, dim=2, index=target[:, :, None]))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "id": "zE0iunOh9rxs"
      },
      "outputs": [],
      "source": [
        "def translate(words, max_len=MAX_OUTPUT_LENGTH):\n",
        "    # Convert words to a tensor of indices\n",
        "    words = input_vocab.to_matrix(words)\n",
        "    words = torch.tensor(words, dtype=torch.int64, device=device)\n",
        "\n",
        "    with torch.no_grad():\n",
        "        # YOUR CODE HERE\n",
        "        # Compute translations in form of indices.\n",
        "        # Note: use greedy translation to ensure determinism.\n",
        "        # translation = ...\n",
        "        translation, _ = model.translate(words, greedy=True, max_len=max_len)\n",
        "\n",
        "    # Convert translations back into strings\n",
        "    return output_vocab.to_lines(translation.cpu().numpy())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "id": "sske-BHZ9rxs",
        "outputId": "2f0e5713-d5d7-422e-b1d0-a03758358862",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Sample inputs: ['משתמש:צלף/!' 'סימן קריאה' 'תבנית:!!']\n",
            "Dummy translations: ['íőőʻőőمעקس野δ4陳:0e8ą', 'ţțțɱxוקỏ•??ıο×рt-≤w', 'íőőʻőőمעקس野δ4陳:0e8ą']\n"
          ]
        }
      ],
      "source": [
        "print(\"Sample inputs:\", all_words[:3])\n",
        "print(\"Dummy translations:\", translate(all_words[:3]))\n",
        "trans = translate(all_words[:3])\n",
        "\n",
        "assert translate(all_words[:3]) == translate(\n",
        "    all_words[:3]\n",
        "), \"make sure translation is deterministic (use greedy=True and disable any noise layers)\"\n",
        "assert isinstance(translate(all_words[:3]), list) and isinstance(\n",
        "    translate(all_words[:1])[0], str\n",
        "), \"translate(lines) must return a sequence of strings!\"\n",
        "# Note: if translation freezes, make sure you used max_len parameter"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oi6Iv31S9rxs"
      },
      "source": [
        "## Scoring function\n",
        "\n",
        "LogLikelihood is a poor estimator of model performance.\n",
        "* If we predict zero probability once, it shouldn't ruin entire model.\n",
        "* It is enough to learn just one translation if there are several correct ones.\n",
        "* What matters is how many mistakes model's gonna make when it translates!\n",
        "\n",
        "Therefore, we will use minimal Levenshtein distance. It measures how many characters do we need to add/remove/replace from model translation to make it perfect. Alternatively, one could use character-level BLEU/RougeL or other similar metrics.\n",
        "\n",
        "The catch here is that Levenshtein distance is not differentiable: it isn't even continuous. We can't train our neural network to maximize it by gradient descent."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "id": "K9AW0Pv29rxs"
      },
      "outputs": [],
      "source": [
        "import editdistance\n",
        "\n",
        "\n",
        "def get_distance(word, translation):\n",
        "    \"\"\"\n",
        "    A function that takes word and predicted translation\n",
        "    and evaluates (Levenshtein's) edit distance to closest correct translation\n",
        "    \"\"\"\n",
        "    references = word_to_translation[word]\n",
        "    assert len(references) != 0, \"wrong/unknown word\"\n",
        "    return min(editdistance.eval(translation, reference) for reference in references)\n",
        "\n",
        "\n",
        "def score(words, bsize=100):\n",
        "    \"\"\"a function that computes levenshtein distance for bsize random samples\"\"\"\n",
        "    assert isinstance(words, np.ndarray)\n",
        "    batch_words = np.random.choice(words, size=bsize, replace=False)\n",
        "    batch_translation = translate(batch_words)\n",
        "    distances = [\n",
        "        get_distance(word, translation) for word, translation in zip(batch_words, batch_translation)\n",
        "    ]\n",
        "    return np.array(distances, dtype='float32')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "id": "F90fsOhl9rxt",
        "outputId": "20de8904-783f-45b2-998e-c2534f1fe45a",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[18.8, 18.8, 18.8, 18.6, 18.9]"
            ]
          },
          "metadata": {},
          "execution_count": 20
        }
      ],
      "source": [
        "# should be around 5-50 and decrease rapidly after training :)\n",
        "[score(test_words, 10).mean() for _ in range(5)]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CQ6TZkKE9rxt"
      },
      "source": [
        "## Supervised pre-training\n",
        "\n",
        "Here we define a function that trains our model through maximizing log-likelihood a.k.a. minimizing crossentropy."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "id": "4wFcLk959rxt"
      },
      "outputs": [],
      "source": [
        "import random\n",
        "\n",
        "\n",
        "def sample_batch(words, word_to_translation, batch_size):\n",
        "    # choose words\n",
        "    batch_words = np.random.choice(words, size=batch_size)\n",
        "\n",
        "    # choose translations\n",
        "    batch_reference_translations = [word_to_translation[word] for word in batch_words]\n",
        "    batch_translations = [random.choice(references) for references in batch_reference_translations]\n",
        "\n",
        "    return batch_words, batch_translations"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {
        "id": "kdfjyhsS9rxt",
        "outputId": "20b6e9aa-ea84-47f4-a16b-97df8d7e4f2d",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Source:\n",
            "['רדומישל ויילקי' 'רמת מרקאגנט' 'יום מיוחד']\n",
            "\n",
            "Target:\n",
            "['radomyśl wielki', 'markagunt plateau', 'a special day']\n"
          ]
        }
      ],
      "source": [
        "bx, by = sample_batch(train_words, word_to_translation, batch_size=3)\n",
        "print(\"Source:\")\n",
        "print(bx)\n",
        "print()\n",
        "print(\"Target:\")\n",
        "print(by)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "metadata": {
        "id": "3xcnyaq39rxt"
      },
      "outputs": [],
      "source": [
        "from basic_model_torch import infer_mask\n",
        "\n",
        "\n",
        "def compute_loss_on_batch(input_sequence, reference_answers):\n",
        "    \"\"\" Compute crossentropy loss given a batch of sources and translations \"\"\"\n",
        "    input_sequence = torch.tensor(\n",
        "        input_vocab.to_matrix(input_sequence), dtype=torch.int64, device=device\n",
        "    )\n",
        "    reference_answers = torch.tensor(\n",
        "        output_vocab.to_matrix(reference_answers), dtype=torch.int64, device=device\n",
        "    )\n",
        "\n",
        "    # YOUR CODE HERE\n",
        "    # Compute log-probabilities of all possible tokens at each step. Use model interface.\n",
        "    # logprobs = ...\n",
        "    logprobs = model(input_sequence, reference_answers )\n",
        "\n",
        "    # Compute elementwise crossentropy as negative log-probabilities of reference_answers.\n",
        "    reference_logprobs = torch.gather(logprobs, dim=2, index=reference_answers[:, :, None])\n",
        "    crossentropy = -torch.squeeze(reference_logprobs, dim=2)\n",
        "\n",
        "    # average with mask\n",
        "    mask = infer_mask(reference_answers, output_vocab.eos_ix)\n",
        "    loss = torch.sum(crossentropy * mask) / torch.sum(mask)\n",
        "\n",
        "    return loss"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "metadata": {
        "id": "Eidx8i5s9rxt",
        "outputId": "dd490fda-3a00-4837-91f5-874e33b50850",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "loss = 5.206414222717285\n"
          ]
        }
      ],
      "source": [
        "# test it\n",
        "loss = compute_loss_on_batch(*sample_batch(train_words, word_to_translation, 3))\n",
        "print(f'loss = {loss}')\n",
        "\n",
        "assert loss.item() > 0.0\n",
        "loss.backward()\n",
        "for w in model.parameters():\n",
        "    assert (\n",
        "        w.grad is not None and torch.max(torch.abs(w.grad)).item() != 0\n",
        "    ), f\"loss is not differentiable w.r.t. a weight with shape {w.size()}. Check compute_loss_on_batch.\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CSx1nIFr9rxt"
      },
      "source": [
        "## Actually train the model\n",
        "\n",
        "Minibatches and stuff..."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "metadata": {
        "id": "NRHXz5ZW9rxt",
        "outputId": "c884cfc2-df79-49eb-a7ad-3904d94b27e7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 330,
          "referenced_widgets": [
            "7a0597ecb1c64093a0499f77005e0be9",
            "e4080d6755494094bb2b24acff9ed811",
            "2de2c4bddd3d4474a840fb23eb527538",
            "d0f5c0e99a73402f803073e77a9ce0f9",
            "10008724e14b48a8a84f54b961ed1e75",
            "fe6759547d0b44879b8c18dfe08ec022",
            "871d2b2889ae4862a958d949e6b54e7a",
            "b0fd92d27a5c4fba93d0a582b3ed205b",
            "eb658d1f4fb04a5ab212d7607ddb7f5e",
            "15c83816dddd4df3a1f3e0bfc9c7911e",
            "8d319226a12d4f21ad4a0dd9bd3ad9a3",
            "56684b96d98f464eae622fa17065a54a",
            "03cf0c10675045e4b26e4e4f89ce74a4"
          ]
        }
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Output()"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "7a0597ecb1c64093a0499f77005e0be9"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "  0%|          | 0/5000 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "2de2c4bddd3d4474a840fb23eb527538"
            }
          },
          "metadata": {}
        }
      ],
      "source": [
        "from IPython.display import clear_output, display\n",
        "from ipywidgets import Output\n",
        "from tqdm.auto import tqdm, trange\n",
        "\n",
        "\n",
        "output_display = Output()\n",
        "display(output_display)\n",
        "\n",
        "loss_history = []\n",
        "editdist_history = []\n",
        "entropy_history = []\n",
        "opt = torch.optim.Adam(model.parameters())\n",
        "batch_size = 32\n",
        "\n",
        "for i in trange(5000):\n",
        "    loss = compute_loss_on_batch(*sample_batch(train_words, word_to_translation, batch_size))\n",
        "\n",
        "    # train with backprop\n",
        "    opt.zero_grad()\n",
        "    loss.backward()\n",
        "    opt.step()\n",
        "\n",
        "    loss_history.append(loss.item())\n",
        "\n",
        "    if i % REPORT_FREQ == 0:\n",
        "        with output_display:\n",
        "            clear_output(wait=True)\n",
        "\n",
        "            current_scores = score(test_words)\n",
        "            editdist_history.append(current_scores.mean())\n",
        "            mean_loss = np.mean(loss_history[-10:])\n",
        "            mean_score = np.mean(editdist_history[-10:])\n",
        "            print(f\"mean loss = {mean_loss:.3f}, mean score = {mean_score:.3f}\")\n",
        "\n",
        "            plt.figure(figsize=(12, 4))\n",
        "            plt.subplot(131)\n",
        "            plt.title('train loss')\n",
        "            plt.plot(loss_history)\n",
        "            plt.grid()\n",
        "            plt.subplot(132)\n",
        "            plt.title('val score distribution')\n",
        "            plt.hist(current_scores, bins=20)\n",
        "            plt.subplot(133)\n",
        "            plt.title('val score (lower is better)')\n",
        "            plt.plot(editdist_history)\n",
        "            plt.grid()\n",
        "            plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RGYvH47Q9rxt"
      },
      "source": [
        "__How to interpret the plots:__\n",
        "\n",
        "* __Train loss__ - that's your model's crossentropy over minibatches. It should go down steadily. Most importantly, it shouldn't be NaN :)\n",
        "* __Val score distribution__ - distribution of translation edit distance (score) within batch. It should move to the left over time.\n",
        "* __Val score / training time__ - it's your current mean edit distance. This plot is much whimsier than loss, but make sure it goes below 8 by 2500 steps. \n",
        "\n",
        "If it doesn't, first try to re-create both model and opt. You may have changed it's weight too much while debugging. If that doesn't help, it's debugging time."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 27,
      "metadata": {
        "id": "50zrI-pQ9rxu",
        "outputId": "c8f83ec8-c7c3-4743-9c4e-55008ad0683f",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "מפרש קדמה -> kedama good\n",
            "the information -> the niformanton\n",
            "ביאליניצ'י -> bilanichi\n",
            "סונו ג'וי -> sono jo\n",
            "ויוויאן פוקס -> vivan poks\n",
            "מנורת שמן -> shemen shaman\n",
            "רובין אליס -> roben allis\n",
            "פרנצ'סקו גוארדי -> frances godari\n",
            "צ'איאונו -> chiono\n",
            "תאופיק אל-חכים -> the pichal chail\n"
          ]
        }
      ],
      "source": [
        "for word in train_words[:10]:\n",
        "    translation = translate([word])[0]\n",
        "    print(f\"{word} -> {translation}\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 28,
      "metadata": {
        "id": "9w8yX4Kn9rxu",
        "outputId": "a8a2b477-81c7-440a-dba0-b2ac96901c92",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 66,
          "referenced_widgets": [
            "1b445bf445424d2994fedcdbc526294b",
            "71cebf62bff3491ca3550a1d51c45f2c",
            "96c94bece44643f68b4dd634018878fb",
            "0227351061d54ef691a4957995e02c0b",
            "bef84e8fa7c142c3a551d101129c3cb0",
            "5b4a3c14447348e2a59b52770e55b7ae",
            "753853f9495b44b796907ef9bbec49ba",
            "72eb109eb34f45bd93f0ffa1002ffc46",
            "d61c491a23f44ad398ead9393e8ee000",
            "72857c50d07d4372a7c8aa5533b2fd9d",
            "24259531823c46c3b45de1fcbf981464"
          ]
        }
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "  0%|          | 0/407 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "1b445bf445424d2994fedcdbc526294b"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Supervised test score: 7.030817706732247\n"
          ]
        }
      ],
      "source": [
        "test_scores = []\n",
        "for i in trange(0, len(test_words), batch_size):\n",
        "    batch_words = test_words[i : i + batch_size]\n",
        "    batch_translation = translate(batch_words)\n",
        "    distances = [\n",
        "        get_distance(word, translation) for word, translation in zip(batch_words, batch_translation)\n",
        "    ]\n",
        "    test_scores.extend(distances)\n",
        "\n",
        "print(\"Supervised test score:\", np.mean(test_scores))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ugvkPI169rxu"
      },
      "source": [
        "## Self-critical policy gradient\n",
        "\n",
        "In this section you'll implement algorithm called self-critical sequence training (here's an [article](https://arxiv.org/abs/1612.00563)).\n",
        "\n",
        "The algorithm is a vanilla policy gradient with a special baseline. \n",
        "\n",
        "$$ \\nabla J = E_{x \\sim p(s)} E_{y \\sim \\pi(y|x)} \\nabla log \\pi(y|x) \\cdot (R(x,y) - b(x)) $$\n",
        "\n",
        "Here reward R(x,y) is a __negative levenshtein distance__ (since we minimize it). The baseline __b(x)__ represents how well model fares on word __x__.\n",
        "\n",
        "In practice, this means that we compute baseline as a score of greedy translation, $b(x) = R(x,y_{greedy}(x)) $.\n",
        "\n",
        "![img](https://github.com/yandexdataschool/Practical_RL/raw/master/yet_another_week/_resource/scheme.png)\n",
        "\n",
        "Luckily, we already obtained the required outputs: `model.translate` with `greedy=True` and `greedy=False` and we only need to compute levenshtein using `get_distance` function.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 29,
      "metadata": {
        "id": "PYBaNPjD9rxu"
      },
      "outputs": [],
      "source": [
        "def compute_reward(input_sequence, translations):\n",
        "    \"\"\" computes sample-wise reward given token ids for inputs and translations \"\"\"\n",
        "    distances = [\n",
        "        get_distance(word, translation)\n",
        "        for word, translation in zip(\n",
        "            input_vocab.to_lines(input_sequence.detach().numpy()),\n",
        "            output_vocab.to_lines(translations.detach().numpy()),\n",
        "        )\n",
        "    ]\n",
        "\n",
        "    # use negative levenshtein distance so that larger reward means better policy\n",
        "    return -torch.tensor(distances, dtype=torch.int64, device=device)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 33,
      "metadata": {
        "id": "BnH-8sG59rxu"
      },
      "outputs": [],
      "source": [
        "def scst_objective_on_batch(input_sequence, max_len=MAX_OUTPUT_LENGTH):\n",
        "    \"\"\" Compute pseudo-loss for policy gradient given a batch of sources \"\"\"\n",
        "    input_sequence = torch.tensor(\n",
        "        input_vocab.to_matrix(input_sequence), dtype=torch.int64, device=device\n",
        "    )\n",
        "\n",
        "    # YOUR CODE HERE\n",
        "    # use model to __sample__ symbolic translations given input_sequence\n",
        "    sample_translations, sample_logp = model.translate(input_sequence, greedy=False, max_len=max_len)\n",
        "\n",
        "    # YOUR CODE HERE\n",
        "    # use model to __greedy__ symbolic translations given input_sequence\n",
        "    greedy_translations, greedy_logp = model.translate(input_sequence, greedy=True, max_len=max_len)\n",
        "\n",
        "    # YOUR CODE HERE\n",
        "    # Compute rewards and baseline.\n",
        "    # Be careful with device: rewards require casting to numpy, so send everything to cpu.\n",
        "    rewards = compute_reward(input_sequence.cpu(), sample_translations.cpu())\n",
        "    baseline = compute_reward(input_sequence.cpu(), greedy_translations.cpu())\n",
        "\n",
        "    # YOUR CODE HERE\n",
        "    # compute advantage using rewards and baselineto_one_hot\n",
        "    # Yet again be careful with the device: advantage is used to compute gradients, so send it to device.\n",
        "    advantage = (rewards - baseline).float().to(device)\n",
        "\n",
        "    # YOUR CODE HERE\n",
        "    # Compute log_pi(a_t|s_t), shape = [batch, seq_length]\n",
        "    # Hint: look at how crossentropy is implemented in supervised learning loss above.\n",
        "    # Mind the sign - this one should not be multiplied by -1 :)\n",
        "    logp_sample = torch.gather(sample_logp, dim=2, index=sample_translations[:, :, None]).squeeze(\n",
        "        dim=2\n",
        "    )\n",
        "    # Policy gradient pseudo-loss. Gradient of J is exactly policy gradient.\n",
        "    J = logp_sample * advantage[:, None]\n",
        "\n",
        "    assert J.dim() == 2, \"please return elementwise objective, don't compute mean just yet\"\n",
        "\n",
        "    # average with mask\n",
        "    mask = infer_mask(sample_translations, output_vocab.eos_ix)\n",
        "    loss = -torch.sum(J * mask) / torch.sum(mask)\n",
        "\n",
        "    # YOUR CODE HERE\n",
        "    # Regularize with negative entropy. Don't forget the sign!\n",
        "    # Note: for entropy you need probabilities for all tokens (sample_logp), not just logp_sample\n",
        "    entropy = -torch.sum(torch.exp(sample_logp) * sample_logp, dim=-1)\n",
        "\n",
        "    assert entropy.dim() == 2, \"please make sure elementwise entropy is of shape [batch,time]\"\n",
        "\n",
        "    reg = -0.01 * torch.sum(entropy * mask) / torch.sum(mask)\n",
        "    return loss + reg, torch.sum(entropy * mask) / torch.sum(mask)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZO4sOvJe9rxu"
      },
      "source": [
        "### Policy gradient training\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 37,
      "metadata": {
        "id": "s86YtJXh9rxu",
        "outputId": "0b6258af-bbae-4393-c72d-e0d806b9d1fe",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 330,
          "referenced_widgets": [
            "15431de86be6410fadc00f2a91140b91",
            "82170e57a38649ef9a83963fbb203de0",
            "69560af419f1475ab14258a656d60681",
            "816175c58d1e4f1a9d075625334a60a5",
            "d7d4c7cc6dda46189c6977489efc71b7",
            "df11bb5436274efcba592438647bf15a",
            "1826302cbf9043f0911f001a85dc74ec",
            "e2e7053c3f76474492116e56709e9cf1",
            "62b2f9742d07437a89362ec42cf59e2b",
            "df1efb6dcb8044bf824808fe8a905315",
            "293364819f86406e9706cfba7d44e728",
            "50543448c11f472f9775cba92c864fa9",
            "2f87d2ce228a419389f7ae8010ba6185"
          ]
        }
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Output()"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "15431de86be6410fadc00f2a91140b91"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "  0%|          | 0/5000 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "69560af419f1475ab14258a656d60681"
            }
          },
          "metadata": {}
        }
      ],
      "source": [
        "entropy_history = [np.nan] * len(loss_history)\n",
        "opt = torch.optim.Adam(model.parameters(), lr=1e-5)\n",
        "\n",
        "debug_list = []\n",
        "\n",
        "output_display = Output()\n",
        "display(output_display)\n",
        "\n",
        "# Feel free to use more steps if necessary.\n",
        "for i in trange(5000):\n",
        "    loss, ent = scst_objective_on_batch(\n",
        "        sample_batch(train_words, word_to_translation, 32)[0]\n",
        "    )  # [0] = only source sentence\n",
        "\n",
        "    # train with backprop\n",
        "    opt.zero_grad()\n",
        "    loss.backward()\n",
        "    opt.step()\n",
        "\n",
        "    loss_history.append(loss.item())\n",
        "    entropy_history.append(ent.item())\n",
        "    debug_list.append(current_scores)\n",
        "\n",
        "    if (i + 1) % REPORT_FREQ == 0:\n",
        "        with output_display:\n",
        "            clear_output(wait=True)\n",
        "\n",
        "            current_scores = score(test_words)\n",
        "            editdist_history.append(current_scores.mean())\n",
        "            mean_loss = np.mean(loss_history[-10:])\n",
        "            mean_score = np.mean(editdist_history[-10:])\n",
        "            print(f\"J={mean_loss:.3f}, mean score={mean_score:.3f}\")\n",
        "\n",
        "            plt.figure(figsize=(12, 4))\n",
        "            plt.subplot(131)\n",
        "            plt.title('val score distribution')\n",
        "            plt.hist(current_scores, bins=20)\n",
        "            plt.subplot(132)\n",
        "            plt.title('val score / traning time')\n",
        "            plt.plot(editdist_history)\n",
        "            plt.grid()\n",
        "            plt.subplot(133)\n",
        "            plt.title('policy entropy / traning time')\n",
        "            plt.plot(entropy_history)\n",
        "            plt.grid()\n",
        "            plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JmrG0CBu9rxv"
      },
      "source": [
        "__Debugging tips:__\n",
        "![img](https://github.com/yandexdataschool/Practical_RL/raw/master/yet_another_week/_resource/do_something_scst.png)\n",
        "\n",
        " * As usual, don't expect improvements right away, but in general the model should be able to show some positive changes by 5k steps.\n",
        " * Entropy is a good indicator of many problems. \n",
        "   * If it reaches zero, you may need greater entropy regularizer.\n",
        "   * If it has rapid changes time to time, you may need gradient clipping.\n",
        "   * If it oscillates up and down in an erratic manner... it's perfectly okay for entropy to do so. But it should decrease at the end.\n",
        "   \n",
        " * We don't show loss_history cuz it's uninformative for pseudo-losses in policy gradient. However, if something goes wrong you can check it to see if everything isn't a constant zero."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m8__jC3T9rxv"
      },
      "source": [
        "### Results"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 38,
      "metadata": {
        "id": "H-Ovy4Cs9rxv",
        "outputId": "03f5746b-5cea-492d-99bf-f8db63da49f7",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "מפרש קדמה -> kadama\n",
            "the information -> the niforman\n",
            "ביאליניצ'י -> bilinici\n",
            "סונו ג'וי -> sono gay\n",
            "ויוויאן פוקס -> vivin poks\n",
            "מנורת שמן -> saman san\n",
            "רובין אליס -> roban lian\n",
            "פרנצ'סקו גוארדי -> francos gardi\n",
            "צ'איאונו -> chiono\n",
            "תאופיק אל-חכים -> theopil car\n"
          ]
        }
      ],
      "source": [
        "for word in train_words[:10]:\n",
        "    translation = translate([word])[0]\n",
        "    print(f\"{word} -> {translation}\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 39,
      "metadata": {
        "id": "OQwarlFJ9rxv",
        "outputId": "34c364af-c340-41fe-9b8a-4387d0ff6235",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 66,
          "referenced_widgets": [
            "d4794b4d68944c3e94ae6133e6d5dd26",
            "d677bd7504894a49947bd94e88f5f5f3",
            "9b14323dc89349a8b43d636261d9c406",
            "f4ee6312a3f540b8802e0955905ae183",
            "06c4b51625954ed99e6b6239a7160e27",
            "1cf58205a6ef46b2b7e6f1a9d1ec810d",
            "549e2f92a9e047df96b20279118201e2",
            "310d161a4e734a708e85cf8dfc413088",
            "30e427c09c2b471baa0a823eab7f9de1",
            "6600572b484644c3865a54015f8aa0a9",
            "5153fd99a12c4362806f9ad33cc26be2"
          ]
        }
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "  0%|          | 0/407 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "d4794b4d68944c3e94ae6133e6d5dd26"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "SCST test score: 6.424684906240394\n"
          ]
        }
      ],
      "source": [
        "test_scores = []\n",
        "for i in trange(0, len(test_words), batch_size):\n",
        "    batch_words = test_words[i : i + batch_size]\n",
        "    batch_translation = translate(batch_words)\n",
        "    distances = [\n",
        "        get_distance(word, translation) for word, translation in zip(batch_words, batch_translation)\n",
        "    ]\n",
        "    test_scores.extend(distances)\n",
        "\n",
        "print(\"SCST test score:\", np.mean(test_scores))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xobu7apf9rxv"
      },
      "source": [
        "## Bonus: make it actually work\n",
        "\n",
        "In this section, we want you to finally __restart with EASY_MODE=False__ and experiment to find a good model/curriculum for that task.\n",
        "\n",
        "We recommend you to start with the following architecture\n",
        "\n",
        "```\n",
        "encoder---decoder\n",
        "\n",
        "           P(y|h)\n",
        "             ^\n",
        " LSTM  ->   LSTM\n",
        "  ^          ^\n",
        " biLSTM  ->   LSTM\n",
        "  ^          ^\n",
        "input       y_prev\n",
        "```\n",
        "\n",
        "__Note:__ you can fit all 4 state tensors of both LSTMs into in a single state - just assume that it contains, for example, [h0, c0, h1, c1] - pack it in encode and update in decode.\n",
        "\n",
        "\n",
        "Here are some cool ideas on what you can do then.\n",
        "\n",
        "__General tips & tricks:__\n",
        "* You will likely need to adjust pre-training time for such a network.\n",
        "* Supervised pre-training may benefit from clipping gradients somehow.\n",
        "* SCST may indulge a higher learning rate in some cases and changing entropy regularizer over time.\n",
        "* It's often useful to save pre-trained model parameters to not re-train it every time you want new policy gradient parameters.\n",
        "* When leaving training for nighttime, try setting REPORT_FREQ to a larger value (e.g. 500) not to waste time on it.\n",
        "\n",
        "__Formal criteria:__\n",
        "To get 5 points, we want you to build an architecture that:\n",
        "* _doesn't consist of single GRU_\n",
        "* _works better_ than single GRU baseline.\n",
        "* We also want you to provide either learning curve or trained model, preferably both\n",
        "* ... and write a brief report or experiment log describing what you did and how it fared.\n",
        "\n",
        "### Attention\n",
        "There's more than one way to connect decoder to encoder\n",
        "  * __Vanilla:__ layer_i of encoder last state goes to layer_i of decoder initial state\n",
        "  * __Every tick:__ feed encoder last state _on every iteration_ of decoder.\n",
        "  * __Attention:__ allow decoder to \"peek\" at one (or several) positions of encoded sequence on every tick.\n",
        "\n",
        "The most effective (and cool) of those is, of course, attention.\n",
        "You can read more about attention [in this nice blog post](https://distill.pub/2016/augmented-rnns/). The easiest way to begin is to use \"soft\" attention with \"additive\" or \"dot-product\" intermediate layers.\n",
        "\n",
        "__Tips__\n",
        "* Model usually generalizes better if you no longer allow decoder to see final encoder state\n",
        "* Once your model made it through several epochs, it is a good idea to visualize attention maps to understand what your model has actually learned\n",
        "\n",
        "* There's more stuff [here](bonus_pytorch.ipynb)\n",
        "* If you opted for hard attention, we recommend [gumbel-softmax](https://blog.evjang.com/2016/11/tutorial-categorical-variational.html) instead of sampling. Also, please make sure soft attention works fine before you switch to hard.\n",
        "\n",
        "### UREX\n",
        "* This is a way to improve exploration in policy-based settings. The main idea is that you find and upweight under-appreciated actions.\n",
        "* Here's [video](https://www.youtube.com/watch?v=fZNyHoXgV7M&feature=youtu.be&t=3444)\n",
        " and an [article](https://arxiv.org/abs/1611.09321).\n",
        "* You may want to reduce batch size 'cuz UREX requires you to sample multiple times per source sentence.\n",
        "* Once you got it working, try using experience replay with importance sampling instead of (in addition to) basic UREX.\n",
        "\n",
        "### Some additional ideas:\n",
        "* (advanced deep learning) It may be a good idea to first train on small phrases and then adapt to larger ones (a.k.a. training curriculum).\n",
        "* (advanced nlp) You may want to switch from raw utf8 to something like unicode or even syllables to make the task easier.\n",
        "* (advanced nlp) Since Hebrew words are written __with vowels omitted__, you may want to use a small Hebrew vowel markup dataset at `he-pron-wiktionary.txt`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "HjOSzs5Y9rxv"
      },
      "outputs": [],
      "source": [
        "assert not EASY_MODE, \"make sure you set EASY_MODE = False at the top of the notebook.\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U_Vyaoic9rxv"
      },
      "source": [
        "`[your report/log here or anywhere you please]`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mN41Cy4p9rxv"
      },
      "source": [
        "__Contributions:__ This notebook is brought to you by\n",
        "* Yandex [MT team](https://tech.yandex.com/translate/)\n",
        "* Denis Mazur ([DeniskaMazur](https://github.com/DeniskaMazur)), Oleg Vasilev ([Omrigan](https://github.com/Omrigan/)), Dmitry Emelyanenko ([TixFeniks](https://github.com/tixfeniks)) and Fedor Ratnikov ([justheuristic](https://github.com/justheuristic/))\n",
        "* Dataset is parsed from [Wiktionary](https://en.wiktionary.org), which is under CC-BY-SA and GFDL licenses.\n"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [],
      "name": "binpord_practice_rl_for_seq2sec_clean.ipynb",
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.7"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "7a0597ecb1c64093a0499f77005e0be9": {
          "model_module": "@jupyter-widgets/output",
          "model_name": "OutputModel",
          "model_module_version": "1.0.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/output",
            "_model_module_version": "1.0.0",
            "_model_name": "OutputModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/output",
            "_view_module_version": "1.0.0",
            "_view_name": "OutputView",
            "layout": "IPY_MODEL_e4080d6755494094bb2b24acff9ed811",
            "msg_id": "",
            "outputs": [
              {
                "output_type": "stream",
                "name": "stdout",
                "text": [
                  "mean loss = 1.359, mean score = 6.536\n"
                ]
              },
              {
                "output_type": "display_data",
                "data": {
                  "text/plain": "<Figure size 864x288 with 3 Axes>",
                  "image/png": "iVBORw0KGgoAAAANSUhEUgAAAroAAAEICAYAAACwF1f6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd3Rc1bXH8e+WRsVF7rbcLRt3XEGmY2Rjqikh9BBCjUMeIZQkPIeahBK/FEJ7eeDQW+iEYpoNCGNsbNx7w1VucrfkonreH/dKHkmjPtKMpd9nLS/NzG1b8ty5e87d5xxzziEiIiIi0tDERDoAEREREZG6oERXRERERBokJboiIiIi0iAp0RURERGRBkmJroiIiIg0SEp0RURERKRBUqJ7BDKzp8zs3hpum25mN4Y7JpFIM7M0M8uIdBw1ZWYvmNmD/uNTzWxFGPf9iZld4z++1symhXHfV5nZ5+HanxxZjsTzzsz+bGa3+Y+jLn4zu8vMnqnBduvMbExdxFQTZpZgZsvNrH0k41CiW8/C8UZ0zt3knHsgXDGJSHRxzn3jnOtX2Xpm9gcze6UK+zvHOfdibeMysxQzc2YWCNr3q865M2u7b5H64CddPwOejnQs5XHOPeycq9cGqdK5SahzvbqccznAc8D4cMRYU0p0o0xt3lQiEhnRet6aR5/z0iDV8Ly7FvjYOXcwzOHUSLR+dtRW0O/1GnCNmSVEKhZ9ANYjM3sZ6A58aGbZZnZn0LemG8xsA/Clv+5bZrbVzPaa2VQzOzpoP8G3ONPMLMPMfmNmmWa2xcyuq2I8MWZ2j5mt97d9ycxa+ssSzewVM9tpZnvM7HszS/aXXWtma8wsy8zWmtlVYf5TSSNlZv9tZm+Xeu0xM3vcf3ydmS3z33trzOwXVdyvmdk//Pf5PjNbZGaD/GVNzOzv/nmw18ymmVkTf9kFZrbEPwfSzWxA0D7X+fEuBPabWcDMTjCz6f76C8wsrYKYhpvZXP93eQNIDFpW4naqf5xN/rorzOx0MzsbuAu43P88WeCvm25mD5nZt8ABoJeVLVkyM3vS/32Xm9nppX6v4Jad4Fbjqf7PPf4xT7RSpRBmdpL/ebHX/3lS0LJ0M3vAzL71f5fPzaxd5f+DUpca03kHnAN8XUHMA/xj7vFjuMB/vaf/Woz//F9mlhm03ct2uByipZk9a971eJOZPWhmsf6ya/33/z/MbCfwhxAxFJ9zVsG1uBwjzGypme02s+fNLPhz5Twzm+/vZ7qZDSmKnVK5CSHOdX/d6/33wm4z+8zMegTt35nZzWa2ClgF4JzLAHYDJ1QQc91yzulfPf4D1gFjgp6nAA54CWgGNPFfvx5IAhKAR4H5Qdu8ADzoP04D8oE/AXHAuXgXt9blHD8duDHoGKuBXkBz4F3gZX/ZL4APgaZALHAs0MKPcR/Qz1+vE3B0pP+u+tcw/gE9/Pdvkv88FtgCnOA/HwscBRhwmr/uMf6yNCCjnP2eBcwBWvnbDgA6+cv+1z8vuvjHO8k/7/oC+4Ez/HPrTv98ife3WwfMB7oBTfztd/rnYIy/3U6gfYh44oH1wO3+vi8B8kqd1xn+437ARqCz/zwFOMp//AfglVL7Tgc2AEcDAX//wef9tf5nRtGxLwf2Am2Cfq/gz6jiY3D48yoQtPxaYJr/uA3eRe1q/9hX+s/bBsX2g/+3beI/nxDp911j/0cjOe/87bcDI4KeB59rcf6x7sI7R0cDWRy+3m0AjvUfrwDWAAOClg33H7+HVxrRDOgAzAJ+Uer8u8U/R5qEiDH4nAt5LS7nd1sHLPb/Nm2Abzn8mTIcyASO9/dzjb9+QtC2oXKT4HP9Qv/vM8CP/R5getByB0z2j90k6PUPgF9H6v2tFt3o8Qfn3H7n305xzj3nnMtyXo3LH4Ch5re2hpAH/Mk5l+ec+xjIxrs4VuYq4BHn3BrnXDbwe+AK82455AFtgd7OuQLn3Bzn3D5/u0JgkJk1cc5tcc4tqekvLRLMObcemAtc5L80GjjgnPvOXz7JOfeD83wNfA6cWoVd5+F9cewPmHNumXNui986cz1wq3Nuk/9en+6fd5cDk5xzk51zecDf8C6sJwXt93Hn3Eb/vP0p3i3Rj51zhc65ycBsvAtwaSfgXVQf9c/bt4Hvy4m9AC8BGGhmcc65dc65Hyr5fV9wzi1xzuX7sZeWGXTsN/Au2mMr2WdVjAVWOede9o/9b2A5cH7QOs8751b6f7M3gWFhOK7UQiM678BLurPKWXYCXqPPBOdcrnPuS+AjvC9s4LUEn2ZmHf3nb/vPe+I1BC3wW1vPBW7zr+mZwD+AK4KOs9k594R/jlRWQlHRtTiUJ/2/zS7goaDYxwFPO+dm+vt5Ecihei2tNwF/9v8f84GHgWHBrbr+8l2lfq8svL97RCjRjR4bix6YWayZTTCzH8xsH943LYDybvHt9N90RQ7gnayV6YzXqlRkPd63tGTgZeAz4HUz22xmf/EvsvvxPohuAraY2SQz61+FY4lU1Wsc/nD+if8cADM7x8y+M7NdZrYH74JS6a1v/4L1JF4rUqaZTTSzFv62iXitjKWVOD+cc4V452mXoHU2Bj3uAVzq3xbc48d3Ct5dj1D73uT85g7f+hDr4ZxbDdyG94U308xeN7POFf2+peIKJdSxK9tnVZT+TCnad/DfbGvQ46p+VkndawznHXh3GJLKWdYZ2Ogfs0jw+/drvBbgkXi39tPxWrhPA77xt+uB9yV2S1A8T+O17IaKvzIhr8UVrB+87+Dzugfwm1J/p25U77zvATwWtP0uvJb68v5viiQBe6pxnLBSolv/XBVe/wneLYIxQEu8WwjgvaHCaTPeG7dId7xbKtv8lp4/OucG4n2TPg+vpyrOuc+cc2fgfZAsB/4V5rikcXsLSDOzrngtTK+BN1QN8A5eC0+yc64V8DFVPC+cc487544FBuLdHv0dsAM4hHdbtrQS54eZGd6FYVPwboMeb8Qr/WkV9K+Zc25CiH1vAbr4+yzSvYLYX3POneLH44D/CXH8EpuUty9fqGNv9h/vx7tNWqRj0OPK9lv6M6Vo35tCrCvRpTGcdwAL/ThC2Qx0s5IdOIPfv1/jtWSn+Y+nASfjJbpFdb8b8VpK2wXF08I5d3TQPis7jw6vWMG1uBzdSsVedF5vBB4q9Xdq6t91CRVTqBg34pVgBO+jiXNueiXbDQAWVBBznVKiW/+24dXEViQJ70TZiXfBebiOYvk3cLtfZN/cP84bzrl8MxtlZoP9Avp9eLdPCs0s2cwuNLNmfozZeKUMImHhnNuO11LyPLDWObfMXxSPdwt/O5BvZucAVRrWysxGmNnxfkvIfryLbKHfAvMc8IiZdfbvppzoX9zfBMaa1/ErDvgN3nt+ejmHeQU438zO8veTaF6nsq4h1p2B96Xy12YWZ2Y/Bo4rJ/Z+Zjbaj+kQcJDD59w2IMWqP7JCh6BjX4p3IfrYXzYfr4QpzsxS8eqHi2z3j13eZ9jHQF8z+4l5nYQux0twPqpmfFLPGsl5B9579LRyls3Eu8twp//+T8Mru3kdwDm3Cu/8+ynwtV9CsA24GD/Rdc5twSvt+LuZtTCv0/dRZlbeMStU3rW4gk1uNrOuZtYGuBt4w3/9X8BN/v+HmVkzMxtrZkWt26Vzk1Dn+lPA783vHG9ep7tLK4m/C17N7ndV+oXrgBLd+vdn4B6/6f+35azzEt4th03AUuruDfIc3m2RqcBavA+hW/xlHfHqj/YBy/BO4pfx3jN34H1L3IX3gfHLOopPGq/X8O5oFN8+dc5lAb/GuxDuxrvz8UEV99cC74N+N965tRP4q7/st8AivBrZXXitpTHOuRV4F7Qn8FqgzgfOd87lhjqAc24j3p2Yu/AuEhvxWq/KfM76+/gxXseUXXjlQO+WE3sCMMGPYStekvp7f9lb/s+dZja34j9BCTOBPv4+HwIucc7t9Jfdi9fSthv4IyX/Dw7463/rf4aVqO/z93EeXnKyE68j0XnOuR3ViE0ip0Gfd76XgHPNH+Gh1L5y/eOd4x/7n8DPnHPLg1b7Gq9ccGPQc8OrcS7yM7wvCEv93/1tyi+lqEx51+LyvIaXaK/BKw150P/dZgM/xysl2Y3XqezaoO1K5CahznXn3Ht4/0+vm1dWuRjvb1WRnwAv+vXXEWEly7REREREGi4zexjIdM49GulYGjK/hX4BMNLvlBeZOJToioiIiEhDpNIFEREREWmQlOiKiIiISIOkRFdEREREGqRAXey0Xbt2LiUlpcJ19u/fT7Nmzeri8DUSbfFA9MUUbfFA9MVUXjxz5szZ4ZxrH4GQKlWV81WkMTnSz9do+1yE6Isp2uKB6Isp2uKB0DFVdr7WSaKbkpLC7NmzK1wnPT2dtLS0ujh8jURbPBB9MUVbPBB9MZUXj5mFnPUqGlTlfBVpTI708zXaPhch+mKKtngg+mKKtnggdEyVna8qXRARERGRBkmJroiIiIg0SEp0RURERKRBUqIrIiIiIg2SEl0RERERaZCU6IqIiIhIg6REV+QIZGbPmVmmmS0Oeu2vZrbczBaa2Xtm1iqSMYqIiERavSe6m/Yc5JHPV7Btf2F9H1qkIXkBOLvUa5OBQc65IcBK4Pf1HZSIRNZXyzP58IfcSIchEjXqPdHduvcQj3+5mswDSnRFaso5NxXYVeq1z51z+f7T74Cu9R6YiETU1FXb+XhtXqTDEIkadTIzWkVizPvp6vvAIo3L9cAb5S00s3HAOIDu3bvXV0xSD1LGT6p0nXUTxtZDJBIJCYFY8goiHYVI9Kj3Ft0Y8zLdQmW6InXCzO4G8oFXy1vHOTfROZfqnEtt377cKcJF5AgTH4gh34FzusiKQERadL1EV6egSPiZ2bXAecDpTlc6kUYnIeC1X+UWFJIQiI1wNCKRV++JrhWVLugSLBJWZnY2cCdwmnPuQKTjEZH6Fx/rJ7r5SnRFIIKlC8pzRWrOzP4NzAD6mVmGmd0APAkkAZPNbL6ZPRXRIEWk3iXEeZf1nHx1+BaBSJQu+Km1anRFas45d2WIl5+t90BEJKoEt+iKiFp0RUREGoz4gBJdkWBVSnTNbJ2ZLfJvh86u1QFVoysiIo1MObMZDjOz74qurWZ2XG2PEx/UGU1EqteiO8o5N8w5l1qbA1pRi64SXRERaTxeoOxshn8B/uicGwbc5z+vlaIOaDl5SnRFIJLj6Nb3gUVERCIk1GyGeFV8LfzHLYHNtT3O4RZdzRohAlXvjOaAz83MAU875yaWXiF4pqXk5GTS09ND7qho6t9Dhw6Vu04kZGdnR1U8EH0xRVs8EH0xRVs8IhLVbgM+M7O/4TU8nRRqpapeXwGW7fQS3Fmz55G1NnqGF4u2z8ZoiweiL6ZoiwdqFlNVE91TnHObzKwD3tBFy/1vp8X85HciQGpqqktLSwu5o427DsDUr4iLT6C8dSIhPT09quKB6Isp2uKB6Isp2uIRkaj2S+B259w7ZnYZ3sgpY0qvVNXrK0CLDbvh++n0HzSYtH4d6ibqGoi2z8ZoiweiL6ZoiwdqFlOVShecc5v8n5nAe0CNC+ZjYjTqgoiICHAN8K7/+C1qcW0touHFREqqNNE1s2ZmllT0GDgTWFzxVhUcUKMuiIiIgFeTe5r/eDSwqrY7TNDwYiIlVKV0IRl4zx8tIQC85pz7tKYH1Di6IiLS2PizGaYB7cwsA7gf+DnwmJkFgEP4dbi1oXF0RUqqNNF1zq0BhobrgH6eq5nRRESk0ShnNkOAY8N5nOLhxZToigCaGU1ERKTBONyiq+HFRCCSia4yXRERkbDSzGgiJUUg0fV+KtEVEREJL426IFJSvSe6ppnRRERE6kRcrGGoRlekiFp0RUREGggzIxCjFl2RIuqMJiIi0oAEYtSiK1IkAqUL3k+nJl0REZGwi4sxJboivvpPdFGLroiISF2JU+mCSLHItejW94FFREQagUCMhhcTKVLviW4xZboiIiJh57XoasIIEVCLroiISIOiGl2Rw1SjKyIi0oBoeDGRwyLWoisiIiLhp0RX5LDI1eiKiIhI2MXFmDqjifgiULrg0TC6IiIi4RcXCzl5SnRFICKlC6pdEKktM3vOzDLNbHHQa23MbLKZrfJ/to5kjCISGQHT8GIiRSLXolvfBxZpWF4Azi712njgC+dcH+AL/7mINDKBGFONrogvglMA1/eRRRoO59xUYFeply8EXvQfvwj8qF6DEpGoEBeLhhcT8QXq+4AqXRCpM8nOuS3+461Acnkrmtk4YBxA9+7d6yE0T8r4SWHZz7oJY8OyH5GGKC4GcjRhhAigURdEGiTnnKOCCiHn3ETnXKpzLrV9+/b1GJmI1DWVLogcFrFEV5ULImG3zcw6Afg/MyMcj4hEQCDG64zmVCMoEplE10yJrkgd+AC4xn98DfB+BGMRkQiJi/H6weQV6EorEplEF5TpitSCmf0bmAH0M7MMM7sBmACcYWargDH+cxGJAqGGBPRfv8XMlpvZEjP7SziOFRfj9YXREGMiEeiMBl6HNOW5IjXnnLuynEWn12sgIlJVLwBPAi8VvWBmo/BGSxnqnMsxsw7hOFDAb8LKzS+EhHDsUeTIFbkWXRERkUainCEBfwlMcM7l+OuEpa4+LjjRFWnkItSiq3F0RUSk0esLnGpmDwGHgN86574vvVLwcIDJycmkp6dXuNOC3BzAmPrtdDo0jY7BlbKzsyuNuz5FWzwQfTFFWzxQs5gik+iqTVdERCQAtAFOAEYAb5pZL1dquATn3ERgIkBqaqpLS0urcKczt0wBchh+7Aj6JCfVRdzVlp6eTmVx16doiweiL6ZoiwdqFpOGFxMREYmMDOBd55kFFALtarvTohpdzY4mEqlEVw26IiIi/wFGAZhZXyAe2FHbnRbX6GrUBZFIlS6oRVdERBoPf0jANKCdmWUA9wPPAc/5Q47lAteULluoiaLhxXLylOiKqDOaiIhIHatgSMCfhvtYAbXoihSL0PBiatMVERGpCxpeTOQwTQEsIiLSgBTPjKZEV0QTRoiIiDQkh0ddKIhsICJRIHIjSatJV0REJOwCKl0QKRahzmimPFdE6lTK+EmVrrNuwth6iESkfhWXLqgzmkjVW3TNLNbM5pnZR7U9qLqiiYiI1I24WO+nhhcTqV7pwq3AsrAcVUW6IiIidULDi4kcVqVE18y6AmOBZ8JxUEPj6IqIiNSFgN+YpCmARapeo/socCeQVN4KZjYOGAeQnJxMenp6uTsrKMgnN89VuE59y87Ojqp4IPpiirZ4IPpiirZ4RKTxMTPiAzHqjCZCFRJdMzsPyHTOzTGztPLWc85NBCYCpKamurS0clclburnxMU5KlqnvqWnp0dVPBB9MUVbPBB9MUVbPCLSOCXExmh4MRGqVrpwMnCBma0DXgdGm9krdRqViIiI1JhadEU8lSa6zrnfO+e6OudSgCuAL51ztZqbW6MuiIiI1B0luiKeCE0BrDmARURE6kpCIEajLohQzQkjnHPpQHptD6oWXRERkboTH4jROLoiRKxFV4muiIhIXYlXi64IEKFEF5TpioiI1JX4WNXoikAEW3RFRESkbiQEYjW8mAgRa9FVg66IiEhd0agLIp7ItOiiRFdERKSuxAdiNAWwCCpdEGlwzOx2M1tiZovN7N9mlhjpmESkfml4MRFPhFp0DacmXZGwM7MuwK+BVOfcICAWb6IXEWlENLyYiEctuiINTwBoYmYBoCmwOcLxiEg9U4uuiKdaE0aEi2p0ReqGc26Tmf0N2AAcBD53zn1eej0zGweMA+jevXtYjp0yflJY9hNtxxI5Eml4MRFP5KYAFpGwM7PWwIVAT6Az0MzMflp6PefcROdcqnMutX379vUdpkijY2bPmVmmmS0Osew3ZubMrF24jpcQF6tEV4RIDi+mJl2RujAGWOuc2+6cywPeBU6KcEwiAi8AZ5d+0cy6AWfi3YUJm/jYGI2jK0IEE10RqRMbgBPMrKl5t05OB5ZFOCaRRs85NxXYFWLRP4A7CXNFX3wghkIH+arTlUYuMjW6mgFYpE4452aa2dvAXCAfmAdMjGxUIhKKmV0IbHLOLaiopC+4pj45OZn09PQK95udnU1G5loAvvjqaxICkS8XzM7OrjTu+hRt8UD0xRRt8UDNYopgoqtUV6QuOOfuB+6PdBwiUj4zawrchVe2UCHn3ET8L6ypqakuLS2twvXT09MZ0LoHrFzK8SedTKum8WGIuHbS09OpLO76FG3xQPTFFG3xQM1iitg4uspzRUSkETsKr9PoAjNbB3QF5ppZx3DsPD7gXd41O5o0dipdEBERqWfOuUVAh6LnfrKb6pzbEY79x8d6ia5GXpDGLkItuiIiIo2Hmf0bmAH0M7MMM7uhLo+XEBcLqEVXJCItuiIiIo2Jc+7KSpanhPN4RS26GmJMGruITRihcXRFRETqRkJApQsiEMHSBeW5IiIidSNeia4IEKkJI1SkKyIiUmeKW3Q1YYQ0cmrRFRERaWCKhxfLU6IrjZtqdEVERBqYeLXoigAaXkxERKTB0Ti6Ip7I1OiKiIhInSkaR1eJrjR2mhlNRKQCKeMnVbh83YSx9XKccB5LGj6NoyviiVDpgooXRERE6kpxZzS16EojF6HOaKgzmoiISB3R8GIinojV6CrPFRERqRvFpQsaXkwauYgNLyYiIiJ1IybGiIs1tehKo6fhxURERBqg+NgYjbogjV7kShdUuyAiIlJnEuJilehKoxe5zmiROLCIiEgjER8bo+HFpNGLWKIrIiIidSc+oNIFkYiNo6sWXRERkboTH4hRZzRp9CpNdM0s0cxmmdkCM1tiZn+s7UHNUO2CiIhIHUpQi65IlaYAzgFGO+eyzSwOmGZmnzjnvqvpQZ2D3AJluiIiInUlPhCjmdGk0as00XXOOSDbfxrn/6tVlrpo014ACgsdMTEq2BUREQk3rzOaEl1p3KpUo2tmsWY2H8gEJjvnZobj4GrTFRERqRsaXkykaqULOOcKgGFm1gp4z8wGOecWB69jZuOAcQDJycmkp6dXut/09HRio6RFNzs7u0ox16doiyna4oHoiyka4vHP02eAQXjfJ693zs2IaFAiUu80YYRIFRPdIs65PWb2FXA2sLjUsonARIDU1FSXlpZW/o4+nQTAyNNOIy42YnNWlJCenk6FMUdAtMUUbfFA9MUUJfE8BnzqnLvEzOKBppEOSKSxM7PngPOATOfcIP+1vwLnA7nAD8B1zrk94TpmQkDj6IpUZdSF9n4LEWbWBDgDWB6Og785e2M4diMiPjNrCYwEngVwzuWG88IpIjX2Al4jUbDJwCDn3BBgJfD7cB5Qw4uJVK1FtxPwopnF4iXGbzrnPgrHwe9+bzFXHd8jHLsSEU9PYDvwvJkNBeYAtzrn9gevFFxq1L1793oPUmomZfykSIcgNeScm2pmKaVe+zzo6XfAJeE8poYXE6naqAsLgeH1EIuI1F4AOAa4xTk308weA8YD9wavVLrUqN6jFJHSrgfeCLWgun1givoKbN+WQ/bB/Ij3GwiOKVpEWzwQfTFFWzxQs5iqVaMrIlEvA8gIGhnlbbxEV0SilJndDeQDr4ZaXq0+MBzuKzAteykztm6Ihn4D0dJ/oVi0xQPRF1O0xQM1iyniPcF2ZudEOgSRBsM5txXYaGb9/JdOB5ZGMCQRqYCZXYvXSe0qf9z6sIlX6YJI5BPdBRnqJyMSZrcAr5rZQmAY8HCE4xGREMzsbOBO4ALn3IFw7z8hEEt+oaOgUNVJ0nhFPNG9480FkQ5BpEFxzs13zqU654Y4537knNsd6ZhEGjsz+zcwA+hnZhlmdgPwJJAETDaz+Wb2VDiPGR/wLvFq1ZXGLCI1uhOvPpZxL88BYN/BvEiEICIiUm+cc1eGePnZujxmcKLbJD62Lg8lErUi0qIbfBdFd1RERETCL8FPdHMKNGmENF4RSXRLf7P8wwdLIhGGiIhIg6XSBZEIJboj+7Qr8fyF6evI0+wtIiIiYVPcoqtEVxqxiCS6ZlbmtbGPfxOBSERERBqm+Fi16IpEfNSFIiu3ZXMwV3VEIiIi4ZAQp0RXJGKJboemZVt1B9z3KXsO5EYgGhERkYYlPtbrD5Or0kBpxCKW6I4bnBDy9fveV8c0ERGR2irqjJaTp0RXGq+IJbq9W4ce0++AyhdERERqrXjUBQ0vJo1Y1NToFgnzVN8iIiKNUoKGFxOJvkT3i+WZfLt6B4fy9A1URESkpuI1vJhIZBPd1B6tQ75+1TMzSftrev0GIyIi0oAUDS+mRFcas0BEDx5bduSFIlv3HarHSESkIinjJ0U6BBGpJg0vJhLhFt2EQOgOaUUO5ObXUyQiIiINS0LR8GJKdKURi2ii+9dLh1S4fPrqnfzqtbm8PGNdvcQjIiLSUKhGVyTCiW6HpETm3DOm3OU3vjSbjxZu4d73l7Bx14F6jExEROTIFq9RF0QiP+pC2+YJPHtNaqXrTZy6hhEPTdE0wSIiIlUQG2MEYkzj6EqjFvFEF+C0vu0rXefl79azPSuHdTv310NEIiIiR774QIxadKVRi4pENxAbQ+8OzSMdhoiISIMSH4jhkKYAlkYsKhJdgCl3nMa/flZ5CcM5j33Dxl0HyM0v5OxHpzJ15fZ6iE5EROTI07V1E37Ynh3pMEQiJmoSXYAxAzpUab03Z29k5bYslm/N4q73FtVxVCIiIkemYd1asTBjLwWFLtKhiEREVCW6ZuVPIBFs855DXDnxOwC2aWIJERGRkIZ3a012Tr5adaXRiqpEt6remZtBVo43mURegePmV+dq5iYREZFShnVvBcD8DXsiHIlIZByRiW5pkxZtiXQIIiIi5TKz58ws08wWB73Wxswmm9kq/2frcB+3Z9tmtEgMMG/j7nDvWuSIEHWJ7ls3nUi75vE12jb1wcls2nMwzBGJiIjU2gvA2aVeGw984ZzrA3zhPw+rmBhjaLdWzFOLrjRSUZfojkhpw+x7zqjRtjuyc7nltbnk5GtwbBERiR7OuanArlIvXwi86D9+EfhRXRx7ePfWrNyWxX6/5E+kMYm6RLe25m7YQ797Pi1+Pmf9LqWnrEYAACAASURBVFZszYpgRCL1z8xizWyemX0U6VhEpFzJzrmi2rutQHJdHGR4t1YUOli0aW9d7F4kqgUiHUB5HrtiGLe+Pr/G20+c+gPH9mjDxf83A4B1E8aGKzSRI8GtwDKgRaQDEZHKOeecmYUcA8zMxgHjAJKTk0lPT69wX9nZ2SXWyc71dvtO+hwObahZaWBtlY4p0qItHoi+mKItHqhZTFGb6F44rAvnDu5En7s/qdH2D3+8PMwRiRwZzKwrMBZ4CLgjwuGISPm2mVkn59wWM+sEZIZayTk3EZgIkJqa6tLS0ircaXp6OqXX+duCr9gXl0RaWuUTM9WFUDFFUrTFA9EXU7TFAzWLKWoTXYC42PBVVuzan8vmPQfp0bYpSYlxYduvSBR6FLgTSCpvheAWou7du9dTWA1TVYY2bMx3lPT3qdAHwDXABP/n+3V1oOHdWjFjzc662r1I1GpwNbrlOeaByZz3xDRueHE2Hy3czJn/+JrCUjPFfLt6B3kFmhNcjlxmdh6Q6ZybU9F6zrmJzrlU51xq+/bt6yk6kcbLzP4NzAD6mVmGmd2Al+CeYWargDH+8zoxrFsrtu3LYctejUwkjUtUt+gCdGnVJKxDhs1Zv5v5G/aQW1DIizPWcd3JPQFYvaeABz+dybiRvbjr3AFhO55IPTsZuMDMzgUSgRZm9opz7qcRjkukUXPOXVnOotPr4/jDuntD9M7bsIdOg5vUxyFFokKlLbpm1s3MvjKzpWa2xMxurY/Aivzn5pO5/uSevPdfJ4VlfwWFjvxCr9X2jx8uLR6RYV+O17q7Zns2Szbv5f35myrd1wcLNpMyfhJrtmfz2ZKt1YrjtZkbWLdjfzWjF6mYc+73zrmuzrkU4ArgSyW5IjKgUxLxsTHM36jxdKVxqUqLbj7wG+fcXDNLAuaY2WTn3NI6jg2A9kkJ3Hf+QADaNU9gR3ZOrfcZXLFQUFi6k6sx9vFpgNchrkh2Tj4Hcwton5RQ/Nqb328E4OL/m87uA3l8fvtI+iaXWxZ5+PiFjrveW0SbZvHMvbdmYwaLiIhUVUIglqO7tNBUwNLoVNqi65zb4pyb6z/OwhuyqEvFW9WNF68fwVlHh3eYwfiAsfdAHuv3ea28U5ZtK142a+0ubnxxNt+s2s5Z/5jKiIemlKjhdXhJ8u4DeYCXDFfHrv25tQ1fpFzOuXTn3HmRjkNEosOwbq1YtGkv+eqLIo1ItTqjmVkKMByYWRfBVObozi15+upUZt0dvpKmMY9MZeifPuf9H/LKLLvs6RlMWbaNq5+dVVwnfPKEL4uXu1KNwaWfV0dufng+eNJXZJIyfhKZ+w6FZX8iItIwDOvWioN5BSzevC/SoYjUmyp3RjOz5sA7wG3OuTJnSW0HtD5SZGblFMe9e3fJTnJz584la21spfsoDMqI09PTWbg9n0fm5HDvCYkc1erw9jX5G/1jjpfgvvrpNIZ3CG9fw2j8P4u2mKItHhGRIiP7tCcxLobXZq5nWLdWkQ5HpF5UKRMyszi8JPdV59y7odYJx4DW1fJp5WMz1pWiuJ9e+R3sOjwu4fDhw0lNaVPp9gWFDj77uHhfUz9cCqzFte1J2qm9itcr/TfakZ2Dc5SoEy7t5XXfw/ZMBg0aTNrA8JZ5NJTBo+tStMUjIlKkdbN4Ljm2K29+n8Fvz+pHh6TEKm+7YOMetu47xFlHd6zDCEXCryqjLhjwLLDMOfdI3YdUNZ/ceioXH9M1IsfOLyjkm1Xbywy+fclTM1hQhR6t780rOaJDjHk/Kyt9SH1wCiMemlLhOmaVHl5ERBqpG07pRV5hIS/PWF+t7f744RJ++9aCMuPPi0S7qtTongxcDYw2s/n+v3PrOK5KDejUgr9fNjQix+599ydc/eyskMv+8OESlmzeS8r4STz99Q9c9M9vOZRXUGKd3761oPjx/pz84uS0qHNbOLjaFAyLiEiD1LNdM8YMSOaV79ZzMLeg8g3w7ibO27iHrEP5rN91oI4jFAmvqoy6MM05Z865Ic65Yf6/j+sjuCPRkk37mPGD19L750+WM2/DHvrf+ynTV+8Iuf6tr88rfrw6M7v48TPfrOGeaQfILyjk40VbqpG4hm7SPZCbX2IoNecc972/mLkbdldxvyIi0hDceEpPdh/I4525GVVa/8vlmcV3HBdmaHgyObI0mCmAT+hVeW1sfcgtKCQ/xK2d575dF3L9KcsyifGbdN+cncEXy7ZxMLeAByctIyPbMe7lOfzXq3NDTkhx9qNTeeP7DQD8/fMVfLn88NBo416eUzwUWn5BIQPv+4z7P1hcvDy/0PHSjPX8+J/TNcyZiEgjclzPNgzp2pLnpq2tUinCF8u20bFFIolxMSzYuLceIhQJnwaT6L4+7sRIh1DsnTllvyVPWbat3JnQnp66pvjxDS/OZsB9nxY//3J5JgA7g5LRr1du55lv1rB8axb//c4i1u7YzxNfrub6F2aXqNEtKpnIK/A+yN6aHfrb+xdBYweLiEjDZmbccEpP1uzYX3yNKc+hvAKmrtzBmIEdOLpzS7XoyhHniE90X7r+OB740aBIh1HCqqAShGCPf7GKbTUc3za4cuGa52bx4KRlxc9H/S095DbPfLO2xPPyOqo54NvVOziQG3rCi4JCR2GhY+v+Qu56bxG/em0uKeMnsb+aE2TUl0N5BRoQXUSkAucO7kTnlon87fMVFSavM9bs5GBeAacPSGZI15Ys2bxPn69yRDniE92Rfdtz9Qk9AEjt0RqAp356TCRDKte78zZx/MNf1GjbVduyqr3NY1+sKvE8v8Dx1Qrv23tw4pyx6wBXPTOTi/53Op8u3soTQdut37mfo+76mIufms4/5+fw2swNfLRwCwA//qe3/jEPTCY7J5/bXp/Hhp2Vd1TYtOcgqzOr//sE27r3EFc/O5P9eWVvu/W/91OueiYic5qIiBwR4mJjuO/8gWzec5ALnvyWq575jm9X7yjTH2TK0m00jY/lxF5tGdrVm3Bi9fbQjTki0eiIT3SDPXfdCD781SmcPagTr954fKTDCasXqzgUTEWji+UXOq57/ntufm1uidf3HfJaZldsy+KmV+bw98kr+XrldgBO+2s6APM27CnTIrxiWxYPTlrKrv25vD9/E/+Zv5m73ltUaYwnT/iSMY9MrdLvU55HJq/gm1U7mLE5dKvyzLW7arV/EZGG7uxBnfh2/GjGn9OflduyueqZmdz/wZLi5c45vlyeyal92pEYF8vgri0BWKg6XTmChHfqrAhrkRhXfCKe3Lsds+46nS+WZ/L7dytPvhqKz5eWrbctPWzZpIVbeCRoaLY1IWqHr3luFusmjK30eFbFMYBDKSh0ZOw+QI+2zXh3bgafLdnK01enVmnbN8upNy7tpRnreOHbdXz527TqBygSJinjIzfBjUhFkhLjuOm0o7j2pBQmfLKcF6avY0CnFlx5XHeWbN7Hlr2HuOOMvgD0bNuMpIQACzL2cNmIbhGOXKRqGlSiW1qHFok1SsAakvIusNNXH57sYqrfelvZtjn5Zf+Y5rchr9/pJcvVmbDi0SkreeLL1Xz9uzTueNMbW3h/Tj4bdh1gQKcWIbfJKygktooH2ZGdw33vL6l8RRGRRi4xLpZ7zxvID9uzue/9xfRNTuKbVdsxg1H9OwAQE2MM7tqShRlq0ZUjR4MqXQilR9umJZ6veugcnrhyeISiiR7XvfB9tbfZeiBEouvnnP/yO759s2oH5z3xDSurUFM83R9vODMrp/i1cS/P5pzHvikeGm32ul3MWe+N9Zu+IpM+d3/CT589XH9bOqLUBycXP774/6ZX/kuJiAgAsTHGE1cOp3OrJvzylTl8sGAzx3RvTbvmh6edH9y1Jcu37iMnv2qTTYhEWoNPdE/u3Y6PbjmFO87oyzM/SyUuNobzh3bm+WtHRDq0BmF9iM5nizft44InpxU/X52ZRXZOPje++D3vBg1QvtGfYefpr38ofm3mGq+2tqgl/pKnZhQnrNc+7yXnRQkywKvLcvk8aIzhHdmHh2ELFVtlnHNMD9EhoyjezKyajZohInIkaNU0nolXp5Kdk8+a7fs5fUCHEsuHdm1FXoFj+ZbadSgWqS8NPtEFGNSlJb8+vQ9jBiYXvzaqfwdm3X16BKNq2A7lFbLnQC4vf7eeMY9MZdD9nzFlWWZxiQIcbsmdsuzwOI6FfoLZ955P+OOHh8sOKpqqctzLc0gZP6nEzG8VWb9zPynjJ/H9urId1t6ancFPnpnJ+/M3l1l26l++4riHajZqhojIkaJfxyQeuWwobZrFM3ZwpxLLhhR1SNN4unKEaNA1upXpkJTIZaldq9yxSapn2J8mV75SKcG56vNBs8kVTX5RkarcStu271DxSBLvzt3EiJQ2zFyzk/hADIO6tGSdX2ucsbtka/CKrYdbL5xz5BU44gON4nuiiDRCZw/qxFlHd8RK9Yno0qoJbZvFsyBjL1dHKDaR6mjUiS7AsG6tlegeAWrbpzA3v5Db3pjHx4sOlzmYwYHcfC6f+B3gjcm8fMs+AP72+UpO6NWW1BRvaumzHj08HFrP338MwF8uHqKexyLSYJVOcoteG9y1JYvUIU2OEGqSkiPCMQ9U3jqcXcFMbX3v+aREkgvw3txNDLzvs+LnU1duL9Ex7sp/fVfh8e58Z2GlMYmINDRDurZiVWZWubNpLt60l9khSsNEIqHRJ7rljVTVLzmpfgORWqtu/ezBSsoh8gocr8/awPTVO8pdJ1Sdr4hIQza0a0sKndfxOFjG7gPc+vo8zntiGpc8NYN7/rOo3GS4Juas38VbszeyalsWhVXskyHS6EsXigxtH8tNZw7ll6/OpUfbprz28+O5+P+ms64GPfel4RhfyWQjlz41g7dvOrG4xEFEpLrM7HbgRrwqrUXAdc65qB3ipWhipnv+s4jBXVrRrU0Tsg7l8/J36zHg5lFHcSivkOe+Xcu0VTv4e9AERTW1OjObq5+dxQG/Y3JSQoBh3VvxpwsH0bNds1rvXxouJbq+lgnGOYM7sfRPZxEbYyQEYvns9pGMeHBK8RS5IqEElzuIiFSHmXUBfg0MdM4dNLM3gSuAFyIaWAU6JCXyq1G9mbl2J9+u3sG2rEM4BxcN78Jvz+pHl1ZNADhjYDK/eXMBlz41g18NSyCthsc7mFvAza/OpUlcLC9dfxxrd+xn/sY9vD9/Mw9+tJRnNVyoVKDRJ7rH9mjt/UyOBaBp/OE/SUIglll3j+G8J6axOjObX4zsxdNT10QkTole1ZgMTkQklADQxMzygKZA2fENo8xvz+pX/Dgnv4ADOQW0bhZfYp0TerXl09tO5exHv+GbTbncEWI/L3y7lr0H87l1TJ9yj3X/B4tZmZnFi9cdR2pKG1JT2nBpajc6tkjk75NXsihjb3Ers0hpjT7R7ZucxLoJY0lPTw+5PDEulil3nFb8XImulLZw017OKTXWpIhIVTjnNpnZ34ANwEHgc+fc58HrmNk4YBxAcnJyuderItnZ2ZWuU5/6JeUxbVM+k7/8iriYw00DBYWOv311gOw8aHVgAz1axJbZ9ttNeby5KJfzj4qjcPMS0oO+AhxV6GgagPvenMGtxyRWK6by/ka5BY5pm/I5sXOAJoGyzRjZuY7EAARiwt/EEW3/b9EWD9Qspkaf6NbGeUM68dHCLQCcM6gjnyzeWskW0hBl7lPpgojUjJm1Bi4EegJ7gLfM7KfOuVeK1nHOTQQmAqSmprq0tLQK95menk5l69SnguRtfPnibJp0G8wpfdoVv/7dmp1k53mj20zZ3pyXLzi+xHartmXxyhffcnzPNvzj+uMJxJbtP7/KVvGPKStp33c4R3eueqtueX+jO99ewJtLM2jXJYVfp5VsZc4rKCTtr+l0bJnI6+NOIC5EPLURbf9v0RYP1CymRj/qQk3NvfcM/nH5sOLnnVp6NUn3jB0QqZAkQvIKCiMdQjEz62ZmX5nZUjNbYma3RjomEanQGGCtc267cy4PeBc4KcIxhdWJR7UlYJC+IrPE658v2UZ8IIbbx/Tlm1U7mLbq8Ag3ew/mMe7lOTRLiOXxK4eHTHIBrj05haTEAI9/sarWcb4+awNvzs6gaXwsb87eWGZkh8lLt7Fpz0HmrN/NY1NqfzypH0p0q+n7u8cw667TadMsvsS3ufxCL9kJvp3xz6uOqff4pP7VxS2sWsgHfuOcGwicANxsZgMjHJOIlG8DcIKZNTVvhobTgWURjimsmsYH6N8mlq+CEl3nHJ8t2cqpvdtxU1ovurRqwp8/WUZhoaOg0HHb6/PYuOsA/7zqWJJblF+W0LJJHNed3JPPlmxjmT/hz4HcfL5Yto3pq3eEnBp+9/5cNmUX4tzhZYsy9nLfB0s4pXc7HrpoEBm7DzL9h50ltntt5gY6t0zkkmO78r/pqyscerK63pmTwaas6Gk0aUhUulBN7ZMSQr5+4bAuvDRjPaf16wAfLgXg3MGdeP66EbRvnsB5T0yrdN/N4mPZn1v5NLYSZaIoz3XObQG2+I+zzGwZ0AVYGtHARCQk59xMM3sbmIv3RXUefplCQzK4fSz/Xr6fjbsO0K1NU5Zu2cemPQf59em9SQjE8tuz+nL7Gwv4cOFmVm3L5qsV23ngR4M4rmflQzfecHJPnp+2lrveW0TrpvF8u3oHOfle0tiueQJjB3dk9IBkVmzdx5Slmcxev4tCBy+umsplqd0Y3b8Dv3x1Du2axfPYFcNolhDgDx8s5Y3ZG4tLLdbt2M+01Tu4fUxffj6yJ3M37Oa2N+bz6W0jaVOqE1517czO4bdvL6BPqxiuOr/idddsz+bRKau4YkQ3Turdrszyndk57M8poHvbprWKqSFRohsmx/ZozboJYwF44srhtGwSB8Cofh3KrPvr0b15/MvVJV579ppUTuvbnt53f1L3wUpYxZQ360iEmVkKMByYGWJZceeW7t2712tc0nCkjJ9Ub/sp+nxtiJxz9wP3RzqOujS0fSz/Xu6VL1x9YgqfLdlGjMGYAckAXDi0CxOnruX+D5aw50Ael6d246fHV+2zqWXTOG44tSePTllF9zZN+cnx3Tm9fzJZh/L4cOFmXv9+Iy/OWA/AgE4tuHlUb3Zv2cCi7AAPTlrGg5OWERdrvPmLE2nb3GvMumh4F16buYHd+3Np3Syef3+/gdgY4/IR3WgaH+CJK4dz0f9O53dvLeCZa1JDTpdcVV+v3I5zsHJ3IfM37mFYt1Zl1skrKORf36zh0SmryM0vZMvegyET3V++Opc563fz81N7cduYPiTGle3g19go0a2l/7l4cJk30vlDO1e4zS2n9+H/vv6Bhy8azO/e9qaRPd0/2Yv07tCc1ZnZJV77+6VD+c1bC8IQtYRTNKa5ZtYceAe4zTm3r/Ty0p1b6jk8EWlkkpsaPdo25asV27n6xBQ+X7KV1B5tihPLmBhj/Dn9uea5WQzv3oo//ejoaiWPt4zuw6Wp3ejcMrHEducM7kR2Tj7fr9tF3+Sk4jF+09O38GDayazYmsW7czMY2q0Vw7u3Lt7ustRuvDB9Hf+Zv4mrju/B27MzGN2/Ax1bemUUR3duyfhz+vOnj5byxbJMxgwseQ0vbdOeg/zl0+Xce95A2jUveWf4y+WZtGsez/5DuTzzzRqe/EnJsseV27K4/Y35LNm8j7OP7kiX1k14dtpa1u3YT0rQZBmrtmUxa+0u+iUn8dTXP/DZkq38+ceDOaFX2yr/HRsi1ejW0uUjunPhsC6VrjfljpEAtEgMEBcbw6qHzuXS1G6VbndZatfixxcf27XcKYslcqLt/8TM4vCS3Fedc+9GOh4RETMjrW97pv+wg5Xbsli+NYszjy6ZHI7s047nrx3Bc9eMICFQvZbI2BijS6smIZPj5gkBRvXrUJzkBuvXMYnfnzuAc0sNETmwcwsGd2nJG99v5POlW9m5P5eflGphvvrEHrRqGsdHCyse9riw0HHHG/N5f/5m3vh+Y4ll+QWFTF25ndH9O3Ba1zg+WbyVjN2HZ2TdcyCXa5+bxbZ9h3jqp8fw1NXH8vNTexFj8M7cjBL7ev37jQRijFduPJ5XbzyegkLHFRO/47WZGyqMr6FToltPOvjF9J1DnGihPH7FcEZ0jOXhiwaXeL2odr5jUHH+mAEVf5OUumVR1Kbrd2Z5FljmnHsk0vGIiBRJ69+BQ3mFPPCR12XgrKM7llhuZozq36HMxBORcvmIbizfmsWET5bTpVUTRvZpX2J5XGwMZw3syJRlmRzKK79/zfPT1zFz7S5aNY3j7TkZJTrBzd2wh32H8hnVrwNn9AhgwAvfrgO8Dnvj31lEZlYOz107grMHecl4x5aJnNqnPe/MySgeGSInv4B352Zw5tHJtE9K4OTe7fjstpGc0rsdD05aWiJ5bmyU6NaTFolxPH7lcF664bgqrT+wcwtuHpZIIDaGB340iKv8b5JvjDuBy1O7MeP3ozm+Zxv6JSeR4hed/2pU70r3e2qfsjU9Ujsx0XUWnQxcDYw2s/n+v3MjHZSIyIm92pIQiOGbVTsY0KkF3dpEd4epC4Z1JjEuhozdB7nyuG7Ehhhh59whXmnEN6tCj8CwOjOL//l0OWMGdOCucwewdsd+5m7YXbz8y+WZBGKMk/u0o22TGMYO6cTr329k36E8Xp25gU+XbOXOs/sxpGvJut1Lju3K5r2HmLHGGxnisyXb2H0gjytGHG51bhIfy4SLvcayu95bXCLBbkyi6xLdwF0wtDMdkqo3ewvA1Sf04CG/Zff4Xm35n0uGYGa88YsT+ez2kcVDr/Rq34z+HZMq3NeTVx7D/PvOKH5+ZiV1RVK5aOqM5pyb5pwz59wQ59ww/9/HkY5LRCQxLpYTj/LqRc86OvqvPS0S4zh3cCcCMcZl5ZQannRUW1o1jePjRVvKLMsrKOSONxfQLD6Wh388mLGDO9E0Ppa35xwuOfhqeSYjUtrQItHrwH7jKb3Izsnn4UnLeOCjpYzs254bT+lVZt9nDEwmKTFQvK/XZ22ga+smnFKqg1rX1k3577P7M3Xldt6du6nGf4sjmRLdBuD6U3ry5E+Gc9HwLrzzy5OY+rtRPH/tCGbdfTr/V2os35ZN40o8n/iz1DL7a11qHalYfECnkYhIVRR1vD57UMdK1owO9503kHd+eVJx+WFpcbExnDkwmclLt5UpX/jfr1azMGMvD180mA5JiTRLCHDOoE58tGALB3ML2LTnICu2ZTGq/+GSiMFdW3J8zza8/v1GkhID/P3SocSEaElOjIvlgqGd+WTxFhZv2sv0H3ZyeWq3kOtefUIPju3RmgcmLWV7VuObyVNX6AiLi7ValxPExhjnDemMmdEsIUD3tk0Z1b8DHZISOWdwp5C3W8rz2BXDmHrnKNZNGMsFlYweIR4luiIiVXPliG58dMsp9O/YItKhVEmrpvEMDTHcV7BzB5ctX5i/cQ9PfLmaHw3rzDlBHd0uObYrWTn5fL50K18t9ybQGN2/5DCkt4zu4yW5lw0rd+x+gEtTu3Eor5CbX5tLjFFuB/eYGON/Lh7MgZwC/vDhkkp/59o6lFdAbn70TH6hK3SErXroXF6+4fD83q3qoDU1ttSt9aJbJKFcOKwLSf7ygqB6nuQW5Z9sANeelMKsu07n2pNSQi6/edRRPOO3Hg/sdGR8wFXVpcdWPnqGiIhAIDaGQV1aRjqMsDq5dztaNjlcvrA/J5/bXp9HxxaJ/PHCQSXWPb5nG7q1acJbszNIX5FJ19ZNOKp98xLrnNKnHfPuPYPT+pbs/Fba0K4t6d2hOet3Higx9FkovTskccvo3kxauIXv1uwsd71wuPJf30XVUKhKdKPMzLtOZ/kDZ4d1n+/+V8lp00Pd2gCYePWxJZ5fdZxX1N6ueQIz7xpDgj/ay3+f3T/k9h1aJPKHC47mouFlh1vr37EFYwYms27CWD6+9VTevunEMuucc4Tcyiqte5R3qBARkboTFxvDWUcnM8UvX3hw0lLW7zrA3y8bWjx5VJGYGOPiY7ry7Q87mLpqB6P7dwg5JFogtvL0zMy45FhvCNLgTmjl+fnIXnRISuCRySvrrGPaks17mbdhD9NWbY+azm9KdKNMQiA27DOZhPr2PPV3o/h2/GgAJvx4MB/+6hTOLDXUS1E9b9Gtk6K37M9O7MG6CWN58ifDQx7v5BCztRSWesP37lDyG+y/fpbKE1eG3l+RQIxF3exIn902UqULIiKN3LmDO5GVk88fPljCv2dt5KbTjip3ooaLj+mKc5CbXxhy9tTquPakFJ64cjinD6h8P4lxsfxX2lHMWruLGT/UTavuW7O9znG7D+Sxdsf+OjlGdekK3Uj886pjeOqnh1tsu7dtWjx49hXHdWdw17LJcNHyoqHNBrbxEvBArPft87whnbnvvIFltiv6hlmRVk3jSyStZwxMrvQbbLi+G/7itLI9WGvqqPbNKl9JREQatKLyhde/38igLi24fUzfctft1qYpJ/RqQ0IgpngUippKjIvl/KGdqzyL3BXHdadji8RKW3U/XrSFO9IPsPdgXpVjyc0v5P35m4pHf5q7YU+l26SvyGRhRuXr1YYS3Ubi3MGdqt3LtSgZ/ekJPQD45bAEptxxWokZa3r60w/2q2RYs6qehIv/eBa/TDuq+PnATi34+ndp1Yq7yM2jjirz2pXHdef35wwofl7bFuKq3F4SEZGGLS42hnMHdyQxLoZHLx9e6Z2+hy8azL9+lhr2O7iVSYyL5ebRvZm9fjfTVoce+7ew0PHI5JXsOuT4ZtX2Ku/7y+WZ7D6Qx+/O6kdSYoA563dXuP7mPQcZ99IcfvbcLDL3HarW71EdukpLlSXEWpmSg1H9O/DRLadwxYiSHbKWP3A2152cwjd3juKGU3pWuf62eUKgeLSHcSN78fGtp9K1tVcDW9SJ7Y1xJ/DJracy794zmHBqVqB1LAAADrhJREFU2ZnmRvXzCvh/d1bZWuI//9gfj7hnmyrFE8qQEK3fIiLSuN09diCTbz+tzHUylF7tmzOyks5mdeWy1K50bll+q276ykxWZ2YD8NXyqie6b8/JoH1SAqf1bc/w7q2Zt6HiRPeJL1cBcDC3gP9+Z2Gd1fRWmuia2XNmlmlmi+skAjniDerSskyLbWJcLPeffzTd2jTl3vMGEldOy+djVwzjnV+W7Cw3oFML/nPzyfzurH6AN3zaWzedyMv+rHLH92rLgE4taN0sno7NYvj16JIzwv3rZ6ks+1PFHfqeu3YEU+4YWeK12BjjmhO91utbT+9TZpsnfzKc9/7rJK4/uWeF+xYRkcaneUIg6md7A68v0K9G92Hehj2kryybyD799Ro6t0wkNTmWr1duL55muCLbs3L4akUmPx7ehUBsDMd0b8WKbVlkHQpd+rBux37enJ3BT47vzvhz+vPViu28/v3GWv9uoVSlRfcFILzDAIj4LhzWhWN7tC7z+rBurUokxyNS2tCqaej5z+84sx9jBhyeZScQG0OTeO92UMdyBvlulhCgd4eS5RbnDu7EHy8cxLoJY4tLMor8aFhnzhvSmeHdWzN2SCfuGTsg7KNjiIiI1IdLju1K19ZNePCjpew9cDgZXbBxDzPX7uL6U3pyTHKAHdk5LNm8r9L9vT9/EwWFjov9PjrHdG+Nc954wqE89sUq4mKN/xp1FNecmMJJR7XlwY+WsnHXgfD8gkEqTXSdc1OBXWE/skg9+Oz2kUz771EVrlNUE1zRvBqdWx0ukYiLjeHGU3vVe22ViIhIOMQHYvjrJUPZuOsgP39pdvGsbhOnriEpMcAVx3VnULtYzOCrFZkV7ss5x9tzMhjatSV9k70GpGHdW2EGc9eXTXRXbcviP/M3cc1JKXRISiQmxvjrpUOJMeM3by2goAotyNWhGl1p0Fo2iaNr66YsuO9MFv3hzJDrDO3qzXrTOkSL8ej+HRg7pBPjRoZvpAYREZFIO/Gotvz9sqHMWreL216fz9od+/lk8RauOr4HzRMCtIg3hnRtRXolie6SzftYvjWLS4JmZmuRGEffDknMDVGn+8jklTSLD3DTyMMdxru0asL9FxzNrLW7uOqZ71gXxqHJAuHakZmNA8YBJCcnk56eXuH62dnZla5Tn6ItHoi+mKItHjgc086dh3tsVjfGeOe4qn88JzbLJD3dq1daujkfgAN7d/KzoYnMnzW9WvGIHOlSxk+KdAglhCueaBuLWySSzh/ame1ZOfzpo6XM2bCb2BjjupNTipen9W3P41+uYtf+XNo0K9sYtOdALr95cwHNEwKcP6RTiWXH9GjFRwu3UFjoiieqWrxpL58s3sqtp/ehdan9XXxMFwoKC3lw0jLOenQqt43py89P7Vnr0Y3Clug65yYCEwFSU1NdWlpaheunp6dT2Tr1KdrigeiLKdrigcMxvbJ+NmRuA6hRjKNLPd8zbxMsnE9ycjJpaRVPZBEqHvn/9u4/yKqyjuP4+7u4SCK6CyIRICwNrSLpsq6iIzKkk4JamxMmlj9CbKZSs2mawhjNfv2hMznijKOWaZqGluXg2KSZUvqHoSK/Vm1lRRRIWfwBhhWGfvvjPEvX3b3sPXfvPefcw+c1c2bPffbuOZ9zz3Pu8+y955xHRERqwUUzm9j6zn+45fENnH3MeMYUXNvyqcMPZcmj63li/TbaWz486um7u3az4JdP8/Ib73L7gmP7XEcz/bBGlj61iZe27WTKmBG8/4Fz5bIOGg+oZ+FJfS/qNjPOOfYwZjcfylXLOrjmob/z4Np/cNfCGX06xXHo1AXJhS+0DTxIRRw9/33uV6dDRERE8u27cw5nyfwWFp9xxIfKjxp3MKOGD+UvnR++O8Ou3e/z1btWsmbTdm44d3q/I6L2XGjec/rCnU9uZNWr2/n+Z47koGH1fZ7fY8xBw7jl/DZuPq+V5jEjaDig+HNLUcrtxZYCTwLNZrbZzBYOao0iVdB7+OLBmjvto3zlpCauPPOIgZ8sIiJSw+rqjPaWcX0+la2rM2Z9YjR/fXHbnovEdu7azTeWruKJ9W9wzeePKjoY1eRDhtNwQD3PvrKdTW/9i2sf6mR282jaWz5WUqY508Zy3TktJQ84VcyApy64+7mDWoNIDaofUsfiM/oObywiIrIvmd08mvtXbWHt5u1sfvvf/PgPz7P1nV1cdeZUzm6bUPTvzIzpExpY+erbfO/+ddQZ/OSsTw664xpXxc7RFRERkfjMrAG4FZgGOHCRuz+ZbiqRyKwpo6kzuPiOZ3jz3feYNu4gbjrvGFoP63sP/N6OmdjI8s5tdHXv5IftRzKuoe9optWmjq6IiEi6lgAPufs8MxsKZH94LdlnNA4fyvGTR9GxZQc/aj+SL86YyJC93Xi+QE9nuG1iI+fNmFjNmEWpoyu5seySE1m3ZUfaMURESmZmBwOzgC8DuPt7wHtpZhLp7WcXtGFEo4rG0TZpJBfPbOKCEybtucg7aeroSm4cPaGBoyc0pB1DRCSOJmAbcLuZHQ2sBC539z13zK/1+9RD9jJlLQ9kL1Ol8sw8EDas62bD4COVlUkdXRERkfTsB7QCl7n7CjNbAiwCrux5Qq3fpx6ylylreSB7mbKWB8rLpJuEioiIpGczsNndV4TH9xF1fEWkAtTRFRERSYm7vw5sMrPmUHQK8HyKkURyRacuiIiIpOsy4O5wx4UNwIKU84jkhjq6IiIiKXL31UBb2jlE8kinLojkjJnNMbNOM+sys0Vp5xEREUmLOroiOWJmQ4AbgbnAVOBcM9NYxiIisk9SR1ckX44Dutx9Q7jx/D1Ae8qZREREUmHuXvmFmm0DXhngaYcAb1R85eXLWh7IXqas5YHsZSqWZ6K7j672ys1sHjDH3S8Oj88HZrj7pb2et+cG9EAz0DnAorP2OldC3rYpb9sD6W1TIsdrOWq0fYXsZcpaHshepqzlgf4z7fV4rcrFaKW8QZjZM+6emZPvs5YHspcpa3kge5mylqeYwhvQl6JWtiuOvG1T3rYH8rlNg1WL7StkL1PW8kD2MmUtD5SXSacuiOTLFmBCwePxoUxERGSfo46uSL48DUwxs6ZwT875wAMpZxIREUlFmvfRLflr04RkLQ9kL1PW8kD2MqWax913m9mlwMPAEOA2d3+uAovO2utcCXnbprxtD+Rzm5KQxdcta5mylgeylylreaCMTFW5GE1EREREJG06dUFEREREckkdXRERERHJpcQ7ukkOT2pmt5lZt5l1FJSNNLNHzGx9+NkYys3Mbgi51ppZa8HfXBiev97MLhxEnglmttzMnjez58zs8jQzmdkwM3vKzNaEPD8I5U1mtiKs995wURNmtn943BV+P6lgWVeE8k4zO63c16hgeUPMbJWZPZh2JjPbaGbrzGy1mT0TylKrR0lK8nhNSn/7s9bEeW+rFUW26Woz2xL21WozOz3NjLUgC8ds1upn3LY3gTyx2t4kldr2JpSl5LZ3r9w9sYno4piXgMnAUGANMLWK65sFtAIdBWXXAovC/CLgmjB/OvBHwIDjgRWhfCSwIfxsDPONZeYZC7SG+RHAi0TDtKaSKSz3wDBfD6wI6/kNMD+U3wx8Lcx/Hbg5zM8H7g3zU8O+3B9oCvt4yCD33beAXwMPhsepZQI2Aof0KkutHiU1JX28JrhdffZnrU3EeG+rlanINl0NfDvtbLUyZeWYzVr9JGbbm0CeWG1vwvuupLY3oSwlt717m5L+RDfR4Und/XHgrV7F7cAdYf4O4HMF5Xd65G9Ag5mNBU4DHnH3t9z9beARYE6ZeV5z92fD/D+BF4BxaWUKy90ZHtaHyYGTgfuK5OnJeR9wiplZKL/H3Xe5+8tAF9G+LouZjQfOAG4Njy3tTP1IrR4lSMMJZ1TM97aaUGSbJJ5MHLNZq59ltL3VzhO37U1EzLY3LbH3WdId3XHApoLHm0NZksa4+2th/nVgTJgvlq0qmcNX7NOJ/pNLLVP4mmI10E3U+XoJ2O7uu/tZ9p71ht/vAEZVMk9wPfAd4IPweFTKmRz4k5mttGjoXMhIPaqyWsxciv72Zx4Uq5O17tJwGtBttXY6RgqyfMxmon6W2PYmkSNO25uUOG1vEuK0vUXt0xejefTZd+L3VzOzA4HfAd9093fSzOTu77t7C9EIWscBhye17v6Y2ZlAt7uvTDNHLzPdvRWYC1xiZrMKf5lWPZKy7XV/5kGO6uRNwMeBFuA14KfpxpFKUNurtrdEFWl7k+7oZmF40q3hq2TCz+4BslU0s5nVEx1od7v777OQCcDdtwPLgROIvm7vGUykcNl71ht+fzDwZoXznAh81sw2En3tdjKwJM1M7r4l/OwG7id6U0p9nyWgFjMPqMj+zINidbJmufvW0CH4APg5+dlX1ZLlYzbV+hmz7U1MiW1vEuK2vVUXs+0tKumObhaGJ30A6Lni/UJgWUH5BRY5HtgRPh5/GDjVzBrD12anhrLYwvkuvwBecPfr0s5kZqPNrCHMfwT4NNG5S8uBeUXy9OScBzwW/qN6AJhv0R0QmoApwFNx8wC4+xXuPt7dJxHVj8fc/UtpZTKz4WY2omee6LXuIMV6lKAsHK8VtZf9mQfF6mTN6mnQgrPIz76qliwfs6nVzzLa3mrnidv2Vl0ZbW9VldH2FjfQ1WqVnoiuSn+R6HyUxVVe11Kir7v+S3RuyUKic04eBdYDfwZGhucacGPItQ5oK1jORUQXM3UBCwaRZybRx+xrgdVhOj2tTMBRwKqQpwO4KpRPJuoUdgG/BfYP5cPC467w+8kFy1occnYCcyu0/2bz/ys/U8kU1rsmTM/11Nk061GSU5LHa0Lb0+/+rLUpzntbrUxFtulX4ThaS9TAjU07Z9anLByzWaufxGx7E8gTq+1NYf8N2PYmkCFW27u3SUMAi4iIiEgu7dMXo4mIiIhIfqmjKyIiIiK5pI6uiIiIiOSSOroiIiIikkvq6IqIiIhILqmjKyIiIiK5pI6uiIiIiOTS/wAxJT/HkAVCywAAAABJRU5ErkJggg==\n"
                },
                "metadata": {
                  "needs_background": "light"
                }
              }
            ]
          }
        },
        "e4080d6755494094bb2b24acff9ed811": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2de2c4bddd3d4474a840fb23eb527538": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_d0f5c0e99a73402f803073e77a9ce0f9",
              "IPY_MODEL_10008724e14b48a8a84f54b961ed1e75",
              "IPY_MODEL_fe6759547d0b44879b8c18dfe08ec022"
            ],
            "layout": "IPY_MODEL_871d2b2889ae4862a958d949e6b54e7a"
          }
        },
        "d0f5c0e99a73402f803073e77a9ce0f9": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_b0fd92d27a5c4fba93d0a582b3ed205b",
            "placeholder": "​",
            "style": "IPY_MODEL_eb658d1f4fb04a5ab212d7607ddb7f5e",
            "value": "100%"
          }
        },
        "10008724e14b48a8a84f54b961ed1e75": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_15c83816dddd4df3a1f3e0bfc9c7911e",
            "max": 5000,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_8d319226a12d4f21ad4a0dd9bd3ad9a3",
            "value": 5000
          }
        },
        "fe6759547d0b44879b8c18dfe08ec022": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_56684b96d98f464eae622fa17065a54a",
            "placeholder": "​",
            "style": "IPY_MODEL_03cf0c10675045e4b26e4e4f89ce74a4",
            "value": " 5000/5000 [01:56&lt;00:00, 52.24it/s]"
          }
        },
        "871d2b2889ae4862a958d949e6b54e7a": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b0fd92d27a5c4fba93d0a582b3ed205b": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "eb658d1f4fb04a5ab212d7607ddb7f5e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "15c83816dddd4df3a1f3e0bfc9c7911e": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "8d319226a12d4f21ad4a0dd9bd3ad9a3": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "56684b96d98f464eae622fa17065a54a": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "03cf0c10675045e4b26e4e4f89ce74a4": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "1b445bf445424d2994fedcdbc526294b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_71cebf62bff3491ca3550a1d51c45f2c",
              "IPY_MODEL_96c94bece44643f68b4dd634018878fb",
              "IPY_MODEL_0227351061d54ef691a4957995e02c0b"
            ],
            "layout": "IPY_MODEL_bef84e8fa7c142c3a551d101129c3cb0"
          }
        },
        "71cebf62bff3491ca3550a1d51c45f2c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_5b4a3c14447348e2a59b52770e55b7ae",
            "placeholder": "​",
            "style": "IPY_MODEL_753853f9495b44b796907ef9bbec49ba",
            "value": "100%"
          }
        },
        "96c94bece44643f68b4dd634018878fb": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_72eb109eb34f45bd93f0ffa1002ffc46",
            "max": 407,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_d61c491a23f44ad398ead9393e8ee000",
            "value": 407
          }
        },
        "0227351061d54ef691a4957995e02c0b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_72857c50d07d4372a7c8aa5533b2fd9d",
            "placeholder": "​",
            "style": "IPY_MODEL_24259531823c46c3b45de1fcbf981464",
            "value": " 407/407 [00:03&lt;00:00, 111.11it/s]"
          }
        },
        "bef84e8fa7c142c3a551d101129c3cb0": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "5b4a3c14447348e2a59b52770e55b7ae": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "753853f9495b44b796907ef9bbec49ba": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "72eb109eb34f45bd93f0ffa1002ffc46": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d61c491a23f44ad398ead9393e8ee000": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "72857c50d07d4372a7c8aa5533b2fd9d": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "24259531823c46c3b45de1fcbf981464": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "15431de86be6410fadc00f2a91140b91": {
          "model_module": "@jupyter-widgets/output",
          "model_name": "OutputModel",
          "model_module_version": "1.0.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/output",
            "_model_module_version": "1.0.0",
            "_model_name": "OutputModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/output",
            "_view_module_version": "1.0.0",
            "_view_name": "OutputView",
            "layout": "IPY_MODEL_82170e57a38649ef9a83963fbb203de0",
            "msg_id": "",
            "outputs": [
              {
                "output_type": "stream",
                "name": "stdout",
                "text": [
                  "J=-1.060, mean score=6.474\n"
                ]
              },
              {
                "output_type": "display_data",
                "data": {
                  "text/plain": "<Figure size 864x288 with 3 Axes>",
                  "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsYAAAEICAYAAABcYjLsAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdeXhcVfnA8e+brU26L2noSmhpi7TskUIRiFL2TRQFRAUEUcSdHwqigiJSFUURRauUXUCRRdkXCaXQldJCC3RPm65Jl+x78v7+uHeSyWT2PZP38zx9OjP3zLlnJnPvvHPue84RVcUYY4wxxpj+LivVDTDGGGOMMSYdWGBsjDHGGGMMFhgbY4wxxhgDWGBsjDHGGGMMYIGxMcYYY4wxgAXGxhhjjDHGABYYR0VESkVkW6rbES0RuV9EfuHePlFE1sax7hdE5DL39uUisjCOdV8qIi/Hqz6TWfr6cZlsIjJJROpFJDtJ+1sjIqXJ2JdJDhFRETnYvf0XEflJqttk/Evm36evf1dbYNzPqeqbqjo9VDkRuUVEHg6jvjNV9YFY2yUixe5JN8er7kdU9bRY6zYmXYjIuEDBvHfQkQiqulVVB6tqR7zr9v7x7bW/GapaFu99mfSgql9X1VtT3Q4PESkTkatS3Q5vIvJXEbnaz+Nhfb/GIlF/n0z8rrbAuI/x/vClE3HY58n0SzEcl2cBLyZ5n8b0eyk6fs4Eno/0Sfb9mlz99o0WkR+KyBM+j/1BRO5yb18hIh+KSJ2IbBKRr4VZr4jInSJSKSK1IvK+iMx0t+WLyG9FZIuI1IjIQhHJd7ed515qrHZ/6X7Mq85yt73vAQ0ikiMix4nI2275VcEuUYrIUSKywn0tjwMDvbb1uPzs7me7W3atiJwiImcAPwIuci+9rnLLlonIbSLyFtAITPbzK11E5G739X4kIqf4vK45Xve9fzUvcP+vdvd5vG9qhojMFpFlbt3LRGS217YyEblVRN5yX8vLIjI69F/QpFJ/Oi5dZ+Hni1JEPJ//Ve7n/yLPserucxdwn4iMEJFnRaRKRPa7tyd41RPwOBCfnp5Qx4yIfNl9j/aKyE98j1+vclcDlwI/cNv+X6/3a457+xYR+ZeIPOzu630RmSYiN7p/owoROc2rzmEicq+I7BTn/PQLSVIKSKZz/y43isgH7mfoPhHx/o74qohsEJF9IvIfERkXoJ4eVwlE5HwRWekebxtF5AwR+ZyIvOPzvO+LyDMB6gz4dxf3+0BE7nDbvVlEznS33QacCNztfgbvdh9XEblWRNYD60O9Prf8t8U51+wRkd+ISJaI5LnlD/MqO0ZEGkWkMMBrORyoVtVtPo9H8v16hQQ4/3mdH65zj6GdInKFv79PGGVHich/3b/dMvd9D5QWGc53tYrIN0Rkvdv2W0VkijjnyloR+aeI5HmVP8f97FS7ZQ4PsO/EUNV++Q84EOfDNsS9nw3sBI5z758NTAEEONkte7S7rRTYFqDe04F3gOHucz8GjHW3/QkoA8a7+5sNDACmAQ3AqUAu8ANgA5DnPq8cWAlMBPLd5+/F+VLNcp+3Fyj00548YAvwPbfuC4E24Be+rwWYDlQA49z7xcAU9/YtwMM+dZcBW4EZQI5bfxlwlbv9cqDda98XATXASK/XNcervq59uPtWIMdr++XAQvf2SGA/8CV335e490d5tW2j+97mu/fnpvpzZ//suPRqUy6wx/Na/WxX4GCv+6Xu8fQrt335wCjgs0ABMAT4F/C013MCHge+x1iIsocC9cAncM4pd+CcR+YEaPv9uOcYr8fKPeVxjvVm9++SAzwIbAZuct+XrwKbvZ77FPBXYBAwBlgKfC3Vn9dM+Of+XVa7n+ORwFt0fz98yv2MHu1+5v4ILPD3GfX+mwPH4pzrT3WPhfHAIW4d+4CPedXxLvDZAG0L+HfH+T5ocz8r2cA1wA5AvD7PV/k5pl5xX2d+mK/vdbf8JGAd3d9vfwZ+5VX2O8B/g7zPNwC3B9h2C+F9v4Y6/7UDP3fLnuVuH+Hn7xOq7GPuvwKcY78C97vXT9uLCfJd7fU+PgMMdV9PC/AaMBkYBnwAXOaWPQqoBGa5f9fLcD6jA5J1TPTbHmNV3QKsAC5wH/oU0Kiqi93tz6nqRnW8AbyM8ws0lDacL6hDcA7QD1V1pziXQb4CfEdVt6tqh6q+raotOAHjc6r6iqq24Xzp5ON8QXvcpaoVqtoEfBF4XlWfV9VOVX0FWI7z4fZ1HM4H//eq2qaqTwDLArS9A+fkcKiI5KpquapuDPF671fVNara7rbdV6XXvh8H1uIc3LE6G1ivqg+5+34U+Ag416vMfaq6zn3P/gkcGYf9mgTqR8clwEnAKlWtC/f9ATqBm1W1RVWbVHWvqv5bVRvdem7D+cL0FslxEKjshThf+gtVtRX4Kc6XXSzeVNWXVLUdJ6AvxAnE23C+lItFZLiIFOG8h99V1QZVrQTuBC6Ocf+m293u53gfzmfoEvfxS4H5qrrCPSZuBI4XkeIQ9V3pPu8V91jYrqofuXU8jnOsICIzcAKrZ30rCPPvvkVV/6ZOnvwDwFigKETbblfVfe5nPJzX9yu3/Fbg917vzQPAJSIi7v0vAQ8F2e/ZRJ5G0eP7NYzzXxvwc7fs8zg/ZgONIfJb1u2R/yzOeaZRVT9wX2usfq2qtaq6BueH2MuquklVa4AXcAJigKuBv6rqEvd8/ABOIH1cHNoQln4bGLv+QfeH/AvufQBE5EwRWexeLqnGOUBDXopX1f8Bd+P0QlWKyDwRGeo+dyBOj4yvcTi9up46OnF+oY33KlPhdftA4HPuZYZqt32fwDkp+Kt7u6p6f4lt8VMOVd0AfBfn12uliDwW6LJZgHb542/foeoMR4/3zKtu7/dsl9ftRmBwHPZrEq8/HJcQII0ihCpVbfbcEZECcQb0bBGRWpzLmsOlZ5pBJMdBoLLj8HqtqtqI0xsei91et5uAPdo9ELDJ/X8wzvuaC+z0el//itODaOLD+3PsfY72PQbqcf7u3seAPxPxf0yBE2R9wQ0ovwT80w1KfYXzd+/6vLqfSQh9nvd+reG8Pr/vjaouwTlGSkXkEOBg4D/+digiw3F+lL8dom3B2hrO+W+v+0PTI9jxHqhsIU4Ptfe+Q33Ph8P3ePe972nngcB1PufRicQnbghLfw+M/4XzoZ6A00P1DwARGQD8G6eHqEhVh+N8gUmgiryp6l2qegzOJYhpwPU4l2uacS6D+NqB82HA3b/gfBC2e1frdbsCeEhVh3v9G6Sqc/3UvRMY7/WrFpxLQoHa/g9V/YTbHsW5bOu7/x5PCVSXy9++d7i3G3Au1XgcEEG9Pd4zr7q3+ylr+pb+cFxCdIGx73FxHU6P0CxVHYrTCw1hvicR2Al45y570jgCibU32VsFTo/RaK/3daiqzojjPvq7iV63vc/RvsfAIJy/e6jzbAX+jyncqz+tOD2dXyBwL2usf/dwvrPCeX2B3htwgvwv4gT4T3j/aPVxOvA/DTwDTMi2xnr+i0AVTprFBK/HJgYo26ONcVIB3OZzHi1Q56pwUvTrwFhVq3DyeO7DyWf70N2Uh5NSUAW0i5PQH9bUIyLycRGZJSK5OIFfM9Dp9jbNB34nzhRN2W6S+gCcS5ZnizPQLRfny66FwL8uHwbOFZHT3XoGusn0E/yUXYTzIf+2iOSKyGdw8r/8tX26iHzKbVMzzq+4TnfzbpxLm5F+ZsZ47ftzOLmdnmBgJXCxu60E53KtR5W778kB6n0emCYiXxBn0NNFOAFPr0typm/pD8eliByEkzP3Ye9quuwm8OffYwjOcVotIiOBm0OUj9YTOK9ttjtI5haCfyGH0/awqOpOnEvGvxWRoeIMfpoiIr4pIyZ614rIBPczdBNOugPAo8AVInKke0z8EliiquUh6rvXfd4p7t9rvNur6vEgzhWcNlX1O6grDn/3cD6D4by+68UZ5DoRJ4/4ca9tD+P8eP+i+5oCOQt4LkRbQ32/Rn3+i4QbvD8J3OJekToE+HKQp4T6ro7U34Cvu+drEZFBInK2iAyJU/0h9evA2PUPYA5el2vdXL1v43wx7sf5Vev3EokfQ3H+sPtxLrvsBX7jbvs/4H2cHN99OL2xWaq6FufA+iNOD9a5wLluLl8vqloBnI8zkrUK5xfW9fj5e7p1fAYnGX4fTt7kkwHaPgCY67ZhF05Qe6O77V/u/3tFZEXwt6CHJcBUt87bgAtV1XMJ9ic4vQr7gZ/R82/Q6JZ/y72c0iO/yK3jHJxgZS/OwKhzVHVPBG0z6Sujj0vCyze8BXjA/fx/PkCZ3+PkPe8BFhPl1G+huHmB38LJ/d2Jk49YifNDwZ97ccYqVIvI03FowpdxAoMPcP6GTxA4RcVE7h84QegmnBSIXwCo6qs45+l/4/zdpxBGbreqLgWuwMkJrgHeoOcVvoeAmTiBZTCx/N3/AFwozowVdwVoZziv7xmcgbsrcYLbe72eX4EzJkKBN/3tw73SdDrBj82Q368xnv8i9U2cQXG7cP5WjxLgWA/1XR0pVV2OM6DybpzXuQEnfkkaz+hNY4wxSSIiz+MMeIp4TtN0ICKDgWpgqqpuTnV7TPREpBxnpoVXk7jPfJwfVker6vpk7TdSIqI4n/ENQcrMB3ao6o8DbD8W51j3e6W2LxCRXwEHqOplqW5LMliPsTHGJF8ZzjRQfYaInOteWh2Ek+f4Ps40SsZE6hpgWToHxeEQZ/aKz+DVixxAolKcEkJEDhGRw91UhmNxZhl5KtXtShZbOckYY5JMVX+d6jZE4Xycy6qCMw3dxWqXHE2E3B5qAT6d4qbERERuxZmj//ZgV03ctJK+ZghO+sQ4nPzn3+KklPQLlkphjDHGGGMMlkphjDHGGGMMkORUitGjR2txcXEyd2lMWnvnnXf2qGphqtvhTzjHa0NDA4MGDUpOg9KAvd7MFur1pvPxCn3vmE2ntoC1J5S+2J5ojtmkBsbFxcUsX748mbs0Jq2JiN9VCNNBOMdrWVkZpaWlyWlQGrDXm9lCvd50Pl6h7x2z6dQWsPaE0hfbE80xa6kUxhhjjDHGYIGxMcYYY4wxgAXGxhhjjDHGABYYG2OMMcYYA1hgbIwxxhhjDGCBsTHGGGOMMYAFxsYYY4wxxgAWGBtjorRw/R6eWt+a6mYYY1JkZUU1q7fXpLoZxsSVBcbGmKgs2rSHZze1pboZxpgU+fSf3uKcPy5MdTOMiaukrnyXTMU3PBeyTPncs5PQEmMyU5YInZrqVhhjjDHxYz3GxpioZImggKpFx8YYYzKDBcbGmKhkiQBYr7ExxpiMYYGxMSYq2e7Zo9N6jI0xxmQIC4yNMVERt8e4w7qMjTHGZAgLjI0xUcnOcgJj6zA2xhiTKSwwNsZExY2L6bDI2BhjTIawwNiYDCMi80WkUkRWez12pIgsFpGVIrJcRI6NdT/dg+8sMDbG9F0PLd7Cxqr6VDfDpAkLjI3JPPcDZ/g89mvgZ6p6JPBT935MPIGxdsZakzHGpM5Pnl7NebZQiXFZYGxMhlHVBcA+34eBoe7tYcCOWPdjqRTGmEzR0NqR6iaYNJGxK98ZY3r4LvCSiNyB84N4tr9CInI1cDVAUVERZWVlASvcuNVZDnrhwrcYOkDi3Nz0VF9fH/Q9yTT2evuXK+5byiXHTuK0GQekuinGpIwFxsb0D9cA31PVf4vI54F7gTm+hVR1HjAPoKSkREtLSwNWuG3xFvhgNccdfzxjhg5MTKvTTFlZGcHek0xjr7d/eX1tFa+vraJ87tmpbooxKRMylSLAQJ7fiMhHIvKeiDwlIsMT20xjTIwuA550b/8LiHnwnWe6NpvG2BhjTKYIJ8f4fnoP5HkFmKmqhwPrgBvj3C5jTHztAE52b38KWB9rhZZjbIwxJtOETKVQ1QUiUuzz2MtedxcDF8a3WcaYaInIo0ApMFpEtgE3A18F/iAiOUAzbh5xLLqma7MuY2OMMRkiHjnGXwEej0M9xpg4UNVLAmw6Jp77sXmMjYmOiMwHzgEqVXWmn+3nA7cCnUA78F1VtfnEjEmCmKZrE5GbcA7aR4KUudpdUGB5VVVVLLszxqQRyzE2Jmr30ztF0dtrwBHuvONfAf6ejEYZY2IIjEXkcpxfvJeqBu4yUtV5qlqiqiWFhYXR7s4Yk2bEk2NskbExEQkw17j39nqv79VBOPOQG2OSIKpUChE5A/gBcLKqNsa3ScaYvsDTYxzkd7ExJkoicgFwOzAGCDh/WiRzj0N4czVHOpdztHM/p9u80Z72tHcqP1zQxBc+lscxRamb1TZd3590kaj2hPyLBxjIcyMwAHhFnG6jxar69bi3zhiTtjw5xjYrhTHxp6pPAU+JyEk4+ca95h13y4U99ziEmKv5xecAwp/LOdLykbQlmdzXMXjwYEpLS9lV08zel1/jnxvhuotKU9astHl/XP2lPeHMSuFvIM+9cW+JMaZP6Z6VIsUNMSaDuTNDTRaR0aq6J1H7eWtDwqpOqA2VdVz7yLv882vHM6wgN9XNMRkgpsF3xpj+yzOPsc1KYUx8icjB4l6OFZGjca7Q7k3kPp96d3siq0+YP7y2gbW76yhbVxm3OtVSuvs1WxLaGBMVm67NmOgESFHMBVDVvwCfBb4sIm1AE3BRsEHumWDL3gZGDx7AoAHJDUuCva2CJLElJl1YYGyMiYpN12ZMdILMNe7Z/ivgV0lqDqsqqnlpza5k7c6vk39TRsmBI3jimtkpbYcxlkphjImKTddmTGY4/09vUdfcnupmsHzL/oifE2mf7r0LN7Oxqj5omczumzehWGBsjImKTddmTN+2t76F5raOgNs7O5Wf/XcN5//pLe58ZV0SW9Zbc1sHz6zcHtP5pq2jk1uf/YAL/vRWWOXFMin6JUulMMZEpWu6NusxNqZPOuYXr3LUpOEBt2+oque+t8oBJ93ie6dOS1LLevvl8x/y4KItjB48gBMOHt1reyTxcmNr4B8DxliPsTEmKt2D71LcEGNMWP76xkYuf7GBhpbutIl3t1ansEXh21XTDNAr5SMRvbp2SuvfLDA2xkTFpmszpm95cNEWwOn9PfyWlwKW+8+qHZT+5vV+f2xbJkX/ZKkUxpiodM9K0b+/PI3pa554Zxu1QQbbXf+vVbS0d9Lanj6r94Q6y0Qy97B3STt9GV/WY2yMiYpYjrExGS2VQeOijXu5/63NvR73TZ2IpFc33LI2oLh/s8DYGBOV7lkpUtwQY0xEQh2y8crbrW9pZ8vehoie8+b6KgAu+dtibvnvB722h3O+uXfh5oj364/YtBT9kqVSpJHiG54LWaZ87tlJaIkxoWXZPMbGZLRYj+xL/7aYVdtqIvreWlVRzYlTC3s9/uHOWgDuf3szZ8w8oNd2T8Bc19zGrc9+wN8WbGLxj06JruGmX7MeY2NMVGxJaGMyk2cp5HBTCto6/Ocir9pWE/G+A+1y2/4mAJZu3hf8+e7/3jNvxKsNpn+wwNgYExULjI3pmzbvCS/NINwj+8J73o6+MRHuM1B6g+fhOb99I/Q+IjhnLdm0l2sfWWF5x/2IBcbGmKh0z0qR4oYYY8LiCe72NbQGLRdpaq2/nuHKuubIKomRJ26trGuJua57F3YP+vvK/ct47v2dNPSjRUGaWjv41Ysf0dLef16zNwuMjTFRsRxjY/qmZIwpO/a213o99sMn3uO/q3YEfV6kHbPRDJAL9Zz73y6PuM5Mcs8bG7mnbCMPufNe9zcWGBuTYURkvohUishqn8e/JSIficgaEfl1HPYDWCqFMZkmUNgYrKe5trktZL2PL6/gW4++G7TM6h3B85LjHdOHe/a6eN6iuOyvvqWdin2NcakrUTw9xW0d/fPcboGxMZnnfuAM7wdE5JPA+cARqjoDuCPWndh0bcZkNt9j+9n3Avf2ltz6KuDMP9weYDBeOPaHSPMIJJLzUCT5wp6Sq7fXRtagAC766yJO/PXrcanLJIYFxsZkGFVdAPgO3b4GmKuqLW6Zylj3Y6kUxvRNifgx29rRyTtb9nPJ3xbzu1fWhSy/r6GVP77bTE1Tz55mf02rbgwcLAfsQU7TKYjX7IhPgB2NZeX7wh542Z/ZPMbG9A/TgBNF5DagGfg/VV3mW0hErgauBigqKqKsrCxghVWNTq/QBx9+yKi6DQlocvqpr68P+p5kGnu9mSXy9Kee5X76zBr21Lfy/VOn+S1d5Q5821BZH7Lmvy7YyDu7O/jHkq1cUzolaNkjf/5K1+1krrmRaet7fO4vTjqIrYcQnAXGxvQPOcBI4Djg48A/RWSy+lxTVNV5wDyAkpISLS0tDVjh9uomWPA/pk2fTunHJyWs4emkrKyMYO9JprHXm5lCxcWeANpfubteWx8wMF68aW9Y+29sbWfxxvDKBvPu1v28tGYXEPtiJIFkWGxswmCpFMb0D9uAJ9WxFOgERsdSYbbYdG3GZKJog0HPbA6hTgnXP/Fe1xRva3fVcu0jK6La9wV/fjvh06gl6vS2qya509lFpJ+f0y0wNqZ/eBr4JICITAPygD2xVGg5xsZktmiP7FCD29btquu6/fTKHTz3/s6w9ylBQudwB9WF+7oSmUpx3O2vsXB9TKfghMu0VJJwhQyM/U39JCIjReQVEVnv/j8isc00xoRLRB4FFgHTRWSbiFwJzAcmu8fxY8BlvmkUkcrqmpXCAmNj+gLPsbq9uil4wRgDolc/jH5sb8jzSYC2RXseSuX564v3LuGhxf1zruB0Fk6P8f34TP0E3AC8pqpTgdfc+8aYNKCql6jqWFXNVdUJqnqvqraq6hdVdaaqHq2q/4t1P54loa3H2JjMUtfcDqRmKsZE7jLSuiv2NSX8PfjFsx8kpF5V5R9LtoY1v3TgOuLYoD4kZGAcYOqn84EH3NsPAJ+Oc7uMMWnOcoyNyWxb9iZmaq+YLtEHON9EsgJeJLtvauvOYa5tbuNn/11Dc1v6L5W8Yms1P3rqfX705PuRP7mfplB4RJtjXKSqnqSgXUBRnNpjjOkjxD172Mp3xmSm6594z+/jS8KcfcLDd6W3dbsDT+cW8nQSZipFXXN7rzmSu8oGqHrxjvagu/7Dq+u5761y/rW8wu/2hpZ2lpf79iMGl6izpyd4D7ZaYbqr2NdIZwp6XmIefOfmKQZsuYhcLSLLRWR5VVVVrLszxqSJbFsS2ph+6aJ5i3lm5fawy0ey0tuaHTX8d1XgFfYCuePltez0menh6FtfCVDav7+81xJ0e5u7ol+g9LHvPPYuF/5lUZ8ORoG0mJViQ2UdJ/76df5clvw58qMNjHeLyFgA9/+AmfaqOk9VS1S1pLCwMMrdGWPSTXeOcYobYoxJuu88tjIh9bZ1KN969N2In7e7tqXX8xI1/iFQ2oZnVbt0S7V47cPd1EWRa5zKWSk8VxWWbI6sBz4eog2M/wNc5t6+DHgmPs0xxvQVWZZKYYwJU/ENzyV8H21BfqXvqmnmmZVOT7RqdCkGCTnVJfj0uXVfI1c+sJzr/rkqsTuKs294zW2dbCFXvnOnfioFRovINuBmYC7OyllXAluAzyeykcaY9OPpMU5FDpgxfZmIzAfOASpVdaaf7ZcCP8TJqK0DrlHVvhXZJEi0nZiX/G0xm/d0DyZcs6OGE6cWRhSXqltaBPbWtzBoQA4Dc7N5fNlW8nLSc1mIJncBlC17G0OUNB4hA2NVvSTAplPi3BZjTB+SZbNSGBOt+4G7gQcDbN8MnKyq+0XkTJxl2mclqW1pLdjl/WA9uvFcaU6AY37xKkdOHM7T157AD//tzPwwdtjAuO0j0crWVvKTZ1bz6vdPZkBOdqqbk1ZCBsbGGONP18p3lkphTERUdYGIFAfZ/rbX3cXAhES3KV1dMm9x2GU1yaPGVlZUh132//61ym/Oc6LaHOq0fMt/1lCxr4kd1c0cNHpQQtrQV1lgbIyJioizOKutfGdMQl0JvBBoo4hcDVwNUFRURFlZWcCKmluCz7qQjhb5TA3X2dkZ8DXu2V/X67E5c1/gx8fl09HZc0DcqlXv0bE9m3afYDXY+7d9u5OjvG79er/lW9z3d9GiRYzK75la8cQ7/ueE7uzUgPusr6+nrKyMx9e2ctjobA4dFX7P7po9zutta3NyqRsauqfIKysro6nJWf1wyZIlbBnUs61bK5znbNq4kTLtnprO055g3t7RTm2LcsZBuWG3NZj9+/aHfH/izQJjY0zUssQG3xmTKCLySZzA+BOByqjqPJxUC0pKSrS0tDRgfQMWvQbN8UspSIWc7GwGTDyMCSPy4cWe08A1dGQDPeci3lDdSWlpKdmvvQgd3cHxEUcczolTC50Bey93/+4oLS2FF/0PFBw3bhxUbGXatGnwwepe5QcMGADNzRx//PGMG57f88kB6szOyuKEE08iN7t3jnJZWRmlpaVc/uJzvLC5jfK5Z/utA5zpzSaMKGBgrhM856zfA8uXkJubB62tDBo0GOrrutqcv+x1aGykevCBjCoczKmHdi9HsajxQ9i8iclTplB68pRe7QnmcneQ5dwrTg1aLiT3/Ro5aiSlpcf6LRJOe6KRntnixpg+QbDp2oxJBBE5HPg7cL6qRraiRgC+8/z2RY2tHVzyt8V+50YO9BN90ca9PVawA7j/rfKIFyoJ1QXg6SN4cfUufvx0eCvOtXZ0MvWmF3hjXVVEqRnrd9d1TcHW2NrOnN8t4LteU+iFO9Xa3Bc+4qsPLrdB1F4sMDbGRC1LLJXCmHgTkUnAk8CXVHVdqtvTVwQ6Fz3wdnmvx177qJKL/OQvh3M+CxVz/vzZD3h48daQ9Xi7bP5SPv2nt3qtnBdoTuRT71zApX9fAkBLm9M7sXhz+IG+71zMdhbvZqkUxpioiSRuEn1jMlWAaVBzAVT1L8BPgVHAn90Apl1VS1LT2r6jodV/EJkVpAswkt/1yegD8O3V//mzHwQs+962mh73qxsjX8TDw/lBkMIVPdKIBcbGmKg5OcapboUxfUuQaVA9268CrornPqvq+t7Au3h5/v1dca0vkSvC+Z5O1+7qPaDQ19Ly0KvDhZr9IhWncVWlua2T/Lz0mi7OUimMMVETbPCdMX3B7uXm8aIAACAASURBVNq+n1+cCBf+5e3QhcK0K8B73NDS7vdxf+4p2xjxfr/20DsRPyeYZJ3R73ljIx/76YvsrXd+tC1YV9Xrh0Aq+rCtx9gYEzWblcIY05f5piME1/tcV7Ev8Ipy33ns3a5lqMP14c7aiMoH4ntalihCzGDPeHzZViaOLODeNzfzu4uOZFh+5NOz/cd9b3bXtjBq8AC+PH8pQNDZN5LBAmNjTNQsx9gYk0mC/c73bPMOMv3NjuERaVCcCIlK+/Cs9gfwz2UVfPWkyQnZTyq+Xfp1YFx8g/95Bb2l+peLCZ/9PZMvS8RyjI0x/Uoic4zBmZPYMw9yrLsK94JeZIMQM/ukbznGxpioCdj8l8aYfiFZ8eCc3y3gqgeW09SuLN+yP6Lnvr62ku3VTb2C9817eq685xtw+xucF+jlHnTj86Gfq8rkG59j/sLNoZqcdiwwNsZEzXKMjTGZJNjZ7PHlzvLIyRgQ9vbGvexuiHz1pCvuW8aZv1/Q6/HWCFZi8ry+dT4D4T7Y28G63aFnyfDo1ODTzaWrfp1KYYyJjQAdFhgbYzLElB89H7pQiqgqt7/wEZ87ZkLQcrXN4c+C0V231233/yff3c7ehlbe2rCHDb88i18va+bXy3oH3Yn8CthZ3ZS4ygOwwNgYEzVn5btUt8IYYzLfjppm5i3Y1DWbQzj2NrT6fXyTT2pFIG+sqwqr3Maqek757Rtd92P5XvDulW5u979oSyJZKoUxGUZE5otIpYis9rPtOhFRERkdj31l2awUxvQJiR4w1p/c8OT7oQslwKV/c5awDrVYRyr4LmUdCd/P5ml39u6VTiYLjI3JPPcDZ/g+KCITgdOArfHakS3wYYwxyVG+N/Ccyb4iPS3H+zQeqLr/rNpBY2vkqR7JZIGxMRlGVRcA/n6+3wn8gDhODWmpFMYYYwLx7g1eWVHNtx99lx8/3fNiZrp9h1iOsTH9gIicD2xX1VUS5JqqiFwNXA1QVFREWVlZ0HpVO9lVWRmyXKaor6/vN68V7PUak0rJjBdjSc/w90x/cx17lsbeVRP+8uTRrNgXKwuMjclwIlIA/AgnjSIoVZ0HzAMoKSnR0tLSoOVz3nqBUaNGU1paEoeWpr+ysjJCvSeZxF6vMamztTby6driwV9Qu2Jr8PmUIwlgey1XHeKpf31jI/saWrnxrI+FvY9YWCqFMZlvCnAQsEpEyoEJwAoROSDWii3H2BhjEuOj/f5nZAgnCI10sGWo0/htz30YcFttUxsNPnnD7X4GZUfb93v7Cx/x1wWbonx25KzH2JgMp6rvA2M8993guERV98Rat7PAR6y1GGOM8RVLGsGlf18ScNv6EIt0/O3NyFar+3PZxl6PvbRmV9ftsrWVnDS1MKI6U8l6jI3JMCLyKLAImC4i20TkysTty6ZrM8aYREjUtGyn+pkOLd578lxJVIXL71vG/LcCB9tXPbCcB94uj3MLohdTj7GIfA+4Cuc9fR+4QlXDz6o2xsSdql4SYntxvPaVhaVSGNMXpGIQk4lNXUvy9uUvrzieKvY1cujYoX63ba9u4ub/rEno/iMRdY+xiIwHvo1zSXYmkA1cHK+GGWPSn4gFxsYYkwgf7Ev+qm+Joj1up/d3Rqw5xjlAvoi0AQVA+OsUGmP6vCyBztQMnDbGRCDdgxETvl218b8wr8AdL62lrSM+J3S//SV95KJF1IGxqm4XkTtwVtFqAl5W1Zd9y3nPizpp0qRod2eMSUMCdFiPsTFpzw7TvifZwzfufn1D3OoK5/OWrp/JWFIpRgDn40wDNQ4YJCJf9C2nqvNUtURVSwoL+86oRGNMaM7Kd2l6djPGGBOWRJ/GH1y0haWb/S3Imn5imZViDrBZVatUtQ14Epgdn2YZY/qCLJuVwhhjUiZeqQ+hMm3ikQXx+1fX96wzTVMrYgmMtwLHiUiBOGvMngIEngHaGJNxRMTmMTamD7ALO5lp6k0vJGU/8fz4eD6L4XwmvYNnVeWah9/h7Q0xT8EfVNSBsaouAZ4AVuBM1ZaFu5SsMaZ/sOnajDGm7ws1OPOdLcGXhPZ1zxu9F/2IVUt7Jy+s3sUV9y+Le93eYpqVQlVvBm6OU1uMMX2MiA2+M8aYvu7XL62Na30bKusDbluyeR/FNzwX1/3Fk618Z4yJmk3XZkzfYNO1mWD+sWRrqpvg15a9jUnfpwXGxpioCZZKYUxfYIepMeGxwNgYE7UsW/nOmIiJyHwRqRSR1QG2HyIii0SkRUT+L9ntMyYdba9uSsp+LDA2xkRNbLo2Y6JxP3BGkO37gG8Dd8Rrh3aUmr7u4nmLAWcQ3u4ErP7nYYGxMSZqWdglWmMipaoLcILfQNsrVXUZ0BbHfcarKmNSYk99S9ftWb98LWH7iWlWCmNM/2azUhiTWiJyNXA1QFFREWVlZX7LbajuSGKrjIk/36+a+vr6gJ/3WFhgbIyJWpaI5Rgbk0KqOg93DYGSkhItLS31W27o1v2w+O0ktsyYxBo8eDCBPu+xsFQKY0zUBJuuzRhjTOawwNgYEzWblcKYvsEOU2PCY6kUxpio2awUxkRORB4FSoHRIrINZwXZXABV/YuIHAAsB4YCnSLyXeBQVa1NUZON6TcsME6SdFr+MJy2lM89OwktMX2d02Oc6lYY07eo6iUhtu8CJsR5r/GtzpgMZakUxpioZWGpFMb0BXaYGhMeC4yNMVETyzE2pk+wo9SY8FhgbEyG8bfcrIj8RkQ+EpH3ROQpERkej31lWY6xMcaYDGKBsTGZ5356Lzf7CjBTVQ8H1gE3xmNHgl2iNcYYkzksMDYmw/hbblZVX1bVdvfuYuI0sMd6jI3pG+wHrDHhsVkpjOl/vgI87m9DuMvLerS3tdHeIQlZljMdJWoJ0nRlr9cY099YYGxMPyIiNwHtwCP+toe7vKzHv9a+jEhHQpblTEdlZWX95rWCvd5MotZlbExYLDA2pp8QkcuBc4BTNE7fkiLQYV+4xqQ9O0pNpknUjz0LjI3pB0TkDOAHwMmq2hivem1JaGP6BjtMTaZJ1EfaBt8Zk2Hc5WYXAdNFZJuIXAncDQwBXhGRlSLyl7jsC+cL1y7TGmOMyQTWY2xMhgmw3Oy9idhXljj/dypkSyL2YIwxxiSP9RgbY6ImbjBsU7YZk97UsoxNhknUhcqYAmMRGS4iT7gran0oIsfHq2HGmPTX3WNsX7rGGGP6vlhTKf4AvKiqF4pIHlAQhzYZY/oIzy9rC4yNSXN2iJoMk6iPdNSBsYgMA04CLgdQ1VagNT7NMsb0BeLmUlgmhTHpzQ5RY8ITS4/xQUAVcJ+IHAG8A3xHVRu8C3mvpDVp0qQYdtet+Ibn4lKPSbxw/lblc8+OSz0m+bIsx9gYY0wKpON0bTnA0cA9qnoU0ADc4FtIVeepaomqlhQWFsawO2NMuvGcQGy6NmPSmx2iJtPsakjMhzqWwHgbsE1Vl7j3n8AJlI0x/YTNSmGMMSYVWjrSLDBW1V1AhYhMdx86BfggLq0yxvQJ4jWPsTHGGJM0CfreiXVWim8Bj7gzUmwCroi9ScaYvsJmpTDGGJMKCeowji0wVtWVQEmc2mKM6WPE5jE2pk+wBT5MpnmxvI2vJaBeW/nOGBO1LEulMKZPsN+uJtPUtqRZjrExxrhxMZ0WGRtjjEmidJyuzRjTz9mS0Mb0DXaEGhMeC4yNMVHzrHxn07UZY4zJBBYYG2OiZjnGxvQNtgiPyTSWSmGMSTs2XZsxxphUSNTXjgXGxpio2XRtxhhjUsF6jI0xaSfXPYM0tXaktiHG9CEiMl9EKkVkdYDtIiJ3icgGEXlPRI6OdZ+hgoiDRg+KdRfGJJWELhIVC4yNMVEbnOucmqqb2lLcEmP6lPuBM4JsPxOY6v67Grgn0Q2yqz6mrxk5MDGhsQXGxpioDc5zA+PG1hS3xJi+Q1UXAPuCFDkfeFAdi4HhIjI2tp32fmhyYXcvscXFpq/JSlCXcUxLQidC8Q3PpboJEUtmm0Ptq3zu2UlqiTFePcaN1mNsTByNByq87m9zH9vpW1BErsbpVaaoqIiysjK/Fb5X1d7j/vQRWdQ2Nnbdb2xqirHJxiRXW3t7wM97LNIuMDbGxEZE5gPnAJWqOtN9bCTwOFAMlAOfV9X9se6rINcZgLffAmNjUkJV5wHzAEpKSrS0tNRvuc6PdsM7y7vun11yMM+s3A4NDQBMOWAEvzppMlc+sNzv841JNzk5OQT6vMfCUimMyTz30zt/8QbgNVWdCrzm3o9ZlghDB+ZaKoUx8bUdmOh1f4L7WNR8UyW++cmDuxboAbj7C0dzyseKYtmFMUllg++MMWEJkL94PvCAe/sB4NPx2t+IglxLpTAmvv4DfNmdneI4oEZVe6VRxCLLJ0GzcMiAeFZvTJ9lqRTG9A9FXl+su4C4dQ0NK8hjv/UYGxM2EXkUKAVGi8g24GYgF0BV/wI8D5wFbAAagStS01Jj+h8LjI3pZ1RVRcTvGPRwB/J41NfXo005VNRoQgZBpJv6+vp+8To97PUmhqpeEmK7AtcmvCEpMmlkAVv3NYYsd+TE4eysaWJ3bUsSWmWMwwJjY/qH3SIyVlV3utM+VforFO5AHo+ysjKmTBzO8i37EjIIIt2UlZX1i9fpYa83c/ibji2cHM0TDh7FWxv2RrSvOz53BMcWj+TL85dQvrd3AKxhrlmm2DRyJvksx9iY/uE/wGXu7cuAZ+JV8bD8XKobLMfYmL5GwoiMv3ri5IjrvfCYCUwaVUBBXox9b6p861MHx1aHMRGywNiYDOPmLy4CpovINhG5EpgLnCoi64E57v24GFGQR11LO20dnfGq0hgTZ9F2vJZOHxPXdkREhGJbqtokmaVSGJNhguQvnpKI/Y0YlAtATVMbowfbyHZj0pH6yUlIdJpCOD3SQZ8fn2YYExHrMTbGxGRYvhMY21zGxvQtsQau0fr8MRNDF8Lp5Z5cODixjTHGhwXGxpiYjCjIA2z1O2PS2dhh+aluQpcDw02PUGX88HxGDspLbIOM8WKBsTEmJp7A2Bb5MCZ9HTZhGM9+6xNRPffMmQfEtS1jIlxMJDc7dNf2Ly84LNrmGNNDzIGxiGSLyLsi8mw8GmSM6Vs8OcZ7622uUWPS2czxw3rclzCzeOOZcrHuF2cydGBuWGUjGXiXZQnJJk7i0WP8HeDDONRjjOmDDhg6kNxs8TtfqTEmvVw8PY8nvzE7pjpGBUltCLRtzJABfPjzM8jLyWJofvBx/8WjCgD49FHjgfAC+FTlS5vME1NgLCITgLOBv8enOcaYviYnO4sDRw1iU1V912Plexr4zJ/fsgF5xqSZMw7K5ehJIyJ6jndgesgBQ3jtupOZXjTEb9kXvnti9/O8gtUvzJpEfl42ABNGFPDMtScE3p9PlBvOgiDh9n4bE0qs07X9HvgB4P8IoecSs5MmTYpxdyZZim94LtVNMH3I5NGD2LSnoev+0s37WLG1mvWV9Xy8eGQKW2aMCWTiyALW7q7je3Omhf2ciz4+keEFvXuFf33h4Zx12FgGDwgvrDhi4vCA23xD3HQKerMEOm01vrSQqKsEUfcYi8g5QKWqvhOsnKrOU9USVS0pLCyMdnfGmDQ2uXAwW/Y20O4u8rG7thmwAXnGpDNPYDH9gJ59W1867kC/5Y+fPKprW9GwgT22XXDU+KBBcW52BOGGT8ATVgCUpNj5U4cUJWdHJmViSaU4AThPRMqBx4BPicjDcWmVMaZPmVw4iLYOZdv+JgB213kCY0ulMCZdTR3jzBE8bnh3kFs+92xu/fTMngXdoPMLsyaR4wa4owfn+Svi1+wpo7jyEweF3S7fWSvCGaw3ICe8cCaWJabfu+U0vnicXflOF4laoCbqwFhVb1TVCapaDFwM/E9Vvxi3lhlj+ozJ7ujxTXucPONdNc4MFTVN1mNsTLr63qnT+Pc1x3P4hMBpDeHyzQuG7hSIG848hIG52WHVUz73bGaMc2bPGOmmbEwcGXoO5hnjhgbMe+7ZpugNHZjr93WazGLzGBtjYuZZnWpTlZNnXFlnqRTGpLvc7CyOOTD0GIDvnzqNIyYM4+TpgdMh/YWL4Qya8+eHZxzCP746K2gesr8WXBpGb66lBztmTxmV6ibELO1yjL2papmqnhOPuowxfc/IQXkUDhnAC6t30dGpXTnG1mNsTN83pXAwz3zzE2HPPxyrvJwsZk8ZHfHzzojzQiT+aKKu3ydZJnR8J+olWI+xMSYubjjjEN7Zsp8/vb6BqjonlaLaAmNjMtKAnJ6pEf4CrfjNJtGznjsvOsJvqTFDBvp9PJhzDh8bVYtM6iXqJ4oFxsaYuPjM0eP55PRC7inb2DWdkQ2+MyYz3XjWIXyjdErX/WTm3l5w1ASmFQ3u8Vi0u7ec4b4rUQGsBcbGmLgQEc48bCxNbR2AM9+npVIYk5mGDszlB2ccElbZQNkH15cM5JJjJ7Hq5tOCPv/606f3eqx4VPjLRc8cPzTgtkjDYguk00d2gtYBt8DYGBM3J07tzgssHj3IBt8Z04+FiiFnjM7m9s8cxrD84LnLvvMsA/zqs4dz/OTwBpA9c+0nom5jrH56zqGJ3UGU0mnRlGhlp/PgO2OMARg7LJ+D3blRpxcNsVQKY0xCjBiUx6NXH9d1P1iM5N2zePns4h4/4CNaeCSAIwPMnnHDmYdw5KTYp8LzdsSEYXGtry8bEN4MgBGzwNgYE1efnF5Ifm42UwoHU9vcToetn2qMSbBwUxxGDR7AQ1fO6krPGJSX3WPFvguPmRD0+f5mpRg3fCA/O28GD185q8fjXz95Stz7ZZ/5Znfvd9HQAUFKBpefl6CoMommjkjMa7DA2BgTV9+dM42nrz2BUe7KWLWWZ2xMv+RZ7S6SfOBkKfAKDPO8Vs079dDuJZ9zIshhvWx2MZ+Y2nuKuSw3YPddKTCYyYWDWHD9J0OWO+bAEWHX6esrJ4S/EmF/Y4GxMSauBg3IYfoBQxhe4OQN2gA8Y/qmcJZZ/naQJZbPP3I85XPPZlhBcuY/hp7jHMKR6MF0h40fxjWlU/jzpcdE9LxJowoS1CJHXphLaPdH9s4YYxJieL7TQ1Ld1EZLewf1Le0pbpExJlxzP3MYz3/nxJDlvn/adMrnnp2EFvk3frizXLQnvH3oylmcd8S4kM8LNFOGd5h883kzem8PEUhfcmzP1feysoQfnnEIY4dFPsdyOH5+fu82mthYYGyMSQhPL9G+hhY+/9fFfObPb/nNN25u62Dd7rpkN88YE8TFx05iSuHg0AWTZN6X/Pe4egbWecerkaRAQOCBe5d8fGLX7ZOnBV4O29uxB0Wf3hAp1fRMU0kWW+DDGBMzEfmeiKwRkdUi8qiIJKYbA7qmYPrDq+tZVVHNut31PPvejl7l/vT6Bk67cwHfe3wl7R2diWqOMaYPO22G/+Wev3z8gYAzqC4SM8Y5cxuXFI/oFWAt+dEpvPmDnjm+c7xyj4P59JHj+danDmbZTXMiak+0PnFw5Etnx8OIOKfHfPXE9Ml5tsDYmH5CRMYD3wZKVHUmkA1cnKj9jR+ezxETh7NqWw2fOHg0hxwwhLv/t6FXubK1VeTnZvPUu9tZvGlfoppjTNoQkTNEZK2IbBCRG/xsP1BEXhOR90SkTESCT5XQj1114mTK557dY2aJQF3Ah3jNhzxr8iiW3nQK5xzeO+2iaOhAJo70yfENlHvhQ0S47rTpFA6JfsaISGRF0Dt+9UmTwyrnSU8JZulNc7h8djHXnTot7P0H871Tp/HstwLPN51MFhgb07/kAPkikgMUAL27cONkYG42z1x7Aq9+/yTu+eLRnH/keNZX1lPT1MbP/ruGLXsbqG5sZfWOGj5X4nzvr6+0lAqT2UQkG/gTcCZwKHCJiPiuAnEH8KCqHg78HLg9ua3MPMt/PIenvnFCj8fGDIn8gtlBUaYueFI+xg4byBvXl/ot8+r3T+px/8TxOdx50REB64xl3GCw53rOx8HkZmdxy3kz4jpIcOb49JijOSd0EWNMJlDV7SJyB7AVaAJeVtWXvcuIyNXA1QBFRUWUlZUFrbO+vj5kGYBtwP4dzuC7+f99g/tWtLB/93bGD85CFSZ27mZwLrzx7joOatvS6/lLdrYzc3Q2uVnQoZCfk5pVm8J9vZnCXm9CHAtsUNVNACLyGHA+8IFXmUOB77u3XweeTnSjMom/Vd1GR5hq0btSp85JowpY/bPTOeyWl8LtRAZg3PB8bj73UM6YeQBjh/Xukf3JOYfi29V95WEDOPpjoVM4brtgJjc9tTr8xoQQyes69/BxfOexlUndp0dBgiJYC4yN6SdEZATOF/BBQDXwLxH5oqo+7CmjqvOAeQAlJSVaWloatM6ysjJClfHI3bCHee8tQUdMAtbTNGAkNQMHMChvO5ef90leqVxCfadSWjq7x/PW7qrjnhcXcNNZH2Pt7jo2VNbz9LUn+N9JgkXyejOBvd6EGA9UeN3fBszyKbMK+AzwB+ACYIiIjFLVvb6VJerHbDL4a0s4bQtVZteulojKt7V2r9D5/urV5FV9BECnV7S2ft06ypo3dz/JK5CrrKwKq90HAWvf3cJaP9umtG9h6VKn/sbGRsrKyqivr2fhmwsDvhbPftduDW9KzK1buz92K1asCFiuvLw8ZF3x/gy9+eabDIygw2P4AGFyfnNCPssWGBvTf8wBNqtqFYCIPAnMBh4O+qw4GePm3L2/rRqAD3fWsX53PbMmjyI3O4upYwbz31U7UNUeUyIt3LAHgC37Gli9vYaPdtWxr6GVkYPCnzDfmD7m/4C7ReRyYAGwHejwVzCRP2YTrUdbXnwOIHjbwikDPFu1CnZs67ofqnzum69AmxMcz5w5k1J3oF9Hp8JLzwMwddo0So87sPtJLz3XFRyPGVNIaWlk8xR7Xot3GzdU1sPCNygoKKC0tJSysjKOPu4EeM25sPf+LacxZGBu13MLC539bl1UDh+s8buby2cXc//b5QBMmjQRyjcBcNRRR8PitzliwjBWbavp8ZwDi4th4/qgze/xnvq8Fm+XHDuRR5dWBNzucdJJJ1KQlxO0Lm9lP5zDyqVvJ+SzbDnGxvQfW4HjRKRAnMjzFODDZO3ck8/3nnsS3rqvkU17Gpg9ZRQA04qGUNvczpMrtrOpqp7G1nY2VNbxlhsYb93XxNZ9jQAs3dyr44yKfY08tLh3GoYxaWY7MNHr/gT3sS6qukNVP6OqRwE3uY9VJ6+J6emXFxzGny89OmS5Uw4ZE/U+UpOkFZ4hA/3PBNHpZxpMj1BLP/sbvDfJd+BhDAKlSHwryMIw4RhekLiOEesxNqafUNUlIvIEsAJoB97F7WlKhqH5OeTlZLG3obXH455lVKcWOXOmXvevVQwvyKVoyEDWVdaRm+X8fl9VUU1jq9NptnjTPs6YObZHPf9cXsEf/7eBM2YcwOjBefzkmdUcOXEEp88oYu2uOkqKRyb6JRoTjmXAVBE5CCcgvhj4gncBERkN7FPVTuBGYH7SW5mGvjBrUuhCwJmHjeWjW8/gkJ+8GPE+esxukSTHHhTduWmW+7zRYc6AceLU0cxbsClomfsu/ziHTUj8IDjfnO9ocowTxXqMjelHVPVmVT1EVWeq6pdUtSX0s+JDRCh0T4ae5aJHD85jepEzhdKMscMYlp/LuUeMIz83m237G5k2ZgitHZ0cOKqga2npATlZLNnce1q33bXNAGyqqufN9Xt4ePFW/rtqBw8u2sKFf1nEqorYOtya2zqoqOueZ/nL85dy+/NJ63A3GUJV24FvAi/hXLH5p6quEZGfi8h5brFSYK2IrAOKgNtS0tg+bGBu8J5Sfy6ensfx7hWsUGLtWQ4agHsFicH245lb+ezDxob80fD9U6dx4tTei5SowpTC7pk2Phmgt/0HZ0wPWn+kNJ0iYR8WGBtjkmbMUCcwPnzCcEYPzuPEqYVd+cTDCnJZ/uM5/PGSo3j+2yfy6nUn84+vzuK6U6dx2fHFXXWcNuMAPtpVS3Ob03t88zOreeKdbVTWOTH+xqoGfvOSM7ylYn8jm/c0AHDHy/6GvITvvrfK+clbTSxYV4WqsmLLfj7cFd30co2t7fz46ffZ79N7bvoHVX1eVaep6hRVvc197Keq+h/39hOqOtUtc1Uyf8D2R985ZSoApxyYE3LJ53h5+Xsn8W2fdIJody0inB5gARSPbJ+UCX/7evIbs3s/6DrXa77nL3nnWvuRlx15aJmktz0sFhgbY5LGMwBv7NCBPHb18fz0nJ7Tt+a6J9QRg/IYOyyfUYMH8K1TpnalWYjACVNGoQo7a5qpa27jocVbePa9HVTWOrHDyx/s4v3tNYwenMf2/U1s3duICLy5fg/v+wwyicTeeqf+Hz31Pjtrmqlvaae2KbzR4L5WbKnm4cVbefXD3VG3xxgTH5fNLqZ87tnk+gSPiezVHDc8n9k+q9YF290gr1zhcBbgiMbQgTkh2wFw66dnhqzLk6ccqK5k/QCJhgXGxpik8QzAKxo2kIPHDGZEmDNLTBzhnGTHDh3YNaH8zuomVmytplNh2/6mrh7jN9ZVAc7SrC3tnby3vbpr2dQlfgbthaul3Umj2La/iceWOaOsa5ujC4w9aR9ro+xx7uv+uayCs+96M60vpxoTzMnTeqclxI1XzBhrAJmKQ0xRvlE6peu2P2kcF1tgbIxJnq4e42GRrTg1bng+WeJMrj/OnRx/R00zy8udXOOKfY3sbXACY1Wn/tkHO7mCzW2dlBw4kvHD83k3hjzj6qY2hg1wzub/+8jp6a1tao+qrt11bmC8O/rAeHt1E7956SPaOzpDF04z722vZs2OWvY3RvfDwphkCBZT3vPFY7j+9Pjm3SZSrIHogJzww8Unrzkh4v1lpVGkHHVgLCITReR1EflARNaIyHfi2TBjTObx5BgfMDSywDgvJ4uPjR3K1XJNxgAAIABJREFUEROGc4AbVO+sbmKZGxi3tHeiCiPcQX2zDhrZ1csMcOCoAo6cNJyVW6MPjGua2hg5UJg4Mp/V22uB6HuMPWkfH7k9xmVrK/nT6xsiquPpd7fzp9c3snzL/q7HdlQ3df1YSAeVtc0s3tS7l77G/UFR4U6/F8h3HnuXBxeVJ6Blpj9Y/uM5LL7xlITUPTA3u2vgcCJnsvAEpF+eXZywffjy9PKOHpzHqptP477LP86YCM7Z3rNahNNjPXn0oKgGSyZKLD3G7cB1qnoocBxwrZ/13o0xpsvM8cMYMjCH6QcMifi5/75mNtefPp2BudmMHJTH1n2NrKyopnhUdwA86yCnl3jW5FGMH9GdhzdxZAFHTxrB9uqmrjSGSNU0tTEoR7q+DAFa2ztpbutgf0Mrl9+3lMow6/a0oaquhX0NrTy5Yjt3vbY+6HykvjZU1gPw5voqZyEC4I//28CX5y+NuRe5ua2jK6c6Fn9fuJkv/n1J14wiHp77FfuDB8bPrNzBT5/xv3CBMaGMHjyg64d0NHJDDCL71CFjuPHMQ9zlnOPI6zSQm53Fpl+exQ+8eqePnDgcgHyvYPL4yaM474juAXLxIQzLzw04U4Wviz8+kdsumOk+M3gPsHfAfOSk4WHVPybMaeliFXVgrKo7VXWFe7sOZ9qZ8fFqmDEm88wYN4z3bzmdcVEMHhmYm02O+0U1dthAXl9bSXNbJxccNaGrzAVHj6d0eiFzPlZEQV4Oowc7OcyTRhZwlHvyfXfr/t6Vu8r3NLAiwPaaxlYG5ToLkXirbW7j/e01lK2t6tF7G8zu2mby3J6gj3bVsr+xlZb2TnZFELSvr3R6m19YvYvZc1/j4cVb2La/kcbWjq6e6Gjd+co6zv3jwtAFQ9hd20x7p/K2u0iLh2fQYsW+poDP9f6R4JmBxJhkuzTINGhZWcLXTp4ScOGNSAXKJsjKkh65xnd87gj+880TGOU1F3BeThZ3XXJUwLoD9dz6fTjCvORPHDyam889lLmfPZxLZ/WcsSKeKc5Lb5oTx9oCi0v/v4gUA0cBS/xs61rHfdKk8CbnNiaVim8IvSRl+dyzk9ASE8jYYfms2eGkM3z6qHHc+eo6AA6fMIzTZxzbVW78iAIaWzsYPTiPIQNzyM4S1uyo7bU4iMfcFz5i7e46Xv+/0l7bapramFwgXb3dIs6XTW1Te1cPaLi90btrWzi2eCQLN+xh3a46qt1c2/K9DWH9aOjsVDZU1pOXk8WmKmc6upUV1WyvdgLNdyuqmTk++kn631y/hx01zbR29FwBq7mtg9qmtrAvq+6td6aje2NdFWce1v2e14bRY9zoFQy/t60m6kUQjInFjHHOcXTQqEEhSiZPfl42h08I3cvqmS/e2+8vOpIJI/LjusLfw1fN6v2g1w6OmDi81zzy4aYUHz5hWNdqqckS8+A7ERkM/Bv4rqrW+m5X1XmqWqKqJYWFCRzFaYzpN8YNdwKz4lEFHDhqEKMG5SHSezWloyYO58iJwxERBuZmUzyqIOhMEFv2NbKnrncKgapS29xOQa509RgXu1+Utc1tXYFxpZ/n+qurqq6FGeOGMiAnix01zVQ3OQFk+Z7gqQUe26ubaG7r5MJjnN7ynCxh695GdlY7gXmwXvFQ6prb+GiXm0Pd2rO/53evrOOsuxaGPZvEHjcdo2xtVY/ndKVSBMkxbmjpHti4LIK86X8ur+DvbwZf3Qvghfd3Rp0jbuLvoSuP5d/XHJ/qZvRyybETeem7J3Wt0NmXrPzpaV23PYHop48aT0nxyOA9uXEeB/fIVbN49fsn9Uh7C2cXpdML+eUFh8W3MWGIKTAWkVycoPgRVX0yPk0yxpjgxrozU3zcXeZ5woh8Rhbk9coJ/Ok5h/LQld29GdMPGMI6dyaIuuY2Fm3sHhimqmzb10hdSzut7T1zdOtb2unoVAblCpMLBzG5cBBzPubk3dU2eQXG7qC6HdVNrN7eu5fj0aVb+frD79Da0UnR0IGMHjyAPXUtVDd09xiHw5NG8dmjx/Pq90/ivCPH8cHOWprcXtZYBhm+606BB1Db0vPrc+H6PeypbwnrBwDA3oZWBuZmsau2mS17G+noVDo7NWhg/P62Go697VW27O3ettTPSoeBzF+4uWs6vUB21TRzzSMr+Pc728Ku1yTWiVMLOebA9LsqICJRjYnoK/wGqHHMf1B1BicePGYIZdd/kqljBof93PuvODamK1/RimVWCgHuBT5U1d/Fr0nGGBOcZ7o3T2B81KQRzPBzAs3Kkh4rPk0rGsKWfY00tXbw4KItXPr3xV1BWm1TO3VuL2V1Uyut7Z2cd/dCXv1gd1eqw6BcGJCTzf+uK+Wij090ntfcvdBHpTsN229fXsfXH36nV3vmL9zMS2ucqd6cwDiPXbXNXfv1rNLnbenmfZx915s0tXanFqzf7Qy8O7hwCAePGcKkkQXUu3UcPmEYm/Y0sMTPbBDh8M6TrmxSvv7QO2zd20h9S3tXT7InfcObqvZIJensVPY1tHb1sFfVt3DmHxZw56vraO9U8rKz2F7d1DVw0OPtjXuorGvp+mExeEAOm/bUd/W0B9Pc1sH6yvqQC6/srHFSTnbX2oJyJnUG5TnZrEVuapLnTJWVFXuX7aNfPS7mOmKZQS3UU71zpkMN1Eu2WHqMTwC+BHxKRFa6/86KU7uMMSagWZNHcsLBo7pGS9987qE8+JVjQzwLphcNQdWZ0WHd7jo6la6ZJLzzXfc3tLGxqp73ttXw7Hs7uoLnQbndJ/Ch7oCb2qa2rsDZ02O8u7aZytqWHukDFfsaWe/OJAFQNHQAowcPYGNV92Nb/PQYL1xfxZodtV35w+BM81Y0dADD3BxCzypTAN/61FQmFw7iS/OX8uHOXtltftU0tvHZe95m3e46VmzZz0h34ZX3qjp4cc0u3txQxbtb93f1JPsL4J9ZuYPjb3+tK42jpqmNjk7l4EKnh2hvfQvrK+t5c70zEG/6AUNo61C27+85AM8z24YneJ1WNJid1c0sWL+HWb98tVeuord1u+vo6NSQKRKegDhUoG1MIh02YRh3XnQEt3/GSRc4aPQgvnbyZP7+5ZKo61z4w0+y7KY5HD9lVNR1xHPAXKAFPkqKR4Rdx58vPbpr2e5kiGVWioWqKqp6uKoe6f57Pp6NM8YYf8YOy+eRq46j0J2+J9zVoaa5l0TX7q7rCkg9aQHbvAPjxtauXOTlW/Z39UAW5HgFxvluYNwjx9gJsvfUt9Da0dnViwvw+tpKgK7ZMcYMGciowXldQdrYYQPZsreRto5Orn1kBa9/5JQvd1MKqhtbu+pasXV/15RN4ExH53HEhGE8dvVxtLZ38soH4S05vax8H+9s2c+rH+5mzY4aTnLzKTdWO73Uu2qaWVa+nyyBvOwsNu+p71XHI0u20Klw12vrASeNAmCKe+m0fG8jqt2r/Z0+o4icLOF3r6ztUc//t3fm8XFV1x3/ntHMaBZppNG+2pIXLMkGbwJsDI7ZbQIBUkJNGoILadokpGvaQrbStGmTtiGhTRra9EObprSUJmkCSdoADYaQgDEG22Bjgw3yLtuyZK3WfvvHWzSSRtbisUaaOd/PZz568+a9N7/77rtPZ84795x9Jx3D2DqXi0oi9A8ant7dyKCB/3xldJjE4KDh3146wHN7raqH3X2D9PRb2l9paB6V1cLpp5MdPTy75wSPxzmmokwHty6vcPMgiwj3b6ilqmDqE/0qoiH3vghjG6YT4Vz8uO49eYyvH6+gx0cur3aXb7iwlN+79oJzUDM5tPKdoihpw9y8EH6vh72NbW44gOM1jE0d1tLZ66Y8O9xyxl0Ox0zyzvR68Gd4hmWlaOnqo6d/gCY7G4OTlQHgZ3tOUJUf4sHbl/Fb75lPRTQ4bLLgJdV59PQP8vfP7ufHrx/jCz/aTf/AoOtFbrYNzZPtPRw41cXKuUMeF8dj7MsQCrIyKcoOsKg4e8Lp43bbnuXNe07S0tXH8jlRsjK9HO+y/qsda+3mtYMtLCqJMK8w7J67d5s66ejpZ//JDrY2tFCVH+LZvSd540irmwd5vu0xfsc2eJ046GWVUT5x5QJ+sP2oG/ZhjHE9xo22YVxj/5hxPM1P7jg6ytDd/NYJPvuDN3jQzk4C0N7dz5HTZ7jt4Rf59y0Hh20fm0f6H57fzxee3D0qrlxRUomxwhXi2a2JKCM9EaPaKcgUz0b+bKJzQ08CNYwVRUkbvBkeFpdF+MnrjXTZMbuO9/BwS5d7g27p6mNvYxsBn3WLdLy9saEUIkIk6B3mMQYrnKLZLk/teE17+gfY8k4zay8opLogzH0bavB4ZJhhvPHiOWRnevnqM2+R4RHeberkyZ1H3bAFJ1zDybMcaxgXZmWS6fVQkhNw4xNXVkV59UDLqBjeeOy2U9+9bGd/qC2NuDmgwTJS95/ooKYkm+qCMO82ddI3YMVg//mPdvP4K4fI8Ah/d8cKV6PT9rn5IbweGRWXnBP08eHVVs5TJ+TjRHsP7d2Wl33IY2wZxgdOdZEd8NLe3c/PbG+6wyMvNLjp8xzazvTxlv2DZtuIHwixoRRO/PRkMl8oymxnIoZrIqo0j7z7/MbaeYD1hOz+G2rP/QvOA2oYK4qSVly1qGhYvK4TF3yo5QzV9iNMJ5Ti6tpigr4M11sZaxiDFWfsZKVwvB97G9vdWFzHy/vawdOc6Rvg8gXDUz7lxxifFdGgO6HvjksqWViUxdd/to8221BstkMpXj3Qgj/D4+ZXBWuyTmVeiLKcoRzIF1dF6ejpZ29jO4eau3jh7eFFNrp6+13P6+4Rscg1pdnDjPaGU50cbe2mKj9MdUGYg81dvHmsjfbufn688xjf23aEKxcVUVcWwesRGlu7XY9xQVYmuSHfqLjknKCP3JCVZs85T/tiYrCdYiexs9g3LCnB7/Ww/dBpvvNiA1f9zWYu+eIzvLCvid+9+gJWzcvjCjsMpK27383eMbJoi+MxPtXZwzF7eaSxrSjK1BnLqL69vpKGL703YUVRzgdqGCuKklZcU1fsLmdlet0Y44amTuYXZhHyZ3DANgQXl0W4unaoHGpmxvBjZQd9blaKhXb2hdgJb47n+Jf7mvCIVao6lsIY4zMa9nPPFdWsnBtl02XVvPeiUvbHeFlbbMN4y7vNLC6PEPANF/MnN9XxhzFlY+vt1FfbDjTz4NNvcfe3t9LS2cufPrmLN460css3fsHvPPYabd19HGzuYqkds1wRDRIJ+IYZxoftCXJVBSHmF2bRP2j4/qtHAGjv6aepo4dfvbiSDI9QlJ1JY1s3TR29iFiPS3NDfteD7JAT9JHhEaIhv2v0O4axP8PjerqjIT/FEUvLBcXZLCrOZvfRNh75RQMDxrBuUSE3LS1j05oqHvvoaneSTtuZvpiJfN3uZD4Y+jFkjPXyesSN6VaUVCLTTmHpy5i4+zcv7Cc/7OfzNy6e8vdevqCADI+w6bKqKR8jWSSk8p2iKMpsoaYkm/LcIK1n+qgpyeZEezcHT3XxTlMnH7x0DruPtvGiHfNaU5LNdXUl/GjnMWD0JL9IwEtLZy/tPf1cUJzFy+82uxX5ADfW+IV9TVxYkUtOcLiXpMCeJOP1CGF/BlmZXr73scsAuKa2mK8987a77enOPhqaOtl+6DR/vL5mVLuuWDi8gFJFNEhxJJOtDS28cbSV3v5B/uapvTy65SDfefEA/YOGlq4+3rT13raygh2HTlNbGrG1Wd7saMhHix3GUZUfpiQngAj8+5aDRAJegv4MBgatZPwAxTkBjrd1E/JnEA358WZ4XG+6gwhkB7zu8R2P8Y5Dp8kP+8kN+dh/spOQPwOPR6iIhjje1sO8wjB1pRGe3HmUrt4B7t9Qw2++Z/7wPomZFPn2iQ5ygj5az/Sx/eBp2ov62Xeig+Pt3RRlZ7o/itYvKeHld5vp6B3xy0dRZjkfW7eAvkHDnavnjr+xjd/rYdvnrh21/q/WBimct2RCxyiKBNj/F7MzUZl6jBVFSStEhN9aN58PXjqHoohlHP10VyMA1y8uITfkcyfiXViey4KiLG5ZVsZVNUWjjhUJ+tw0b9UFlrc5dsJbc2cvZ3oH2HG4lTVx0ic5XlkrpGC40b24LEKpbYTOKwjT3NXL9149jEfg1uXlE2pn/dw8frm/yQ1jeGzrIbIDXoK+DAqy/Jxs7+G5t6xMDtfXFbO0Isdtp6Otvmqo6EJVQZjiSID6uVF6BwZZWpnLV29fxkMbl7nFVUoiAY61dnOqo5d8O+1bbsj667e3ycr0urHQeWE/zZ29GGN4/u0m1iwocH9AhO3Z+hVRK0RkXkEWdWURNz48XsonJ41eq+0xdsIvvvCj3dzw0M/5+KOvcrqrj8VlEXef+zbU8NL9V5Pln1n5VBXlXAn6M/jj9TVkesf+0eeMt/EyRRSFPG6KzISQyLxwCUQNY0VR0o47V83l0zfUUpQd4GRbD0/tbqS2NEJlXsjN4VuZF3TTHn1t43Ie2XTxqOPMLwi7k+KiIR+LyyJuGeTsTC/Nnb3saWxjYNC4oQqx5LrhBKPj7USE968o56LyHIojAVo6e/n+q0e4YmEhJXaBk/Gor4rS1NHrhgsMDBquqS3mxU9fzVduXwZYJZTn5IUoigT44b2Xc8clcwDctl9abRnGeWG/+w90w5JSwComctmCAtbExE6X5AQ43trN4ZYzbuGCXHu/wuxMCrL8wzznjmG8p7Gdpo4erlhY4Hp9nTRWNSURcoI+KqJB6myD1u/1xK2KFQla++w70UF7dz91ZRE+d2MdtaURPlBfSabX+rfn7BvweSjPDSakqIKizBacH7I5QR8Pf2gln7+xjnmFE69Kl0hm2shTw1hRlLSlMDuT9p5+tja0cP1iK/bY8W6umDN+AvpLqoe8wDlBn2tsZXiE6sIwpzp73YltdaWRUft7PEKeHToQj09dt4gffGIN0bCPvY3tHDl9ZljM83jUx5TYfe9FljF7ZU0RWZleau1sD00dvXE9r9fWFrO+yud6iKryh3Il33hRKZV5Qa6qKR61X0kkQGfvALuPtbG43Gpz1P6xEQ37KM0JxjGM+9zJgVcsLHS9viG/5eW65/Jqnvn99+DN8Ljp25ZW5MT1ggV9GXg9wqt2WewFhVncuWouj2y6mL98/4XctLQMwPUYz8kLTTgPtqKkCovLIjxwUx1fvX0pRZEAd8fkDU4mD39oJZdUJ7c0uMYYK0oaISK5wD8BS7AeZN1tjHkxuaqSR1FMIvw7V1kxeHm2kbo8jod3JMvn5JJhe2Jzgj4uqrAM4/ywn4KsTI63dbP7aBvZAa8bDjCS+YVhKqKhuJ85Bls05HfLRtfGMbDHorY02431vXtNNYeau9xY4MJsK1vE6a6+YQa0Q1EkwMYaP+W5lu7YogNFkQA//6Or4n6n480eGDRcVG6dQ8fwzwtncvPSMvoHh3IG54X9tHT18ov9TcwvtGKYHa+vE0rh93pcD3Z2wMcty8q4bP7wDB8OVho9H7vsktILiod7wT66dh5vH2+nviqPrEzvsKqBipIuiAib1iTXGI5XfGT9khLWLylJgpoh1DBWlPTiIeB/jTG3iYgfSGurwJn8dtfqueTHxPsCLJ+Axzic6WVJWYQdh1vt9GOWIZiflUle2M+bx9rYfayNutLImF7Jb324Hq/n7A/voqGhtG5OXt+J4M3wcOvycrIyvSytzOX7H1/jfiYiLCrOZsu7zVx8lvKsAV8Gd66aO2FPdUlkKMzjQtuDnhu09OeFfPzKyoph20dDfgYGDdsOtHC17Z12PMZOKMVIvrZx+Vk1RAJWGEtO0Dcs8wdYmS1+eO/lANy5eu6wWGNFUaafmfbARg1jRZkCVff9eNxtGr703mlQMnFEJAdYC2wCMMb0Ar1n2yfVWbuwkIc/tJJrYoy+y+bns+Pw6Ql7Zi+uynMN44KsTMJ+a2Jbftia3Ha6q4+Nl1SOuf9E8nk6Htfy3KBrNE6UL9564Zif1VdFOdTc5VanG4s/u2ViM9FhyGOcE/RRmWd5m50YaiekIhYnpru9u99NeRcZMflusjj7LyjKOmuYRLzsHoqipDdqGCtK+lANnAT+WUSWAtuA3zHGuMlyReSjwEcBiouL2bx581kP2NHRMe42M50A8MLP9wxbt6kafvnC86O2jdfeRZ5B3r/Qx65tLyIirJ/rIS/QxunjbfQPGvoHB/C0HmXz5pNT1nj8iDXBr9DXm9DzvdxnqKv38Pzzz8X9fCr92ztgPR6tCA3w3HPWcRtOWVkkWk8cGXUeDp/sd5d7TjSwefNhGg9Z7W09dWJK7R04Y2UVCQ+0T2r/VLieFWW24DwJK8mJH2aWLNQwVpT0wQusAD5pjNkiIg8B9wGfczYwxvwj8I8A9fX1Zt26dWc94ObNmxlvm1RirPZ+IGbZ+bixtZvgc/tp7uzl4zfVuaEaU8HsOcG3Xt/K5UuqWbdu0fg7JIip9m/9W7/kfcvKWLe6CoDiY218eevPWbF4EetWDc+nmnf4NA9u+wUAt169muqCMB07j/Ivu15jYVUl69bVTfr7Hz+yjV2nGlm7dCHrrpg34f3S7XpWlGRybV0x3/jgCq5bPHoSbzJRw1hR0ofDwGFjzBb7/XexDGPlPFCSE+CB9029clQsRXbltyXlsyMe9rt2kRKHeYVh3r+8nLUjipDAUCiF3+uh0p6g6ISLTDmUIjAUSqEoysxERNxsOTMJNYwVJU0wxjSKyCERWWSM2QtcDexOti5lfOpKIzz6kUtZPW90kZDZQKY3gwd/dVnczxzDeF5BGK+dW3Uoj/HUKtE5+zsxy4qiKBNF8xgrSnrxSeBREdkJLAP+Isl6lAkgIqxZUJCSRShCfi8Bn2eYERuxy0WH/FPz3ayal8eViwopjUysEMp0IyLrRWSviOwTkVFPbURkjog8KyKvichOEZmdtXUVZRaiHmNFSSOMMduB+mTrUJRY7t9QO6yKXVV+mHuvXMC1dVOLPbyqpjhu8ZGZgIhkAN8ArsUKb9oqIk8YY2Kf3nwWeNwY800RqQN+AlRNu1hFmWF87sY61iw4v0/O1DBWFEVRkspdl1UNe+/xCJ+6fvomGU4zlwD7jDHvAIjIY8DNDA9rMoATUJ4DHJ1WhYoyQ7lnGir0qWGsKIqiKNNHOXAo5v1h4NIR2zwAPCUinwTCwDVjHWw2p1icSVpA9YxHuuhRw1hRFEVRZhZ3AP9ijPmKiKwGviMiS4wxgyM3nM0pFmeSFlA945EuenTynaIoiqJMH0eA2FKIFfa6WO4BHgcwxryIVYemYFrUKUqao4axoiiKokwfW4GFIlItIn5gI/DEiG0OYqVTRERqsQzjqZdOVBRlwqhhrCiKoijThDGmH7gX+CnwJlb2iV0i8gUReZ+92R8AvyEiO4D/ADYZY0xyFCtKenFOMcYish54CMgA/skY86WEqFIURVGUFMUY8xOsFGyx6z4fs7wbWDPduhRFOQePcUwuxg1AHXCHnW9RURRFURRFUWYd5xJK4eZiNMb0Ak4uRkVRFEVRFEWZdchUw5ZE5DZgvTHmI/b7O4FLjTH3jtjOzbEILAL2jnPoAqBpSqJmB9q+2U2i2zfXGFOYwOMlDBE5CRwYZ7NU7++RaHtTm/HaO2PHK8zKMTuTtIDqGY/ZqGfSY/a85zGOzbE4EUTkFWNMypas1fbNblK9fbFM5GaSTucDtL2pzmxv72wbszNJC6ie8UgXPecSSjGRXIyKoiiKoiiKMis4F8N4IrkYFUVRFEVRFGVWMOVQCmNMv4g4uRgzgEeMMbsSoGnCYRezFG3f7CbV2zdZ0u18aHtTm3Ro70xq40zSAqpnPNJCz5Qn3ymKoiiKoihKKqGV7xRFURRFURQFNYwVRVEURVEUBZhBhrGIrBeRvSKyT0TuS7aeRCMiDSLyuohsF5FXkq0nEYjIIyJyQkTeiFmXJyJPi8jb9t9oMjVOlTHa9oCIHLH7cLuI3JBMjckm1ccsxB+3qXKNw+TGsFj8rd3fO0VkRfKUT43JjmsRud9u714RuT45qkeTqH4Tkbvs7d8Wkbti1q+0r/t99r4yBT0fEJFdIjIoIvUjto97Xse6p9iT/LfY6//TnvA/WT1/LSJ77HPw3yKSOx16xtDyZ7aO7SLylIiU2euT0lcxn/2BiBgRKUimHpnCmEzUtQOAMSbpL6zJe/uBeYAf2AHUJVtXgtvYABQkW0eC27QWWAG8EbPur4D77OX7gC8nW2cC2/YA8Klka5sJr3QYs3Y7R43bVLnGbf0THsPADcD/AAKsArYkW3+C2ht3XAN19nWdCVTb13tGstuQqH4D8oB37L9Rezlqf/ayva3Y+26Ygp5arKJem4H68c7r2e4pwOPARnv5YeBjU9BzHeC1l78cc37Oq54xtERiln8beDiZfWWvr8RKpnAA+56XxGvnASYxJhN57RhjZozHWMtLz0KMMc8DzSNW3wx8217+NnDLtIpKEGO0TRkincdsSlzjMOkxfDPwr8biJSBXREqnR2limOS4vhl4zBjTY4x5F9iHdd0nnQT12/XA08aYZmNMC/A0sN7+LGKMeclY1sS/Ms41Hk+PMeZNY0y8Srdjnde49xTb43gV8N04bZuMnqeMMf3225ewai+cdz1jaGmLeRsGnCwISekrm68CfxSjJdl64nHerx2YOaEU5cChmPeH7XWphAGeEpFtYpXJTlWKjTHH7OVGoDiZYs4D99qPlB6RWfwIPQGkw5iF+OM21a/xsdqXyn0eb1zPtvZOtt/Otv5wnPWJYrJ68oHTMUZtIvTcjeXNTJoeEfmiiBwCfg34/BS1JKSvRORm4IgxZseIj5J57UxmTCa0r2aKYZwOXG6MWQFsAD4hImuTLeh8Y/9iTKV8gN8E5gPLgGPAV5IrR5kGzjpuU/AaH0aqt88m5cZ1mvTblBBtBxAoAAACkUlEQVSRzwD9wKPJ1GGM+YwxptLWcW+ydIhICPg0Q8b5TCCpY3KmGMYpX17aGHPE/nsC+G9myCO588Bx5/Gq/fdEkvUkDGPMcWPMgDFmEPgWqduHEyHlxyyMOW5T9hq3Gat9KdnnZxnXs629k+23s62viLM+UUxWzymsR/jeEesnjYhsAm4Efs3+8ZBUPTaPAr8yRS2J6Kv5WPG6O0SkwT7GqyJSkiQ9UxmTCe2rmWIYp3R5aREJi0i2s4w1CWDUjNAU4QnAmaF6F/DDJGpJKCPiKW8ldftwIqT0mIWzjtuUvcZtxmrfE8CH7Znqq4DWmEf3s5azjOsngI0ikiki1cBCrIlFM5XJ9ttPgetEJGo/qr4O+Kn9WZuIrLJjND9MYq/xsc5r3HuKbcA+C9wWp20TRkTWY8XQvs8Y05VMPSKyMObtzcCeGC3T2lfGmNeNMUXGmCpjTBVWuMEKY0xjMvTY52eyYzKxfTXWrLzpfmHNfnwLa2bhZ5KtJ8Ftm4c1S3IHsCtV2gf8B9Zjjj6swXQPVkzP/wFvA88AecnWmcC2fQd4HdhpD9DSZOtM8jlK2TFrty/uuE2Va9xuy4THMNYs82/Y/f06MZkGZstrsuMa+Izd3r2MM7t+BrRj0v2GFWu7z379esz6eixjZD/wdewquZPUc6u93AMcxzKcznpex7qn2GPxZVvnfwGZU9CzDysOdbv9eng69Iyh5Xv2+d0JPAmUJ7OvRnzewFBWimRdO5Mek4m6dowxWhJaURRFURRFUWDmhFIoiqIoiqIoSlJRw1hRFEVRFEVRUMNYURRFURRFUQA1jBVFURRFURQFUMNYURRFURRFUQA1jBVFURRFURQFUMNYURRFURRFUQD4fz8SY00vQceoAAAAAElFTkSuQmCC\n"
                },
                "metadata": {
                  "needs_background": "light"
                }
              }
            ]
          }
        },
        "82170e57a38649ef9a83963fbb203de0": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "69560af419f1475ab14258a656d60681": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_816175c58d1e4f1a9d075625334a60a5",
              "IPY_MODEL_d7d4c7cc6dda46189c6977489efc71b7",
              "IPY_MODEL_df11bb5436274efcba592438647bf15a"
            ],
            "layout": "IPY_MODEL_1826302cbf9043f0911f001a85dc74ec"
          }
        },
        "816175c58d1e4f1a9d075625334a60a5": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_e2e7053c3f76474492116e56709e9cf1",
            "placeholder": "​",
            "style": "IPY_MODEL_62b2f9742d07437a89362ec42cf59e2b",
            "value": "100%"
          }
        },
        "d7d4c7cc6dda46189c6977489efc71b7": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_df1efb6dcb8044bf824808fe8a905315",
            "max": 5000,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_293364819f86406e9706cfba7d44e728",
            "value": 5000
          }
        },
        "df11bb5436274efcba592438647bf15a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_50543448c11f472f9775cba92c864fa9",
            "placeholder": "​",
            "style": "IPY_MODEL_2f87d2ce228a419389f7ae8010ba6185",
            "value": " 5000/5000 [03:02&lt;00:00, 31.72it/s]"
          }
        },
        "1826302cbf9043f0911f001a85dc74ec": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e2e7053c3f76474492116e56709e9cf1": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "62b2f9742d07437a89362ec42cf59e2b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "df1efb6dcb8044bf824808fe8a905315": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "293364819f86406e9706cfba7d44e728": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "50543448c11f472f9775cba92c864fa9": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2f87d2ce228a419389f7ae8010ba6185": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "d4794b4d68944c3e94ae6133e6d5dd26": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_d677bd7504894a49947bd94e88f5f5f3",
              "IPY_MODEL_9b14323dc89349a8b43d636261d9c406",
              "IPY_MODEL_f4ee6312a3f540b8802e0955905ae183"
            ],
            "layout": "IPY_MODEL_06c4b51625954ed99e6b6239a7160e27"
          }
        },
        "d677bd7504894a49947bd94e88f5f5f3": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_1cf58205a6ef46b2b7e6f1a9d1ec810d",
            "placeholder": "​",
            "style": "IPY_MODEL_549e2f92a9e047df96b20279118201e2",
            "value": "100%"
          }
        },
        "9b14323dc89349a8b43d636261d9c406": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_310d161a4e734a708e85cf8dfc413088",
            "max": 407,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_30e427c09c2b471baa0a823eab7f9de1",
            "value": 407
          }
        },
        "f4ee6312a3f540b8802e0955905ae183": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_6600572b484644c3865a54015f8aa0a9",
            "placeholder": "​",
            "style": "IPY_MODEL_5153fd99a12c4362806f9ad33cc26be2",
            "value": " 407/407 [00:03&lt;00:00, 125.79it/s]"
          }
        },
        "06c4b51625954ed99e6b6239a7160e27": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "1cf58205a6ef46b2b7e6f1a9d1ec810d": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "549e2f92a9e047df96b20279118201e2": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "310d161a4e734a708e85cf8dfc413088": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "30e427c09c2b471baa0a823eab7f9de1": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "6600572b484644c3865a54015f8aa0a9": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "5153fd99a12c4362806f9ad33cc26be2": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}