09a_auto_encoders.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "GTY-Msjal7zU"
   },
   "source": [
    "# Generative models - auto-encoders\n",
    "\n",
    "### Author: Philippe Esling (esling@ircam.fr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Z3M5ezRyrjA3"
   },
   "source": [
    "In this course we will cover\n",
    "1. A brief introduction to [generative models](#generative)\n",
    "2. A formal presentation of [Auto-Encoders](#ae) (AEs)\n",
    "3. An explanation of how to [implement AEs](#implement)\n",
    "4. An [application](#application) of AEs for modeling images\n",
    "4. An practical exemple of [convolutional denoising AEs](#denoising) for image data **(exercise)**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"generative\"></a>\n",
    "\n",
    "## Generative models\n",
    "\n",
    "### Supervised refresher\n",
    "\n",
    "Until now, we have mostly discussed models that are developed for _supervised_ and _discriminative_ tasks. To formalize this problem, we have a set of data $\\{\\mathbf{x}_{i}, \\mathbf{y}_{i}\\}_{i\\in[1,n]}$, where the $\\mathbf{x}_{i}$ and $\\mathbf{y}_{i}$ are linked. Therefore, we want to approximate this relation through\n",
    "\\begin{equation}\n",
    "    \\mathbf{\\hat{y}} = \\mathcal{F}_\\mathbf{\\theta}(\\mathbf{x})  \n",
    "\\end{equation}\n",
    "where we train the parameters $\\mathbf{\\theta}$ so that $\\mathbf{\\hat{y}}\\approx\\mathbf{y}$. The existence of a label $\\mathbf{y}$ (\"correct answer\") defines a _supervised_ problem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Going unsupervised\n",
    "\n",
    "In some cases, we might only have a set of data $\\{\\mathbf{x}_{i}\\}_{i\\in[1,n]}$, and still be interested in learning some underlying properties or structure of this set. In that case, the problem is _unsupervised_ as we have to learn without a given answer. \n",
    "\n",
    "Here, we can turn to _generative_ models [[1](#reference1)], which allows to create new data instances based on the observation of existing examples. Although these models are more naturally defined in a _probabilistic way_, we will assume that we have some simple _code_ $\\mathbf{z}$, which allows to control the properties of the generation, and need to learn\n",
    "\\begin{equation}\n",
    "    \\mathbf{\\hat{x}} = \\mathcal{F}_\\mathbf{\\theta}(\\mathbf{z})  \n",
    "\\end{equation}\n",
    "where we still need to learn $\\mathbf{\\theta}$, so that $\\mathbf{\\hat{x}}$ have similar properties to that of the examples in $\\{\\mathbf{x}_{i}\\}_{i\\in[1,n]}$.\n",
    "\n",
    "Now the problem to solve is how we can learn directly from a set of data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Kg83oWxMm92f"
   },
   "source": [
    "<a id=\"ae\"></a>\n",
    "\n",
    "## Auto-encoders\n",
    "\n",
    "One way to understand a set of data is to try to _compress_, or _simplify_ the corresponding dataset. So the idea is to learn simultaneously how to _encode_ our unlabeled input $\\{\\mathbf{x}_{i}\\}_{i\\in[1,n]}$ and to _decode_ the corresponding representation. This idea give rise to the notion of **auto-encoder**. \n",
    "\n",
    "### Architecture \n",
    "\n",
    "The auto-encoder is an unsupervised architecture originally proposed to perform _dimensionality reduction_ [[3](#reference3)]. As its name indicates, we will try to train this model to learn an efficient _encoding_ $\\mathbf{z}$ of unlabeled input data $\\mathbf{x}$. The only way to learn efficient parameters is to also learn a _decoding_ function to _reconstruct_ $\\mathbf{x}$ from $\\mathbf{z}$.\n",
    "\n",
    "<img src=\"images/auto_encoder.png\"/>\n",
    "\n",
    "As shown here, a first model $\\mathcal{E}_\\phi$ _encodes_ the input into a _latent code_ $\\mathbf{z}$ in order to provide a low-dimensional representation of the data. A second model $\\mathcal{D}_\\theta$ designated as the _decoder_ aims to generate outputs from $\\mathbf{z}$ that are as close to the original inputs as possible."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Formal definition\n",
    "\n",
    "The latent code $\\mathbf{z}$ can be seen as a compressed abstract representation, and may be used as an intermediate space for analysis or generation. This helps to govern the distribution of the data through a simpler and higher-level representation, while enhancing the _expressiveness_ of the generative model.\n",
    "The behaviour of an auto-encoder can be formalized as:\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\mathbf{z} &= \\mathcal{E}_\\phi(\\mathbf{x}) \\\\\n",
    "\\mathbf{\\hat{x}} &= \\mathcal{D}_\\theta(\\mathbf{z})  \n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "with the _encoder_ $\\mathcal{E}_\\phi$ and _decoder_ $\\mathcal{D}_\\theta$ functions parameterized respectively by $\\phi$ and $\\theta$. As we can see this defines the reconstruction relationship\n",
    "$$\n",
    "    \\mathbf{\\hat{x}} = \\mathcal{D}_\\theta(\\mathcal{E}_\\phi(\\mathbf{x}))  \n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training\n",
    "\n",
    "The training of an auto-encoder consists in finding the optimal functions of encoding $\\mathcal{E}^*$ and decoding $\\mathcal{D}^*$ by evaluating the \\textit{reconstruction error} $\\mathcal{L}$ between $\\mathbf{x}$ and $\\mathbf{\\hat{x}}$, such that\n",
    "\\begin{equation}\n",
    "    \\mathcal{E}^*, \\mathcal{D}^* = arg\\,min_{ \\phi, \\theta}{\\mathcal{L}}(\\mathbf{x}, \\mathcal{D}_\\theta(\\mathcal{E}_\\phi(\\mathbf{x})))\n",
    "\\end{equation}\n",
    "\n",
    "As the latent space usually has a smaller dimensionality than the input, it acts as an incentive for the network to find the main attributes of variations in the dataset (and also explains its use for _dimensionality reduction_)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Variants and discussion\n",
    "\n",
    "There are several variants of auto-encoders, such as denoising auto-encoders or variational auto-encoders. Each address some downside of the basic AE model. For instance, the deterministic nature of the basic auto-encoder implies a point-wise mapping of the latent space, meaning that not all the latent positions can be leveraged to produce relevant reconstructions. Because of this reason, there is no way to ensure that the latent space could allow a robust generalization and that any random $\\mathbf{z}$ would generate a meaningful output."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "cAxMjUCBUa43"
   },
   "source": [
    "<a id=\"implement\"> </a>\n",
    "\n",
    "## Implementation\n",
    "\n",
    "Here, we discuss how we can implement and train a simple auto-encoder network in Pytorch. As discussed earlier, an AE is composed of two parts, an **encoder** and a **decoder**. The goal of the encoder is to \"compress\" the dataset, representing its principal features with a very small code, while the goal of the decoder is to learn how to reproduce the initial input from this code. Hence, we will first need to use some basic imports and definition to setup our problem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "e1_Y75QXJS6h"
   },
   "source": [
    "### Import Pytorch and other libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "YfIk2es3hJEd"
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torchvision\n",
    "from helper_plot import hdr_plot_style\n",
    "hdr_plot_style();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "iYn4MdZnKCey"
   },
   "source": [
    "### Load the dataset\n",
    "\n",
    "To start with a pragmatic and simple to understand example, we  will try to train the basic AE using the Fashion MNIST dataset. This dataset contains images of size 28x28 pixels, with different pieces of clothing. The following code allows to load (and eventually download) the dataset, by using the `torchvision.datasets` module. We also plot some randomly selected test examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "YZm503-I_tji"
   },
   "outputs": [],
   "source": [
    "import matplotlib\n",
    "dataset_dir = './data'\n",
    "# Going to use 80%/20% split for train/valid\n",
    "valid_ratio = 0.2\n",
    "# Load the dataset for the training/validation sets\n",
    "train_valid_dataset = torchvision.datasets.FashionMNIST(root=dataset_dir, train=True, transform=torchvision.transforms.ToTensor(), download=True)\n",
    "# Split it into training and validation sets\n",
    "nb_train = int((1.0 - valid_ratio) * len(train_valid_dataset))\n",
    "nb_valid =  int(valid_ratio * len(train_valid_dataset))\n",
    "train_dataset, valid_dataset = torch.utils.data.dataset.random_split(train_valid_dataset, [nb_train, nb_valid])\n",
    "# Load the test set\n",
    "test_dataset = torchvision.datasets.FashionMNIST(root=dataset_dir, transform=torchvision.transforms.ToTensor(),train=False)\n",
    "# Prepare \n",
    "num_threads = 4     # Loading the dataset is using 4 CPU threads\n",
    "batch_size  = 128   # Using minibatches of 128 samples\n",
    "train_loader = torch.utils.data.DataLoader(dataset=train_dataset, batch_size=batch_size, num_workers=num_threads)\n",
    "valid_loader = torch.utils.data.DataLoader(dataset=valid_dataset, batch_size=batch_size, shuffle=False, num_workers=num_threads)\n",
    "test_loader = torch.utils.data.DataLoader(dataset=test_dataset,batch_size=batch_size,shuffle=False,num_workers=num_threads)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we are going to check the properties of our different sets and also plot some random examples, in order to better understand what type of data we will be working with."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The train set contains 48000 images, in 375 batches\n",
      "The validation set contains 12000 images, in 94 batches\n",
      "The test set contains 10000 images, in 79 batches\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABG0AAACBCAYAAAB6p0ctAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAABDp0lEQVR4nO2debxO5fr/LzaKEt/a5mETUUIkUlQkc0oaCbuiQYaDzonoNKtvpEinktKcNJzU+TUYShOpHEMyhETI2KjIsO3fH732/f3cH3ut/ext7/2s59mf9+vl9brWvu9nPetZ97iW63Ndxcws04QQQgghhBBCCCFEpCge7wsQQgghhBBCCCGEEIeilzZCCCGEEEIIIYQQEUQvbYQQQgghhBBCCCEiiF7aCCGEEEIIIYQQQkQQvbQRQgghhBBCCCGEiCB6aSOEEEIIIYQQQggRQfTSJg/07dvXWrZsGe/LEEIIIYQQQgghRBITqZc2TZo0sfnz59vo0aNjqp+WlmZz584t4Ks6lPPPP98WLFhQ6N8rDiW3fQZ5+umnbe7cufbll18WwJUJIYQQhY/WxcRHbRg/EuVZRAhRtCgRS6UffvjBvvnmGzMzq1+/vrOPO+44a9y4cb5dzJIlS6xnz57Wu3fvmOrv37/fvv/++8P6zssvv9zWrl1r//3vf2Oq3759e5szZ85hnycZufDCC23YsGGWkZFhf/75p73zzju2efNmmzFjRoF9Z277DHL11VebmdmUKVPy+7IiT2GNaZEYnHLKKfbYY49Z5cqVbcOGDZaZmWnFixe3GTNm2IQJE+J9eSKE7t2725AhQ6xYsWJWvHhxe+yxx6xx48Y2atSofP0erXF5Q+ti4qM2TE7C2jXqzyIiHK2L0UDPGvlLTC9t3n77bbv22mvN7K9FIMt++umnC+7KYuCHH36w9PT0wzrHkUceaUceeWTM9fv06WPXXXfdYZ8n2ahbt65dccUV1rFjR9u7d6+VLl3apk2bZq+++mq8L01kQ1THtIgPS5cudZvUMWPGuL/fdddd1rZtW/0vYkTp3r27XXjhhda5c2fbu3evlShRwm655RZr3rx5vn9XUV/j8oLWxcRHbZic5Ge7xuNZRASjdTE66Fkjf4nppU1mZmau/h5G79697dprr7X9+/fb77//bj/++KONGTPG1q1b5+qUKlXK/vWvf1mDBg2sdOnS9sgjj9gLL7zgylNSUuzNN9+0o446yvbs2WNdunQ55HtatWpl99xzj/3+++/Wu3dvGz9+vNWvX982bdpk6enptm/fPpsyZYq1bt3a9uzZY7/++quZmW3dutV69uyZ7bWfdNJJtm7dOvvzzz+9v8d6niuvvNIGDhxoBw8etAMHDtg999zjvHbS09Nt5MiRduutt9qll15qtWvXtoyMDJs3b56NHj3a9u3bl+t7XZiceOKJNm3aNNu7d6+Zme3Zs8fuvvtuK1++vJn99VZ1/PjxVqlSJStRooTt2rXLRowYYWvWrHHnmDVrli1btsxSU1MtLS3NSpUqZZMnT7Znn33W+66bb77ZunfvbhkZGfbNN9/Yww8/7JXH8l1FnVjGdCxjyCy8X2d5cLz99tvuZUDWeOnfv7/NmzfPfV9Oc0OxYsXsb3/7m3Xr1s1Klixpe/bssVGjRnn/wxHrNYvYeO2116xDhw42d+7cmMZVSkqK3XfffXbOOefYwYMH7fPPP7dff/3V2rdvbwMGDLClS5fG8dckH9ddd5316NHDzbsHDhywu+++2/vfxLDxaRbbfJnbtVL8hdbFxEdtmJzk1K5m0X0WEeFoXYwOetbIfzJz+le/fn1nT5kyJdu/x/KvbNmyma+99lpmiRIl3N8mTZqUWadOHXeclpaWuWXLlszWrVtnmllm8eLFM19++eXMChUqZHtOvJ7s/v3nP//JnDlzZmarVq0yzSyzRo0amcWLF3fl6enpriynf5MmTcqsVKlStmU5nefCCy/MnDRpUmbJkiXdvXj55Ze9z6Snp2du3rzZ+9vFF1+c+eSTT+bqPsfjX9myZTMfeeSRzJo1awbWOeKII5xdqlSpzEmTJnnlaWlpmVu3bs087bTTXNtPnz49s2LFiq7Orbfemjl48GB33LBhw8xZs2Zljh49OlfflZs+lIz/cjOmw8ZQLP06LS3tkPbh8RLL3PDPf/4zc9iwYe44NTU1c8aMGdnODTmNe/079B+3U5kyZTJffPHFzMaNG7u/5TSubr/99sy+ffu64759+2b+9ttvmWlpaXH/fcn2LzU1NXPChAmhdWIZn7G0q1nu1kr9M3e/tS4m9j+1YXL+y6ldo/wson/B/7QuRuufnjXy919MnjZZGrSwv48aNcrat29/SJ3Zs2fbvffea2Zmf/zxh+3YscOaNWtmX331le3Zs8cGDx58yGeeeOIJ+/TTT83M7ODBgzZr1iyrW7eu7dixI5bL9ShTpoz16tXLtm3bZmZmGzduzPU5zP5665qSkuLOk1suv/xy6927tx08eNDMzHbt2mXXXHONTZkyxXsDOHHiRO/49ddft/bt21tqaqrt3LkzT99dGOzatcuGDh1qXbt2tQsvvNBKlixp69evt3//+99m9tf/WAwbNsz1kT179tgxxxxzyHkeeeQRW7hwoZn91fYzZ860unXr2vbt2y0lJcUaNGhgvXr1cvW//vpre+aZZ6x27drub7F+V1EmljGdRdgYirVf50ROc0OxYsWsUaNGdtlll7m/7dy50wYOHGgDBw60O+64I+ZrFsH069fPzjvvPCtevLjVq1fPvvjiC1u7dq2ZxTau6tWrZ3feeac7fu655+zSSy8tvB9QhDjqqKNyXBNjGZ+aLwsOrYuJj9owOcmpXc2i+SwiwtG6GC30rJG/xPTSJhbuvfde93ImiIMHD9qQIUPs3HPPteuvv96OPfZY2717t40bN84yMjJcPXYrysjIsOLF85boat26dXl+0YIMGDDAHnvssTx/fvfu3a6z4d8OHDjg/W3JkiWHfHbZsmVWo0aNSL+0MfvLBfHNN990x927d7fLL7/cpk+fbg888IAtWbLE2rdvbwcPHrSSJUtmq2lkV7qMjAwrVqyYmZmlpqba+vXrD/nM0qVLvY1NrN8lYiNsDMXar3Mip7mhYsWK1qJFi2xjqyxfvjxX1yyCeeqpp7yYNu3atbOnn37aLr/88hzHVWpqqm3YsOGQcy5btqxQrr2o8dNPP1nVqlVD68QyPjVfFixaFxMftWFyEtauZtF8FhHhaF1MXPSskTOFmvK7dOnSduDAAZs5c6ZNmDDBbrvtNvv888+tb9++hXkZuaZkyZJWr169w3r4KFOmzCGTfZkyZSwlJcX7W5MmTQ75bOPGjd3/dkeVm266yUqU8N8Bzpo1y1q3bm1mf739njp1qht0+/fvd3rTWNm5c6elpaUd8vdTTjnFO86P7xKxEUu/jiX2VU5zw/bt2+3DDz+0tm3bHvJv0KBB+fujhOP999+377//3lJTU3McVzt37rSaNWseco5GjRoV2vUWJXbt2mVVqlSxsmXLHlJ23HHHmVls41PzZcGhdTHxURsmJzm1q0hMtC4mJ3rW+ItCfWnTr18/69+/v/e3008/3Xbv3l2Yl+GxZcsWa9q0qTsuU6aMdevWzavTq1cvmzZt2mGdZ/r06TZx4kQrWbKkmZkdffTRNmXKlEO8d6688kpr1aqVO7744ottz549tmvXrtz/uELk5JNPttGjR7tBlZKSYnfccYd98MEHZmZWtmxZK126tJn95XZ41113eb8zFjIyMmzlypXewGnYsKGNHDnSq5cf3yViI5Z+vW3bNm/z2ahRIxs6dKh3npzmhszMTFuwYIH9/e9/9ybpWrVqeS7lIn8pX768nXLKKfbzzz/HNK7Wrl1rV155pTvu27evNWzYsFCvuShx55132vPPP29VqlRxf+vUqZMLYBrL+Ix1voxlrRQ+WhcTH7VhcpJTuxY2ml/zD62LyYeeNf4iZnnUww8/bI0aNbL69evb3Llzbc6cOZ4bfSzs27fPypcvb2+//bYdeeSRVqpUKZs1a5ZzRaxataq9+OKLVrVqVduwYYO98MIL1rFjRxs5cqTt3r3brrrqKlu2bJkNGTLELrroIjMzdz1mf0WKfumll8zM7LbbbrO2bdt65VmfRWbNmmVdu3a1999/30qXLm0//fSTTZ482avTqVOnHKOB53SeN99808qWLWsfffSRHTx40DIyMuyee+45RIs3YsQI69Onj40bN85SUlJswYIF9o9//CNX9zkebNq0yZYvX26zZ8+24sWLW/Hixe2ll16yN954w8zMxo0bZ2+99ZaVKFHCUlJS7LnnnrPvvvvO3nnnHevevbulpqYe0vZ169a1m2++2X7//Xfr2LGj/fLLLzZmzBgbOXKkffbZZ3bw4EHbsmWLDRo0yJ555hnbt2+fjRs3Lsfv2rdvn/Xs2dOlbsc+8tZbb9lDDz0Ut/tYmOQ0pmMZQ7H0671799oHH3xg8+bNs4yMDPv222/tlVdesQcffNC1a05zg5nZ448/bunp6fb+++9b8eLFLSUlxVauXOnJMmMd98InK/J+5cqV7bzzzjMzc+N4+PDhlpGREdO4uvvuu23s2LE2ZMgQ279/v7377rv21FNPxfnXJS9Lly61O++80yZPnmypqam2b98+W7BggV1zzTVmFtv4jKVdzWJbK4WP1sXER22YnIS1a1SfRURsaF2MFnrWyD+K2V8RiUUATZs2taZNm9rUqVML/LvS09Nt7dq1uQqqJIQQUea+++6zhx9+2LZs2RLvSxFCCCGEECLhyLdAxMnK4sWLbfHixfG+DCGESDgqVqxop556ql7YCCGEEEIIkUf00iYiXHXVVTZixAjbs2ePbd++3S6++GL7448/4n1ZQgiRKy666CIbMmSIZWZm2t69eyMRvE0IIYQQQohERfIoIYQQQgghhBBCiAhSqNmjhBBCCCGEEEIIIURs6KWNEEIIIYQQQgghRATJVUyb7du324YNGwrqWkQAaWlpVrFixXw5l9owfiR7O1auXDmmevv37/eOS5T4v2moWLFiXtnBgwedvX379sO4uvwh2duwqJCM7ViyZElnly1b1isrU6aMs4888khnFy/u/7/NgQMHnP3bb795ZTt37syX68wvkrENiyJqx8QnUduQ9xuZmdlHi6hWrZp3jHMofibo82Zme/bs8Y63bt0a83UWFonajrGCe00zsxNOOMHZvBYi2K67d+/2ytavX58/F5dPJGob4v7FzCw1NdXZu3btcjaPMWy3jIwMZ+Ozg1n4c0ZKSoqzjzjiCGfjfsjM7Mcffwz+AflMUDvm6qXNhg0brHnz5vl2UXzjkLDJLxHA33a4v+XLL7883Mtx5HcbithJ9na85ZZbAstwMuWXL+XLl3d2qVKlvLK9e/c6e+LEiYd5hYdPsrdhUSHK7cjrIh7zRgSpUqWKs9u0aeOVNW3a1NknnXSSs0uXLu3Vw03JrFmzvLKnnnoq5KoLnyi3oYgdtWPik0htiPMpPqyZHfqQlsW9997rHderV8/Z+J9QPD/jvmflypVe2X333Zfj9TEF/VyUSO2YF4477jjv+J133nE2/scGv8DBfShnE+7Xr1+23xXrC8H8JlHbsGrVqt7x1Vdf7eyPPvrI2TxG8SXLr7/+6uw///zTq4cvgfAzZmbHHHOMs+vUqePsHTt2ePWeffbZ4B+QzwS1Y1yzR+WmE+OA6tq1q7NHjBjh1WvQoIGzsaF//vlnr96xxx7r7CZNmnhllSpVcvaUKVOc/eijj3r1lixZEni9if7SSYhYqFWrlrNvvPFGr2zRokXOxv9V6tixo1dv+fLlzq5du7ZX1rBhQ2c/99xzzubxLEQigxu8MG8z/p+X119/3dm4KcFNiFnw/yLy/zziQ0znzp29sttuu83Z33zzjbM7dOiQ7bmz+96wl05CCJHf8IsZfJES9JLGzH94u/LKK72y7777ztk4p7G3ANK+fXvveOrUqc7etm2bs8OeHcJ+i/g/8D7hPRo7dqxXr3Hjxs7GNkVPKjP/PxMvuOACr6xu3brOXrt2rbN5bWUPc+HTrVs37/iee+5xNo6xMI8o5LHHHvOOBwwYEFgXX85UqFDB2b///rtXrzBf2gShmDZCCCGEEEIIIYQQEUQvbYQQQgghhBBCCCEiiF7aCCGEEEIIIYQQQkSQuMa0Ya6//npnjxo1yitDnT8GLmVNJ2oZO3XqlKfrQB3bJZdc4uwePXp49TZv3uxsjm+Tnp6ep+8WIpHAoKarV68OrIda0Ndee80rwyB/Q4YM8cq+/vprZ5988snO/vTTT3N/sUJEFIxjEKbZZl02jgnUZXOWA1wnUZ/PGU04EDiC8XQwPkPLli29egsWLHA2x2BQTBshREEQlPwjLO4LxsA08+Pt4bz28ccfe/Uwm9SWLVuc/ccff3j1jj/+eGdzYHeMufHqq686+/333/fqYeKGsN+Sn8lPokhuAjQH3SeOv8aBZrPgmIkYU5UD3LZr187ZGNMmLIZNWKy3ZG/HIH755RfvGH/7ihUrnI3P/2Z+W2O7cQDlsLZZtWqVszHGbX4Gdc4v5GkjhBBCCCGEEEIIEUH00kYIIYQQQgghhBAigsRVHoXpSs38VN4//fSTV7Z3715no7sgu5lhnvb8cC0Lc+f+n//5H2dfdNFFXhmmkmvRooWzi3LatyD3xrB2yqurYFDKv3Llynn10O2RXVuxb+3bt8/ZTZs29eqhHKCopWA888wznc3p8WrWrOnssmXLOpvbAMfREUcc4ZUde+yxzm7WrJmzJY8SyUrYPMeu2Tg/orQJx5uZP5dhWtoffvjBq4du/+hObOZLqTBF7caNGwOvNyylblF1AxdC5D9Bc0j37t294z59+jibnzMw3S8+Z4TtDVNTU52N+xUzsw0bNjib5258FujVq5ez+/fv79XDVNSPPvqoV7Zo0SJn4+/nvXaizq95XSNq1arl7BtvvNHZ/PyFayG2Ka99p512mrP5mXDYsGHOxj4zc+ZMr97u3bsDz4EU1XXxmGOO8Y7xeQIlUZxKHcH9y86dO70yfO7g9wZVqlRxNkrL+RxRQJ42QgghhBBCCCGEEBFEL22EEEIIIYQQQgghIohe2gghhBBCCCGEEEJEkEKPaXPqqac6m1Nyr1+/3tmcKhR1bKhDZM1frHFTsF5YKjn8Xo5XgrpEjg2AmkpMW1y/fv3A70o2+L6ijhDvJbanmX/PMY4C6o3NfA3kt99+65Xh+VEPedxxx3n1fvvtt8Dzt23b1tlz5sxx9tlnn+3Vq169urOnT59uRRXWC6Oe+4wzznA2p/Zr1aqVs1euXOmVtWnTxtkcC0fkHRxjYXFHLr74YmfjnMafwbaeMWNG4PkKWq8dZT14rHEGwvTunTt39o4x1hvOr5yyFNsH50bUgJuZrV692tmcWhPHN7Z/z549vXoPPPCAs8PaIGrtI4RIHDi2Bc5JmK67S5cuXr1169Y5m+dkTAGN82np0qW9ergHxPNhamK+jsWLF3tluPfE2Dr8XUceeaSzhw4d6pVNnjzZ2fPmzXN2ssytYb8DY/9wTNE6deo4G+MkciyToPNzKniMbcJxi/D8Y8eOdfbo0aO9erh+3nbbbV7Z7NmznR22/icz3BZ4jGOF3w3gcwGOZ35ewM9xjCr8HM4r3F+iQPSuSAghhBBCCCGEEELopY0QQgghhBBCCCFEFCl0edQll1zibHaxR5kMu0ChqxS6d7N7I7qWoWtT2PnCpFOY6pndMfG7MHW0me+eXq9ePSuK8H0NSoc9aNAg77hu3brORpdSTmOLbcNpZ7Ft0C1x69atgdfL0oPjjz/e2R9//LGzOQ1chw4dnF3U5FE4JjhdN1K1alVns2tio0aNnD1//vzA82PKRJE72M0T515Muf7KK6949dBt+8MPP3Q2p2dEGdVVV13llaEL97vvvutslkWyvC4vJINbOKeoxTkFU20zOD8effTRXlnQmsmu+Lt27XJ2pUqVvDKUWGHb9evXz6vXo0cPZ5955pmB14t9sqi6hCcTKCc288cizh1C5Adhsl6cg3hdwc/xPIll2H/5u/797387G1N58/k+++yzwHPgnMfzMIKpj3l969u3r7NRHpWMjB8/3jvGfcamTZu8Mnz+wvteqlQprx63Vxacnh3vOz8HYopoXD+5Hu6Zpk6d6pVdd911zsY9UrKkbo+FsP0g7l/4ORD3ERhOA6XjOX0XhmzAe3ziiSfmcNWFjzxthBBCCCGEEEIIISKIXtoIIYQQQgghhBBCRJBCl0ehu3RYdiF2l0Z5U1i2J5TMoEtVXt3KwjJVseQqFs455xzv+KOPPsrTdSU6KCdDOZSZ2XfffedsbMPvv//eq1elShVnn3zyyV4ZSpjQLfG8887z6h111FHOPuuss7wydK/D6O4NGzb06gW5WBYF0PWUJYKYUQ0zIJxwwglevc2bNzt7y5YtXhlmLAqS1xU1Ys2QhPXC5CfYt5944gmvjI9jIS0tzTtGd1904caMAGZ5k8uErQVRcyUOux5cZ2666SavDOdDXN/M/N+P8yGvTehSjK7A6E5s5ruPszs/jmGcG7kNateu7exJkyZ5ZYMHD3a2JFEFT17d62OdYy6//HJnn3LKKV4ZSy2FKCwqV67sbM5WGbYu4vyH6xHKXrgMz8HyG5TZoMzJzJ9PUQrC0g2c83mPVZTmUMwyauaHOkAZt5m/PuG+kddFlBtje3CWWQy/gPtVMz/LIsrcuM/gWstyuPT0dGfjfilqe5iChCVpGEYBx3DYczfeY35ewHvO2TXx/Pg8FxbyIV7I00YIIYQQQgghhBAiguiljRBCCCGEEEIIIUQE0UsbIYQQQgghhBBCiAhS6DFtML0v6+kxRRprNYNiF3BaterVq+f4mdyAmke+3jDdd1A6Qry+ZCdMT3/99dc7m1M5r127NtvzlStXzjv+9ddfnc3xMfC7Ud9aoUIFrx6207Jly7wy1CDjdzVt2tSrt3DhwmyvtyiAulPW6eJ4wXgbeC/NzKpVq+ZsTA1uZjZhwgRnJ2PK76x+mhvtMtZFbT2PN9T+ciyUgQMHOhtjD4XFsAmL74XjCFNDm/ka4TvuuMPZw4cP9+qF/RYkP2KVxYOw+fCWW25xNqfaRr01r4uoucZ5k+8LxkzAFO+s2V6zZk2232vma/6xb/G6iP2JY7jhHM7zgAgnaEyEjYG8jg/8HMfpGD16tLMxxsacOXO8ekuWLMnTdwtxuGCswrB4NJw+GPd8OGfyGMBjrMfzM8bp4HNgXZxPOWZH2HMGjj+0k2WvhLFlcJ9o5u89cW3i47DUz7jfxPbguDUYp4+fNfgZNAuOP4TXxJ/B/XFRZfv27d4x9mEcR7xnwTGMNq+XeMxjDPsSxk5VTBshhBBCCCGEEEIIERN6aSOEEEIIIYQQQggRQQpdHnXsscc6e9OmTV4Zut+jyyGD9SpWrOiVTZs2zdno7sZuhegqxanB0H3ysssucza6XJr5acPCzo+w/CMRiTXNbpgc4OWXX3b2Pffc49VD6cb06dOdzS6LeD7uB+juhqlIuQ1RUsCurSj5aN68ubMxnbiZn1K8qIGuxyw9Q3dgdHUMk3jMnDnTKzvzzDOdzTKMZCBrrglLZ86utNjvwz4XVta1a1dnxypjCJuTw0C5VJcuXZyNkiCzcDdmBOcVvjc4ZqMmnQq7no4dOzqbfxPKDnkM4FjC+xImlcN+well8bvY5Rzr4n3mfobjns8xaNAgZ48ZM8aEf79YxhhEfvdtXhdRinDvvfd6ZbhmYhuynE5kD0otevbs6ezHH3/cqxd2P4MkM7lJ8R6UtjrW74oamHoZ9+YcqqB+/frO5jUHJRph4RqCJMpcL2wNxnGPn2OpTGpqqrN5rUYJTr169QLrJSonnXSSs/ne4prGz1WrVq1ydtC6ZeZLeXEO5OeJHTt2OJulZ3j+oDTuZn7/xOcTM1+yXFQlxHiPzfz2xucMlC+Z+e2G+6P333/fq9evXz9nf/XVV14ZzgP4PBLrnrQwkaeNEEIIIYQQQgghRATRSxshhBBCCCGEEEKICKKXNkIIIYQQQgghhBARpFBi2mDaT4wHwnpP1Aayrh/1gZgS9Z///KdXD+OjoIYQU/mZ+XpITuuFKd1mz57t7Oeee86rh5pjTueHOlfUUZ5wwgmWTIRpnMN00lu2bHE2p8z+8ccfnX322Wc7+4wzzvDqLV++3NmvvfaaV4axAfA68HvN/BTsrDOtXbu2s1ED+cEHH3j1zjrrLGezFpbT2CUbmCad4yJgfAxM3YjtZmZWs2ZNZ994441eWatWrZx9ww03HN7FRpCg8YNxBFiHHQTeKzM//SGPnVq1ajkbY840bNjQq4d6erwOvibUeXNcDpzLcV58/vnnvXqYAhz7i5nZ0qVLnY33LK9xdqIAatexvXnOwBgHfN9RQ496br4vODbxHNxWeD5en/Eaca7k78J+x/r/K664wtlFNaYNxx3JSxyb1q1bOxvXHzN/P8PpjrFtMI0qxn3jz/GcnOxrWl4Jiv3Ce4JLL73U2RjjcdiwYV49nB8xjlCs18DwWhO0Pzv99NO9Y+wznNY9SmCsGlxz+J4cffTRzuaYIfgMgvcH1y0z/16izamEcSzy3I3nxLmbYyTiWshtiL8T478kS0ybZs2aOZvjr2HcIo6dFjQWeU3DPSquY3huMz92EI5ZM3+uxHNwjEd8HuU5FGPahPXPZIbXQRwf2J7cD3CMrVu3ztm4nzQz69+/v7N5LP7000/OxphjHL80CsjTRgghhBBCCCGEECKC6KWNEEIIIYQQQgghRAQpFHkUuhsh6J5r5ru4sfsSuncvWrTI2ZwuumXLls5GSRS7maHLHLqjmfluhi+++KKzb775Zq9ejRo1nM1uyAimlWO3/0Qkr2kf0e3zuuuuCzwfyo/QbXTx4sVevU6dOjl7wIABXtm3337rbEwt/8UXX3j10CXv5JNP9sqwL61YscLZ7CqLLpbozmlm9u6771oyg9I2dj1Ft0WU2aDk0Mx3G0VXbDN/3OI4ShaCXNRxTLBkafDgwc7GVJfshs+yTwTvJY6j8847z6uH8zC6qLIkBt20WdqKbYrjkn/Xhx9+6GyUqJr5qRexz916661evURyJ77gggucjfeIfzuuTzwGgtKI8ljEtTZMQoFlnO4Sz4Htj/MfH4eti2GpzJOZvK6fbdu2dTa6c48fP96r165dO2dXrlzZK8N1DF3077//fq8eys7D5FCJkg66IOBxhPMtjhWWxB9//PHO/uSTT5zdqFEjr97//u//Onv69Ole2euvv57tdYRJ0hmUZHTv3t3ZvA/CPRJer1m0UuJiymuc/3gdXL16tbM5zTDu/TG1MEsy8J6jBJQl4tgPeN+I4Fo4ZcoUr6xOnTrObtCggVeG63PQc1YigyEK+HkR1zv+7bi2bN26NfAc+DyG/YRlbnifuc+jlAq/l1O34/zIeyQsq1atmrM3b95sRQWeu7BtcOxwPbx3se49+P7j3Iuf47AqUUCeNkIIIYQQQgghhBARRC9thBBCCCGEEEIIISJIocijhg4d6myUQLE7P7prbtu2zStDlyi0UXZh5kthMNsTuwmjmxxmUjEz69Kli7PZ9RtBtyyO+I6/E7+rRYsWgecrLLJczXLjShvk2snnQLezW265xStDl1KUSWBkdjPfPRelAhgZ3Mx3C+fsOEHZNTjTBvYXznQyf/58Z2MbcvR4dI9s3ry5V5bs8ii8L+x6GpR9geuh6ym7LYaNv0SnWLFi7veyHBTvw8SJE72yIAkQz5noqs3u3ViG2aPCMj+hzeMeXYvDMhLh97L7Ks4PYdn+MLsJZ3J74IEHnD1t2rTAc0QBnLNwLeH7h9fNkgyUKWEGQ56vcR5FuRXfE5QpcZ/BumFu/9je7N6N63DHjh2dPWPGDEsmuJ3CpEN4X1HWgbIIM78NMZsakx9rDl4TS37/+9//OjuvkijOtpmI8G/ndS2LefPmeccjRoxwNkokP/vsM68e7o+xnpm/j8F9EM7lZr7sEmVzZmZ169Z1Nq4p69ev9+rhPBDlbH04t+D6xH0NfytL7jEDI85d/LtxjsO1j+vhOOLrwP6De2CWiuJ1nHvuuV4Z7okxq26ygNljebzhXMn3DCWJuIfk7LG4BwnLjoljgCVwGCYD+wU/m6KsCjPkmvlrMo5TDueQzHDGLgTXPt6X4LzL9zUI3r9OnTrV2SiRzM0zcmEhTxshhBBCCCGEEEKICKKXNkIIIYQQQgghhBARRC9thBBCCCGEEEIIISJIocS0wTgumFL20ksv9eph+lKOM4P6btQNom3m6wZRT8rnQ40cp3BD7SRqG1m7j1pW1HmbmT366KPOxjStX375pcWbLJ1eWOpX1o8GafswDpGZ2YMPPhj4mblz5zq7evXqzub0tHiP6tev72xO3Xj33Xc7m9NlorYb+wim8Tbz0zpyHA3UPm/cuNHZrKnEY9SJFwWwjXfu3OmVoU4b64Wli87N+ROdzMzMQ/p+FphyF1PEmpn95z//cXbTpk2dXa5cOa8epurlmFsYvykorTeDunFuMxwrHKMKU3Oivpzr4XXwfcEyTHeMaSHNfD04x7QJutfxguesLH755RfvGNcxngOxHXHODktHjDFzWLsflsIZ7x9eE8edwnWcwX6D8RmiHNMm7J4ElfFeIazvBZV17drVO8Z1jPcbSFCMHL5ehH8XxnPDuBJh381zTIUKFZzdrVs3rywsfkF+kZv9TV7g34vnDEs3e+GFFzob90R8j9asWeNsvJdmfopoHFOzZs0KvKYTTzzRK8M9EvYTvjevvvpqNr8iemBMF9w38FqFcybPtUF9hvd8eH58fuBxj3GJcD028+dkPB/Pn2vXrnU2tw3O3xy3MxnAOY/vLa/9CMYPwtgyGDvGzB+nuL/BODVmfiwc3rfwmpwF9ztsK14zsS6nCi8qcCw/bG+8P9wPwmI3Inj/+bkVU8Y3adLE2cuWLcvhqgsfedoIIYQQQgghhBBCRBC9tBFCCCGEEEIIIYSIIIUij0K3pPfeey9bm7n66qu94ylTpjib5UwIuk6hCy5/Bo/ZFQ6vNyydH7q4TZ482Stj1/wokhsXYZQzoVv/yJEjvXoff/yxsxctWuSV4X1t3LixszmFHrqsoosw17v55pudjWnazPyUmOwKh6CLMKYhNzM755xznI0pxFetWuXVQ/kHy6PC3PWSjbC01UHpLc38duVzoCszymKShaAxiH2P0xii+yxKCXkeQ/f9HTt2eGWYJhHnMU57j/0XbXbhRgkoSwPwOlBGxSlyw1LLYr/A/oNjz8ysdu3azmZJAd+DeFO1alVn4+/gtQrdhtmtGj8XJotB8F6ySzK66fM4RekAroucyhbXYJ7/8Ls5pXVUCVsng8rymioU7wmnb77oooucjf2A092GtX2sa/7KlSud3bNnT68M2/viiy92Ns8JOK+zVOCVV15x9qBBg2K6plgJkrjEKpdC2UWY9I/HDqbe7tWrl7N5PsTvatOmjbM5VfuZZ57pbO4Lbdu2dXb37t2djXJZM19Ki6mjzfx93IoVK5z9+eefWxDcjlGSnOI6g+sHp8Jevnx5tvXM/DkUxzD+3Sx4LuQxgPecZR2x9lNcx3jORPnGsccem+35Ehkci7w/x/uEa6mZ309xn8Htjf0X7dWrV3v1UHrO4xnPiesdy9XxuYbbGPe2KH0sSvA6FhSmhJ8DcA/E7YvgWOT1GSVRGDolantGM3naCCGEEEIIIYQQQkQSvbQRQgghhBBCCCGEiCCFIo9CVzB0EWQXMXSBQlmMme/2hO5QnD0qyEWcvyvMVRZd69Bm6Qa6b7H7JILuk3yOeLiXZv12dpXG6xw+fLhXhlmc1q9f7+zPPvvMq4cugZwdBSOwo5yC70mXLl2cjS7WmJnKzHddZtffF1980dnossrfhRKABg0aeGUoPUHJArvgobs4Z5TgaPXJDEsh0H0cx1GYzJDPUa1atWzPkWywGz66+3LGOZxD582b5+ynn37aq4dzKEuRypYtm21ZrPf473//u3d8//33O5szcqArKo6VsCxsfB1YhtfLbq4o48RMhWbxl6yyezyOD/x9nI0Cs9phu5n5a2HY2opgWZhUke8tzp1Brst8fs7wgWP/hBNOCLzGZKZmzZrecXp6urNREoNSYzNfCnHrrbc6GzMQmZl9+OGHzua2iRV0Qf/666+9MpREoSSDZQO4t8FrMjvUFT4/CZKAhUnDcPyhxJKvE2WrLO/H8Ydy/iuuuCKwHo6xzp07e/VQ2sSZxPA6UBZ76qmnevVwDcA9nJmfFeqrr76yWMhrfyoMcO+FcyaPN5TZ8FyLbYOyel6PcO5GOQWvs5i5iM+B14jzMO8FMHsUS6BxPcW2YZkWr8mJAmbf4t8QJIEy82Xe+EzC0hpe/7LYtGmTd4x9gT+Dz364PuPzDn+OZej4nMBtXFTg+RmPcV/CWX7xnoetKzhOGcyIiGMnSvLPLORpI4QQQgghhBBCCBFB9NJGCCGEEEIIIYQQIoLopY0QQgghhBBCCCFEBCmUIBGoTUONWFgcGNZ1ol4zTJOPWsEgDT7DMTZQ54rfxRpF/C2cljfo2qOgkcu6F3z/lyxZ4mzWOM+aNSvbc7H+MihWhpmv78W4OGlpaV49jOGBqbxRx2/mp1nn9L6oEcVr4lgPqE3m+DOoN8c2xDgrfPzCCy94ZYmqJc4LHM8H4wCh9prvyc8//+xsTpO4dOlSZ3PMhGSCtbh4j1h3j/PQ0KFDnb1t2zavHt5XHmM452E/5/GBMVMqV67s7D59+nj1MJbMxo0bvTLUgyPbt2/3jlG/zmOR55Kgc+M8zylC402tWrW8Y4yfgH0b50kz/7fzGMN2xDHG6wyen/sTEpY+E+PHhaX8xr7M+nNcn5Mhpg2OF+x7vN/AsdOtWzev7O6773Y2rqfDhg3z6m3dutXZOGabN2/u1cN4WF988YVXtmrVKmdjnABMIW1mNnDgQGdzKmHcG8yfP9/ZPD9jW3MK3YIka1xxfBeMgcEptKtUqeJs7LPcfzEODDN48GBn49zD8b84XkYQ2E+w3cz8+GW4zv6///f/vHqYApz75Pfff+9s3EdzHCqM94P3yez/+ldYDK2Cgvf++MyA8St4HsP4hxxXDWPE4TrL6wzOhfjbOWZK2HVg38L1ntNXz5gxI9vvMvP3trhHxfWYz59IYDwffsbC9ZTjS2E/bdWqlbN37tzp1QtKB96iRQuvHj6v8HqHazLO8+PHj/fqXXDBBc7m/c2vv/7qbI7PWVTgeFlBab5XrFjh1atbt66zw/o5tjU/++I8gLG/OJ5bFJCnjRBCCCGEEEIIIUQE0UsbIYQQQgghhBBCiAgS2Ry67IIW5MIflpYUba4XJNnKrm4W6IbK1xEme4qH62gsYJpeMz/FKMujguQtnEZy8eLFzkZXezM/ldopp5zibE6FiC7dKNm67LLLvHroVj516lSvDM+J18HSAOwjnIYP3W3RRZx/17hx45zN6ZlR8pHssKsxyj+w/7BrIrotsoskSgKSjVKlSrl0pGPHjvXK0FWX72ujRo2cffzxxzsbUwKb+W67Dz74oFfWoUMHZ6NLL48BlFzVqVPH2Tzu8Xq5zfD82PYrV6706uG80qBBA68MxxzOzzwW0f15woQJXtn5559v8YTnAnSxR0kZu07j3INyTjN/PsPxxmsVjisciyzLw7HIazC2MbYjz99YFmtad5ZFort4lAmSXTdp0sQ7Rhd9ljMh2A+GDx/ulWEa6Tlz5jgb3cPN/Dbk+9q6dWtn9+jRw9ksp8CxM3PmzMDrHTNmjLN53/T4448Hfq6gKFGihJMXsowZ70ubNm28MpTqxZrqmcH59+WXX3Y2u9i3bNnS2Shh5bGC449Tpnfs2NHZ69atczbLSHHO4b560003ORvHNs7XZr7sjcMWZIFSu8KC51PcZ+NcyHMcSgGxLcz89a5x48bZ/p2/G6UzLF+tV6+es1k+iNeLax+u72Zm06ZNczb+LjNfIoZ9s1KlSl69KMo8gsB+j/PSjh07vHq4prGkGNsB+zY/p+H6iTbvK1CyxuMI+xeuuyzfw/kR134zX/rFe6uiAo9THC8oD+V09tg2n3/+eeD58R5zuACc53GMsaQxCsjTRgghhBBCCCGEECKC6KWNEEIIIYQQQgghRATRSxshhBBCCCGEEEKICBLZmDYcByYozkxYmjC0WQuKx5zKG78L9Ysc7yFI4x9lypYt6zS9nHJ1zZo1zua0g5hyDe8Xp9pDrTim6zPzNYWotee04bNnz3Y2xrZ47rnnvHqY8m/AgAFe2e233+5s1BWzVhW/G+PsmJm9++67zsa0i8uXL/fqoQaXtcRRSztckKC228xvY9SnchtgDA/U+JtFU1OaX5QqVcr1P45LEZQu3cyP29KwYUNnc1wBPP7oo4+8ss6dO2d7fk6JiSljMeYCx05AjTbHZkC9fu3atZ2dFc8nC5yTOZYExjhBPTLGjjAze/vtt5390EMPeWUY/yceVKhQIbAsTE+PYyAsxgbeP+4L2F5Yj/X5uO6GpcXEODu8LmJMDNb14zqC33XiiSd69cK06VEC1xa0+fdgKmZeW998801nY8phjnO1aNEiZ7/xxhvOxrFs5u9feCziOonpiDleXKxgrA9e++KRurZEiRKub/J4xxTXHF+EY9FlwbEncE3jNNAYiwL3RZx2HeN0YHpt3svi9WNsPzM/zhCObY6pgusI76lxjsC5adasWV69BQsWOHvhwoVeWdb+hsd5YcBxKfD34X3gtsY9Ks9xuEfFtM/Yd8z8eRfTbnPKYZy7eb+N8VWwHsdNOvPMM53N8eLwvmNMEB73iQTOjziX8TqDcAzOdu3aZVuPxwCeE+87x5XjPoTgeoprN8ZEMvNTvHP7YF/gmC1FFZwbMRYbg8+BQTHmzPzxwe2L9x/3WGExzOKFPG2EEEIIIYQQQgghIohe2gghhBBCCCGEEEJEkMjKo1jOFCSPYndxdH9DNzOuh25U7EaLrlMoHWCZAp4jqmm9mTJlyjgZEN8TdNnj34qgVAFd/szMTj75ZGevXr3aK8OUfT/88IOzO3Xq5NUbMmSIszGVN6dKRbdyTo969tlnO3vFihXOPv300716S5cudfZLL73klWEKdGxrlrKga2tRkkMxPEaxD2GfYVdTvLfs3s3u6cnE77//7uQQp512mlc2f/58Z/N9xbkGbZbVoIzx1FNPDTwHtg2nyQ1KncnuvehmzOfA86MbM18vuvmja6yZ7+aP18Qu03379nX2Lbfc4pWxxKCwQUmRmf97w9KS4jGvMyirQjf9sPUOXX7x82b+mslrAKb7DRrbfI0sIcE073gdLBmKKtwvR40a5eyHH37Y2bz2/e1vfws8x7XXXuvsZs2aOXvcuHFePUztfOONNzr72Wef9ephKm+8PjNfJjly5EgLAmUjPP9gX8JxGYW5+s8//3TpYVk+g3JMTl2NYwd/B/923O+w6/ymTZucjbIVTlWMEtGwPWrYnICyWHT7Z7kPXj+X4dyJe94qVap49VDq98knn3hljz76qJmZ9enTxwobbkNsD5yfeK3CvSdKBPkc2B5hczKOhzBZEvYPM39uxLZmWSHOjSgvNfPXEOxL2J8TDRyn+BzIax/eP9zjm5l169bN2bjP4PYJSlXP58NnRO53OMZwPuRxhOEiWCqNYxjnHw7ZgM8ryQ5KEvGec8gEfObiPSWC0kV+bsX2wPEcJouLF/K0EUIIIYQQQgghhIggemkjhBBCCCGEEEIIEUEiK48Ki4KP7kvoMmxmdtJJJzn7mWeecfYrr7zi1UM3Q3aBRZc5/F50YePrSJQsNz///LPLhPSPf/zDK0NXS46Cj+6g6N7NkqXU1FRnt2zZ0ivDyPd4PpQ5mfmZLDAyOMvYvvnmG2ezOzq6R6J78pNPPunVw6we7N6NbproVonutWaHutgiiSKbKwhwXGF7s6QFXXm3b98eWFaUQHdQdmvHDFvYL9kdF8czygXN/Ew3mAGGXf7xu9DlnPs1zn88d6MEB8+Bc6tZcAYXM3/OQdnXFVdc4dXD7C5PPfVU4PniAc6NZsFZJzh7BP4mbCs+B95Pbkd080W3f26DoAx/Zv4YDpPPYl9g93NsR1w/ozzOsT04iwW6Zl9yySXO/uCDD7x6N910k7MHDRrklb322mvORpkvy5c6dOjgbFyrWrVq5dVDmRZKpcz8rENhYFvz/INlLBuJN8WKFXP9e/HixV4ZjgkeY5i5CefesKxILLXAfo/nCJN64rjkMYXn47KNGzc6G8dYmDwqVmnIGWec4R3jXmrChAnZXmNY1paCgucM3FfgWshrFcvVEFyrsA1ZuobH+Bm+x2EhFIIyzvI+FOd4Pj+eIyzLYCLB+4cs+H7hmsl9GTO5odyM71+QHK5JkyZePXweCpOLYhuw9BjXCgwjYeb3Ddy3JXI7Hi54T2Idi5yBD8FxzxI3lNqhrexRQgghhBBCCCGEECIm9NJGCCGEEEIIIYQQIoLopY0QQgghhBBCCCFEBImsYI51a6jzw5SyY8aM8erhMWoZOe4Inp91jqhF7N27t7M5RgvqkcNil8RD7xvEvn37bP369Wbma+TNzC644AJnd+nSxStDXTb+bk6xhhp3vq8VK1Z0Nt5j1Gebmbs+M7M2bdo4m2NetG/fPvAc06dPd/bzzz/vbI7hUKNGDWdzere8pO/mfhClti9o+LcGpRvl2FAYx4a1yWEp/JKBrPvCsSEwRTfGGTHzUyHivePUy5jGlVMv4zk5xhCCenpMk8j68rD4XtgPsB6fo27dus7G+Z7BWAasTV64cKGz8zJ+CxKOaYO/H+8ztpuZH8+C7wueA+cevrdYhmsfx8oIS0GM4H3ntZXHN4Jpz7FfxDumTfXq1W3s2LFmduh4eOONN5zNKVinTJnibIyj0KlTJ6/eggULnP2vf/3LK8OxOGDAAGdjXzYz++yzz5zduXNnZ7dr186r16tXL2dzDJtY9f/YXziGA4JjHWNNxYvMzEz3uzg+Gh8juA/AmBrcLytVquRsjneDczjOxbwuBo173t+ErZn4OYz1h3tjs/D05ThucZ3llNMYww9jnMWbsNTLuL/kOGC4L+V7gmUYz4JjWwSdn/sLjg9uQ1yTg649u2tEMO4YflcUY3HECt7DsP0CxijhFNo4PjB+Ke/P8RjHM98/7msItit+rk6dOl49HG98PrwOHLNB8X2KAtgPcAzwPhfbOixFN+6deIzhd+FzatjYixfytBFCCCGEEEIIIYSIIHppI4QQQgghhBBCCBFBIiuPYtB9DN0Ahw4d6tU77bTTnL1ixQpnc9pFdI9C9yozX0Jz0UUXOZtTPaPrXqLIYFJSUpzLHcodzMzuu+8+Z99///1e2TnnnOPsc88919mcShhd19hlEV3L0ZWYXXrRZR8/w9KZ9957z9mcNhzTgVevXt3ZLI9iWdXhkij9oCBgl09sL3QhDUtbyXKN7777Lj8vMXIEpcydPHmys9ldHedCdLnle4f3mfslznnYTjwXYtug2y5fN34Xu4jjNfL5EXQFZjdmnK/RlZ/ndZSQRA1OoY1jAtuO2wrvBY+dIEkUnwPbC124+ZrC0nWjqzC2Kbsa43fxdaAkB/tCvN3At23bZhMnTjQzs7vuussru/nmmwM/hzJclI5w2t5bb73V2Z9//rlXhmvVW2+95ezzzz/fq3fNNdc4e/Dgwc5OT0/36vFaiIRJopCwVMUIyq3QrTzRyO99QGGyadOmeF9CXOC5hde/LFi+ifLHMJkvyul4fsJxhPMiS1sxVTTLTXEvimsazx14TSz1QkkPridB+4pEAH8/3iP+TWvXrnV2zZo1vTJsV2w7lr6iNBXbjqVsvBYiuP7htfM+CGWMvKfD78NnKO4LRQm8J/hswW0Ytu9BwuRR2PbYX8L2q/EielckhBBCCCGEEEIIIfTSRgghhBBCCCGEECKK6KWNEEIIIYQQQgghRASJbEwb1u6jHhR195j+1sysWbNmzm7dunWevhs1qqgXZr0c6g1ZG4uEacILm4yMDKfx5XuM9wu1uGZmc+fOzdZm8Lc2b97cK2vbtq2zUWfMWniM57F69Wpn8/ViDAFMJWtmVq9ePWevWbPG2WGax6Kcrjs/YP0n3j9sO9Z2oxady6KQRragKFGihOu3Tz75pFeGsbk4xWFQjBiO1xSUWtbM14ejnp6120Gach4beI2sB8c0jGGxxDDWCuvX8RjHOsfleuqppyyIeOuTOe16UPwYjtMTa+w0PB//VrzveC85RSaufXxvUROOGnP+XWGpx4NiQcQ7Hsr+/ftdnIF+/frF/DmM9YbxJTCFtJnZWWed5ewbbrjBK8P2xvbl9O4Y3ws1+Dh+C4KwuQP1/xjrToiChtMy4/jDGEW8r8MYhxwfA/cpOCdzbBGOiZHdZ8z8+Q9TT5v5cyHue3gPhL+Lx3r58uWdjfFZEnnvir8J2473+OvWrXM2x7RZv369s3Ee5fUI+wLuW8LiyoWlU8e9GcdSwvUO12oz/7dhH4z3uhhP8PkuLS3N2RyXDZ8fwmK2YT/gOQE/F/TcEhXkaSOEEEIIIYQQQggRQfTSRgghhBBCCCGEECKCRFYexaBbG9rsvoQp8fLDRTAszS26DXN6NySqroroTmlm9umnnzqb3buDpGbowmbmp0X/4osvvDI+DgLvObqK8v2vU6dO4HWgrCpWotpOiQK7nuLYRHfiypUre/XQbZRhyU8yceDAAecyza60GzZscPaqVau8sh49ejgbZUlVq1b16oVJNnHuwvvPbcGu2lnwWMHvYvdhLAv7LnRZ5RS8KNfE1JmvvPJKtteXHWH9rDDge4n3EN1z2cUXXanZrTcsXXsQKIlCmZOZWYUKFZyNbvlmvnsxjm1ubxzr/JvxGMc2u74nCh999FFM9Z544okCvpKCIyx98EMPPZStLURhg/MOyogWLlzo1UM5Z9ieHiX2vFeuXbu2s3Ee53kM125ej/GZAaWPLVq08Opt3brV2Sj7MfN/C87D8ZYCHw64/0f5GodsQDp06OAd4/4B+wLLknA9xXvG8je8t7ze4XqNn2OZOEpply1b5pVhO6KMLpHb8XDBtgl7NsN9XZh0Dcc639eg7wrbQ8eLotsjhBBCCCGEEEIIISKMXtoIIYQQQgghhBBCRJDIyqPYHSrITYzrYSTpMNcy/FxY1qCwzB3oOsUZNBKBsGw/LE/AY3SNZ3dQdK9H+ZLZodkwskBJm5nvxoZtwy6qLCkR8YWlTOjmGibjwX4Y1ieTmdGjR3vH2O8xg52ZL8nYvn27s/ne4ZzEZZw1I4sweRS6GXNb45zALucos0H3YZbmlCtXztnsloqZjDCjQpg8is8RJvMoDDjzB7rHh0mgEJYDYzuESc+CXOfZXXznzp3O5rEXJHEMktDx95r5fQF/C67bQgiRE5zdDveRJ5xwgrMfeeQRr978+fNjOv+YMWMO4+oOhbMC8n42C16rcS6/4oorvLJGjRo5G/fDYVl0og7uD/EeYaY6M/+Z5KWXXvLKevfu7exYs/fi/eNnEswyxfsWXCexrWrVquXVGzBggLNRemfmy7sw1ANnxSqqBEnJmTA5U9geC/c2uKeOt6Q+O+RpI4QQQgghhBBCCBFB9NJGCCGEEEIIIYQQIoLopY0QQgghhBBCCCFEBIlrTJuweBWshUftPpaxNi3WNGFBn2GCUuPy5zB+R27OH0/yqtdDbSBrP/lYFB04rhOOP4x7wekUMbYFa1IxFWYys3nz5pjrvvDCCwV4JclDvGPYMKhVZ4LieJn5YwdjQ5n5YwfXKo4zg+fE9ZPHG8aW4ZTfyI4dOwLrYawe1p9jLKVEjrsghIgvHE8RnxFwnsT03GZ+TJvCjHsWFMOGCduX89qA8yn+fk43nUhg7DdcMznuGaZTv/32272yPn36OBvj0Rx99NFePVy7MJ7b999/79U78cQTnY0x9cz8VOSYrptjESKPP/64d4y/GddxjttUVMFnTn6uxzFx1FFHBZ4j1rThWC/suT5eyNNGCCGEEEIIIYQQIoLopY0QQgghhBBCCCFEBImrPIpduNFtOyxVKH6O0wcXJJgKzMx3tWvQoEHg56IqjxIiP3n11Ve94+HDhzsb3TyXLFni1evZs6ezWTIxe/bsfLxCIeLHDz/84B1XqFDB2bt27XI2u+TiuoNu2maHphHPgscRuv+WL1/e2ZxeFsvYTR+P8XMsU9iyZYuzWX6A58fUqV9//TX/BCGECISfH3DeRAk2p9pGwvbmWBbrHp7rBc2ZDNbjORPncv7NKAcpV66cs3FtSTR69OjhbGxH/u24loQRJlOKFZRioZ1XOG14tWrVnI0SvXPPPderN378+MP+7kQB27dKlSqB9XBchcmZUPbE9x+/C8dRrH2sMJGnjRBCCCGEEEIIIUQE0UsbIYQQQgghhBBCiAiilzZCCCGEEEIIIYQQESSuMW3C0uvNnDnTOx48eLCzMTYAa9iCzsl/Rw0pa/dR+4b6Uo6zU7NmTWd/++232X4vn0+IZIXjz9xwww3O/vPPP529aNEir97WrVudzWkxMZ2iEInMuHHjvOM2bdo4u0aNGs4+66yzvHpr16519tVXX+2VoRYe1xmOn4Ax4VjPjWAKWV4z8RjHKcd6mz59urM5TeuMGTOcjet4//79A69JCCGYp59+2jt+4403sq0XtocIewbB+TSve3iMcRNrOvGwep988ol3jOmncU6eM2dOrJcYOa666ipn16lTx9mpqalevddffz3wHLj+xRq3CNuY2wCfA/l8QX0j7Lny0Ucf9cq++uorZ2Pa8DVr1gRee7LTrFkzZ3ft2tXZ/By+ceNGZ+N9ZCZPnuxsjqGH+xSMo/Tee+/l4ooLB3naCCGEEEIIIYQQQkQQvbQRQgghhBBCCCGEiCDFzCxmv7/t27fbhg0bCvByRHakpaV5bpCHg9owfqgdEx+1YXKgdkx81IbJgdox8VEbJgdqx8RHbZgcBLVjrl7aCCGEEEIIIYQQQojCQfIoIYQQQgghhBBCiAiilzZCCCGEEEIIIYQQEUQvbYQQQgghhBBCCCEiiF7aCCGEEEIIIYQQQkQQvbQRQgghhBBCCCGEiCB6aSOEEEIIIYQQQggRQfTSRgghhBBCCCGEECKC6KWNEEIIIYQQQgghRATRSxshhBBCCCGEEEKICPL/AZRIHTcbwqhQAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 1440x360 with 10 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(\"The train set contains {} images, in {} batches\".format(len(train_loader.dataset), len(train_loader)))\n",
    "print(\"The validation set contains {} images, in {} batches\".format(len(valid_loader.dataset), len(valid_loader)))\n",
    "print(\"The test set contains {} images, in {} batches\".format(len(test_loader.dataset), len(test_loader)))\n",
    "nsamples = 10\n",
    "classes_names = ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat', 'Sandal','Shirt', 'Sneaker', 'Bag', 'Ankle boot']\n",
    "imgs_test, labels = next(iter(train_loader))\n",
    "fig = plt.figure(figsize=(20,5))\n",
    "for i in range(nsamples):\n",
    "    ax = plt.subplot(1,nsamples, i+1)\n",
    "    plt.imshow(imgs_test[i, 0, :, :], vmin=0, vmax=1.0, cmap=matplotlib.cm.gray)\n",
    "    ax.set_title(\"{}\".format(classes_names[labels[i]]), fontsize=15)\n",
    "    ax.get_xaxis().set_visible(False)\n",
    "    ax.get_yaxis().set_visible(False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VEdCXSwCoKok"
   },
   "source": [
    "## Basic auto-encoder\n",
    "\n",
    "We recall here that in order to define an autoencoder, we will need an `encoder`, which compresses the images into a small latent vector, and a `decoder`, that reconstructs the original image from this code. Here, we will start very basic and define the encoder and decoder as simple `Dense` layers. To define the model simply, we will use the [Keras API](https://www.tensorflow.org/guide/keras/custom_layers_and_models) defined in the `tensorflow.keras` module (Note that we pre-loaded the `layers` submodule"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "class AE(nn.Module):\n",
    "    def __init__(self, encoding_dim):\n",
    "        super(AE, self).__init__()\n",
    "        self.latent_dim = encoding_dim\n",
    "        self.hidden_dim = 256\n",
    "        self.encoder = nn.Sequential(\n",
    "          nn.Linear(28 * 28, self.hidden_dim), nn.ReLU(),\n",
    "          nn.Linear(self.hidden_dim, self.hidden_dim), nn.ReLU(),\n",
    "          nn.Linear(self.hidden_dim, self.latent_dim), nn.Sigmoid()\n",
    "        )\n",
    "        self.decoder = nn.Sequential(\n",
    "          nn.Linear(self.latent_dim, self.hidden_dim), nn.ReLU(),\n",
    "          nn.Linear(self.hidden_dim, self.hidden_dim), nn.ReLU(),\n",
    "          nn.Linear(self.hidden_dim, 28 * 28), nn.Sigmoid()\n",
    "        )\n",
    "\n",
    "    def forward(self, x):\n",
    "        encoded = self.encoder(x.reshape(-1, 28 * 28))\n",
    "        decoded = self.decoder(encoded)\n",
    "        return decoded.reshape(-1, 1, 28, 28)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we can see that the model depends on a given `encoding_dim` variable, which defines the size of the latent code. Therefore, we can instantiate our model arbitrarliy with `64` dimensions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "0MUxidpyChjX"
   },
   "outputs": [],
   "source": [
    "latent_dim = 64   \n",
    "model = AE(latent_dim) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The only remaining part that we did not discuss yet is what type of _loss_ (defined as $\\mathcal{L}$) we can use to train our model. First, we will simply rely on the _Mean Squared Error_ (MSE) loss, which is defined as\n",
    "\\begin{equation}\n",
    "    \\mathcal{L}_{MSE}(\\hat{\\mathbf{x}}, \\mathbf{x}) = \\mid \\hat{\\mathbf{x}}, \\mathbf{x} \\mid^{2}\n",
    "\\end{equation}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "9I1JlqEIDCI4"
   },
   "outputs": [],
   "source": [
    "# Loss function that we will use\n",
    "loss_fn = torch.nn.MSELoss(reduction='sum')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7oJSeMTroABs"
   },
   "source": [
    "<a id=\"application\"> </a>\n",
    "\n",
    "### Training the model\n",
    "\n",
    "Train the model using `x_train` as both the input and the target. The `encoder` will learn to compress the dataset from 784 dimensions to the latent space, and the `decoder` will learn to reconstruct the original images.\n",
    "."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "h1RI9OfHDBsK"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(5675.9932, grad_fn=<MseLossBackward0>)\n",
      "tensor(4074.0925, grad_fn=<MseLossBackward0>)\n",
      "tensor(3881.9771, grad_fn=<MseLossBackward0>)\n",
      "tensor(3564.0066, grad_fn=<MseLossBackward0>)\n",
      "tensor(3160.0903, grad_fn=<MseLossBackward0>)\n",
      "tensor(2828.1121, grad_fn=<MseLossBackward0>)\n",
      "tensor(2869.4741, grad_fn=<MseLossBackward0>)\n",
      "tensor(2570.3743, grad_fn=<MseLossBackward0>)\n",
      "tensor(2457.3076, grad_fn=<MseLossBackward0>)\n",
      "tensor(2376.8555, grad_fn=<MseLossBackward0>)\n",
      "tensor(2323.8416, grad_fn=<MseLossBackward0>)\n",
      "tensor(2240.7024, grad_fn=<MseLossBackward0>)\n",
      "tensor(2229.1489, grad_fn=<MseLossBackward0>)\n",
      "tensor(2182.6277, grad_fn=<MseLossBackward0>)\n",
      "tensor(2156.1914, grad_fn=<MseLossBackward0>)\n",
      "tensor(2125.0405, grad_fn=<MseLossBackward0>)\n",
      "tensor(2037.0834, grad_fn=<MseLossBackward0>)\n",
      "tensor(2029.6140, grad_fn=<MseLossBackward0>)\n",
      "tensor(2042.3490, grad_fn=<MseLossBackward0>)\n",
      "tensor(1917.9260, grad_fn=<MseLossBackward0>)\n"
     ]
    }
   ],
   "source": [
    "# Learning rate\n",
    "learning_rate = 1e-4\n",
    "# Optimizer to fit the weights of the network\n",
    "optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)\n",
    "for t in range(20):\n",
    "    full_loss = torch.Tensor([0])\n",
    "    # Forward pass: compute predicted y by passing x to the model.\n",
    "    for i, (x, _) in enumerate(train_loader):\n",
    "        y_pred = model(x)\n",
    "        # Compute the loss.\n",
    "        loss = loss_fn(y_pred, x)\n",
    "        # Before the backward pass, zero all of the network gradients\n",
    "        optimizer.zero_grad()\n",
    "        # Backward pass: compute gradient of the loss with respect to parameters\n",
    "        loss.backward()\n",
    "        # Calling the step function to update the parameters\n",
    "        optimizer.step()\n",
    "    print(loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wAM1QBhtoC-n"
   },
   "source": [
    "Now that the model is trained, we can test it by encoding and decoding images from the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "Pbr5WCj7FQUi"
   },
   "outputs": [],
   "source": [
    "encoded_imgs = model.encoder(imgs_test.reshape(-1,28*28))\n",
    "decoded_imgs = model.decoder(encoded_imgs).reshape(-1,28,28).detach().numpy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By plotting the images, we can see that the model is able to perform an adequate (yet somewhat blurry) reconstruction of the input images. The interesting point is that this reconstruction comes from a code of only `64` dimensions, whereas the original images have `784` dimensions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "s4LlDOS6FUA1"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABLQAAAD7CAYAAABzGs0JAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAACKFklEQVR4nO2deZxVxZn3nxZEUVlkX5sdlUUFNxQVUKO44Bg1EX01Juhk1MSE6CQTkxhmQlwTZ2Je4xJH805M3BKj0SRuKG6IKyAgIM1Oszc0iAiiUO8fzi1/9eOe4nZzu/ue07/v59Ofz3OpuufUqae2e6jfU2Vm5kwIIYQQQgghhBBCiJSwV0MXQAghhBBCCCGEEEKImqAXWkIIIYQQQgghhBAiVeiFlhBCCCGEEEIIIYRIFXqhJYQQQgghhBBCCCFShV5oCSGEEEIIIYQQQohUoRdaQgghhBBCCCGEECJV6IVWApdeeqktXrw4M/cplBYtWtiSJUusurraqqurbfny5Q1dpD1Cfky/H+XD9PvQTH7Mgh/lw/T70Ex+zIIf5cP0+9BMfsyCH+XD9PvQTH5Mux+bNnQBRGmxefNm69mzp/+c1obd2JEf0498mA3kx/QjH2YD+TH9yIfZQH5MP/JhNsiSH53+dv1r2bKl6927d2buU9u/5cuXN3gZ0lC/8mP661Y+zEb9yo/pr1v5MBv1Kz+mv27lw2zUr/yY/rqVD7NRv/Jj3fxph1YCH374oX344YeZuU9jRX5MP/JhNpAf0498mA3kx/QjH2YD+TH9yIfZQH5MN6mKodWxY0e74447bOnSpVZdXW0LFiywG2+80Vq0aLFL3ptvvtmqq6tt69atdtlll5mZ2ahRo+zll1+26upqe/XVV23QoEHBd8rKymzu3LleR7pjx46CyjV8+HB78cUXbcOGDbZmzRp77LHHbNCgQTZhwgTbsmWLLV261Fq3bl3r+1xxxRW2efNme+CBB+zYY4+1Z5991tavX29VVVX28MMPW+fOnfN+74ADDrArr7zS/vKXv1hlZaWtX7/e3nvvPRs3blxBz1VXyI/p96N8mH4fmsmPWfCjfJh+H5rJj1nwo3yYfh+ayY9Z8KN8mH4fmsmPWfFjfdHg28QK+evXr5+rqKhw48ePd61bt3Zm5jp27OgmTpzoZs6c6dq1a5f3exMmTHCXX365mzhxops3b5479dRTnZm50047zV111VV7vO3unHPOcfPmzXMnn3yyMzO3zz77uMsvv9xVVVW5F154wV122WVF2d43YsQIV1lZ6SorK93ZZ5/t9tprL7fvvvu67373u+7dd991ZWVlu3zn+OOPd7/4xS/cSSed5Pbbbz9nZq5Xr17uH//4h7viiisKqvdibz2UH9PvR/kw/T6UH7PhR/kw/T6UH7PhR/kw/T6UH7PhR/kw/T6UH7Pjx3r8a/AC7PavrKzMvfvuu+6UU07Jmz5u3Dj35JNP5k2bMGGCq6ysdH//+99dixYtanTf3Tm1devWbvHixa68vHyXtDFjxjjnXFEb9qpVq1z//v13SXv88ccT6ybfX4cOHdy7775blDqQHxuXH+XD9PtQfsyGH+XD9PtQfsyGH+XD9PtQfsyGH+XD9PtQfsyOH+vzLxWSw1GjRtnSpUtt0qRJedPvv/9+69Kli/Xt2zdv+iuvvGJjxoyxzZs3F7VcY8eOtYceesiWLVu2S9pTTz1lU6ZMKer9Jk2aZPPnz9/l32fOnGnl5eUFX2ft2rXWoUOHYhatIOTHz0mzH+XDz0mzD83kxxxp9qN8+Dlp9qGZ/JgjzX6UDz8nzT40kx9zpNmP8uHnpNmHZvJjjrT7sT5JxQutIUOG2MsvvxzNM3nyZBs6dGjetBdeeMF27txZ9HINHjzYpk6dmpj++uuvF/2e+di5c6eVlZXt8u/77ruvXXfddfb222977e38+fPtqaeeqpdyMfJjnDT4UT6MkwYfmsmPuyMNfpQP46TBh2by4+5Igx/lwzhp8KGZ/Lg70uBH+TBOGnxoJj/ujrT4sT5JzSmHzrloej7H1jXOueh9G6JMOZo1a2YvvfSSVVRU2FVXXWVz5861jz76yFq2bGmHHnqoPfTQQw1SLvmxZpSiH+XDmlGKPjSTH2tKKfpRPqwZpehDM/mxppSiH+XDmlGKPjSTH2tKKfpRPqwZpehDM/mxppSqH+uLVOzQmj59uo0aNSqaZ+TIkTZt2rR6KtHnzJo1y4499tjE9FhaXXPBBRfY/Pnz7ZJLLrG3337bPvroIzP7/LjQ1157rUHKJD/WnFLzo3xYc0rNh2byY20oNT/KhzWn1HxoJj/WhlLzo3xYc0rNh2byY20oNT/KhzWn1HxoJj/WhlL0Y32SihdakydPtvLycjvllFPypo8bN85WrlxpCxYsqNdyPfroozZ27Fjr1q3bLmlnnnmmDRs2rF7Lgxx44IE2e/bsvGknnHBCPZfmc+THmlNqfpQPa06p+dBMfqwNpeZH+bDmlJoPzeTH2lBqfpQPa06p+dBMfqwNpeZH+bDmlJoPzeTH2lCKfqxPUvFCyzlnY8eOtbvvvtvGjx9vrVq1MjOzDh062MSJE238+PE2bty4ei9XdXW1XXvttfb888/7N8n77LOPjRs3zm655Rb77W9/W+9lyvGPf/zD/vmf/9m/LS4rK7PDDz/c7rnnHnvwwQcbpEzyY80pNT/KhzWn1HxoJj/WhlLzo3xYc0rNh2byY20oNT/KhzWn1HxoJj/WhlLzo3xYc0rNh2byY20oRT/WNw1+1GKhfx07dnR33HGHW7ZsmauurnYLFy50N9100y7Hcnbr1s2tWLHCVVdXu61bt7otW7a46upq//e3v/0t7/UnT54c5NuxY0fwef78+W6vvfba5Xsnnniie+mll9yGDRvcypUr3X333efat2/vJkyYkPf4zpre549//KPbvHmz++STT9yKFStct27dgrTcM958883BfU444QT3yiuvuHXr1rnq6mr35ptvuu9///uuRYsW/p4//vGPff6ysjL3wQcfRMv26quvyo/yo3yYAR/Kj9nwo3yYfh/Kj9nwo3yYfh/Kj9nwo3yYfh/Kj9nxYz39NXgBMvt30003uYsvvrjBy6E/+bGx/8mH2fiTH9P/Jx9m409+TP+ffJiNP/kx/X/yYTb+5MeG+0uF5DCtjBgxwmbMmNHQxRB7iPyYfuTDbCA/ph/5MBvIj+lHPswG8mP6kQ+zgfzYcOiFVh3QsmVLu/32223+/PmJAdpE6SM/ph/5MBvIj+lHPswG8mP6kQ+zgfyYfuTDbCA/Njx6oVUEBgwYYGvWrLHq6mqrrq62iooK27x5c4MErBO1R35MP/JhNpAf0498mA3kx/QjH2YD+TH9yIfZQH4sPcrsc+2hEEIIIYQQQgghhBCpQDu0hBBCCCGEEEIIIUSqaNrQBagpa9eutaVLlzZ0MRolPXr0sA4dOhTlWvJjw5B1H3bq1KmgfJ9++mnwuWnTL4bCsrKyIG3nzp3eXrt27R6Urnhk3Y+NgSz6cO+99/Z2ixYtgrT99tvP2/vuu6+399or/H+1zz77zNsffvhhkFZVVVWUchaTLPqxsSEfZoO0+pHXHM7lF8507do1+IzjKH4n6ftmZlu3bg0+r169uuBy1gdp9WGh4FrTzKxfv37e5rkQQZ9+/PHHQdqSJUuKU7giklY/4hrGzKxdu3be3rx5s7e5j6HvduzY4W38/WAW/63RpEkTb++zzz7exjWRmdn69euTH6CIFNOH9UHqXmgtXbrUjjrqqKJdjxsUEpsU0gA+WzGe5e23397ja+Qoth9FYWTdh9ddd11iGk4y/GKqdevW3m7WrFmQ9sknn3j79ttv38MSFoes+7ExUMo+5HkRP/MCDencubO3R44cGaQNGTLE24cccoi3mzdvHuTDxdpzzz0XpN13332RUjcMpexHURjyYTZIkx9xTMUfsma7/oDNceONNwaf+/fv7238Tzoeo3HtM3fu3CDtpptu2m35mLr8bZQmH9aGtm3bBp//8Y9/eBv/04dfbuE6dPr06UHaZZddlvdehb4orQvS6scuXboEn7/xjW94++WXX/Y291F8AbVp0yZvb9u2LciHL8jwO2afB5bP0adPH2+vW7cuyPc///M/yQ9QRIrpw/pAkkMhhBBCCCGEEEIIkSpSt0Or2NTkjTW+PT/zzDO9/W//9m9BvgEDBngb3+hWV1cH+dq0aePtww8/PEjr2LGjt++9915v33nnnUG+GTNmJJY37TvMhCiEnj17evuqq64K0qZNm+Zt3Fp/2mmnBfnef/99b/fq1StIGzRokLd///vfe5v7sxBpBv83Nya75S3ojz32mLfxfx/xfxvNkuUULMHA3Qqnn356kPbTn/7U2x988IG3Tz311LzXznff2A4zIYSoC3gXFu6aStqRZRbu1Pg//+f/BGmLFy/2No5rLJtCvvSlLwWf77//fm+vWbPG27HfD7FnEZ+DdYT1c+uttwb5Dj30UG+jP1FOahYqB84+++wgrW/fvt5esGCBt3lu5VAbYlfGjBkTfP75z3/ubexjMXkoctdddwWfr7zyysS8uBOrffv23v7oo4+CfPW1QyttaIeWEEIIIYQQQgghhEgVeqElhBBCCCGEEEIIIVKFXmgJIYQQQgghhBBCiFTR6GNoMf/yL//i7R/96EdBGsYVwVPRWD+O2unRo0fXqhyomT3//PO9fe655wb5VqxY4W2Op3XppZfW6t5CpAk8MW3+/PmJ+VB3/uc//zlIw9ODvvOd7wRps2fP9vbAgQO9/dprr9W8sEKUKBgzJRYfgmNAYJ/AGBB8tDTOkxgPhI+R51NGEYzfhbFghg0bFuR74403vM3xXhRDSwhRVySdLh6LM4Vxd83CGJ84tr3yyitBvq5du3p71apV3t6yZUuQr3fv3t7mk2Mxxs+f/vQnb7/wwgtBPjwZOvYsxT5dvZSoycmPSXXE8R75BLscHKMVYzjzyXknn3yytzGGVixmViy2ZJZ9uDs2btwYfMbnnzNnjrfxHYBZ6G/0HZ/OGPPPvHnzvI1xtdN22mBDoR1aQgghhBBCCCGEECJV6IWWEEIIIYQQQgghhEgVjV5yiEeOm5mdeeaZ3t6wYUOQ9sknn3gbt9/y1s1NmzZ5uxjbNWMSiQMPPNDbX/7yl4M0PA726KOP9nZjP7o1adtwzFe13YKbdHRvq1atgny4nZi3i2P72r59u7eHDBkS5EOZTWM6Rvm4447zNh9vW15e7u0WLVp4m+sf+9E+++wTpLVp08bbRxxxhLclORRZJTbGsdwBx0aUC2J/MwvHMTxWfuXKlUE+lNHg9nyzUJ6IR8wvX748sbyfffZZYlpjllYIIYpP0jhyzjnnBJ8vueQSb/Nvjfbt23sbf2vE1obt2rXzNq5ZzMyWLl3qbR6/8ffARRdd5O3LL788yLd48WJv33nnnUHatGnTvI3Pz2vtNI6xtZ0jevbs6e2rrrrK2/z7C+dC9CfPfUceeaS3+Tfh9773PW9je3n22WeDfB9//HHiNZDGPC+2bNky+Iy/KVBm2LRp8usTXMNUVVUFafjbg98ddO7c2dsYsoGvIfKjHVpCCCGEEEIIIYQQIlXohZYQQgghhBBCCCGESBV6oSWEEEIIIYQQQgghUkWjjKE1dOhQb48ePTpIW7Jkibf5uG/UzKLumTXGhcZownyx42DxvhwbCXXQHIsENdyzZ8/29kEHHZR4ryzCdYu6ZaxP9KlZWO8YuwXjG5iFmuuFCxcGaXh91F+3bds2yPfhhx8mXn/UqFHenjRpkrdPPPHEIF+3bt28/cgjj1hjhOMTYOyIY4891tt8NO/w4cO9PXfu3CBt5MiR3ubYW2LPwD4Wi3V03nnneRvHNf4O+vuJJ55IvF5dx4go1RgUhcY0icXXOP3004PPGFsSx1Y+dhx9g+MixpswM5s/f763+Whs7N/o+wsvvDDI98tf/tLbsfovJd8IIdIHx9LBcWnYsGHePuOMM4J8ixYt8jaPy+vWrfM2jqnNmzcP8uEaEK83Z86cIB+WY/r06UEarj0xlhffa9999/X2+PHjg7R77rnH21OmTPF2FsbX2DNgnDGOYdynTx9vY1xWjpuUdP0BAwYEnzGOEsdIw+vfeuut3v7xj38c5MP586c//WmQ9vzzz3s7Nv9nHfYHfsa+wu8H8LcB9mf+zYDf45h4+D0cV7jNiPyoloQQQgghhBBCCCFEqtALLSGEEEIIIYQQQgiRKhql5PD888/3NktWUHbGWwpx6yFKJni7MG7XxK2CsevF5Ijbt2/3Nm9vxnvtt99+QRpKPvr372+NFa5blm3m+Pa3vx187tu3r7dxmzYfR4/+4ePj0T+43Xf16tWJ5WVJT+/evb39yiuveJuPcj311FO93Zgkh9gncOs106VLF2/zVt/Bgwd7+/XXX0+8Ph57LGoOb53G8ffAAw/09qOPPhrkQynESy+95G0+YhmliV//+teDNJRFPP30095mqTHLVmtD2qUWfMQ8jidr1qxJ/B6OjQcccECQljRnsrRl8+bN3u7YsWOQhrJF9Ntll10W5Dv33HO9fdxxxyWWF9tjY5ZZZAmU6JuFfRHHDiGKQUwqj+MQzyv4PR4rMQ3bL9/rL3/5i7e3bduWeL2pU6cmXgPHPR6LkY8++ihvmczMvva1r3kbJYdZ47bbbgs+4xqjsrIySMPfX1jnzZo1C/Kxr3KgP83COuffgevXr/c2zp+cD9dL999/f5D2zW9+09u4Pio0TEFWiK0HcQ3DvwNxLYEhajAkw+7uhaFQsJ4PPvjg3ZRamGmHlhBCCCGEEEIIIYRIGXqhJYQQQgghhBBCCCFSRaOUHKIEIXYCHksQUDIYO5UQJWi4RbG2WzVjJyqyjLEQRowYEXx++eWXa1WuLIAyTZQYmpktXrzY2+jHZcuWBfk6d+7s7YEDBwZpKAvE7b6nnHJKkG///ff39gknnBCk4ZZVPIlk0KBBQb6krctZB7dzs+wWT/7EU3r69esX5FuxYoW3V61aFaThqXpJctXGSKEn+WG+mKwL2/Zvf/vbII0/F0KPHj2Cz7iNHmUReHKNWe1kaLH5oJS26MfKgvPMtddeG6ThWIjzm1n47DgW8tyEW/Rxaz1uzzcLJRksj8E+jOMi13+vXr28/X//7/8N0q6++mpvS2ZYP9RWtlLoGHPBBRd4+7DDDgvSWL4sRH3RqVMnb/PJyrF5EcdAnI9QTsZpeA2WtaF8DaWDZuGYivIqlkPhuM/rrMYyjuJp2GZh6BAMi2AWzk+4buR5ESX86As+CR3DmeB61Sw8DRhlo9xecK5leemll17qbVwrldL6pT5gqSeGJ8E+HPvtjfXMvxmw3vkkaLw+/p6LhVIRX6AdWkIIIYQQQgghhBAiVeiFlhBCCCGEEEIIIYRIFXqhJYQQQgghhBBCCCFSRaOMoTV48GBvc/wOPOaUdeFJcVL4aNRu3brt9js1ATXWXN5YjImk44SxfI2BWPyOf/mXf/H2xx9/HORbsGBB3uu1atUq+Lxp0yZvczwevDdq6tu3bx/kQ1/NmjUrSMOYB3ivIUOGBPneeeedvOXNOqhx57gA2F8wvg/Wo5lZ165dvd2lS5cg7Ve/+pW3uY1khVw7rUm8BMyLsTy4v2GsAY6/9K1vfcvbGO8sFjMrFlMQ+9HSpUuDNIxJ8O///u/evuaaa4J8sWdBihEfsb6JjYXXXXedtzt27Bjkw9gOPC9ifAccM7lOMD7LgQcemPf7ZmYVFRV572sWxhjBdsXzIrYljhmJ4zePA2L3JPWJWB+obf/A73FcoB//+Mfexpg+kyZNCvLNmDGjVvcWYk/B2Kix+FcYX9AsXPPhuMl9AD9jPh6jMS4QXwPz4pjKMYJivzWw/6GdhfUSxrLCdaJZuPbEuYk/49qfwfUm+oLjZGFMUP6dwb9Bc3CsMywTfwfXx42ZtWvXBp+xDWM/4nUL9mG0eb7Ez9zHsD1hrGbF0CoM7dASQgghhBBCCCGEEKlCL7SEEEIIIYQQQgghRKpolJLDNm3aeLuysjJIQzkLbuFlMF+HDh2CtIceesjbuIWUt+ni1kM+2hO3I3/1q1/1Nm5hNguP/YxdH2FJVVqJyYGwLmIym4cfftjbP//5z4N8KId65JFHvM1bgfF63BZwCykeJ85+RKkObxdHGdVRRx3l7aqqqiBfy5YtrTGCW/lZyonb63HrcEw29eyzzwZpxx13nLdZ2pQVcuMNj0MIb1HHdh/7XiztzDPP9Hah0qDYuBwDJYhnnHGGt1FqZxaXByA4rnDdYJ8tJTlirCynnXaat/l5UMrLfQD7EtZJTHqKbYKPh8d7sYwD82IdcxvDfs/X+Pa3v+3tG264wcTnYJ2xNDiJYrdtnhdR4nPjjTcGaThnoh9Zpip2BeVLF154obfvvvvuIF+sLpMkaLH1FoPrXJ6TC7lXKdK6dWtv4/qcQ4AcdNBB3uY5B2VPsTAoSbJ/zhebg7Hf4/dYgtauXTtv81yN0rb+/fsn5ksjhxxyiLe5XnFO499V8+bN83bSvGUWyuNx/OPfEuvWrfM2Sznx+jjf8VoJ2yb+NjELwwA0Zlk+1rNZ6HP8rYGSQLPQd7hGeuGFF4J8l112mbdnzpwZpOE4gL9JCl2TNna0Q0sIIYQQQgghhBBCpAq90BJCCCGEEEIIIYQQqUIvtIQQQgghhBBCCCFEqmg0MbTw6G6MPcTactQicxwR1CPjsebXX399kA9jMaFmGY/iNQv113wsJx7L+vzzz3v797//fZAPYxzwcbyoqUfddr9+/SxrxOIqxGIzrFq1ytvvvPNOkLZ+/Xpvn3jiid4+9thjg3zvv/++t//85z8HaRiLBMuB9zUz69atm7dZ296rVy9vo+b6xRdfDPKdcMIJ3mb9PR9FmyVmzZrlbY7BgvF48Phl9JmZWXl5ubevuuqqIG348OHevuKKK/assCVKUv/B2CUc+yEJrC+z8Ahj7js9e/b0Nsa4GjRoUJAP43dgObhMGFuC4wDheI5j4wMPPBDku+aaa7yNbcbM7L333vM21llt43o1NBgrA33N4wXGU+E6x5gdGDuC6wT7Jl6D/YTX4/kZy4jjJN8L2xzHGxk7dqy3G3MMLY51VJu4Wccff7y3cf4xC9c0GHvELPQPHoWOcSb5ezwuZ3lOqy1JsaZ4PfCVr3zF2xhT9nvf+16QD8dGjFlWaBkYnmeS1mbHHHNM8Bnby6RJkwoqR0OBsbFwzuF6OeCAA7zNcYrwdwjWEc5bZmF9oo2/LczCvsjjN14Tx2+OyYpzIfsRnxNjTmUhhtYRRxzhbY73iDHSOFZjUl/kOQ3XqDiP4bXNwjhl2GfNwnESr8ExZfH3KI+fGEMr1jazDs+D2D/Qp9wWsI8tWrTI27ieNDO7/PLLvc19ccOGDd7GOIccL1nkRzu0hBBCCCGEEEIIIUSq0AstIYQQQgghhBBCCJEqGo3kELfvIbjd3SzcNsrbAVEyMW3aNG+jxNDMbNiwYd5GmSFv3cRtqLjF0yzctvvHP/7R2z/4wQ+CfN27d/c2b+tH8GhYltGkldoe34xbqb/5zW8mXg8lfbgVe/r06UG+0aNHe/vKK68M0hYuXOjtyspKb7/11ltBPtzmOnDgwCAN29OcOXO8zdvPcesybpM2M3v66actq6BUlLdz4zZglK2hjNcs3IqN8gazsN9iP8oSSdIP7BMsA7z66qu9jUdWs7yF5dQI1if2o1NOOSXIh2MxbvtmqRlKH1gyjn7FfsnP9dJLL3kbpd9m4fHJ2O5+8pOfBPnSsk3/7LPP9jbWDz83zk/cB5KOAue+iHNtTJaEaXxcNV4DfY9jH3+OzYuYD8f4xkBt589Ro0Z5GyUSt912W5Dv5JNP9nanTp2CNJzHUPpyyy23BPkwnENMYpgk78k63I9wrMW+wmEmevfu7e1XX33V24MHDw7y3Xzzzd5+5JFHgrTHHnssbzliIR4YlDmdc8453uY1EK6PsLxmpXekff/+/b2NYyDPg/Pnz/f2unXrgjRc/y9btszbLHPCekdpNYdewLbA60YE58J77703SOvTp4+3BwwYEKTh/Jz0WyutYMgP/r2I8x0/N84tq1evTrwG/h7DNsKyUaxjbvMoT8T7YqgAs3Bs5PURpnXt2tXbK1assMYEj1/oH+w7nA/rr9D1B/sAx1/8HocrEvnRDi0hhBBCCCGEEEIIkSr0QksIIYQQQgghhBBCpIpGIzkcP368t1FWyPIY3AK9Zs2aIA23GKKNUiazUFqGpxLytnvceoqnfZmZnXHGGd5mOQWC2xz5VBJ8TrzX0UcfnXi9+iS3fbMmW9STtkvzNXAr53XXXRek4TZtlB7hKSJm4bZ3lODgCRZmodSCT3FLOgWKT4TCNsOncr3++uveRj/ySSe47fioo44K0rIsOcQ64e3cSScEcT7czs3bgGP9LwuUlZX5Z2aZNdbF7bffHqQlyep43ET5A0smMA1POYydUIg293vcsh87PQ/vy1vCcXyInUyLJ3HxiaO//OUvvf3QQw8lXqOhwfEK5xKuOywzy5xQ+ocn7fJYjWMoShi5PlD6x+0F88ZkNOhrlkzgPHzaaad5+4knnrCswb6KyfGwblEqhVIjs9CPeOonU4w5B8vEMvp3333X27WVGebaa0wCW8rwc/O8lmPKlCnB53/7t3/zNsqOp06dGuTD9THmMwvXMLgGwnHcLJQyowzVzKxv377exvlkyZIlQT4cB0r9RFkcX3B+4lPI8Xk5jAWeFIzjFz87jnM493E+7EdcDmxDuAZmCTaW46STTgrScE2MJ8BnATzhnPsbjpNcXyjzxTUkn3CO64/YCc7YB1hSiqFnsE3wb1OUKuIp7mbhnIz9lMOjZB0+XRLBuY/XJjj2ct0mwevX+++/39soO67Jb+TGjHZoCSGEEEIIIYQQQohUoRdaQgghhBBCCCGEECJV6IWWEEIIIYQQQgghhEgVjSaGFsaNwiPhv/KVrwT58AhyjmuFsSRQp4y2WahTRu06Xw/1uHwMK2q1UUvNsUJQN48xJczM7rzzTm/jUetvv/22lQI5XXAsfgVr1pO0xBj7zMzsP//zPxO/M3nyZG9369bN23zMPNbTQQcd5G0+fnnixIne5mOvMZ4EtpNhw4YF+fBoZo7bg7EWli9f7m3WcONnjE2RddC/VVVVQRrGhMB8HGOC42YVev0s4Jzbpe3nuP76672NR72bmT311FPeHjJkiLdbtWoV5Fu7dq23Oc4fxovDcS42JmCsCvYb9hWOiYdHbGNMC86H5eB6wbQNGzZ4G492NgtjUHAMraS6bgh4vMqxcePG4DPOYzz+oQ9xvGYfJsU341gh+L1YzBIsE8e5w3mcwTaDsWBKPYZWrF6S0ni9EGt7SWlnnnlm8BnnMV5zIEkxubi8CD8Xxo/EWDaxe/MY0759e2+PGTMmSMvFS+F4lMWiJmub2sDPiteMHRf/T//0T97G9RDXT0VFhbexHs3MBg0a5G3sU88991ximQ4++OAgDddH2Ea4bv70pz/leYrSBGNI4dqB5yocN3m8TWo3vObD6+NvCO73GAsN52OzcFzG6/EYumDBAm+zf3AM51jBaQfHO65XnvcRjFWGsawwVpVZ2E9xbYNxsczC2Fu8ZuE5OQe3OfQTz5mYt3Pnznmv1xjg+KHoc6wjbguxWLEI+oB/t/bo0cPbhx9+uLdnzZq1m1ILM+3QEkIIIYQQQgghhBApQy+0hBBCCCGEEEIIIUSqaDSSQ9zm98wzz+S1mW984xvB53vvvdfbLBFEcCsiHgHK38HPvL0Uyxs7jhe3jd5zzz1BGktdSpWabL1HiSDKZX74wx8G+V555RVvT5s2LUjDuj300EO9zUfh4jZw3HrP+X7wgx94G49aNQuPtubtpQhuvV+4cGGQNmLECG8ff/zx3p43b16QDyVVLDnMbYFN6/HkhcLyJdwinHQ8tVnoU74GSgNQZpYlkvogtj0+ihi3paM8l8cylMWsW7cuSMOjjnEswzo3C7dwo82yCJRWs+QGy4HSRD7qPnZEPLYNbEPY98zMevXq5W2W6nAdNCRdunTxNj4Dz1W4DZ+lCvi9mMwMwXrkLf4oe+F+iuMXzot8FD3Owbz9H++NYQRKndg8mZRW2+O+sV5wbjIz+/KXv+xtbAt8bH3M/4XO+XPnzvX2hRdeGKShz8877zxv85iAYztLcB599FEzM/vOd75TUHkKJWmeLVSCiFKmmJyW+85tt93m7YsuusjbPBbivUaOHOntp59+Osh33HHHeZvbwahRo7x9zjnneBvl52ahNH3FihVBGq7h5syZ4+0333zTkmAflpKE2yycZ3D+QCmimdn777+fN59ZOI5iH8Z/N0seD7kPYL2zVKrQtorzGI+bKImqK/luQ4F9kdfmWEc4l5qF7RTXGOxrbL9oz58/P8iHoRy4P+M1cb7j8A/4m4b9i2tblBM3NngeSwr/w78FcB3EPkawL/L8jDJDDElUSmvGUkY7tIQQQgghhBBCCCFEqtALLSGEEEIIIYQQQgiRKhqN5BC3V+KWW952iVsKUWZmFm4jxO2FfMphkuyC7xXbfo7bVdFmORRuh+TtyAhuR+ZrNNSW7dzzs/wAy3rNNdcEaXja4JIlS7w9derUIB9uteWTvPC0EJQocb2cccYZ3kbZAp6gaBZKAnhL/R//+Edv4zZwvhdKawYMGBCkoZwLpUC8rRUlGHz6Ue5kFd5unjVYXoSSDOxHMekuX6Nr1655r5FFWN6C2+j5dFRsS1OmTPH27373uyAfjqMs72vRokXetELr+V//9V+Dz7fccou3+eQo3N6NfSV2WiiXA9OwvLx1HKXReKquWcNKwbn/Y//AZ+NTk/DkVfSZWTgXxuZWBNNi8l+uVxw3k6QAfH0+iQr7fr9+/RLLmHXKy8uDz5deeqm3UWqG8n2zUF70k5/8xNt4Wp6Z2UsvveRt9k+hoKxj9uzZQRrKDFHmxHIcXN9gmcy+kJfUVpqZRJKkMia1xP6HkmWWwKAMnENmYP/DEBljx45NzIfPfvrppwf5UC7Ip11iOVBmPnTo0CAfjv+4fjMLTy+cOXOmFUJt21J9gWsvHDe5v6F8jcdb9A+GquD5CMdvlCjxPIun7PE1sIw4FvNaAE855LACOJ+if1j6yHNyGsATIrn8SbJCszBsAv4eYakaz385Kisrg8/YDvg7+NsP52f8rcPf47AOePoi+7cxwWM0fsa1CZ9Gj/XOYzaC/ZTBk3ux75SarLpUyfavWyGEEEIIIYQQQgiROfRCSwghhBBCCCGEEEKkCr3QEkIIIYQQQgghhBCpItsBYQDUwaIeNRZ3ijXkqA2PxQBBbXJSzA+GY/qgph7vxZpofJb169cnXh/LUSp63Fx9sA9mzJjhbY6r8Nxzz+W9Fmu+k2LzmIXxBDAOV48ePYJ8GDPokEMO8TbGDTEzu+eee7yNRxubhbp0LBPHl8FYCPgdszDGBfoRYzvx5z/84Q9BWk77Xyq+rys4dhjGHMM4DxwLobq62tt81PF7773nbY7PkjVY+4/1xHE+cCwaP368t9esWRPkw7rlPobjHrZz7h8Yp6lTp07evuSSS4J8GLtq+fLlQRrGoEDWrl0bfMaYGdwXeSxJujaO9XzUd0PSs2fP4DPGasG2jWOkWfjc3MfQh9jHeKzB63NbQmLHX2O8SrwG3tcsbMcc6wLn56zE0ML+gm2P1xzYd8aMGROkTZw40ds4n37ve98L8q1evdrb2GePOuqoIB/G33vrrbeCtHnz5nkb45KMGjUqyPetb33L2xgnyyxcG7z++uve5jEa/T1//nyrD3L9iuNJYcydpUuXBmmdO3f2NrZZbr8Yd4q5+uqrvY3jDsca5Pg8SWAbQZ+ZhbEScZ7929/+FuQbMmSIt7k9Llu2zNu4jua4dxhfDOvJbNe2Vd/w+h9/N2C8HB7LMN4qx3HEmJQ4z/I8g+MhjgEcpylWDmxfON9j/zUze+KJJ/Leyyxc2+IaFedjvn5awNhh/BsL51OOZ4ftdPjw4d6uqqoK8uHcir45+uijg3z4W4XnO5yTcYy/7bbbgnxnn322t3lts2nTJm9zLODGBMfow8/Yr+bMmRPk69u3r7dj7Rz9zb99cRzAeIMcP1LkRzu0hBBCCCGEEEIIIUSq0AstIYQQQgghhBBCCJEqGo3ksDbwts4kSUzsaHG0OV+SDDJf3hy4tZvLEZOTxY5Qb2huueWW4DMeE86SwyTZGB8HPX36dG+jhMUsPA71sMMO8zYfZ4xb2VEG+dWvfjXIh1KN+++/P0jDa2I5WHKD7YSP08Ut7Ci74Of6xS9+4W2US5p9IaPire1Zg58PJVXYdnirL24D5i3HKLHJIs2aNfNHit96661BGm6B57odPHiwt3v37u3tWbNmBflwO/x//ud/Bmmnnnqqt3GrPPcBlDH26dPH29zvsbzsN7w++n/u3LlBPhxXBgwYEKRhn8Mxmvsiygp+9atfBWlnnXWWme063tQHKKc0CyUrKNFkOQKOOyiPNgvHMuxvPFdhv8K+yDJX7Is8B6N/0Ydcl5jG7RY/4/dYaowSjFInKZzB4YcfHnxG6QtLBBFsC9dcc02Qdu+993p70qRJ3kbJhVnoR67b448/3tvnnnuut1mihH3n2WefTSzvDTfc4G1eO919992J36sLmjZt6iW7HBYA62TkyJFBGspfUVrGbZv7H4Jj78MPP+xtlqwMGzbM2ygJ576C/e+ll14K0k477TRvL1q0yNssy8Yxh9vptdde623s2zhWm4UyUg4F0tDwmIprbRwPeZxDeS36wyyc7w499NC8/873Rkkay8L79+/vbZbkYnlx7sP53czsoYce8jY+l1kou8T22bFjxyBfWqRT2O5xTFq3bl2QD+c0lumjD7Bt8+80nD/R5jUFSkC5H2HbwnmX5bA4NuLcbxbKKXld1Zjgfor9BWXX+DvSLPTPm2++mXh9rGcOwYFjPfYxlgmL/GiHlhBCCCGEEEIIIYRIFXqhJYQQQgghhBBCCCFShV5oCSGEEEIIIYQQQohUoRhaETjuVFJcq9gxn2iz7hw/49G8fC/US3NsmaSYIqVOixYtfAwBPjq9oqLC23x0MB6binXGR+ZifAo8dtcs1DBjbA88qtzM7Pnnn/c2xtL5/e9/H+TDo3uvvPLKIG3ChAnexjgGrI/He2NcLzOzp59+2tt4dPL7778f5EPdP8cuyB3hnXUtNsaRMAv9i1p4rn+MGYQxRcyyX2fNmjXz7Y/j4GA8Bo5nhHGiBg0a5O39998/yIefX3755SDt9NNPz3t9Ptoaj37HOC8crwXjQnA8GIwP0qtXL2/n4oflwHGZ49dgXCWMf4DxaszM/v73v3v7v/7rv4K0XLwxrs/6oH379olpsfgd2AdiMX2w7rgdoK8wH8cDwXk3dqw1xvXieRFj8HAcEZxD8F4HH3xwkC8WB6PUwLkFbX6mv/3tb97mufWvf/2rt6dMmeJtjqs3bdo0bz/++OPexr5sFq5huC/iPPnhhx96m+NTFgrGFuK5r76PoG/atKlvmxhb0Mxs2bJl3uZYRhz3MgfHusE5DWNtmYWxb3BNhPFAzcK4QG3btvU2r2Wx/BhH1CyMaYZ9m+M34RzCa2ocI3Bseu6554J8b7zxhrffeeedIC23tmkoOA4OPiPWBfsb16g8zuEa9eijj/Y2th+zcOxt0aKFt3GuMwvHb15vY0wnzMex2o477jhvc3xKHGMxBlFaY7bi2IjjGM8zCMf7Pfnkk/Pm4z6A18Q65ziW3H4QnE9x7sb4a2ZmH330kbfZN9gOOD5UYwbHR4z9yODvwKSYlmZh/2Afow9wnRWLmyi+QDu0hBBCCCGEEEIIIUSq0AstIYQQQgghhBBCCJEqJDmMwBLBJMkhSzBwSylu3eR8uC2Rt6bjVkSUprBMBa/BW1lLmf32289L67hecCtsTJaD8h/cSmtmNnDgQG/Pnz8/SMOjd1euXOnt0aNHB/m+853vePv+++/3Nh93jlINPuL8xBNP9PacOXO8fcwxxwT53nvvPW8/+OCDQdorr7zibfQ3y8Nwu3hDb8NvKLiPYvvB9sLbt7FeWTLBko+s8dFHH3mJ0ZFHHhmkvf76697musXxBm2Wq6E0eOjQoYnXQP/wcfdJR2Dztnncvs/XwOujPIDLi/IZ3G5uFspnsEwsRfja177m7euuuy5Iy0l3YvKFugJlembhs8aOFsfPPM+gVBFlL7H5DrfQ4/fNwjmTx//mzZvnTUPfchlZljV37ty85WAJXinD7fJHP/qRt3/96197m+e+7373u4nX+Od//mdvH3HEEd7+xS9+EeRbtGiRt6+66ipv/8///E+Q79xzz81bPrNQevzDH/7QkkApFo8/2J6wXzb0eL1t2zZ/vDvL0VDejG3ZLOw7+Az83LjWYSlKZWWlt1EGhmses1ByHVujxsYElJmjjIblc1h+TsMxENe8nTt3DvKhdPbVV18N0u68805vX3zxxVbfsB/RJzhG8VyFa0+U3fI10CexcRn7Q0zqh23ELBwf0d8s1cXxEWXbZuE8gu0J23SawH6KvwN57sO6w/W9mdmYMWO8jWsM9g1L85Ouh78Ruc1hH8OxkPsRhl/h8APYh3H84RAo+FulMYAyX6x3DkWCv7l4TYmgHJh/t6JPsD/H5KbiC7RDSwghhBBCCCGEEEKkCr3QEkIIIYQQQgghhBCpQpLDCLHTWnA7IG7BNzM75JBDvP3//t//8/ajjz4a5MNtu7ytHLeh4n1xWyiXI02nsVVXV/sT+77//e8Habh9mU9rwS3WKJlgGWC7du28PWzYsCANT2jB66F00Cw8cQlPsGB56AcffOBtlnjgtmPc9v/f//3fQT48fYolE7j9Gbcr45Z1s123rSNpkqMWE+xX6GuWe+HW+LVr1yamNTZwizXLRfA0SGyXvM0d+zNKcM3CE9nwpDKW0uC9UMbB7RrHQB6/UdqG18Dx1Sz5tDGzcMxBKeXYsWODfHgS2X333Zd4vfoGx0Wz5NOR+JQjfB70E18D65J9iNvmUUbD9Z90Cq1Z2IdjcnRsByzpQB/i/Fnq/Rx9wqctodzh/PPP9/aLL74Y5Lv22mu9/e1vfztI+/Of/+xtlM6zJPDUU0/1Ns5Vw4cPD/Kh9BHlh2bhCXkx0N88/mAaS7EakrKyMt++p0+fHqRhn+A+hicM4rjLp3QiLF/Cdo/XiEmnsV9yn8Lrcdry5cu9jX0sJjksVG517LHHBp9xHfWrX/0qsYwNAY8buLbAuZDnKpaBIjhXoR9ZEoqf8Ttcz7HQJEmno/M6FMd5vj5eI3YiblrgtUMOriucM7kt44mjKN/kukuSlx5++OFBPvwtFJNfY/2znB/nCQzLYha2C1yzpdWHxQLrpdC+yKfFItjvWTqKEla0dcphYWiHlhBCCCGEEEIIIYRIFXqhJYQQQgghhBBCCCFShV5oCSGEEEIIIYQQQohU0bjFsbuBNbKoK8Yj4W+44YYgH35G7TTHOMLrs64atc94FDHHg8L4B7E4SaixLgW2b99uS5YsMbMwJoeZ2dlnn+3tM844I0jDWBD47HxMKsbU4Lrt0KGDt7GeMSaEmfnymZmNHDnS2xxj50tf+lLiNR555BFvP/DAA97muDHdu3f3Nh/Rirr3QuG2UGr+ryv4OZOODOdYdBg3i2MhxI7gzQq5uuFYNEOHDvU2xjYyC48zxvrbtGlTkA+PY8e4AHxNjmuGYPwOPOqYY1rEYgpiW8B8fI2+fft6G8d8BmOncCyEd955x9u16b91BcfQwmfHOkafmYXxc7hO8Bo47nC9YhrOfRybB2MLxeJ3YJ3z3Mr9G2nTpo23sU2UQgytbt262a233mpmu/aHxx9/3Nt8lPq9997rbYzdMnr06CDfG2+84e3f/OY3QRr2xSuvvNLb2JbNzKZOnert008/3dsnn3xykO+iiy7yNsfMKjTeCLYZjhuDYF/H2HYNgXPOPxPHY+TPCK4BMIYPt8uOHTt6m+Nr4fiN4zDPi0n9ntc2sTkTv4dxRXFtbBbGYmIfYr/FeRbXzWZhvFCMp1gK8NiPz4jrS449iOtSrhdMw/g5HEsn6frcZrB/sB9xTk4qe74yIhjrEO+V1tg/WH+xtQLGQ2rfvn2Qhv0D4yXz2hw/Y3/muuN2hqBP8Xt9+vQJ8mF/4+thObDPJsUTayxgW8A+wOtc9Df/hkNw/cR9DO+Fv1NjfU98gXZoCSGEEEIIIYQQQohUoRdaQgghhBBCCCGEECJVSHJYA3BLJm6rHT9+fJDvyCOP9PacOXO8zUcn43ZD3K5oFkrSvvzlL3sbt16bhdth0yQra9Kkid/KihIiM7ObbrrJ27fcckuQNmLECG+fdNJJ3j7xxBODfLgdlLcCo1wDt+jzVnmUwuB3WJL2zDPPePuvf/1rkPbBBx94u1u3bt5mySFLFfeUNLWFYsLbqNFXuC07duw0S6AWL15czCKWJEnH3t9zzz3eZhkIjoe4lZ3rD+ua2yWOe+grHg/RP7gdnsuN92LZBZaRr4/gFnuWB+CYjRIZHttRllVKoDTELOwT6Df2E9YD950kmSFfA32FsgguE86tPNbi1nv0J2/dx3txOVDihu2gFKQVa9assdtvv93MzH72s58FaT/4wQ8Sv4fSdpRkYV8xM/vJT37i7TfffDNIw7nqySef9PZZZ50V5Bs3bpy3r776am9feumlQT6eC5GYzBBB38VCKqCEEaUaaaLYa4D6pLKysqGL0GDw+MLzXw6WRaOkOCadR5kqj1HYj3BsZMn4xo0bvc0yblyL4pzGYweWieWTKJXDOSVpXVHq4LNj/fDzLFiwwNvl5eVBGvoU/cZScpR6o99YGspzIYLzH5ad10AoDeb1HN4Pfz9xO2hsYL3g7wv2Y2ztg8Qkh+h/bDOx9ar4AtWSEEIIIYQQQgghhEgVeqElhBBCCCGEEEIIIVKFXmgJIYQQQgghhBBCiFShGFoROFYIas8xzgceX29mdsQRR3j7+OOPr9W9UQ+P8QlYm4v6ZtbhI7H4Ew3Bjh07fEwBrmesM9T+m5lNnjw5r83g8x511FFB2qhRo7yNcQ049gbGD5o/f763ubwYswSPhDcz69+/v7crKiq8HdNYs68aazys2sBac6w79BvHkcC4F5zW0MfA1zVNmzb17fa///u/gzSMB8jHFCfFpOL4cElHxJuFMSkwfgfHi0iKY8F9A8vIMSjwKOVY/EKM78QxM/Az9nWOA3jfffdZEg0ZDwGPaDdLjlfFMcEKjdWI1+PnxDrHeuQjrnHu43rF+BMYz4KfKymul1ly3JlSiL306aef+tgml112WcHfw9iSGM+me/fuQb4TTjjB21dccUWQhj5HH2Odm4UxBTHmB/bfuiA2dmC8EYytKURdg/OFWdj/MC4ar+swpirH48G1Co7LHM+IY/Dk+45ZOAZu2LAhSMPxENc+vA7C5+K+3rp1a29jTKi0rl3xedBvvL5ftGiRtzmG1pIlS7yNYyjPR9gOcM0Si2PJbQ7BdRnHbcP5Dudqs/DZsP2VwrzYkODvux49enib40Dib4hYjEhsCzwm4PeSfruIZLRDSwghhBBCCCGEEEKkCr3QEkIIIYQQQgghhBCpQpLDGoBbRdHm7YB4pG0xttzGjqnHbfh8RCtSylt/cYuymdlrr73mbZZMJEk4cVuomdnKlSu9/dZbbwVp/DkJrHfcfs0+6NOnT2I5UKpYKKXsq1KHt3Nj38Tt+Z06dQry4VZshiV0WeOzzz7zMgTeor506VJvz5s3L0g799xzvY1Svy5dugT5YlJoHL/QB+wPlj/k4L6C9+Jt+ZgWuxduA0fJiFkogcYjsB999NG85ctHrK3VNVyPWH+43Z23zKM8gbfJo7ylUDklygxROmhm1r59e2+jzMUs3K6PfZt9jX2dnxk/Y99mOUmaePnllwvK99vf/raOS1J38LyL/Nd//VdeW4j6BscelOa98847QT6UScfW9Ri2gtfKvXr18jaO5TyW4dzN8zH+bkA58dFHHx3kW716tbdRTmcWPguOxQ0pr98TcO2PclAOgYKceuqpwWdcO2A7YKkfzqdYXywnxXrl+Q7na/weh11AafqsWbOCNPQhylLT6sNigf6J/TbDdV1MEop9nes26V6xNbT4gsbdUoUQQgghhBBCCCFE6tALLSGEEEIIIYQQQgiRKiQ5jMDbC5O2XnI+PO0gtl0Tvxc72S52whRuReSTntJC7GQ6lvzgZ5Sc8BZrlK2gJNBs11ObcqBU1CzcGor+4W3fLNMSDQfLA3HreEwWh20w1h6zzo9//OPgM7Z7PG3VLJQ5rV271ttcfzgucRqf7pQjJjnE7fvsbxwTWMaB8jXcls+St1atWnmbt3rjqXt4+k9McsjXiEmn6ho+nQrlJjFZIcISe/RBTMqZJEVhCUZVVZW3ue8lyYaTJKl8X7OwHeCz4LwthBCFwCex4jqyX79+3r7jjjuCfK+//npB17/hhhv2oHS7wifY8no2B8/VOJ6PHTs2SBs8eLC3cT0cO+2tlMH1IdYPnqZqFv4eefDBB4O0iy++2NuFnjKPdce/R/A0RF6z4DyJfurZs2eQ78orr/Q2SlnNQskkhk7h0xsbM0khGpiYRDC2zsL1Da6pGzJMRZrQDi0hhBBCCCGEEEIIkSr0QksIIYQQQgghhBBCpAq90BJCCCGEEEIIIYQQqaLRx9CKxcfh2BsYKwTTWAdb6DGfSd9hko625+9hvKCaXL+hqa0+GLXIrDfnz6JxwHHksP9hnB0+Ehlj6bD+HY+yzjorVqwoOO8f/vCHOixJdmjImFkMxsZgkmIGmoV9B2PRmYV9B+cqjmuF18T5k/sbxrLCeFfMunXrEvNhbDCOdYFx29Ia40UIURpw/Fb8nYBjZa9evYJ8GEOrPuMsJsXMYmLrcp4fcEzF59+6dWsNS1caYKxJnDM5zuK8efO8PWHChCDtkksu8TbGvzrggAOCfDh3YfzIZcuWBfkOPvhgb2P8TjOzjRs3envDhg3e5rinyN133x18xmfGeZxjxDVm8Dcn/7bHPrH//vsnXiP2TgD7HOaL/bYXX6AdWkIIIYQQQgghhBAiVeiFlhBCCCGEEEIIIYRIFY1ecsiyCJRCxI77xu/hEa91DR7laRZuXx0wYEDi90pZcihEsfjTn/4UfL7mmmu8jVunZ8yYEeS78MILvc0ypOeff76IJRSi4Vi5cmXwuX379t7evHmzt3mLO847KH0wC6UKCPcj3E7funVrb/Px8JjGshf8jN9j2c+qVau8zXIevD4efz579mx+BCGEiMK/IXDsxNAGbdu2TbxGbH2OaYWu4zlf0rjJYD4eN3E852dGiVWrVq28jfNLmjj33HO9jT7k58a5JEZM+lcoKG9Eu7bg3Gdm1rVrV2+j5PWkk04K8t122217fO80gT7u3LlzYj7sVzGJIEoJ2Qd4L+xHhbazxo52aAkhhBBCCCGEEEKIVKEXWkIIIYQQQgghhBAiVeiFlhBCCCGEEEIIIYRIFY0+hlbseNxnn302+Hz11Vd7G2ORsF426Zr876hX51ghqLNFLTvH9SovL/f2woUL896XrydEVuF4V1dccYW3t23b5u1p06YF+VavXu1tPtYaj0QWIs384he/CD6PHDnS2927d/f2CSecEORbsGCBt7/xjW8EaRh7A+cZjtWCMSg5dgSCR8DznImfsZ9ybMlHHnnE23zU+hNPPOFtnMcvv/zyxDIJIUQ+fve73wWfH3/88bz5YuuI2O8QHFNru47HmFqxexVapldffTX43KFDB2/juDxp0qRCi1hSfP3rX/d2nz59vN2uXbsg32OPPZZ4DZz/Co2Rhv7l+sffgXy9pHYR+1155513BmkzZ8709oYNG7xdUVGRWPbGwBFHHOHtM88809v8W3z58uXexrpk7rnnHm9z3E5cq2DstmeeeaYGJW68aIeWEEIIIYQQQgghhEgVeqElhBBCCCGEEEIIIVJFmZmlSou2du1aW7p0aUMXo1HSo0ePYGvxniA/NgzyYTaQH9OPfJgN5Mf0Ix9mA/kx/ciH2UB+TD/F9GF9kLoXWkIIIYQQQgghhBCicSPJoRBCCCGEEEIIIYRIFXqhJYQQQgghhBBCCCFShV5oidRw9913W3V1tS1evLihiyJqiXyYDeTH9CMfZgP5Mf3Ih9lAfkw/8mE2kB/TT2186PRXd39Dhgxxt99+e4OXIyvP06RJE7d48eJG9cxZe56G8GEpPHfWnkd9Mf3Po76YjedRX0z/86gvZuN51BfT/zzqi9l4HvXF9D9PTXyoHVp1TMuWLa1NmzYNXYyikbXnKYSsPXPWnqdQsvbcWXueQsjaM2fteQola8+dtecphKw9c9aep1Cy9txZe55CyNozZ+15CiVrz5215ymErD1zmp5HL7SEEEIIIYQQQgghROooaCvXzTff7Kqrq93WrVvdZZdd5szMjRo1yr388suuurravfrqq27QoEHBdwYMGOAefPBBt2bNGvfhhx+6iooKd+utt7rWrVsn3qd58+bu+uuvd3PmzHEbNmxw69atc08++aQbPny4e/HFF/N+p2PHju6OO+5wS5cuddXV1W7BggXuxhtvdC1atNgl7xVXXOE2b97sHnjgAXfssce6Z5991q1fv95VVVW5hx9+2HXu3DmxbAMGDHC///3v3bJly1xVVZWbM2eOu/HGG91vfvMbd/TRRwd5x4wZ4zZs2OA2b97sPvnkE1ddXR38zZgxoyj1PGPGDP88u7sO/1vz5s3d/fff77Zs2eJ27Njhtm/f7iZPnhzUdc6H1dXVbufOnW7Hjh3us88+cxs3bkx8Hvbhhg0bXGVlpduwYYP79NNP87YD9uGyZcvcnDlz3KpVq9zatWvds88+644//njXpEkTV1VV1Wh9yP9+5ZVXuuuvv94tWbLEbd++3e3YscOtX7/eXXLJJUF/GTBggHv55Zfdjh073M6dO93OnTvd1q1bAz/y86AfN27c6LZt2+a2bdvmtmzZ4rZs2ZK3L6MfN23a5DZv3uw2b97s1q1bF/hw8eLF6ovqi6n2ofpiNvyovph+H6ovZsOP6ovp96H6Yjb8qL6Yfh/yX03er9T2ebAd1OTdz8CBA90jjzziVq5cuYsPayAbLSiT/5swYYK7/PLL3cSJE928efPcqaee6szMnXbaae6qq67y+UaPHu3mzZvnLr74YteqVStnZq579+7uxhtvdHPmzHGdOnXa5dr777+/mzp1qrv++utd+/btnZm5pk2buuOPP949//zzbvv27bt8p1+/fq6iosKNHz/eV1bHjh3dxIkT3cyZM127du12+c6IESNcZWWlq6ysdGeffbbba6+93L777uu++93vunfffdeVlZXt8p3y8nK3YsUK9/Wvf93tv//+zsxcy5Yt3Te/+U23ZcsWd9ppp+WtrxEjRuzS8IpZz7u7x4QJE3Zp3BMmTHDf+ta33PLly926devcV7/6VWdm7owzznC33Xabr+t8PvzKV77iZs2aVbAPc9f42c9+5iZPnuy2b9++SztgH44cOdItXrzY/fnPf3azZs1y7dq1c8OGDXNTpkxxl156qVu8eHGj96GZuRtuuMEtXLjQvfTSS66iosKdeuqprmnTpu6aa65xc+fO9f2F/ThixAj32GOPFdwXL7jgAjdv3jx36aWXutGjR/v2wd9HP5511lluwYIF7vLLL3c33nijmzlzphs9enTgQ/VF9cUs+FB9MRt+VF9Mvw/VF7PhR/XF9PtQfTEbflRfTL8P89W12e7fr9TmeWrz7mfkyJFuwYIF7vzzz3d77723M7NdfFjg/Wte6ZWVle7vf/973h1QZubatm3r5s2b53r06JE3fezYse5vf/vbLv9+1113uWuuuSbvd5o1a+ZmzZoV/FtZWZl799133SmnnJL3O+PGjXNPPvlkXgetWrXK9e/ff5e0xx9/PO/1LrroIvfrX/86731+9rOf7bI7rRiNe3f1XNvGPWHCBPfRRx+5OXPm5L12s2bN3Jw5c/L6MHevQnzI7YB9mLsG+nDfffd18+bN875BHzZv3ty99dZbfoBqzD40M/f222+7jRs35r1+rq7z9UW81+78mO/76Mfc97Evsg/Rj+hD9UX1xSz4UH0xG35UX0y/D9UXs+FH9cX0+1B9MRt+VF9Mvw+5rvP5kN+v1OZ5avPuJ19fzP1xX9zdX61iaL3yyis2ZswY27x5c970iy66yB544AFbunRp3vSHH37YOnbsaF26dPH/tt9++9npp59uv/71r/N+Z/v27TZ48ODg30aNGmVLly61SZMm5f3O/fffb126dLG+ffvukjZp0iSbP3/+Lv8+c+ZMKy8v3+Xfp0yZYqNHj7aTTjrJmjRpEqT99Kc/tdmzZ+ctw56wu3quLXvvvbeZmR1++OF5r719+3a766679tiH3A7Yhw8//LD17dvX1qxZ43143nnn2ZNPPul9gz7cunWr/cd//If/fmP24X777Wd9+/a1p59+Ou/1c3W9p30x3/fRj7nvf+UrX/F9kX1o9oUfu3btGvjQrHH7UX0x/T5UX8yGH9UX0+9D9cVs+FF9Mf0+VF/Mhh/VF9Pvw9q8X6kNtenL+fpiDvbh7mha8yKbvfDCC7Zz587E9AEDBtjFF19s//qv/5p846ZN7aCDDrKVK1eamdkhhxxi06dPt88++6zgcgwZMsRefvnlaJ7Jkyfb0KFDbcGCBQVdc+fOnVZWVrbLvy9dutTGjh1r3//+9+2OO+6wHTt22JIlS2zKlCn229/+1jZs2FBwuQtld/VcW9q3b2/z58+37du3J+ZJ8mHTpk2tWbNmdtZZZ+3Wh4W0g+bNmwcvJI888kibPHlykAd9+Nprr+32+RqDDw855BBbvXq1TZo0qcZ9EX2Y+5zkx0L78qhRo3xfzOdDsy/8+Oyzzxb0jI3Bj+qL6feh+mI2/Ki+mH4fqi9mw4/qi+n3ofpiNvyovph+H9bm/UptqM27n6S+mKMQH/prF17Uwtm6dav94Ac/sLvuuqvg7zRp0qRWjnTORdPzNdTaMm3aNLvwwgvN7PO31gcffLCNGTPG3nzzTRszZozNmzevaPeqS8rKynZbb0k+HDFihF1++eV2ySWX7PId9mEh7eDaa6+1Tz/9dLflLRZZ8WGTJk1260Oz/D6I+TB37ZwfC+3L1157rfpiLVBfTL8P1Rez4Uf1xfT7UH0xG35UX0y/D9UXs+FH9cX0+7C271dqSm3e/RSTWkkOd8eUKVPsy1/+co2+88EHH9jQoUNtr70KL9L06dNt1KhR0TwjR460adOm1agshfDpp5/arFmz7MYbb7Tvf//79qMf/ajo9yiUQiYNZP369VZeXh6t62L4sJBrsA/feecdO+GEE4I86ENO2xPS7MMPPvjAOnfuvNv+sqd+LPT76Md8PjT7wo/F9KFZuv2ovvg5afah+uIXpNmP6oufk2Yfqi9+QZr9qL74OWn2ofriF6TZj+qLn5NmH9bm/UptqE07SOqLOWriwzp5uieeeMK6du1qX//61xPzDB482Fq3bu0/b9q0yd588027+uqrE78zceJE++53v+s/T5482crLy+2UU07Jm3/cuHG2cuXKguWGMV555RU78MAD86bt2LHDmjdvnjeturra2rZtu8f3j7FmzRrr1KnTLv/esmXLvHWzbds2W7x4cbSuhwwZYoMHD97Fh/g8u/NhvnbAPqyqqrJevXr5cv7lL3+xMWPGWP/+/c0s9GHz5s3t5z//ebwyImTJh5s2bbLKyko76aSTEq85ceJEKy8v38UH/DwxPyb1ZfTj4MGDbfr06b4vsg/NvvDjihUr9siHZtnyo/rirqTNh+qL2fCj+uKupM2H6ovZ8KP64q6kzYfqi9nwo/rirqTNh7V5v5KjJs9Tm3c/+fpijtr4sMaR+PNF0Oe//v37u4ULF7r/+I//cJ07d3Zmnx8Recwxx7i77rrLLViwwHXv3j34Trt27dycOXPczTff7Lp06eKj73/pS19yL774onvmmWdcy5Ytg+/069fPLViwwI0fP94fEdmhQwc3ceJEN3PmTNeuXbuinBCwaNEi9/TTT7thw4a5Jk2aODNzBxxwgLvooovcsmXL3HHHHZf3ek2aNHGzZ892l156qWvatKkzM9e1a1c3btw4984777gTTjhhj+rZ7PPTHqdPn+7OO+88Z/b50aKXXHKJW7RokVu2bFneUyvGjx+/27oeOnToLj7cZ5993MKFC92LL77ofYjPc9ZZZwXXzbWDBx54wL322mvumWeecW3atAnawYknnhj4cOTIkW7RokXuT3/6kz+G9ZhjjnFvvPGG+93vfudPrWjMPjQzd8stt7iVK1futr9wX8w9z89+9jN3zz33uAULFrijjjoqeB7si8cff7xbuHCh+/nPf+7Gjh3rXnzxRffss8+6k046KejL2BfPPPNMV1FR4caNG+ePRD7ttNMCH6ovqi9mwYfqi9nwo/pi+n2ovpgNP6ovpt+H6ovZ8KP6Yvp9aFa79yu1eZ7avPsZOXKkq6iocOedd57be++9nZnt4sNC6sUKydStWze3YsUKV11d7bZu3eq2bNniqqur/V++4zjNPj/C8dZbb3UVFRVuw4YNbu3atW7y5Mnu6quvdvvtt1/e77Rq1crdfPPNbsGCBa66utqtW7fOPf30095x+f46duzo7rjjDrds2TJXXV3tFi5c6G666aa8R1/+8Y9/dJs3b3affPKJW7FihevWrVuQlnu+m2++Ofjec88954YPH+6eeOIJt3r1ardp0ya3bt0699e//tUdddRR0frr1auXe/TRR93KlSvdhg0b3Ny5c91dd921y/dqW89m5g4++GD36quvuurqard69Wr36KOPuuHDh7sJEya4LVu2uN/85je7XHvjxo1u69at7rPPPnPbt2/PW9f5fDh16lQ3bdo0t2rVqrzPwz6sqqpyixYt8s+frx2wD5ctW+bmzp3rVq9e7aqqqtzUqVPdxRdf7Jo0aeJ27tzpdu7c6T799NNG5cObb7457/VzPty5c6f75JNP8vYX9mNVVZVbs2aN27hxY+LzoB83btzotmzZ4j766CO3efPmxL6Mfty0aZPbvHmz27x5s1u/fn3gwx07drhPPvnEbdu2TX1RfTF1PlRfzIYf1RfT70P1xWz4UX0x/T5UX8yGH9UX0+9Dfp58dV3I+5WaPE+sHezu3c/AgQPdo48+6latWrWLD3fs2OGqq6vdD3/4w2g5y/7XEEIIIYQQQgghhBAiFdRthDAhhBBCCCGEEEIIIYqMXmgJIYQQQgghhBBCiFShF1pCCCGEEEIIIYQQIlXohZYQQgghhBBCCCGESBV6oSWEEEIIIYQQQgghUkXThi5ATVm7dq0tXbq0Xu61117h+74mTZp4u1mzZnltMzPnvjg48rPPPsv772Zme++9d95rm5l9+umnu7X5+nVNjx49rEOHDkW5Vn36sRQpKytLTON2Ukyy6MOmTb8Yxtq2bRukYT1v37498RrY1/fZZ58g7ZNPPvH2xo0bvV2ffY/Joh8LJdZ3kLrsR8UgrT7k+sfPOBc2b948yIefeW5Ngvvs5s2bvY1z4c6dO4N8/LkuSasfa0uh/S9GqfXNxubDrJJWP8bGVIR/a+CYGhvzcI3EvyG2bt2a9xp8vfrqs2n1YQz0J//Wa9mypbfxN2FsnN2yZUvw+eOPP/Y2+q0hx9ms+HG//fbzNj4Pr2FwrbJjxw5vcz9Cv/JvjaR2snLlyiAf9tm6pJg+rA9S90Jr6dKldtRRR9XZ9bFBYUM2M2vdurW3u3bt6u3u3bsH+fCHblVVVd5/NzPr2LGjt1u1ahWkrVmzJq9dWVkZ5MMf2NiJ6oK33367aNdCP/LAXexBmAcevH6x7xV7ltikhhT6oqQ29VZXPmxI2rVr5+2LL744SMMF4PLlyxOvse+++3q7X79+QdrChQu9/eSTT3p73bp1NS9skciiH2NgH0ab+xEuHrgfldqP6DT5MLYIw0V4586dvT1kyJAg36BBg7yNP8TYL3gvXsS++OKL3l69erW3cUFvZrZt27bE6xebNPkRKfQ/VTgf9jnsi4XOfWbh4j9WDkyry/VNWn0oQtLqRxxDzcIXUNjfysvLg3wDBgzwdtKLKbPwtwaOm2Zm7733Xt5r4BhqFv7HXl2OqWn1IZM0Th544IFBvlNPPdXb6Cf+3YJjIdfRu+++6230G6+B6vM/etLqR653XMdcddVV3sYXkWZmS5Ys8Tb+xxuvTbBd9OrVK0jD3yF4/euvvz7I9/777yeWv5gU04f1QepeaBUDbLD4ksosnDBOOOGEIK1///7exreW3ChxUPrwww+9zf8z0qVLF2/z5IENNtY5Xn31VW9z41uwYIG362syqg01KQ8O6uhHvkbsf5rqEyxv0s49zlfXL/iyAL5sHjlypLfPOOOMIB++nNq0aZO3+X+48AU17wrBvogvsZ577rkgH36vIdtcFoj9iEZiP3L5Ow25oy4NxF5a4eKK0/Al1pgxY7w9evToIF+nTp28jT+c2IcHHHCAt/E/bMzMevbs6W18uVxRURHkw/mO58ykubCxjbOFPm+hfTF2Pb4Gzt2xHQVJ831N7t3Y/CpKE3xJZRbfsYrrmz59+nj7ggsuCPKdcsop3sZxDW2zcEzltc99993n7alTp3p7w4YNQT5c3+BaKt/9GguxF/rob5z7Lr/88iDf1772NW/jWMh1uv/++3u7uro6SLvlllu8PWnSJG+zn3ANpPXQF+ALZfwtYGY2fvx4b5933nnejim4EF5/xHZV4jsC9PfixYuDfP/+7//ubXw/0NhRDC0hhBBCCCGEEEIIkSr0QksIIYQQQgghhBBCpAq90BJCCCGEEEIIIYQQqaLRxNBCvSvGA7n00kuDfEOHDvV2mzZtgjQ8QQ3jMrDWGeNmYT6OFfLRRx95m2OFYOBAjFHCmluMEcRB8x588EFvv/XWW96OnfZWasROf8G6jZ0kUax757vv7u6VVF7WX8diLiXFCqtJObIWRwT16rHTQjGGAMYMwJgGZmFMH44xgf0bY1vEApI3BpLadm3B+uQ4TdgHcPzi+yYFYuW02Al5jYmk+uIgxXi4AsaINAtjZR1//PHexhgQZmEsiVhsSczHaUcccUTefNxn8QAIbgf4ua4PUkkTSfGqeJzjtpFEUpwsvlfsRKhC+2axxyIhakvSOowPmcLfFxzL97DDDvN27969vc2nHM6fP9/b+LuG+w2OjxwUHsdUHNtnzZoV5Js7d663+VlwvG1MY2rSCb9m4W+zUaNGeXv48OFBPlyjJq1ROI1/w5177rnexnbw97//PciHB4yxnxrTuBlbG3LbbtGihbdjsT+x/nD9GvtNwn7ENoTX6NGjR5AP43DhewQuR2NDO7SEEEIIIYQQQgghRKrQCy0hhBBCCCGEEEIIkSoajeQQt+8NGDDA22eccUaQr3379t5myQTKJNDmfLi9F7c28rG9uH2RtyUmbV/E7Y+cxhJJ3IqIx36uXLky8V6lAG7jjUkOcVt17EhvvgY+L6YlHbvKxPLxVu+ke3F5Y5KJpKOB+dhd3L7K10/7Eb2xdoBbhHm78L777utt3KbLYD72IfYj3M4dK2Op9aliEOtHtb0GjonoA/bjtm3b8l6D+2Ksj6FUCtNYMp5lCWLMh1g/HTp0CPINHDjQ2yxtR5k+Hg+PPuN7x+TXuJU/5pvDDz888V7Tpk3z9pw5c4K0devWeRvn4MZAofJ4tHncRN9hn2XJDfouJs9IkhNzPpbgFDr+pCnEQn2R1BeLMcY3dpLW7ihBMwvHzYMPPjhIw3EU2y/Pi+vXr8973xgc3gTnYCwjSh3NQjn5jBkzgrQXX3zR2ytWrCioHGmE1xsoFUV/mpmdddZZ3sb5E33L4DjJYybOfTwW9uzZ09sXXHCBt7t06RLkQwkiz4ubN2/2dtb7fez5eB5DGS72Pwx5YBb6q9DQMCzfT5L6d+vWLcgX+y3TmNEOLSGEEEIIIYQQQgiRKvRCSwghhBBCCCGEEEKkCr3QEkIIIYQQQgghhBCpIrMxtDguB2rZUWPMx2FiTAiO34FadozZgdpjM7N27dp5OxafAD/zvTAOVyzuBT5X586dg7QRI0Z4++mnn/Z2VVVVkI/vXV8kPVdSXB2zMBYU+qPQ+Fex+8baDGqd+fuxI85RV43lxWvzNTjWGt6vVatW3uZYCBjXjdtkrA2lHewr3LZRe45xB9hPqEmvrKwM0rBeMWZFocfXp5natBvuR7H+gfEKunbt6m2OVYZxOTCWBF8v1o9w7MC4h9xXMK5SLCZejFgsr4Y81jwWz+HAAw/09nHHHRfkO/bYY73N9Yqf8Xoc5wNBX3A+vAbHqUiivLw8+Ix+47gzr7/+urdjcZ6yQCxWYywv+pT7WFJcQo5V1bZtW2+zj5PijbAP8HuxsuMcEIvXxbEksx4rJkehsdNi40MMxeHKD/4W+NrXvhakYcxbXGOYhWs7jNPE8xGuI7GfcjvHPoFrSLNwjMV4oexHXEsdeuihQRp+78knn8x73yzAv7H+6Z/+ydscB61Tp07exvUMxz/CsQvnKo7lhP6IxeFC/x522GFBGo7JU6ZMCdIee+wxb3NMysYE9w/sY0m/P82S40JyX8TfDeh7BtM4TjeXUXyOdmgJIYQQQgghhBBCiFShF1pCCCGEEEIIIYQQIlU0Gskhbu/FI1RZPoHbBqurq4M03IaZdIy8WSiPwW3ALFHC7aW8DRW3FqMsIradm7c24pZXlB+++eabVgrkys5ba7HOWO6Az4vbeLle+JoI+hz9yD7Araa4VZfLFNtOjNtS0T+8DR+lFSy3atGihbfRpyyvmz17trcrKiqCtJyMKq3SQ64vrEvsHyxRwnzYp1jKijIzrqOkrfy8/T8m3UgLMbkgPxPKCTAfy2mx/rhu8Vjpgw46yNssp8WxE/s2+wDHVPYxlhG/V5MxNeb/pHvxeBH7Xl0Tk0v36dPH28OHDw/yoRyU5QhYzzjesdwE+ynWAdcxjs+xPoZpOL+bhdv1eY5H2fCGDRsSy5tWYnJX/Bwb57C/8XyE9Yl9vX379kE+TGPZJ8t8c7D0Cn3M40/SugjlT0xMjpgFkub3QmWAsbAYojCwj/Xv39/bLNPD3xf8WwPHVOyXLHNKktryb43Y+gbBdsFrKezfLHk6++yzvf3CCy94m/tzGsH6Hz16dJB2xBFHeJvHSZSbIhziBf2B/uR1FI6FPK7jnIxpLJFMWkeZmb377rvenjdvXt6yZxX0Mf+GQ8kvwjL6pHUdywWxj8V8jO2Cr8H+F5+jWhFCCCGEEEIIIYQQqUIvtIQQQgghhBBCCCFEqsis5DAmbcHTnBjcDshbD1GegFvmWVKC2wgLlWBs2rQpSMNr4vbj2ClAvK0ft5TiCRy8NRa3q9anHCbnI64/lNgxuNUZ64VlJVhPvF0T6ywmTcR2gpKbAQMGBPlw+2fsetieYqfVcBo+W/fu3fN+xyw8XYZlQUuWLDGz7EgIsJ+iPI23c+NWe2xn+O9m4VZslrvh1vtCfZ1WeCsz1lnsxD+sC5YjxGTXKGPALfosrUg66Ynlati+WXKIJwNhn+JroGQ8Ni5jGVnCEzttMZe3FPoi+gPlMTjOcD4mSZbKW/KTTqxjX+PYzdfA+sd65XkR2xK32549e3p7+vTp3m6o034LpVBJc6FjFNdLTOqfBObr2LFjkIbzE48rOCfjuoXzrVu3ztvcF7FNYj4G+ze3E257aaPYY0ihp2CaJc8NMWl61iSe+cB6wtNX+fcEh4xAsD5RhsbSalyb4JqP+31sXY950f/8eyLWNlCu3qFDB29nQXKIfjvkkEOCNAwBwmMLtgOcW3jdnvQbMSYJZ38mncLOaw/8TcPrnkGDBnl7/vz5iffKIrFxCesW/RMLV4Tf4bVTbN2IvoudBC3yox1aQgghhBBCCCGEECJV6IWWEEIIIYQQQgghhEgVeqElhBBCCCGEEEIIIVJFZmNoMah9jh2pjmmsW0U9OMZzYG086tpRB4sxXMzicZOS4nnwNZDYkcuoveeYUg0Ry6WsrMw/Pz9TLNYUasBRw8x6ZowLwM+bFDOMj9nF+AcYwwfjaZmFMVm4PeER8RiTgPNhGmvsMRYQloOPVT722GO9XVFREaTl4r8VGhul1MF+hT7k4+CxP6PNxxIffvjh3mbNOx4Dj/EsOE5ZmuOD5MYAHguS4iOZhX0M4/Jxf8Z+xf104MCB3u7WrZu3uW4xRg6OAbHYTnzcMpYfr8HjPN8bwXaH7YLjUeC9OPZMKbUTrIcePXp4G2ObmcVjRyA4dnGcOqyHWGxJrEuOcYT3xrYaG0+xXZmFMfLw+TneSyn5aXdgXcRibcXia+H30I9ct7iWaNu2rbdxHjQzO+igg7zNMa5wnE66r1kYj4+fZdmyZd7GOZ7nAJwfuO2mITZJbF1XaBrXXVLb5vEUfR+Lx4b1yGu2WFpSfJ5Y3+NnLrV+iutXjEvIvxNwXcd9DP2AzxeLFYfrQY4fif2K57ekMSHWtrgt4JyP8YoXLlwY5Cs1XxUCrl9wjjQL64HHkqRn5XU7jlexNoE+5DEOP2M7498+eE2M/2UWxgZ+6qmnvF3qsSWLAa45uN1jGvqO1y34Gfsv/gY0C9dWfA38fYFtIRbLNY19qq7QDi0hhBBCCCGEEEIIkSr0QksIIYQQQgghhBBCpIpMSQ4LPa40dowwShUK3Q7IEhvcFoxbD3mrKW4b5C3CuC0Yn4W3HvJR2QiWH++F2/hLAd7iGZO0YBpuwY3VH2+jR5+jzduocUsubsflrbr4GduIWVjvKM/gbbwo2WLwGpiPpUz9+vXzNh8v/N577+X9TlpBv2E752OtsT9jnbPUAdsVb9NG2RNvH84CZWVlvo2x5Cd2VDTWLfYB7oudO3f2Nl8f5Qkoz+BxDuV9CEv9sEwxGTKOgSzjQKkiXwPLEZPXYV2xFKEhZb881mJ/wTED50uzsA/wNZKkSLFxHeuHv4/5UB5oljx+8b1atGiRmIYSRDxifvny5UE+nv/rm5pIq2Lyy6RrcH/GusV+xfWA42Pfvn29zfMiyrjnzp0bpOUk8GbhvMhyKFwzsYwe2w0+F443fC9+lly7boiwCzFi5UE/xfJhO+B8STJMXoegPIbTsF5RrsvrXLw3j4WF9ufYmr3UJDc4B2F4itWrVwf50D88L2I9oYSM6ytJBsjzFtYfjo1myXMa9xXsp7zOxXJgqA4ub0OPqbUBnzsmtcT+YBb6MElCahaulzAf1xVej/2bNA7w+igmKe3du3feMmVRcsg+wLrmMAdY71gv3AewbrHeZ8yYEeQ76aST8uZjsEy8Ro1Jjxsz2fh1K4QQQgghhBBCCCEaDXqhJYQQQgghhBBCCCFShV5oCSGEEEIIIYQQQohUkdkYWrGj0lHvz9p11KtzjImkWCGsuU06Sj4W0yUWUwmPZOXyogaX9bgY1wV1wBwfpaFhTT8+Bx91jDE2sC742WMxqbBe8Bpct3gN1E5j3BWz8Fhf1jbjNfFZWL/OcYcQbIcYw4ljoWE5+vTpE6Tl4jWUWqyQ2oL9GfsbH6uM8SFidYx9kdscxhlC3Xwa40Hkwznn65OfHdsvxx3A+CoYN4P7M/YXTuvVq5e38d4cBwfjWOB4y7G1cBzAWFhm4RiIz8VjBY63PFZWV1fnvdfatWstCY4DlRvr66svxuIrYQxGjPfCcx32HU7D58N8PD5hf8ExOBbPDuPRmIV1jj7kMmEaPzM+Z/fu3b09a9asIB/HRSslCm073GexnmLxFLH+OEYojofY74cNGxbkw7iTHFcJxwTszxxDC9dF7GNsN1he9KlZ+Jy8BsuN56U2L2J5uGyxGFpJabwmxTT0xaGHHhrkO/PMM729YMGCIG3NmjXenj17trdXrFiReK9Y7B/s69w2cd7ltsRxuRoa7C8Yrw/nFbOw3letWhWk4ZwUi9uJ+XB85bUsjsv8ewXnUPQPj384lrAf0V8HHXSQt0utXxUKlhtjAfIaMhZ7FcG642vgOIb9lNdiOI+xD5PmKv5dhNdnH2JbxXUaxzPNIugfjo2FdYv1x+sWrGscG994440g34gRI7zNaxOc7/BeeD2zbMbyLQbaoSWEEEIIIYQQQgghUoVeaAkhhBBCCCGEEEKIVJEpySFuG2Q5EG6Xrqys9DZv3cQtgElSESa2RRG3ofKW+aStjJyG24e5vPwZQfkbbmnmLaQNcexxWVmZf2auVywPb8nEusB6Z5kBXpNlDLhFHSUTuFXaLJRUocyC6wu32/PW3549e3obt/iyjAO3IfM18Dljx65jeVl6l0vjbcZpBZ8dJQgsVUiSlvFWbOzr3BexD2P9cf3jNvVSO0q8ULi/YbvkrfKDBg3yNrY3HudQ1saSQ+wfCMtIUK4Rk09UVVV5m+UO2BZQDsfHbR9xxBHe5rEd2wluA2e5Ad6bpTosk6xPuP/379/f2zjGsWQldg1s6/jcLLFJkvXGpG+x+RnLyPWP7Yx9iNJslP1w+8Z5oz77c5JMJybfQZ9gPu6LWNexI8OxPvv16xekoRwY50yu50mTJnmb+zN+RqkLy1Sxj/E8jm0h9iyYD8cisy/CT9TVvMg+S+orTKxPIDxe4/Vj/QPrefTo0d5GeZVZOMf17t07SMM+hm2CJTYrV670NvsJ10F4PeyjZmbLli3z9gcffJBYxoaA/YjrMJS987ob64L7Bz4T1kWsH7HEPuleLNnEesffTfxcKAnla2A7RJ/yWirWT0sJ7H99+/b1Nq9fcHzluQqvEZMconwM65jnnNi4jn7DsYzbC5afy4tpOC/i7+Wsgv0tJmHGNJ7vOnXq5O3Fixd7G38fmoXjAMtKk94dcJl4LhSfox1aQgghhBBCCCGEECJV6IWWEEIIIYQQQgghhEgV2dAf/S+4RZO36E2fPt3bXbp08TbKZsxCOQjLLpJOAOHtt0nbxXm7bez0F3wWlM/x1nHcrsrSvZkzZ3p72rRp3l66dGnivRoC3j6LW2HZj0l1xrKh2MleuM0T65NPO8N8uG2eQb/yNm1sGyhXwlPbdgf6lZ8lKR+2cbMvtn6n9dQZJkleHJPY4HdYDoW+5jaH8gzc6tvQUodiUVZW5vsSj3ko5UVpkFm4FR+3qPPJdHhSIEuDsT1j2+S6xe3wKK3gbd+xvpgko+MTQVEywtvtsZ2gBIdPN8VxmaXruWvW17ibdJKYWSgZwzTuA7w1HkFfoT/5O0mn+vK8mHTqk1my5JfnXMzHfsfvoe9ZBo7SgIaS5SO1katxP8L6jMmzY6dh4dyC8yK3LfzM8zOeCorz/XHHHRfkw1NFeezAPodjAj/XokWLvM3z5/Lly82s/qT4Mb8lrVliJ1XG5PEoIeb1wGGHHeZtHMf5XuhDXh/hvIv3Pf/884N8eHooy62xfeLpo9xn8V68fo2F3dhTcj6J9X9ubzhX4fPxibnobz61DOfaWCiVJHkZ+wrrncdl/B7KG3mNhOMhr1/x5Hjsp4VKZ0sN7H+xtTrKB1lSmnTKKMvFcOxKkg6ahfMk1yveC/tO7PcT/5bEvHj6NP52Nmv434t1TUz+i/XOYy/me+edd7y9cOHCIB++O+DfK0myUn7HkJXfHsUmnaONEEIIIYQQQgghhGi06IWWEEIIIYQQQgghhEgVeqElhBBCCCGEEEIIIVJFpmJoIawxxeOfX3rpJW9XVFQE+Y455hhvDx48OEhDrT7GFig0BghrojFmB+uqk7TUseeaP39+kPb22297G48ORb27WcNoovfaay+v9WYtMj47a4dRf4wxCWKxBTgeAx4xjXp/jDlhFurjMVYP32v16tV572sWxopBm58LddtJsX7MwjbDbRc18RxXIvcspRzTIBYzJgbWHfsGP2M74zaH1+By4Gfs67Hy1fZZGorcuILxo/jzoYceGqSVl5d7G9ssH22N/ZRje2A/iMVwwu/hGMBHYHfs2NHb3BaS/Ijxv8zCZ+GYO/g9bEMY/8UsPGae43y9//77u5SnLonFE8T4IDh+xMYJLjfWA7bzQuMS8b1wjuOxENsF3ouvgXHvuO9hXowVwjGgYrGoGgKs91jbQR/H6o/9g8+P7blHjx5BPqwXjA/HfXHFihXe5jUS3nvVqlXe/uCDD4J8+D2+PrY7nMfxembhXHjwwQcHabn6aYh5ke+JfsP64XaJ/RTHO7OwHnCdg/3BLGw/GN8MYyiZhe2e6x9jMeH3ePzHsXvevHlBGo7leC9e2+RineUrB8flKiaFzNvcFzH+FT5TbG3C8W/R/7jm43vh74ZY38Z7cTmwHcbWuegrXl9i+dH/HK+L58JSBccWnCM5zmIspi22ndiaIsk3sTUqjx1JczCvo3Ds4DQsI67t+F7cLrJOUuwy7rPYtjFGJMdQxfGWx8okP+J60ixsQzy3NmZK99etEEIIIYQQQgghhBB50AstIYQQQgghhBBCCJEqMis5ZHCbJB4FzeDWvkMOOSRIw23Vsa2huPUQt/7yVlPcwhuTzOE2Vz7mE48wnjFjRpCG27RxW3YpyJ+cc/6ZeetmbDsl1hNKJ3lbLMqI+vXrF6ThFuJjjz3W2yzPSJKV4nHhZqHkkMEtvujHxYsXB/mw/HzENm5ZRYkpb3+OSXXq61jyPaG27RL7H9cJti3cns/9CLf+8hHzeBwzXo9lSLE6TpJKlQJlZWW+7HiEtFnYV3h7NNY1jo2xLeoxySFvgecy5sC2zUeLY1vge2G9Y1vgcRklkrEj4bENsRQE65HrNHe/upIcxq7L8iWso5h8AmU+PF5jnaOvuY9hGtZ/rLzcV/AaMTkPEjuePCaPqS9JKJO7b0y2HJNFox/Zp+gTbgsoMzzqqKO83bNnzyAfzkdYJp4X0cc8jyeFb+D+jGsk9g9KoNGP/MyVlZXe5vkhV8ZiS0qTfIhjI5cT5w8cT1E6aGbWqVMnb6M0iPOib/i50VfoG5aEYZ3zvIjlRX/yM+N4wWnob+yX69evD/JhvfE1YuNWfRCb95PmLbOwzjgEBc9dObgfYb1g3+Z5C9P4XpiGZeQwKNjvWYacJHOLhc8oZXBOw7mPx0wcd3js4nk/B69zsP2iL3Ds43zsm6Q+gGtX/szruaQwLQ01DzYUPBckhf/hfl9VVZXXZj/ifMS/9ZLWwDz2ltpviFJBO7SEEEIIIYQQQgghRKrQCy0hhBBCCCGEEEIIkSpKX4dUB+B2XJYq4HZD3iqI8hvcfsvSCtxijafQ8DZg3PrLp9BgGm5/ZTkPbnNduXJlkJa0rbwUTmzaa6+9/PZdLk9MeoTbZLFe8LQqs3DL9kEHHRSk4UlHeD3enouSRvQ3+xHbBUoCzZK3gfNpctgOWbqB9YNbT1lugFtg8ZQNsy+eLSsnlCSdWMc+ROkGbtNetGhRkO9LX/qSt7k/Y/3HtvrG6ha3hOM1SsEfZWVl/pl5mzx+5n6JW/GxbfNWbNxWzSesInh9lnEjOAZyPuwfLFEq9ProE95unyQh52dGmRtv7c+Ne/V1sho+A/sXx8nYabpJ7TdGTHoak+zjvVlig8+C32MJRtLJRGZh+bE+UGpqFj5zbE4qNklytRhJp+LxCZ74THgam1m4VonJetE/mBY7bY6lOhj2IXaKGN6L+3PS+onbDI4/vKarK+lGznexkzNjJ3MOHTrU2/379w/yYT3EJCu4jmDJCtYRjt18QiT6hk9UxO9hmfi5MB+f+IU+RJkhz4tYXp6f+dTDhgbrKda+sO/w+IVjG9YRPzsSO8kQ6y92AiLOWzz+YNvgE4+Txkeeb/A3SinLprA9o83zItYRy2RxLMNxgK+Bvk7qU2bxkBno39i9UFLMslZsFyhr5vk566fqcbtMkvrzuINtG8de/r2IYzHfK2ntUwq/E9KAdmgJIYQQQgghhBBCiFShF1pCCCGEEEIIIYQQIlXohZYQQgghhBBCCCGESBWNJoYWalWT4geYhfpW1q3iZ4yTxceYo/YZ0/h6qIOOxSxBDTPrlzEmDce6QF07l7EUyNVH7PhYjqmBumWMj8QxtDD2w4ABA4I09A/WGcePQL05xhZgH2CMA9ZVY73HjsLFmFd8fYw3gtfneF14r6Q4FqUct6C2YFwU7s/Y52KxEDD2FqcV2nfwexx3AP2NZWI/NUR8u7KyMj/eYJ8yC+MK9enTJ0jDZ8RYGVxfOA7FYoBg2+SxDOM94PU4bg/GoWEfoP+xj/GR11hGjm2C5cLrcVwSHNO4r+fuXVd9MXZdHp8KjT+D7RLHRQZ9z+VIavccfwXrkusu6dlibY7nEASvz/ka4rjyvfbay7crHsuwXXJ7Qz/27t3b23379g3y4VzFMVqwr2P9cWxOrGvsO9wXsfwc6wn7Js6tHIcGx2WenzGGJD4X+w1jynAMrboYb5s2berrkuNOsd8QLCfG0OIxGZ8Bn9ssXA9i3fFYiOMa9ilcazDsQ7w3jv9cpzjWcpvDNoNtmNfKuJ7j9SL7tL7hZ2rTpo23sd55nom12aS1O4+9SfF1eZzEa2B/MwvrD+/LZVqxYoW3eR5PiqHGsWLxmqW0FuVnxbEQ+yX3I3wGjn3bs2dPb2N/4zUF1he2ER7/sYw8Z+I1MB+Pydi/cZ7g+2HsRV4jNHR/q2tiz5dUz2bhbzj0MY+HlZWV3ub5ANevPF6I3aMdWkIIIYQQQgghhBAiVeiFlhBCCCGEEEIIIYRIFY1ScohbAFmqgNsuY9tqcRsmbzlGUCrDW2xxSzin4dbW2BHnse2RuNUb7VLa6mu269G+WLe87RLrE7+H8gOzcFs6b19Hn8eOGk+SsvGWbfQjH9Od1J6WL18e5MPtqtwmkyRQvHUZ662ioiJIy201zsrxr7jdF/sE1z+2EfQTS0GSrm2WLLGJfS92vDOmlUJfbNKkia8nHl9QYsdjDW7LR/g4aJRgsMQS2zCOUSxVQB+gBIf9jZIlfpZCZdyYD2XCZmHfx3GfpSDo4+7duwdpuf7MdVEsuP3i87B8IEkGzXUXmzPx+ji+8PiUBM+zOGbGyoF9h/PFJDZYRix7odJEvncx2WuvvXybZl9he4mNc4ccckhiPuw73E+xf6PNsgjsL9juuf5w7OD6wv6N/q+qqgryoUSG11lJ4zKPS/g9li3m2n8x5aXNmjXzfX7w4MFBGj4fzxEoD0W5Eo8T6De+BvaDJAlRvu/lYFk5yk15nER5DM6nvJ7D/sblQJnW6tWrvc3tBSVWLLPcuHGj1RW5dsHlwfbCcwSuPbEPcz/C8CY8BmIdYh/gdop+xL7OPsC+wv0oSQaI/uVyxPoz3gulomZhW26I8ApJcL1iH0P/sp+wbcfCJOCzxtaQ2H9jc1qsf6DN10BJMj8ztk9sPzGZdBZhqWdSm2U/Yn1iGtczrov4GjjGJklRRTLaoSWEEEIIIYQQQgghUoVeaAkhhBBCCCGEEEKIVKEXWkIIIYQQQgghhBAiVTQucez/gjrYWOwE1ngnxbLi2AKoOUa9LB83jNrsWFwS1EuzljYpZgKXv9Ti9pSVlfnn79y5c5CG9cJ1i/ExMG4Dx/dBH3DcDNS2o+6dj8LFe6P2nNsFxi7AWFhmZp06dfI2HnvMcSAwtgbr3jENNfyxWCEYj8Lsi7ZbCr4vBvgc6CeOEYZ+wz62cOHCIN/w4cPzXs8sbEuxWHRJcfrMkseZUvCHc86XNxZrhceepKO6mZh/cHzEfs/Xw1gksVgS2D843gWWA5+FY0TgM3O8ESwvxmTjsQPLiLFS+N7FpJA2xu0S09COjYWFxuHj+sc6wXLw9fB7PC/i2IjtZc2aNUE+rGOOJ4PPie2M2wGmxWLoFJO9997bx6PhGFroA07DGDY457APcA3CMbTwGZPmHLNwbsF65r6CZcK4XmbhXIhxmjjuGsaa5BhdGA8M538+jh7j6vHYXhfzYrNmzaxHjx5mZnbOOecEaXhkO44fZuHzoQ85Ph/6g/tO165dvT1gwABv81oBY1dh/+C1B16f2wumYbwlLLtZGEOQ12IYlwnBedssXNti2c3MXnvttbzXKCbc32MxtLA+sdw87mPsL47zh+0x6Xpmyb9lYvGR+Br4GWPKcnkxHz8zXh/bBY8dOKbW1RhaG7gsOLbguMbPE4vZid/DcYfHyaQ5iNdAmMa/d7C9xNomjut8fSwXfo+fK+twLDT0eey3Kc6t6B9eS3H/Q9B3eN+keUuEaIeWEEIIIYQQQgghhEgVeqElhBBCCCGEEEIIIVJFo5EcJskMeKs5bu3j7e+43RC3gfPR2EnHa/MWz9gRrbjdMOloXrNQesBb8lGGUajMsr4oKyvzz8xSDywfSytwGzQ+H2+H79atm7dZtoJbQGPyFrwmbqnnLcO4PZ6fBaWFuJ2b74VtLbbdHmUKfK/YNXLPXAoSt2KAz4FthLdiJ/Uj7tu4hZelwSgHqG3fSdoSXgo0adLEt3WWGeCzsywiaSs+91lspyy1xWvGZNF4zZj8rVAZJPY/3hKeJGXicqEfefzB9sVbzHPjXrHbQa5ssetyWbD+8Fm5HWB9cZ3gnJQkc8fysc3b6RFuS1gOLCOXl2VaCD5z7GjsmISnrti5c6d/xlNOOSVIw3rnvojz0+DBg73NfQDHNu6L2IZjMie8N9ZZnz59gnyxcAsoPUNJLsqw+DPLfZJkMbwOQn/npIA5cs9WzHnx448/tmnTppnZrvLH/v37exvXKGZh/eNagdd8vXr18javI7p06eLtmCT64IMPzpvGawqUSLJvsI9hqIWY/BolT5wX2wv7GmWG3Ja4zMUk1++5fWC75/Kg1BbLxvlw3ItJ+JJCHvD3sK9zPmxDLKlKkn/zGIPzIo+v2Oew3WF75DKWEjHJIfqay4/9lNciSXNLTHIWk/3j9WNhMZJ+f5qFYz7Pz/g99D2326zD9YL+wnbCvwOxnrAu+Xrou5isFO/FfVHkpzRHFyGEEEIIIYQQQgghEtALLSGEEEIIIYQQQgiRKhqN5DAmM0RwWz9v10za1s5bnjEfbjWNbUmNnaaFWxRZmhjbipgkjykF2Zlzzm9F5+3WWC8sDcPnwNNz2rZtG+TD7/HWXaxr3CbKW0PxGljPfD08pZFPOUQ5BW715lO5EN7+jNuEsa64bhDeylpqMrc9BZ8ndipd0nb9mMyVpUdYz7XtR6Vc/zt37vQSApaToSyGxx6UyMTk0+wThE8AzMH+STr1lds5pnE50AfYh9k3KHOL9SPsiyzfwlPckiTpxW4TSdeLbZPHOiq0D/A8ljTfcd3hNWL5knzNaXg9libi6VOFnjjKc0hDzJPYF7nd4DPxycAoZUPJIUu3sP74tDiUVuDcytdIkgFyn0XZIsucFi9e7G0cA/DEQy4vn1aJJ/f17NkzsRx4wiKPCTlpfjFPjfr000993f7lL38J0nC9gbIms/CEWRxbWSY5e/Zsb/P6L6nv8L2wLWF/4PqPjR241sFyxE5PY6kUnnKIadxHYxJxPAmz2CSNAVjPLDfHz7ETKfEznthtliynZYlpUpnYVzg/cRqftpmD20zSmovvjW28vLw8yIf1wdK7hvxdwu0NJdyxdsnjGlKo3xBsE7G5j+c7/B7a/NsK13fsw6R5vLHJ3WJjD9YL1x/6BMcAHg9x/IpJg2P9ISZDbsxoh5YQQgghhBBCCCGESBV6oSWEEEIIIYQQQgghUoVeaAkhhBBCCCGEEEKIVNFoYmghqH1lPT7qUfn4Z8yLWnCOMZEUd4A1t6hvZh0sfkZNOt8rdpQranDx3qUQQwvBeA5mZp06dfI2xydIeiY8PtfMrE2bNt6OHeuci6Fhtmv9oR8xbghryhcuXJj3emZhrCyMT8GaevRrTB+PMSc4zg2WkeNK5OLNlJrviwHWZaFH5HLfRu06t6WkuD01ISmeXSlQVlbm+wjHqcHYG9zeuK5zcGwMrFtu91gXGMMhdrw3jodclzhe8L3wM+Zjf2OcCY65w3GNcnBd4FzB5airY+ZzbSwWVyw2f7B/kULj1OH1YjEmYjFFko6iNwvHVyxvTfoUj685uG7w3nz9uhpHy8rK/PNzPLu+fft6G+MtmYU+wJg43J8xNgz7B7+H4xz3AYxJtWrVKm9zPBmc+5YsWRKkLVu2zNtLly71Nj8z3hvvZRbOpxhTLBYL8w9/+EOQluv7SWNZbXDO+f7PManQT/jcnIbzE683cG3D/QO/h2MQ9+2kcZ3nt9j4kDTvcpkKjfsZGztwHuc1Fq6J6gru//iZY00lxZPkeRHbc/v27YM0rGvsE+wDbDNY77F4XTEf4HPxeIjl52ugj3MxAM12HSexHFynxeyDNYXrFdcHOOdzPvQhrw2w/mLxzbCO0Ob4V1j/3MeS4izz7ydsS/zbF79XaBzRLP6eYJJ+c3F7xbqOraXYrwj6AH3KfbEh+0opox1aQgghhBBCCCGEECJV6IWWEEIIIYQQQgghhEgVmZUcxra6xrY24xZelobgVkHcXp90LDtfg8vEW6eT7hWTqCRJYMxK+zjPHTt2+O3+eMS9WVi3/OxJx+nyFmiUMbLEBP2D1+NjzPEaKEvibbx4fDVLDFDGgdv8+bnQjzHpI25lZdkOpqH8EO+dxa2qWF/cx5L6H2+3xjbHckRMY0lMbYhJAxqCnTt3+rbEsqHY1mncHh3bKh+TvGF7xPYcO44er4HyG7Owj3G/x76O9R47Vjm27T8mK4xJJnPXKPZ2/dw4wdfFOkY5iFmybIivgeNrTJoXk9hg30mSjpvFx7ikuZt9jc/J8yz2bxwf+LlYaofUpdQiN57FZE5cL9hfUI7I6wPs3xs2bAjSUKZeXl7ubR4DcL7GMvG8NW/ePG/PmjUrSFu8eLG30afcPtFXLC3DZ0HZP/dZnMdxrjaru76Ye6aY9Iv7B/YjrAeuf/QTryOwT6Ada0v47LGxkO+VVGd8jdi4ktR+OF+hcrq6gp8pJkfG9ofyMpa2o4+5P2M7xTRet+C9cQzg+sN783iLcyaOgbh2NQvbK4+p2DZwfYzhAczCMbUYa6liwT488MADvY0+5DkhFqIGfYDXiLVXbDs1GZOSpMG8jsJrcv3j+gifBeXcZtmXHLKPq6qqvI39LxaOIrZ2QGKhEnCM4XrOYr0XA+3QEkIIIYQQQgghhBCpQi+0hBBCCCGEEEIIIUSq0AstIYQQQgghhBBCCJEqMhtDK0YshhbGConFj0FtMscWQF0s2hzXCL/HcR9Qq4vlZe1s0vG+ZqG+O3aNhsA552MD8PHVqLuPxVJAvX/Pnj2DfKgBj8WzicXSwaPFk2KwmYXxBDgWArYnbGuxo1tjR8RjWiymBcZFwHJkMYYW1kmhx4Iz2Oa432MMlmKQ5E9Oqy927Njh4+lgLBozsyFDhniby4YxbDDmBMczwrGMr4FxB1q3bu1t9gF+D6/PcXViPsZrYD6O/YP3jsWBSjqe3SyMWcLtJzdeFDv2S+75uE3Fxj8sdyzmDj43x0VJmic5JlVSvBwek5LGeLPk8ZrrH33NPky6Hs+fsX4ai32xJ3z22Wc+ds3s2bODtLZt23p7wIABQRr2xQULFnib5yPsi126dAnSMIYKzmlcfxh7q127dt6eP39+kG/GjBne5jk+aUzl8QF9wvG11qxZ422MG4b/zt/DuF54/WLPi4XEs2OwLcZik2G7j813xZ5nuB0kXT8WJ6tQYuUrhXgyeE8eD7GeMJ4Ujy+4BuQ6wvkjFmsqKa4etwtcD3L8q9h8iuAzc/9NivcTi6fW0MTW0p06dfJ27LcE+gbHQr4m9u3YXIXzHfsQv8djApYR65jnZvR1bN2D5cB1WWME+wuvRxD0MY4JNVmP4ZgQi7mc9ThmtUU7tIQQQgghhBBCCCFEqtALLSGEEEIIIYQQQgiRKhqN5BC35SVJT3YHbvPDrfssscGtv7iNM3asK5cjaWsob1dFmQAf6Vvb56wvcuXjra/r1q1L/A5vvcyBkguzcGtorF743klg/fH18Bq8vTRpizVvI48dA43XwHyx4+75uXLPnJXtqYXKGFDShtvk8Rhls7C/8XHVKC+ObRculKSxqKHYsWOHlzhMnTo1SMO64DZ75JFHeru8vDzvd8ySt95zXtxuzdv3UYKBfXv16tVBPpQXcR/A68dkhfgZ249Z2Oew7Dxmofxq5cqVea9fX77HvsJzENYR+onlXTin8RyEn2N9BesO5RTcJtC/XEcof4hJW9BvLLFJOl6bZeCxMbk+fMfSOWxvLClHqVDv3r29zXIRlNzFjqDHumBp4vLly72N9fDuu+8G+SorK/Ne2yxsT9gueIzBtsBpHTt2zHsNHCvMQrkjlr0u2dP2EZsjGkq2VWgohNizx+btUpgLmVyZeIzC9hwLM4HfY5neihUrvM3rEZxPsA9wn8W1KObjekaZMP9ewfLi9bB/cXlXrVoVpOG9ee5ASsnHWBaW5uE8GWuz+Kw8tyb95uJ/x3Lwb4ukfEyShJ+fC5+F12lJoXL4ubIIPi/7J6k/c91iH8Y+xmuT2NiO14yFucG1WiykQmNDO7SEEEIIIYQQQgghRKrQCy0hhBBCCCGEEEIIkSoapeQwtmUb01jehtsI0ebr4TZ/zMfbSXGbY6Gn3/A2x6RTqvhzKW31Nfu8bDkJQexEMy43bq/E+uNtlzFpXtJW+dhpW7G6LEbdJp1wVJPrx06yrKtTufaUmJ8KBbf889ZclJ9gO4ud4sblwK3ZxZDulqLsIvfMLI/DumBZL8p3unbt6m0+CQZPQFy8eHGQhvIv3NrNdYRjKqbhiW58DZYLJp1QwxIllM2xDA3HGbwXSw7xhFSu09xptw0hA+d6xTpK2u7OsGQlqb64H+E18TuFjrtcfvQFS3GwzbG0AtsqXqMYcuI9ZefOnb6fLVmyJEhDSQPLEfEEZexjKDUyC5+X+0PSaZWxuQQlN2vXrg3yoSySZVlY1zE5P67BeD323nvv5b0G90X8zCdN5+5XauujUqQmJw8W45qlBJczdlI69jGUAldVVQX5sG++8sorQRrOd3h9vheepoayMz4hD8vP/SjppEweD7H8gwYNCtJQ+h+bx3mcLlWSQhewdBrlzDjnm5l169bN21gPMRkgEhszY7JOLCPP1RUVFd4eOHBgkIbtthRD1NQX7A+cM3Au6dy5c5AP6z32WwP9yO0p6dTlmExVfEHDr+CEEEIIIYQQQgghhKgBeqElhBBCCCGEEEIIIVKFXmgJIYQQQgghhBBCiFTRaGJoJcE68aSjq81CXSxqWDlWC4IaZtauoy42pqXFWBexI0D5Gvi5lDW3sRhXMWKxtgqNTxY7brqhYpAV416lHEML4yjEjmUv1G8YW4bj5WAsCuw7HFsA+zpfA/ttUpyZmoDliMXmaAg4ngbGxmD/YPyINm3aeJtj8+A4xLF/kuD2ymNnDo6dh/eKxS7AGBRcXmwbGOvJLPQXxt7isRfTOBZGXVNo3Zklx4LhPoDPwPMiXr/Q8RTHAK479FOsLcXiWOK9Y/FGsOzc9vHeDTF/xto298UPP/zQ21ju2DHwXGfo19jzYp1hXD0eu/DevG5JGue4TNw2ELzm1KlTvY1xhczCWF5oC1FbsH9g3zMLY0316tXL2/Pnzw/yrVixwtscnxLH0SQ73+ccsd81sbVhUnxZs7DPcuxKHI+w3/NzxWKVNiRcFoybhPMCjx8Yxw/XQ2ZmlZWV3o7F5cR6xjQeI/Ez12tSLEie0+bNm+ftAQMGBGnHHXect7E+OCZh1onF0MW1BM/BCMbw5N8MsZiR2E/Rp7F5UHyBdmgJIYQQQgghhBBCiFShF1pCCCGEEEIIIYQQIlU0SslhbDs9btdHKZNZeEwnbrvno2hrI1GKSY1Q7sHbHFu1auXtUpMv1Sf87LWRLdYkrRSJlbdUt3cnbZXmfAzmRbkbH6V7wAEHeBvrp2vXrkE+7Ecsc8Jr1la6id/DsvP40NBHJ3Odx2RouO0dt+izH2MShCRi/Tnmg2L02SQ5q1lym4zVW32PI1yWJImYWfJR7+hPs1BCuXLlyiAN+wf6PtZ+sYwsyUQJf0xGjzaXNyaTWL9+vbdRQsDXwPI3xFzAchFcZ8QkRVifXH/4HIWOL9zfYlL/pHy1BdsJlwPHI6ybmHwrJsEUdU/a1lRJFCrNw/ZbXV0d5MPPLC8udXAMNTPbsGGDtzHMA49hxQjZUBdwuXAuQP+inNTM7P33309MQ8khEgsbEwtHUZs1RWxdsnDhwiAN2yPO9+zDLII+5nkR6wzl7ByOAr/XsWNHb8f6AP7uMAt/e2A+nrdK6fdcKaEdWkIIIYQQQgghhBAiVeiFlhBCCCGEEEIIIYRIFXqhJYQQQgghhBBCCCFSRWZjaBUa94F17Rgz5Ze//GWQVl5e7m3UyOIRnWZmbdu29TbG8GGdNupiOTYWarjR5mPv58yZ423WtaeFmB485sfGDtdNLH5NbWM/1QVYNrRjZYzFqcCjiG+++eYg31VXXeVtPF778ccfT7z+2rVrg7TnnnvO2xjroiZtE+MYYT7W15da+8bycWwx/IzjF8ZAMguPui5GzJ26rqNYjKCk45OLEe+irsBYQ3zcOs5xAwcO9DbPJXPnzvU2HzvevXv3vPdt2bJl8BnjHOFciH3DLIwBxf0DY15hvuXLlwf5MIZWly5dgjR8ziVLlnj7nXfeCfJhTJRSiFmB64VC50yOAVdoX0RKpV9y7JmkI+55nYVtpqH7Ym2IHfueNrKynsNyv/XWW0Fahw4dvD1lyhRvz549O8iX5nhuPN6+8MIL3sb2+sgjjwT5sC+WEhx374EHHvA2rht5PsI5gttyof5NWvcWo2/ErvHMM88En3v06OFtnBfffPPNIF8pzIXFBuuJn++ll17yNsbQ6t+/f5AP28JDDz2U99pmZk899ZS3eY00ZMgQby9evNjbvDbJog+KgXZoCSGEEEIIIYQQQohUoRdaQgghhBBCCCGEECJVlJlZqvb8rl271pYuXdrQxWiU9OjRI9hOvSfIjw2DfJgN5Mf0Ix9mA/kx/ciH2UB+TD/yYTaQH9NPMX1YH6TuhZYQQgghhBBCCCGEaNxIciiEEEIIIYQQQgghUoVeaAkhhBBCCCGEEEKIVKEXWkIIIYQQQgghhBAiVeiFlhBCCCGEEEIIIYRIFXqhJYQQQgghhBBCCCFShV5oCSGEEEIIIYQQQohUoRdaQgghhBBCCCGEECJV6IWWEEIIIYQQQgghhEgVeqElhBBCCCGEEEIIIVLF/wc+MiTbVDFDPQAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 1440x288 with 20 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "n = 10\n",
    "plt.figure(figsize=(20, 4))\n",
    "for i in range(n):\n",
    "    # display original\n",
    "    ax = plt.subplot(2, n, i + 1)\n",
    "    plt.imshow(imgs_test[i, 0])\n",
    "    plt.title(\"original\"); plt.gray()\n",
    "    ax.get_xaxis().set_visible(False)\n",
    "    ax.get_yaxis().set_visible(False)\n",
    "    # display reconstruction\n",
    "    ax = plt.subplot(2, n, i + 1 + n)\n",
    "    plt.imshow(decoded_imgs[i])\n",
    "    plt.title(\"reconstructed\"); plt.gray()\n",
    "    ax.get_xaxis().set_visible(False)\n",
    "    ax.get_yaxis().set_visible(False)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Improvements\n",
    "\n",
    "Even though this very basic example seems to work, several improvements can be made over the original model. First, we can see that the overall framework does not depend on the exact nature of the `encoder` and `decoder`. \n",
    "\n",
    "***\n",
    "1. We rewrite the original class to accept any type of architecture for these (see code below). \n",
    "2. Fill the missing code to have a new AE model\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "class AE(nn.Module):\n",
    "    def __init__(self, encoder, decoder, encoding_dim):\n",
    "        super(AE, self).__init__()\n",
    "        self.latent_dim = encoding_dim\n",
    "        self.encoder = encoder\n",
    "        self.decoder = decoder\n",
    "\n",
    "    def forward(self, x):\n",
    "        encoded = self.encoder(x)\n",
    "        decoded = self.decoder(encoded)\n",
    "        decoded = decoded.reshape(-1,1,28,28)\n",
    "        return decoded\n",
    "\n",
    "######################\n",
    "'''\n",
    "class Reshape(nn.Module): \n",
    "    def __init__(self, *args): \n",
    "        super(Reshape, self).__init__() \n",
    "        self.shape = args \n",
    "         \n",
    "    def forward(self, x): \n",
    "        return x.view(self.shape) \n",
    "\n",
    "def construct_encoder_decoder(input_shape, latent_dim, hidden_dim, output_shape):\n",
    "    # Encoder network\n",
    "    encoder = nn.Sequential(\n",
    "          #nn.Flatten(),\n",
    "          nn.Linear(input_shape, hidden_dim), \n",
    "          Reshape(-1,28*28),\n",
    "          nn.ReLU(),\n",
    "          nn.Linear(hidden_dim, hidden_dim), nn.ReLU(),\n",
    "          nn.Linear(hidden_dim, hidden_dim), nn.ReLU(),\n",
    "    )\n",
    "    # Decoder network\n",
    "    decoder = nn.Sequential(\n",
    "          nn.Linear(latent_dim, hidden_dim), nn.ReLU(),\n",
    "          nn.Linear(hidden_dim, hidden_dim), nn.ReLU(),\n",
    "          nn.Linear(hidden_dim, output_shape), nn.Sigmoid(),\n",
    "          #Reshape(1,28,28)          \n",
    "    )\n",
    "    return encoder, decoder\n",
    "'''\n",
    "######################\n",
    "\n",
    "class Encoder(nn.Module):\n",
    "    def __init__(self, input_shape, hidden_dim, latent_dim):\n",
    "        super(Encoder, self).__init__()\n",
    "        self.input_shape = input_shape\n",
    "        self.hidden_dim = hidden_dim\n",
    "        self.latent_dim = latent_dim\n",
    "        self.fc1 = nn.Linear(input_shape, hidden_dim)\n",
    "        self.fc2 = nn.Linear(hidden_dim, hidden_dim)\n",
    "        self.fc3 = nn.Linear(hidden_dim, latent_dim)\n",
    "        \n",
    "    def forward(self, x):\n",
    "        x = nn.ReLU()(self.fc1(x.reshape(-1, 28 * 28)))\n",
    "        x = nn.ReLU()(self.fc2(x))\n",
    "        x_encode = nn.Sigmoid()(self.fc3(x))\n",
    "        return x_encode\n",
    "    \n",
    "\n",
    "class Decoder(nn.Module):\n",
    "    def __init__(self, latent_dim, hidden_dim, output_shape):\n",
    "        super(Decoder, self).__init__()\n",
    "        self.output_shape = output_shape\n",
    "        self.hidden_dim = hidden_dim\n",
    "        self.latent_dim = latent_dim\n",
    "        self.fc1 = nn.Linear(latent_dim, hidden_dim)\n",
    "        self.fc2 = nn.Linear(hidden_dim, hidden_dim)\n",
    "        self.fc3 = nn.Linear(hidden_dim, output_shape)\n",
    "        \n",
    "    def forward(self, x):\n",
    "        x = nn.ReLU()(self.fc1(x))\n",
    "        x = nn.ReLU()(self.fc2(x))\n",
    "        x_decode = nn.Sigmoid()(self.fc3(x))\n",
    "        return x_decode.reshape(-1, 1, 28, 28)\n",
    "\n",
    "\n",
    "\n",
    "latent_dim = 64\n",
    "hidden_dim = 256\n",
    "img_size = 28*28\n",
    "\n",
    "#encoder, decoder = construct_encoder_decoder(img_size,hidden_dim,latent_dim,img_size)\n",
    "encoder = Encoder(img_size,hidden_dim,latent_dim)\n",
    "decoder = Decoder(latent_dim,hidden_dim,img_size)\n",
    "model = AE(encoder, decoder,latent_dim)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Second, we only used the train dataset, whereas the use of a validation and test sets allows to ensure that we do not overfit our model.\n",
    "\n",
    "***\n",
    "1. Write a separate train function, that can be called on different sets\n",
    "2. Re-write the training loop to test overfitting\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch: 1 | train_loss: 4728.1826171875 | valid_loss: 3377.00927734375\n",
      "epoch: 2 | train_loss: 4038.50732421875 | valid_loss: 3399.588134765625\n",
      "epoch: 3 | train_loss: 3652.517578125 | valid_loss: 2836.024658203125\n",
      "epoch: 4 | train_loss: 2988.138671875 | valid_loss: 2425.7646484375\n",
      "epoch: 5 | train_loss: 2803.21044921875 | valid_loss: 2296.773681640625\n",
      "epoch: 6 | train_loss: 2552.16650390625 | valid_loss: 2042.8253173828125\n",
      "epoch: 7 | train_loss: 2676.5 | valid_loss: 2243.31787109375\n",
      "epoch: 8 | train_loss: 2547.62255859375 | valid_loss: 2079.4404296875\n",
      "epoch: 9 | train_loss: 2492.85009765625 | valid_loss: 2084.994384765625\n",
      "epoch: 10 | train_loss: 2365.85009765625 | valid_loss: 1952.626220703125\n",
      "epoch: 11 | train_loss: 2317.473388671875 | valid_loss: 1905.071533203125\n",
      "epoch: 12 | train_loss: 2271.8291015625 | valid_loss: 1855.13720703125\n",
      "epoch: 13 | train_loss: 2261.1240234375 | valid_loss: 1812.0244140625\n",
      "epoch: 14 | train_loss: 2291.6943359375 | valid_loss: 1931.838623046875\n",
      "epoch: 15 | train_loss: 2244.8828125 | valid_loss: 1749.9288330078125\n",
      "epoch: 16 | train_loss: 2166.368896484375 | valid_loss: 1728.418701171875\n",
      "epoch: 17 | train_loss: 2065.79638671875 | valid_loss: 1690.25830078125\n",
      "epoch: 18 | train_loss: 1982.7806396484375 | valid_loss: 1585.57470703125\n",
      "epoch: 19 | train_loss: 1973.1529541015625 | valid_loss: 1585.3863525390625\n",
      "epoch: 20 | train_loss: 1892.642333984375 | valid_loss: 1519.71875\n"
     ]
    }
   ],
   "source": [
    "######################\n",
    "# Learning rate\n",
    "learning_rate = 1e-4\n",
    "# Optimizer to fit the weights of the network\n",
    "optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)\n",
    "for t in range(20):\n",
    "    full_loss = torch.Tensor([0])\n",
    "    # Forward pass: compute predicted y by passing x to the model.\n",
    "    for i, (x, _) in enumerate(train_loader):\n",
    "        y_pred = model(x)\n",
    "        # Compute the loss.\n",
    "        loss = loss_fn(y_pred, x)\n",
    "        # Before the backward pass, zero all of the network gradients\n",
    "        optimizer.zero_grad()\n",
    "        # Backward pass: compute gradient of the loss with respect to parameters\n",
    "        loss.backward()\n",
    "        # Calling the step function to update the parameters\n",
    "        optimizer.step()\n",
    "    \n",
    "    with torch.no_grad():\n",
    "        for i, (x,_) in enumerate(valid_loader,0):\n",
    "            # forward and calculate the loss\n",
    "            y_pred = model(x)\n",
    "            # Compute the loss\n",
    "            valid_loss = loss_fn(y_pred,x)\n",
    "            \n",
    "    print('epoch: {}'.format(t+1), '| train_loss: {}'.format(loss), '| valid_loss: {}'.format(valid_loss))\n",
    "######################"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "r4gv6G8PoRQE"
   },
   "source": [
    "## Exercise: Denoising AE\n",
    "\n",
    "Imagine (for the sake of argument), that we choose an encoding dimension which is of same dimensionality as the input one. Then, one huge problem is that nothing prevents the AE from simply learning the _identity_ function (try to imagine why). An autoencoder can also be trained to remove noise from images. This type of _regularization_ prevents the model from learning this degenerate situation.\n",
    "\n",
    "In this exercise, you will need to create your own denoising AE, by relying on a noisy version of the Fashion MNIST dataset (adding random Gaussian noise to each image). You will then train an autoencoder using the noisy image as input, and the original image as the target.\n",
    "\n",
    "Let's reimport the dataset to omit the modifications made earlier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "id": "uJZ-TcaqDBr5"
   },
   "outputs": [],
   "source": [
    "class GaussianNoise(object):\n",
    "    def __init__(self, mean=0., std=1.):\n",
    "        self.std = std\n",
    "        self.mean = mean\n",
    "        \n",
    "    def __call__(self, tensor):\n",
    "        return tensor + torch.randn(tensor.size()) * self.std + self.mean\n",
    "    \n",
    "    def __repr__(self):\n",
    "        return self.__class__.__name__ + '(mean={0}, std={1})'.format(self.mean, self.std)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "aPZl_6P65_8R"
   },
   "source": [
    "Here, we create two new train and test sets by adding random noise to the images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "id": "axSMyxC354fc"
   },
   "outputs": [],
   "source": [
    "# Load the datasets and use our Gaussian noise transform\n",
    "train_valid_dataset = torchvision.datasets.FashionMNIST(root=dataset_dir, train=True, transform=torchvision.transforms.Compose([torchvision.transforms.ToTensor(), GaussianNoise(0., 0.1)]), download=True)\n",
    "train_dataset, valid_dataset = torch.utils.data.dataset.random_split(train_valid_dataset, [nb_train, nb_valid])\n",
    "test_dataset = torchvision.datasets.FashionMNIST(root=dataset_dir, transform=torchvision.transforms.Compose([torchvision.transforms.ToTensor(), GaussianNoise(0., 0.1)]),train=False)\n",
    "# Create loaders\n",
    "train_loader_noisy = torch.utils.data.DataLoader(dataset=train_dataset, batch_size=batch_size, num_workers=num_threads)\n",
    "valid_loader_noisy = torch.utils.data.DataLoader(dataset=valid_dataset, batch_size=batch_size, shuffle=False, num_workers=num_threads)\n",
    "test_loader_noisy = torch.utils.data.DataLoader(dataset=test_dataset,batch_size=batch_size,shuffle=False,num_workers=num_threads)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wRxHe4XXltNd"
   },
   "source": [
    "Plot the noisy images.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "id": "thKUmbVVCQpt"
   },
   "outputs": [
    {
     "ename": "RuntimeError",
     "evalue": "DataLoader worker (pid(s) 15704, 22172, 1456, 15416) exited unexpectedly",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mEmpty\u001b[0m                                     Traceback (most recent call last)",
      "\u001b[1;32mc:\\Users\\33689\\anaconda3\\lib\\site-packages\\torch\\utils\\data\\dataloader.py\u001b[0m in \u001b[0;36m_try_get_data\u001b[1;34m(self, timeout)\u001b[0m\n\u001b[0;32m   1162\u001b[0m         \u001b[1;32mtry\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1163\u001b[1;33m             \u001b[0mdata\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_data_queue\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mget\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mtimeout\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mtimeout\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   1164\u001b[0m             \u001b[1;32mreturn\u001b[0m \u001b[1;33m(\u001b[0m\u001b[1;32mTrue\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mdata\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mc:\\Users\\33689\\anaconda3\\lib\\multiprocessing\\queues.py\u001b[0m in \u001b[0;36mget\u001b[1;34m(self, block, timeout)\u001b[0m\n\u001b[0;32m    113\u001b[0m                     \u001b[1;32mif\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_poll\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mtimeout\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 114\u001b[1;33m                         \u001b[1;32mraise\u001b[0m \u001b[0mEmpty\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    115\u001b[0m                 \u001b[1;32melif\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_poll\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mEmpty\u001b[0m: ",
      "\nThe above exception was the direct cause of the following exception:\n",
      "\u001b[1;31mRuntimeError\u001b[0m                              Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-27-5eb5e6334540>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[0mx_test\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mlabels\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mnext\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0miter\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mtest_loader\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 2\u001b[1;33m \u001b[0mx_test_noisy\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mlabels\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mnext\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0miter\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mtest_loader_noisy\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      3\u001b[0m \u001b[0mn\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;36m10\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      4\u001b[0m \u001b[0mplt\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mfigure\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mfigsize\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;36m20\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;36m4\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      5\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mn\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mc:\\Users\\33689\\anaconda3\\lib\\site-packages\\torch\\utils\\data\\dataloader.py\u001b[0m in \u001b[0;36m__next__\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    679\u001b[0m                 \u001b[1;31m# TODO(https://github.com/pytorch/pytorch/issues/76750)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    680\u001b[0m                 \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_reset\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m  \u001b[1;31m# type: ignore[call-arg]\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 681\u001b[1;33m             \u001b[0mdata\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_next_data\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    682\u001b[0m             \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_num_yielded\u001b[0m \u001b[1;33m+=\u001b[0m \u001b[1;36m1\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    683\u001b[0m             \u001b[1;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_dataset_kind\u001b[0m \u001b[1;33m==\u001b[0m \u001b[0m_DatasetKind\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mIterable\u001b[0m \u001b[1;32mand\u001b[0m\u001b[0;31m \u001b[0m\u001b[0;31m\\\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mc:\\Users\\33689\\anaconda3\\lib\\site-packages\\torch\\utils\\data\\dataloader.py\u001b[0m in \u001b[0;36m_next_data\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m   1357\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1358\u001b[0m             \u001b[1;32massert\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_shutdown\u001b[0m \u001b[1;32mand\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_tasks_outstanding\u001b[0m \u001b[1;33m>\u001b[0m \u001b[1;36m0\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1359\u001b[1;33m             \u001b[0midx\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mdata\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_get_data\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   1360\u001b[0m             \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_tasks_outstanding\u001b[0m \u001b[1;33m-=\u001b[0m \u001b[1;36m1\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1361\u001b[0m             \u001b[1;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_dataset_kind\u001b[0m \u001b[1;33m==\u001b[0m \u001b[0m_DatasetKind\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mIterable\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mc:\\Users\\33689\\anaconda3\\lib\\site-packages\\torch\\utils\\data\\dataloader.py\u001b[0m in \u001b[0;36m_get_data\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m   1323\u001b[0m         \u001b[1;32melse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1324\u001b[0m             \u001b[1;32mwhile\u001b[0m \u001b[1;32mTrue\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1325\u001b[1;33m                 \u001b[0msuccess\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mdata\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_try_get_data\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   1326\u001b[0m                 \u001b[1;32mif\u001b[0m \u001b[0msuccess\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1327\u001b[0m                     \u001b[1;32mreturn\u001b[0m \u001b[0mdata\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mc:\\Users\\33689\\anaconda3\\lib\\site-packages\\torch\\utils\\data\\dataloader.py\u001b[0m in \u001b[0;36m_try_get_data\u001b[1;34m(self, timeout)\u001b[0m\n\u001b[0;32m   1174\u001b[0m             \u001b[1;32mif\u001b[0m \u001b[0mlen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mfailed_workers\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;33m>\u001b[0m \u001b[1;36m0\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1175\u001b[0m                 \u001b[0mpids_str\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;34m', '\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mjoin\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mstr\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mw\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mpid\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mw\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mfailed_workers\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1176\u001b[1;33m                 \u001b[1;32mraise\u001b[0m \u001b[0mRuntimeError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'DataLoader worker (pid(s) {}) exited unexpectedly'\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mpids_str\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[0me\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   1177\u001b[0m             \u001b[1;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0me\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mqueue\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mEmpty\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1178\u001b[0m                 \u001b[1;32mreturn\u001b[0m \u001b[1;33m(\u001b[0m\u001b[1;32mFalse\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;32mNone\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mRuntimeError\u001b[0m: DataLoader worker (pid(s) 15704, 22172, 1456, 15416) exited unexpectedly"
     ]
    }
   ],
   "source": [
    "x_test, labels = next(iter(test_loader))\n",
    "x_test_noisy, labels = next(iter(test_loader_noisy))\n",
    "n = 10\n",
    "plt.figure(figsize=(20, 4))\n",
    "for i in range(n):\n",
    "    ax = plt.subplot(2, n, i + 1)\n",
    "    plt.title(\"original\")\n",
    "    plt.imshow(x_test[i].squeeze())\n",
    "    ax.get_xaxis().set_visible(False)\n",
    "    ax.get_yaxis().set_visible(False)\n",
    "    plt.gray()\n",
    "    ax = plt.subplot(2, n, i + 1 + n)\n",
    "    plt.title(\"original + noise\")\n",
    "    plt.imshow(x_test_noisy[i].squeeze())\n",
    "    ax.get_xaxis().set_visible(False)\n",
    "    ax.get_yaxis().set_visible(False)\n",
    "    plt.gray()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Sy9SY8jGl5aP"
   },
   "source": [
    "### Define a convolutional autoencoder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vT_BhZngWMwp"
   },
   "source": [
    "In this example, you will train a convolutional autoencoder using  [Conv2D](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Conv2D) layers in the `encoder`, and [Conv2DTranspose](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Conv2DTranspose) layers in the `decoder`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "id": "R5KjoIlYCQko"
   },
   "outputs": [],
   "source": [
    "class DenoisingAE(nn.Module):\n",
    "    def __init__(self,encoder,decoder):\n",
    "        super(DenoisingAE, self).__init__()\n",
    "        #self.latent_dim = encoding_dim\n",
    "        self.encoder = encoder   \n",
    "        self.decoder = decoder\n",
    "    \n",
    "    def forward(self, x):\n",
    "        encoded = self.encoder(x)\n",
    "        decoded = self.decoder(x)\n",
    "        return decoded\n",
    "    \n",
    "encoder = nn.Sequential(\n",
    "    nn.Conv2d(1,3,(3,3),padding= 1),\n",
    "    nn.ReLU(),\n",
    "    nn.Conv2d(3,16,(3,3), padding = 1),\n",
    "    nn.ReLU(),\n",
    "    nn.Conv2d(16,32,(3,3), padding =1),\n",
    "    nn.ReLU(),\n",
    ")\n",
    "\n",
    "decoder = nn.Sequential(\n",
    "    nn.ConvTranspose2d(32,16,(3,3),padding=1),\n",
    "    nn.ReLU(),\n",
    "    nn.ConvTranspose2d(16,3,(3,3), padding=1),\n",
    "    nn.ReLU(),\n",
    "    nn.ConvTranspose2d(3,1,(3,3),padding=1),\n",
    "    nn.ReLU(),\n",
    ")\n",
    "\n",
    "autoencoder = DenoisingAE(encoder,decoder)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Write down the full optimization loop to optimize your denoising auto-encoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "id": "QYKbiDFYCQfj"
   },
   "outputs": [],
   "source": [
    "######################\n",
    "# Learning rate\n",
    "learning_rate = 1e-3\n",
    "# Optimizer to fit the weights of the network\n",
    "optimizer = torch.optim.Adam(autoencoder.parameters(), lr=learning_rate)\n",
    "loss_fn = torch.nn.MSELoss(reduction='sum')\n",
    "\n",
    "for t in range(20):\n",
    "    full_loss = torch.Tensor([0])\n",
    "    # Forward pass: compute predicted y by passing x to the model.\n",
    "    for i, (x, _) in enumerate(train_loader_noisy):\n",
    "        y_pred = autoencoder(x)\n",
    "        # Compute the loss.\n",
    "        loss = loss_fn(y_pred, x)\n",
    "        # Before the backward pass, zero all of the network gradients\n",
    "        optimizer.zero_grad()\n",
    "        # Backward pass: compute gradient of the loss with respect to parameters\n",
    "        loss.backward()\n",
    "        # Calling the step function to update the parameters\n",
    "        optimizer.step()\n",
    "    \n",
    "    with torch.no_grad():\n",
    "        for i, (x,_) in enumerate(valid_loader_noisy,0):\n",
    "            # forward and calculate the loss\n",
    "            y_pred = autoencoder(x)\n",
    "            # Compute the loss\n",
    "            valid_loss = loss_fn(y_pred,x)\n",
    "            \n",
    "    print('epoch: {}'.format(t+1), '| train_loss: {}'.format(loss), '| valid_loss: {}'.format(valid_loss))\n",
    "######################"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "A7-VAuEy_N6M"
   },
   "source": [
    "Plot both the noisy images and the denoised images produced by the autoencoder to check that your implementation is correct"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "id": "sfxr9NdBCP_x"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Figure size 1440x288 with 0 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "######################\n",
    "x_noisy, labels = next(iter(test_loader_noisy))\n",
    "######################\n",
    "encoded_imgs = autoencoder.encoder(x_noisy).detach().numpy()\n",
    "decoded_imgs = autoencoder.decoder(encoded_imgs).detach().numpy()\n",
    "plt.figure(figsize=(20, 4))\n",
    "plt.figure(figsize=(20, 4))\n",
    "for i in range(n):\n",
    "    ax = plt.subplot(2, n, i + 1)\n",
    "    plt.title(\"original + noise\")\n",
    "    plt.imshow(x_noisy[i].squeeze())\n",
    "    ax.get_xaxis().set_visible(False)\n",
    "    ax.get_yaxis().set_visible(False)\n",
    "    plt.gray()\n",
    "    ax = plt.subplot(2, n, i + 1 + n)\n",
    "    plt.title(\"decoded\")\n",
    "    plt.imshow(decoded_imgs[i].squeeze())\n",
    "    ax.get_xaxis().set_visible(False)\n",
    "    ax.get_yaxis().set_visible(False)\n",
    "    plt.gray()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### References\n",
    "\n",
    "<a id=\"reference1\"></a>\n",
    "[1] Rezende, Danilo Jimenez, and Shakir Mohamed. \"Variational inference with normalizing flows.\" _arXiv preprint arXiv:1505.05770_ (2015). [link](http://arxiv.org/pdf/1505.05770)\n",
    "\n",
    "[2] Kingma, Diederik P., Tim Salimans, and Max Welling. \"Improving Variational Inference with Inverse Autoregressive Flow.\" _arXiv preprint arXiv:1606.04934_ (2016). [link](https://arxiv.org/abs/1606.04934)\n",
    "\n",
    "[3] Kingma, D. P., & Welling, M. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114. (2013). [link](https://arxiv.org/pdf/1312.6114)\n",
    "\n",
    "[4] Rezende, D. J., Mohamed, S., & Wierstra, D. Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint arXiv:1401.4082. (2014). [link](https://arxiv.org/pdf/1401.4082)\n",
    "\n",
    "[5] Gulrajani, I., Kumar, K., Ahmed, F., Taiga, A. A., Visin, F., Vazquez, D., & Courville, A. (2016). Pixelvae: A latent variable model for natural images. arXiv preprint arXiv:1611.05013. [link](https://arxiv.org/pdf/1611.05013)\n",
    "\n",
    "[6] Van den Oord, A., & Vinyals, O. (2017). Neural discrete representation learning. In NIPS 2017 (pp. 6306-6315). [link](http://papers.nips.cc/paper/7210-neural-discrete-representation-learning.pdf)"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "TensorFlow_3_Autoencoder_Dimensionality_Reduction.ipynb",
   "provenance": [],
   "toc_visible": true,
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3.9.4 ('base')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.4"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "vscode": {
   "interpreter": {
    "hash": "4dcb06daf87d0588a14713ac9c655a46c30ed7eee687eb6e320f01c9e0bc4e8b"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}