Bob_training.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Client/User A, Bob\n",
    "In this jupyter notebook, we simulate how User A obtains the base_model from the server. After obtaining the base_model, User A trains the model on his data. Then,  weights were extracted from the Dense layer. These weights are encrypted, serialized and stored in a pickle file to send to the server. After getting the updated model from the server, User A check the new model's accuracy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Import Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import time\n",
    "from sklearn.metrics import confusion_matrix\n",
    "import tenseal as ts\n",
    "import pickle\n",
    "from fractions import Fraction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating User A's dataset. \n",
    "User A dataset consists of only digits 0 to 4 from the MNIST dataset. The dataset is split into 3 main parts: Server data, User A data and User B data.  \n",
    "In this section, User's A data is made to contain only digits 0 to 4."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of original training examples: 60000\n",
      "Number of original test examples: 10000\n"
     ]
    }
   ],
   "source": [
    "(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()\n",
    "\n",
    "# Rescale the images from [0,255] to the [0.0,1.0] range.\n",
    "x_train, x_test = x_train[..., np.newaxis]/255.0, x_test[..., np.newaxis]/255.0\n",
    "\n",
    "print(\"Number of original training examples:\", len(x_train))\n",
    "print(\"Number of original test examples:\", len(x_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Slicing the MNIST original dataset to retain only part of the dataset.\n",
    "x_train = x_train[20000:40596,:,:]\n",
    "y_train = y_train[20000:40596]\n",
    "x_test = x_test[4000:7139,:,:]\n",
    "y_test  = y_test[4000:7139]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x_train: 20596\n",
      "y_train: 20596\n",
      "x_test: 3139\n",
      "y_test: 3139\n"
     ]
    }
   ],
   "source": [
    "# Length of the new dataset\n",
    "print(\"x_train:\", len(x_train))\n",
    "print(\"y_train:\", len(y_train))\n",
    "print(\"x_test:\", len(x_test))\n",
    "print(\"y_test:\", len(y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Filter the new dataset to retain only digits 0 to 4.\n",
    "def filter_01234(x, y):\n",
    "    keep = (y == 0) | (y == 1) | (y == 2)| (y == 3) | (y == 4)\n",
    "    x, y = x[keep], y[keep]\n",
    "    return x,y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of filtered training examples: 10509\n",
      "Number of filtered test examples: 1603\n"
     ]
    }
   ],
   "source": [
    "# Filtered dataset\n",
    "x_train, y_train = filter_01234(x_train, y_train)\n",
    "x_test, y_test = filter_01234(x_test, y_test)\n",
    "print(\"Number of filtered training examples:\", len(x_train))\n",
    "print(\"Number of filtered test examples:\", len(x_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAV8AAAD8CAYAAADQSqd1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAB+KUlEQVR4nOydd5wkVbX4v7eqOnfPTE/OaSdszoFNLDlnEEUwYMCsqE+fT31PDE/9Kaj4EARFkgio5AxL3IXNOc7O7OzknHs6d9X9/dGzCYaNMzvds/3lsx+mqyucOn3r1L3nnnuOkFKSIEGCBAlOLcpYC5AgQYIEpyMJ45sgQYIEY0DC+CZIkCDBGJAwvgkSJEgwBiSMb4IECRKMAQnjmyBBggRjwEkZXyHERUKIKiFEjRDiByMlVIIECRKMd8SJxvkKIVRgD3A+0ASsA26QUu4cOfESJEiQYHxyMj3f+UCNlLJWShkCHgeuHBmxEiRIkGB8o53EsXlA4yGfm4AFRzrALCzSiuMkLjnyBPASkkExVtePRZ0AeOjtklJmjMW1EzoZnljUS+L5GZ5jaSsnY3yPCSHELcAtAFbsLBDnjvYlj4s18o1Tfs1Y1wnAcvnv+lN5vYROhifW9ZJ4fobnWNrKybgdmoGCQz7nD207DCnlfVLKuVLKuSYsJ3G58UNCJx8moZPhSejlw4wXnZyM8V0HlAshSoQQZuATwHMjI1aCBAkSjG9O2O0gpYwIIb4OvAqowN+klDtGTLIECRIkGMeclM9XSvkS8NIIyZLgeBACNdVN55WVeHMEUgNHi8S9J4C5vptIQzMY+lhLOboIgeJ00n/pFHonKoQm+IfdzbzXhrUDLAMGaSuaMXr6MLy+8a+fBDHNqE+4jTRC01DsdkR6KnqqE6kpqL0+xKAPo68fw+eD8ZyjWAgUmw0l1c3AvHx8lw5wTmENSZqflxsm07DNTXJ1HqnbXYigjtB1hD847oyx4nCgpLrxV2bRemGYG2et5ReZ29ClgSqi3jRdGgD8ZNoM3morp6MnCd2Uj6MtG2uHD7FjL0YoPK70cihqUhJkZxDKTUbzhlH2tWD09SMjkbEWLaZQ7HaUzHSkw4YUAmlRiSRZ0DwhFH8Y4Q8iPV6kz4cMBkdMf3FnfNWMdHzT82m4WOOaM9dQaOnhzs3nYN+QSfZqL+qWGgyvd6zFHDWE2YycXErTkiTu/uZdTDYFcCrRSYefZm7CmGuwKajw3/uuwhO00DNgh9p0yu4YRO/tHx+GRgj06WU0LXFwzU3v8LB7DVmqjfDQO9eQh9/jTzI285OMzdEPy+C9gImf77sMyzcLUVvaMbx+ZDh0au/hFBCcW07tTYIdF9zN73qm8+KvzsL9Zi16e8dYixZTyCkTqLnKiWVKH1ZThGlpTdxX8C7fb5vL642VePdlk75JkFztw9TQSaS5ZUSuGxfGV3G5aPjGNOTsARYX7ONjaQ+RqQ6Sq0UwIThv8S76F1r4zVUXsWXDNCoe7Efu3DsuH6j6/5xD1pIW7i57mBnmECZh+tA+U81h7i9/DICAFNTNTeHuxeewZccc0tepZLzXgb5n76kW/eRRVNQkJ7XfmUzp0nruL/4rpVqI2oiF/2g8n7UbyhGpIf50xj841+Y7cNhrfge1wSxMIsLHXdXMsoR4qOIfrH82m/9Ydx3pL1pxP7Nt/Ly0hUCx29l3rcZ1M9diEipfdm9g9ZdK2FdcRv6vosZXP3s2g7nmA4cl1QUw1XWMmHGJRYTJTOD8GdRfCV9c+C4mJUJXeBvOwXQ8ISvVu/NYuTGdieZpOBsE0gaaW9J7sY/yL9ez4dXJlD5iJlJbd9KyxLTx1QryGZiXR+tChfmLdnFR2jZmWpuYYjJjoKFgoln38ap3Kpc7t/P53BW87ujlWfssHPvmkrPKj/LOprG+jRFBTUqi+8opFJ1Vz1cL3mKuRQdUtoRgc6CArYMF5Fr6uDJpMwqSv/UsZWtvHjfnv8eF9ma+U/AqryVP46X8ydSUZ1L+Jz96e0dcDUHV5CQ8Z1cQLAjR6XXyo9prAKipzca5x0TB7gjNyyx0604ganx3hCJ8e/31qLudKEF45vx6Ppv3PufZm7jU3k/zzDf4k3kZvqwZ5P5pAzIUinu3lepy0XP5ZM6YWcVVKRsASFaszHE3UOUuBkUlfN4s6q5QySztOnBcdUsK1oZiMjfmY39ze9SFNx4QYqjtTKR7qkpkyiCfnbSeMmsbd9aeS8u+dJx7NTQf5LdEMA2EEFJi6g2g202E3GZ6Oxz0ZNsxzeql2pZD2V8FRn3zSXXwYs74CosFxW4Hs4mBeXk0Xabzwjl/YKJpfzyf+bD9a8NJ3L9nEb4JFhY5qvlG+tt87sKVPN0/m8etZ1FaV0CksSWuh9vCZIacTPzX9vGX0ieZZDIRlpKaiME97eezqqEYfZ+TSGqEvdMy0BSd11fNIKlG4TfnO7BOepFstZ/vpq3h0+7V/Lt4Ni+vOwvXyjB6d0/c6EbYbfSVqhCUDGxOI9IaXVhVvtmH1t6FiOgY5+ViEtEXSlCG+Xf/PJJed5D5Qg16RyfN4YX8dpkDb9m7XOOs5ZbkOs6Ys5c7ci6k72Enek8fyPjQx7AoKiLNTefFQf6Qs5w5Q49NWOoMRiwIHYSq0jbfzOfOfJP/TNuBQdQ3vmUSPNc/m0dzF1LZWoLW1osMBKJtJI5RbDbIy6bpigjfmf8Kk61NeAwbf2o4G9/z2VSu9cCmXR/qiBiAAKwmM/lbM9k1oYBFc6uYNmE9z2w5F3dv/0npJuaMr5xRQdtsJ54S+NplL3O5czuFmg2DD/dGDCQb/cVob6Tw/L/O5q8Lzmb+3D08Uvw6k9K3kf/xHn4//VyKvmmgt7XHVS/vUJSyIjqWpPP0rN8O6cJgX0Tnyle/SfnDYUo2Vx8YMjcNHVPOagDUxzO4u+xaGi5y8LMbHuUcWwu3pm5j7m9r+el/fY6UlSqR1rYxurPjI9LcQu7tww+J9TOm03CBk9su/CdLrc2EpZl2PcK/n1tCyYb+A37O3NvfR3s8j/sXX4V+23N83LWX6WYz12eu497sy1AGvRiB+DW+Wk4WnhlZ3LHgH5SZAoAVgC4jxNOr55G/RgdFEHZK7MrhvbYZZpiRsZH/vnwjV1deyc6aPFy7TeTc8f4Y3MnIIYryaFuaSs0Ff2JFQOOrmz5J+iN2bM9vINNoHMayHI4Mh4g0NjHpVxHWfmsSC6/Yy/Rvb6FhVwmMF+M7cMMZdF0e4NdzHyVF9TLNPEDy0GRSrxHgtrZzGYyYmelq4rWOSexpyEZtN2O1Q++ZXnKTvKSafSgIDCTXOGtxTfXzv9fdSP7jESJt7WN8h8ePWlbC3pvS+czlb5KlRn+uV33J/L+9FzHp9m5kS/sRh4d6dw+qz8eEJjc/SLmBa5eu4ZdZ65ln6aflXIkSKcL+dHwY349Czcigq9KBc34XFznqSVYsPDRQxK/euYzJD7ZitBx+f3pbO+53dR76n8up++EqPuVezVxLG3/+owe+X4bYvDtuX9TemXk0nS9YYm3HpURHif1GgK/WfoyClyWOlVXooRAV9zTxQPdF3DXtLEpyu3ii8rED+wP8ofRfvJ1dzsMFZ6A9XYze2BKXcyjK1Ins+WwK377kBdYFJd/57ZcoWt2PqKtCP84Rn97ZRfqmYn6Xdx5PL72Hr5ZNJ6k184QnMGPK+AbcCqVZXVzl6MNAomADoNvw8/Hdn6R1VS6mQcEa1xSsXYLsNgOzJwICunHRXGzDpOqQGz1fkmJlnqUZeVYv4bXZaD4/+sDAGN7h8WO4nYRyQ3w6ZQMmYWFDEO6sPw/Pa9k4a1Yd3T9p6BheL4bXS/6bubxWNJFfZq3HrpgontBOb0EeDk2LW2ODotJ37gQ6F+p8q2QtbsVKWOpsHSwgZZuG3tD0oXuTkQh6Zxcp7ys8vm0u6bMHuSVlJz8oepmfJ92MSVUhDvWhpqfRW25i6tRakpVoj1ch6prp8DpJ6Q6i9/UDEKlvJGdlKoONNnrS85k3/VYwGahmgwnZnTxY/jgXOfZgFAr+8JkrKXnSitjXjOHxjNXtHR+Kiup0UH95KhPn7mOatZFPr72ZCav6kDv3YpzAi0RGIpgHdQyPiVINvJkqSanJMB6MLwKEkAdcDAaSQSPIxmAqvodzmfBm/UfOxOa+CL5rFlCnZLGnPECuJrAKjSzVzF9nPMzXJn6TjI40iCfjKwT+LBvOlEEy1OgI4F+982hanceER6rRj3NiyPbsWlqWLKR3ToBkxcz5Wbt5MCcXYbMh4+Wh+gCK2UTbxSG+OucdvpYSjeCoj0TY1J2HuyqINIbXkYxEiDS34F5ZxIu5U/mGu5rF1jDhJBWLzYoeDJ7K2xgRZH4WnnKdb+W/fth2FUE4oiI+0F7Eqi24VoELyNy/b3oavReUU/sLO2dY4KakveR+8kH+t+YzpHv88WF8hUBx2DEqClly9SauSN3EKm85BfeZoHbfiPTgbcKMPwvC6c4TztEQU2WEgimQavEdeFv3GgG+33I+tz76eVIeXXvUEBj7U2uY/NMGPvW/3+UP3XOoDYexCI1ZZoXuGRJfRfopuIsRQlFRU934v9LLbVNeOLD5qQ1zyNhkoHd2ntBpU7fDpVtuBqDU0kHELhHmD4erxQNC01BysvjKnHe41b0HiLaZS5Z/E/lAJqblG446mWjpl/T6bARlGICWJQr+BeWjLvto0Dc5ifTiHpZYAwe2GUj6DIPBKjfqQOAIR0fRu7pJemwNjeE0DCQmoXKhvZ++CggVpo6m+COGmp6Of8lEkv7Qym3Zr7PeW8qf3zsb7c0NI/ryEJM99JdaT/j4mDC+QtNQszJJP7OVG7NWHdh+W9u5vPPmdEr/0XHMM/J6Vw9ZrzTy+s+WcumL3+YH7XMAWDJ/Jx1zTKhpcdKAUlPovaCCL5WuYKmtFY8R4tddMyh4UZD8Xt2Jnzcs8QbMKCh4DQtKBPiI3mFMIwRqehr7PpnHZGs0mZ5PhvhlxzJylmu439l3TKdxv1vH4I5UnvXmASCzggRSY2tAeKy0nRfhY0UbATCG/ru/v5AL3vs6Ex4bQDa2HtuJpORHa6/iz32lBzZNWFJPxyzbaIg94giXA2+Oxn/kvcKb/iIefutMJv55cMSvk5XsIew68VTGMWF8leQkBpaUcE3+JmaYD8Yd7h1Ix9Ym0Ktqjvlc+2cmk97YTe5b8O+182iK+LkufT3+wjCkpozCHYw8wmajv0xhmrURt2KlUxc8tPkMnDu7TmriUBig69Gf3aqEkSqgjFku7BNGy8/DO6eQrLOaKdV6CMowVWGN51fMJXl73zHrKNLahmsfPNJ8BgCTCtrwZSsg4k8nZUXtzLbVHbbtpc5ppLxug501GIPHboBc62zcW7WEJzw5AFyfsx7PBB2tuHAkRR4VDKeVYIpgqknybn8Frr0KxubYq24WE8aXzDTarg1yiXMHWWo0rCws9ejk2QlKqPf1k7R8NxMej7AllM0yWzfpOf1EMpNGVvZRQlrN+AvDOEQEA4P6iJv8pzXo7B6R8xsYFJi6MWw6whR/bgfv1BwaLlR4bfJTlJkstOg6Lw7MZOKdrcjqY+v17ie5Lkz15mhq6m8XvIanREeo6miIPWoITaMyqYNc9eCwOix1tu3NJ/WBVchg8LgWj+Qu78L8WhI/23gZABfaa8kt76R3QW7Mv5giyTaC7qjLZGVjKbYuY8TObWgC1KgeuwYdaN4THzXGhvE9hFbdz7sBM99uWcYVWVsYqIigZWed0LmE00kg3cQMcxsm4uth+iBew4JzVw+G/+h+u2NlgSWMxR0Ae3wMJw8lmKJizj4YYndv11Ief/5MIvVNUUNzXOfSULKjel1iDUBKGETMPRofiZqSjP/i2cxy1pOmHjQGX228AEeV+QhHfjT6rmrSt/kx77IDkKpaqEzpwFMU+3ppn2cjf2EzT3tTybrHivu1PSNyXi0nm7aFKr846ykGjAD255LIfOtD9SOOmZjTZEvExj86F7LzV9P5c/VShC4ITsw7oXOFizLonqKSoWrURAy6e5xoHXEU7XAIOmJElr0G3IIpOQd9fzHeiRkWtbyUnqmCT09cC8C+SIBX6iZRsDx4Qqv1BooVPjM1uihFQUGI+PKBGyX5lP5oF+fYa3AKEz16kC83nsPOv06h4MUTXAQgJVpVI9mrgjw0UES/EeJLWW8x7cpdMf9i0i1g08KsGyzF2jqI0T8yz7x/Wj4U+ZhtaeQJTzlJ9UGMthNPUhRzWhwwrNR60rA/vYbgqjQc9Sq65cTE9BRZUab3YxEmnhuYidZigZ6+kRX4FKCgoB51Hc7R0bKz8ObCuem7UVDYEoJAvwVGsDd9Kuibk4la6eHKpM0AvOCZRqgmCfO2uuM+l5aXizfP4Oqk+M0BEkmx8EDhCvK16AimPmLjnQ2TyXqrFWP77hM+r97VjbWuh+3ePAJSMsus8LWcNxAxPkcgVbBrIdqDLoQvMDIx7ELQU2mmLLsTl2LwRPNcTJ0+jMCJPzsxOa2rDPU8Cv+4BSXJhXTaOZEFn91TBf+c/TcMFP62cTHZm2TcrVNXRsDo7qdvaTFJs7r5cnI9BvDLhstw7TTHzfJiABSVzisCfHfyW1SYokPquzacRfbGE/tte5cW4i7toUiLGhQDAylj27gMR3goH0WXEeId7yyKntORXSPT1g2poMuobnRif35At0jSLV78umnEhnbCbMYz38/FmTto1820vp/HhP4GTsabHHM939mWPq7J3YSYNQWkJNLWjr73xIrG6jbJFLOGggIeDZN35BzvpwpjKObZqoQI5qcgzMfvw1McDjhjOjO+v4X7pz58YBHLvt5UzANxNMQWAsVq4XuzXuOmpOiCil4jgH2nlaTaE0sH2TVDsCC7ActQas4rq67CtdGKjIRHTOxTyeZgJs81T8Py5taTXs2pOBxEMpLItfRhiqP3kaVXsHcgnS9kvkMoz43icp3U+dSkJMJLpvKrBU/x8aSdbA4UUviaD6On96TOG3PGtymisXGgCKWl82B6v5PIurV/wQaG+NAKn1hGhCOYujUCUsXAIE3x0j7fgpJ07A1JWCyIedPovGE6jf9h8Om09yjSJP1GgJ92zkS84SZ9U/z5wFNULxZhIix1GiMmUmp0lLrj672r6Wn033gGs8+s4vrUtQe2V+3Lwdmsx1VaSWFI2nU/YaljoBDW1RFZxSXyc+idZOc813YcQuFVXzI/rr76I1cNxgqpu8LUbMmnSPPRdK4NffqEkzqfSHNTf5GZKeY2nh+cwB07zkOraTnuid0PEhPGVwTDGJ1WPIaJPsNGZ9AJ/gBSP3Gjq6anYVgNgjLCrnAYS4+CaSCO1usHgjjrYEOgmE49SKoawJg7gJ7tRliOXC5bWCxo+XlEFk2h6VwXvos8vDL/z0w367TrBs8OTuDR1QvJeacHsSMOk6oPYWDQrTtwNPuPecWfsFhQy0oYXDyBrkuC/DDvJRZagwRlmJd9LpxVZpyNw9eCi1WUoM7Tnin4ZBirCOMwh6K9vZMccgfzkumriGY7qwpbuK95Gd3v5ICM7RGkY3cHqdsE/YZKyoJ2uqbZUN3uEzqXmp6GrzKTuYuq0BH8s2Uu6uqkEcmFHRPG12jvpPxRL5sDhWSpgyxNqyY4txzF6Tzhc/adV05GYS9dRoiv7P4kJf/uRlkRP5Mqelc32U/v5VerL+bR/lnkqyaemXcvHfOSUEqPHOiuFOZRf1MRN//5WV766m/YtPABslQLqhDc1nQZtz9xDRO/twtjW9VJTRjECvI4jIxSWsjub2XysV++yoZldzPJZEJBYWdY5QcPfpbCxxpgzbZRlHbk0Vp7+dO/L6U2YqbU1MPC9H1EZpZF82KfBP2lZvJmRyNjvrLtRtoeLSb/l+/H/Kggsq+e9E39/KTxcn5R8QyeM/0MnnliS8b7ziun/gadR4pfZ/ngZBpWFFL4WN2IyBkTE26Gz4fYsodfP3s155yzmduyX8e4U+GtmxfA+u3HdS6haagZ6aR8qYHvF77C7pAb8ZcMaBmZWL9TRTTzVjeVd2Xw5y+ezYLz97LQKrj7+3ex9VsFrOqfwIoVUxESTAMCe6ukvxxQwMgJcMcZD3O2rROrsER93sCUR75O7rsRStZWoQ8OxvxDdDQUFPK0AequsGOfuwglIknbHkDr96N0DxBpbkEryKdvYT4DRQr+LINPnPceP0rawmRTALuIjiA+U3cem16bRMl91ei9vXGnl0hjC6V3BXjyinl8PvU9znRVsXzCYtJ3WeEESyNVPziHy6au50r3RtYHVeTyVDLW9JzUBNOpRNQ20f2LCr79leu5uGIH/h+aqe+Ygbp1mBqPQ+WpBs6dSPL6FiL1jQe+8qcqpLi9vOG38++fX0Dp2mYirSOTmjYmjC9SIoNBMjZJVk0sRs+GG5I38fezzyfHMhOtqhG96+gruxSrFSU7k64z8/hKzpO8PDCdf22bzaSNbeiDcVify9BR9jaRsXoyX069keVn3MMsi4VKUw1n2GpxnR3AkAo9ITst3mTmu9tQkJTaOllibccurDRF/KwJFPDbPeeTuyKCY1srkWPQZcxiGLzvKWeO5W0KNRtZapilZ2+j1Z9E2FDZtyQd3e9E+JPRvEVEXAbJ+X2Uu7spdPTyefcqMlQNy5Dh3RQyWFU1gdKVQfSurrgzvAAYOnpnJ/9cP4/kBX6uTtpEzwV+rH0TcG11HXO9MTUrE6Mgk66ZLj41613ChsrP915O8+YcStcNIprjp/Cm4fNh39LIwMulvLRsCheV76L6c2byXpmKq7of0diO3tuLVlRAJMdNX5GdtktDWHsyUBtbUGxWIrMr6Jse4azMFn5Rcykp23vRW9pGrPJLbBjfIVK2dtM5J50XKir5dNI+rMu6aLSnk+0swb7DigyGDvM3Gf0eZCSMMJtRnA7IzqB/Ygpd5wdIVQd5ctdMCv6lEdl3YtESsYDe10/6+x0II5NHJ81ioaOaYi1IpcnEb3OiFQaUQ7xHBgZhqdOpS2ojBss9c3iyfgbmf6RiX18bd6F2hyElMhLhpT1TKLZ2cVPSDpIVK38ueOfALv0VAQJSEpYQkApWYQwZ2/0hUjZ6jQD1EUm3YePu1vNJ2mrGsm77cafojDUyV2j8O2sm183YyP+b+xTf6/wkmY5sUhWBCISQg4PI0MEoDuFwIFQFGYkgHHb85Zl0T7WQemkzN6as5dbaj9G9PJeKR2rRO7vQ4yjHsYxEiydkPyeQSgnv2Cbw06VPc1voWpKzU3FXObA29NE7K4P+UgXfhBA3Tl/HW+mLSHLYkRWFNFxgZdbkGopt3ax+eRpJ7VUnPcl2KDFlfPVd1ZTfF+HBrZcz7Wd3s3zmQ3im62y8KZNvvftJrI1m1EPmQgpe7EFpaEGfXEzTIifGkn6+VPkKX06ppfL1W8h7VsP6/Jqxu6ERQt+zl9SGZl5vXMoDZ51P8dJ6nq185iP33x4y8V97P07be3mk7jTI3N2HsXX1CcVKxxoyEqHsK3Xc941LCF5v4juphy8iSFasJB/lHLe1ncvLW6aSsslM7lO15HSujSvD8lGkvbqXxswyfpx6BY+UvMJl1/0fKy+z8njXGbxVU4H7jUIcrQfvs2OuiaDbwNqpwNx+clM6OC+lmV9mr2F90Eb1uiIqHqmNrzjwDxBpbSP7UR/G6gJ+8s0r+c5ZrzD74n2YRDQXyCeSH6HUZMJnhDnjoe9SUjtAZGYZwR/3sXryX3hoYDJ3bTmLisc7MIYS0Y8UQh7lbS+EKAAeBrIACdwnpbxTCHEb8EVg/zTzD6WULx3pXEkiVS4Q5x75eiYzanoqXeeXMP1rW/l8xrtUmoJUR0z06E5CUkUVBrpUqA9l4NGtJGs+CkzdFJt6CEuF+7vOZMfPpuPc1Eyk6chrr9fINxiQPWMWxXgsOgGiMa5OJ0qaG39ZBp2zLFzwidVc415PgeYjIAUf2/QFIrqCv8lF2T8DmBq6kD4f0nv8K3GWy39vkFLOPcHbOimOqhNFRV82g9aFVpR5fVSmdzDB2cV0eyPXOw8OjQ0MVgUs/G/dpdQ0ZKJ1mMl7O4K104/S74OBQYzunmOetR5LncDR9SI0DaW4gIGZmZR+dxeXpW1hgqmTVCVEu25jbzgTr3EwUibP1INDhBgwrORpffQZNgLShAmd7977RXJXDCLW7zyifuLi+RECxWJBFOSiux14C+x48lWUMPizJZGh1CaG1UBJDVGR286vS57i2zXX0/56PrkrvCgbdh9Xr/dY2sqx9HwjwHellBuFEC5ggxBif6r830spbz9miY4BGQ5FayW9a+a9ghm8NbmCmYWNLHDvo9TcSYGpm1w1yD8HpqMjMCkRfIaZ9wYr+HNfLjVtGZi2Oije1oTe2XX0C8YLUmJ4PBiDg1gHveR1Z/Oy5QyezpiPtBggIW29itmA9C4ddXM1kROcbIl5DB1LTQe5kQz62pLZk5LCTkcFTzkkP846JL5VCpQBDddehdw2HWtPCPN7OzCCwbh3MQyHjEQwGltIjuhsenoqq9KnIAq9LC2uBeBj6evI03rZ6C9mx2AuUHrY8Zs7c+npcqH0mSh/24NS2zwuRgRIGe18VNeCECTVZ+DKTgMg7LZhmBUMVdC8TGPS9FYuydhGdSiTzpfzyV3lRdtZNyqVTY5qfKWUrUDr0N8eIcQu4MQy3RwjMhIhUtdA0e+7kFMmUDu3nA0zSnHn9jMxrYNZyQ3cveoc0A++cLU+ldQdMGFbP8bm9xkHTWZ4pIxOPnZ1U3CEyLl4mZU+USKNTSiNTaSuOL7jxrteZDBIpK6B3N80RD8vnMGaM6cDUHd+KjYtzI6aPOx7zYgPKCN1d4SsnV3o1bVIGBduqg8hZbTg5VDdNXXonzCZcZTNZSBoZV8wg/trF5P/4G707p5R08Nx+XyFEMXALGANsBj4uhDi08B6or3jk1tv9wEMnw/WbSNjvSBjKJNSN/CG4qbC2Hj4zkMTccY47NEkSHCiiFVbyFs91En5rUIYqKBj+IUSUo5Pg3sMyHCIrLtWwZ8UNqOSyt7jrm58vBzzIgshhBN4ErhVSjkA3ANMAGYS7Rnf8RHH3SKEWC+EWB/mBLvu+5cYD/2TkchhnzGGloPGieEdEZ2MMxI6GZ4Re34+8Awd2Hbovzhh1NrKB3U0yhyT8RVCmIga3kellE8BSCnbpZS6lNIA/gLMH+5YKeV9Usq5Usq5Jo68LPZ0IaGTD5PQyfAk9PJhxotOjmp8hRACuB/YJaX83SHbcw7Z7Wrg+JaiJUiQIMFpzLGEmi0BVgDbODhf8UPgBqIuBwnUAV8ampw70rk6AS8w1mEI6YfIUCSlzBgrQWJUJzCGeokhnUBstRUPUDVW1z+EWNJJrLSV435+jmp8RxohxPqxjJWMFRkOJRbkiQUZDiVW5IkVOSB2ZIkVOfYTC/KciAwxkdUsQYIECU43EsY3QYIECcaAkzK+QoiLhBBVQogaIcQPjvGw+07mmiNELMhwKLEgTyzIcCixIk+syAGxI0usyLGfWJDnuGU4YZ+vEEIF9gDnA03AOuAGKeXOEzphggQJEpxGnEzPdz5QI6WslVKGgMeBK0dGrAQJEiQY35xMSsk8oPGQz03AgiMdYBYWacVxEpcceQJ4CcngmGVlikWdAHjo7RqrEKKEToYnFvWSeH6G51jayqjn8xVC3ALcAmDFzjGlTzyFrJFvnPJrxrpOAJbLf5/SDPQJnQxPrOsl8fwMz7G0lZNxOzQDBYd8zh/adhjjZSngSJLQyYdJ6GR4Enr5MONFJydjfNcB5UKIEiGEGfgE8NzIiJUgQYJRQ1ERJjNC00BRx1qa05YTNr5SygjwdeBVYBfwTynljpESLEGCBCOPMnMyDf+zgO/t3kDOShs1t89DWOK39xjPnJTPd6hs0BFLByVIMBYIkxk1zc2+L0wgqc4geY8X1m4ba7HGHG+Rk0nnVLPA4sWV9Sbdcxz0XjITx0ubR7Q4ZDyhut0YZfk0XOSi8IV+5KZT04eMuxVuisuFWl6KMn0iasrRSiWeXqgZGYi5UzGWzESZOvG0HlIKswkj3c3SKzbRflaE7mnOsRYpJgg7FC7K2IGCwiyzwnVZ62lboCDM5rEWbexIS6FrupMLrlpL29Jk1PLSox8zAsSd8Y3MLGPXf6QR+p0P3+KKsRYnZhCaRtelZZTdU83PH76f7v+no7pP35eTMJvRk618JeNtrpy1mZ7p8ZMsfDSxdYT5/fZzCQ/VrJhiaaF0XiPCcvoa30hGEj2zDH6RtZLPf+lFdn0/7ZRcN+6Mr9brx15n4o9lT9D+2QBdtyw8rXt4B1BVPMWCXEsfOgIpxyz0MibQ+/rQdtbzn7XXsrs/a6zFiRnUgE6gx3qg3FauGuLGvNXRybfTFGXjbib+oQOf1Lk5uYpvLHyDhv9ZhOIY3fjhuDO+iseLq8EgQzX4+pS36V0SRMvNBnF6GxsZjpC1NszOwRwKNB/XF28gMrEQNS11rEUbG6REBoMMhCyUubowZftQMzJO63aiuFx4863Mm1KLaagmokkIUlTfGEs2tshgEDq60aXEqVjJN3cTTjZAGV3zGHfGVw56cTaH0KXkC8m1nD9xF0Z6Moi4u5WRxdCxr6yiweMmT7XzlZQdDBZYEa7T2NepqlhUnfmuvUzKbocM92ndTkROJn0TFH6S/wImkRgtjjVx3RJrw2FqBjJQ+gaHr8Z6GiE0ja6rpzA9tQUDiY4kqdqD0dk91qKNCcJiQUlzMz21mTnWRgodPRgOC0I5fXu+TZdlkbq0jTLT6etiiCXi2vgWaRqlrm6MZMdp3aMBQCgEUwRJmh+FqIFRuz0YvtNzSCnDEeSglzpvGh7j9J1MOhTvLD+3FL871mLEHFp2Ft4zJ2I6xS6puLZYdsWMTQ0hNeW07tEAoAg8E3QKLYf0dMORuCoJPqIYOoTCeEIWQqi4TT68+XZQT7/htjCZ0c+ezbTCFuZYG49+wGmGTE2mt1I74AfviTixtSqgj275+Lg1vuppPHHyQYSmobpTmDy9gWnWRgzkgdns0xmp64R0lbBUyTQN0Feunnaz+sJkRs3KYN+VZq7K2kT+6XX7x0QkxcZgiY46NGJsCKaRtiOMDEdG9bpx+1PoUqJLA0PG7ftj5JhRSc01Sbw34Xbcig0QKKf7y0kIhEnDbgpjEjrF5k4GJwURp5m/M7J4KtVXmFl5ze0kK2ZUcXrd/4lQ60vH8vI6Rrv7krBc4wDdaSacH0RFYCB5O2Bi4eovIn3+sRZtzFCTk9ArCvle0SvMMPtZ651A3osahj8w1qKdMrTsLDpnWLl82fohwyv4U28lE5//Gk2RIGGpszKQxbfeuhEZOH30cjQWp+yl/ZuLUOz2Ub1O3BlfYTYTdkXf3hF0goaG0CXSOH2H2YamYLJGUISgKeLnyZ552F91If2nr/FF04g4zcy09KEiaPK7ce4dHPWhZCwhdQPNL6kdTKdJD/OHnsn8afMy8pYL+gwzBgYdkSSsTabTSi9Hw6KECTsY9ZjwuDO+0mHDm62iCoHHCNEftiJCiYazn/cDRby6exJpf1mFkejNoADteoRmXzJKIDTW4pxS9M5OMtYNsO/5Uv7dP5v7XziPgkc1kl7bRViqKCioGIjRnVeKOxqCaaRviyBD4VG9Tlw7gHaGHVR3Z5DT2X3ax/kCiUm2YTCAB3oXsmdHPuW715520R9y807ytptYcU86pZGN0R6u04GOwMAgLFXE6aWSo1LrTcf60gakkYh2OIxIVjLdc3QsQmGyyUt5WiekpyTifCExyXYIRkE2zWeZsQgFQwowxGlneIHoMutwCMPrjS6jPcSg6KM+pRSfGIjD9DRaxJ3FCqaamTapARMqVqFiVcPI0zB2M8ERUFSCWXZM0/oxkWgbCWKTuDO+vnSVnxQ+h0moVIUV9vano/QnlhdDwu2wH8VmxZepcU3plkQ8eIIT4xRkSow742vyS5YPTsHAoFN3EYpo6FkpiETvN+F2GEJxOfFlC/4zbRNaoueb4Di5JG0rdT+bj+Jyjep14s742ttC3LtpKQEZYbK5m6nprXgLHaflstHh2OXPRfYlchlIJbr8XE3MBSQ4Tnp0J/YWkVhe/EFMfQHM+6LJoNMVMzmWfnTT6d3j060Kuan9AGzvz8XSmXgRJfhopJS85plGVVijIZiGuY/TczLyI/DpFiwDBnKUjW/chZpFnGaCuWEUIaiPRFjTXYx7TQuR0OkVw3koAwUa6yY/DVjY3pxDem3C/w2gJ+YBhkUGgjz1yDJePWcSPQN2yp+rJ3I6L8j5AGXWNrpmgftZ86gWFY074xvItHDWtJ1YhUaZSaEsqZOGlBJoUECeftHialkJ/sxoz99IhA4lOAJaSRHB4jR6yy0IA3o3ZiAktF7mQNEL0fwSa6+OY2srvsnZ2Or60KtqxlrsU87LPdOp+E0t+uDgqF4n7oyvGjCo7stAz5c860vlnboyJvT2YZymvZxAcSqB7OgKv1bdj1ZjI2XHwGlthqXPj7Vb8qgnkxtc7WMtzilFmMyomek0fayYUAoY2sGWEE4xUFKDFGS0AOAwRUeL3nB0jsAbMtPusxJszUPL9CP2ZpCxMQ3Hk2tO+X2MJX7dhN7eMerXiTvja+4L0bA3g+CUCA82L0bZ6kJv23Pa+qw8BWZsmR4AqsPJpOyRyE07xliqscXw+bB36jzeOp/rnM+MtTinFGHS0DPdVF5XxSez1pCt9h/4zqWESFV0UlULAMrQlI+BQY8ePJBMvF1XSFV0bi8/k6eT51H+5Km/j9OBuJtw06oaKf1XmIA0qNpSSO57gVH1y8Q6vmxBYWovQRnhn93zMQ+eniOAQ5GRCKrfoLk/eaxFOeUYfj9q3yDNg8mUmrqYZTEO/CszaQcM7wf5f51nsTGYgksxH9gv0+xBOhJ5U0aLo/Z8hRAFwMNAFiCB+6SUdwohbgO+CHQO7fpDKeVLoyXofoxBL5a9HVy25XPkrJCY1ldzOpsbW4eksTcFu2LmR9mvc617JraxFioGsLYOElmTynOTsnihfgrO+rjrZ5wYUqI3tZL83RL+I/lLSPXY7lsdDLHLMpnfmg5Gyqj+MJP6ezmdZlLK3rqZjBctJLF61K91LG6HCPBdKeVGIYQL2CCEeH3ou99LKW8fPfGGQdeRgSC93Vm4e8IYHs8pvXyskVIbpK/OxebZQR7rW4Jl4HR+FR1EtPeQvdbBf6d8AlcduKtOn9GRDIfQd1QhgGMNwtzvtFM+sO10MLymtn4y1tiZZvs62W9ouNd1nJL7PqrxlVK2Aq1Df3uEELuAvNEW7CPlMWQ06UW/CSWUSJlo3lpHalEldy4+j5XvTGVCvee0nmzbj97egdbeQekbYy1JglgnUltHSm0dKY9EP5+qF85xTbgJIYqBWcAaYDHwdSHEp4H1RHvHvcMccwtwC4CVEcgMb+jo3T2U37ombifZRlInencP7odW0/KwQqlcjUzoZFyR0MuHGS86OWZHmBDCCTwJ3CqlHADuASYAM4n2jO8Y7jgp5X1SyrlSyrkmhnf2nxBxamRgFHQih0YDCZ2MOxJ6+TDjRSfHZHyFECaihvdRKeVTAFLKdimlLqU0gL8A80dPzAQJEiQYXxzV+AohBHA/sEtK+btDtuccstvVwPaRFy9BggQJxifiaD5CIcQSYAWwDQ5Edf0QuIGoy0ECdcCXhibnjnSuTsALdJ2M0CNA+iEyFEkpM8ZKkBjVCYyhXmJIJxBbbcUDVI3V9Q8hlnQSK23luJ+foxrfkUYIsV5KOfeUXjQGZTiUWJAnFmQ4lFiRJ1bkgNiRJVbk2E8syHMiMpwmkecJEiRIEFskjG+CBAkSjAEnZXyFEBcJIaqEEDVCiB8c42H3ncw1R4hYkOFQYkGeWJDhUGJFnliRA2JHlliRYz+xIM9xy3DCPl8hhArsAc4HmoB1wA1Syp0ndMIECRIkOI04mZ7vfKBGSlkrpQwBjwNXjoxYCRIkSDC+OZl8vnlA4yGfm4AFRzrALCzSiuMkLjnyBPASksExKwIXizoB8NDbNVYhRAmdDE8s6iXx/AzPsbSVUU+m/sF12AvEuaN9yeNijTz1mVdiXScAy+W/60/l9RI6GZ5Y10vi+RmeY2krJ+N2aAYKDvmcP7TtMMbLOuyRJKGTD5PQyfAk9PJhxotOTsb4rgPKhRAlQggz8AnguZERK0GCBAnGNyfsdpBSRoQQXwdeBVTgb1LK07t4WIIECRIcIyfl8x0qGzSypYOEQKgqistF78WV9FYqhEv9Hy1Dv5nknSq5z9VjdHVjBE6vBOvKzMmEUm3oFoX+EhMDFQYyNXTge3OdhaRaSNvUh9y5FxkOHeFscY4QCM0EgJqbhXdKNoFUFbPHAAHeLJWeBWGEX8XaopL3jg9t/e7Trs0kiFZ5Dpw/g8FcDW++IDTho23MfixVNnLfC6C+tXFEZIi56sVqZgZGbgYDE5x0XBLkpmlr+Z/0bRhIlKGiKIf+vS4o+W7Zx/C05OHapGDUNx7p9OMCNSUZsjIw7BZaliXjz5LoNknOxDZ+VfYi59p8B/b9Zdc0HquaQyjJTZalEq2xk0h7ZzT/73hBCNTkJMhIw1+aCsBgronumRI1I0Ck3wwCMgq62Dj9IarCFh7uXsw7xmyKqlwQDMZ1LuSPQrHbEXYbwnZIVb9wGOnzI0vzEYN+RDAMQKSpeVzq4INoxYXobhfBLDuNF6gklfRybdEOfpG5jabIIKmKGbtiHvbYWyfP5XV9PsV7cok0t5y8LCd9hhGm5/xSfNf1s2He3UC0rLVf6vQYEdKHlOKTYaxCRUFhlkXw5rQnuOTr19Jzfx5JDU3jvhF5l1TS+LEIi8pr+UHWY0w1e3ApZhQUDAzCh9z/99I287WF69g+x8X/Xnkpzc+UkvtYGL2rewzvYGRRLBYCc8uov8TEzo//34GS6MNjYY4F8rOWk3tDHytfm4Pi8YzL3q+cVEJfpYv+CQf1Ye2SpFSHSP1JPRu2l2Jr0cCAoj/0Yni9YyjtqaHmi3nMOquKvxa9hEWYDnTiglLnl+3n8dn0FcwxD18H8Y7stVx7URp7wxPIvX2cGV/F4cBTqHBdyVZWBVU+u/JzqK0WTAOCnFVBWhdbMDRwNkgC6QKpgq84zAPn3s9fy//B+bO/h3trBfqOWMi6Nzoo0yfSPk/ju/Ne5nLnLpIVFROmIx5jFybmWnzcV/YYn7viRppMleT87v1TJPHoIExm1Mx02i8qQl7VzdVF73FF0maUj9CFgYHPCNOmQ5oqSVXMfNm9gccuOId8tRzWbjvFdzD6dM1MYvDCQR6e97cD29oiydQEs7kuaSt9BRoBqWJIhb9cvoyVry+k4M3giA2rYwk1K5M93y/lvy55mkscNdjEwfJDvYaflYEstvx2Jj/smIyQkmCKieaPhXE4A5S6e3iy7GUAflP8FH/4+LnsHYGywTFlfCFaCv0fO+bxSM8Sct4VWLtCaL4w6u568v3FoAi0Tg9Gkg2pKXTOcrBnaTbzLQGSKnvoWJhK2nic9lNUtNxsqm9wU7FwH2fZ9xwYCQCEpc5LvnTCUkOXAlVI0tRBKk3d5GoWTJjI1VRuyF/HHxa5UB9KRe/tjzv3gzCZGbxiFp58FX+OJHd2K98ofpPZlhbyNRv9RgCrUAlInX4jOgJ4on8Ou71ZhAyNDSsrmbawhp8XPkeFyYqvKEIg04Z1jO9rJFHsdoKLJtF9RoQrJ+xixiGj6EpTN7MtHWSoFrIOVIk3+Fb2cjxnW1ifVE5l72TkjppxNT8gzGa0Ai8zrQ1kqnaCMsL9/eVs9BSyvrWA0I5kyta3Int6keEIJqeDHK2EYLKVHZUpUBY9T66qMsneyl7cJy1TbBlfXSelJgjYyFjXB1X7DgwHdUCs2nLg7/2kJM2lNZSCgcGZeXt5foqbtFMt9yijuFwoaW6CJelcfdEqvpa28oDhrYkY9BlWfIaFX+++iIihIGV0KJWd5OETueu4xFFDuhr1+13u2EP/RBtvpc5ADHqRwfgxvkLTUDPTab4swuLKKi5L28K1zv35q20EZZjlvnysSpi2cDLbvfkAvLh5OpYWE0JC2W+3sOXn06jOyaDC1I8za5BQUtK4Mr7CbqN1kYVLZ27gY+61h31nF2bsQ0bXGKqNoKAwyaTw/wqe5dGkufyr9hzyO9PRO7rGjwE2DEJ+ExsCxXTrndSF8/nDxnMw1dpI3WmQ8tpuIr2H1P/1ekl6PYAxoYCQK/nAZgUFk4iMiEgxZXyNQAD17Y2kvX2wZEYC6L9kCq1nGZh6Vb7v2kGOaiMso0bz6hVfwbLbhqUPsu5bf9jDohUV8OuvXU3Dhe/y4/StACQrZorMXWCKqZ/+mFDT02i5spi/L7uL+ZbD/fphqVMVVvj1XTdgGpQk1YfQ3twAQAXrDux3OrQrYbEQKA5xQ9oq5lqO/eWar9n4RuomLrt1C1/w3UrmSgf6rupRlPTUEWlppfLrHv56zZVYPAZJG1spb92JDIVAyg+XixeCnismY9zQzUOTfw9Hce2dCPH3BB6BrqATzTdmy8xHjZTle3CvSwJd547nbuC3pgPjRSrbBxC+DmQ4jP4RvRQFiUmohCUoAlTic0JSBkO4miO85plGqrKODFXyjj+H2/eeT1t9Gs5ajcJ/1yIjOjIQGNbQKi4X0iKxi+Apl/9UIX1+kraYaVuSwr/DJv5Sv5Ren40LCnfj182811KCIQ9/Ts4rqOIa93rmWjRKTBGCl/TT600jaZwYX6TEGBwk/cU9oBvog96P7NWraakEZ5YQur6XWye8RYVpdGxKXBtfNT2NgXSNEksH7wdcvL9nAtlb49OwHAm9uwe6e6If6hsPm8s/Ur8mnJ+Gnh1krqP2wDafDNGn20GPvz6g9Ptx1vTz+PNn8o/SeaSlDNK1I4OkvZDfbuBo8hBpbRv+YCFQbDZ6rppKzoR2CrR+wIK30UV6T/y4Xo4F6feTudnPf7z1cURQIalaxeSTPFd8BkpE4GiSqB845sXCM9i6JI/vFb3CMpuPm8tX80D+RSSNyR2MElIeNcpHKy7EMzObxksk/1m2ksW2OixDk3MGkn8O5vPHnWdTMAL1guPa+BrFOQwUK8yxNnJP51mkrDaT/NL202JoeUSEQLFYaJtpZ25pFefZPIRlNAytJSLZ5stHhCNgxNeLyggEYEcVJT+3wLRyfHmpTHh21YHvj3Q3wmxGyUhD3NDJ9ya8RqnJRL8RwL1dwVbfc8SXWLxhBAIo72xiUnMJIhBCb+9EhkNkpqWCIdEP9W0Okel2s8+YxL8umc+y/Lf5mruKuzMuHAPpxw7V7aZvfi4tZxmsuOj35Kh24GBURFPEz+93n4vrOdeIXC+ujW/j+UlMuLCWMpPG5u48bD0Sw+MZa7HGHDUzg4ElJfznNx9jia0RDkk+8vPmS9n06iQKa+M31EwGg7B+O7b1x36MmplB+/n5PDL5DspMFvZFQvyw4UqyVnZj1JzyZGWnBL1m3+Gf94+ehtu3txc1CEHjg33i0wPFamXX7RP48vw3uCF505DhPZzzVnyDnGfMOJ5cPSLXjE/jq6iEz5uFtqCXpWnVfKbuQnzPZpG9Y3z1YI4LRUUunEbzMjuByX7OKtvBIlsjKcrBn/iO7qmsWzmRCS/2x6nX98SRFhP+DIFVRMdF24I5VD9VQV5HFTISHmPpYoNgimRJSg0A32tdhKNp/M2fDIealkp4UiEzyhpZ6KgmSz3YWQnKCCsDDr707mcoeE7BtbWdyAgt4oo/4ysEqtNB3TIT1xbtwCR01q+uoGydB9o6x1q6MUOYNPZeZaNsTj1XZW/mQkfVwRWBRpiqsIW/rF1K7noDsbvutDK+Wl4unikZRGYO4lAEm0IGj7YtIHOdD8MzOO5XRB4LyvSJGPkBZlrrUVB4s6ECV+f4d+Dtn1xruMDMDzK3UKoNog25GuoiPl73VvJIwwJyX9FwbWrG6By5laFxZ3wVmw3ysrjqklVcn7KO+7qWUfJ0ADbvRo+MTPxdPCFMZoRJQ0lL5e2P/5aMA29tC7qUhNGpjWj8uvESyu8Po+3Yg34aLCPdj9A0PPPyaTpX8PoZd5OsWLir7Vy2v1/GhI1bMILjN+oB5RAXwhEW0whNo/7yVC6s3Mgsc3Q6N1DrIqt9fI8IhKYRqSyg4Xwzr33itxRqUR9vBB2PEeKx/rncv3YJGStMuP+5ipG2LnFnfENnTKLxljCPpr/PD1rP5Y0VM6jYtQddPz0dDh1fmEP4gn5ennMf+ZoTXRoYQ/3a9SHBPW0X8P76Sip/sA3h70A/jXp5QtPouWkeuTfXcm/R0xRqNjYEYe3rUyh/sJWIz3f0k8QxwQtnIzWBpSeEeG/zsPuoSUlEppTwtRuf53LnLoJSo1OPYGtTMPUFxu0ISbFa6frkLNTrOvnv0heHDC8MGAH+2j+N+5+8kOJn+pnUVI/R2zsqeogr4yvmTaN1kYVvTFtOmw6vr51OwVsGRv/AaTV0VN1ujOJcam5yMXVuLZ/IXkuGakGXBkEZod8IURVO4vPP30LqFkH5Li/GODc0H0TLy8UzL5/cm2v5Rv5yCjSFsNT55EvfoOj9CEZbx1iLODooKmpqCp2XV2D+eDsZNi/NnmT6diw8EA4iAFurILUqTDBFZfAT/Zzv2E2qorE9ZOLTa2+hZOUgSm3LuJtDUaZOpGeWm67Zknnzq/hU1vvMMHdhYOO9gIn/qbmR9vXZlD7VC7VNGH4/cpRG1HFjfNWkJJoXudDm9rLUvof/bbmEtI0Kzi2NRE4jd4OalER4ShFti+zceN47XJ20kSJNB0wYSNYGrbw2MJdn9k6n4FUdx7aWaLpAoivepN2KHFqkoXT0Ygx4xp1h1rKz8M7Mo+k8wb1FT1OgKehItodUct8G+662cdvrVawWZHYGAxd5eaTyCYq1EHURM//KnY8hBYqQGFLwfkcJ9aUZSEeEB6Y9QZEWzYrXHHFj2ehErd49bEhaPCIsFpSkJCJlubTNctA3J8Rn577P5937Oy1mmiJ+flp7Dd1v55C3LoixZdeoyxUXxldoGvrkYqZcv4sf5L1MbTid2j9OJPO9RiKNTWMt3qlBCBAK+uRi6q6w8durH+YCWw/Rfow25N8Nc9vea+lenkvBb6KhZAdeS4pK20X5ePMgnBTtAuW+k4xrRxfs2TsWdzQ6CMHAomKaLjJYfuEdFGrRnBY7QhF+3XgJyeta0Jtbx1jI0UNJddM3KZmtS+4aSq1pxW2GWVkfiMvLXgfTDzsSA4POiIukOgMZGif+XkVFzc1mcGoWPZ8bZGn+Ri5K2cql9kHAjoGkXg/wrGcG+p+yKF5XNyK5eo+FmDe+wmJBKcyj/UcBfpTzFg91L2L5I2eQ+/yWcdt7ORSttJhwdjKDhTbaLw1y84xV/I9zJ1PNQThknVKXEeIzt3wbe20P9u7dB4aLWkE+gYosms4y818fe5LZ1gZcQ4lBLvR/D/NACtqeU39fI42wWFAzM6j/ZCFnXbeBP2W8c8DwAuwJZ7J1azET+3eP2jAyFmi7tJCcG+o+MqexgkAVyoHcIPsxCZXL9lzGvldLKHh2Q9xORGrZWaCqYNIwkh1U3ZLEollV/Hf2U5SZohnvTEIFVFp1H+eu+irWVU5y3+7DvmsLkdCpSyQU88aXaeXUn5/Ef1c8RooSYEd/Du7qMIY/cFr4ebsW59A5z6B8ShM35mzhfMduMhSBaein2x4WPNc/m3/umk1B2KB3TgYmXxq259bRf+MCeiYLKPVyefkmltlqyVA1wlJhXTAZR7PA3OmN+xWBasUEBqan0zFHYdk5W7g5fQWlH2jZDiUIyWF6L5mEszmIqaUfvbp2+BPGMbpZkGr56GgWAwnyw7+4Lg0KHb3syihCsdvRhxLOxBVC0H1eCf1lCoGcMM5ML18qe5OznTuZYQaN6Mt4wAiwLWziC2u/SuoLdtxbe2BvY3TxzgdPabGg5uXQemEO2Y/tRO/rHzFxY9v4KiqeUif2xV1c7Gjhgf5JVO3LYWK9B2OYBgQcWMMv8nOg34Pe0Rl3jUiYzCjJLvSyPDoW69ywYDU/yYxm6NKwYSAPpAPs0+3sGczEaLLTMUswWBFGBBRKPLPpviTAxRU7uMa9gaXWCBEs1IbDrAsUcnftWbj3hBDtH73qKaYZqtem5mbRvjST7nk618xbw6+z1xEdERy+UitD9VCR307V+dmYm6y46q1k2MyIlk4Mz+CwD148IoaauvERr9Sw1PFJHQVQEShCYBdmDCRnuGpYXVZEcHYp5rV7MAbjLAZaKPSVKxQvrefWwuWcafVgERofbAtbQk7uajmH5FcdpL1z0HUpNA2EgrBaUFJTkHYrKAqDJclol3UhV+Wh7DUwvL4RyYMd08ZXTU7Cm61wWf4uqsIaf/n7JVS+PoCxffdHHqPY7cjJpez+qpm0lZlkPDoQdyVi1OxMehfn8+n/eZ6Pu6pxKhYYGkaqQkEFwkPPxNm2AGeXvIZS8jqqUA6Gml178Hz7h5qGlPy8+VI2vzKJgp9HfcLxOpstzGbU3GxqPpfLjz72Ly5x1JOsfHRW3llmhecrn4NK6DcCbAkl8fO9lxN4qJzU9V3oVTWnUPqxo0XXWeGbQLapD6sIk6F6qTQZKCjc4GpnzvQH+e6PrkN+txBlT118lRaSBpofipw9nG/z81Hm7ae1lxO5O5uMHR1IjwdhMiMjYdSMdLBZCRal0niOBVnmw2iyY5gla6c/yLKLv0fuSgumLXvRBwZOWtyYNr6esyvwzPdzS+oqHuufRcbWMGw7mOLOWDaLQJoZa2eIrmk2+ucHWVSxl4UpbzM3lMLTe5aCyQRxZnzrP1nI1Ct2c51rDxZhOqwXE/6Ino0BICEyjDntN0KsCabxjVc/Q9oGhcJN8b+8WFSW0LI0lX/c9AfKTDp2cezp0JMVKwssXh6e+HfW/SSX2lAGb3VWEvhVLtbVe0bkwRorUqpDvLd6MhS+cWCbLiXbw4KbVn8B+yoHua924C9OIeJQ6alUOf+6tfw0awV2YaZEU7mv7Alu/u0n6X98Oml/XXWEq8UYUlL4l92s0GbxmUssPFT05rC7PVL5KDW3JzFgWPlVzcW0taUgfBoXzd/CTGcDxaZOMtVBXEqYgIz2mt2KjYdv+QO3LLoJ45WpZP7p5HOjxKbxFQI1LZWmiww+PW0t6YqZWbY67r1eR1sy+8BuRmEAu2OAVq+VkuxGPpW1g7n2Wnp0J7977VIKtoaRgfgbTlp7JHu6M7AXRRM4H7owQhlabv/BxRKqEAcWV+z/bk9Y8pJnOi+2TKH3vWwK14exN/RBW2fc9nj3489z0T9RZ7r5wy6G4fDJEPvCCulqmBRFwyJM5KgmzrO347O2UGru5LbpN1G4zQ5xbHztezopUDOZon+DqfP2YddC1HvcdK3LImObJKmqF72qBlt3GsJqxd6UyluB+Xzr1rdxagKTUMlRbXyn+DW+NemzZJaXxpVvXO/uIe9tH1sCk7nkYie/LH2KIi2MXZiGXBCQo9rJUSPAIJS9TE1BNv26jU8kr6NAU7AJM9Hk6YcnUC/VIhiGgrVnZGZJYtL4ClWFNDefP2MFt6ZuxiRMLLF6eeOsPxKQB2dx01SJRSiEpYGKoM8wqAqn8beWJZQ/6kXsrsOIwzIorsYIDTVumH30fQdlmMCQsTUBAQk9hpnqUBZv9E1m+fZJuNeZKPhzfLsZPkjEoSBSDv620arNOrURaIu48Bg2AvLgw9On26nyZZOkBZhub2CiuZ1UJUKGasGumik2dRFMk2Ae+YoFp5LIvnos++qZ8BLs/f4iIlZwNEuK/7Y6mlB8aL8DeW2bmsnZqPHYzXO4OmkTBZqCRZi4wObFVdZH/8wMnDX74sr3q6zcTGFNFi2BCfzq45ewwL2PUnMnF9u7MAn1QMViIBpyZt/vcjry6On9YCr9Dclk7fGMyMgxJo0vQgFNxakGsIjow2ASKvmHhA4dvj88MFDAX/YuwbMhnQl/a0Q27kTGWXHIE+Hv/dPY5c3BE7GgIGnxJtNQlcXEe3oRvgATu6vGZZrNpHdqUYMlcE70c48eZF0wk2+9cRM5bysk7x5A1B8Sz5vupn9mBslv1fDGBYtpP0MycVoj9034J1mqjb3hDMrvrD1qsu14Ivc3xzY0lpEIK2+YyV/+Ywn3Ln2YZbZoCOcXyt/jzmvOIflVF7rHE1cGONLWTuZd7XjugreLp/D0glwyfvVnyrVBkhXzgV7wsWAgGTSCfPO5r1L8Uhi5YWQq9B5VAiFEAfAwkEV0geJ9Uso7hRC3AV8E9qcS+6GU8qWREErqOqLPw/+9ehGvzGzkU3mrKTe38Z2qj9PWczC3vm2jneQ6HUtPBDUQIaM/QOZAA3pbR9xV5T0U295u8t5JZ7r+LcrmNPCpvNVc7Ygakso3vojSbjkwq521xsDR7EfxR4Pi7RGDSb429JY2MOS4TZcofT7sjYPMXXcTJe4etuwtIPNtE5Peb4PefqTXh37IQgHh85Hc1YveP4D71T0kV+ex94oizmn/KpFuG649Kjnd68d1DPCRkHvryXxzFl/i09yz9O8stPbxMddugtNMLM+bg6gNxm1EiN7cRsryQX7R8BmkKuittNEz04DkaPsozO7hOyWvcXfj2dR3pxL0Hxz9yJCCqctE0Qt+Jja1IHv6Rmz0eCzmPwJ8V0q5UQjhAjYIIV4f+u73UsoRqGD/AaSBMeAh7x2D1voibsstRLcZJO9WSe8/6G9x7+hDNLWj9/QBoMexwT2Mrl6cO6HQn0pbYxG35RXy44xoQ8l5TcPac9Co2Ha0oHd0xaV75WSQwSBqWzem50vZl+Qmt9kgZV0LkX3DJ0aXQR19yHjo3T2ouk7O+xX4qx1Ye3VsTf3jp1LvCWAEAqRu6UUYKXzV91mmTq1nVkojTX430qwhhIjbSVoZDqF39yBW9SCArOYinG2ZhJxmENCflst3Sj6Ns0EhqU+ihg6ZY4lILD0hlJWbT31WMyllK9A69LdHCLELyBthOT54UQyPB9uza/kIRwMwfivR6r290NuLqbqWrNeOvO/p2U+LDpUjbe2k3d9+YNvx6ELv68fy8roDNT7i1bCMJMbW3SRvV0l9NYWmj1Wwo6IYw64zKTIwrvSz3y++/7d3AZljIMdx+XyFEMXALGANsBj4uhDi08B6or3jD2XiEELcAtwCYOXDpTlORxI6+TAJnQzPKdeLoaN3dZPx59VkQDSfSIyNKMdLWxl+AfgwCCGcwJPArVLKAeAeYAIwk2jP+I7hjpNS3ielnCulnGs6pJbY6UxCJx8moZPhGTO9SBn9F2OGF8ZPWzkm4yuEMBE1vI9KKZ8CkFK2Syl1KaUB/AWYP3piJkiQIMH44qjGVwghgPuBXVLK3x2yPeeQ3a6GEShknyBBggSnCUIeJXZPCLEEWAFs4+Ac1w+BG4i6HCRQB3xpaHLuSOfqBLxA18kIPQKkHyJDkZQyY6wEiVGdwBjqJYZ0ArHVVjxA1Vhd/xBiSSex0laO+/k5qvEdaYQQ66WUc0/pRWNQhkOJBXliQYZDiRV5YkUOiB1ZYkWO/cSCPCciwzFPuCVIkCBBgpEjYXwTJEiQYAw4KeMrhLhICFElhKgRQvzgGA+772SuOULEggyHEgvyxIIMhxIr8sSKHBA7ssSKHPuJBXmOW4YT9vkKIVRgD3A+0ASsA26QUu48oRMmSJAgwWnEyfR85wM1UspaKWUIeBy4cmTESpAgQYLxzcmklMwDGg/53AQsONIBZmGRVhwnccmRJ4CXkAyKo+85OsSiTgA89HaNVQhRQifDE4t6STw/w3MsbWXU8/l+cB32AnHuaF/yuFgj3zj6TiNMrOsEYLn89/DpwUaJhE6GJ9b1knh+hudY2srJuB2agYJDPucPbTuM8bIOeyRJ6OTDJHQyPAm9fJjxopOTMb7rgHIhRIkQwgx8AnhuZMRKkCBBgvHNCbsdpJQRIcTXgVeJVjD8m5RyZOprJEiQIME456R8vkNlg0akdNBxoaiELphNx2wTvvwIwhlNoy26zCTVKmT/bTOGz3fKxRot1LRUyM6gf4qbvjIFJGhDt6eEwdZjkFTlQW7eGVd1thIkGBOEQHE66b90Cj2TFMIuAyFB9QkMM5j7BLkrfKirt49qWanYLKB5BBSrFSUni73nq1xw5kY+n/7uUPlweNnn4nf7LiC4ZzLWfT3Q24/R74nr8jCKy0V4ahFd02z0TY0wqbIBX9hMc1cKAJpJp6/Xiic/mVzLNLTmHoy+/nFZNDPBkVGsVkRyEiS7iKQ7McwqHCUOQRgSEZFoHQPQ58EYGIjbWm3HhBCoqW6MklzaLgnxqRlrmG5rREew2VtEsbWLlX1lbDSmUrDJkjC+h1FRzL7LU3nzY78hXTGjCoExlGztYruHZZMfY83dSXz5pc+RtSqT1NWtH1nXKx4Izy2n7osGry7+DSv8pfy7bQ5NXSmIxmiBpZL59Vw1dTPnX7iH+29ayJNPLSX/LT/Kik1jLHmCU4qiIiYU0b44lZ6FIe5c8hhLrV04lcMnpBQEqlDQpYGBpN8IsCWUxBfe+ByZK7NIW9eFvqt6jG5i9FEsFgJzSqm7TKX63HsIyghBGcErDTJVD1PNHs6yV3PHtWGaHkrC8HpHTZa4M77SpKLbJVYhUMWHX+t2YWaBdYBnrriTx5fO54ntcyj7VPwa374JFowug/Pf/iYTf+NF+AJUhjqQQ5V5hcPGE6UXc+eCq/j3l24n+Xof95QtY1JjIZH6xnHlhhCzppD+p2bOdu/GoXx070wR0ZexIQ/OJ2/yFfHv7bOQUrCkvIYJ9i4e2nIGk/6rjUhzy6jLPppoOdk0f6yU677wJmc6d5Oh+MjVBE0RgTdiEJbqYfur6OgIVMClKMy1DPLCBX/k7SUV3LtnCQXfLkJvbB6XlZyrfzGT+Qur+FPei6jCztQ3vkzquxYyVvfSNy2F6d/ewt1573Fb9utcc/Z/kLrKQqSuYVRkiTvjK3SJEhRcveNTXJy7k4WOamaYB3Ap5gP7WIVGpQkuSNrO5vz8ESv1PBak7vBh77IgFRP6rpoPl3XpBItnkLxgIfdcu4xzkndx/sRdrLt4JlkPdGAEAmMj+CggpCSga8y0NlCqRT7Uq9uPMjTWNg4p+7jA2kj+7F50BGfZq3ApYarLMuk1J58S2UcT6bAxUK5zRdJmslWd6rCNn9afx5qtZZj6VJTQ8L4HwyQJp+rMmVLL74qe4SLHLqrys9mTWgFN4yvnljCZEZWllMxq5tzUXWwMFvDNvYtIf9NC+qpOjL11GLPmYRIGqlBIVS1IBRimgzdSxJ/x9YewdUgCT2Vx/4JUdk7M4b/zXsQuJMoH9JShesmxDdA0NqKOCGLVliNWcIahUugbfLxSPZllc6q4InUTr82fTPZjFhhPxnfQz8YtE3jAupSzk3cxzdJK6jA2wqWYUVAOuKMA8jUbX06pBaApEmZbKBNfxHSqRB9dFAVp19kYKMRrWHiru5Kdr1Qw6ZluZEPLR/r/FYcDObGYHZdXUPdpJxNNXgqsPeyymdBUBRk+xfcxigizCW9ZEtdkr8ahBLm34Uz6ns4j951mInUNCLOZnilQaW9Dlwb9Rgiz14DQ6Ckh7oyvXlVDRlUNANkv51N93kRWfL+KAlct0Yi3BF7DghgwgTF+XA4Aes0+yr++jz2LZvD2mXMQC/r4WuU7H9rvMmcVWepHv7I+t+dGOt7Mo+jZLoymulGU+BTR1knpP1L408prcTWHse1qo6Dx/aOO+AyvF7W+jax1dnZ/PBeHqKM56EYNRDjVRRZGHSkxeXRcSoAN3mJaVudSfM8qIkP3qVgsvHLDbynW7HToPh4bmI5zZzd6V/eoiRRXxlfMmYK3yIknL2pkfYsHubpyFVc5q7EI64HJhLCEHaEI16/5EhlP2XGyeowlPwWoKtPzm8nTenmufxa570qMcTprrW6poag+Gfmsg2ec53zo+zsvuZLkBR3cN/nvVJqibaUmHOTB3kX8c9080tZpFGwawNhbH9eRMPvRPR7Ma/eQsdUKwSD6MYZZqulp9J9dxpz/3Mgljj3c2bWUZ9fOZuLO7eOu7RiBINYdTfzqxauwdCuUvNaPcDoZPG8y7fMVzjhrB/mahccHM7ir9izCT2WS0bwVGRq99hHTxlctKyFQnMpAiZmwUzBYaCAyg2Sn9QPwxcJ1nO/YjVuxHjhGl9Gh5terbsD+vpOUtU2Mv2mDwxEmM0p6Khekb2Cdv5Tn9k2jcHs3enh83rnh9UZnoT+0mD1KWuECmrNTaalIJkPp5uft57F8bwXaDif5O3Sctf2Ipnb08WJgpIy6Fo4nvFBR8c8tpWOewlfS30EFXm2YSNpGdVzFyB/A0NF7+8hcD1oggtAlnR+fSs+SEGdPrOLHOa9gEU62+gpoq09j0nvd6P7AqE5Yx7Tx7ZuTRdtSyccWr+Js1y4KtD4yVOOAsY369MwHfHtBqeOTOm26iu+ZLHJXdI/aTGUsIExmhElDyUjDOzmbq5zVfKH2WoyNyehV74+1eGOC0DQ0v4HqU2mLJDNgWHnrxdmUvOSBtVGdGEc5x7hGUVGsFhR3CnVLNOYsrqLUZGK530VwWwr573XH9QT1sAgBQgFD4qz3E0420zcpieLPVPNQwfNMMdsAJwA9IQfqoIIY8KK6k0HXkaHwqLyQYtr4dl/j40fTX+OmpP2ZKzWUI6SjeNabx937lmE8lEnOigaMUfTXjDXCZCa0bBq9FRb65oS4fekTuBUrTf3JWHrHWrqxQWgacvYk0v9rH7/Oe5W94Uxu/+UnKX2nOa5jvUcStayYtnMy8Zzp57Y5/+Q6ZxthqfO9Bz9Hyase9J17xlrEEUdNdUNmGv7CZC66/R0+nbwJkxC4FRuqOHxu4Cc5r3LrdcvRrxPcsOHzBBpdJNUoZN27fsRdVDFtfG0rnfwteRE3TXti2O9bIkFadDvVoWx++uZVpG1QSd3lw7SvDr2nd1T9NWOFXDSD1kUOlCW9XFC4kTxLL+WWNuZZuqkJC75cvoLX3JPZWLmAzLWC1A3d6LtrxlW870chbDb2XeHk22m7CEgT/2qbS8a7rRit7WMtWkwQvHgezWdrfOWSV5lsbSZX6+ePvVP4y/MXUPpyP0pd67jr9QqLheabJlJyzV5mJ+/mxuRNZKp2IuisC0o+veYzRIIaE4taqV1RhLlP4MszuPPKB/njzMfpm+bgiY55dNTOwLG7E9nVgz4wMCKyxbTxdVeHqWtMo31ykCz18JjOXeEwtzVczfbmXMK9FgpfBufWJiL1jePax9t2hgPTmd3839THmG+RKAh2h4P8sfsMnqmdTqrDh00Lk1vRSVtaMr6sDDJykjC/v2NcxfwOh9A0Qtlhck29hKRKu89JSlvruL/vY2UwV8M8YYBvuKMr2BoiEWp8maTsBqW2Bb1/ZIxKLKGmp+Gp0Lmr5EnyNScduuANv4W3PZN4umYGya/ZMfklDUXFFL7rRevz4StO4RtpN3LRlB1cm7qOL2S/y5eumEBGRg5pm22w1TMinZmYNr62fb1YmzNZ4S/iOmfbAZeDgcELAzOof7SM8hfqD6xQGs9G9wBn9vKDia8wyxwhLKHfCHFP57m89dQcih+pxzc1l9bJJkJneHh1yf/x2LS5PDBzIZUNOYj65nExu/+RSAP0g8Heihj/vf3jwTCDWTvYtw1JBVVIgikCFIFQBFKKcTVKiuSlgSuMWQh6dR+veIu4a+/Z+N7NoOSZDmTjXgyfD9fQ/jpg2QUVy82s/ewctlyby+8rn2DHZXexNO8mOk1pZNYnRV9UJ6mnmDa++p69ZK1z89+lV3LdufcC0dVLJmHiu2nbebDkHDJy0yDOl4ceD9m/tfA/F9zIM+ftYt9AKua703Cuq6egez2RcAhzSxu5b6iIe01c/u3vc9aVG3ntzP/j94+dS/U3JqLs2Dduk+7IUJjUDRq1SzKZaR2/E60nSuquILUTU2Bu9HOpycRvct6m8zvLuXDhV3G/5CD9nejocbwgNu5i4q9Lue6pb9NfopHzdj/pLZ0YPfXokfBHGlAZDpH+wDrUN/K5df43qLx1B8/O+Btvlhdz55XnkPXddIz6ppNKQhTbawilxLGlheK/K0x76Js86snBQKJLAwWFy89fQ/U3NXpuXohitR79fOMAbXcDJU/10vzrcqx/cONcV4/e1X2wR2voyHAIw+ul8JUB3n52Njft/AxfyXibfbdC32VTQBm/i1GEAWGpoh8tnddpiHlHI2VP+Jn/q28wa82nOWfbx/luy9mkqiq/mv0MWZ/fx+5v56Hl542bNiIjEWR9M641DeS91IGoacDo7ok+L0fpucpIBKOlDfeaVrbfO5WmiI2LHPX8ZOIL7PtfO6K85KT0FNvGF4g0t2BZuZPSpzz8tW4Ja4PiwJr9r6e/y3fmLKfnnACKO2XcNJgjoXf3YGzZhfX5tZhfXU+krf0jE6CI7TXkrArSsyGTXFXys1nP01euIEwxPeA5YaSu42jT6Y/YcYgQOY4BRH4OisuF0MbnPR8PemcnYtVWsh/YguWVJDyvZbN85Qx+2bmQyeY2flz4PBcv3kTv4gJUZ+wVpTxRDK+XSGsbelUNhsdzXAmDjEAAvaWNjBVtdBsOMlUH59n6+PH0l4ikWBEfzGlwHMS88UVKDJ8Psb2G/uXZfH/PdRhD/+VqFj7u2s2fFv6DcHEWanLSqCbCiDeMQADL9kYyNhtsDLm4wtFOMF0ft4ZIBoNYX1jLnsFMijQ/n85+n7ZzMpGVRSgpyYe/nE/XdiIlhtdL+n2ryPnd+0y8o4F3bj+Dv3YvQUXylYy3sX2xBbLST4vOzIlgYNCn2xGGRJ7EEv7YN75DGIEA+XdvRrk3g2VbbiAgo28vl2JmibWfsj9WsfunFXiuP2L1+tMOvaOT5DXNfPvuL7E9LLDlDeJfNnmsxRpV1m8s4w/dSzjH1sOjP7iDsnuqab0/g+o756KWlxI+bw7+K+aNtZgxQaSlFfcz23h+zzQ2Bwop0VQeq3wM78R01Iy0sRYvJukzIjy07wy0/sCHswweB3HVBTJ8Plx7+mh8M5PvpJ7Pl7LeYoYZTELls+kr6JtjY41RSfILjuiKlHE0a3vCSInR10/WWj99up0kewBfRlIc13w9OiXPRnhjy0JeyFxExCEJFwZJSfGSU97Jvl86SHX1YlZ1qq+dTcmDAuuuZiKtbWMt9tgwNLJMe97G3c4zmTb1UWaYzRiaQJyuo4OjYADhiBqNrjkJxtz4atlZGOluwhn2aDmTfj9K/0dnjzdMKsl1Otu6c+hJd6IQXfY3ywy3ZL9Dy+RkFJcTwx8AGf8h42p6GjIYwvD6Tvwtq+toniBhVOymML32kZUx1tDe2MD+PpualUnH5RPonWzGX+DlExUbSNW8ZGgDnDexiWW7v0duOAdtwDOqVQvGDCEQZjOKzYox6B3e3ykl7teqqZpfzrayAmaYx0+0w0mjqChOB+G8FKziYHrJYFgD/eTix8fc+LZdWYpyeTfvz/oL+yIBbqm6kbZVBSAkyGHevEJimOHhSY8zyRwCDiZRP8tmcGnuNt5yzUB0dZ/siykm6LmwHGdzCPOWfei9J7ZuWEl1U3dxCtnqAGFdRRlHeVqPht7eQdpfO0jXNBS3m9VZ04m4bXRPtcKtj1NywT5qLCUUGuWI9zaPtbgjjupyQVY63onpONc3EGnvHPYlrnd1Y+qvpCGUBiSM735UdzLhSYXU3GCiQBsAHHgNhUCbAxEePKlzj7nx7a+Q/L+KVwDIV008UPl3esoPGtQPhgypSFQkZSaJSRwufljqh5WOGQ8MlCp0XWpgNudRdFsW1NQd14otZepEWs5M5Wc3/51u3UHr5mzKn60Zd8tIj4aMRNC7uhB9/WiqQk5NCn9qu56K/9zBoqvfYP3ZhXT9bj7Ot6vQ+/rHWtwRo+/iySg3d/Djsof4+rM3k/9mAZYX131oPzU9jXCSQaF5fOVDESYz3TfNoWu+zqTfd0PvANLrPaZEOarbTfcllXScHebVc3/PBM3GgwOZ/G73uZT9I4DR3nlSso258VUDgppgFr22VpIVM7mahfxD5gGND+SgUg5JmL4/6mE/b/mdPNM0HfeA56RmIWOJzI1hGt1WCmY0sfvb2SRvmU3K3jD2qi70vXXD+rUVqxUlOxPv5CxaF2lkzG1jkbWFa3Z8hpTd0XC10xIpkeEQMgyyq4ekdSrrn5jOxrN6+NSEtfzflYVM2pEG48j4BpMF1+bu4GzbIGcs3M2G/skUvvjh/fxzSrAWephiaaYmHEENGOOjhpsi8GULls7czcqvTcbakYV7j07K6mYijYfUuBECecZ0TE3dSL8fMlKpvyoDMa+fL5atp8IUDb37W/1izC+koFXXRFNOngRjbnwdTfDPulksnFLNNJMPu3J8pV10KfHJMDvDDu5qOofuLZkk9zWd1CxkLGFdvpUM9yxqs9P56eJn+Gv+Uhp2Z5GWmU2GxQyGAeEIIhBCOmygCHSXlb4yB+0LJefO38qnMt5jRSCPwbeyyN3hGTe6ORlkOESkqZmcu7uoc88mvXKA/zrjZZ5KOXesRRtRpArJWrSX91+5L3P9jOgiCun3R3fQNHA5aF1k5oKiLeRqfv7UvQRzf2hUS+icSoSEHOsAd1/6AG95JvHPzXOQSj4pW23R50ZVMFx2Gs9zkLLHhnlQx5Ovcdn173OjezXTzVZ8RohVQRttm7Ipe7YGvfPker0QA8Y3455VKO9N5H9uv4rflP+Lci38kYURh6PLCPHSYCX33HslyXURSpsGTmrJX6whg0GS/74a91N2Hlp4JS2f1/nC2W9x67Xb+PdgLmGp8X5/GW/XlPO92a9hEjpp2iALLNHZ+//rXsTN791M8YMK+e9tTCSZ+QByRgXBnAjF5i6qg9ljLc6IY+mTPNM6k5uT9lJm0vjPKa/ys/93ObZNNoQBgXRJ0YIm/jnhdxSoBk94Knn/ZwtwbtmOPg4mIGUwSOEDNazZNo8b7lrDzzLX8csLNjJwXoC/D0ziyeZZFLq6uT3/72SqDnxGiDA6VqFhESbASr/hZ7kvizu//wnK1zcQGQHDC8dgfIUQBcDDQBYggfuklHcKIW4Dvgjsl+SHUsqXTkiKPXXYvlPMFy74Fv4sSSQzxMKKWn6e/zz52uGGeGtI55GeRTy7cRboAlOfSvomSf579cjBQWQozPhwOByO4fdjXruHitYcXq48i79PPhfbGV2oiiTf1cf3Zr9GqbmDB9uXsL0zG0+3A/d6EynVISob+pCtHeOuNMyhCE1DSU7Ct2ACHXNMiAhkbB2+5xZ2qAzmKaRf2kRlcjVfSdlOqTbI7uD4mi8AcG/tpfXpAtq/FyJD1bjIUc/ERffRMt+NLhUcSpAirZcCTeFZbwF37zmT/Pf3nfSQOpYwuntwbBDc/Ntvk35tI18seJdrHb3clLSLC507sQpJmhINAfpT3yQaA6mcl7yDPt3Oo81nUL07j7zl4FpZM6LzAcfS840A35VSbhRCuIANQojXh777vZTy9pMVwggEYPtuck1TCKVZCaaa2LZzEueVloP18CGyGNSwtqnk7dYRBpg8IWy726KZzcZzXO/+UjG7fST1pGNvzaKjNx1DgSpXBptzSkGArUXF2i3J7TVI3toJrR0jln80lhEWC6Sn0nCxQn5FK4qQNE52H7LD0P8lqKYgqUk+Ppm3lgxtgLDUeGxgBn/edCaV/QPjazKyrZOMDVau2vRFbipbyzmOXcwyK8wy92NgEJY6nbrBt5ou4M0dE0nZYEbv3DOuXFMyEkHv7iXn7S5a1AJ+VPBJfpA5fHY/U6MZzS94KXUOIiJwtAjy6yM4V9eh9/SNqF6OanyllK1A69DfHiHELiBvxCQ49FqbdmACTOwv6nFsjINpgWPH0Im0tUNbO5lHqQs6fh6foyPMZiKpDs5fsJVJjlZKzR1MrIwOylxCYldU7CIaRdOq+6kOJ/N0z1w0RWdXXzbVjVkUPa5A28gMKWMFvbsHZf0gqX+ewd0fO5vdU3L4XOa76CiAisdwscVXxJpnplP+jhdl4zbkODK8+5HhEPrOPWSdYKWO0dDIcfl8hRDFwCxgDbAY+LoQ4tPAeqK94w8FogohbgFuAbAyzqP7j5GETj7MyepE7+1FrOql7gyV+hmLGahw0VsZdSOYZ/fy8dKNfC9tJwBf33cdNa+XUvDLNUNHt1JOa7TI4ojczcgxEm1FhkOYX1lHxWsqTcDPlfkf2idPXxWNBjkZYU8R4+X5OWbjK4RwAk8Ct0opB4QQ9wA/J+oH/jlwB/C5Dx4npbwPuA8gSaTGw2876iR08mFGTCeGjqhpwN1qJ2VTdPykv+ZguXMJLyedBYC1M0RxWxt6HPTwRrStDN1vvC8+Gi/PzzEZXyGEiajhfVRK+RSAlLL9kO//ArwwKhImSHCcGB5P1D/edrB223531n5i3+wmGO8cdXpXRLNr3A/sklL+7pDtOYfsdjWwfeTFS5AgQYLxiZBHiRAQQiwBVgDb4MBysh8CNwAzibod6oAvDU3OHelcnYAX6DoZoUeA9ENkKJJSZoyVIDGqExhDvcSQTiC22ooHqBqr6x9CLOkkVtrKcT8/RzW+I40QYr2Ucu4pvWgMynAosSBPLMhwKLEiT6zIAbEjS6zIsZ9YkOdEZBh/UeUJEiRIEAckjG+CBAkSjAEnZXyFEBcJIaqEEDVCiB8c42H3ncw1R4hYkOFQYkGeWJDhUGJFnliRA2JHlliRYz+xIM9xy3DCPl8hhArsAc4HmoB1wA1Syp0ndMIECRIkOI04mZ7vfKBGSlkrpQwBjwNXjoxYCRIkSDC+OZmUknkcXm+kCThi6WCzsEgrjpO45MgTwEtIBsesUmAs6gTAQ2/XWIUQJXQyPLGol8TzMzzH0lZGPZ/vB9dhLxCxlax6jXzjlF8z1nUCsFz+u/5UXi+hk+GJdb0knp/hOZa2cjJuh2ag4JDP+UPbDkNKeZ+Ucq6Ucq5pXBcsP3YSOvkwCZ0MT0IvH2a86ORkjO86oFwIUSKEMAOfAJ4bGbESJEiQYHxzwm4HKWVECPF14FVABf4mpdwxYpIlSJAgwTjmpHy+Q2WDTqx0UIIECRKcxox5Ac0ECRKMPYrLBUV5hNPtiIiBtn73uC62KiwW9DMmY2rzQHcvelf3KZchsbw4QYIEhOeWM3B7iBcfvZc7/v5nREnB0Q+KU4SmoRTlc/8j/8fuHyXRc2H5mMiR6PmOY9SkJBq+MhXLkmimu6zvS4zaBuQ4rmIM4L1uAc3nSn5xzpP06XbuevxyCl73IlZtGWvRxhTF5UJoGuj64UVVhaBjlpV5KR3oUmLIMQvbPTXMmkTdhS7SFTNCjF0hjHFlfMW8aXRPddIz46BC1YAg5z0dx4qqES37fCpR09MgI5VIih11aw2G13tMxwmXE29pmP+pfBUVyZ/TrkFrMo9f4ysEWnYWrYsFF8/dwvXODnwyxEMLOmkfyCB71VgLOEYoKmp5Cc2XZKKEIPudHtjhOVjtWyh4poSY6Wo88nnGCb5cG8Z0DwBqsxVn89g8DzFvfBWrFSUlGZmaDOEIIhRGDnrRu3sO209NS6V5kQv7Re3snP44PiOMjqQqbONTrq8wcU8GYtCLjMRXrWPF5SI8sYCeyTb8mYKSxpRjNr4yyYEjw8fVjqiu/phixmS1gsczmiKPGUIz4Z1VwKy5NXwt4y0MNKxC4xNFG/i/yeeSPdYCjgFC01BSkulYksGEq6vZ05VJZKvtoL9RCBSrhQUTa1lgrxlLUU8Z/jSV80r2YGDgrAfL3o4xqYAe88Y3smAS+y6zcvfVf+XV/mm8UjcJ9f0Ccv+4HhkORXcSgqbPTmT6tTv5XcGLbApaeWFgAYO6BZPQ+f7ZL/DQystJDQSJ1MfX291zwWQ6r/fzrwW/p9Kkctk7t6A0fWgty7D4C5NJsR98SXlzVBypydA5vsqj70dNczPn5xv4UvoKSjQrxlDhFYsSBk0iNC3uXr4ni1qQR8/CHP764z+w2l/Kpu0lKCvWHvhesdkQJQV8J/fvzDIrhOX4r24XShJc4d6IjsTRZhBpbBoTOWLa+EbOncO+qzX+cvF9lGr9eFw1VE5uo6M8ifdfmIpsbMHw+1GcTsqurOamzFU8MTCZh35/Cak7fCiBCKiCjUmzcW/bG5duh7Zrg3x96ruUDv1STWfbKNBnIN4/iv9SUam7WuHLedsObApcPEDvYDpJVeOvhyMsFqQ7iXLbduxC0m8EqI+Y6NYdLLVX805FBZ7JZcjt1Qeq+I531KQkOs7KJe1TDZSZJE8PpKB61cN3EgKpCqxCR0Glywjxp44LEcHw2Ah9iugz7Fy49UqS2o8S0XHGdPzZVpxVvRg19Qc7fCNAzBpfoWm0zbcwcUodcy2DfKfpIt6qqkCGVDRHmApvJ0Y4gmK3Q0ke12a9TkfExT8a5pL5bgdGbX20lyMEZpsN3e8/6OOKI7JSByg1d2AS0YdGqiBVhWOZEjG7A2SZDr5wcpIH6LUnjZKkY4uak0Xv9DSmWRsxgH97KvjD9nMoTu/he0WvkGkZpN+ai1BE3JdOP1ZEchLeXMEP8ldgE2Y29BZi7f5Ay9F1RFinU3cQ1Py06Rbe3FPBxGD78CcdJxhSwR8ykWx8tE3QcrJpONPJYFmYPC0VV0PLiBrf2Aw1EwIlJRltQS835q6hKQLvvTqdyjt8TP5pCyX/J4k0RxWhpCTTM8PNPGsDy3sn070pE33P3oPDSykxfL64NLyHogz9Z+kBrdd/TMeEfCY8uu3AZ5OiY5ijfvRxhRAESjNoWyyZZY7QErFxb/VS8u41U7OhkC3+IhRhYJjVI59HUaP/xPiY7Y/kpRLI1Vlma8VAsqspG2fT4W8eKSUiEKItkoJPhmmJuHFstiF9x9bG4hWT0ClP68SwfESbEAL/tHySzmnj92c/Rk+lirCObB6JmDS+anISHVdV8F+TXqHY1MlPGi+n5PZtGNuqiDQ1R0OGhoypb2ou13x/ObmqyootEyl+wTfG0o8sAlDFwQcmc7MfY/vuox9o6Ez6VR/3Vi1BQWASKhdk7qR3foje62aOmrxjgZaVSeN5ZnZdexcmofLJd2/B9lgK5vd2ICQowqDS3kbbIjvCMvwDJDQNrSgfdUIRauaYZY0cUWq+rvLds18iWbHSqvtJet9G2sqWw/YRmobhsjHN0oxbsdEdcZJcqyNDI9fDizkkWJUQfyl+jkCqedhdhNlMyxeCfLXkHRRGZ6gUc24HYTJDbhaZN9Yzz9rIT5ovo+ZfFWR51wzbezVMCgvse/EYERSfgjoYHCVVnVqEyYyam0W2Y4AU5ZAXykfdnKISuHQOUgVfhkr3wjAFed38uPglDCRIgyud2ylY3MOjExbQ2zkX26o9h8d7xikDi4qRxX6Uob6ENARIQFUpntPEbFsdbZFkvEU6Qo32dNSUZHyLKuiYZSKYbmA4dRASkzOEbVU2ufcOxO0KL8XlYuDiKXx8yvtc7txFqw7nPv49JqzpR2896E7QCvLxTs2h4RKFDCVCUCp0RVxYe0Kgj1+/uLs6zDfe+yQbzrkL3Syi8wWHhF9qebl0n1XIbTOeYJmtkfcDuahB4AguihMh5oyvmp9Dz4xUflrwNLvD6bxfU0rF+wPIYSZJ1IwMfJkqRdoAO8PJmPsUhD+EOqkcz6RUzP0RLC0D6Luqx+BOTg5htRAqTmdu0loyVD/9BjzhmYgaiCAVFcVqQXGngElDT09ioMxJ6zk6qBKH28t/TnybebZ9FGhhwIqBJF+zoYoWGtOqeSJ1AjZTzP38J4TQJYYhDkQ3mNpNONoCCJPGsoxqslUvZnTKJzbTc/lkhIRAqkLf1AhTJu6j0tVOptlDjS+TzZ15hBQnJ1pea8xRVJQ0Ny0XRrg0eTM5qo2acJD0zRLFF0Lk5RDOSsZbYGOgUGGwNMLF87bgUqJtwa156ZtgJWOTGeL05XM0bM2DOLe60c+RDOYrpJQWHrQRioqRkULXbFhia6Q+YuOJjnm4ayIjPhqIuafPMyOLtnMjnG0LcG3Npbg2WZHrN0a//IAvLjSlgP4KyNUs/KVnCtZuQFFovjCDadfvZPW+EpLfSSc9Do2v4rDTNcXGua4dFGlmasIRbl99IRP9gyhpqZCewsCkVEJOhZ4pcM0Fq/hJxlpMQsXAQB8yHqo4fFi1M5TGm10TMfmMcdO7cW1tR1mce+Bz6naJaWsdIjmJWfY1pCqQq4a5v/xxfnnrecx0NnCGrZZKk4ouJUEZodOQPGrMZ9XKGeSs88XtQhTFaiGcl8ry8/5AvmZBQcVAoIYkvtIUvFka3TMNPrVsBVckbaLSZGATZiDaTuZY67jjQi+Zr7lgHIyKhkO0dZO23UFASnyz/XR1p5M6FMmgupMZLHIycU49LkXlj63nseWtCoqfWTXiI+qYMr6K1UrXVI3vLHwRgN0tWaR2Rm9ZsdsR+TlI20Fjsu8CC5MX1qKgcG3KeqZ/s5HQN1QucdTzQP90VnVOJLUqPt/eRqab5KtaKFAHAQsZqsENs9by5m8qMKtOKlNa+EnOA6iASQisQsUkTAB4jBA7ww7CUqXc1E+OenDS7VsbPk7eX0zY3tyAPl5CrgJBxCG30l+mELlmIr1TJQss3biUaJvJUFVuz3kXAHUoeuQxTyF37DgP04okcv+6hfzQemQkjsOsygrpmG2nWLMf2FRhMvO7395FthrELgQmoWAVGtowj3+qEmJeQQO9puRTKfUpRW/vwL5d44XBSr464x3+lTqLVvdc8v9Vz76biyg8u54ny59Dw8KaNZVU/qOL0XhSYsr4UlGMvzDMJc4dgI3PT32fZ1Kms2fRfLAa5Ob24Lb2Hdj9i+lrWWSP9mpTlRDl5nbW+UtYtPKrOFY6KNkRwLSzaVQUN9oYVhOX5q7DpUR7+3ah8gn3Ws5y7UIVBimKnwz14OSRLiV/7ivlqZaZ1NVlkrzNhOaXeC8Y5L+mvcwNrqivT9cV1KAxrmJdjQEP9hbBH3omc2vqTm689k06Qi7KbR3Yh15IMBQxIqAmHGRdoJAHGhfT92weWTVhbPs60I9x5WAsIyIGSkQSQUcj+oJREEw26ViEjf1Big0RHw5F4BQmLCJqBvZG/DzYs4ht/5xMXt+uMbuHU4HR28e9f7qSvplhCgq7uOjT7/Pk3Fksm7CVK9M2oqGyLigxDYpRi3mOKeMrTSrogt3hdOoiYRY6qplU1kxLkRuT0Mk29WEVUUVkqF6KNIldmOk3Avy24zxqPWnsbcsg9WUb6e82YnR2o/viN/rBkNEJJAUFi1CYZIIppiAGkrCElQErHsNGWziZ3f4cnt04C2e1ifwanaRVtUivj7rMqbxVMIkbXR0YyOijF5MxLieO4Q+QvC/CfZuXcPWZm/lO6lYsQotONA4ZoKAM02MEecozlVfap7CnMQvHdiuFzzeit3Wgx6mb4YMIjw97p5sXvGmcY2vDIjSCMkIYyfqgnb2hTHb7c9jam8f1ues5y15DsabRbfj5Q8e5vLx+OhPf7MEYjP8X0ZEw/H5yn63HNFhI48IMbipcw5UTt3BO8k6mmbswsPGTfVfibAT8ozN6jinjq9S1kra+km84PonVHuKWSStZat/DMls3m4IO/rfuUpp7k1EUyeycRr6e/QaVphDbQy62/2I6rg3NlDZtBhiTtdojiToY5JHq+Vw9ZzMu5aC3ySDay23XQ3xn22cY6HFgbTCTtS5MxcsbD/Ro99+/vU2ysS0ftVBh3K4uMHQcK6so6yrinzPmcGPyevK1w+M3e4wIz3qm8Lf7LyHvzT7KN0fnEeK9nXyQSGMTyYbBf6y4nr+e9QC5qofaSCp9uoNf77yQ8NYUsleHcVR18r8/uQTrGc9S7OzkbX8uKx+bzcR7txxz7pC4RkoiTc24H2nDvWsyd3RdiSzz4p7so1TrwVANgnfkkLWxlkjb6Cw4iSnjq/f0kvHoJjKftIBQeM1cwevKRFAUMAxMYR/F+iBoGrXLJvGX7+n8IPtVnumbjWt9E5GW1rG+hRHDqK6j6FuZXPzd72It8JDmjPbg9fuzcDT5EYYkv74DGWhDhkLIUHjYiBCIRl3p0sBAIuGjw9XiGL2vH7GpipWfn8eT88/Bv8zDtsUPHvj+zs4z2fjD2eSv2o4+znt1kZZWJn3Xy//O/SyKLjF3eBFePwWD7Uj/0MpPdwoWe5hycxsR4EfPfJLidf7Tw/AeiqEjNu6idJcNoWnc/6szcZ/ppShpL5ae4KguNokp44uU0djKo4S4CE0jYhGYFJ3qcBrPbZ/ORO/euF/FdigyHEJvbaPkmUxCKQ50swuAlDXNyP4BMCSRQe9Rfbepu3w05rsx5kV1Mz2/mT2Ly8lbMeq3cMqRwSBqdQOWikkMBLUDoWcAQcOEuTcYXe04jvzdwyIlel8/1i0NII1oFsBQ+OB9KyqByfkUpbVRqoUAM846ganbG5fzIyeLjETA78dz1RxKy1rJ0Ab4futStPZ+9FFyOUCsGd9jRSgEUxQcapCGcBq23VZkeLwNIKONQn17I7ZDth3vXWp7mkjNLycow5iEypLUvWyePH6rFOh9/RgmgWY+aEY8RoiekB0lpGOMcKB8LKN/RPY6oar0TLZwXkoLyYqVLt2PqyWC6B88xRLGEKpK20LBt3K24FCCvLh9KpP6akc0l8MHidupl4gd7GqIjnASrgbjtEsVeKzoXd3YWwLURiAsdUotHaSnjc98vvvxFAomZR/0060MZLGpJR/R2DZ+/d7HgTBp+JYOsthVTa8R4E1/Ec5dPWNSxyxWEKpKxuROJlubqArkkrLOMuqx3nFpfIVJQ1ncy0JHDR0hF8nV3nGzYGA0MLf0cfUzt1J7Gr6f+o0Q337jk2Q8akPv6R1XrqkTQZ1UTsuXZnLnnCdYYm1ndSCDH797DXR0x+3CkpGkM5LE+v4istYOjHp+i7g0vgDJtgBWEcavm1F7vcjTaDh53Og6pgFBeH/o2hjWrToVhCr8fCx7Pf+/vbuLbaoO4zj+fbpu62BjqGNhsAUHggY1olFiApFbYwy+XIFGLzTRC0nQ6IVBY4xeq5dEjOiNiV4AxvgSEg0GiS8BYTDYHC8DN2Ew9gJMunXrOY8Xp8ayFruxnpeuzydpsrZs/4ff2men5/Q8BRh0hNruOHWHL5R94wVvwD7rhrmnapD6WIKTqYXMP1RZ1o1XqquJNTbwwIIeulONHDrXTKy3H/V5g64km6/E49ycSDInliKtMWTkqr2cNIA3QPyh207ydJ33Erpj3BujmD79Z8iVRcNoQ5xX7viBhgrvhIu+8Xpu6Ryb3VPMCojVJEg31vPo/DbOpeaTOjcX50K/73+sS675xhIJaGpka+sOVldXEsO2ZqZEBSdzdtNs/XTaWCLBwJN3snxOPxPqkHQn2HLoceb2lW9juYYIGpOcEYkqAlJyraB44nHStVWsqr7EvHhwc4wLJi4iLSKyR0Q6ROSYiGzO3P62iJwVkbbM5RH/y/2PA3x8eSF7jq/AGbR9eVPVnWpkYKAu7DL8UVHB6AKhPp6kJz3KJ5fvovnDSqrabasXIFZbS7oGGuPeAddT6VHahptJnOov7XkWM6SpcSqHx/hxdBGDE3MDW3cqf+7SwKuquhJ4EHhJRFZm7vtAVVdlLt/6VuV1fHPxbqpPJcr6gTNdeweXM6djln2SRRY3DhW4XHRq2De0jETnWdwS/Ow+X0xMkLikfHphDcPuGG/2rqfn12bcoUtlvfGi4+NUDI+w9/IdDKaCa74F3+erqn1AX+brERHpBBb7XdhUHOltZsEJt6wfOFPiKrEJGHETHPlzMbftK915F4VUpKBvYj4xUTrOL6R1+IS9DTHDHRuj7vRVft93O1+u/4Oju2+n9bsruCOz+62HhWgqhds/wPenV1BVlSaeDGa33LROshCRW4F7gd+ANcAmEXkWOIC3dTyc53teAF4ASDBn8t0zEutJUN91peT2+vqZST56+QpNP6d4a+1jJLoSVP7VG7mZBsXIxE0mafniDLtYR7JJiaUp3aHoGcV+rOj+dpYeEHa+sYgW55eSzMeP54+bTLJkQ4d3JaCD91NuviJSC+wAXlbVKyKyFXgXb3TAu8B7wHOTv09VtwHbAObJzTP+TavjEkuNczDVSN0ZkOM9Jdd8i51JIc7fV6k+fJqKd1qov3ge53y/30tOW1EyUcXpH6BlVyVaU+1dL/Gj+L48VlRL+tWAb8+fgE87n1LzFZFKvMb7maruBFDVC1n3fwR87UuFk6jjoCN/s3nvUyzrSpX9S6YpcR2cwSHk56FZf+6+Tozb28pMSSjYfEVEgI+BTlV9P+v2psz+YIAngKP+lDhJppGseH4okOWMMcYPUmifj4isBX4C2vlvGOEWYCOwCm+3wxngxaxmfL2fdRG4CgzMpOgiaMiqYYmqhvZZ4RHNBELMJUKZQLQeKyNAV1jrZ4lSJlF5rEz7+VOw+RabiBxQ1fsDXTSCNWSLQj1RqCFbVOqJSh0QnVqiUse/olDPjdRQxqe1GGNMeKz5GmNMCMJovttCWHOyKNSQLQr1RKGGbFGpJyp1QHRqiUod/4pCPdOuIfB9vsYYY2y3gzHGhCKw5isiD4tIl4icFJHXA1ozkhPZsuoLPJPMupZL7pqWSe6alkn+dYuTi6r6fgEqgFPAUqAKOAysDGDdJuC+zNd1wHFgJfA28FoQ//eoZWK5WCaWSTRyCWrLdzVwUlW7VXUc+Bx4zO9FVbVPVQ9mvh4BIjORjZAyAcslH8skl2WSX7FyCar5LgZ6s67/RcC/xEkT2cCbyHZERLaLyE1B1pIReiZgueRjmeSyTPKbSS5lccBt8kQ2YCuwDO/06D68iWxlx3LJZZnkskzym2kuQTXfs0BL1vXmzG2+u95ENlV1VNUFPsJ7CRO00DIByyUfyySXZZJfMXIJqvnuB5aLSKuIVAEbgK/8XvT/JrJl/bPgJrJdK5RMwHLJxzLJZZnkV6xcpvVJFjdKVdMisgnYjXeUcruqHgtg6TXAM0C7iLRlbtsCbBSRVWRNZAuglmuEmAlYLvlYJrksk/yKkoud4WaMMSEoiwNuxhgTNdZ8jTEmBNZ8jTEmBNZ8jTEmBNZ8jTEmBNZ8jTEmBNZ8jTEmBNZ8jTEmBP8A84a51eAe3B8AAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 25 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Display a sample of the User A's dataset. \n",
    "digits = {}\n",
    "for i in range(5):\n",
    "    digits[i] = np.where(y_test==i)[0][:5]\n",
    "#print(digits)\n",
    "\n",
    "fig, ax = plt.subplots(5, 5, sharex='col', sharey='row')\n",
    "for i in range(5):\n",
    "    for j in range(5):\n",
    "        ax[i, j].imshow(x_test[digits[i][j]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting the base model.  \n",
    "In this section, we retrieve the base_model and train it on User A's data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "flatten (Flatten)            (None, 784)               0         \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 10)                7840      \n",
      "_________________________________________________________________\n",
      "dropout (Dropout)            (None, 10)                0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 10)                100       \n",
      "=================================================================\n",
      "Total params: 7,940\n",
      "Trainable params: 7,940\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# Retrieving the base model.\n",
    "new_model = tf.keras.models.load_model('saved_model/my_model')\n",
    "\n",
    "# Check its architecture.\n",
    "new_model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[303,   0,   1,   0,   0,   3,   4,   1,   1,   0],\n",
       "       [  0, 349,   2,   2,   0,   1,   1,   0,   3,   0],\n",
       "       [  1,   3, 299,   5,   1,   3,   6,   2,   6,   0],\n",
       "       [  1,   2,  10, 272,   0,  15,   0,   5,   4,   0],\n",
       "       [  2,   1,   1,   0, 277,   1,   2,   1,   3,   9],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0]], dtype=int64)"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Creating a multiclass confusion matrix before training.\n",
    "predictions = new_model.predict(x_test)\n",
    "confusion = confusion_matrix(y_test, np.argmax(predictions,axis=1))\n",
    "confusion"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "51/51 [==============================] - 0s 1ms/step - loss: 0.3041 - sparse_categorical_accuracy: 0.9357\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "e:\\UDM\\M2_privacyinAI\\code_final\\venv\\lib\\site-packages\\keras\\backend.py:4906: UserWarning: \"`sparse_categorical_crossentropy` received `from_logits=True`, but the `output` argument was produced by a sigmoid or softmax activation and thus does not represent logits. Was this intended?\"\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.30413129925727844, 0.9357454776763916]"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Evaluate the base_model to see the loss and sparse_categorical_accuracy before training.\n",
    "new_model.evaluate(x_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Compiling the model.\n",
    "new_model.compile(\n",
    "    optimizer=tf.keras.optimizers.Adam(0.001),\n",
    "    loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
    "    metrics=[tf.keras.metrics.SparseCategoricalAccuracy()],\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/100\n",
      "83/83 [==============================] - 1s 3ms/step - loss: 0.4545 - sparse_categorical_accuracy: 0.8634 - val_loss: 0.1834 - val_sparse_categorical_accuracy: 0.9607\n",
      "Epoch 2/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.3508 - sparse_categorical_accuracy: 0.9000 - val_loss: 0.1481 - val_sparse_categorical_accuracy: 0.9669\n",
      "Epoch 3/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.3187 - sparse_categorical_accuracy: 0.9092 - val_loss: 0.1311 - val_sparse_categorical_accuracy: 0.9707\n",
      "Epoch 4/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.2781 - sparse_categorical_accuracy: 0.9207 - val_loss: 0.1189 - val_sparse_categorical_accuracy: 0.9707\n",
      "Epoch 5/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.2655 - sparse_categorical_accuracy: 0.9229 - val_loss: 0.1071 - val_sparse_categorical_accuracy: 0.9719\n",
      "Epoch 6/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.2334 - sparse_categorical_accuracy: 0.9275 - val_loss: 0.1021 - val_sparse_categorical_accuracy: 0.9750\n",
      "Epoch 7/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.2340 - sparse_categorical_accuracy: 0.9293 - val_loss: 0.0965 - val_sparse_categorical_accuracy: 0.9750\n",
      "Epoch 8/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.2156 - sparse_categorical_accuracy: 0.9345 - val_loss: 0.0951 - val_sparse_categorical_accuracy: 0.9775\n",
      "Epoch 9/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.2098 - sparse_categorical_accuracy: 0.9358 - val_loss: 0.0924 - val_sparse_categorical_accuracy: 0.9744\n",
      "Epoch 10/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1987 - sparse_categorical_accuracy: 0.9381 - val_loss: 0.0915 - val_sparse_categorical_accuracy: 0.9769\n",
      "Epoch 11/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1963 - sparse_categorical_accuracy: 0.9344 - val_loss: 0.0897 - val_sparse_categorical_accuracy: 0.9782\n",
      "Epoch 12/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1939 - sparse_categorical_accuracy: 0.9335 - val_loss: 0.0886 - val_sparse_categorical_accuracy: 0.9769\n",
      "Epoch 13/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1943 - sparse_categorical_accuracy: 0.9361 - val_loss: 0.0855 - val_sparse_categorical_accuracy: 0.9782\n",
      "Epoch 14/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1745 - sparse_categorical_accuracy: 0.9397 - val_loss: 0.0868 - val_sparse_categorical_accuracy: 0.9769\n",
      "Epoch 15/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1779 - sparse_categorical_accuracy: 0.9389 - val_loss: 0.0860 - val_sparse_categorical_accuracy: 0.9788\n",
      "Epoch 16/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1787 - sparse_categorical_accuracy: 0.9393 - val_loss: 0.0870 - val_sparse_categorical_accuracy: 0.9775\n",
      "Epoch 17/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1719 - sparse_categorical_accuracy: 0.9380 - val_loss: 0.0863 - val_sparse_categorical_accuracy: 0.9788\n",
      "Epoch 18/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1708 - sparse_categorical_accuracy: 0.9401 - val_loss: 0.0844 - val_sparse_categorical_accuracy: 0.9807\n",
      "Epoch 19/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1641 - sparse_categorical_accuracy: 0.9455 - val_loss: 0.0866 - val_sparse_categorical_accuracy: 0.9800\n",
      "Epoch 20/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1659 - sparse_categorical_accuracy: 0.9435 - val_loss: 0.0867 - val_sparse_categorical_accuracy: 0.9807\n",
      "Epoch 21/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1591 - sparse_categorical_accuracy: 0.9445 - val_loss: 0.0826 - val_sparse_categorical_accuracy: 0.9807\n",
      "Epoch 22/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1603 - sparse_categorical_accuracy: 0.9431 - val_loss: 0.0810 - val_sparse_categorical_accuracy: 0.9794\n",
      "Epoch 23/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1600 - sparse_categorical_accuracy: 0.9421 - val_loss: 0.0823 - val_sparse_categorical_accuracy: 0.9813\n",
      "Epoch 24/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1486 - sparse_categorical_accuracy: 0.9446 - val_loss: 0.0826 - val_sparse_categorical_accuracy: 0.9788\n",
      "Epoch 25/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1592 - sparse_categorical_accuracy: 0.9425 - val_loss: 0.0797 - val_sparse_categorical_accuracy: 0.9794\n",
      "Epoch 26/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1527 - sparse_categorical_accuracy: 0.9447 - val_loss: 0.0803 - val_sparse_categorical_accuracy: 0.9819\n",
      "Epoch 27/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1532 - sparse_categorical_accuracy: 0.9438 - val_loss: 0.0820 - val_sparse_categorical_accuracy: 0.9807\n",
      "Epoch 28/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1507 - sparse_categorical_accuracy: 0.9444 - val_loss: 0.0810 - val_sparse_categorical_accuracy: 0.9813\n",
      "Epoch 29/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1475 - sparse_categorical_accuracy: 0.9469 - val_loss: 0.0833 - val_sparse_categorical_accuracy: 0.9807\n",
      "Epoch 30/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1379 - sparse_categorical_accuracy: 0.9489 - val_loss: 0.0812 - val_sparse_categorical_accuracy: 0.9819\n",
      "Epoch 31/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1539 - sparse_categorical_accuracy: 0.9439 - val_loss: 0.0784 - val_sparse_categorical_accuracy: 0.9813\n",
      "Epoch 32/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1437 - sparse_categorical_accuracy: 0.9492 - val_loss: 0.0817 - val_sparse_categorical_accuracy: 0.9807\n",
      "Epoch 33/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1452 - sparse_categorical_accuracy: 0.9461 - val_loss: 0.0784 - val_sparse_categorical_accuracy: 0.9819\n",
      "Epoch 34/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1455 - sparse_categorical_accuracy: 0.9460 - val_loss: 0.0820 - val_sparse_categorical_accuracy: 0.9819\n",
      "Epoch 35/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1399 - sparse_categorical_accuracy: 0.9481 - val_loss: 0.0801 - val_sparse_categorical_accuracy: 0.9819\n",
      "Epoch 36/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1399 - sparse_categorical_accuracy: 0.9468 - val_loss: 0.0839 - val_sparse_categorical_accuracy: 0.9813\n",
      "Epoch 37/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1388 - sparse_categorical_accuracy: 0.9499 - val_loss: 0.0830 - val_sparse_categorical_accuracy: 0.9813\n",
      "Epoch 38/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1342 - sparse_categorical_accuracy: 0.9519 - val_loss: 0.0825 - val_sparse_categorical_accuracy: 0.9813\n",
      "Epoch 39/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1318 - sparse_categorical_accuracy: 0.9527 - val_loss: 0.0784 - val_sparse_categorical_accuracy: 0.9832\n",
      "Epoch 40/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1377 - sparse_categorical_accuracy: 0.9483 - val_loss: 0.0792 - val_sparse_categorical_accuracy: 0.9819\n",
      "Epoch 41/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1367 - sparse_categorical_accuracy: 0.9501 - val_loss: 0.0772 - val_sparse_categorical_accuracy: 0.9819\n",
      "Epoch 42/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1333 - sparse_categorical_accuracy: 0.9536 - val_loss: 0.0783 - val_sparse_categorical_accuracy: 0.9807\n",
      "Epoch 43/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1337 - sparse_categorical_accuracy: 0.9493 - val_loss: 0.0827 - val_sparse_categorical_accuracy: 0.9819\n",
      "Epoch 44/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1361 - sparse_categorical_accuracy: 0.9492 - val_loss: 0.0795 - val_sparse_categorical_accuracy: 0.9832\n",
      "Epoch 45/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1330 - sparse_categorical_accuracy: 0.9481 - val_loss: 0.0828 - val_sparse_categorical_accuracy: 0.9819\n",
      "Epoch 46/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1241 - sparse_categorical_accuracy: 0.9545 - val_loss: 0.0802 - val_sparse_categorical_accuracy: 0.9813\n",
      "Epoch 47/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1279 - sparse_categorical_accuracy: 0.9544 - val_loss: 0.0796 - val_sparse_categorical_accuracy: 0.9807\n",
      "Epoch 48/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1279 - sparse_categorical_accuracy: 0.9538 - val_loss: 0.0809 - val_sparse_categorical_accuracy: 0.9794\n",
      "Epoch 49/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1286 - sparse_categorical_accuracy: 0.9519 - val_loss: 0.0799 - val_sparse_categorical_accuracy: 0.9819\n",
      "Epoch 50/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1261 - sparse_categorical_accuracy: 0.9543 - val_loss: 0.0808 - val_sparse_categorical_accuracy: 0.9813\n",
      "Epoch 51/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1226 - sparse_categorical_accuracy: 0.9552 - val_loss: 0.0790 - val_sparse_categorical_accuracy: 0.9807\n",
      "Epoch 52/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1287 - sparse_categorical_accuracy: 0.9521 - val_loss: 0.0798 - val_sparse_categorical_accuracy: 0.9800\n",
      "Epoch 53/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1258 - sparse_categorical_accuracy: 0.9521 - val_loss: 0.0807 - val_sparse_categorical_accuracy: 0.9807\n",
      "Epoch 54/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1228 - sparse_categorical_accuracy: 0.9540 - val_loss: 0.0805 - val_sparse_categorical_accuracy: 0.9807\n",
      "Epoch 55/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1213 - sparse_categorical_accuracy: 0.9550 - val_loss: 0.0807 - val_sparse_categorical_accuracy: 0.9807\n",
      "Epoch 56/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1236 - sparse_categorical_accuracy: 0.9516 - val_loss: 0.0790 - val_sparse_categorical_accuracy: 0.9800\n",
      "Epoch 57/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1264 - sparse_categorical_accuracy: 0.9532 - val_loss: 0.0766 - val_sparse_categorical_accuracy: 0.9782\n",
      "Epoch 58/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1201 - sparse_categorical_accuracy: 0.9544 - val_loss: 0.0786 - val_sparse_categorical_accuracy: 0.9782\n",
      "Epoch 59/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1185 - sparse_categorical_accuracy: 0.9544 - val_loss: 0.0786 - val_sparse_categorical_accuracy: 0.9807\n",
      "Epoch 60/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1196 - sparse_categorical_accuracy: 0.9543 - val_loss: 0.0755 - val_sparse_categorical_accuracy: 0.9800\n",
      "Epoch 61/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1165 - sparse_categorical_accuracy: 0.9557 - val_loss: 0.0792 - val_sparse_categorical_accuracy: 0.9788\n",
      "Epoch 62/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1167 - sparse_categorical_accuracy: 0.9542 - val_loss: 0.0819 - val_sparse_categorical_accuracy: 0.9794\n",
      "Epoch 63/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1193 - sparse_categorical_accuracy: 0.9548 - val_loss: 0.0770 - val_sparse_categorical_accuracy: 0.9794\n",
      "Epoch 64/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1176 - sparse_categorical_accuracy: 0.9553 - val_loss: 0.0811 - val_sparse_categorical_accuracy: 0.9788\n",
      "Epoch 65/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1164 - sparse_categorical_accuracy: 0.9546 - val_loss: 0.0807 - val_sparse_categorical_accuracy: 0.9807\n",
      "Epoch 66/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1179 - sparse_categorical_accuracy: 0.9551 - val_loss: 0.0815 - val_sparse_categorical_accuracy: 0.9794\n",
      "Epoch 67/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1213 - sparse_categorical_accuracy: 0.9540 - val_loss: 0.0873 - val_sparse_categorical_accuracy: 0.9788\n",
      "Epoch 68/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1099 - sparse_categorical_accuracy: 0.9573 - val_loss: 0.0799 - val_sparse_categorical_accuracy: 0.9819\n",
      "Epoch 69/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1229 - sparse_categorical_accuracy: 0.9519 - val_loss: 0.0853 - val_sparse_categorical_accuracy: 0.9782\n",
      "Epoch 70/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1124 - sparse_categorical_accuracy: 0.9571 - val_loss: 0.0858 - val_sparse_categorical_accuracy: 0.9813\n",
      "Epoch 71/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1162 - sparse_categorical_accuracy: 0.9543 - val_loss: 0.0817 - val_sparse_categorical_accuracy: 0.9807\n",
      "Epoch 72/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1142 - sparse_categorical_accuracy: 0.9563 - val_loss: 0.0800 - val_sparse_categorical_accuracy: 0.9819\n",
      "Epoch 73/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1131 - sparse_categorical_accuracy: 0.9568 - val_loss: 0.0828 - val_sparse_categorical_accuracy: 0.9807\n",
      "Epoch 74/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1166 - sparse_categorical_accuracy: 0.9557 - val_loss: 0.0800 - val_sparse_categorical_accuracy: 0.9813\n",
      "Epoch 75/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1131 - sparse_categorical_accuracy: 0.9540 - val_loss: 0.0856 - val_sparse_categorical_accuracy: 0.9813\n",
      "Epoch 76/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1125 - sparse_categorical_accuracy: 0.9571 - val_loss: 0.0866 - val_sparse_categorical_accuracy: 0.9819\n",
      "Epoch 77/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1120 - sparse_categorical_accuracy: 0.9561 - val_loss: 0.0851 - val_sparse_categorical_accuracy: 0.9794\n",
      "Epoch 78/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1069 - sparse_categorical_accuracy: 0.9555 - val_loss: 0.0846 - val_sparse_categorical_accuracy: 0.9782\n",
      "Epoch 79/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1129 - sparse_categorical_accuracy: 0.9525 - val_loss: 0.0821 - val_sparse_categorical_accuracy: 0.9800\n",
      "Epoch 80/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1089 - sparse_categorical_accuracy: 0.9558 - val_loss: 0.0888 - val_sparse_categorical_accuracy: 0.9782\n",
      "Epoch 81/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1117 - sparse_categorical_accuracy: 0.9573 - val_loss: 0.0851 - val_sparse_categorical_accuracy: 0.9788\n",
      "Epoch 82/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1092 - sparse_categorical_accuracy: 0.9569 - val_loss: 0.0881 - val_sparse_categorical_accuracy: 0.9800\n",
      "Epoch 83/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1066 - sparse_categorical_accuracy: 0.9571 - val_loss: 0.0893 - val_sparse_categorical_accuracy: 0.9794\n",
      "Epoch 84/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1137 - sparse_categorical_accuracy: 0.9552 - val_loss: 0.0859 - val_sparse_categorical_accuracy: 0.9800\n",
      "Epoch 85/100\n",
      "83/83 [==============================] - 0s 3ms/step - loss: 0.1121 - sparse_categorical_accuracy: 0.9555 - val_loss: 0.0894 - val_sparse_categorical_accuracy: 0.9788\n",
      "Epoch 86/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1092 - sparse_categorical_accuracy: 0.9568 - val_loss: 0.0937 - val_sparse_categorical_accuracy: 0.9807\n",
      "Epoch 87/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1105 - sparse_categorical_accuracy: 0.9567 - val_loss: 0.0897 - val_sparse_categorical_accuracy: 0.9794\n",
      "Epoch 88/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1066 - sparse_categorical_accuracy: 0.9585 - val_loss: 0.0860 - val_sparse_categorical_accuracy: 0.9800\n",
      "Epoch 89/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1162 - sparse_categorical_accuracy: 0.9517 - val_loss: 0.0872 - val_sparse_categorical_accuracy: 0.9807\n",
      "Epoch 90/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1096 - sparse_categorical_accuracy: 0.9538 - val_loss: 0.0894 - val_sparse_categorical_accuracy: 0.9807\n",
      "Epoch 91/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1060 - sparse_categorical_accuracy: 0.9588 - val_loss: 0.0897 - val_sparse_categorical_accuracy: 0.9794\n",
      "Epoch 92/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1067 - sparse_categorical_accuracy: 0.9559 - val_loss: 0.0916 - val_sparse_categorical_accuracy: 0.9782\n",
      "Epoch 93/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1142 - sparse_categorical_accuracy: 0.9552 - val_loss: 0.0909 - val_sparse_categorical_accuracy: 0.9800\n",
      "Epoch 94/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1080 - sparse_categorical_accuracy: 0.9577 - val_loss: 0.0923 - val_sparse_categorical_accuracy: 0.9788\n",
      "Epoch 95/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1117 - sparse_categorical_accuracy: 0.9547 - val_loss: 0.0885 - val_sparse_categorical_accuracy: 0.9794\n",
      "Epoch 96/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1080 - sparse_categorical_accuracy: 0.9568 - val_loss: 0.0905 - val_sparse_categorical_accuracy: 0.9788\n",
      "Epoch 97/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1067 - sparse_categorical_accuracy: 0.9577 - val_loss: 0.0950 - val_sparse_categorical_accuracy: 0.9769\n",
      "Epoch 98/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1020 - sparse_categorical_accuracy: 0.9571 - val_loss: 0.0955 - val_sparse_categorical_accuracy: 0.9763\n",
      "Epoch 99/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1042 - sparse_categorical_accuracy: 0.9598 - val_loss: 0.0920 - val_sparse_categorical_accuracy: 0.9807\n",
      "Epoch 100/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1031 - sparse_categorical_accuracy: 0.9598 - val_loss: 0.0926 - val_sparse_categorical_accuracy: 0.9788\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x1f29bb255b0>"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Retraining the base_model to obtain the new weights.\n",
    "new_model.fit(x_train,\n",
    "          y_train,\n",
    "          batch_size=128,\n",
    "          epochs=100,\n",
    "          verbose=1,\n",
    "          validation_data=(x_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "51/51 [==============================] - 0s 1ms/step - loss: 0.0926 - sparse_categorical_accuracy: 0.9788\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.09257857501506805, 0.978789746761322]"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Evaluate the base_model to see the loss and sparse_categorical_accuracy after training.\n",
    "new_model.evaluate(x_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extracting weights & bias."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "flatten 0\n",
      "dense 1\n",
      "dropout 0\n",
      "dense_1 1\n"
     ]
    }
   ],
   "source": [
    "# Checking which layers have weights & bias.\n",
    "for layer in new_model.layers:\n",
    "    print(layer.name, len(layer.weights))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extracting User A's weights from the first Dense layer.\n",
    "bob_weights = new_model.layers[1].get_weights()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Masking the weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "start_time = time.time()\n",
    "\n",
    "K = 500\n",
    "res_A = [(x + K) for x in bob_weights]\n",
    "#print(\" First tensor: Data = {}\".format(res_A))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using CKKS to encrypt the weights.\n",
    "In this section, we use Tenseal's CKKS class to encrypt the weights. Note that protobuf has a hardlimit of 2GB, hence, we split the weights into 2 arrays."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Splitting the weights array into 2 arrays called 'arr0' and 'arr1'\n",
    "arr = np.array(res_A)\n",
    "arr0, arr1 = np.dsplit(arr, 2)\n",
    "#(numpy.dstack((arr0, arr1))) == res_A  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # Note that we use the generated key from key_query file.\n",
    "# # Setup TenSEAL context.\n",
    "\n",
    "# context = ts.context(\n",
    "#             ts.SCHEME_TYPE.CKKS,\n",
    "#             poly_modulus_degree=8192,\n",
    "#             coeff_mod_bit_sizes=[60, 40, 40, 60]\n",
    "#           )\n",
    "          \n",
    "# context.generate_galois_keys()\n",
    "# context.global_scale = 2**40\n",
    "\n",
    "# load : get the data from file\n",
    "key_query_file = open('key_query', \"rb\")\n",
    "context_key = pickle.load(key_query_file)\n",
    "key_query_file.close()\n",
    "\n",
    "# Recreate the key.\n",
    "context = ts.context_from(context_key[\"context\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Encrypt the weights\n",
    "enc_v1 = ts.ckks_tensor(context, arr0)\n",
    "enc_v1_serialised = enc_v1.serialize()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Encrypt the weights\n",
    "enc_v2 = ts.ckks_tensor(context, arr1)\n",
    "enc_v2_serialised = enc_v2.serialize()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # Creating a copy of the context to share with the server. Ideally, the secret key should be drop.\n",
    "# # Since we use the context from key_query file, no need to reserialise the key.\n",
    "\n",
    "#public_context = context\n",
    "#sk = context.secret_key()\n",
    "#public_context.make_context_public()\n",
    "\n",
    "# Context and ciphertext serialization\n",
    "#server_context = public_context.serialize(save_secret_key=True)\n",
    "\n",
    "# # Check is context is private.\n",
    "#print(\"Is the context private?\", (\"Yes\" if context.is_private() else \"No\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Storing CKKS encrypted weights and context in a dict."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Storing the weights and context in a dict so it can be sent as a JSON file.\n",
    "bob_query = {\n",
    "    \"data1\" : enc_v1_serialised,\n",
    "    \"data2\" : enc_v2_serialised,\n",
    "    #\"context\" : server_context\n",
    "}\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Storing the encrypyted weights and context in a pickle file. \n",
    "# Note that this is done only to obtain the size of the encrypted weights on disk, and that pickle is not secure.\n",
    "input_dictionary = (bob_query)\n",
    "file = open('bob_enc', 'wb')\n",
    "pickle.dump(input_dictionary, file)\n",
    "file.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gentry FHE\n",
    "In this section, we use our own implementation of Gentry's FHE schemne to encrypt the weights. \n",
    "Note that this is a basic implementation. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [],
   "source": [
    "r = 3\n",
    "p = 29\n",
    "q = 22\n",
    "#encrypt mask weights A\n",
    "encrypt_A = [(x + (2*r) + (q*p)) for x in res_A]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Storing CKKS encrypted weights and context in a dict. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We only stored the encrpyted weights in dict. Keys are stored separately.\n",
    "bob_query_gentry = {\n",
    "    \"data1\" : encrypt_A,\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Storing the encrypyted weights and context in a pickle file. \n",
    "# Note that this is done only to obtain the size of the encrypted weights on disk, and that pickle is not secure.\n",
    "import pickle\n",
    "input_dictionary = (bob_query_gentry)\n",
    "file = open('bob_enc_gentry', 'wb')\n",
    "pickle.dump(input_dictionary, file)\n",
    "file.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading updated model.\n",
    "In this section, we uload the updated model which have the averaged weights from the server."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "flatten (Flatten)            (None, 784)               0         \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 10)                7840      \n",
      "_________________________________________________________________\n",
      "dropout (Dropout)            (None, 10)                0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 10)                100       \n",
      "=================================================================\n",
      "Total params: 7,940\n",
      "Trainable params: 7,940\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "47/51 [==========================>...] - ETA: 0s - loss: 0.3049 - sparse_categorical_accuracy: 0.9362"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "e:\\UDM\\M2_privacyinAI\\code_final\\venv\\lib\\site-packages\\keras\\backend.py:4906: UserWarning: \"`sparse_categorical_crossentropy` received `from_logits=True`, but the `output` argument was produced by a sigmoid or softmax activation and thus does not represent logits. Was this intended?\"\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "51/51 [==============================] - 0s 2ms/step - loss: 0.2896 - sparse_categorical_accuracy: 0.9389\n",
      "Epoch 1/100\n",
      "83/83 [==============================] - 1s 4ms/step - loss: 0.4424 - sparse_categorical_accuracy: 0.8661 - val_loss: 0.1632 - val_sparse_categorical_accuracy: 0.9651\n",
      "Epoch 2/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.3490 - sparse_categorical_accuracy: 0.8975 - val_loss: 0.1350 - val_sparse_categorical_accuracy: 0.9713\n",
      "Epoch 3/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.3034 - sparse_categorical_accuracy: 0.9127 - val_loss: 0.1177 - val_sparse_categorical_accuracy: 0.9738\n",
      "Epoch 4/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.2771 - sparse_categorical_accuracy: 0.9190 - val_loss: 0.1110 - val_sparse_categorical_accuracy: 0.9738\n",
      "Epoch 5/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.2581 - sparse_categorical_accuracy: 0.9235 - val_loss: 0.1049 - val_sparse_categorical_accuracy: 0.9757\n",
      "Epoch 6/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.2461 - sparse_categorical_accuracy: 0.9256 - val_loss: 0.0980 - val_sparse_categorical_accuracy: 0.9769\n",
      "Epoch 7/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.2411 - sparse_categorical_accuracy: 0.9252 - val_loss: 0.0951 - val_sparse_categorical_accuracy: 0.9763\n",
      "Epoch 8/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.2239 - sparse_categorical_accuracy: 0.9288 - val_loss: 0.0893 - val_sparse_categorical_accuracy: 0.9782\n",
      "Epoch 9/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.2192 - sparse_categorical_accuracy: 0.9287 - val_loss: 0.0894 - val_sparse_categorical_accuracy: 0.9769\n",
      "Epoch 10/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.2144 - sparse_categorical_accuracy: 0.9305 - val_loss: 0.0874 - val_sparse_categorical_accuracy: 0.9782\n",
      "Epoch 11/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.2074 - sparse_categorical_accuracy: 0.9324 - val_loss: 0.0856 - val_sparse_categorical_accuracy: 0.9794\n",
      "Epoch 12/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.2022 - sparse_categorical_accuracy: 0.9326 - val_loss: 0.0895 - val_sparse_categorical_accuracy: 0.9769\n",
      "Epoch 13/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1942 - sparse_categorical_accuracy: 0.9347 - val_loss: 0.0862 - val_sparse_categorical_accuracy: 0.9775\n",
      "Epoch 14/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1806 - sparse_categorical_accuracy: 0.9371 - val_loss: 0.0840 - val_sparse_categorical_accuracy: 0.9788\n",
      "Epoch 15/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1788 - sparse_categorical_accuracy: 0.9379 - val_loss: 0.0861 - val_sparse_categorical_accuracy: 0.9782\n",
      "Epoch 16/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1742 - sparse_categorical_accuracy: 0.9402 - val_loss: 0.0804 - val_sparse_categorical_accuracy: 0.9794\n",
      "Epoch 17/100\n",
      "83/83 [==============================] - 0s 3ms/step - loss: 0.1770 - sparse_categorical_accuracy: 0.9386 - val_loss: 0.0857 - val_sparse_categorical_accuracy: 0.9782\n",
      "Epoch 18/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1628 - sparse_categorical_accuracy: 0.9418 - val_loss: 0.0833 - val_sparse_categorical_accuracy: 0.9794\n",
      "Epoch 19/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1674 - sparse_categorical_accuracy: 0.9425 - val_loss: 0.0846 - val_sparse_categorical_accuracy: 0.9788\n",
      "Epoch 20/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1685 - sparse_categorical_accuracy: 0.9399 - val_loss: 0.0807 - val_sparse_categorical_accuracy: 0.9800\n",
      "Epoch 21/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1642 - sparse_categorical_accuracy: 0.9442 - val_loss: 0.0851 - val_sparse_categorical_accuracy: 0.9794\n",
      "Epoch 22/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1611 - sparse_categorical_accuracy: 0.9413 - val_loss: 0.0806 - val_sparse_categorical_accuracy: 0.9800\n",
      "Epoch 23/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1601 - sparse_categorical_accuracy: 0.9420 - val_loss: 0.0819 - val_sparse_categorical_accuracy: 0.9807\n",
      "Epoch 24/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1562 - sparse_categorical_accuracy: 0.9429 - val_loss: 0.0819 - val_sparse_categorical_accuracy: 0.9800\n",
      "Epoch 25/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1505 - sparse_categorical_accuracy: 0.9436 - val_loss: 0.0830 - val_sparse_categorical_accuracy: 0.9794\n",
      "Epoch 26/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1550 - sparse_categorical_accuracy: 0.9447 - val_loss: 0.0827 - val_sparse_categorical_accuracy: 0.9794\n",
      "Epoch 27/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1509 - sparse_categorical_accuracy: 0.9450 - val_loss: 0.0843 - val_sparse_categorical_accuracy: 0.9794\n",
      "Epoch 28/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1482 - sparse_categorical_accuracy: 0.9459 - val_loss: 0.0826 - val_sparse_categorical_accuracy: 0.9788\n",
      "Epoch 29/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1485 - sparse_categorical_accuracy: 0.9476 - val_loss: 0.0855 - val_sparse_categorical_accuracy: 0.9807\n",
      "Epoch 30/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1496 - sparse_categorical_accuracy: 0.9419 - val_loss: 0.0820 - val_sparse_categorical_accuracy: 0.9800\n",
      "Epoch 31/100\n",
      "83/83 [==============================] - 0s 3ms/step - loss: 0.1410 - sparse_categorical_accuracy: 0.9478 - val_loss: 0.0869 - val_sparse_categorical_accuracy: 0.9800\n",
      "Epoch 32/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1344 - sparse_categorical_accuracy: 0.9506 - val_loss: 0.0846 - val_sparse_categorical_accuracy: 0.9807\n",
      "Epoch 33/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1448 - sparse_categorical_accuracy: 0.9443 - val_loss: 0.0886 - val_sparse_categorical_accuracy: 0.9800\n",
      "Epoch 34/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1456 - sparse_categorical_accuracy: 0.9434 - val_loss: 0.0851 - val_sparse_categorical_accuracy: 0.9794\n",
      "Epoch 35/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1341 - sparse_categorical_accuracy: 0.9487 - val_loss: 0.0836 - val_sparse_categorical_accuracy: 0.9819\n",
      "Epoch 36/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1376 - sparse_categorical_accuracy: 0.9503 - val_loss: 0.0855 - val_sparse_categorical_accuracy: 0.9800\n",
      "Epoch 37/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1394 - sparse_categorical_accuracy: 0.9454 - val_loss: 0.0841 - val_sparse_categorical_accuracy: 0.9825\n",
      "Epoch 38/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1313 - sparse_categorical_accuracy: 0.9481 - val_loss: 0.0847 - val_sparse_categorical_accuracy: 0.9825\n",
      "Epoch 39/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1345 - sparse_categorical_accuracy: 0.9513 - val_loss: 0.0850 - val_sparse_categorical_accuracy: 0.9794\n",
      "Epoch 40/100\n",
      "83/83 [==============================] - 0s 3ms/step - loss: 0.1319 - sparse_categorical_accuracy: 0.9509 - val_loss: 0.0848 - val_sparse_categorical_accuracy: 0.9819\n",
      "Epoch 41/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1313 - sparse_categorical_accuracy: 0.9495 - val_loss: 0.0848 - val_sparse_categorical_accuracy: 0.9825\n",
      "Epoch 42/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1357 - sparse_categorical_accuracy: 0.9460 - val_loss: 0.0827 - val_sparse_categorical_accuracy: 0.9807\n",
      "Epoch 43/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1348 - sparse_categorical_accuracy: 0.9495 - val_loss: 0.0858 - val_sparse_categorical_accuracy: 0.9825\n",
      "Epoch 44/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1284 - sparse_categorical_accuracy: 0.9506 - val_loss: 0.0826 - val_sparse_categorical_accuracy: 0.9838\n",
      "Epoch 45/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1269 - sparse_categorical_accuracy: 0.9503 - val_loss: 0.0852 - val_sparse_categorical_accuracy: 0.9832\n",
      "Epoch 46/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1363 - sparse_categorical_accuracy: 0.9465 - val_loss: 0.0882 - val_sparse_categorical_accuracy: 0.9794\n",
      "Epoch 47/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1259 - sparse_categorical_accuracy: 0.9493 - val_loss: 0.0883 - val_sparse_categorical_accuracy: 0.9800\n",
      "Epoch 48/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1199 - sparse_categorical_accuracy: 0.9547 - val_loss: 0.0907 - val_sparse_categorical_accuracy: 0.9788\n",
      "Epoch 49/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1262 - sparse_categorical_accuracy: 0.9520 - val_loss: 0.0844 - val_sparse_categorical_accuracy: 0.9782\n",
      "Epoch 50/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1284 - sparse_categorical_accuracy: 0.9526 - val_loss: 0.0859 - val_sparse_categorical_accuracy: 0.9788\n",
      "Epoch 51/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1216 - sparse_categorical_accuracy: 0.9538 - val_loss: 0.0865 - val_sparse_categorical_accuracy: 0.9794\n",
      "Epoch 52/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1189 - sparse_categorical_accuracy: 0.9535 - val_loss: 0.0860 - val_sparse_categorical_accuracy: 0.9813\n",
      "Epoch 53/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1167 - sparse_categorical_accuracy: 0.9527 - val_loss: 0.0886 - val_sparse_categorical_accuracy: 0.9788\n",
      "Epoch 54/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1232 - sparse_categorical_accuracy: 0.9527 - val_loss: 0.0832 - val_sparse_categorical_accuracy: 0.9807\n",
      "Epoch 55/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1197 - sparse_categorical_accuracy: 0.9535 - val_loss: 0.0903 - val_sparse_categorical_accuracy: 0.9788\n",
      "Epoch 56/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1198 - sparse_categorical_accuracy: 0.9537 - val_loss: 0.0889 - val_sparse_categorical_accuracy: 0.9813\n",
      "Epoch 57/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1188 - sparse_categorical_accuracy: 0.9544 - val_loss: 0.0939 - val_sparse_categorical_accuracy: 0.9782\n",
      "Epoch 58/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1200 - sparse_categorical_accuracy: 0.9540 - val_loss: 0.0900 - val_sparse_categorical_accuracy: 0.9788\n",
      "Epoch 59/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1162 - sparse_categorical_accuracy: 0.9551 - val_loss: 0.0924 - val_sparse_categorical_accuracy: 0.9794\n",
      "Epoch 60/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1199 - sparse_categorical_accuracy: 0.9517 - val_loss: 0.0934 - val_sparse_categorical_accuracy: 0.9788\n",
      "Epoch 61/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1202 - sparse_categorical_accuracy: 0.9534 - val_loss: 0.0939 - val_sparse_categorical_accuracy: 0.9788\n",
      "Epoch 62/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1180 - sparse_categorical_accuracy: 0.9544 - val_loss: 0.0885 - val_sparse_categorical_accuracy: 0.9800\n",
      "Epoch 63/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1132 - sparse_categorical_accuracy: 0.9558 - val_loss: 0.0959 - val_sparse_categorical_accuracy: 0.9794\n",
      "Epoch 64/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1126 - sparse_categorical_accuracy: 0.9533 - val_loss: 0.0932 - val_sparse_categorical_accuracy: 0.9794\n",
      "Epoch 65/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1201 - sparse_categorical_accuracy: 0.9530 - val_loss: 0.0900 - val_sparse_categorical_accuracy: 0.9794\n",
      "Epoch 66/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1198 - sparse_categorical_accuracy: 0.9508 - val_loss: 0.0925 - val_sparse_categorical_accuracy: 0.9813\n",
      "Epoch 67/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1140 - sparse_categorical_accuracy: 0.9554 - val_loss: 0.0935 - val_sparse_categorical_accuracy: 0.9807\n",
      "Epoch 68/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1120 - sparse_categorical_accuracy: 0.9555 - val_loss: 0.0968 - val_sparse_categorical_accuracy: 0.9800\n",
      "Epoch 69/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1091 - sparse_categorical_accuracy: 0.9570 - val_loss: 0.0915 - val_sparse_categorical_accuracy: 0.9794\n",
      "Epoch 70/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1121 - sparse_categorical_accuracy: 0.9567 - val_loss: 0.0943 - val_sparse_categorical_accuracy: 0.9788\n",
      "Epoch 71/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1164 - sparse_categorical_accuracy: 0.9519 - val_loss: 0.0944 - val_sparse_categorical_accuracy: 0.9794\n",
      "Epoch 72/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1145 - sparse_categorical_accuracy: 0.9534 - val_loss: 0.0915 - val_sparse_categorical_accuracy: 0.9775\n",
      "Epoch 73/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1127 - sparse_categorical_accuracy: 0.9556 - val_loss: 0.0964 - val_sparse_categorical_accuracy: 0.9825\n",
      "Epoch 74/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1128 - sparse_categorical_accuracy: 0.9559 - val_loss: 0.0948 - val_sparse_categorical_accuracy: 0.9807\n",
      "Epoch 75/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1117 - sparse_categorical_accuracy: 0.9545 - val_loss: 0.0914 - val_sparse_categorical_accuracy: 0.9794\n",
      "Epoch 76/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1172 - sparse_categorical_accuracy: 0.9540 - val_loss: 0.0927 - val_sparse_categorical_accuracy: 0.9813\n",
      "Epoch 77/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1136 - sparse_categorical_accuracy: 0.9557 - val_loss: 0.0906 - val_sparse_categorical_accuracy: 0.9788\n",
      "Epoch 78/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1152 - sparse_categorical_accuracy: 0.9517 - val_loss: 0.0953 - val_sparse_categorical_accuracy: 0.9825\n",
      "Epoch 79/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1076 - sparse_categorical_accuracy: 0.9545 - val_loss: 0.1016 - val_sparse_categorical_accuracy: 0.9775\n",
      "Epoch 80/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1143 - sparse_categorical_accuracy: 0.9532 - val_loss: 0.1001 - val_sparse_categorical_accuracy: 0.9800\n",
      "Epoch 81/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1078 - sparse_categorical_accuracy: 0.9551 - val_loss: 0.0962 - val_sparse_categorical_accuracy: 0.9807\n",
      "Epoch 82/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1109 - sparse_categorical_accuracy: 0.9554 - val_loss: 0.0971 - val_sparse_categorical_accuracy: 0.9800\n",
      "Epoch 83/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1038 - sparse_categorical_accuracy: 0.9568 - val_loss: 0.1007 - val_sparse_categorical_accuracy: 0.9807\n",
      "Epoch 84/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1082 - sparse_categorical_accuracy: 0.9558 - val_loss: 0.0966 - val_sparse_categorical_accuracy: 0.9800\n",
      "Epoch 85/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1070 - sparse_categorical_accuracy: 0.9569 - val_loss: 0.1029 - val_sparse_categorical_accuracy: 0.9782\n",
      "Epoch 86/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1066 - sparse_categorical_accuracy: 0.9564 - val_loss: 0.1000 - val_sparse_categorical_accuracy: 0.9800\n",
      "Epoch 87/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1038 - sparse_categorical_accuracy: 0.9597 - val_loss: 0.0969 - val_sparse_categorical_accuracy: 0.9807\n",
      "Epoch 88/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1027 - sparse_categorical_accuracy: 0.9588 - val_loss: 0.1033 - val_sparse_categorical_accuracy: 0.9794\n",
      "Epoch 89/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1033 - sparse_categorical_accuracy: 0.9599 - val_loss: 0.0962 - val_sparse_categorical_accuracy: 0.9782\n",
      "Epoch 90/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1055 - sparse_categorical_accuracy: 0.9578 - val_loss: 0.1050 - val_sparse_categorical_accuracy: 0.9794\n",
      "Epoch 91/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1050 - sparse_categorical_accuracy: 0.9565 - val_loss: 0.1066 - val_sparse_categorical_accuracy: 0.9807\n",
      "Epoch 92/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1012 - sparse_categorical_accuracy: 0.9596 - val_loss: 0.0948 - val_sparse_categorical_accuracy: 0.9794\n",
      "Epoch 93/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1040 - sparse_categorical_accuracy: 0.9588 - val_loss: 0.0934 - val_sparse_categorical_accuracy: 0.9800\n",
      "Epoch 94/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1042 - sparse_categorical_accuracy: 0.9582 - val_loss: 0.0973 - val_sparse_categorical_accuracy: 0.9788\n",
      "Epoch 95/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1044 - sparse_categorical_accuracy: 0.9597 - val_loss: 0.0988 - val_sparse_categorical_accuracy: 0.9775\n",
      "Epoch 96/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1038 - sparse_categorical_accuracy: 0.9593 - val_loss: 0.1032 - val_sparse_categorical_accuracy: 0.9794\n",
      "Epoch 97/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1018 - sparse_categorical_accuracy: 0.9584 - val_loss: 0.0979 - val_sparse_categorical_accuracy: 0.9788\n",
      "Epoch 98/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1026 - sparse_categorical_accuracy: 0.9595 - val_loss: 0.0977 - val_sparse_categorical_accuracy: 0.9782\n",
      "Epoch 99/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.1015 - sparse_categorical_accuracy: 0.9592 - val_loss: 0.0980 - val_sparse_categorical_accuracy: 0.9788\n",
      "Epoch 100/100\n",
      "83/83 [==============================] - 0s 2ms/step - loss: 0.0957 - sparse_categorical_accuracy: 0.9617 - val_loss: 0.0991 - val_sparse_categorical_accuracy: 0.9794\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x1f29bd78640>"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "updated_model = tf.keras.models.load_model('saved_model/updated_model')\n",
    "\n",
    "# Check its architecture\n",
    "updated_model.summary()\n",
    "# Creating a multiclass confusion matrix before training.\n",
    "predictions = updated_model.predict(x_test)\n",
    "confusion = confusion_matrix(y_test, np.argmax(predictions,axis=1))\n",
    "confusion\n",
    "# Evaluate the updated_model to see the loss and sparse_categorical_accuracy before training.\n",
    "updated_model.evaluate(x_test, y_test)\n",
    "# Compiling the model.\n",
    "updated_model.compile(\n",
    "    optimizer=tf.keras.optimizers.Adam(0.001),\n",
    "    loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
    "    metrics=[tf.keras.metrics.SparseCategoricalAccuracy()],\n",
    ")\n",
    "# Retraining the updated_model to obtain the new weights.\n",
    "updated_model.fit(x_train,\n",
    "          y_train,\n",
    "          batch_size=128,\n",
    "          epochs=100,\n",
    "          verbose=1,\n",
    "          validation_data=(x_test, y_test))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "51/51 [==============================] - 0s 1ms/step - loss: 0.0991 - sparse_categorical_accuracy: 0.9794\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.09908999502658844, 0.979413628578186]"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Evaluate the updated_model to see the loss and sparse_categorical_accuracy after training.\n",
    "updated_model.evaluate(x_test, y_test)"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "4def2c97ab8a4ac20a870d28dfe7453688818c63ec73e61b3deb2eaed7804e7f"
  },
  "kernelspec": {
   "display_name": "Python 3.9.7 64-bit ('venv': venv)",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}