Gradient-Descent.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 407,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from mpl_toolkits.mplot3d import Axes3D\n",
    "from sklearn.metrics import accuracy_score\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Intro to Stochastic Gradient Descent\n",
    "\n",
    "### Outline\n",
    "1. [Overview](#Overview)\n",
    "2. [Optimization](#Optimization)\n",
    "3. [Gradient Descent](#Gradient-Descent)\n",
    "4. [Use in Machine Learning](#Use-In-Machine-Learning)\n",
    "5. [Example (Logistic Regression)](#Example-(Logistic-Regression))\n",
    "\n",
    "\n",
    "\n",
    "## Overview\n",
    "\n",
    "Stochastic gradient descent is an iterative form of gradient descent which is used to optimize an objective function which is the sum of differentiable functions.\n",
    "\n",
    "## Optimization\n",
    "\n",
    "In optimization problems we typically seek to minimize or maximize a function usually referred to as an <strong>objective function</strong>  or <strong>loss function</strong>.  Since we can reframe a maximization problem as a minimization problem (i.e. minimize $-f(x)$ rather than maximizing $f(x)$), we will assume that our goal will be to minimize the objective function.\n",
    "\n",
    "### Simple Optimization Example\n",
    "\n",
    "Suppose we want to minimize the function $f(x) = x^2 + 4x$.  We can see by looking at the plot below that the minimum value for y is achieved for an x value around -2.5."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl4VPXZ//H3nYUESFgT9iXsCgqIyCLgvtUNqtWKVrEu\nqFVb+9ifj9ZWra1Va1tr3SrggiuuVeqGewEVlE12CDtBlrDvEJL798ccfGKchAQycybJ53Vdc83M\nWWY+cyY593zP8j3m7oiIiJSUFHYAERFJTCoQIiISlQqEiIhEpQIhIiJRqUCIiEhUKhAiIhKVCoRg\nZm3MbLuZJYedJZ7M7HIzmxin9+piZjPMbJuZ/TIe7ylyqFQgahAzW2Zmu4JisP/Wwt1XuHuGuxcm\nQMZaZvZakNXN7IRDfL27zKwg+KybzewLM+t/EK/zmZlddQhRbgE+dfdMd//nIbxOTJlZfTP7xsx2\nRFtOZvZXM8sNCt18M7ssxnmeCv4OOsbyfSQ6FYia55ygGOy/fRvLNzOzlIOYbSLwM2BNJcV42d0z\ngOzgtd8wM6uk1y6vtsCcOL9nhZhZOjAWmAn8AnjTzLqWmGwHcA5QHxgGPGRmx5bz9ZeZWU4F8gwE\nOpR3eql8KhCCmeUEv9JSguftzGx88CvxIzN71MyeD8adYGZ5JeZfZmanBI/vCloAz5vZVuByM0sy\ns1vNbLGZbTCzV8ysUbQs7r7X3f/h7hOBSm3RuHsBMBpoBjSOshyONbOvzWxLcH9sMPweYBDwSNAS\neSTa65vZuWY2J2ipfGZmhwfDPwFOLDZ/5xLzXWBmU0sM+x8ze6sSPvb+1+tgZhvNrFfwvIWZ5e9v\noQWbF18CFgDD3H00cD3wHzNrvf913P1Od5/v7kXuPhmYAFS4RVaOvCnAw8CNJYY3MrM8MzsneJ5h\nZoti3ZKpqVQgJJoXga+IrETvAi6t4PyDgdeABsALRP7JhwDHAy2ATcCjlZS13MwsDbgcWOnu60uM\nawS8A/yTyOf+O/COmTV299uJrAhvCFpdN0R57c5EVrA3EWmpvEtk5VrL3U8qMf/CErOPBdrtLyiB\nS4FnS/kcjwVFKNptZrR53H0x8L/A82ZWB3gaGO3unwWTdAc+BK5x96JgnteA4UDUFoKZ1QaOITYt\no18D4939e5/H3TcCVwAjzawJ8CAww92jLis5RO6uWw25AcuA7cDm4PZmMDwHcCAFaAPsA+oUm+95\n4Png8QlAXpTXPSV4fBeRf+zi4+cBJxd73hwoAFIOkDcPOOEQP/NdwN7g864DPgGODsZdDkwMHl8K\nfFVi3i+By4PHnwFXlfE+vwdeKfY8CVi1P3855n8cuCd43I1IEU2Lwd/AWGAWkc1Ih/T6RFpj7wNW\ngb+/nHJM1xpYBNQPnjvQscQ0DwefYxXQuLKXk26Rm1oQNc8Qd28Q3IZEGd8C2OjuO4sNW1nB9yg5\nfVvg3/t/5RIpGIVA0wq+7veY2aBiO9vL+hX7SvB5m7j7Se4+Nco0LYDlJYYtB1qWM8735vfIr/CV\nFZh/NHBxsG/k0iDznnLOWxEjgSOAhw/l9c3sgeB1LvRgjR1lmjbFWzdEfnzMLDbs4lJe/h/A3e6+\npYwII4L3f8bdNxzs55CyqUBISauBRsFmiP1aF3u8A/huXLDtOrvEa5RcYawEflSsMDVw93R3X3Uo\nQd19gv/fzvZuh/JawLdECllxbYj8QoUffqYy5w9W9K2LzV8md59EpKUzCLgYeK60ac3sX/b9I9G2\nl6dQmlkGkZXvk8Bdpe0HOhAz+wPwI+A0d99axmdaUfw7B1YA3YsNe7GUWU8GHjCzNWa2/0CFL/cX\nlOBvbgSRTXC/0BFOsaMCId/j7suBKURWILUscqjjOcUmWQikm9lZZpYK/A5IO8DL/gu4x8zaAphZ\ntpkNLm1iM0sLjqgBqGVm6cEKN5beBTqb2cVmlmJmPwW6Am8H49cC7cuY/xXgLDM7OVguNwN7gC8q\nkOFZ4BGgwCM76aNy92v9+0eiZZSzUD4ETHH3q4jsb/lXBbIBYGa3ESlgp8Twl3tnoAfQM7hB5G/w\n38Hj3xIp2FcADwDPWg07hydeVCAkmkuIHJmyAfgT8DKRlR1Bs/8XwCgiv453ENlXUJaHiGz7/sDM\ntgGTgL5lTL8A2EVk88y44HHJX/eVKljZnU1kxb6ByHkLZ/v/7cx+CPiJmW0ysx+cx+DuC4gcmvsw\nsJ7ICu0cd99bgRjPEdls8vxBf5BSBAX5DOC6YND/AL3M7JIKvtSfibSsFhVrtfy2EqPi7uvcfc3+\nWzB4vbvvMrOjg+yXeeS8nfuJFItbKzODRFgpmw9FvmNmLwPz3f3OsLNUZ8FRQeuAXu6eG3YeEbUg\n5AfM7JjguPkkMzuDyGGrb4adqwa4DvhaxUESxcGc5SrVXzPgDSLnA+QB17n79HAjVW9mtgwwIueL\niCQEbWISEZGotIlJRESiqtKbmLKysjwnJyfsGCIiVcrUqVPXu3vJ85d+oEoXiJycHKZMmRJ2DBGR\nKsXMSvYaEJU2MYmISFQqECIiEpUKhIiIRKUCISIiUalAiIhIVDErEBa52Pg6M5sdZdzNFrnEZVax\nYbcFlw5cYGanxyqXiIiUTyxbEM8Q6T3ye4Lr255GpG/4/cO6AhcRuZLWGcBj6r5XRCRcMSsQ7j4e\n2Bhl1INEulIu3sfHYGCMu+9x96VELjfYJ1bZRESqsqc/X8on89fG/H3iug8i6JN+lbt/U2JUS75/\nmco8SrlUo5kNN7MpZjYlPz8/RklFRBJT/rY93PfefN6fvebAEx+iuBWI4BKWvwXuOJTXcfcR7t7b\n3XtnZx/wTHERkWrlyYlLKSgs4roTYn+l1Xh2tdEBaAd8E1w9shUwzcz6ELkyWfHrHreinNfyFRGp\nKTbv3MtzXy7j7O4taJdVN+bvF7cWhLvPcvcm7p7j7jlENiP1Ci4pOBa4KLgWcTugE/BVvLKJiFQF\nz3yxjB17C7n+xNi3HiC2h7m+BHwJdDGzPDO7srRp3X0OkYu+zwXeB64PrjcrIiLA9j37ePrzZZza\ntSldmmXG5T1jtonJ3YceYHxOief3APfEKo+ISFX2/KTlbNlVwA1xaj2AzqQWEUl4uwsKGTVhCYM6\nZdGjdYO4va8KhIhIghvz1QrWb98b19YDqECIiCS0vfuKeGL8EvrkNKJv+8ZxfW8VCBGRBPbGtDxW\nb9nN9SfFt/UAKhAiIglrX2ERj/93Md1b1ee4TlkHnqGSqUCIiCSot2euZvmGnVx/YkeCE4zjSgVC\nRCQBFRU5j366iM5NMzj18KahZFCBEBFJQB/MXUvuuu1cf2JHkpLi33oAFQgRkYTj7jzyaS45jetw\ndvcWoeVQgRARSTD/XZjP7FVbue6EDiSH1HoAFQgRkYTi7jzyySJa1E/nx0e1CjWLCoSISAKZvHQj\nU5Zv4prjO1ArJdxVtAqEiEgCefTTRWRlpPHTY1ofeOIYU4EQEUkQM1ZuZkLueq4e1I701OSw46hA\niIgkikc+WUT92qlc0q9t2FEAFQgRkYQwb/VWPpq3lisGtCMjLZ5Xgy6dCoSISAJ49NNFZKSlcPmx\nOWFH+Y4KhIhIyBbnb+edWav5Wb+21K+TGnac76hAiIiE7PHPFpOWksRVg9qFHeV7YlYgzOwpM1tn\nZrOLDXvAzOab2Uwz+7eZNSg27jYzW2RmC8zs9FjlEhFJJCs37uTN6au46Jg2ZGWkhR3ne2LZgngG\nOKPEsA+BI9y9O7AQuA3AzLoCFwHdgnkeM7Pwj/ESEYmxJ8YvxgyuOb592FF+IGYFwt3HAxtLDPvA\n3fcFTycB+88jHwyMcfc97r4UWAT0iVU2EZFEsG7rbl6ZksdPjm5F8/q1w47zA2Hug7gCeC943BJY\nWWxcXjDsB8xsuJlNMbMp+fn5MY4oIhI7IycsYV9hEdce3yHsKFGFUiDM7HZgH/BCRed19xHu3tvd\ne2dnZ1d+OBGRONi4Yy/PT1rBuT1a0LZx3bDjRBX3szHM7HLgbOBkd/dg8CqgeMcjrYJhIiLV0tOf\nL2VXQSHXn9gx7CilimsLwszOAG4BznX3ncVGjQUuMrM0M2sHdAK+imc2EZF42bKzgGe+WMYZ3ZrR\nqWlm2HFKFbMWhJm9BJwAZJlZHnAnkaOW0oAPgwtwT3L3a919jpm9AswlsunpencvjFU2EZEwPTF+\nMdt27+OXJ3cKO0qZYlYg3H1olMFPljH9PcA9scojIpII1m3bzdOfL+PcHi3o2qJe2HHKpDOpRUTi\n6JFPFlFQWMT/nNo57CgHpAIhIhInKzbs5MXJK7jwmNbkZCXmkUvFqUCIiMTJgx8tJDnJ+OVJib3v\nYT8VCBGROJi/ZitvzljF5cfm0Kx+ethxykUFQkQkDv46biEZaSlcd0JinjUdjQqEiEiMTV2+iY/m\nreWa49rToE6tsOOUmwqEiEgMuTsPjJtPVkYtfj4gsa73cCAqECIiMTQhdz2TlmzkhhM7UjdBrjVd\nXioQIiIxUlTkPDBuAS0b1GZo3zZhx6kwFQgRkRh5b/YaZq3awq9P7UxaStW7BpoKhIhIDOwrLOJv\nHy6gU5MMfnxU1MvbJDwVCBGRGHh9Wh5L8nfwm9O7kJxkYcc5KCoQIiKVbHdBIf/4KJeerRtwWtem\nYcc5aCoQIiKV7PlJy1m9ZTe3nN6F4NIGVZIKhIhIJdq+Zx+PfbaYgR2zOLZjVthxDokKhIhIJRo1\nYQkbd+zl/53eJewoh0wFQkSkkmzcsZdRE5byoyOa0aN1g7DjHDIVCBGRSvLYp4vYuXcfN5+W+BcD\nKo+YFQgze8rM1pnZ7GLDGpnZh2aWG9w3LDbuNjNbZGYLzOz0WOUSEYmFbzfv4tlJyzm/Vys6NskM\nO06liGUL4hngjBLDbgU+dvdOwMfBc8ysK3AR0C2Y5zEzq3qnHYpIjfXQR7ngcFMVuJRoecWsQLj7\neGBjicGDgdHB49HAkGLDx7j7HndfCiwC+sQqm4hIZVqcv51Xp67kkn5taNmgdthxKk2890E0dffV\nweM1wP4zSFoCK4tNlxcM+wEzG25mU8xsSn5+fuySioiU098/WEjt1GSuP7Fj2FEqVWg7qd3dAT+I\n+Ua4e293752dnR2DZCIi5Td71RbembWaKwe1JysjLew4lSreBWKtmTUHCO7XBcNXAa2LTdcqGCYi\nktD+Mm4BDeukcvWgqnUxoPKId4EYCwwLHg8D3io2/CIzSzOzdkAn4Ks4ZxMRqZBJSzYwfmE+vzih\nI5npqWHHqXQxu7yRmb0EnABkmVkecCdwH/CKmV0JLAcuBHD3OWb2CjAX2Adc7+6FscomInKo3J2/\nvD+fZvXSubR/27DjxETMCoS7Dy1l1MmlTH8PcE+s8oiIVKZxc9YybcVm7j3vSNJTq+dR+TqTWkSk\ngnYXFHLPu3Pp0jSTC45uFXacmKlaV9AWEUkAoyYsYeXGXbx4dV9Skqvv7+zq+8lERGJg9ZZdPPrp\nYn50RDOO7VC1u/M+EBUIEZEKuPfd+RS589szDw87SsypQIiIlNPXyzYy9ptvueb4DrRuVCfsODGn\nAiEiUg6FRc6db82hRf10rju+Q9hx4kIFQkSkHF7+eiVzV2/lt2cdTu1a1fOw1pJUIEREDmDLzgIe\nGDefvu0acdaRzcOOEzcqECIiB/DgRwvZsquAO8/phpmFHSduVCBERMqwcO02npu0nIv7tqFri3ph\nx4krFQgRkVK4O3/4zxwy0lK4+dQuYceJOxUIEZFSjJuzls8XbeDm0zrTsG6tsOPEnQqEiEgUuwsK\n+dM7czmsWSYX92kTdpxQqECIiEQxcvwS8jbt4o5zulbr/pbKUjM/tYhIGb7dvIvHPlvMmUdW//6W\nyqICISJSwr3v1Zz+lsqiAiEiUsxXSzfyn6C/pVYNq39/S2VRgRARCRQWOXeOrVn9LZUllAJhZr82\nszlmNtvMXjKzdDNrZGYfmllucN8wjGwiUnON+XoF82pYf0tliXuBMLOWwC+B3u5+BJAMXATcCnzs\n7p2Aj4PnIiJxsWVnAX8dt6DG9bdUlrA2MaUAtc0sBagDfAsMBkYH40cDQ0LKJiI1UE3tb6kscS8Q\n7r4K+CuwAlgNbHH3D4Cm7r46mGwN0DTe2USkZlqwpub2t1SWMDYxNSTSWmgHtADqmtnPik/j7g54\nKfMPN7MpZjYlPz8/5nlFpHqr6f0tlSWMTUynAEvdPd/dC4A3gGOBtWbWHCC4XxdtZncf4e693b13\ndnZ23EKLSPU0bs4avlhcc/tbKksYBWIF0M/M6lhkQ9/JwDxgLDAsmGYY8FYI2USkBon0tzSvRve3\nVJaUeL+hu082s9eAacA+YDowAsgAXjGzK4HlwIXxziYiNctjny0mb9MuXry6b43tb6kscS8QAO5+\nJ3BnicF7iLQmRERibv6arTz26SIG92xRo/tbKotKpojUOPsKi7jltZnUr53Kned0CztOwgqlBSEi\nEqYnJy5lZt4WHh56FI20Y7pUakGISI2yJH87f/9wIad2bcrZ3XXGdFlUIESkxigqcm59fRa1UpL4\n05AjdMb0AahAiEiN8cLk5Xy1bCO/P6srTeulhx0n4alAiEiNkLdpJ/e9N59BnbK4oHersONUCSoQ\nIlLtuTu//fdsHPjzj4/UpqVyUoEQkWrv9WmrGL8wn1tO70LrRjX7KnEVoQIhItXaum27+ePbc+nd\ntiGX9c8JO06VUmqBMLN3zSwnflFERCrfnW/NYVdBIff/pDtJSdq0VBFltSCeBj4ws9vNLDVegURE\nKst7s1bz3uw13HRKJzpkZ4Qdp8op9Uxqd3/VzN4Dfg9MMbPngKJi4/8eh3wiIgdl8869/P6tORzR\nsh7DB7UPO06VdKCuNvYCO4A0IJNiBUJEJJHd/fZcNu/cy7NX9FFPrQep1AJhZmcAfydynYZe7r4z\nbqlERA7BpwvW8ca0Vdx4UkddQvQQlNWCuB24wN3nxCuMiMih2ra7gNvfmEWnJhnccFLHsONUaWXt\ngxgUzyAiIpXh/vfns3rrbl6/7ljSUpLDjlOlacOciFQbk5Zs4PlJK7hiQDt6tWkYdpwqTwVCRKqF\nXXsLufX1mbRpVIffnNYl7DjVgi4YJCLVwoMfLWTZhp28eHVfatfSpqXKEEoLwswamNlrZjbfzOaZ\nWX8za2RmH5pZbnCv9qGIlMs3KzczasIShvZpo+tLV6KwNjE9BLzv7ocBPYB5wK3Ax+7eCfg4eC4i\nUqa9+yLXl26Smc5tZx4WdpxqJe4FwszqA8cBTwK4+1533wwMBkYHk40GhsQqw7L1O7hoxJes3KhT\nO0Squkc+XcSCtdv483lHUC9dvQJVpjBaEO2AfOBpM5tuZqPMrC7Q1N1XB9OsAZpGm9nMhpvZFDOb\nkp+ff1ABksyYs2orN740nYJCnRwuUlVNXrKBRz7J5bxeLTnpsKirDDkEYRSIFKAX8Li7H0WkK4/v\nbU5ydwc82szuPsLde7t77+zs7IMK0KZxHe47vzszVm7mr+MWHNRriEi4Nu7Yy6/GzKBt47rcPfiI\nsONUS2EUiDwgz90nB89fI1Iw1ppZc4Dgfl0sQ5zVvTmX9G3DE+OX8OmCmL6ViFQyd+c3r37Dxh17\neXjoUWSk6YDMWIh7gXD3NcBKM9t/oPLJwFwifT4NC4YNA96KdZbfn92Vw5plcvMr37B26+5Yv52I\nVJInJy7lk/nruP2swzmiZf2w41RbYR3FdCPwgpnNBHoCfwbuA041s1zglOB5TKWnJvPIxb3YtbeQ\nm8bMoLAo6lYtEUkg36zczP3vz+f0bk25rH/bsONUa6EUCHefEexH6O7uQ9x9k7tvcPeT3b2Tu5/i\n7hvjkaVjkwzuHtyNL5ds4JFPFsXjLUXkIG3dXcANL02jSWY6fzm/B2a6QlwsqasN4CdHt+LHR7Xk\noY8XMmnJhrDjiEgU7s5tb8zi2827+efQntSvo0NaY00FAjAz/jjkCHIa1+VXY6azYfuesCOJSAkv\nfbWSd2au5ubTOnN020Zhx6kRVCACGWkpPHzxUWzaWcBvXv2GIu2PEEkY89ds5Q//mcOgTllce1yH\nsOPUGCoQxXRrUZ/fnXU4ny7I58mJS8OOIyLAzr37uP6FadSrncqDP+1JUpL2O8SLCkQJl/Zry+nd\nmnL/+/OZsXJz2HFEarw735rDkvU7+MdPe5KVkRZ2nBpFBaIEM+Mv5/egab10bnxpGlt2FYQdSaTG\nenP6Kl6dmscNJ3ZkQEf10hpvKhBR1K+Tyj+HHsW3m3dz2xszifT8ISLxtCR/O7f/exZ9chrxq5M7\nhR2nRlKBKMXRbRvym9O68O6sNbwweUXYcURqlN0Fhdzw4nRSU5J4aGhPUpK1qgqDlnoZrjmuPcd1\nzubut+cyb/XWsOOI1Bj3vjuPuau38tef9KB5/dphx6mxVCDKkJRk/P3CHjSoncoNL05j5959YUcS\nqfben72G0V8u58qB7Tilq7rwDpMKxAFkZaTxj5/2ZMn6Hdzx1pyw44hUa3mbdnLLa9/QvVV9/vcM\nXR0ubCoQ5XBsxyxuPLEjr03N441peWHHEamWCgqL+OVL0ylyeHjoUdRK0eopbPoGyumXJ3eiT04j\nfvfmbBbnbw87jki187cPFjJtxWbuPe9I2jauG3YcQQWi3FKSI0dTpKUkccOL09ldUBh2JJFq478L\n8/nXfxcztE9rzunRIuw4ElCBqIDm9Wvztwt7MG/1Vn7/5mydHyFSCVZs2MmvX55Bl6aZ3HF2t7Dj\nSDEqEBV00mFN+eVJHXl1ah7/+u+SsOOIVGlbdhVwxeivKSxyHv9ZL2rXSg47khSjC7kehF+f2pml\nG3Zy//vzadu4Dmce2TzsSCJVTkFhEde/MI3lG3bw7BV9aZ+dEXYkKUEtiINgZjzwk+70atOAX788\nQ536iVSQu3PHW7OZuGg9f/7xkfTv0DjsSBJFaAXCzJLNbLqZvR08b2RmH5pZbnDfMKxs5ZGemszI\ny3rTpF4aV42eQt6mnWFHEqkyRk5YwktfreT6EztwQe/WYceRUoTZgvgVMK/Y81uBj929E/Bx8Dyh\nNc5I4+nLj2HPvkKufGYKW3er51eRA3l/9hrufW8+Zx3ZnJtP7RJ2HClDKAXCzFoBZwGjig0eDIwO\nHo8GhsQ718Ho2CSTf/3saBbnb+eGF6ezr7Ao7EgiCWtm3mZuenk6PVo14G8X9tDFfxJcWC2IfwC3\nAMXXpk3dfXXweA1QZTphGdAxiz8NOYLxC/O56z9zdPirSBTfbt7FlaOn0LhuGiMv6016qo5YSnRx\nLxBmdjawzt2nljaNR9awUdeyZjbczKaY2ZT8/PxYxaywi/q04Zrj2/P8pBU89fmysOOIJJTte/Zx\nxTNfs3tvIU///BiyM3VluKogjBbEAOBcM1sGjAFOMrPngbVm1hwguF8XbWZ3H+Huvd29d3Z2drwy\nl8v/nn4YZ3Rrxp/emctHc9eGHUckIewrLOLGF6eRu247j17Si85NM8OOJOUU9wLh7re5eyt3zwEu\nAj5x958BY4FhwWTDgLfine1QJSUZD/60J0e2rM8vx0xn9qotYUcSCd2f3pnHpwvyuXtwN47rnFg/\n6qRsiXQexH3AqWaWC5wSPK9yatdKZtRlvWlQO5UrR3/Nmi27w44kEppnPl/KM18s46qB7bikb9uw\n40gFhVog3P0zdz87eLzB3U92907ufoq7bwwz26FoUi+dJy8/hh17Crly9Nfs2KMLDUnN88n8tdz9\n9lxO7dqU2848POw4chASqQVRrRzevB4PX3wU81Zv5VdjplNYpCObpOaY++1WbnxxOoc3r8dDF/Uk\nWYezVkkqEDF0Ypcm3HVuNz6at44/vzvvwDOIVAPrtu7mytFfk5meypPDjqFOLXX5VlXpm4uxy/rn\nsCR/B09OXEpOVl0u7aftsFJ97dy7jytHT2HLrgJevbY/zeqnhx1JDoEKRBz8/uyurNi4k7vGzqFN\nozocryM5pBoqKnJ+/fIM5ny7hZGX9aZbi/phR5JDpE1McZCcZPxz6FF0bprJ9S9MY/6arWFHEql0\n978/n3Fz1vK7s7py8uFVpiMEKYMKRJxkpKXw1OW9qVMrmcue/ErXtZZq5dFPF/HE+CVc2q8tPx+Q\nE3YcqSQqEHHUvH5tnr+qL0XuXDRiEovWqUhI1ffwx7k8MG4BQ3q24M5zumKmI5aqCxWIOOvcNJOX\nru6HO0GR2BZ2JJGD9tBHufztw4Wcd1RL/nZhT1KStUqpTvRthqBT00zGDO8LRIpE7loVCal6Hvxw\nIQ9+tJDze7XigQt66FyHakgFIiQdm2QyZng/zIyhIyexUEVCqgh35+8fLuShj3O54OhW/OUn3VUc\nqikViBB1bJLBmOH9SDJj6IhJLFijIiGJzd158MOF/PPjXH7auzX3n6/iUJ2pQISsQ3akSKQkR1oS\nOgRWEpW787cPFvLPTxZx0TGtufe8I3VFuGpOBSIBtM/OYMzw/tRKTmLoiEnM/VZFQhKLu/PAuAU8\n8ukihvZpw59/rOJQE6hAJIh2WXUZM7wf6anJXDJqEnO+1bUkJDG4O/e/v4DHPlvMJX3bcM+QI1Qc\naggViASSExSJ2qnJXDJqsi44JKFzd+57bz7/+u9iLu3Xlj+pONQoKhAJpm3juowZ3p+6tVJUJCRU\n7s6f353HE+OXMKx/W+4e3E0nwdUwKhAJqE3jOowZ3o+MtBQuHjmJWXkqEhJf7s6f3pnHyAlLufzY\nHO46V8WhJlKBSFCtG0WKRL3aqVwyahLfrNwcdiSpIdydu9+ey5MTl/LzATnqPqMGU4FIYPuLRP06\nqfzsycnMUJGQGHN3/vCfuTz9+TKuHNiOO85WcajJ4l4gzKy1mX1qZnPNbI6Z/SoY3sjMPjSz3OC+\nYbyzJaJWDeswZnh/GtapxaWjJjN9xaawI0k15e7cOXYOz3yxjKsHteN3Zx2u4lDDhdGC2Afc7O5d\ngX7A9WbWFbgV+NjdOwEfB88FaNmgNmOG96NRRi0uGTWZ92evDjuSVDO79hZy40vTefbL5VxzXHt+\ne6aKg4TMh5ypAAAQIElEQVRQINx9tbtPCx5vA+YBLYHBwOhgstHAkHhnS2QtGtTm1Wv607lpJtc+\nP40HP1xIUZGHHUuqgbxNOzn/8S94Z9Zq/veMw7j1R4epOAgQ8j4IM8sBjgImA03dff9P4zVA1EtS\nmdlwM5tiZlPy8/PjkjNRNKmXzpjh/Ti/Vyse+jiXa5+fyvY9+8KOJVXY5CUbOPeRz1m5aSdPDTuG\n607ooOIg3wmtQJhZBvA6cJO7f69vCXd3IOrPY3cf4e693b13dnbNu7Zzemoyf72gO3ec3ZWP56/j\nvMc+Z/mGHWHHkirouUnLuWTUZBrUSeXN6wdw4mFNwo4kCSaUAmFmqUSKwwvu/kYweK2ZNQ/GNwfW\nhZGtKjAzrhjYjmev6MO6bXs495HPmZi7PuxYUkXs3VfEbW/M4vdvzua4ztm8ef0AOmRnhB1LElAY\nRzEZ8CQwz93/XmzUWGBY8HgY8Fa8s1U1AzpmMfb6gTSrl85lT01m1IQlRBpfItHlb9vDxSMn8dJX\nK/jFCR0YeVlv6qWnhh1LEpTFe4ViZgOBCcAsoCgY/Fsi+yFeAdoAy4EL3X1jWa/Vu3dvnzJlSgzT\nVg079uzjf16Zwbg5azmvV0v+/OMjSU9NDjuWJJhZeVsY/twUNu3cywM/6cE5PVqEHUlCYmZT3b33\ngaZLiUeY4tx9IlDaXrCT45mluqiblsLjlxzNw58s4sGPFrI4fwdP/OxomtVPDzuaJIi3Zqziltdm\nkpWRxuvXHUu3FvXDjiRVgM6kriaSkoxfndKJJy49mkVrt3HOIxOZulwn1dV0hUXOve/O41djZtCj\ndQPG3jBAxUHKTQWimjm9WzPe+MUAaqcmM3TEJF6ZsjLsSBKSLTsLuOKZr3li/BIu7deWF67qS+OM\ntLBjSRWiAlENdWmWydgbBtCnXSNueW0md42dQ0Fh0YFnlGpj0bptDHnsc75YvJ57zzuSPw45gtRk\n/btLxegvpppqUKcWz/z8GK4c2I5nvljGZU9+xcYde8OOJXHw0dy1DHn0C7btLuClq/sxtE+bsCNJ\nFaUCUY2lJCfx+7O78rcLejB1xSbOfWQiXyzW+RLV1e6CQu5/fz5XPzeFdll1GXvDQHrnNAo7llRh\nKhA1wPlHt+KVa/qTnGRcPHIyv3n1GzapNVGtfL5oPWf8YzyPf7aYC49uzavX9qdFg9phx5IqLu6H\nuUo4erZuwLibjuOfH+cyYvwSPpm/jjvO7srgni3U904VtnHHXu55Zx6vT8sjp3EdXriqLwM6ZoUd\nS6qJuJ8oV5l0otzBmb9mK7e+PosZKzczqFMW9ww5kjaN64QdSyrA3fn39FX88e25bNu9j2uP78AN\nJ3XUCZJSLuU9UU4FooYqLHJemLycv7y/gH1FRdx0SmeuHNhOR7pUAcs37OD2f89m4qL19GrTgHvP\n606XZplhx5IqRAVCymXNlt3cOXY24+as5bBmmdx3fnd6tm4QdiyJoqCwiJETlvDQR7nUSk7ilh8d\nxiV92pCUpE2EUjEqEFIh4+as4c635rB2226G9c/hN6d3ISNNu6gSxbQVm/jtG7OYv2YbZ3Rrxl3n\ndlNXKnLQErYvJklMp3drxrEdGvPXcQsY/eUyxs1Zw92Dj+DUrlGv2yRxsm13AQ+MW8Bzk5bTNDOd\nEZcezWndmoUdS2oItSDkB6at2MRtr89iwdrIr9U/DO5G03r6tRpvJVt1N5/WmUx1zS2VQJuY5JD8\nYHv3GV24uG9bkrW9O+ZWb9nFXWPnfLdf6N7zjuSoNg3DjiXViAqEVIriR8y0aVSHKwe244LerahT\nS1snK9vCtdsYNWEJb07/FjP49ak6skxiQwVCKo27M27OGp4Yv4TpKzZTv3YqP+vXhmH9c2iiTU+H\nxN35YvEGRk5YwmcL8klPTeKCo1sz/Lj2tG6kc1MkNlQgJCamLt/IiPFL+GDuWlKTkhjcswVXH9ee\nzk11HH5FFBQW8fbMbxk5filzV28lK6MWw/rncEm/tjSqWyvseFLNqUBITC1bv4MnJy7l1akr2V1Q\nxPGds7l6UHsGdGysrjvKsHV3AS9NXsEzXyxj9ZbddGySwdWD2jG4Z0udBS1xU2ULhJmdATwEJAOj\n3P2+0qZVgQjfph17eWHycp75Yjnrt+/h8Ob1GH5cO87u3kLbzovJ27STpz9fxstfr2T7nn30b9+Y\n4ce15/jO2TrRTeKuShYIM0sGFgKnAnnA18BQd58bbXoViMSxu6CQt2asYtSEpeSu206zeulcPiCH\noX3aUL92zT00c2beZkZOWMq7s1YDcHb35lw9qD1HtNRlPyU8VbVA9AfucvfTg+e3Abj7vdGmV4FI\nPEVFzn9z8xk5fglfLN5A3VrJ/PSYNlzQuxWHNcusEZufduzZx4Tc9Tz9+VImL91IRloKQ/u05vIB\n7WipLrglAVTVM6lbAsUvopwH9C0+gZkNB4YDtGmjK2UlmqQk48QuTTixSxNmr9rCqAlLePbLZTz1\n+VKyM9MY2DGLgR2zGNQpq9ocAVVY5MxatYWJufmMz13P9BWbKCh0WtRP53dnHc5Pj2mtE9ykSkq0\nFsRPgDPc/arg+aVAX3e/Idr0akFUDeu27ua/C/OZkLuezxetZ0NwsaIuTTMZ2CmLgZ2y6NuuUZU6\nt2Llxp1MyF3PhNx8vli8gS27CgDo1qIegzplM6hTFn3aNdJ+GElIVbUFsQpoXex5q2CYVGFN6qVz\nQe/WXNC7NUVFzrw1W5mQu56Juet5btJynpy4lFrJSRzdtiEDO0VaF91a1E+os7a37Crgy8UbmJCb\nz8RF61m+YScAzeunc3q3pgzslM2ADo1pnJEWclKRypNoLYgUIjupTyZSGL4GLnb3OdGmVwui6ttd\nUMhXSzcycdF6JuSuZ97qrQA0qJPKgA6R1kXP1g3IzkyjYZ1acSkauwsKyd+2h7xNu/hy8XomLFrP\nNys3U+RQt1Yy/do3DgpZNh2y69aI/SpSvVTJFoS77zOzG4BxRA5zfaq04iDVQ3pqMsd1zua4ztkA\n5G/bw+dBsZiQm887wdE/AEkGjeqmkZ2ZRlZGLbIz0sgKHmdlpH13y85Mo1Hd7xeTXXsLWb99D/nb\n95C/bQ/rt+9h/ba9kfvvbntZv20P2/bs+957dm/VgOtP7MjAjlkc1aYhtVK02UhqhoRqQVSUWhDV\nm7uTu247C9duY/22YAUerMzzg5X5+u172LOv6AfzRopJLWrXSmbj9r3s2FsY9T3q1079ruAULzBZ\nGbVoUi+dXq0bUr+OdjBL9VIlWxAixZkZnZtmltmNh7uzfc++oFVQrEWwLVJEdu7dR+O6aWRlRgpA\n9v6WRmYtGtdNU2tApAwqEFKlmRmZ6alkpqfSPjvsNCLVi34+iYhIVCoQIiISlQqEiIhEpQIhIiJR\nqUCIiEhUKhAiIhKVCoSIiESlAiEiIlFV6a42zCwfWH4IL5EFrK+kOJVJuSpGuSpGuSqmOuZq6+4H\nPLW0SheIQ2VmU8rTH0m8KVfFKFfFKFfF1ORc2sQkIiJRqUCIiEhUNb1AjAg7QCmUq2KUq2KUq2Jq\nbK4avQ9CRERKV9NbECIiUgoVCBERiapaFwgzu8DM5phZkZn1LjHuNjNbZGYLzOz0UuZvZGYfmllu\ncN8wRjlfNrMZwW2Zmc0oZbplZjYrmC7m11o1s7vMbFWxbGeWMt0ZwXJcZGa3xiHXA2Y238xmmtm/\nzaxBKdPFfHkd6LNbxD+D8TPNrFcsckR539Zm9qmZzQ3+B34VZZoTzGxLse/3jjhlK/N7CWOZmVmX\nYsthhpltNbObSkwTl+VlZk+Z2Tozm11sWLnWRZX+v+ju1fYGHA50AT4Dehcb3hX4BkgD2gGLgeQo\n8/8FuDV4fCtwfxwy/w24o5Rxy4CsOC6/u4DfHGCa5GD5tQdqBcu1a4xznQakBI/vL+17ifXyKs9n\nB84E3gMM6AdMjtN31xzoFTzOBBZGyXYC8Ha8/p7K+72EtcxKfK9riJxMFvflBRwH9AJmFxt2wHVR\nLP4Xq3ULwt3nufuCKKMGA2PcfY+7LwUWAX1KmW508Hg0MCQ2SSPMzIALgZdi+T6VrA+wyN2XuPte\nYAyR5RYz7v6Bu+8Lnk4CWsXy/cpQns8+GHjWIyYBDcyseayDuftqd58WPN4GzANaxvp9K0koy6yY\nk4HF7n4ovTQcNHcfD2wsMbg866JK/1+s1gWiDC2BlcWe5xH9n6epu68OHq8BmsY41yBgrbvnljLe\ngY/MbKqZDY9xlv1uDJr5T5XSrC3vsoyVK4j82owm1surPJ897OWDmeUARwGTo4w+Nvh+3zOzbnGK\ndKDvJexldhGl/0gLY3lB+dZFlb7cUg5l5kRgZh8BzaKMut3d36qs93F3N7ODPia4nDmHUnbrYaC7\nrzKzJsCHZjY/+LVx0MrKBTwO/JHIP/QfiWz+uuJQ3q8ycu1fXmZ2O7APeKGUl6n05VXVmFkG8Dpw\nk7tvLTF6GtDG3bcH+5feBDrFIVbCfi9mVgs4F7gtyuiwltf3HOq6qCKqfIFw91MOYrZVQOtiz1sF\nw0paa2bN3X110MRddzAZ4cA5zSwFOA84uozXWBXcrzOzfxNpUh7SP1Z5l5+ZjQTejjKqvMuyUnOZ\n2eXA2cDJHmyAjfIalb68SijPZ4/J8ikPM0slUhxecPc3So4vXjDc/V0ze8zMstw9ph3TleN7CW2Z\nAT8Cprn72pIjwlpegfKsiyp9udXUTUxjgYvMLM3M2hH5FfBVKdMNCx4PAyqtRRLFKcB8d8+LNtLM\n6ppZ5v7HRHbUzo42bWUpsd33x6W839dAJzNrF/z6uojIcotlrjOAW4Bz3X1nKdPEY3mV57OPBS4L\njszpB2wptqkgZoL9WU8C89z976VM0yyYDjPrQ2R9sCHGucrzvYSyzAKltuLDWF7FlGddVPn/i7He\nIx/mjchKLQ/YA6wFxhUbdzuRPf4LgB8VGz6K4IgnoDHwMZALfAQ0imHWZ4BrSwxrAbwbPG5P5KiE\nb4A5RDa1xHr5PQfMAmYGf2jNS+YKnp9J5CiZxXHKtYjIttYZwe1fYS2vaJ8duHb/d0nkSJxHg/Gz\nKHY0XYyX0UAimwZnFltOZ5bIdkOwbL4hsrP/2Djkivq9JMgyq0tkhV+/2LC4Ly8iBWo1UBCsv64s\nbV0U6/9FdbUhIiJR1dRNTCIicgAqECIiEpUKhIiIRKUCISIiUalAiIhIVCoQIpXIIr2oLjWzRsHz\nhsHznHCTiVScCoRIJXL3lUS6KLkvGHQfMMLdl4UWSuQg6TwIkUoWdHExFXgKuBro6e4F4aYSqbgq\n3xeTSKJx9wIz+3/A+8BpKg5SVWkTk0hs/IhIdwlHhB1E5GCpQIhUMjPrCZxK5Gpov47zxW5EKo0K\nhEglCnr7fJzI9RdWAA8Afw03lcjBUYEQqVxXAyvc/cPg+WPA4WZ2fIiZRA6KjmISEZGo1IIQEZGo\nVCBERCQqFQgREYlKBUJERKJSgRARkahUIEREJCoVCBERier/AwPO5BPqkpS9AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f04437885d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Minimum value =  -4\n"
     ]
    }
   ],
   "source": [
    "figure1()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, we can find the exact minimum through the use of calculus. By calculating the first derivative with respect to x we can find the slope, or <strong>gradient</strong> of y for any given x.  By setting this first derivative equal to zero we are able to identify points at which the slope is equal to zero, and since we know that the minimum value must occur at a point where the slope is zero, by identifying these points and finding the correspoonding y values we are able to identify the x value which minimizes our function.  \n",
    "\n",
    "\n",
    "$$ y = x^2 + 4x $$\n",
    "Calculate the first derivative:\n",
    "$$ \\frac{dy}{dx} = 2x + 4$$\n",
    "Set derivative = 0 and solve for x\n",
    "$$ 2x + 4 = 0 $$\n",
    "$$ 2x = -4 $$\n",
    "$$ x = -2 $$\n",
    "\n",
    "This indicates that the minimum value for y is achieved when x = -2, which is supported by the plot above, and matches our intuition based on that plot.\n",
    "\n",
    "### Simple Multivariable Optimization Example\n",
    "\n",
    "Now lets examine the situation where we are looking to optimize $f(x)$ with more than one input variable.  For simplicity, we'll look at the case of just two variables, $x_1 and x_2$, and minimize the equation $f(x) = {x_1}^2 + {x_2}^2 + 6x_1 -10x_2$, with corresponding plot below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWQAAADuCAYAAAAOR30qAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsvXmYG9WVNv5e7Xvvm91tt/fdGG9tMmyGsCRhIIEfqweb\nQCAEEiAhkzHw5QeEJJhMIBAmmUmIhyVfCBgImECABMK+2RhssN3d7t3t3lfte53vD6mqS1JJqlKr\n3ZKt93n0SLp169aiqreOzj3nPYyIUEABBRRQwPRDNd07UEABBRRQQAQFQi6ggAIKyBEUCLmAAgoo\nIEdQIOQCCiiggBxBgZALKKCAAnIEBUIuoIACCsgRFAi5gAIKKCBHUCDkAgoooIAcQYGQCyiggAJy\nBBqF/QtpfQUUUIBcsOnegXxDwUIuoIACCsgRHJOEfPjwYVgsFoTD4enelaOKxx57DCeffPJR2VZz\nczNWrVoFq9WKX//610dlm/mOwjkrIB3ympDr6+thNBphsViEV29vL2bNmgWXywW1Wj3du4iPPvoI\nZ511FkpLS1FRUYGLL74YfX19GY931113QavVwmKxoLi4GF/60pfw4YcfKh7n9NNPxx/+8IeM9+MX\nv/gFNm7cCKfTiZtuukmyz2uvvYZTTz0VVqsVFRUVOO200/Diiy9mvM2jjZ07d2LVqlWw2WwoLy/H\nGWecgY6OjozHk3POCji+kdeEDAB//etf4XK5hNeMGTOmdHuhUEhR/7GxMVx33XXo7OxEV1cXrFYr\nvvnNb05qHy699FK4XC4MDQ3h5JNPxoUXXoijrdrX1dWFZcuWJV3+7LPP4uKLL8bmzZtx5MgRDAwM\n4Cc/+Qn++te/HsW9zBytra3YvHkz7r//ftjtdnR0dODGG2/M6CHPXzPpzlkBBYCIlLxyCrNnz6Z/\n/OMfCe0dHR0EgILBIBERtbe30ymnnEIWi4XOPPNMuuGGG2jTpk1ERPTmm2/SzJkzk45755130kUX\nXUSbNm0iq9VKjzzyCIXDYbr33ntp7ty5VFpaShdffDGNjIzI2uc9e/aQxWLJ+JjvvPNOYd+JiPbv\n308AaGhoiB599FH6l3/5F2HZ+++/T2vXriWbzUZr166l999/n4iIbr/9dlKpVKTX68lsNtONN94o\nua2dO3fS0qVLqaioiE477TQ6ePAgERFt3LgxZv3m5uaY9TiOo7q6OvrFL36R9DjC4TDdc889NGvW\nLKqoqKArr7ySxsfHiWji93vssceorq6OysrK6Kc//SkREfX09JDBYIg5359++imVlZVRIBAgIqLt\n27fT4sWLqbi4mM4++2zq7OwUzkdZWRkdPnyYiIj27t1LxcXF1NjYmLB/zzzzDJ1wwglJ93/Lli10\nxx13CN/jr6PZs2fTtm3baMWKFaTT6STP2UsvvUSrVq0iq9VKtbW1dOedd8Zs491336WTTjqJioqK\nqLa2lh599FEiIvL5fHTrrbdSXV0dVVZW0re//W3yeDxJ93UaoZRfjvvXcUHIGzZsoFtvvZX8fj+9\n++67ZLVaFRGyRqOh559/nsLhMHk8HnrwwQepoaGBuru7yefz0XXXXUeXXXaZrH3+1a9+RQ0NDRkf\ns5iQfT4f/fCHP6S6ujoiohhCHhkZoeLiYnriiScoGAzSk08+ScXFxTQ8PExERKeddho98sgjSbfT\n3NxMJpOJ/v73v1MgEKD77ruP5s2bR36/P+36jY2NBIDa29uTjr99+3aaN28etbW1kdPppG984xv0\nb//2b0Q08ft961vfIo/HQ3v37iWdThfzQPj9738vjPXDH/6Qvv3tbxMR0QsvvEDz5s2jgwcPUjAY\npHvuuYdOOukkoe/tt99OGzduJI/HQ8uXL6eHH35Ycv/a2tpIr9fTLbfcQv/85z/J6XTGLJdDyCec\ncAIdPnxYIMv4c/bmm2/S559/TuFwmPbt20eVlZX0/PPPExFRZ2cnWSwWevLJJykQCNDw8DB99tln\nRER0yy230L/+67/SyMgIORwOOu+882jr1q1Jz/U0YtoJLt9eeU/IZrOZioqKqKioiC644AIiiiXk\nrq4uUqvV5Ha7hfU2bdqkiJBPOeWUmOWLFy+m119/Xfje29tLGo1GeAAkw759+6ikpITeeeedjI/5\nzjvvJK1WS0VFRVRRUUEbN26kTz75hIhiCfmJJ56gdevWxay7YcMGwcpKR8g/+clP6OKLLxa+h8Nh\nmjFjBr355ptp13/vvfcIAHm93qTjn3HGGfSb3/xG+N7U1CScQ/736+7uFpavW7eO/vznPxMR0SOP\nPEIbN24koog1XltbS2+//TYREZ177rn0hz/8IWa/jUajYCUHAgFavXo1LV++nM455xziOC7pPn74\n4Yd08cUXU3l5Oen1etqyZYtAzHIIefv27THjpTvnN998M91yyy1ERPTzn/+cvv71ryf04TiOTCYT\ntba2Cm0ffPAB1dfXJx13GjHtBJdvr7z3Ib/wwgsYHx/H+Pg4XnjhhYTlvb29KC0thclkEtrq6uoU\nbSO+f1dXF77xjW+guLgYxcXFWLJkCdRqNQYGBpKO0draiq985St46KGHcMopp0j2effdd4XJyVS+\nxksuuQTj4+MYHBzEP//5T6xZsyahT29vL2bPnh3TNnv2bPT09KQ61KTrq1Qq1NXVyVq/rKwMAFJO\nXsaPP3v2bIRCoZhzWF1dLXw2mUxwuVwAgIsuuggffvgh+vr68M4770ClUgnntKurCzfffLPw25SW\nloKIhP3WarW46qqrsH//ftx6661gLHmo7IYNG7Bjxw4MDQ3h3XffxTvvvIOf/exnaY+fR7rr7OOP\nP8bGjRtRUVGBoqIi/M///A+Gh4cBAN3d3Zg3b17COkNDQ/B4PFizZo1wjOeeey6GhoZk71cBuYu8\nJ+R0qKmpwejoKDwej9DW3d0tfDabzTHLwuFwwsUdf9PW1dXhlVdeER4E4+Pj8Pl8mDlzpuQ+dHV1\n4ctf/jJ+/OMf48orr0y6r6eccoowOXngwAFFxxmPGTNmoKurK6bt8OHDwj6mIiKp9YkI3d3dSY9R\njEWLFqGurg7PPfec7PEPHz4MjUaDqqqqtOOXlJTg7LPPxtNPP40nn3wSl112mXA8dXV1+N3vfhfz\n23i9XnzpS18CAPT09ODuu+/GN7/5Tdx6663w+/1ptwcA69atw4UXXoj9+/cDSLxu+vv7E9ZJd46v\nuOIKnH/++eju7obdbsf1118f+dsaPY62traEdcrLy2E0GnHgwAHh+Ox2u/CwKiC/ccwT8uzZs7F2\n7VrcddddCAQC+PDDD2Nm+hcuXAifz4eXX34ZwWAQP/3pT9PepNdffz3uuOMOgVCGhoawc+dOyb49\nPT0444wz8N3vfhfXX3999g4sDb761a/i0KFDePLJJxEKhfD000/j4MGDOO+88wAAVVVVaG9vT7r+\nJZdcgpdffhlvvPEGgsEg7r//fuj1eoHYUoExhgceeAD33HMPHn30UTgcDnAch/feew/XXXcdAODy\nyy/Hr371K3R0dMDlcuH222/HpZdeCo1GXvLoFVdcgSeeeALPPvssrrjiCqH9+uuvx7333is80Ox2\nO5555hkAkYfKVVddhWuuuQbbt29HTU0NfvzjH0uO/9577+GRRx7B4OAgAKCpqQkvvvgiNmzYAABY\ntWoV/va3v2F0dBT9/f148MEHZe23GE6nE6WlpTAYDNi1axeefPJJYdmmTZvw+uuvY8eOHQiFQhgZ\nGcHevXuhUqlw7bXX4vvf/76wbz09PXjttdcUb7+AHIRCH0dOQe6kXmtrK5188slksVjojDPOoGuv\nvZauvvpqof+jjz5K1dXVVFFRQf/5n/+Z4EMWRzUQRfyS999/Py1cuJAsFgvNnTuXbrvtNsl9vOuu\nuwgAmc3mmFemkNof8XGIoyzeffddWr16NdlsNlq9ejW9++67wrIPPviAFixYQMXFxfS9731Pcry/\n/OUvtGTJErLZbHTqqafS/v37hWXp/KFERK+88gqdfPLJZDabqby8nE477TR66aWXiChyDu+++26q\nra2l8vJy2rRpE42OjhJR4u8ntT2Px0MWi4WWLl2asN0nnniCli9fLkQvfPOb3yQiogcffJBWrlwp\nTEz29PRQeXm5pE//iy++oPPOO48qKyvJbDbT7Nmz6Uc/+pEQyeH1eumSSy4hq9VKK1asoAceeCDB\nhxx/bcYfwzPPPEOzZs0ii8VCX/va1+jGG2+M+W3feecdWr9+vXAcjz32mLDt2267jebMmUNWq5UW\nL15MDz30UMrfYpow7T7ZfHsxIkXxq8eElsWll16KxYsX4+67757uXSmggGMZBS0LhTjmXRYAsHv3\nbrS1tYHjOLz66qvYuXMnvv71r0/3bh0VEBFCoRBCoRAUPnwLKKCAowylam95if7+flx44YUYGRlB\nbW0t/vu//xsnnnjidO/WlIPjOASDQfj9fnAcByASLaHRaKDRaKBWq6FSqdJOPhVQQAFHB8ely+JY\nBxEhGAwiHA6DMYZgMChYx7yvSgyn04mysrICSReQbRQuIoU4Lizk4wVEBK/XGwkwj5KqmFjjv/Pr\nHDp0CGvWrEEgEBDa1Wq1YEmrVKoCSRdQwFFAgZCPARCR4J5obGzEzJkzUVJSImtdnqTFojn8eH6/\nPyYEUK1WQ6vVClZ0gaQLKCC7KBBynoMnYo7jwBiDSpU4T8sYUzShl8yS5jgOPp8vph9vSavVaqjV\nasl1CyigAHkoEHKeQhw9AUyQqFLylYtkJB0Oh4V9GB8fh0ajQXFxcYGkCyggAxQIOc8gJkEikvQT\nZ0LI/FhKEL9tt9sNrVYLq9WKYDAoLGOMCQQt9kkXUEABsSgQch5hKtwT2Qa/X/y+8fsSCoUQCAQE\nklapVJLujgIKOJ5RIOQ8QDL3hBQyIWR+nakgRLGVHE/SwWAQwWBQ6MvHSPMEXSDpAo43FAg5h8G7\nJ7q6ulBZWQmtVpuWoKbbQpYD/hjiIzsACKF3LS0tWLBgQSGRpYDjCgVCzlGI3RMDAwMoLy+XRUST\nsZCnE/Ek7XQ6oVKpQEQIBAIJMdJid0eBpAs4VlAg5ByDlHtCpVIJqc/pMJlJvVxDqvC7cDgMv98f\nQ+SFRJYC8h0FQs4RpIqe4C1FOVDSl0c+EVcqkk6VyFIIvysgH1Ag5BxAuugJxphsCxlQbu3mgsti\nMkiXyLJ//34sX768kMhSQM6jQMjTiHgRoGTkoMTqjSdvp9OJpqYmEBFsNpvw0ul0WTuOXIT4XAYC\nAajV6oREFr5fgaQLyBUUCHkawBMDH/KVjgCUWMi8dR0KhdDS0gKHw4GFCxdCrVbD6XTCbreju7sb\nwWAQJpMJVqtV8FlrtdrJH1wOI1W2YSGRpYBcQIGQjzI4joPX60VTUxOWLVs2JZETY2Nj6OrqwuzZ\ns7F48WKEw2GEw2EYDAZUVFQAmFCGczgcCAaDOHDgAIgIFosFNpsNVqsVFosl74ko3XnjSbqQyFJA\nLqBAyEcJYvcEEEkzlntDy42ycLlc6Orqglarxbp161K6JRhjMJlMMJlMGBgYwOLFi6HVauFyueB0\nOtHT0wOXywWVSgWr1Sq4OoxGY14RkdKEl3SJLLt27RKKGxQSWQrINgqEPMWYbBgbv04qSy8cDqOt\nrQ2jo6Oorq6GVqtNIONUxCS2AnninTlzJoCIpeh0OuFwONDW1gav1wutVhvjj9br9bKP5WgjGxmI\n4tA6IhLeAcTERwOFiiwFTA4FQp4iiDWK48PYlLogkhE4EWFwcBCtra2oq6tDQ0MD+vr6YkK/lOyv\nFDQaDUpKSmL0lQOBABwOBxwOB3p6ehAIBGAwGMBxHCwWC0KhEDSa3Li0pjolPD7bsJDIUsBkkBt3\nzTEGOSJASiBF4B6PB42NjdBqtVi7dq1gpR6NTD2dTofy8nKUl5cDiBCRz+dDe3s7XC4X9u3bh3A4\nDIvFIrg7LBZLDHkdLWSbkFOdp0IiSwGTRYGQswglIkBKIA57C4fD6OjowNDQEBYtWoTS0tKYvtMR\nU8wYg9FohNVqhU6nQ3V1NTiOg9vthtPpRF9fH5xOJxhjAkFbrVaYzeYpJyG+nNV0IRVJt7S0wGg0\nChOthUSWAgqEnAWk0yieLPiwt6GhIbS0tKCmpgYNDQ1Zk9+cChLnJwOtVitmzJgBIPIw4f3RXV1d\ncLvd0Gg0Cf7obJ47/l9KtpCNsfjrg+M4wY0RX5GFf5AUYqSPLxQIeZLg3ROff/455s+fD5PJlPVt\nhEIh9PX1wWQyYfXq1TAYDEn75rKWhVqtRnFxMYqLi4W2YDAo+KN5/7der4ff78fIyAhsNtuk4qOz\n6bLI9jniOE5w46SKkU6WyMK7OwokfeygQMgZIt49wXGcosgJOeA4Dl1dXejp6UFVVRWWLFmSdh0p\nQpYj2Tld0Gq1KCsrQ1lZGYDIefX7/fj000+FeOpQKASTySRY0VarVbY/OtuEnG3rPZU7pZDIcvyh\nQMgKkcw9oVars0rIo6OjaG5uRmVlJebMmSObCHLFZZEpGGMwGAzQarWYP38+gMg593g8cDgcGBgY\nQGtrK4hIcInYbDaYzWZJEsomiaYjUKUIh8OKx0uVyPLxxx9j9erVAAqJLPmKAiErQKroCaWxxYA0\nWfj9fjQ3NyMUCuGEE06AyWRCT09PTGWNVMglcs0WGGMwm80wm82oqakBECEzPomlu7sbbrc7Jo7a\narXCaDTmNCGLXRaZIj6UUhwjHV+RZWBgAG+//Tauu+66SW2zgKlDgZBlQE70RKbJHvw4RITDhw/j\nyJEjWLBgASorKxP6ykGm8pv5RuJqtRpFRUUoKioS2oLBoDBpODg4CK/XC7VajXA4jOHh4UmLKh1t\nl0WmSBYj3dvbiw8//LBAyDmMAiGngBIRIP7GlwuewFUqFcbHx9HY2Ijy8nJs2LAhwWpSSvb5Rq7Z\nglarRWlpaUwo4NDQEHp6emJElYxGY4wlLTeJZSpcFtmKzU73sGCMweVywWKxZGV7BUwNCoScBPHu\niXSWkVLSVKlU8Pv9aGpqgs/nw8qVK2E2myX7FizkzKHVamEymTBv3jwAsaJKQ0NDaG9vB8dxMJvN\nAkknE1WaCpdFtsaTM5bL5YLVas3K9gqYGhQIOQ68CNChQ4dQXl4ekzKcCkoIWRxJsGDBAlRVVaUk\nfKV6yPlOyNncl3jLUSyqVF1dDQBCEgufCu5yucAYEyxom80Gk8mU04Qsx9ouEHLuo0DIUcT7iXnJ\nSrmQS8gOhwONjY0Ih8NYvXq1rBtEiR6ykr7HA+T4fcVJLPGiSk6nEx0dHfB4PMKDa2hoKCuiStn0\nScsl5ILLIrdx3BNyMhEgjUaj2AWRisCDwSBaWlrgcrmwdOlStLW1ybaOlFiwx4qFPN2xw1KiSoOD\ng0IaeG9vL/x+PwwGQ4w/erpE/uUQstvtRl1d3VHaowIywXFNyOnC2JRYyGq1WjI0jYjQ19eHjo4O\n1NfXY8mSJYolOCdbdfp4jj/NZuq0RqOB2WzG3LlzAUyIKjkcDoyMjKCjowPhcDjBH52MKLP5uxQs\n5GMDxyUhywljyzRqQgyn04nGxkZYLBasX78+xnpSSsgFC3n6x4r3+fKiSkajEVVVVUIfPoklXlRJ\nnMSS7YdkwYd8bOC4ImQlIkCTIeRQKIS2tjaMjY1hyZIlMbGyUv3TQakPOZ5ceelHo9GYdL1cIuRs\nIptqb3KsbZVKBYvFAovFEiOq5HK5EkSVfD4fBgcHYbVaYTAYJkXSBUI+NnDcELJSjWKVSiU7O47v\nHw6H0d/fj7a2NsyaNQsLFy5MepNNlV84vu/g4CBaWlqg0WiEGNyioqIYTYhcc2nki4UsF1JJLHyU\njdvtRn9/P3w+H/R6fYzynRJ/dIGQjw0c84TMuyeamppQX18PrVYr6wZVaiEHg0H09fUhEAikrWcH\nKHdZKLWQfT4fGhsboVKpsHr1auHfQLwmBBAhGiKCwWDImZp5uUjI2bS21Wo19Ho95syZI4zt9/vh\ncDgyElWSS8g2my0r+1/A1OCYJeR494TT6VQ0wSN3Ui8cDqO9vR39/f2w2WxYsWKF7PEn44ZIBa/X\niz179mDhwoWoqKgAx3FCBWUpTYimpibBzeL1eidlqWUDUxmHPBlMZSIHL6pkMBiEtPlkokp8ZXCx\nqFI4HE5rBLjd7oKFnOM4JglZyj2h1OKVo97GuwNmzpyJlStX4vDhw7LHn4pJPafTiQMHDiAcDuNL\nX/qSrLRctVotpBLzJZn4yAGxpcaTQFFRUVJltWwh2ySazbGyVStQjkUr9QDlOE7wR4tFlYhIcEUl\n+5fjdrsLURY5jmOKkFNFT2RCyMn6e71eNDY2Qq1WY82aNTAYDHC5XIrjlrM1qRcOh9Ha2oqxsTEs\nWrQIhw4dmpRGQrylxmey8XoQLpcrptJHUVFR1itPZ9PNcLT0IpQgU2tbrGjHg3fJif/l6HS6mH85\nOp0O4XA4Z4rPysEalZkcJP+elYtW+F8jonOzPnAWkD+/TgrIiZ7IhoXMcRw6OjowMDCARYsWCaLq\nyfqngpLIiVQW8vDwMA4dOoSZM2eioaFBqHysBOlcIuJMNh7iSh+9vb0IBAIgIpjNZhgMBkUi8vE4\nHl0Wk4FGo4FWq0V1dbUwccj7o+12Ozo7O/Gd73wHXq8Xv/jFL7B+/XqcfPLJMa6o7u5ubN68GQMD\nA2CM4brrrsPNN9+M0dFRXHrppejs7ER9fT127NghJMvce++92L59O9RqNX7961/jnHPOAQDs2bMH\nV111Ffbv398K4G8AbqYMflQHwnjIWD/p8xOPr3mby7M+aJaQ94QsN3oikzA2cf+RkRE0Nzejuroa\nGzZsSNiO0kSSyVrIvDARx3ExZZ2SEXI6UlJ6v0hV+mhra0MoFIqZMORjb3k9CCVC+9lArhJyNpXe\npMbT6/WoqKgQCqh++OGHOPXUU1FbW4sXXngBJ510UgwhazQa3H///Vi9ejWcTifWrFmDs846C489\n9hjOPPNMbN26Fdu2bcO2bdtw33334eDBg3jqqadw4MAB9Pb24stf/rLwz+w73/kOHnnkEZx00kkL\nECHkcwG8ovSYmJpBa50CivJmf8hsIW8JWeye4C3iVDeeRqPJyGXh8/nQ1NQEIsKJJ56YNJY3E7U3\nufsjtpCJCD09Pejq6sL8+fOFhAQemSaGTBaMMeh0OlgsFkG0R1zUtL29PeavNO/vlJowLFjIypGO\n4Pl5lE2bNmHTpk0Jy2tqagQ/tdVqxZIlS9DT04OdO3firbfeAgBs2bIFp59+Ou677z7s3LkTl112\nmRApMn/+fOzatQv19fVwOBzYsGEDiIgYY08A+DoyIWQVg9p4fJWiyjtCFqer2mw2WdKYgHILmTEG\nt9stKLLxlkaq8ZUSspIqIPxkzsGDB2G1WtHQ0CDpD8yEfKYqU0+qqKnf74fdbk85YZirccjZFgM6\nmoTs8XhkF+Dt7OzEZ599hoaGBgwMDAhEXV1djYGBAQBAT08PNmzYIKxTW1uLnp4eaLVa1NbWioc7\nAmCm0uMBAKgwLYTMGKsD8ASAKgAE4PdE9BBjrBTA0wDqAXQCuISIxqLr3AbgGgBhADcR0WvR9jUA\nHgNghAz3TV4RMu+ecLlc6OrqwgknnCB7XbVaHVO9NxXGxsbQ2NgIjuPQ0NAg66+lUpU1pXKdPp8P\nX3zxBZYuXSqZ+SdnjFyIL9br9aisrEyYMBRHDQCRiarBwUHYbLaUVbbTIdtRFtm0kKfSZREPp9Mp\nK8LC5XLhoosuwoMPPpgQsyzX+MkWGGPQGLJ3jhQgBOBWIvqUMWYFsIcx9g8AVwF4g4i2Mca2AtgK\n4D8YY0sBXAZgGYAZAF5njC0kojCA/wZwLYCPIcN9kxeEzGsUh8NhMMag1WoVWbtAhJD9fn/KPn6/\nH4cOHUIgEMCqVauwd+9e2TeN0gtVLiGPjo6iqakJjDE0NDRkTAjpqklMV+q0lPSlw+FAS0sL3G43\n+vr6hLRvcdSAkqrTuehmONraynKy9ILBIC666CJs2rQJF154IQCgqqoKfX19qKmpQV9fn/AgnTlz\nJrq7u4V1jxw5gpkzZ2LmzJk4cuSIeNhaAD2ZHBNjgFp39AmZiPoA9EU/OxljjYhY+RcAOD3a7XEA\nbwH4j2j7U0TkB9DBGGsFsJ4x1gnARkQfAYAc903OE3I4HEYgEAAw8YRW6n4AUrssiAjd3d3o7u7G\n/PnzUVlZOeWWQDpCDgQCaG5ujnk4TAUZ88glLQuNRpOQxcZX+RgcHERbWxuIKIagk00Y5qoPOduT\neumOM53SGxHhmmuuwZIlS/CDH/xAaD///PPx+OOPY+vWrXj88cdxwQUXCO1XXHEFfvCDH6C3txct\nLS1Yv3491Go1bDYbPvroI5x00kkMwGYAD2d0UIxBrZ1eHzJjrB7AiYhYuFVRsgaAfkRcGkCErD8S\nrca7aYLRz/HtSZHzhCwVxqbRaGS7H3gkI2S73Y7GxkaUlJQk9ctOBZK5OMRynXPnzkV1dXXWHw6h\n9k+hmbs6Zl9yCXKqfIgnDHkBeakJw+MhyQRI/xumc1m8//77+OMf/4gVK1Zg1apVAICf//zn2Lp1\nKy655BJs374ds2fPxo4dOwAAy5YtwyWXXIKlS5dCo9HgN7/5jfCA+e1vf4urrroKAFoRsQYVT+gB\nAFMBav2U3I/ljLFPRN9/T0S/T9g+YxYAzwG4hYgc4nMcnbDMuhWT84SsUqkSLrZM6sbFEzJfpsnj\n8WD58uVHPYNJ6hg8Hg8OHjwIo9GYINeZLYTaPxXeeVLONflNOUg2YRifYci7uXgVtslYuNl0f2R7\nUi8d0rksTj755KTXwBtvvCHZfscdd+COO+5IaF+7di32798PAPMy2VcebOos5GEiWptm21pEyPhP\nRPSXaPMAY6yGiPoYYzUABqPtPQDEyv+8m6Yn+jm+PSlynpCzBZ6QxWFjc+bMwdKlS6fFQhS7LDiO\nQ2dnJ/r7+7FkyRLZdfyOVWRq1cbH3nIchwMHDkClUgm18vhMN96K1uv1sreVq5N6ch6meZk2zRg0\nU2Mhp9lC0qlCAAAgAElEQVQsYwC2A2gkogdEi14EsAXAtuj7TlH7k4yxBxCZ1FsAYBcRhRljDsbY\nBkRcHmndN8cVIft8PuzatQs2m022BaqUHOT25wl5fHwcjY2NqKyslEw4mUrwVnIuWsjZeEjysbeV\nlZWCdRgMBuF0OmG329HX1wefzxejqJZqwjBXJ/WO1YrTjAEqzbREWfwLgCsBfMEY2xttux0RIt7B\nGLsGQBeASwCAiA4wxnYAOIhIhMaN0QgLALgBE2Fvad03OU/IqW5MueQXCoXQ0dEBh8OBdevWyZYg\n5K1qub4+ntjk7BPHcRgbG4Pf78fKlSthNptlbWMy4N0VCW36Kone04dsJ4aIyUqr1aK0tBSlpaXC\ncn7CcGhoSJgw5DMMi4qKhAnDXA2hkzNB6HQ6BcH8vAFjUGmO/qQeEb0HINkPfWaSdX4G4GcS7Z8A\nWC532zlPyMkgh/yICP39/Whvb0dtbS3cbrciPVil2Xd8ckiqG42IMDAwIIjGr127dton1ZTGUE81\njmYyR7IJQ5fLBbvdLkwYarVa+Hw+jI6OoqioKK3UZTpkM8pCzlgejyfvXBaMYVpcFtOJvD1aPtIi\n2Y3hcrnQ2NgIo9GIdevWQaPRoLe3V9E2MtW/SGZRe71eHDx4EDqdDieccAJaWlqmxB0iBVdfJ2Ao\ngdE3lrDMwgKw07T8NUyK6cyuS1bhY8+ePUICSygUgtlsFnzRSicMp8NCzjeXRcRCzq3rcqqR84Sc\n7GZKRsihUAjt7e0YGRnBkiVLYmbhJxuZkQ7JLGqO43D48GH09vYKKnGBQECxOlw2LEdvElIu0mRf\n5jBT5KKWhV6vh1arxbx584RxeUnS+AlD/pWqTt7RtpDz14dc0LLIOUhNOsWTJRFhcHAQra2tqK2t\nxYYNGyZ9I2YiGBTf32634+DBgygvL49Jw1Y6djYn3pKRsqOvE7aa+qxsY7LIRf0JMRhjQjFTPsMw\nFAoJkpd8nTw+w7CoqAhWq1X493S0LeT8jLJQQa07utVqpht5QchSECeHeDweNDY2QqvVYu3atVkT\nS5+MhnIoFEJLSwucTidWrFiRcDNkQvZKCJknonA4jPHu1qT9PMZS2WOmQzZJNFvI5kRcOmg0moQJ\nQ5/PB7vdnjBh6PP54PV6odFoFO+f5oWHEPr6zcL3Y7WeHmMAU0+LuND/AjgPwCARLY+2PQ1gUbRL\nMYBxIloVzeRrBNAcXfYREV0fXUeRsBCQx4SsVqsRDAbR2tqKwcFBLF68WLgRsrmNTHzIfGmnWbNm\nYfHixZI3nFKLV8nEGz82H1K3aEZZQp9ctpKzPal3NEMJxWCMwWg0wmg0JkwYjoyM4PDhw8KEoTg2\nOtWEoeaFhxLajlWXxTT6kB8D8F+IKL4BAIjoUv4zY+x+AHZR/zYiWiUxjiJhISBPCFmKvPx+Pxob\nGzFr1ixF8btKbnalVizHcWhuboZer09rqWciRiSXwBljaGxshMfjQd2sengAmELOhH5Dljkwh+0J\n7X2DYzDq2KSqfuQKckXljgc/YajT6bB8eSQaKhAIwG63CyWygsEgzGazQNL8hKEUGQPyXRb5RsgR\ncaGjT1FE9E7U8k1ANGnkEgBnpBojmsmnSFgIyBNCFsPr9aKpqQkejwezZ89GfX297HWVxhXLtZB5\ncaKhoSHMnj1bmPjJJuRayMPDw3A6naipqcHSpUtxpCeiheLRWCVJORm8AULLp5+CMSYQQ1FR0aSk\nMOUiVzWMpwo6nS4mw5CfMHQ4HMKE4Uk9Hwj9nedcB4PouORUnE4VkZS7YGBTYxDI0rJIglMADBBR\ni6htTjSBxA7g/xDRu4iICCkSFgLyiJA5jkNXVxf6+vqwcOFC+Hy+jBXfsknITqcTBw8eRHFxMWpr\na6fMCknn4ggGg2hqakIwGITVakVNTY1AxqngVhfFWMlu9USoV+3s+agotSZMVImtN6vVOm0uATnI\nFiEfzTht8YShkMwhIuSWlhZ4vV5hwtDj8aR01yUr65XzmDqXRVotixS4HMCfRd/7AMwiopGoz/gF\nxtiyTHcsLwh5bGwMBw4cQFVVleCe6OvrE2Q55SJbYWxAbKXnpUuXwmazob29XfFDIhv7wtew49Xh\n9uzZI9lPbCW7NBPhgPGkLIbURJXH44Hdbkdvb68Q7lVUVCQ8DCaLbFu12Rgr275oJQQZ76pYuXJl\nwoSh0+lEd3c3LBaL4Is2m80Jqnn5BMYYVFMgsJUpGGMaABcCWMO3RTWQ/dHPexhjbQAWIgNhISBP\nCDkUCmHVqlUxJWiyKcGZqr+UqP3w8DCam5tRW1uLhoYG4UJX6nNWglR+dMYY1q1bJ/wlTeXeSOa6\nEFvGYvQNjqGmckLsiDEGs9kMs9ksWG98Beru7m6hCjWfNMGXZcplK1oOshmmpoSMte8/Db43KypB\ncOPmyGfRhKHdbkd1dTWsVqsgSdrV1QW3241wOIynn34aADA4OCgIzItx9dVX46WXXkJlZSWv0oa7\n7roLjzzyiOBG+fnPf46vfvWrANJXm/Z6vfjqV7+Khx9+mKWLKkgJBrDcSgz5MoAmIhJcEYyxCgCj\nUSGhuYgIC7UT0ahSYSEgTwi5srIygUizLVIvp7+40vOaNWsS/KlTScjisYkIvb296OzsxIIFCxJu\nMsYYDg/5AJhgZJ6EsQZUM2BGYrsU3JwJrf1+zK9OPkHJV6DmIwaqqqqEpInu7m64XC5oNBqBoNNF\nEuSi3zfbhCxnLO37T4OG+gFEyDgZ+Ek9/p+KOMPQ6XSira0Nb7/9Nq644goEAgG8/fbbMef3qquu\nwne/+11s3rw5Ztzvf//7+OEPfxjTJqfadENDA0/eGVWbFsCmzIecZrPsz4hUBilnjB0BcCcRbUek\nTNOf47qfCuAnjLEgAA7A9UQ0Gl2mSFgIyBNClro55VjIwz/ajPJfCJErGbssxJKdUpWexf2nymXB\nW8h8+rXBYJBV6NRL0qQsBVfYDIvaLbksHSnHbz8+aUIcSXD48GGhuKnYis7lv9dHW1hI+/7Twmee\njHnrOB6poiysVivOOussPPvss3j99dclH3annnoqOjs7Ze27nGrTALB582a8+uqrGVWb5hFxWUxL\nlMXlSdqvkmh7DhHdZKn+ioSFgDwhZCnIJVcxKWdiIft8PuzevTtlpWdxf7mVpIEJ14Ica4kxhv7+\nfoyOjqaNuTaXz0k7npszwayaIGpX2Cy8Z4OU4xEfScBX0bbb7cLfa61WK1h3uSR2BGRfnD5VmJoU\nGU9mPHHIm5KHysMPP4wnnngCa9euxf3334+SkhLZ1aajnzOrNi1Cjrksphx569iTYyHzRDz8o83C\nOnIJmeM49Pb2Ynh4GIsWLcKSJUvSRmcotZDlJoe43W4MDg7C5/OhoaFBcQKMlyZ8705uYtLNzUmX\nhefJWWr5593ZsRJ53Ye6ujosW7YM69evx9KlS2GxWOB0OjEwMIDdu3ejqakJvb29cLvd0xopcLT0\ni1X7RdU5AqmL8vKQU3Fa6WTrd77zHbS3t2Pv3r2oqanBrbfeqmj9rIAxMI02669cRl5YyJm6LOIh\n10LmKz2XlpairKwsxieXCplqX6QSRO/s7MTAwABKS0tRU1OTNgFgX4cPgAFGjS+mPZnrws2ZQJR4\nfl1hM5KVDPu8m2FlXfbJUa/XC/5wj8eDWbNmCWLy7e3t8Hg8MBgMgptDrA0x1ci29oTUWKr9b0Bt\nH45sz1wEVSBSIYisxQitPT/jfXM6nYr1tsVuuWuvvRbnnXceAPnVpqOfM6o2LYCxgoWcL5BLfmIr\nWa1WpyTxQCCAL774Ah0dHVi1ahVmzZo1aXGhTPs7HA7s2rULHMehoaEBRqNx0hbigL9Cdl9X0ABn\nwJh0ebYsZSnwfk5+kmrWrFlYsWIF1q9fj4ULF8JoNGJoaAh79+7F7t270dzcjP7+fni93imzorPp\nspB6CIstY85cBNXYBBnLQSpXRCZZen19EzHszz//vJBVeP755+Opp56C3+9HR0eHUG26pqZGqDZN\nRHjiiSeAiRJHmYEBUKuz/0q3Wcb+lzE2yBjbL2q7izHWwxjbG319VbTsNsZYK2OsmTF2jqh9DWPs\ni+iyXzMZ/qK8sJCloMQXVv6LJzD8o83QP3QbfN++M2F5skrPfr8/I4F6uZAiZI7j0NbWhtHRUSxb\ntky4keSkTkes4wi8oUQrORlcQQMs2vR9PcGpz9LjkUz/gw/14i24cDgsJK4MDg4KCRO8FZ0tgp7K\nSb14MubBk3Eq61gO0gkLXX755XjrrbcwPDyM2tpa3H333Xjrrbewd+9eMMZQX1+P3/3udwDkVZv2\ner34yle+AkwmwgIQXBbTgMcQp2URxa+I6JfiBsbYUkSiL5YhUk/vdcbYwmgJp2NXyyJbqHn+YeCH\nDwrfU1V6VuoTzqS/mJDHxsbQ2NiIGTNmYP369QlRB0onusSk7AhE/rJ6ggaYROTLW8HJSNkZMMKq\n80qOP1WuCyUkqlarUVJSIhSG5SNR7HY7BgYG4PF48Mknnwjhdpmmf0+VyyLYuQ/8NKnaPiwQslzL\nWA5cLldKl8Wf/xwfyQVcc801SfvLqDYNAPiv//qvSV0cbOpSp1MilZaFBC4A8FQ0QaSDMdYKYD1j\nrBPHg5ZFpuCtZM/gKEyQV+k5m5l9qfrzUp0ulyshAYZHugnAj1rDMMq8duNJmQdPyq44SzgVKX/U\nrsOGucoyJtNhMnHITFSSqaamBg6HA6tWrRKs6L6+Pvj9/pjEFTnVPqai4nSwcx/0ruGJ9igZs5D8\n8ynn4eVyuVBeXq58R6cbjMlyMRxFfI8xthnAJwBuJaIxRCJJPhL14TUrgjiWtSySQcnNa6mthOvI\nIIZ/tBkt3/hO2krPSiUyMyHksbExfP755ymlOuWO7Q1rYVTHht15QwYEucSL2hM0ICwxmZcM/a4i\n2PSxs/6eYOTfhJiUc0kzgd+XVOnf4mofPEFL1czLtsuiljli2lgo8ruFzMXQ2iP+47DJBlp6etqx\n5FScnjMnfShkzoExYGpcFpmIC/03gHsAUPT9fgBXZ3vH8oKQk90IfKSFVma+u2/z/wF+fhMA4IT3\nn4ZJ5LpQst1kUOJDDgaDGBsbg9vtlsz6k9qXZGT3UeuEFS9Fyp6gFiatvPjofpcNFr20hebw6xNI\nWdiHdh28wUVYbGuTtZ1UyFamXrJxUqV/2+12HDlyRJDA5Ak6mxZyqbc/5nvAVAK9I0LCYjKWg2QR\nG2K4XK78qxbCY2osZMXiQkQ0wH9mjD0C4KXo1x4AdaKuvGZFRloWeRtlASiTx+zv78cXX3yB5gu+\nDQDwDI7C88tbsro/cn3IAwMD2LVrF4xGI+bPny/Lp5nMQhaTsRTs/sjYvDXLw+HXwx1IUiDWH9su\n7ufw6yXH8wYjz/YmR/alRzOFEmLn07/nzp2LE088EevWrcOsWbPAGEN3dzfa29vR09ODtrY2DA8P\nK0oAEsPe1yV81ruGETBNuMpC5ojfmCfjdNYxAFnqhfmohQxgwmVxlKMspHeF1Yi+fgMA7yx/EcBl\njDE9Y2wOIloWu4ioD4CDMbYhGl2xGTKiTvLCQgakLUQ5scjiSs8nnngiGhsbYamtBBdQFsMsB+nc\nCvFiQB0dHYpE56X6+kMa6DWxxyJlJSeDO6CDWZdoEbv8upSWskaV/DjfbDZh4yJ56dpSSEek75RG\njJtTRz9J2kfOOKnAWESg32q1CjG2HMfBaDRifHxcVvq3GDwRm/zjQhtPxnrHoOCqkGsZ8zhmq4UA\nU+mySLPZRC0LAKczxlYh4rLoBPBtACCiA4yxHQAOAggBuDEaYQEcq1oWyZDKQpaq9MxxHMLhMAw3\n/VKwjj2/vCWt60IukhGyOKxOLAakxOcsRchvN0Vu/mSkHAjF3qi864K3cnnwpBxvGaciZbtPiyJD\nhPR561iMyZJyMvBkzH9ORcrZ9vtqtdqk6d+dnZ3weDzQ6XQxER0ajSbGKgYAj74YZu+I8D3eMgbk\nWcfAMU7IAKCaligLKS2L7Sn6/wzAzyTaFWtZ5I3LQkm2nt1ux8cff4xgMIiGhgaUlZUJY/AEKCbh\n4PbE2GQxlFix8fB6vfj0008xNjaG9evXxyizKSHktNZ3KJEUpYgy3tXAY8ApnQTS70hsdwciN4nd\nl9x68QUZXtmvLDuMRzLLVkzGcseZynRncfr38uXLsX79eixZsgRmsxmjo6PYt28fegfHY9bx6GPD\n2dThyAPPr4+QcUhnRkgn/7wdq+WbAOSUy+JoIW8IWQrxFnIoFEJjYyOam5uxYsUKLFiwIOZijb/J\nDbMiUSjBMXtSUlaa7MGDiHD48GF89tlnqK+vx7JlyxImH5USsvjBwFvHyWD3SosAOf1aeIPKLkqn\nL3l/KVL2BSf2LVNSlmvZpiLp6SiWyqd/m4sqUFM3L6nwPwD49JEwN7/eBqNrUCBi9fx1svdL7qRe\nvlWcBgACA6m1WX/lMvKakMUW8uDgID7++GNYLBasW7dO1qyy6pJ/hypaRDEZKWciqel2u7F79254\nPJ4YCz0eSsLqxNb93/er4Q+p4A/F/nxyreSk+x2IJV6Xf+I7T8rxfbx+Feye1AT/yn4zXvhMvoUm\ndU6UWsf8OEdTMpNHvFUMxBYAMHtHBDKWgpLQQTmTen6/P2XB3ZwFYyCVOuuvXEbeELLUjaVWq+H1\nevHZZ5+ht7cXa9euRV1dnaKb0HDTL6GvSayiIN6GEoU4v9+Pzz//HAsXLsTixYtT/p2crA9ZCv6Q\nJsE65knZ6deK2ib2S0y88YQrRkpLOQUp+4ORy0wuKccT6cGzv5yyf9NXzpI1zmQgl5Bb+hJ97vEl\nsngyNnuGEqzjAx49du3ahS+++AKHDx/G+Ph4yutPjssCyD19aXlgII0m669cRm7vXQoQEcbHxzEy\nMoLly5cLEy1y1xVfoOpNt0MbtY6D2++E9pq7hWVyLWSHw4GDBw8CABoaGmTdvEqsb568/74/9ubz\nh1TQa2JJ3RdkMGhjyVvKUvYG1TBqE7fvDqiRjPudHhWspsj2vP7YY7R71NBrUz9geFL++onyKmCn\nI+PKhuR6wUfTZcETsVk9MZFpDtsTSmOpuYk5D7HfOBgl5JXzl8akf/f396OlpUXwV/MThnyoZDgc\nThmHzz/E85KQGUCqo09RjLH/BXAegEEiWh5t+08A/wogAKANwDeJaDyaYt0IoDm6+kdEdH10nTWY\niLL4G4Cb05W0yhsLWQyn04ldu3YhHA5jxowZisg4mVWqMkxYldxzDwif0/mQOY5DS0sLDh48iGXL\nlkGv18u++JVayO2BlQiGE38yseti3CN9ATu80j/1gF36Znb7EvvzbU5P8svG7lbB7k59WfmDwNO7\nklvLPJEqIWMpKznbURZSY+3rVklaxa6wOYGMveqJY9aEIwk22sBEMQDDrKUAJtK/a2pqsHjxYqxb\ntw4nnHACysrK4Ha70dTUhF27dmH//v0YHR1FIBDIOUH/7GDaXBaPISIEJMY/ACwnopUADgG4TbSs\njYhWRV/Xi9p5caEF0Vf8mAnIGwuZMZZQ6TkYDGJoaEjROLwLIv5vnnrT7cCffg7O50d43A489wBU\nF/0gpcuCFwOqqakRip3yrgU5RKCEkD84POFWCYZV0KoT1xOTsZSV7PEzmPSJD2i3TwWzYWI8V5S8\n49vFcHpUSCVVa3erUGTmBHeFFHhSvnR9rLVMRBi5clNMW/naYgx/kuibTYWpjrLY1y09dnzVFXPY\nHkPGXq0V1rAffq0Z2oBbsI5TIVn6d0tLC8bGxjA4OCiZ/h0IBFLWL8xlEGMg9bSUcEoQFyKiv4u+\nfgTg/0s1RjSRRLG4UN5YyGNjY/joo4+EWnI2my3rIvXqTbcLlnJ43A7uuQckSZOP5mhtbcWqVasw\nZ86cjCpPT6YoarylHD/BB0xEO4itY49/4kHhEFm6UhaxuD1+uccHOOIqPfniDEUpS9kvka/ypw+s\n+NMHE4Tlue5bkvsihpSrYvDGy2K+T5UPeV+3SiBjk0ji1Kz2CNVWeLjDJoGMLcGxCBn7huHXmmFx\nDwpkzFvHcsGnf5tMJsyZMwfr1q3DypUrUVxcDKfTiQMHDuDll1/GFVdcgVAohL1790reK1dffTUq\nKysFvWMgUqDhrLPOwoIFC3DWWWdhbGxMWHbvvfdi/vz5WLRoEV577TWhfc+ePVixYgXmz5+Pm266\nKWuaJsTUWX8hqmUhel2ncLeuRiyxzolqJL/NGDsl2jYTGYgL5Q0h63Q6rFmzBrNnzxZusqmoPK2q\nrIamugbq4sjfzUVd78X0HxkZEaI51q5dm6DMpiRMTu5E3c5PtQilOcxxt0bSGh0YlxAW8isjqWRk\nDSSSshj+AMHukn9j8sRcUi/tFy5fG4nhTeU3FiPbPuR9/SX4sD1JOGHQKEnGVrVL+O7VRojZr430\nC+rMCGily2jJRSgUEv7tabValJeXC+nfZ599NrZs2QLGGO6//35cccUVCetfddVVePXVV2Patm3b\nhjPPPBMtLS0488wzsW3bNgCxFadfffVV3HDDDcK9wVecbmlpQUtLS8KYmYGBVJqsvxDVshC90gkL\nTewRY3cgkpH3p2hTH4BZRLQKwA8APMkYyzjGMG8I2WKxJGg+TEUZJ3bW1YDoLx65HKj47EUEg0Hs\n378fXV1dWLNmTdJojkwm6lJh56cTPt54Uuat5HH3xN+6VC4CMQZGpX3ELglfs9tLcHsniNUTp9zp\ncCdax2LYXcmJOf7nO+Ox1G62ZGRcOjcSWii2krNFyG80meAyJw+7cwZjk2csajfc4bgHNSIHqgtH\nZEz1wUTfcSZIFYes1Woxd+5cLF68GH/84x+xY8eOhD6nnnpqQo3GnTt3YsuWLQCALVu24IUXXhDa\npSpO9/X1CRWnGWPYvHmzsM5kQIyBU2uy/soUjLGrEJns28RPzhGRn4hGop/3IDLhtxAFcaHsrcPO\nuhrMMHFDkcsBzT8fR1lZGU488cSUYkDZdFk8t1uHUDiWUOJJecie/gLzJrGIvRJFQsTEq2SZQ4Yl\nPDiauk86Mi5fUC4QrxhSbcDkCPmNJpPwMukmTrpYNS9MTCBjsbg/T8ZWtQtezggvF+njo8i7XxNZ\nPlnrGIi4UlLFIWei9DYwMICamoiWTnV1NQYGIkJnPT09qKubEDbjK0739PQkVJzu6ZlcOb0IGDim\nzvoroz1h7FwAPwJwPhF5RO0VjEUGZYzNRWTyrj1TcaG8J+SpKnTKvnZ9DCkDQPXnL6e9wbNFyM/t\nlj8RE4w7BbyVLPbhiknZ6ZY+BpcnQphKSdnnj7SJSdkfkB7D7iTYnZFl4p8unozj3RblC6QF1qXI\n2PPTGwAoj7J4+QuL8EoHhy/x93EFDWCYOG6eiC0qJ3xkhI0bhV9jgs07JJCxraZe9v5JQeyykILT\n6ZyU9CY/UT0tYJiWKIuouNCHABYxxo4wxq5BpKSTFcA/ov7i/4l2PxXA54yxvQCeBXA9EY1Gl90A\n4A8AWhGxnI8dcaFk7gGlkEPIvBhQVcUMqIZ6Qb6J2FJ672mwky9NOf5kCfmZXTqoRIcbCjNo1CT6\nDmjUwLgr+fEPjjHo4zjD62cwxkVZeH2AUcLgd3sJZiNLIGC3hwAQLGbpbfOkHL9tf/yEnzMyPiDP\nMhajdG4ZRttHklrGPNJFWTz/6cREok4UkWIQxVKLrWNggojN+glLWarKiidkgE0bcUtIWcZBlR5a\nTlpbWgnSJaxkomNRVVWFvr4+1NTUoK+vT9BfUVJxeubMtPNXaUFgCKuOfqqzEnEhInoOwHNJlh27\n4kLZQjqrWiwGxJ34FXBlVWAVIilUtxP03tNJ11caWxzf95ldkRueizMw410XQ+OxP128lQxIW6mD\no4n9vL4J6zim70jy43C5U8mMcnA4058Du5PD+a9eDku1tF+4pL5EkWUMAJaaiD/U89Mb8PeWufig\nbyX+9IEVT++KfSUj46T76tUk/AY8UpGxhkV+GL0q4tIwhNwIqiITg5O1jnmksmAzUXo7//zz8fjj\njwMAHn/8cVxwwQVCu9yK0/w6k0PuuCyOFo47QtZoNJIWcjIxoNF5J4PTGQBj7Ax6MlJWOqnHR1k8\ns0snkDGPdKQcT8L8d3sa14HXn9jmSeKmcIuI2h1H2i43J7grpOBwJifmUIhw5fubJJfJgXVmIiHz\nZMzjG/+8UvG4hrhMQ7tXA7tX+o+k06+DUyRZatV64QlNkLMnHCFeHxd5D5BeIOOyKrHe+dTB6XSm\nJOTLL78cJ510Epqbm1FbW4vt27dj69at+Mc//oEFCxbg9ddfx9atWwHEVpw+99xzEypOf+tb38L8\n+fMxb948vur0pEAAOKbK+iuXkdcuCx5KJm/UajUCgdj/z263GwcOHIDNZkNDQ0OMT06tVuNQ0WIs\ntDdFnl5eN+B2AmYrQntehmbN12LGysSHHE/EYnCEGPcFAIw5k19UQ2MEXZJ/eU63NHk6o9aux0sw\nGRPPo9tDMJuSWIeuECyWNOI2EhEY8WRsqS6Bq38spq1oViQD09UXa9bzZGydWQZnT0RXOJ6MU0GX\n5qp3RHU5jPqJ39Escl2Y9UGBiK0ivWiejG1aNzxhPWwaN3ycHkUaOwKkh5HcCDEtNJRZtZFM4Ha7\nYybc4iFVcRoA3njjDcl2uRWnswI2PS6L6URuPy5kQGlyhdiHzHEcOjo6UooB8eOrN3wj0hBnKYf2\nvIzPOycqMivxIf/1cxvawyfJ3ncAGLGnd1XEwx+gBDLmrWRnnOuBt5TjreGh4cQN+XyR8+hyheBy\nRZb7/YnHzi93uUK4uvFGXP52YjxsPHgyjoeUZZwMxqoyXHHgFqTSkxG7K+wuJpBxMrj86hirGADc\nfi3UTBQWKGEZA0CIRcjlSP+g7GNIBbkVp/NSCxkRH/J0uCwYY//LGBtkjO0XtZUyxv7BGGuJvpeI\nlhElU+0AACAASURBVN3GGGtljDUzxs4Rta9hjH0RXfZrJsNqzCtCViJSnww8IYv1MBoaGlBcXJyy\nPwD4y2oRNtsQLo/9u7l05J/4uDWEj1tDOBKYm5KQd36qxV8+0eEvn+iglnn2edeF3RU5/vjheVIe\nj0YvBCQMMJ8vcZ+kXBeANPkCgMeT2hXDk3LMvgUntntT980AAGOltM+Y9yXHkzFv/UqRsXVmmaR1\nbKya6HvJPunaiZH4aCa8jKKcD7F1DESI2OVXwyRqt+oDcPtjLTiNKhpvHH03qiO+Yy0ilrTVYspa\nOrecf4b5TMjAtLksHkOi7sRWAG8Q0QIAb0S/gzG2FMBlAJZF1/ktHwaHY1nLIhmUxiIzxjA6Oorx\n8XEsW7Ys7cUqHt+w7BQEvngTqoAHYaMFaq8LYWMkpGiN/TXsKToHek0IrY4ZaHUA/nDkdwlE05r5\nGGK1ihDmotmGKiAsw6AedSCGwDkOEN/Xw2McNJqJmzMQhOC64P24Ph8Hg2FiJZcrelyGxIvU4wnD\nZFILn8XtQOy2efj9HPx+TtKFwZNxOii1jI3VFfD0xlqcYjLmwYfa6bSAN2Fpcri80noeHr8KKkRO\nsM0QECqx2PQh+EI66HQh+MNaGNU+GMkNPzNCjVDk31aWqlbIrRaSrxWnCQxhyg0tCwAXIFJnDwAe\nB/AWgP+Itj9FRH4AHYyxVgDrGWOdOJa1LJJBiYU8Pj4eI5Epx3KId4noVmwEp4uEL/FkzONE77uS\nY+ii8phiMR61ikSf0+x3GqVKe5KJs0AQCZNqUpayVBuQ2iJO5ZURuzAAaTKWspItdVWSZGqaUQlj\ndSJR822mGRPCS1Lra8pK8R37PQntOu3EA8wYlxHt8kpnLQIRMrboRQ8pgYy98IV0sOk88Ie1KNY5\nEeS08LNI2FtVZbkioft0SBeDDORvtRAeHFRZfyEzLYuqaLIHAPQDqIp+ngmgW9SP16w4trUsgOQi\n9eksZF4MqKWlBUuWLIHRaFQ0CRg/vpiU43Gi911sCL4JANCrE/crlUJaPMadsWQcjgu74LhYMg6F\nEl0QYpcBD5+PE6xjcRsAuN2x7VKk7PWE4PWE4BaRrpTveHTUjx+670pol4KlrkqyPYZsRaQcT9Cm\nGZVJyZiHFCnHw+mJ/ccSbx174jSgxfHhvlDEt+wPRwg6yGmhV0VijWdURguZyii5JBdyrO18dllE\nLGR11l+YhJYFAETTprOjnhSHvCJkKaSzkOPFgCwWiyIXRzLC163YCJ+tGmGDFWFD4gUfT8o6TSJh\nJbOS44lYDDEpjzvCCIdjrwsxKTudkfMST8ouV0iYkBNjZCQxUYEn32RwS/iNAeCO0E9wtypSiFdX\nKu0zNlaWwFJXlUDGPLGKyThmuYS1LAUxGfO4NvSbyD6JrGOXh+D0RMg4GcTWMm8di6ur6KK/s14d\nceCbNT7oVX5wUMFm0ggTcNl2WaQr35TPhAxMmYWcCQaikpq8tCbvJ+sBUCfqx2tWHH9aFkBywuTF\ngDo7O2PEgDLxOSebzbYsWCPIJ/ps1VAFJu7osMaAdfQhgERSTua6cLpZWvcEECHlccfEMUiRMk/G\nSuHzTawnJmL+sxQ5j4744HZNhH/dEfqJrG3pa6StYiA5GVtXLJFs11SUQ1udfDwAUBVHCPra0G/g\n8pDwMugnyNkkyvEwGzjBdWExTjzU3AE13AE1ioyRc+GJErMvFHk3a3wIkka4+Ts7O7F7927s27cP\n/f398Pv9ijVYpCDH2vb5fDAapSuK5zoiFrIq668M8SKALdHPWzChS/EigMsYY3rG2BxEJu92Zapl\nkVeTenKjLAYGBtDa2oo5c+agpqYmtlxTBoScCmGVDtBFKj/4bNUwOPrB6UxQh3wxpPweTgYwIRKv\nUU9M8jncLKVPNh52R6RzsnvRESVjrWiSLxjkoNWqYny7Pl8YBkO0eKk7KGoPgST2J5WlDABuVwD3\nVf4WYUfiMl1pCQKjE3HGPBlrSksQGo2NP9ZGU3VDIyMx7ZoK6aw9cbu2ugrB/oEE65gnYx432P4v\nfuv4t6TH4vIQSOLmFV8OPBEXGwPwhdQo0vsQ4DTQkQaGqKuittKG2spI9qzP50NXVxdcLhc+/fRT\noSxTcXGxICivBHIm9bIp0n/UQeBdDEcVUS2L0xHxNR8BcCeAbQB2RHUtugBcAgBEdIAxtgPAQURk\nOW8kIp5gbsBECadXcCxpWSSDOBXa7/ejsbERjDGsW7dO8gLPtlBK0bwV6O/uBD+957NVQ+dLZKST\nVe/hJedpAGI1KOLjiNUqluAr5sETsRTCYYJazQQyjoxNMaQ8OuqHThd7gYtJWQyPJwCTSZfQxoNf\n5vdHtndf5W8njsFmQ9ghwcpRpLKMeTKOh5h0NRXlCA0NJ7QLy9OQMY8bbP8X/+uPzebjU8jNogQZ\ni5ETNKF5S1kb4zuORtNwGli0vhgyFsNgMMBqtcJsNqO2thahUAh2ux12ux1HjhxBMBiExWIRCNpk\nMqW8XtNN6mVLJH66QGDgMrdoM9+utJYFAJyZpP/PAPxMol2xlkXeE7JGo4HP50Nvby86OjqwYMEC\nQQzlaKG6rh793YA27Ic+6ELAYENQY4TZNYCwJlqMUqPDV0oi1vIrYychEIy90VSqiciFVKQsRnzo\n2/h4ECp16gdOIBBOIOWRYS8MxolLgXc/SJEyD/EyMRmngq60BCxedQgTVnI8GWvKyhAaGZEm3STW\nMisqAQPADSdPviDrRK27LSUv4vGx8+Fyc5JZikCiQL/Xr4LWFA0ZjBaJNWpDKNdPVJaeVSUdaiaO\nstBoNCgrK0NZWZmwzOVywW63o729HR6PB0ajUSjJZLPZYqxdOf7oaVVrywLClL/7ngnyipClLqxw\nOIyenh6UlZVh/fr1KSvwThWICJxaB/5Pvz4YqRLhtlTB4IvcpOpQAGFNhIy+UvKhQMpazYSVnIyU\nU1nG8aTMhSmGlHkr2emccEmISZknX583FEPKPHjiFVvHPH5S+1jkg4TfO95KVldFrGJufCyxM5Jb\nxtpFS0CjiXUTWWlFQjsrmpg8VJVHx4tzZ4nJmMeWkhfxG/d5Ce1uL8FiirWUvX4VbFEy9gVVMGjD\nsskYSF0lmndh2Gw21NXVgYjg8/kwPj4eU32aJ+hAIAC9XrqCSeTQ04fF5TIIDGEuf/c/E+QVIYtB\nROju7kZnZyesVmtMTbCp2p7UA8Hv9+PgwYPQaDRYvHgxRvp7BJlFY9AJn6FIIGUxzi7bAwB4qX9t\nUlJ2ujhZvuXx8QDUojCNeFIeG/NDo4m18qQsZZ83hLBElko8Gf9y4ZOxHaw2wJncRcGTMQCoiksS\nSFlVEVlOY7GaFaxcmqRZaYXwzpOymIx5cNZSqMYmLOV4Mg6bJr5/e95bAIA/9m4EkKj77PEB6uiT\nzxfVmy42BVFnifVzpyJjIL1cphiMMRiNRhiNRkEwPhgMCm4OXjh+fHxccHOIQzpdLhfM5vQFVHMW\nVLCQ8wJiMaCVK1fi8OHDitbnZS/l3hiCnkWctdHf34+2tjYsXLgQFRURkiiuqEb/sAM6+OHVWmNI\nWWwl8ziv+hMAwJOta2A0RC4+cQRFOjgcEbIMhzlJUnY5I8tDIS6BlMdGvTGk7HFPEK/eIKpg7Y1Y\n1w+fFJ2TSBEeJobaZgPSzPDzZBwPMRnHEG9pbMgbK60AwomTjZw14jfmSiqhGhtMScYh/QSJXla/\nCwDwVOd6wTr2+ABrNOyc1744oap3YltRP2c6MgYmH4fM180rL4+4bGw2G3Q6Hex2O1pbW+H1emEy\nmaDRaNDY2Cg75K2+vh5WqxVqtRoajQaffPIJRkdHcemll6KzsxP19fXYsWMHSkoiD757770X27dv\nh1qtxq9//Wucc845abagHARMiw95OpFXhExE6OjoQH9/P5YsWYLi4mJ4PJ6MyzjJvTH4/jwhBwIB\nHDx4ECqVKsFNolarMdTXhYqa2dDBD7u2HEXBYTjNEYIx+SOl7NVcIBKhEcVlC/cCAJ44cALUaiaE\nsokt5njwZMwjnpTt476Y72JS5l0VUpYyAPh9IegNGvi8wQkijoJMFjCPK3aFeCu5Mqr34RxPGFtV\nXAJoJSZcS0pBY6OSlnE8EfMI2yLEqxZZwjwZ8wjULoDWPrFcTMbJcNH8L4TPQXVUHCiqPBaiyDsH\nlSIyBrIfh6zVagUXxqxZs0BE8Hq9aGpqwlNPPYVPPvkEp59+Oi666CJ873vfSznem2++KRA9MFHs\ndOvWrdi2bRu2bduG++67L6bYaW9vL7785S/j0KFDWXePEBhC3NEnZMbYIgBifd25AP5/AMWIaFPw\nvrLbiehv0XVuA3ANgDCAm4joNWSAvHr8OJ1OhEKhGDGgqSh0KtWfT5/u7+/H7t27MWPGDKxcuTLB\nH8jrIdfXViIAvUDK2nBk5t2jnxAxUnOJftnNy/Zh87J9UItcDlLPjXgyjofTGdlevAsiFOJiYoaB\nCCmLrWMe9214Aw9tfCfldiRRKRJfsiaKNnGl1UlX5eYulmwPFyVO4vFkHLN+HBmHzBHyDRZFSD6e\njMXWMR9TLgZPxgnbyYCMAWUui3SQCntjjMFkMmH16tX493//d5x//vl49tlnccoppyQZJTmUFjud\nCnDEsv5KByJqJqJV0UrSaxD5T/h8dPGv+GUiMk4lMKQIeWUh86FAYkxVoVMxVCoVfD4fmpubASBp\nSB3flyfv+tpKNB+2w6pyCJZyUK2HR1+MoEqPIu/AxHpcGJyo3teVyz4XPj/2+QrBUhZPzkkhHObg\n8QQT2mIs5WAYGu3EtrzuCHlrdRo8cPbumAQXAOAM/6+9Lw9vozy3P6Nd8iLZTmwndhYvsZPYzkJW\n2mwm7A9NoLSsJaQ0F3pLKC1rbsOlTVsg9BZu2WkhFNoCuSkXml5a8iNJIWyFlDXe7XiJ7XiJbUm2\nds2Mvt8fo288o122ZMeJzvP4sT2SNd/ImjPvvO95z5sGhdsh2xYqSubnlkPpDJ9LBkbJ2JeRDYVN\nnjPms0LnjCkZ88ZpUA4LcrdAMuazcsEEXJgpGVM4ckuhs48WAsORMasKTrNIo2Npt1c8ZAzEph1O\n1GtRc3ppiiMcGIbB+eefD6VSiVtuuQU333xzxGGnq1evFv82cUNN5SAEognXJGIjgFZCyIkIapWQ\nBkMQ5vLFhSlFyOHm6sXjhwzET8herxfHjh1DeXk58vIid4MFrrF8thFNnYCb1wJqgCVqGP0zEIf1\nwmule0IrDyi2LqoB8dsGPv5embifUDpT+4hArgpV6CjM4Y+cpaT8xGVfRNx/LOBzIkzAyDDBpw4e\n3CclZSkZ85nZUI74twdExrxxGkJ1rbAGgaA1I5G9ht3p06Fi5X5v4cg4VKqCkrGPKGD1ZmB2nJYG\nyY6QpYinbfqDDz5AQUEBTp06hQsuuADz58vvVCZLPsclh5CnMQzzqeT330Xws7gGgNTF/zaGYbYA\n+BTAnYQQCwTToI8lz4nJSCgUphQhh8JYPiSxEjLLsmhoaIDb7cb8+fPHrG8+5TIiVz+MYTYTRvUI\nhpENo88sjvMZ1k6H0TMgi5IZEBAEH9sP1zULjweQ0m/eKxfJGAB8nE9Gyjzvw0/PrwMAqLgAz4qA\njI9PY4g5Svbp5Sc8b8gMipK9pjyoHMFKEyBCVJyZDYTwrvWk5UBrl8vdKBkDgDdTeD0FLz9Gr3a0\nScORJuSj0xwDcZExhdkjRN6LZsXfeDHRhByr9SYdSpqbm4srrrgCR48ejXvYaaJBCJOsCHmQELI8\n2pMYhtEA2ATgP/ybngHwCwj1xl8AeATATYlc2JQn5LEgFkI+deoUWlpaUFJSAo1GM67bzLXlwPtN\nwaQMAhgYB5TgMKwVSMLoGZClLigY4hOj5FD48bpG2e8KHy/5OUqaQ6WDknPHc0jwmoSIWOWxR3jO\n6N0El2YMImVvhhD9ql3BhT9vWg40TvmdgydNaKDwpE8XSVlKxhRunREGx2ikLCVjj2aUoBxp0+FR\nGpDhHoxKxsNsJjiiACEMzC4dNpTF46o8iolMWcQaITscDvh8PmRkZMDhcODtt9/G/fffLw473bFj\nR9Cw0+uuuw533HEHenp6xGGniQZB0iLkWHEJgM8JIf0AQL8DAMMwzwF40/9rOIOhuDGlCDlRt0yR\nCJllWTQ2NoLjOCxfvhxarRY2m23cZjCUlDVKDkAmDCrhhB72GWFUDEMJDjxUGNZOF2+LjexgyCgZ\nAAijCIqSw8GnUMtImVNpg6PkwL8JESU7ckuh9sqjZE6bHkTKvCETvCZEHlZCypSMAYDVm0RS9qaN\nWmh6DVkiKXvS5NaanvTpsosOhVsnRK9Ov6pFJbnQSMkYADxKoR4xpBuN7rw+DUAAlqgAH8TGhESQ\nMZDYCBmIPnE6WooNELxfrrhCGFHGcRyuu+46XHzxxVixYgWuuuoq7NmzB3PmzMG+ffsAyIedqlQq\n2bDTRCPObGSicS0k6QqGYWZIPJGvAEBHPP0VwCsMwzwKYCb8BkNj2eGUImQgdO40Xl1xOEIeGBhA\nc3MziouLkZ+fL37Yx5KnDoW15QCgwnuNwtvu9alg0tgw7BNIJF0hkJ0CPvigwLDarzVlheaDSFEy\nAQMmjnxmIClHipJdmbFPSHZnjKYg1J7gFj4pEUvB6k0gIe4MvIaskMfs1grvmcE1WhikZEzhUmcA\n6gxkuAbCkrFsXz6hUMv6p1QkmoyB+AbyjhexTgspLi7GV199FbQ9Jycn7mGnicRkFvUYhkkDcAGA\nWySbf8UwzBIIwXsHfSyKwVBcmHKEHApj0RVLpXIcx6GxsRFer1eMikO9fqKwbj7wXqMORq0bVm+G\nMFmCqGD3pYHzqWBSyW/trepRDa6RG+0MixQl+xRKWQQZGCVHgz1dUEOoueACWKgomQtj2C+FyyCQ\nsTZE56LTMA16d3Bx06nPht4tT2lQMqaPA4Ai4PPvUo/eqg/oZyOTHyVuKRl74R9EGoGMh5x6zM9s\nx6JZ4eV68WCiCHmqeyETABw/OYRMCHEAyAnYdkOYp4c1GIoXZwQhUy1yrD4WKpUKHo8QHQ4ODqKp\nqSmkVSfFWCw7o0XslJT1ag5WbwZYXgGTTiA6KycQTqZKiDBpxAwAwyr5rbuJHS1wjSdKHkkTbm3V\nAcUwVqUPImUpHP7UAPXvkP2tNgNqj00kYgqPzigjZaf/cZcuS0bKlGxdOpNIylIyFp+nykA6O0ra\nUjJ2Q0idjCiF19Ji9C4gGhkPOg3gCYM56nooFKdXC/KZPnEagNA6PbkpiwnHlCPkUCmLsTR6sCyL\nuro6uN1uLFu2DDpdsCxL+nyvN3IjhhQ0xRGJkAcHB6EcaoIr5+vw8EqYdB5Y3Wkw6RzgfMK/ZYTL\nAE8YZKnDa3ul0TOAIEJWgg/4XS6pUAU0p7BKbRApB4LVpMGrlhOUR50ekpQDyVh8vs4IXhH6AkqJ\nWPY6OhOYECTkVAmEY1cLDSjS46VkLP7u08LtJ2E6ZsmgHD1Wq8cAlvfL2nwMeMIg3/cFTp4chFqt\nhs1mEz0jok3qSDZinTg9VQecAv7W6RQhTz3E263ncDjQ3d2N8vJyzJw5M+oHm3bfxYpIOWee59HU\n1ASXy4Vly5ahp6cNtcPlALQiKQNAumaUKCysoBIwqsMrGigCo2QeShlJ8VDJSJlTaIJIORDSKNmm\nH70AaDl50U9KyjbdKBHr2eBcsk2TDQMXvH0gvQhpfHBKw6H059k5IRKmRCyFixiQzgivGYqMxXX6\nydjLq+DlhVOA9SmDyFjIF8/H8ePHkZmZKU4sb29vBwCxZdlkMsVtLj9exKLWmOoDTgkJ9gs/03FG\nEHKsETLHcWhubsbIyAimTZsWs3ZyPK3WUgwPD6Ourg6FhYVYsGCBeCEo1X6F457FgD96S1OzsHu1\n4HwKmHQu0fFqmE0X23WzNJE74saDwCh5RJ0DjTpy1EwhJWIKarIECERM4VRlyEjZrhKiXIfSKJIy\nJWLpcxQI/l+4iJAXtpMMuDgdMiQXr3BkTBFIxkN2NS6rsoDnGTH9pNFoYDKZRBMpjuMwMjICq9WK\n7u5ucByHzMxMkaB1Ol3QhT6RhvFnAyEDkxchMwzTAcFYlgfAEUKWMwyTDcHjYi6Eot5V/saQhHlZ\nTDlCjnWMUyDMZjMaGxsxe/ZsFBQUxOUQN94I2efzoa2tDUNDQ1i8eDHS0tJACAHP88jLy8PJkyeR\nb/8Qffg69BofLC4tsvQCAVrdQqSXoZUrICxe+YkmJejxRsnDTDY06gC/C78vhxQelUGMkkdUo0Sr\nI8F2cC51BjgmdIqCErEUgUQsbvcJxJuhEIicErG4H05IPdlY4VZdrRwtZEYj40GbGoQw2LR4GD4f\nxP8hnYHH8zwYhoFCoYBKpUJ2djays/2ucj4fbDYbrFYrmpub4fF4kJaWJqY40tLSJrQpBACcTmeQ\n1cBUgqCymNSpJ9WEkEHJ7zsAHCaE7GYYZof/93sDvCxmAjjEMEzZWJQWU46QQyFSBMvzPJqbm+Fw\nOLB06VLo9Xo4HI6ERLzhICVwh8OB2tpaTJs2DStWrIBCoRDJmBACrVaLkpISAIDP58TeT9Kh0xDQ\naDlDKxCKxSWcWFl6Z0hLwkCCBgAFM7pmJROQW2aCj1+tGCVpL9FAw0QnZSkRU7gZg4yUbRDIVR/C\nt7NfMRNpIbbbeX/qRjmq6KBkDAA2XwacrA6ZmtHHKRmLv/NquPwk7GKFj7pBPUrQp+x6sYrP8UJn\n2JXLnQCEv+F5Hg0NDaJxPP0M0P+dUqkUCVpqHD9nzhwQQuBwOGC1WtHR0QGn0wmNRgOv14vh4WFk\nZGSMi5xjnac3pQ3qSdB8gcnGZgiz9gDgJQDvArgXZ6uXRTiEi5AtFgsaGhowa9YszJ8/X4yux5KC\niDdC5nkenZ2d6O7uRkVFBYxGo0jEPp9PPJED/+66c5145Z8C8Ri0BDaPGhwPZBlYcD6FSMyEACZ9\n7N11PGFkpMwTZUhSliIcKbt8cuKjM+SkcDMGsEQeEbuIAXpGIF+bbzQH7PAZkKYQtlMipqC/MwEX\nFCcrrGHEm4ZMjSMkGYs/+8nYyymFL5GER8nYMgJ8r3r0wuByuVBTU4MZM2agsLBQ/Oz4fD7ZFwDx\ns6FQKMT/K8MwSE9PR3p6OgoLC0EIwcjICBobG9HT0wObzQa1Wi1G0EajMS7yjEbIU32eHkWSVBax\neFkQCJEuD+C3/sfzJI0hfQBo183Z62URTpZGZWyA8GFtaWmBzWbDkiVLxu0QF29jCCEEDQ0NMJlM\nWLVqFZRKpSwqjmTUQgjB+tkt6OrqQg+q4WUZmNJ9sDgFgsnQja7b6hJIyAcG2fpgaZqPKGRRcjRS\nZn0qWZQsxTA7GoFrlHKSdvu0MlIeYQWy1auCLxguYgAXYpKww2cACWGNaPcTb4Zm9PgoGVP0OYSU\nh1HrDknEgEDGAEKScYnqYzh1Tnz1lUE0YD958iQWLlwIo1GeOqHRMIXP5xNGeElImn62pNEzwzBQ\nq9VIS0vDggULAAipkOHhYQwODqKtrQ0Mw4g5aKPRGFHGGWsL9lSep0cIRF/wBCMWL4s1hJCTDMPk\nAjjIMIzMm4AQQpjAKCEBmHKEHAoqlQoOh1/Da7Wivr4ehYWFKC8vT4iuOJ7n9/b2YmBgAEVFRSgq\nKpJFxUBk1yyv14uGhgao1WosX74cKpULvz9iALWtTtf7YHMrwfsYmAwcaBCkAIHZNaoqUPjzxya9\nO4iUY8GwVy6V0gSQtJfXhCRlLy9XGrg4nYyUR7yj0a9BLSdrm1dYf7pku11CvPTxwNSLkx0lrT67\nsG6j3hOVjAetwgn/vWoXgMViiqGlpQUjIyNQq9Voa2tDVlYWTCZT0IBRCrqNkmMgQdPcMyEELMvK\nNOparRa5ubmiaQ+dQm21WtHZ2Qme54MKhRTRCHm8k0lOBxAAHDc5kT4h5KT/+ymGYd6AkILop+3T\nDMPMAEANU85OL4twoLripqYmDA8Ph4yKpaB53HhePxohsyyL+vp6MAyDmTNnioU7emJGsy8cHBwU\nzYykrnLfXe/E748YoNcCFpsCWRkCuVqdwr/OqBfIUgECn9/3wgcGChAxgg7crVIhP3a1Qn5saqWc\nwL0+VRApUwxLSFavDO4EdHE6sBEGVVKipbCzofXgDu8o2Wdq/Wb/bOhouH9k9DU8/vl3Hv/SOA7g\neCIhYwEsy6K5uRmZmZlYsmQJGIaBy+WCxWJBT08PGhsbxRRDVlZW2BRDOIJmWRadnZ3IyMiQFQjp\nFy0USqdQ8zwvFgppJ2l6ejpMJhM8Hk9ELXSsbdOnNQiZlKKev21aQQix+X++EMDPIXhW3Ahgt//7\nfv+fnN1eFoFwuVzo7e1FaWkpysrKEn6bFq2oR7v9SkpKkJ+fj9bWVnAcJ+a1I5ExTa+4XC6cc845\nIacIf3e9kNv83SE9aLScmeafSO1SiXm27DQuiJQBIRKU7p73MTJSZn1KGSmzvCKIlKWwevQA9NCq\n5CTt4tUyUh72CMQoLaRR9NmNSNME65/tHoF407Wjj0nJGAB6bQYYdaOvKSVjNzt6oOHI+KYN8vTO\n8PAwGhoaUFJSIsraAIgDRmfOnCm8jscDq9UqOgEqFAqYTCbxK1SKQaFQwOl0oqamBrNmzUJ+vtB+\nTe+aIuWhlUql+NqAQO60UDgwMACWZcUBpyaTCenp6eLnzGazTe0Bp/A3hiQnZRENeQDe8L+XKgCv\nEEIOMAzzLwD7GIb5HoATAK4CUl4WInw+H44fP46hoSGxup0M0NvMQAQ2eeh0OhBCoFarceLECbjd\nbuTk5IQ9MUZGRtDQ0ICZM2eGTa9IcfP5LvzukB5elgBQiBpNUwYB7wPMDvm/MzttlDADSTkapKRs\ncWkBaKFVyd8DD6cKSco0RUDhZNUyUh7xCBcdh1cjI2VKxtKfA9fs8AqvPewWyE+jHD1hKRlTiSoC\nYAAAIABJREFUIhZ+Hq3Um608fvSN0Vw3IQQnT55ET08PFi1aFFUiptVqkZeXJzqoUUKkSgqfzyeS\nY1ZWFjQajXjnE5iPDsxD07RGNCVHRkaGGGXr9XpkZGTAarWiq6sLdrsdGo0GCoUCX3zxxbgJ+cCB\nA7j99tvB8zy2bduGHTt2jOv14oXQGDLxhEwIaQOwOMT2IQgTREL9zdntZTE8PIz6+nrMmDEDS5Ys\nQX19fdL2FYooQzV50Hxxfn4+jEYjLBYL2tra4HA4YDAYkJ2djaysLBgMBnR2duLUqVOorKyM68S5\n+XwXHntTi2GbcOJmpClgtTHgfQSmgOa1QIIGAIXkUFRK6YddBVXAHXhglOzhFGFJedglj+z1ajlR\nO1l1yIGVDq8GobJHds/oYmghk5IxALg8wmu5ALglgbbHSwAQePzbOI6A54Vb3zs3jz6R53k0Ngp1\nmmXLlo1JHqZWqzF9+nQxquZ5XiTorq4uOJ3CnU1RUVHIOx+KUIXCaEoOnuehUqlgMBhgMBhkUXx9\nfT0OHDiAo0ePYt26dbj++utxyy23BO84Aniex6233oqDBw+isLAQK1aswKZNm7Bw4cK4XmdcIJMW\nIU8aphwhE0LQ0tICs9mMqqoqpKeny9ID8b5WvOmNSE0eNFesVCpFydOsWbPEgpHFYkFLSwssFgt0\nOh0KCwvHtI7bLxOivF+/IUSRPE9gylTCahv98AaScyhwPCMjZY6HjJRDpS6kpEzz2Dp1MJm5WJVI\nyjbP6K28Xj16JyclXQBI0/Aht/dZ1cgwjK6DkjEQiowRRMbSqBgQGiZqa2sxc+ZMFBQUJNRnOycn\nB0ajUczh5ubmisGDx+NBRkaGGEEbDIawY8kiKTl4nofdbofJZALHcbLna7VaLF26FLfccguKioqw\na9cucRZePDh69ChKS0tRXFwMALjmmmuwf//+CSVkQgg49uwys5hyhMyyLNRqNVauXDlmXbH0b+Ix\nibHb7aitrcX06dNDNnmEyxVTTarNZoPH48GSJUug0WhgNptx/PhxOJ1OpKenIysrK+KJGoi7rhCY\n58F9o8eQkS6cmNYAmwhK0D4ij5LjIWWrP+LWquUniZtloFPLI5kRlwIjLg0MWvl2F6sMqy3tH1Yj\nTSd/0OEWjsfmVMDpF2Bk+m8oKBlTIhZ+Hq3MX17VgqysLBCSJr6fAwMDaG1txYIFC4IkbYkAzRfP\nnj1bHBCanZ0tKm5okY7+32k3X1ZWliwHLIW0UMjzPGpra2EwGGAymcTPH/0MUnKmTm/SbsJ4cPLk\nScyaNSocKCwsxCeffDLGd2Xs4M8yd6EpR8harRZz586VbRvPXL1YCJkQIg46DdXkASCixIhOIWEY\nxi9nE/aZnp6O2bNngxACu90uRtAulwvp6eliikOv10c8xv/4NouH/kxv71RibtmYObqmQIJWSlkZ\nDKTLVwe9JQrZNg+rCEnKXi54jU4PI5LyiHN0J1LitbtGt1MCDnodiUrulFlCwB7hdVh/JOX1+sDz\nBHdsHoHFAjFlpNfrReJaunRpxBTCWDE4OIjjx49j4cKFIT0kGIZBZmYmMjMzxf+7tJvPbrdDp9OJ\nF+bAbj7arFJYWCimKAB5BE0/l++8846YMpmqSKIO+bTFlCPkiRjjJIXb7UZtbS18Ph9WrlwJlUoV\nc5MHIHhoNDU1obi4OOw4HYZhxGKNlKDNZjOam5vhcrmQkZGBrKwsZGdnQ68flYp5PB7U1dXhW0vS\nUVpait2vMfDxBGlpKgyPjJKelJwBwSNATsqjYLlgUg7cRgtn7gChhF4bfAL1mxXQB6jZHG4F7E6C\nNL18DQ5X8DQYKdye0GTs9Qo/77yG/k9HU0YejwfHjh2DSqWCRqPB559/HpH44gUhBO3t7bBYLDjn\nnHNidn4L1c3ndrthsVjQ3d0t6+ZTqVRiswpVXVAERtC/+tWv0N7ejueee27MxzRRg0wjQUhZJG4w\nRKxgGGYWgD9AUFsQCJ18jzEM8zMA/waAmpD/hBDyd//fnJ3mQolCLITc29uLtrY2zJ8/Hy0tLQAg\ni4ojkTFVgNjtdixdujSi33IgpARNfRFsNptokOR2u5GZmQmVSoXBwUGUl5dj2jTBZe0nVwl521++\nStdBkJGhlpGzFNkmqpeFLEoORcoDFgJNgLJLq5Efv8vDQK8lsDkCtrshkrLdOUqqDpdAyoFEDAAO\np1CgAwCVSiJnC4iKzUNuPPRvoYtyNH87b9488T0CIGqMKfFpNBqRoMM1gYQCx3Goq6uDXq/H0qVL\nx0XsDMOElNq1tbWhu7sbGo0GLS0tMi20VGrncDhwyy23YPbs2XjrrbfG5dm8YsUKtLS0oL29HQUF\nBdi7dy9eeeWVMb/emDB5ETIH4E5CyOcMw2QA+IxhmIP+x/6bEPJr6ZPPenOhUCb1QHzFsWiDTmmT\nx8qVK6FWq0VtMZUhRRsuWV9fj7y8PMybN2/cUb30Vnfu3LngOA719fWwWq3Q6XRoaWlBf3+/GEHr\ndDrcd61wbPf+Tip9I8jMlEdvZisvI2KlUr5WKRECgJeFjJQ9XiKSss1B/N8BfYiMwKkhHwyBEbGT\n+IkXSDMw4jYKt3v0f0SJ2OsVtrFe3k/EwWRMCEF3dzd6e3uxZMkS2V0FEKwxppFpPE0g1Dhqzpw5\nor44kfD5fGhvbwfHcVizZo3YAGW1WmE2m9HW1gYAaGpqgs1mw8svv4wf/OAH2LZt27g/cyqVCk8+\n+SQuuugi8DyPm266CRUVFYk4rJhBQOCbhJEhfr+KXv/PNoZhGhDZmyJlLhSIeIt04Qg5sMkDEKLi\n9PR0fPHFFyLpmUymoJOUEIKuri709vaioqIiKZ1SlOzz8/NRVVUlaqRpBF1fXw+v14vMzExkZWVh\n15Ys6HQq3PMsi/QMLUZGgpsxMjM1IinzPJGRMseRkKTsdstPFJ1OHhm6/KmFwNZXp2u0WSUQA4Oj\nFw/pRYISMSCQcSQiFo5h1KUtVkmbTqfDjBkzxEJcqCYQGkFTSWNraysqKiqSMiaJZVnU1NQgKytL\nplEPJbVraGgQo9ff/va3KC0tRXV19bjXcOmll+LSSy8d9+uMFYQgWSmLWMyFAAAMw8wFsBTAJwC+\nDuA2hmG2APgUQhRtwdlsLgREHuM0VkIO1+RBpUZlZWXgOA4Wi0Us3iiVSpGgtVotGhsbkZ6eLiow\nEgnaxEDziFISkFo/FhUVwefzYWRkBBaLBXV1dWBZFlvXZSI7OxuP/kVoyyU+gvRMIYylJK1UBrjP\nSYhZrQqOuNRqifzMT9ChvAcoWTsc0U8ul3OUlD0eiR2olwfH8njsR3qEI2JhH0LUWlhYOK6cZ2AT\niNfrhdVqxeDgIOrq6kS9udvthk6ni3meYyygap7AzsFAEELwP//zP3jhhRfwv//7vygqKoLH4xmT\nBPS0BCHgkxMhx2IuBIZh0gH8L4AfEUJGGIZ5BsAvIOTRfgHgEQA3JXJhU5KQQ4FacMZaPVepVCIh\n0yaPWbNmBTV5SAt3arVaZgbj9XphsVjEgk5aWhrUajVGRkbiykNGAzUd0mg0WL58edSIT9rSKyVo\ns9mMq1ecBMuy+MN7FXDYPUhLl79fxqzRW3sfT0RSph1TUmI2mz3QaILXotONbnM4WPh9n6DTjX7c\npMQLAE6nFwaDkE4JJGKb1Ylf36aIGomeOnUKbW1tYVUO44FGo0F2djZ6enowY8YMzJ07V2wCaW9v\nByFETHGMZ6QTleVVVlZGvMPieR67du1Cc3Mz/vGPf4jvjVarTYqCZDIgtE5PjuyNYRg1BDJ+mRDy\nOgAQQvoljz8H4E3/rylzoUCMxcGN4zixySRck0ckUlUoFBgcHIRKpcK6devA8zwsFgtOnjyJxsZG\naLVaMYLOyMgYU16PqjQCTYfigZSgASE3WV4+DLPZDIvFghf+UR78N6rR4w6MnFWSx2g+lxKzw+6F\nwz89SaeXf7yGBgUZFiVep1OePrGYhcdZLweO5fHbnxjhdnthNjvQ2WmBzWYLqY7w+XxobW2F3W7H\nsmXLEhqtUjgcDtTU1GDu3LliKkuaOqBObRaLBSdOnADP8zKCjkaShBB0dHTAYrFEPQabzYZt27Zh\nwYIFeOONN6a0CX1ETJ7KggGwB0ADIeRRyfYZEj/kKwDU+n8+e82FgLGPcZKC4zh0dXWJbaGxNnlQ\nWCwWNDU1iQUdGkFL85AulwtmsxmdnZ2w2WzQ6/WiUD8tLS3i69OOwOHh4bhVGtFA86EajQZDQ0O4\n92oz0tPTYbVagwhaSsaKQGKWdOjRuR3SiNnsJ2CtJDJ2u1i4XaO+Fl638DPrYbHnZ8Hz+PR6PQoK\nCsT0g1QdMTIyAo1GA5fLhezsbCxatCgp5EQj70j54lBObZSgu7u7wbKsmNfPysoKstKsr6+HRqPB\nkiVLIgYBJ06cwA033IDbbrsNW7ZsmdJ+x9FACMBzE0/IEHLFNwCoYRjmS/+2nwC4lmGYJRCC9w4A\ntwjrTJy5EBPnZIHTQqXNcVxQNEzlQJFyboAQiXR2duLEiRPIyclBRUWFLFcMRJeztbW1wWq1oqKi\nIqh6H2m/TqcTFosFZrMZDocDaWlpMn8Luk+n04m6ujpMmzYNc+fOTcpJ19PTg87OzpC395RMaAT9\n9P/NhVLSvheJmCnUGoGEPS55FEx/Z70s/rR7ZtDfxQOr1Sq+T3ToKL0riVe+FgqEELS2tsJms6Gy\nsnJckbc0r2+xWODxeJCZmYn09HT09vaisLBQbKUPhw8//BB33nknnnnmGXz9618f81omEOP64Gbn\nVZGN1/01UWsR8dpvij+LJYc8GThjCLmtrQ16vV6MTkOBNnnQNuXh4WHMmzcvZs9ih8OBuro65Obm\nYs6cOeMiStqlRUnP6XSKt+CU7JPR2stxnGiqM3/+/JiKoNQ0h5IJdTWjahO1Wo1r7+yS/c3t3z4p\nu71PJKiapa+vD1VVVbKLIo2gLRbLuPTFLMuitrYWGRkZKCkpSfhF0efzoaenB62trWIXYTifC0II\n/vjHP+LFF1/Evn37MHv27ISuJYkY15uWlVtFNlz1eqLWIuIvT5WdtoR8RqUsIuWQpU0eOTk5GBoa\nitmzmGpae3p6sGDBgoQUjKRdWrNnzxZlTl6vFzqdDvX19cjMzBQj6ESkLGw2G+rq6jB79mxZ6200\nUNMcejtO86Vmsxnt7e0AgF23CETidDrR19eHysqqpPjx0tt7pVKJ5cuXBxFsLPpi+p6GI2iqcigq\nKgrbXTle9PX1oaenBytXroRer5e1z1OfC4/Hg48//hgnTpyAzWbDoUOHpr7pfByYLB3yZGJKEnIo\nUNF8ILxeL+rr66FQKMQmDzrteXBwEF6vV8zrhko/UDtDvV4fk8JhLKATIebMmSNG+KGka0ajUSST\neKr40gtKVdX4iTIwX8pxHMxmM1paWuD1eqHX69HT0yMWtMbTMSYFlbTNmjUr5gtKoL44FEFL9cUD\nAwNob2+PqnIYK3w+H1paWuDxeGQa6VDt801NTfjoo4/Q398PhmHw85//HL/61a8SvqbTFULrdPA5\nfSbjjCFklUoFt1s+py1Ukwct3Gm1Wpx77rli2oC2JEtJb2RkBMePHw9qu00UqAcCtfKUXhBCSddo\nVNrV1SWr4mdlZYXNb9Kuw1glc2OB2+1Ge3s75s6di5kzZ4p6bdpNxjCMLMUxljX09/ejvb193I0Y\noRpALBYLent7cezYMRBCUFBQAJZlxdl3iQK9CzKZTFEn27S3t2Pbtm24++67cc011wAQCslnFQjg\n4yZN9nYxgMcgiN6fJ4Tsnoj9TklCDje4lKYfpE0ey5cvh1arDZpvR080qWcEjUoHBwfR3NwMnueR\nl5cHn88n2n4mCm63G3V1dTAajVi2bFnUE1/aKUaPkeZ1T5w4IepgKempVCox8k7mrTctDkojSpVK\nJZOE0XZf2lAjPZZoBE09QZxOZ1IkbVqtFjk5Oejt7UVBQQEKCwthtVrR19eHpqYmqFQqWQQ91gsa\nlc0VFxdHlS8eOXIE9957L55//nmsXLlS3D4WG82pDBo8TTQYhlECeArABRC67v7FMMxfCSHJm4Lh\nx5Qk5FCgsrdYmzxCgU5jGBwcRElJCfLy8sSotKOjAwAitk7Hiv7+fjGfTQk2XoTK69KotLW1FV6v\nF4SQpEX3dOKGz+eTWYqGQmC7L8uyso5HStDZ2dky0vN4PKitrUV2djYWL16cFLUJzRdLiTI/P1+8\no6It1P39/Whubh4TQdPjjJYGIYRgz5492Lt3L956660Jd1c77UAI+MlJWawEcNw/ygkMw+yF4FeR\nIuRYwTAMLBYL7Ha7OHU6Hs9in8+Hjo4ODA0NyearSUkvMNJTKpVi/jmWCj6N3DmOw/LlyxMa7dGo\n1Gg0wm63w2g0IisrCxaLBZ2dnbI2b6PROK5bcZrLpfrgeIkyXMcj9Y1QKpXQ6/WwWCwoLy+PKmUc\nK2gaJFJePVQLdeBa6Z1JIEETQnDixAkMDQ1FteVkWRb33nsvrFYrDh06FHW+39mAJOqQo3lZFACQ\nyoa6AaxKxkICMSUJOZAA7HY76urqQAgRJ4nE0+RBdb85OTkR0weBkR7NP/b09KChoQE6nU4k6MDp\nDyMjI6ivrxcLUsmI9mhXnzQqlhKJ2WxGb28vmpqaRDkY7SKMlaD7+vrQ0dGRUFMdjUYjkh7Nq/f2\n9sJkMqG1tRWdnZ2yqHS8eV1CCI4fPw6HwxE1uo+0ViCYoGm0bzQa0dfXB7VaHdWW02w2Y+vWrVi/\nfj2efvrphPuguN1urFu3TvS5+Na3voVdu3YJrfRXX42Ojg7MnTsX+/btG/MdW3JA4EtOyiImL4vJ\nwJTUIdMJHrTJo6enB2VlZWhra8Py5ctjbvIghKCnpwddXV0JGelDO/PMZjPsdrs42NTtdsNsNsc9\n0DRWSLv6KioqYpLIUbWB2WzGyMhIxIsJ3UdTUxNYlsXChQsTppyQgtqKqtVqlJeXi8REL3wWiwXD\nw8OiMiLWOxMpaGHNaDSiuLg44RdGr9eLU6dOobW1FQqFQtbqHSrN1dTUhJtuugk7d+7ElVdemZQL\nNdW8p6eng2VZrFmzBo899hhef/11ZGdnY8eOHdi9ezcsFgsefvjhRO56XAfDMMwBAInPtwmEfHGE\n/Z4L4GeEkIv8v/8HABBCHkrCWuT7nqqEPDIyIjZ5UM/hTz75BCtXroypyYMa9qjVapSVlSWcYAgh\nsFqtqK+vF8e5S2ecJaoVmja70LltYz2hw11MsrKywDAM6urqkJ+fj1mzZiWFNGgaRDqLLhwoQdOL\nSazNH1SHHc1FbTyghvi0PkDz5RaLBVarFQqFApmZmaivr4dKpcKDDz6I3//+9zjnnHOSsp5AOJ1O\nrFmzBs888wy2bNmCd999FzNmzEBvby82bNiApqamRO5uSvZ1MwyjAtAMYCMEk6B/AbiOEFKX9H1P\nRULmOA4ffPABysrKZN4BH3zwAfLy8qIW3QYHB9HS0jIuw55oGBgYwPHjx8U1Sqd+mM1meL3eMeuK\nA/cxnuJgKNA2b5risNlsMJlM4nsbbcZfvBivpI1G+xaLRUbQ0nRMX18fTpw4kbS7FEBoPurq6grq\nHpSCZVl0d3fjzjvvxGeffYaioiJccskl2LVrV1LWRMHzPJYtW4bjx4/j1ltvxcMPPwyTyQSr1QpA\n+J9nZWWJvycIU5KQAYBhmEsB/AaC7O0FQsgDE7LfqUjIgBDRUUMgmqLgOE6cpmC1WqFWq5GTkyOe\nmFSU73K5sHDhwqTYFPI8L8q0KioqwhKtVFdsNpvh8/nEKC8rKytixE6PI9o+xoPA90oaldIZf9Ih\nrOPdR0VFRcKKnDRFRAma53kolUqUl5eLEX8iQQiRHUek/53X68Vdd90Fj8eD5557TrzDWbNmTULX\nFA5WqxVXXHEFnnjiCaxZs0ZGwLQInEBMWUKeLExZQvZ6vfD5fBELd/TEpATNsixycnJQUlIi8wpI\nFGhxccaMGXHf2gdeTBiGkUns6G240+lEbW0t8vLyMHv27KSkD1wuF2prazF9+vSQnh2B0b7H4xFV\nHdSsPxo8Hg9qamowbdq0cfuChIPX60VNTQ0yMjKQlpYGq9Uqy5dTC8/x7Jt6XmRmZkbNSQ8ODmLr\n1q248MILcc899yS8eBcrfv7zn8NgMOC5555LpSxOM0xJQna73ejt7cW0adNE7XA4UOlRf38/iouL\nxVypy+USvSJiJZFI+wg3zWOsYFlWjPJotK9Wq2Gz2VBRURE0eThRoGmQBQsWxLwPqQG+xWKJ2uZt\nsVjQ2NiI8vLypDU7UFVLqHwx/QxQAyK9Xi9eTEIVNMOBNnvE0nhTX1+Pbdu2YdeuXdi8efOYj2ss\nGBgYEGcEulwuXHjhhbj33ntx5MgR5OTkiEU9s9mc6NbsFCHHiSlJyB0dHbjxxhtht9vxta99DdXV\n1fj6178eRIQulwv19fViRV0akUjn0JnNZnAcJ+pJo6UMpJBO8ygrK0tKazLP86irq4PL5YLBYIDd\nbo/LWzkWSE3ex5sGkXoB03QMbfOm73lVVVVCPZ6l6O3tRWdnJ6qqqqLqeQkhokOctKBJU0fhCJpq\n0WPJex84cAC7du3CH//4RyxatGhcxzYWHDt2DDfeeKOoyb/qqqtw//33Y2hoCFdddRU6OzsxZ84c\n7Nu3L9EXyBQhx4kpScgUNpsN77//Pg4ePIgPPvgAOp0OGzZswIYNG/DVV1+hoKAAa9asiangRVuR\naeTEMIxIeOG0r4mY5hENNA0ibcKgJEIvJlJv5ezs7LibCmgeMycnJyn+yzzPY2hoCC0tLeA4ToxI\n6VeiLmI0J+12u6PmcsNB6ltNG42kihODwYCuri4MDAxg0aJFES9cPp8PTz75JA4cOIB9+/Yl/DPS\n1dWFLVu2iOZDN998M26//fbTSV+cIuQ4MaUJWQpCCE6dOoX9+/dj9+7dUKvVKC0txYYNG1BdXY35\n8+fHrVelhDc8PCzOVMvJyYHBYEBbWxtGRkZi1v2O5XhoGiTaBGtq3UjX63a7Y07HDA0Nobm5Oanp\nA9qeTN3spKPsqRQsVOt0PKD54uzs7IReVKSKE/qlVqsxe/bsiHcnHo8Ht99+O1QqFZ555pmkFJB7\ne3vR29uLc845BzabDcuWLcNf/vIXvPjii8nWF8eKFCHHiTOGkCl++9vfIjs7G1deeSWOHz+OgwcP\n4vDhw2hubsaiRYuwYcMGnHfeeZgxY0ZcJy2NSE+dOgWz2QyDwYBZs2YhJydnzCqDcGBZFg0NDVCp\nVCgvL4+boKTWnWazGSzLytIx1IKUTj6prKxM2mBM2tkXyceBdrtJ8+XxNH5Q7W+yfDsAgWCPHTsm\nSv/oe+t0OsW7E5PJBIPBgMHBQWzZsgWXX345fvSjH01Y8W7z5s3Yvn07tm/fnuxiXaxIEXKcOOMI\nORw4jsPnn38uErTFYsHq1atRXV2NtWvXIjMzM2pxkOYm6aQNs9mMoaEhmW1ndnb2uPKvw8PDaGho\nSOi0jcCRTDzPiyS9YMGCpHTdjSd9EKrxQ2oqL/0/0U7LWPLFYwUl/FB3EdLJL42Njfjxj38MlmXx\nzW9+Ez/84Q9RWlqalDUFoqOjA+vWrROba5KsL44VKUKOE2cNIQfC4XDgww8/xMGDB/Hee+9BqVRi\n3bp1qK6uxsqVK2URI8uyaGxshEKhQHl5eRC50Ih0aGhIJDwa4cWaI6Vt4P39/aisrEwauVDiyM/P\nF1MHiTQeAkZz0omStEnli1QVYTKZMDIyAkIIKisrkzZ5OdYCISEEb775Jnbv3o377rsP3d3d6Ozs\nxCOPPJKUdUlht9uxfv167Ny5E9/85jdlDR9AUvTFsSJFyHHirCVkKQghGBoawuHDh3H48GF88skn\nyM/Px4YNG2A0GnH8+HFs37495oiV53kxwrNYLCLh5eTkhLwF93q9qKurg8FgwLx585Jyi0vHzA8N\nDaGyslKW96bGQ4ERKW2oiYdQaaEzWTlpQohosUrfJ1p0owXNROSPpQZElZWVURt1Hn30Ubz33nvY\nu3dv0tImocCyLC677DJcdNFFuOOOOwAA5eXlqZTFFEWKkEOAEILm5mbcfffd+PTTTzFz5kwUFxej\nuroa5513HgoLC+M66aWENzw8DJ1OJ3YQer1eNDU1obS0NGn+CpTw09PTUVJSEpXwpTIwm80WE+FR\nvffg4GAQ4ScSNH0gbUl3OByynC4dYjvWNm+O48SGkmgDTl0uF7Zv3w6TyYTHHnssKV2T4UAIwY03\n3ojs7Gz85je/EbfffffdydYXx4oUIceJFCGHQV9fH1544QXcc889YBgGX331lZh/PnXqFFauXIkN\nGzZg/fr1MJlMcZ30tGrf2dkJt9uN7OxsTJ8+Pexcv/HAarWioaFhzIQfqDKg07GlJkl0TJROp0ta\nhA9AVJ1E8ooIpzihBB3tQuF0OlFTU4O5c+dGbfbo6+vDDTfcgGuvvRa33nprUroNI+GDDz7A2rVr\nUVVVJb7nDz74IFatWpVsfXGsSBFynEgR8hjgdrvx0Ucf4eDBgzhy5Ah8Ph/Wrl2L6upqrF69OupJ\nT8c3mUwmzJ07F06nE0NDQ0GmQ9nZ2WP2d6AR68DAACorKxNG9IFt0y6XCyzLIj8/H8XFxUnz1aDG\n/gsXLowrXyxtALJYLPB6vbJp3tJaAdVKL1y4MOpk8S+++AL//u//jl//+te48MILx3xs4XDTTTfh\nzTffRG5uLmprawHgdNIXx4oUIceJFCGPE4QQWCwWvPvuuzh48CA+/vhjTJs2TdQ/V1VVyQiEOs2F\ny7EGmg7RKnk8Y6NYlkVdXR30en1SI9be3l50dHRgzpw5YpqDDl+Nt+MxHKjnxfTp0xPi3SFt86Yd\nmkajET6fDw6HA4sXL44oASSE4I033sB///d/45VXXkF5efm41hMO7733HtLT07FlyxbP/w7DAAAS\nuklEQVSRkO+5557TRV8cK1KEHCdShJxgULXEoUOHcPjwYRw7dgzl5eVYu3Ytjh49ik2bNuHiiy+O\nOZKUzsqzWq1QqVSyglsg2VLZXCzDNMcKn8+H5uZmeL3eILP6UB2PY236oOmWZDatUMN6j8cDpVIp\nGxYbeEHx+XzYvXs3Pv30U7z66qtJj047Ojpw2WWXiYR8GhXrYkWKkONEipCTDJ/PhwMHDmD79u3I\ny8uD0+nE8uXLUV1djXXr1iEnJyeuqM/j8cgUEVJPi6GhoaTL5txuN2pqapCbmxtTxEoN2qWWqJEu\nKBTd3d3o6emJmC8eL2j0nZubK7rzUdc92qgCCH7NLpcLf/nLXzBnzhw88sgjSdFuByKQkCfAvzjR\nSBFynJiSM/WmEhQKBbxeL1599VWsWrUKXq8X//znP3Ho0CE8++yz8Hq9WLNmDaqrq3HuuedGJVKt\nVosZM2ZgxowZYsFtYGAAn3/+OXw+H3JycsR25EQrHWibdTyG+IEDTekFpbu7O+ToKEKIOM162bJl\nSdMXUze4efPmiUMOAGFY7LRp00TpGsuy+Nvf/oannnoKAwMDsFgs2L9/P6688sqkrCtWRJuIk8LU\nRCpCnmQMDw/jyJEjOHjwID766CNkZmaK+eclS5ZEjcToWCKqCqAFrKGhoZAt02OBVMNcVVWVsDbr\nQJMkm80GlmWRnZ2dNM9qAOL0kFi6+/71r3/htttuw2OPPYbq6moMDg7CZrOhqKgo4esKRCplcfYh\nRcinEejQ1UOHDuHQoUP48ssvUVJSIhK0VEMsNR8KN5ZI2jJtNpsBQOZgF0+B0GAwoLS0NGkFQjp/\ncM6cOaI7nNQkKSsra9wRPyEEra2tsNlsqKqqinixI4Rg3759ePrpp7F3716UlJSMa99jQSAhn0b6\n4liRIuQ4kSLk0xg+nw+NjY2i/rmjowNLly7F6tWr8fbbb+OHP/whli9fHvNtfbh8bk5OTsiOPBp9\nx2LAPlYQQtDd3Y2+vr4gj+RAz2oa8dMiYTwRP8dxqK2tRVpaGkpLSyNG3jzP4xe/+AXq6+vx8ssv\nj3sa+Vhw7bXX4t1338Xg4CDy8vKwa9cuXH755aeLvjhWpAg5TqQIeQqBZVns3bsXO3bsQFFREdxu\nt8ygP5JFZygEekRIO/KsViu6u7uTOhSU53k0NjYCAObPnx/1whJokhSoiAj397TZI5aJ1na7Hf/2\nb/+GsrIyPPTQQxNSvDuDkSLkODElCfk///M/sX//figUCuTm5uLFF1/EzJkzAQAPPfQQ9uzZA6VS\niccffxwXXXTRJK82sdizZw/OPfdcLFy4MMigX6/Xiwb9y5YtiyuCpAXCwcFBdHZ2guM4sbg13hFX\noUDVGvn5+XG3olMESgIVCkXQUAHqrbFw4cKokW5nZyduuOEG/OAHP8DWrVsnpGh24MAB3H777eB5\nHtu2bcOOHTuSvs8JRIqQ48SUJOSRkRGxk+rxxx9HfX09nn32WdTX1+Paa6/F0aNH0dPTg/PPPx/N\nzc1Jq9SfTiCEoL+/H4cPH8ahQ4fw2WefYfbs2WL+uby8PCYPCzpAtaCgQNaCzLKszMFuPJEjnakX\nj1ojFlBf5aGhIQwPD4MQAp7nsWDBgqjywn/+85/48Y9/jKeffnrCJkDzPI+ysjIcPHgQhYWFWLFi\nBV599VUsXLhwQvY/AUgRcpyYkvdj0rZWh8Mhnmj79+/HNddcA61Wi6KiIpSWluLo0aM499xzJ2up\nEwaGYZCfn4/rr78e119/PXw+n2jQ/8tf/hItLS1YvHixaNCfn58vIygqaZMONzUajTAajSgqKpI1\nfLS3t8c04ioQhBB0dXWhv78fS5cuTbgsT6PRIC8vD9OnT0djYyM4jkNOTg76+vrQ0tIizsqTTvog\nhODll1/Gnj178Ne//hVz585N6Joi4ejRoygtLUVxcTEA4JprrsH+/fvPJEJOIU5MSUIGgJ07d+IP\nf/gDjEYj3nnnHQCC+czq1avF5xQWFuLkyZOTtcRJhUKhQFlZGcrKynDrrbeC4zh89tlnOHjwILZt\n24bh4WGsXr0a69evx0cffYS1a9di48aNYVMTSqUSOTk5omaXjrjq6+tDU1OTbMRVqMGgPM+joaEB\nCoUCy5YtS5paw+v14tixY7JW64KCAplJUmtrK4aHh/Hkk09CqVTC7Xbj8OHDcefgx4uTJ09i1qxZ\n4u+FhYX45JNPJnQNKZxeOG0J+fzzz0dfX1/Q9gceeACbN2/GAw88gAceeAAPPfQQnnzySezatWsS\nVjl1oFKpsGrVKqxatQr33XcfHA4H3nrrLezYsQMGgwGffvopPvvsM9GgP1prt1qtRl5enqi+oHri\njo4O2O12cawRJfCamhrMnDkThYWFSTtGqgoJbPYAhDuItLQ0pKWlYdasWWIXntvthlKpFC9MyRpl\nlUIKseC0JeRDhw7F9Lzrr78el156KXbt2oWCggJ0dXWJj3V3d6OgoCDs39599934v//7P2g0GpSU\nlOD3v/+9eLt+phcH09LS4PF48Otf/xqbN2/G4OAgDh8+jD//+c+4++67MWPGDDG9sXDhwqgRrV6v\nFydjS8ca1dTUwGazIScnByqVCl6vNymOcP39/Whvb0dVVVVUVUh7ezu2bNmCO+64A9dddx0YhgHP\n8xNea4j385rCmY8pWdRraWnBvHnzAABPPPEEjhw5gtdeew11dXW47rrrxKLexo0b0dLSEvZEe/vt\nt3HeeedBpVLh3nvvBQA8/PDDZ3VxEBhtoKAGSQ0NDaioqBAN+gsKCqIqEKT54oqKCni93nGNuIq0\nHzoBvLKyMqqy5L333sM999yD5557DqtWrRrzfhMBjuNQVlaGw4cPo6CgACtWrMArr7yCioqKSV1X\nApEq6sWJ0zZCjoQdO3agqakJCoUCc+bMwbPPPgsAqKiowFVXXSU6kD311FMRT3apj+3q1avx2muv\nATi7i4OAcHtfWlqK0tJSfP/73wfP8/jyyy9x8OBB3HrrrRgYGMCqVatEgySj0SgjaJovViqVYr7Y\nYDCIdx/SEVetra1RR1yFA8dxYhfhkiVLog6pffHFF/GnP/0Jf/vb32S528mCSqXCk08+iYsuugg8\nz+Omm246k8g4hTFgSkbIycA3vvENXH311fjOd76D7du3Y/Xq1fjOd74DAPje976HSy65BN/61rcm\neZWnB9xutzgg9siRIwCAtWvX4rzzzkN2djbef/99fOMb34g5XxxqxBXNP1M1RCBcLhdqamowa9as\nqM0eLMviJz/5CQYGBvDiiy8mzQkvhSCkIuQ4MSUj5HgQrThIf1apVLj++usnenlTEjqdDhs3bsTG\njRtFg/5//OMfeOqpp/Dhhx9i1apV8Hg82LBhQ5BBfyhoNBrk5+eLQ2SpGqKtrQ0OhwPp6emixE6v\n14s65liaPaxWK7Zu3Yqvfe1reOKJJ5Km7qD485//jJ/97GdoaGjA0aNHsXz5cvGxM70ukcL4ccYT\ncrTi4Isvvog333wThw8fFiOxsRRbztYTkeqRq6ur8cILL6Curg4ejweHDh3C448/jtraWsyfPx/V\n1dWorq7GnDlzouafDQYDDAYDCgsLxRl5Q0NDaGhogN1uByFEdIOLhJaWFnz3u9/Fjh078O1vf3tC\nOu8qKyvx+uuv45ZbbpFtr6+vx969e1FXV3dW1iVSiBGEkHi+zii89dZbZMGCBeTUqVOy7bW1tWTR\nokXE7XaTtrY2UlRURDiOi/ha9fX1pLGxkaxfv57861//ErfX1dXJXqu4uDjqa51J4HmefPnll+S/\n/uu/yMUXX0wWLVpEvve975E//elPpLOzk9jtduJwOKJ+2Ww28umnn5KPP/6YdHZ2kmPHjpF3332X\nvPPOO+TLL78knZ2dZGRkRHz+m2++SRYtWkQ+/fTTSTnuwM/Bgw8+SB588EHx9wsvvJB89NFHk7G0\niUS8/HLWf53xEXIkbN++HR6PBxdccAEAobD37LPPxl0cBIAFCxaE3H62FwgVCgUWL16MxYsX4667\n7oLH4xEN+p9++mmwLCsOiD333HNDTgfxer2oqalBTk6OGGHn5OSgpKRE9LMYHBzE8ePH8dhjj4Fh\nGLS3t+PNN988LYp3QKppKYXYcFYT8vHjx8M+tnPnTuzcuXPc+0idiHJotVrRAIkQIhr0HzhwAD/9\n6U9hNBpF/43FixejtrYWdrsd5eXlmD59etDrqVQqTJ8+HdOnT4fX60VWVha6u7uxaNEiXHbZZXjp\npZewZMmShB5DLHWJFFIYC85qQo4XqRMxsWAYBiaTCZs3b8bmzZtFg/6DBw/id7/7Hd5//33wPI/v\nf//7yM/PR05OTtii3NDQEG688Uacf/752LNnDxQKhXgbmGjE2rQkRaoJJIVYkCLkODBRJ+IZbskY\nFtR3YuvWrUhPT8fAwADuv/9+HD16FDt37sSJEydwzjnniBH29OnTwTAMGhoasG3bNtx///24/PLL\nxeLd6TR3btOmTbjuuutwxx13oKenBy0tLVi5cuVkLyuF0w1xJp1TiILAYk68BUKO40hxcTFpbW0l\nHo+HLFq0iNTV1U3E0k8r2O12wrKsbJvX6yXvv/8++elPf0rWrl1Lli9fTi6//HKyYMEC8uWXX07S\nSuV4/fXXSUFBAdFoNCQ3N5dceOGF4mO//OUvSXFxMSkrKyN///vfJ3GVE4ZJL5JNta8UIScIiToR\nP/roI9nfBlbnUxjFyMgIefTRR0lNTU3S93XXXXeR8vJyUlVVRS6//HJisVjExx588EFSUlJCysrK\nyIEDB5K+limESSe4qfaV6tQ7zfDaa6/hwIEDeP755wEAf/zjH/HJJ5/gySefnOSVnd1I+Z6MCadH\nvmgKIbltSymkcIbgwgsvFKekrF69Gt3d3QDCyxpTSGEsSBHyaYZEVONvuukm5ObmorKyUtxmNptx\nwQUXYN68ebjgggtEP+AU4scLL7yASy65BEBok/mzWdaYwviQIuTTDCtWrEBLSwva29vh9Xqxd+9e\nbNq0Ka7X2Lp1Kw4cOCDbtnv3btGOdOPGjdi9e3cil31G4Pzzz0dlZWXQ1/79+8XnpHxPUkgmUrK3\n0wyJsGRct24dOjo6ZNv279+Pd999FwBw4403YsOGDXj44YcTtOozAxPle5JCCuGQKuqdoejo6MBl\nl12G2tpaAIDJZILVagUgKGuysrLE31OIjgMHDuCOO+7AkSNHZB2D8Q5FOMuQKurFiVSEfBbidGqY\nmCpIpO9JCimEQ4qQzxLk5eWht7cXM2bMQG9vL3Jzc2P6u66uLmzZsgX9/f1gGAY333wzbr/9dpjN\nZlx99dXo6OjA3LlzsW/fPmRlZSX5KCYPE+F7kkIKqaLeWYJNmzbhpZdeAgC89NJLMXtvqFQqPPLI\nI6ivr8fHH3+Mp556CvX19akiYQopJANxdpKkMAVwzTXXkPz8fKJSqUhBQQF5/vnnyeDgIDnvvPNI\naWkp2bhxIxkaGhrTa2/atIm8/fbbpKysjPT09BBCCOnp6SFlZWWJPIQJwX333UeqqqrI4sWLyQUX\nXEBOnjwpPpbqvksIJr3zbap9pYp6KcSMjo4OrFu3DrW1tZg9e/aULxKOjIwgMzMTAPD444+jvr4e\nzz77bKr7LnFIFSriRCplkUJMsNvtuPLKK/Gb3/xGJDGKqVoklB6Hw+EQjyHVfZfCZCHeCDmFsxAM\nw6gBvAng/xFCHvVvawKwgRDSyzDMDADvEkLKJ3OdYwHDMA8A2AJgGEA1IWSAYZgnAXxMCPmT/zl7\nALxFCHltEpeawlmAVIScQkQwQti4B0ADJWM//grgRv/PNwLYH/i3IV5LxzDMUYZhvmIYpo5hmF3+\n7dkMwxxkGKbF/z1hcg2GYQ4xDFMb4mszABBCdhJCZgF4GcD2RO03hRTGglSEnEJEMAyzBsD7AGoA\n+PybfwLgEwD7AMwGcALAVYQQc5TXYgCkEULs/qj7AwC3A/gmADMhZDfDMDsAZBFC7k3KAYVf22wA\nfyeEVDIM8x8AQAh5yP/Y/wPwM0LIPydyTSmcfUgRcgqTAoZhDBAI+d8B/AGTkP5gGGYeIaTF//Nt\nANYTQr7FMEwFgFcArAQwE8BhAPMIIXyy15TC2Y1UY0gKEwqGYZQAPgNQCuApQsgnDMPkEUJ6/U/p\nA5A3QcvZzTBMOYTI/wSA7wMAIaSOYZh9AOoBcABuTZFxChOBVIScwqSAYRgTgDcA3AbgA0KISfKY\nhRBy5rb9pZBCGKSKeilMCgghVgDvALgYQL8/VQH/91OTubYUUpgspAg5hQkDwzDT/ZExGIbRA7gA\nQCPGoNhIIYUzEamURQoTBoZhFgF4CYASQjCwjxDyc4ZhchCnYiOFFM5EpAg5hRRSSOE0QSplkUIK\nKaRwmuD/AwZbwlpoiMGzAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f044359ecd0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Minimum value =  -34\n"
     ]
    }
   ],
   "source": [
    "figure2()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can again use calculus to identify a minimum, however, as there are multiple input variables we need to take the partial derivative with respect to each of them:\n",
    "\n",
    "$$\\frac{\\partial(f(x))}{\\partial{x_1}} = 2x_1 + 6$$\n",
    "$$\\frac{\\partial(f(x))}{\\partial{x_2}} = 2x_2 - 10$$\n",
    "Setting each to zero and solving we get $x_1 = -3$ and $x_2 = 5$\n",
    "\n",
    "Plugging these values into our equation we get -34, which is indeed the minimum value for $f(x)$.\n",
    "\n",
    "### Generalizing\n",
    "\n",
    "From this we can generalize to say that we want to find the vector $\\theta$ that minimizes the function $f(\\theta)$.  This can be accomplished analytically (by calculating the partial derivative of the function with respect to each element in $\\theta$ and finding the points where $\\frac{df(\\theta)}{d\\theta}=0$) for some cases, however not all optimization problems can be solved in such a way. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gradient Descent\n",
    "\n",
    "For situations in which there is no nice, closed-form solution to the optimization problem, gradient descent can often be used to minimize the objective function.  In gradient descent we follow the gradient of the function to be optimized until we arrive at the minimum value.  This is done iteratively by initializing the vector ($\\theta$) and then updating each value in $\\theta$ proportional to the gradient calculated at that value of $\\theta$.  This process is repeated until the gradient is zero, indicating that the minimum has been reached (when implemented as an algorithm it is typical to assign some threshold $\\epsilon$, and terminate the update loop once the gradient is smaller than $\\epsilon$).\n",
    "\n",
    "Fortunately, functions that we want to optimize are typically <strong>convex</strong>, that is to say a line segment drawn between any two points on the surface will lie \"inside\" the shape in multi-dimensional input spaces, or \"above\" the shape in the case of a one-dimensional input space.  A simple way to think of a convex function is one that slopes towards a minimal point.  The parabola in figure 1 and the parabaloid in figure 2 are examples of convex functions.  This property is important for optimization because it guarantees that a local minimum is the global minimum.  Gradient descent leverages this fact and follows the gradient to find the nearest minima, which in the case of a convex function is the only minimum, making it the global minimum.\n",
    "\n",
    "### Gradient Descent Algorithm\n",
    "\n",
    "As an example, let's perform gradient descent over the two example functions used earlier: $f(x) = x^2 + 4x$ and $f(x) = {x_1}^2 + {x_2}^2 + 6x_1 -10x_2$.  This will allow us to take a examine the algorithm for 1-D and 2-D input spaces.  From this we should be able to generalize to n-dimensional input spaces.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 493,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Minimum value of f is -4.0, achieved at -1.999999663\n"
     ]
    }
   ],
   "source": [
    "# One-Dimensional example for optimizing f(x) = x^2 + 4x\n",
    "\n",
    "def grad_descent_1D():\n",
    "    # function is x^2 + 4x\n",
    "    x = np.random.randint(-5,5)    # Initialize parameter to random value\n",
    "    max_iter = 1000                # Set a maximum number of iterations to prevent infinite loops\n",
    "    l_rate = .1                     # The learning rate\n",
    "    eps = 0.000001               # Threshold value for terminating update\n",
    "    for i in range(max_iter):\n",
    "        grad = 2*x + 4             # calculate the gradient for the function\n",
    "        x -= l_rate*grad           # subtract the gradient frin from the parameter (take a step towards the min)\n",
    "        \n",
    "        if(abs(grad) <= eps):\n",
    "            break       \n",
    "    return x\n",
    "\n",
    "min_x = grad_descent_1D()\n",
    "print \"Minimum value of f is %s, achieved at %s\" %(((min_x*min_x) + 4*min_x),min_x)\n",
    "\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 494,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Minimum value of f is -34.0, achieved at [-3.0000004   4.99999795]\n"
     ]
    }
   ],
   "source": [
    "# Two-Dimensional example for optimizing f(x) = <x,t(x)> + x*[6,-10]\n",
    "\n",
    "def grad_descent_2D(verbose = False):\n",
    "    # function is  x1^2 + x2^2 + 6x1 -10x\n",
    "    x = np.random.random_sample(size=2) - 5    # Initialize parameter to random value (map to -5:5)\n",
    "    max_iter = 1000                # Set a maximum number of iterations to prevent infinite loops\n",
    "    l_rate = .1                     # The learning rate\n",
    "    eps = 0.000001                 # Threshold value for terminating update\n",
    "    coef = np.array([6,-10])\n",
    "    \n",
    "    for i in range(max_iter):\n",
    "        grad = 2*x + coef             # calculate the gradient for the function\n",
    "        if verbose == True: print \"Iteration %s: x = %s, gradients = %s\" %(i,x,grad)\n",
    "        x -= l_rate*grad           # subtract the gradient frin from the parameter (take a step towards the min)\n",
    "        \n",
    "        if(np.linalg.norm(grad) <= np.linalg.norm(x) * eps):\n",
    "            break       \n",
    "    return x\n",
    "\n",
    "\n",
    "min_x = grad_descent_2D()\n",
    "#min_x = grad_descent_2D(verbose=True)  # Uncomment to see values at each iteration\n",
    "print \"Minimum value of f is %s, achieved at %s\" %((min_x.dot(min_x.T) + np.sum(min_x*coef)),min_x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are relatively trivial examples and not truly representative of the problems seen in machine learning applications.  For a better perspective on gradient descent, and to properly introduce stochastic gradient descent, let us investigate a use case of gradient descent in the context of machine learning\n",
    "\n",
    "## Use In Machine Learning\n",
    "\n",
    "Gradient Descent is often used in machine learning to optimize coefficients in order to minimize a given objective function.  One example of this is the <strong>logistic regression</strong> classifier.  In logistic regression we estimate the probability of a binary classification, given a set of input values.  Put another way, we seek to estimate:\n",
    "\n",
    "$$P(Y = y | X = x)$$ \n",
    "where x is the vector of features, and y is the classification.\n",
    "\n",
    "This is accomplished by using the logistic function $\\frac{1}{1+e^{-t}}$, a plot of which is shown below in figure 3.  For logistic regression $t$ is a linear function of our input features.  Commonly it is of the form $ w_0 + w_1x_1 + w_2x_2 + ... + w_dx_d$ where $d$ is the dimensionality of the feature vector.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 495,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEICAYAAACktLTqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl4VPXd/vH3h5AQAllYwpKwb7K7oeBWqSu4lLbaVtzq\nUlGrbe3iUutP+1jbp9burUpdqPpgRQWrWGm1VsVaRQFlC5thJ2FfEiCEbJ/fHzPYMQKZwCRnlvt1\nXbmYOefMzD2T5Oab75w5x9wdERFJLi2CDiAiIrGnchcRSUIqdxGRJKRyFxFJQip3EZEkpHIXEUlC\nKvcUYmY9zGy3maUFnSVRhF+vPodxuzvN7LGmyNTUEjm7/JdpP/fkY2argc5AbcTiAe5eGkyizzKz\nwcBTQN/wornAt9198WHe34+Bfu5+eWwSNuqxRwOT3b1bDO7rLWAUUBOx+Gx3f+9I7/sgjzeaGGWX\n+KKRe/K60N3bRnw1abGbWctG3qQU+BrQMfw1HZgS61wJ6uZ637smKXZJbir3FGJmvczM9xexmfU2\ns7fNbJeZvW5mD5rZ5PC60Wa2vt7tV5vZWeHLPzazqWY22czKgavMrIWZ3WFmK8xsm5k9Z2btD5TF\n3Xe6+wp3rwWM0F8Z/ZroeQ8ys7fMbKeZFZnZFyLWdTCzl82s3Mxmm9l9ZvZOxHo3s37hy+eZ2eLw\n61ViZj8wszbA34GC8BTObjMrCL8+kyPu51QzezecYZ2ZXdXI5/Cp71142Vtm9o3w5avM7B0z+6WZ\n7TCzVWY2NmLb9mb2ZzMrDa9/sRHZvxB+3XaGH3NQxLrV4ddhgZmVmdmzZpbZmOcmTUPlntr+AnwA\ndAB+DFzRyNuPA6YCecDTwLeALwKnAwXADuDBQ92Bme0EKoE/AD9r5OM3yMzSgZeB14BO4YxPm9lR\n4U0eBPYAXYCvh78O5nHgenfPBoYCb7j7HmAsUHqwv5LMrCehEv0DkA8cA8yL0VOMNBJYRugvoV8A\nj5uZhdf9H5AFDCH0OvwmyuwDgGeAW8LZZwAvm1lGxGZfBcYAvYHhwFVN8NykkVTuyevF8Ehrp5m9\nWH+lmfUATgDudvcqd3+H0NRIY7zn7i+6e5277wVuAH7k7uvdfR+h/zAuPtSUjbvnAbnAzcBHjXz8\naIwC2gI/Dz/PN4C/AePDbyxfBNzj7hXh+f4nD3Ff1cBgM8tx9x3u/mGUGS4FXnf3Z9y92t23ufuh\nyv33Ed+7aB8DYI27Pxr+a+hJoCvQ2cy6EirxG8K5q919ZpT3+TXgFXf/p7tXA78EWgMnR+Z191J3\n307oP9JjGpFZmojKPXl90d3zwl9fPMD6AmC7u1dELFvXyMeov31P4K/7iwlYQmi6pfOh7iQ8gpwI\nPGVmneqvN7PTIqYNihqZsQBY5+51EcvWAIWERqIt6z2PQ70GFwHnAWvMbKaZnRRlhu7Aiugj8+2I\n791xjbjdxv0XIr6vbcOPv93ddzTivvYrIPR67b/fOkKvUeGBHheoCD+mBEzlnro2AO3NLCtiWfeI\ny3sI/RkPQHiUm1/vPurvarUOGBtRTHnununuJVHkaRF+vML6K9z93xHTBkOiuK9IpUB3M4v8We8B\nlABbCO2VErmnSORrUD/HbHcfR2ha40Xguf2rGsiwjv/uFXS49oT/jfx+dYnytusIfa/zDrCuoeyl\nhP7TBiA8zdOd0OsncUzlnqLcfQ0wB/ixmWWER6EXRmyyHMg0s/PD89Z3Aa0auNuJwE/Dc8yYWb6Z\njTvQhmZ2tpkda2ZpZpYD/JrQHP2SI3haLcwsM+KrFfA+odHkbWaWbqFd/y4EpoSnL14g9BpkmdlA\n4MqD5M0ws8vMLDc8PVEO7P9rYBPQwcxyD5LraeAsM/uqmbUMv4nbqKkLd99CqFAvD79m1xDlfxju\nvoHQnP9DZtYu/Dp8LsrszwHnm9mZ4Z+D7wP7gHcbk1+an8o9tV0GnARsA+4DniX0i4u7lwHfBB4j\nVCp7gPUHvptP/I7QvP1rZrYLmEXoTb4DySP0Rl0ZoSmLvsAYd688guczHtgb8bXC3asIlflYYCvw\nEHCluy8N3+ZmQnP+Gwm96fgM4dfgAK4AVlto76AbCL1+hO/rGWBleEqqIPJG7r6W0HTO94HthN5M\nPfownt91wK2Evl9DaFzBXkHoPYOlwGZCb5BGk30ZcDmhN4O3EnotLwy/rhLH9CEm+YSZPQssdfd7\ngs4SFDO7H+ji7ofaa0Yk7mnknsLM7AQz62uh/dPHENq18TN71iQzMxtoZsMt5ETgWuCvQecSOVKN\n/VShJJcuhOacOxCacrnR3Ztid8R4lk1oWqKA0Pzzr4CXAk0kEgOalhERSUKalhERSUKBTct07NjR\ne/XqFdTDi4gkpLlz52519/qfOfmMwMq9V69ezJkzJ6iHFxFJSGa2puGtNC0jIpKUVO4iIklI5S4i\nkoRU7iIiSajBcjezSWa22cwWHWS9mdnvzaw4fDaWxhyiVEREmkA0I/cnCJ1l5WDGAv3DXxOAh488\nloiIHIkGy93d3yZ0JLuDGQc85SGzgLzwmV9ERCQgsdjPvZBPn71mfXjZhvobmtkEQqN7evToEYOH\nFhGJP+7O3upadlRUs2NPFTsrqtlRURX62lPNcT3zOK1/g59DOiLN+iEmd38EeARgxIgROqiNiMS9\n2jqnbG+onHeGyzl0eX9hV4eW11tWVVN30Pu8cXTfhCj3Ej59arJu6BRcIhLHNpdXUrx5N9v3l/Oe\nT5f0fy9XU15ZzcGOr9iyhZGXlU5eVgbtstLp3j6L4d1yaZeV8cmy/f+2a5MR2rZ1Bhktm35HxViU\n+3TgZjObQuisO2Xh03qJiATK3Vm/Yy9FpWUsKilnUWkZRaXlbNn12ZNttclICxVxm3TaZWXQvX3W\np8s5K1TO7bIyQpfbpJPdqiWh08rGnwbL3cyeAUYDHc1sPXAPkA7g7hOBGYROIVZM6FyVVzdVWBGR\ng6mtc1Zt3U1RaTmLSkJlXlRaRnllDQBpLYz+ndryuf75DCnIYWCXbDq0bUW7rHRys9Jp1TIt4GcQ\nWw2Wu7uPb2C9AzfFLJGISAOqaupYvmkXi0tDo/FFJWUs2bCLvdW1AGS0bMGgLtlccHQBQwpyGFqQ\ny1FdsslMT64CPxSdiUlE4lpFVQ1LNuxiccTUyvJNu6iuDU2Et8lIY0hBLl87oTtDC3MZWphD3/y2\npKel9gfwVe4iEjfK9lZTVFoWGpGXhObHV2zZTV34Dc12WekMLczlmlN7M7Qgl6GFufRsn0WLFvE5\n7x0klbuIBObDtTt4b8W2T4p87faKT9Z1yclkaGEOY4d1ZWhBDkMKcynIzYzbNzDjjcpdRJqVu/PW\nsi089FYxs1fvAKBnhyyGFuZ8MrUypCCHjm1bBZw0sancRaRZ1NTWMWPRRh5+awVLNpRTkJvJPRcO\n5svHdiM3Kz3oeElH5S4iTaqyupZpH67nkbdXsmZbBX3z2/DAxcMZd0xhs3yYJ1Wp3EWkSezeV8PT\ns9bw+Dur2LxrH8O75TLx8uM4Z3AXvQHaDFTuIhJT2/dU8ef/rOLJd1dTXlnDKf068OuvHsMp/Tro\nzdBmpHIXkZgo3bmXR95eyZTZa6msruPcIZ25cXQ/jumeF3S0lKRyF5EjUrx5NxNnruDFj0LHCxx3\nTCE3ju5Dv07ZASdLbSp3ETksC9bv5KE3V/Dq4o20atmCy0f15Bun9aZbu6ygowkqdxFpBHfnvRXb\neOitFbxTvJXszJbcNLofV5/Siw7aLz2uqNxFpEF1dc4/l2ziobdWMH/dTjq2bcUdYwdy2cgeZGdq\nH/V4pHIXkYOqrq1j+rxSJs5cwcebd9O9fWvu++JQLj6+W0odYTERqdxF5DP2VtXy3Jx1PPL2Skp2\n7mVgl2x+d8kxnD+sKy1T/GiLiULlLiKfKNtbzeRZa5j0ziq27ani+J7tuHfcEM4Y2En7qCcYlbuI\nsHlXJZPeWc3Ts9awa18Npw/I56bP9+PE3u2DjiaHSeUukuJmLNzAd5+dR3VtHWOHdeXG0/sytDA3\n6FhyhFTuIils+vxSvvvsPI7pnscvv3I0vTu2CTqSxIjKXSRFvfhRCd97bh4jerZn0tUn0LaV6iCZ\n6LspkoKmzV3PD6bOZ2Tv9ky66gSyMlQFyUbfUZEU89ycddw+bQEn9+3AY1eeQOsM7a+ejLTDqkgK\neeaDtdw2dQGn9uvI419XsSczlbtIipg8aw0/fGEhpw/I59ErR+gTpklO5S6SAp56bzV3vbiIMwZ2\n4k9XHK9iTwGacxdJcpPeWcW9f1vMWYM68+Blx9KqpYo9FajcRZLYY/9eyX2vLOHcIZ35w/jjdELq\nFKJyF0lSf5q5gv/9+1LGDu3C78cfS7oO+JVSVO4iSejBN4t54NVlXDC8K7/52jEq9hSkchdJMr//\n18f8+p/LGXdMAb/6ytE6RG+KUrmLJAl357evf8zv/vUxXz62kAe+cjRpLXSY3lQV1X/pZjbGzJaZ\nWbGZ3XGA9blm9rKZzTezIjO7OvZRReRg3J1f/3M5v/vXx1x8fDcVuzRc7maWBjwIjAUGA+PNbHC9\nzW4CFrv70cBo4FdmlhHjrCJyAO7OL15dxh/eKOaSE7rzi4uGq9glqpH7iUCxu6909ypgCjCu3jYO\nZFvoVC1tge1ATUyTishnuDs///tSHn5rBZeO7MHPvjSMFip2IbpyLwTWRVxfH14W6Y/AIKAUWAh8\nx93r6t+RmU0wszlmNmfLli2HGVlEIFTs972yhD+9vZIrRvXkvnFDVezyiVi9jX4uMA8oAI4B/mhm\nOfU3cvdH3H2Eu4/Iz8+P0UOLpB53539eXszj76ziqpN7ce+4ISp2+ZRoyr0E6B5xvVt4WaSrgRc8\npBhYBQyMTUQRiVRX59z9UhFPvLuaa0/tzT0XDtbJq+Uzoin32UB/M+sdfpP0EmB6vW3WAmcCmFln\n4ChgZSyDikio2O96aRH/N2sN13+uD3edP0jFLgfU4H7u7l5jZjcDrwJpwCR3LzKzG8LrJwI/AZ4w\ns4WAAbe7+9YmzC2ScurqnDv/upAps9dx4+i+3HbuUSp2OaioPsTk7jOAGfWWTYy4XAqcE9toIrJf\nbZ1z+7QFTJ27nm+d0Y/vnT1AxS6HpE+oisS52jrn1ufn88JHJdxyVn9uOWtA0JEkAajcReJYTW0d\n339+Pi/NK+V7Zw/g22f2DzqSJAiVu0icqqmt45Zn5/G3BRu49dyjuOnz/YKOJAlE5S4Sh6pr6/jO\nlI+YsXAjPxw7kOtP7xt0JEkwKneROFNVU8e3nvmQV4s2cdf5g/jGaX2CjiQJSOUuEkf21dRy09Mf\n8fqSTdxz4WCuPqV30JEkQancReJEZXUt33z6Q95Yupl7xw3hypN6BR1JEpjKXSQOuDs/eH4+byzd\nzE+/NJTLRvYMOpIkOJ1/SyQOzFi4kb8t2MAPzhmgYpeYULmLBGz7nirufmkRwwpzuUF7xUiMaFpG\nJGA/nl5EeWU1T39lpE5mLTGjnySRAL1WtJHp80u5+fP9GdjlM6dAEDlsKneRgJRVVHPXi4sY2CWb\nG0drOkZiS9MyIgH5ySuL2baniklXnUBGS42zJLb0EyUSgLeWbWbq3PXccHofhhbmBh1HkpDKXaSZ\n7aqs5s4XFtKvU1u+dYaO8ihNQ9MyIs3sf/++lI3llUy98WQy09OCjiNJSiN3kWb0bvFW/vL+Wq49\ntTfH9WgXdBxJYip3kWZSUVXD7S8soFeHLL539lFBx5Ekp2kZkWbyi38sY932vTw7YRStMzQdI01L\nI3eRZjB79XaefG81Xz+pJyP7dAg6jqQAlbtIE6usruX2qQsoyG3NbWMGBh1HUoSmZUSa2G9eX87K\nrXuYfO1I2rTSr5w0D43cRZrQvHU7efTtlYw/sTun9u8YdBxJISp3kSayr6aW26bOp1N2Jj88b1DQ\ncSTF6G9EkSby4BvFLN+0mz9fdQI5melBx5EUo5G7SBMoKi3jobdW8OXjCvn8wE5Bx5EUpHIXibHq\n2jpufX4BeVkZ3H3B4KDjSIrStIxIjP1p5goWbyhn4uXHk5eVEXQcSVEauYvE0PJNu/j9v4q5YHhX\nxgztEnQcSWEqd5EYqamt49bn59M2syX/84UhQceRFBdVuZvZGDNbZmbFZnbHQbYZbWbzzKzIzGbG\nNqZI/Jv0n1XMX1/Gj78whA5tWwUdR1Jcg3PuZpYGPAicDawHZpvZdHdfHLFNHvAQMMbd15qZdg+Q\nlLJyy25+9dpyzhncmQuHdw06jkhUI/cTgWJ3X+nuVcAUYFy9bS4FXnD3tQDuvjm2MUXiV12dc9vU\nBbRq2YL7vjgUMws6kkhU5V4IrIu4vj68LNIAoJ2ZvWVmc83sygPdkZlNMLM5ZjZny5Yth5dYJM48\n9d5q5qzZwd0XDqFTTmbQcUSA2L2h2hI4HjgfOBf4f2Y2oP5G7v6Iu49w9xH5+fkxemiR4KzdVsH9\n/1jG6KPyuei4+mMekeBEs597CdA94nq38LJI64Ft7r4H2GNmbwNHA8tjklIkDrk7t09bQFoL42df\nGqbpGIkr0YzcZwP9zay3mWUAlwDT623zEnCqmbU0syxgJLAktlFF4sszH6zjvZXbuPO8QRTktQ46\njsinNDhyd/caM7sZeBVIAya5e5GZ3RBeP9Hdl5jZP4AFQB3wmLsvasrgIkEq2bmXn81Ywsl9OzD+\nxO4N30CkmUV1+AF3nwHMqLdsYr3rDwAPxC6aSHxyd+58YSG1dc79Fw3XdIzEJX1CVaSRpn1Ywszl\nW7h9zFF0b58VdByRA1K5izTCpvJK7n25iBN6tePKk3oFHUfkoFTuIlFyd37010Xsq6nj/ouG06KF\npmMkfqncRaL08oINvL5kE98/ZwB98tsGHUfkkFTuIlHYunsf97y0iKO753HtqX2CjiPSIJW7SBTu\nmV7Enn21PHDxcNI0HSMJQOUu0oB/LNrAKws28O0z+zGgc3bQcUSionIXOYSdFVXc9WIRQwpyuP70\nvkHHEYmazqEqcgj3vryYnRVVPHnNCaSnaSwkiUM/rSIH8cbSTbzwUQnfHN2XIQW5QccRaRSVu8gB\nlFdWc+cLixjQuS03ndEv6DgijaZyFzmAn72yhM27Knng4qNp1TIt6DgijaZyF6nn3x9vYcrsdVz3\nuT4c3T0v6Dgih0XlLhJhz74a7pi2kD4d2/Ddsz5zMjGRhKG9ZUQi3P+PpZSW7eX5608iM13TMZK4\nNHIXCXt/5Taeem8NV53cixG92gcdR+SIqNxFgL1Vtdw+bQE92mdx67lHBR1H5IhpWkYE+NVry1i9\nrYK/XDeSrAz9Wkji08hdUt6Ha3fw+H9WcdnIHpzct2PQcURiQuUuKa2yupbbpi6ga04md4wdGHQc\nkZjR35+S0v7wxscUb97Nk9ecSHZmetBxRGJGI3dJWQvXlzFx5kq+cnw3Th+QH3QckZhSuUtKqqqp\n49ap8+nQJoO7zh8cdByRmNO0jKSkh99awdKNu3j0yhHkZmk6RpKPRu6ScpZuLOePb37MuGMKOHtw\n56DjiDQJlbuklJraOm59fgE5mencc+GQoOOINBlNy0hKefTfq1hYUsaDlx5H+zYZQccRaTIauUvK\nKN68m9+8vpyxQ7tw/vCuQccRaVIqd0kJtXXObVPnk5WRxv+M03SMJD+Vu6SEJ95dzYdrd3LPhYPp\nlJ0ZdByRJqdyl6S3euseHnh1KWcO7MQXjykMOo5Is4iq3M1sjJktM7NiM7vjENudYGY1ZnZx7CKK\nHL66Ouf2aQtIb9GCn35pGGYWdCSRZtFguZtZGvAgMBYYDIw3s898pC+83f3Aa7EOKXK4nv5gLe+v\n2s5dFwyiS66mYyR1RDNyPxEodveV7l4FTAHGHWC7bwHTgM0xzCdy2NbvqODnM5ZwWv+OfHVE96Dj\niDSraMq9EFgXcX19eNknzKwQ+BLw8KHuyMwmmNkcM5uzZcuWxmYViZq788MXFgLwv1/WdIyknli9\nofpb4HZ3rzvURu7+iLuPcPcR+fk6Cp80nefnrOffH2/ljrED6dYuK+g4Is0umk+olgCRf9N2Cy+L\nNAKYEh4ddQTOM7Mad38xJilFGmFjWSU/eWUxI3u357KRPYOOIxKIaMp9NtDfzHoTKvVLgEsjN3D3\n3vsvm9kTwN9U7BIEd+dHf11IdW0d9180nBYtNB0jqanBcnf3GjO7GXgVSAMmuXuRmd0QXj+xiTOK\nRO2leaX8a+lm7jp/EL06tgk6jkhgojpwmLvPAGbUW3bAUnf3q448lkjjbdm1jx+/XMRxPfK4+pTe\nDd9AJInpE6qSNO5+aREVVbX84uKjSdN0jKQ4lbskhRkLN/D3RRu55az+9OvUNug4IoFTuUvC276n\nirtfWsSwwlwmnNYn6DgicUEn65CEd+/LRZTtrWbyN0bSMk3jFRHQyF0S3OuLN/HivFJu+nw/BnbJ\nCTqOSNxQuUvCKttbzZ1/XcjALtl8c3S/oOOIxBVNy0jC+ukri9m2p4rHv34CGS01ThGJpN8ISUgz\nl2/huTnruf5zfRjWLTfoOCJxR+UuCWdXZTU/nLaAfp3a8u0z+wcdRyQuaVpGEs79/1jKhvJKpt14\nMpnpaUHHEYlLGrlLQnl3xVYmz1rLtaf05rge7YKOIxK3VO6SMCqqarhj2kJ6dcji++ccFXQckbim\naRlJGL98dTlrt1fw7IRRtM7QdIzIoWjkLglh7prt/PndVVx5Uk9G9ukQdByRuKdyl7hXWV3LrVMX\nUJDbmtvHDAw6jkhC0LSMxL3fvv4xK7fsYfK1I2nTSj+yItHQyF3i2vx1O3nk7RVcckJ3Tu3fMeg4\nIglD5S5xa19NLbdNXUCn7EzuPH9Q0HFEEor+xpW49eCbK1i2aReTrhpBTmZ60HFEEopG7hKXikrL\neOjNYr58bCFnDOwcdByRhKNyl7izcsturn1iDu3aZHD3hYODjiOSkDQtI3GlePNuLn10FrV1ztPX\njSQvKyPoSCIJSeUucePjTbsY/+j7gPPMhFEM6JwddCSRhKVyl7iwbOMuLntsFmbGM9eNol8nFbvI\nkdCcuwRuyYZyxj86i7QWxpQJKnaRWNDIXQJVVFrG5Y+9T6uWaTwzYRS9O7YJOpJIUtDIXQKzqKSM\nSx99n9bpaTx7vYpdJJY0cpdAzF+3kysef5/szHSmTBhF9/ZZQUcSSSoauUuz+2jtDi5/7H1ys9J5\n9noVu0hT0MhdmtXcNdv5+qTZdGibwV+uG0VhXuugI4kkJY3cpdnMXr2dKx//gPzsVkyZoGIXaUpR\nlbuZjTGzZWZWbGZ3HGD9ZWa2wMwWmtm7ZnZ07KNKIpu1chtfn/QBnXMzmTJhFF1zVewiTanBcjez\nNOBBYCwwGBhvZvUP+LEKON3dhwE/AR6JdVBJXO+u2MrVf55NQV5rpkwYReeczKAjiSS9aEbuJwLF\n7r7S3auAKcC4yA3c/V133xG+OgvoFtuYkqje+Xgr1zwxm+7tW/PMdaPolK1iF2kO0ZR7IbAu4vr6\n8LKDuRb4+4FWmNkEM5tjZnO2bNkSfUpJSDOXb+HaJ2fTq0MbnrluFPnZrYKOJJIyYvqGqpl9nlC5\n336g9e7+iLuPcPcR+fn5sXxoiTNvLtvMdU/NoW9+W/5y3Sg6tFWxizSnaHaFLAG6R1zvFl72KWY2\nHHgMGOvu22ITTxLRv5Zs4sbJHzKgS1smX6vD9ooEIZqR+2ygv5n1NrMM4BJgeuQGZtYDeAG4wt2X\nxz6mJIrXijZyw+S5DOyazdPXjlKxiwSkwZG7u9eY2c3Aq0AaMMndi8zshvD6icDdQAfgITMDqHH3\nEU0XW+LRPxZt4Oa/fMTQwlyevOZEclvrvKciQTF3D+SBR4wY4XPmzAnksSX2XlmwgW9P+Yiju+Xy\nxDUn6oTWIk3EzOZGM3jW4QfkiE2fX8p3n53HcT3y+PPVJ9K2lX6sRIKmww/IEXnxoxJumfIRx/ds\nxxMqdpG4oXKXwzZt7nq++9w8RvbuwBNXn0AbFbtI3NBvoxyW5+as4/ZpCzilb0cevXIErTPSgo4k\nIhE0cpdGe+aDtdw2dQGn9uvIY19XsYvEI5W7NMrkWWv44QsLGX1UPo9eOYLMdBW7SDzStIxE7an3\nVnP3S0WcMbATD19+HK1aqthF4pXKXaIy6Z1V3Pu3xZw9uDN/vPRYFbtInFO5S4Me+/dK7ntlCecO\n6cwfxh9HRkvN5onEO5W7HFRNbR0TZ67gl68t57xhXfjdJceSnqZiF0kEKnf5jMrqWqbOXc8jb69k\n7fYKLjy6gN989WhaqthFEobKXT6xq7KaybPW8vg7q9i6ex/HdM/jrvMHcdagzrRoYUHHE5FGULkL\nW3fv48//WcVT761hV2UNp/XvyI2jj+GkPh0IH+VTRBKMyj2Frd9RwaNvr+TZOevYV1PHmCFd+Obo\nfgzrlht0NBE5Qir3FPTxpl08PHMF0+eVAvClYwu5/vS+9OvUNuBkIhIrKvcUMm/dTh56s5jXFm+i\ndXoaV5zUk+tO60NBXuugo4lIjKnck5y785/ibTz0VjHvrthGbut0vn1mf646uRft2+gUeCLJSuWe\npOrqnNcWb+Sht1awYH0ZnbJb8aPzBjF+ZA8dc10kBei3PMlU1dTx0rwSJs5cwYote+jZIYuffWkY\nFx1fqEMGiKQQlXuSqKiqYcoH63js3yspLatkUNcc/jD+WM4b1pU07aMuknJU7gmurKKaJ99bzRPv\nrmb7nipO7NWen355GKMH5GsfdZEUpnJPUJvLK3nsnVU8PWsNe6pqOWNgJ745ui8jerUPOpqIxAGV\ne4JZs20PE2euZNrc9dTU1XHB8AJuHN2XQV1zgo4mInFE5Z4gFpeW8/DMFbyyoJSWLVpw8YhuXP+5\nPvTs0CboaCISh1TucWjr7n0UlZazqKSMotIyikrLWbOtgjYZaVx3Wh+uPbU3nXIyg44pInFM5R4g\nd6e0rJKikjIWlZazuLSMRSXlbCyv/GSb7u1bM7QglytG9eQrx3cnNys9wMQikihU7s2krs5Zs72C\nRSVlLCrjLl2TAAAFVklEQVQtY3F4ZL6johqAFgZ98tsyqk97hhbmMrgghyFdc1XmInJYVO5NoKa2\njuItu1lUUh6aVikpZ/GGcnbvqwEgPc0Y0DmbcwZ3YWhhDoMLchnUNZusDH07RCQ21CZHqLK6lmUb\nd7EoPDdeVFLG0o272FdTB0Dr9DQGdc3my8cVMqQghyEFuQzonK3zkIpIk1K5R6m2zinbW03x5t0U\nhefGi0rL+HjzbmrrHICczJYMKcjlypN6MqQgl6GFOfTu2FafEBWRZpeS5V5ZXcuOiip27KlmZ0UV\nOyqq2VFRVe/yp/8t21uN+3/vo2PbVgwtzOGsQZ0ZUpDD0MJcurVrrU+FikhciKrczWwM8DsgDXjM\n3X9eb72F158HVABXufuHMc76GXV1Tnll9afLec+By3lHxf4ir6Kyuu6g95mVkUa7rAzystJpl5VB\nYV5r2mVl0C4rnbysDHp1zGJoQa52RRSRuNZguZtZGvAgcDawHphtZtPdfXHEZmOB/uGvkcDD4X9j\n7s1lm/nJy4s/GU3X+YG3a2GQ96mSzmRIQc4nJR1Z2O3apH9S6Dpyoogkg2hG7icCxe6+EsDMpgDj\ngMhyHwc85e4OzDKzPDPr6u4bYh04r3U6g8IlHSrkjIjL6eHSziA7syUtNNctIikqmnIvBNZFXF/P\nZ0flB9qmEPhUuZvZBGACQI8ePRqbFYBje7TjwUvbHdZtRURSRbPuj+fuj7j7CHcfkZ+f35wPLSKS\nUqIp9xKge8T1buFljd1GRESaSTTlPhvob2a9zSwDuASYXm+b6cCVFjIKKGuK+XYREYlOg3Pu7l5j\nZjcDrxLaFXKSuxeZ2Q3h9ROBGYR2gywmtCvk1U0XWUREGhLVfu7uPoNQgUcumxhx2YGbYhtNREQO\nlw5wIiKShFTuIiJJSOUuIpKEzP0gn99v6gc22wKsCeTBj0xHYGvQIZqZnnPyS7XnC4n7nHu6e4Mf\nFAqs3BOVmc1x9xFB52hOes7JL9WeLyT/c9a0jIhIElK5i4gkIZV74z0SdIAA6Dknv1R7vpDkz1lz\n7iIiSUgjdxGRJKRyFxFJQir3I2Bm3zczN7OOQWdpSmb2gJktNbMFZvZXM8sLOlNTMbMxZrbMzIrN\n7I6g8zQ1M+tuZm+a2WIzKzKz7wSdqbmYWZqZfWRmfws6S1NQuR8mM+sOnAOsDTpLM/gnMNTdhwPL\ngR8GnKdJRJwveCwwGBhvZoODTdXkaoDvu/tgYBRwUwo85/2+AywJOkRTUbkfvt8AtwFJ/460u7/m\n7jXhq7MInYwlGX1yvmB3rwL2ny84abn7Bnf/MHx5F6GyKww2VdMzs27A+cBjQWdpKir3w2Bm44AS\nd58fdJYAXAP8PegQTeRg5wJOCWbWCzgWeD/YJM3it4QGZ3VBB2kqUR3PPRWZ2etAlwOs+hFwJ6Ep\nmaRxqOfr7i+Ft/kRoT/jn27ObNL0zKwtMA24xd3Lg87TlMzsAmCzu881s9FB52kqKveDcPezDrTc\nzIYBvYH5ZgahKYoPzexEd9/YjBFj6mDPdz8zuwq4ADjTk/fDESl5LmAzSydU7E+7+wtB52kGpwBf\nMLPzgEwgx8wmu/vlAeeKKX2I6QiZ2WpghLsn4tHlomJmY4BfA6e7+5ag8zQVM2tJ6A3jMwmV+mzg\nUncvCjRYE7LQCOVJYLu73xJ0nuYWHrn/wN0vCDpLrGnOXaLxRyAb+KeZzTOziQ3dIBGF3zTef77g\nJcBzyVzsYacAVwBnhL+388IjWklwGrmLiCQhjdxFRJKQyl1EJAmp3EVEkpDKXUQkCancRUSSkMpd\nRCQJqdxFRJLQ/wfXRUTdX7WiswAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f0443662750>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "figure3()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a result, this function outputs a probability (value between 0 and 1) based on a linear combination of features.  In order to find the vector of weights $w$ which maximizes the probability of correctly classifying an instance, we minimize the loss function across our training data: \n",
    "\n",
    "$$\\frac{1}{N}\\sum_{i=1}^{n} ln(1+e^{(y_i<w,x_i>)}) $$ \n",
    "\n",
    "where y is the classification of the instance (-1 or 1), w is the vector of weights, x is the vector of features, and N is the number of instances or observations in the training set.  The gradient of this function can be computed by taking the partial derivative with respect to each element of the weight vector:\n",
    "\n",
    "$$ \\frac{yx}{1+e^{(yw^Tx)}}$$\n",
    "\n",
    "With this we can create our update function, which will update the weights at each iteration of our algorithm:\n",
    "\n",
    "$$ w = w_{i-1} + \\alpha(\\frac{yx}{1+e^{(yw^Tx)}})$$\n",
    "\n",
    "where alpha is our learning rate, which can be used to adjust how quicly our algorithm follows the gradient.  Often times, the value of alpha is degenerated over time to prevent \"overshooting\" the minimum, as with a large learning rate the algorithm could oscillate over the minimum rather than reaching the true minimum.\n",
    "\n",
    "\n",
    "Below we will step through an example of using gradient descent on a toy dataset. The algorithm below can implement three types of gradient descent:\n",
    " * \"Standard\" Gradient Descent: Where we iterate through each training example and update our weights each time\n",
    " * Batch Gradient Descent: Where we vectorize the weight update calculation and update our weights based on the mean gradient calculated\n",
    " * Stochastic Gradient Descent: Where we randomly select a training instance (or group of them for mini-batch stochastic gradient descent) and update our weights based on the gradients calculated at the random selections.\n",
    "\n",
    "### Example (Logistic Regression)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 496,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(850, 25) (150, 25) (850,) (150,)\n"
     ]
    }
   ],
   "source": [
    "# Create Fake Data for Classification\n",
    "\n",
    "from sklearn.datasets import *\n",
    "\n",
    "num_samples = 1000\n",
    "test_size = 150\n",
    "\n",
    "X, Y = make_classification(n_samples=num_samples, n_features=25)\n",
    "\n",
    "X_train = X[:(num_samples -test_size)]\n",
    "X_test = X[(num_samples -test_size):]\n",
    "Y_train = Y[:(num_samples -test_size)]\n",
    "Y_test = Y[(num_samples -test_size):]\n",
    "print X_train.shape, X_test.shape,Y_train.shape,Y_test.shape\n",
    "\n",
    "\n",
    "\n",
    "# Define Logistic Function\n",
    "def logistic(x):\n",
    "    return (1/(1+np.exp(-1*(x))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 497,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Do some preprocessing\n",
    "\n",
    "# Map from {0,1} to {-1,1}\n",
    "temp = Y_train\n",
    "Y_train = np.array([-1 if k <1 else 1 for k in Y_train])\n",
    "Y_test = np.array([-1 if k <1 else 1 for k in Y_test])\n",
    "\n",
    "# Add Offset Column to feature sets\n",
    "temp = np.zeros(X_train.shape[0]).reshape(-1,1)\n",
    "X_train = np.hstack((X_train,temp))\n",
    "\n",
    "temp = np.zeros(X_test.shape[0]).reshape(-1,1)\n",
    "X_test = np.hstack((X_test,temp))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 513,
   "metadata": {},
   "outputs": [],
   "source": [
    "def gradient_descent(x, y, alpha = 1.0, eps = 0.001, batch_size = None, stochastic = False, max_iter = 100000):\n",
    "    \n",
    "    if batch_size == None:\n",
    "        batch_size = x.shape[0]\n",
    "    w = np.array([1.0 for i in range(x.shape[1])])\n",
    "    b_i = 0\n",
    "    y = np.array(y)\n",
    "    gradient = np.copy(w)\n",
    "    for i in range(max_iter):\n",
    "        if np.linalg.norm(gradient) <= np.linalg.norm(w) * eps:\n",
    "            break\n",
    "        if stochastic == True:\n",
    "            batch = np.random.permutation(x.shape[0])[:batch_size] # uncomment for stochastic gradient descent\n",
    "            #batch = np.random.choice(x.shape[0], batch_size)\n",
    "        else:\n",
    "            batch = range(b_i,batch_size)\n",
    "            if batch_size > 1:\n",
    "                b_i = (batch[-1] + 1) % x.shape[0]\n",
    "        \n",
    "        gradient = np.mean(np.tile(-y[batch]/(1.0+np.exp(y[batch]*w.dot(x[batch].T))),(x.shape[1],1)).T*x[batch], \n",
    "                axis=0)\n",
    "        w -= alpha * gradient\n",
    "        \n",
    "        \n",
    "    print \"Number of Iterations: \", i\n",
    "    return np.array(w)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 514,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def predict_prob(x, weights):\n",
    "    y_pred = logistic(weights.dot(X_test.T))\n",
    "    return y_pred\n",
    "    \n",
    "def predict_class(x,weights, threshold = 0.5):\n",
    "    y_prob = predict_prob(x, weights)\n",
    "    return y_pred_prob >= threshold"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 519,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of Iterations:  81\n",
      "Accuracy:  0.933333333333\n"
     ]
    }
   ],
   "source": [
    "# Batch Gradient Descent\n",
    "weights = gradient_descent(X_train,Y_train)\n",
    "y_pred_prob = predict_prob(X_test, weights)\n",
    "y_pred = predict_class(X_test, weights)*1\n",
    "Y_test_acc = np.array([0 if k <1 else 1 for k in Y_test])\n",
    "print \"Accuracy: \", accuracy_score(Y_test_acc, y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 520,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of Iterations:  6240\n",
      "Accuracy:  0.933333333333\n"
     ]
    }
   ],
   "source": [
    "#Stochastic Gradient Descent\n",
    "weights = gradient_descent(X_train,Y_train, batch_size=10, stochastic=True)\n",
    "y_pred_prob = predict_prob(X_test, weights)\n",
    "y_pred = predict_class(X_test, weights)*1\n",
    "print \"Accuracy: \",accuracy_score(Y_test_acc, y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 521,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy:  0.96\n"
     ]
    }
   ],
   "source": [
    "# SkLearn Logistic Regression Classifier\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "clf = LogisticRegression()\n",
    "clf.fit(X_train,Y_train)\n",
    "y_pred_sk = clf.predict(X_test)\n",
    "print \"Accuracy: \",clf.score(X_test,Y_test)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "* Gradient Descent is useful for optimizing functions with no simple closed form solution\n",
    "* Batch gradient descent works well for reasonably sized data sets\n",
    "* Stochastic gradient descent:\n",
    "    * Useful for large and/or high dimensional data sets (due to vectorization limitations)\n",
    "    * Typically requires more iterations than batch gradient descent\n",
    "    * Can be useful for situtations where there are multiple local minima (non-convex)\n",
    "* Gradient Descent useful in classifiers such as logistic regression and in neural nets.  Useful when seeking a set of values to optimize a function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Utility Functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def figure1():\n",
    "    x = np.arange(-10,11)\n",
    "    y = x*x + 4*x\n",
    "    plt.plot(x,y)\n",
    "    plt.xlabel(\"X\")\n",
    "    plt.ylabel(\"Y\")\n",
    "    plt.title(\"Figure 1 - Plot of y = x^2 + 4x\")\n",
    "    plt.show()\n",
    "    print \"Minimum value = \",np.min(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def figure2():\n",
    "    from matplotlib import cm\n",
    "    x = np.arange(-30,31)\n",
    "    y = np.arange(-30,31)\n",
    "    x,y = np.meshgrid(x,y)\n",
    "    z = x*x + y*y - 10*y + 6*x\n",
    "    fig = plt.figure()\n",
    "    ax = fig.add_subplot(111,projection ='3d')\n",
    "    surface = ax.plot_surface(x,y,z,cmap=cm.coolwarm,linewidth=0, antialiased=False)\n",
    "    fig.colorbar(surface, shrink=0.5, aspect=5)\n",
    "    plt.title(\"Figure 2 - Plot of Convex Surface\")\n",
    "    plt.show()\n",
    "    print \"Minimum value = \",np.min(z)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def figure3():\n",
    "    x = np.arange(-5,6)\n",
    "    y = 1/(1+np.exp(-1*x))\n",
    "    plt.plot(x,y)\n",
    "    plt.title(\"Figure 3 - Logistic Function\")\n",
    "    plt.show()\n",
    "\n",
    "#figure3()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 512,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "11714\n",
      "(150,)\n",
      "0.626666666667\n"
     ]
    }
   ],
   "source": [
    "# Used for Validation of Gradient Descent Algorithm\n",
    "# Copied from http://nbviewer.jupyter.org/gist/vietjtnguyen/6655020#Nonlinear-Transformation\n",
    "def gradient_descent2(z, y, w_h=None, eta=1.0, max_iterations=100000, epsilon=0.001):\n",
    "    N = 50\n",
    "    if w_h == None:\n",
    "        w_h = np.array([0.0 for i in range(z.shape[1])])\n",
    "    \n",
    "    # save a history of the weight vectors into an array\n",
    "    w_h_i = [np.copy(w_h)]\n",
    "    \n",
    "    for i in range(max_iterations):\n",
    "        #subset_indices = range(z.shape[0])\n",
    "        subset_indices = np.random.permutation(z.shape[0])[:10] # uncomment for stochastic gradient descent\n",
    "        \n",
    "        grad_E_in = np.mean(np.tile(- y[subset_indices] /\n",
    "                                        ( 1.0 + np.exp(y[subset_indices] * w_h.dot(z[subset_indices].T)) ),\n",
    "                                    (z.shape[1], 1)).T * \n",
    "                            z[subset_indices], axis=0)\n",
    "        \n",
    "        w_h -= eta * grad_E_in\n",
    "        w_h_i.append(np.copy(w_h))\n",
    "        \n",
    "        if np.linalg.norm(grad_E_in) <= np.linalg.norm(w_h) * epsilon:\n",
    "            break\n",
    "    print i\n",
    "    return w_h\n",
    "            \n",
    "weights2 =  gradient_descent2(X_train,Y_train)  \n",
    "y_pred_prob2 = predict_prob(X_test, weights2)\n",
    "y_pred2 = predict_class(X_test, weights2)*1\n",
    "print y_pred2.shape\n",
    "print accuracy_score(Y_test_acc, y_pred2)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}