t81_558_class_12_02_qlearningreinforcement.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_12_02_qlearningreinforcement.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 12: Deep Learning and Security**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 12 Video Material\n",
    "\n",
    "* Part 12.1: Introduction to the OpenAI Gym [[Video]](https://www.youtube.com/watch?v=_KbUxgyisjM&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_12_01_ai_gym.ipynb)\n",
    "* **Part 12.2: Introduction to Q-Learning** [[Video]](https://www.youtube.com/watch?v=uwcXWe_Fra0&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_12_02_qlearningreinforcement.ipynb)\n",
    "* Part 12.3: Keras Q-Learning in the OpenAI Gym [[Video]](https://www.youtube.com/watch?v=Ya1gYt63o3M&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_12_03_keras_reinforce.ipynb)\n",
    "* Part 12.4: Atari Games with Keras Neural Networks [[Video]](https://www.youtube.com/watch?v=t2yIu6cRa38&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_12_04_atari.ipynb)\n",
    "* Part 12.5: How Alpha Zero used Reinforcement Learning to Master Chess [[Video]](https://www.youtube.com/watch?v=ikDgyD7nVI8&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_12_05_alpha_zero.ipynb)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Google CoLab Instructions\n",
    "\n",
    "The following code ensures that Google CoLab is running the correct version of TensorFlow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    from google.colab import drive\n",
    "    %tensorflow_version 2.x\n",
    "    COLAB = True\n",
    "    print(\"Note: using Google CoLab\")\n",
    "except:\n",
    "    print(\"Note: not using Google CoLab\")\n",
    "    COLAB = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 12.2: Introduction to Q-Learning\n",
    "\n",
    "### Single Action Cart\n",
    "\n",
    "Mountain car actions:\n",
    "\n",
    "* 0 - Apply left force\n",
    "* 1 - Apply no force\n",
    "* 2 - Apply right force\n",
    "\n",
    "State values:\n",
    "\n",
    "* state[0] - Position \n",
    "* state[1] - Velocity\n",
    "\n",
    "The following shows a cart that simply applies full-force to climb the hill.  The cart is simply not strong enough.  It will need to use momentum from the hill behind it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step 1: State=[-0.54298351  0.00115394], Reward=-1.0\n",
      "Step 2: State=[-0.54068426  0.00229925], Reward=-1.0\n",
      "Step 3: State=[-0.53725693  0.00342733], Reward=-1.0\n",
      "Step 4: State=[-0.53272719  0.00452974], Reward=-1.0\n",
      "Step 5: State=[-0.527129    0.00559819], Reward=-1.0\n",
      "Step 6: State=[-0.52050433  0.00662467], Reward=-1.0\n",
      "Step 7: State=[-0.51290287  0.00760146], Reward=-1.0\n",
      "Step 8: State=[-0.50438161  0.00852126], Reward=-1.0\n",
      "Step 9: State=[-0.4950044   0.00937721], Reward=-1.0\n",
      "Step 10: State=[-0.48484139  0.01016301], Reward=-1.0\n",
      "Step 11: State=[-0.47396841  0.01087299], Reward=-1.0\n",
      "Step 12: State=[-0.46246627  0.01150213], Reward=-1.0\n",
      "Step 13: State=[-0.45042007  0.0120462 ], Reward=-1.0\n",
      "Step 14: State=[-0.43791831  0.01250176], Reward=-1.0\n",
      "Step 15: State=[-0.4250521   0.01286621], Reward=-1.0\n",
      "Step 16: State=[-0.41191426  0.01313783], Reward=-1.0\n",
      "Step 17: State=[-0.39859848  0.01331578], Reward=-1.0\n",
      "Step 18: State=[-0.38519838  0.0134001 ], Reward=-1.0\n",
      "Step 19: State=[-0.37180672  0.01339166], Reward=-1.0\n",
      "Step 20: State=[-0.35851456  0.01329216], Reward=-1.0\n",
      "Step 21: State=[-0.34541053  0.01310403], Reward=-1.0\n",
      "Step 22: State=[-0.33258017  0.01283036], Reward=-1.0\n",
      "Step 23: State=[-0.32010531  0.01247486], Reward=-1.0\n",
      "Step 24: State=[-0.30806361  0.0120417 ], Reward=-1.0\n",
      "Step 25: State=[-0.29652811  0.0115355 ], Reward=-1.0\n",
      "Step 26: State=[-0.28556694  0.01096116], Reward=-1.0\n",
      "Step 27: State=[-0.27524312  0.01032383], Reward=-1.0\n",
      "Step 28: State=[-0.26561434  0.00962878], Reward=-1.0\n",
      "Step 29: State=[-0.25673298  0.00888136], Reward=-1.0\n",
      "Step 30: State=[-0.24864606  0.00808693], Reward=-1.0\n",
      "Step 31: State=[-0.24139526  0.0072508 ], Reward=-1.0\n",
      "Step 32: State=[-0.23501706  0.0063782 ], Reward=-1.0\n",
      "Step 33: State=[-0.22954281  0.00547425], Reward=-1.0\n",
      "Step 34: State=[-0.22499885  0.00454396], Reward=-1.0\n",
      "Step 35: State=[-0.22140666  0.00359218], Reward=-1.0\n",
      "Step 36: State=[-0.21878297  0.00262369], Reward=-1.0\n",
      "Step 37: State=[-0.21713985  0.00164313], Reward=-1.0\n",
      "Step 38: State=[-0.21648478  0.00065507], Reward=-1.0\n",
      "Step 39: State=[-0.21682075 -0.00033597], Reward=-1.0\n",
      "Step 40: State=[-0.21814623 -0.00132548], Reward=-1.0\n",
      "Step 41: State=[-0.22045518 -0.00230895], Reward=-1.0\n",
      "Step 42: State=[-0.22373702 -0.00328184], Reward=-1.0\n",
      "Step 43: State=[-0.22797653 -0.00423951], Reward=-1.0\n",
      "Step 44: State=[-0.23315378 -0.00517725], Reward=-1.0\n",
      "Step 45: State=[-0.239244   -0.00609022], Reward=-1.0\n",
      "Step 46: State=[-0.24621747 -0.00697347], Reward=-1.0\n",
      "Step 47: State=[-0.25403939 -0.00782191], Reward=-1.0\n",
      "Step 48: State=[-0.26266974 -0.00863035], Reward=-1.0\n",
      "Step 49: State=[-0.27206323 -0.0093935 ], Reward=-1.0\n",
      "Step 50: State=[-0.28216923 -0.010106  ], Reward=-1.0\n",
      "Step 51: State=[-0.29293174 -0.01076251], Reward=-1.0\n",
      "Step 52: State=[-0.30428945 -0.01135771], Reward=-1.0\n",
      "Step 53: State=[-0.31617585 -0.0118864 ], Reward=-1.0\n",
      "Step 54: State=[-0.32851945 -0.0123436 ], Reward=-1.0\n",
      "Step 55: State=[-0.34124405 -0.0127246 ], Reward=-1.0\n",
      "Step 56: State=[-0.3542691  -0.01302505], Reward=-1.0\n",
      "Step 57: State=[-0.36751021 -0.01324111], Reward=-1.0\n",
      "Step 58: State=[-0.38087966 -0.01336945], Reward=-1.0\n",
      "Step 59: State=[-0.3942871  -0.01340744], Reward=-1.0\n",
      "Step 60: State=[-0.40764024 -0.01335314], Reward=-1.0\n",
      "Step 61: State=[-0.42084563 -0.01320539], Reward=-1.0\n",
      "Step 62: State=[-0.43380952 -0.01296389], Reward=-1.0\n",
      "Step 63: State=[-0.44643872 -0.0126292 ], Reward=-1.0\n",
      "Step 64: State=[-0.45864146 -0.01220274], Reward=-1.0\n",
      "Step 65: State=[-0.47032831 -0.01168684], Reward=-1.0\n",
      "Step 66: State=[-0.48141298 -0.01108468], Reward=-1.0\n",
      "Step 67: State=[-0.49181321 -0.01040022], Reward=-1.0\n",
      "Step 68: State=[-0.50145146 -0.00963826], Reward=-1.0\n",
      "Step 69: State=[-0.5102557  -0.00880424], Reward=-1.0\n",
      "Step 70: State=[-0.51815998 -0.00790428], Reward=-1.0\n",
      "Step 71: State=[-0.52510506 -0.00694507], Reward=-1.0\n",
      "Step 72: State=[-0.53103883 -0.00593378], Reward=-1.0\n",
      "Step 73: State=[-0.53591681 -0.00487798], Reward=-1.0\n",
      "Step 74: State=[-0.53970243 -0.00378562], Reward=-1.0\n",
      "Step 75: State=[-0.54236732 -0.00266489], Reward=-1.0\n",
      "Step 76: State=[-0.54389151 -0.0015242 ], Reward=-1.0\n",
      "Step 77: State=[-5.44263607e-01 -3.72094556e-04], Reward=-1.0\n",
      "Step 78: State=[-0.54348081  0.00078279], Reward=-1.0\n",
      "Step 79: State=[-0.541549    0.00193182], Reward=-1.0\n",
      "Step 80: State=[-0.53848261  0.00306638], Reward=-1.0\n",
      "Step 81: State=[-0.53430464  0.00417797], Reward=-1.0\n",
      "Step 82: State=[-0.52904639  0.00525825], Reward=-1.0\n",
      "Step 83: State=[-0.52274728  0.00629911], Reward=-1.0\n",
      "Step 84: State=[-0.51545456  0.00729272], Reward=-1.0\n",
      "Step 85: State=[-0.50722291  0.00823165], Reward=-1.0\n",
      "Step 86: State=[-0.49811404  0.00910888], Reward=-1.0\n",
      "Step 87: State=[-0.48819611  0.00991793], Reward=-1.0\n",
      "Step 88: State=[-0.4775432   0.01065291], Reward=-1.0\n",
      "Step 89: State=[-0.46623461  0.01130859], Reward=-1.0\n",
      "Step 90: State=[-0.45435414  0.01188048], Reward=-1.0\n",
      "Step 91: State=[-0.44198927  0.01236487], Reward=-1.0\n",
      "Step 92: State=[-0.42923038  0.01275889], Reward=-1.0\n",
      "Step 93: State=[-0.41616983  0.01306055], Reward=-1.0\n",
      "Step 94: State=[-0.40290112  0.01326871], Reward=-1.0\n",
      "Step 95: State=[-0.389518    0.01338313], Reward=-1.0\n",
      "Step 96: State=[-0.37611358  0.01340442], Reward=-1.0\n",
      "Step 97: State=[-0.36277956  0.01333402], Reward=-1.0\n",
      "Step 98: State=[-0.34960543  0.01317413], Reward=-1.0\n",
      "Step 99: State=[-0.3366778   0.01292763], Reward=-1.0\n",
      "Step 100: State=[-0.32407975  0.01259805], Reward=-1.0\n",
      "Step 101: State=[-0.31189033  0.01218942], Reward=-1.0\n",
      "Step 102: State=[-0.3001841   0.01170623], Reward=-1.0\n",
      "Step 103: State=[-0.28903082  0.01115328], Reward=-1.0\n",
      "Step 104: State=[-0.27849515  0.01053567], Reward=-1.0\n",
      "Step 105: State=[-0.26863652  0.00985862], Reward=-1.0\n",
      "Step 106: State=[-0.25950904  0.00912749], Reward=-1.0\n",
      "Step 107: State=[-0.25116142  0.00834761], Reward=-1.0\n",
      "Step 108: State=[-0.24363708  0.00752434], Reward=-1.0\n",
      "Step 109: State=[-0.23697416  0.00666292], Reward=-1.0\n",
      "Step 110: State=[-0.23120564  0.00576852], Reward=-1.0\n",
      "Step 111: State=[-0.22635946  0.00484618], Reward=-1.0\n",
      "Step 112: State=[-0.22245866  0.0039008 ], Reward=-1.0\n",
      "Step 113: State=[-0.21952148  0.00293718], Reward=-1.0\n",
      "Step 114: State=[-0.21756149  0.00196   ], Reward=-1.0\n",
      "Step 115: State=[-0.21658763  0.00097386], Reward=-1.0\n",
      "Step 116: State=[-2.16604342e-01 -1.67115330e-05], Reward=-1.0\n",
      "Step 117: State=[-0.21761155 -0.0010072 ], Reward=-1.0\n",
      "Step 118: State=[-0.21960466 -0.00199312], Reward=-1.0\n",
      "Step 119: State=[-0.22257458 -0.00296991], Reward=-1.0\n",
      "Step 120: State=[-0.22650757 -0.003933  ], Reward=-1.0\n",
      "Step 121: State=[-0.23138525 -0.00487768], Reward=-1.0\n",
      "Step 122: State=[-0.23718442 -0.00579916], Reward=-1.0\n",
      "Step 123: State=[-0.24387695 -0.00669253], Reward=-1.0\n",
      "Step 124: State=[-0.2514297  -0.00755275], Reward=-1.0\n",
      "Step 125: State=[-0.25980435 -0.00837465], Reward=-1.0\n",
      "Step 126: State=[-0.26895731 -0.00915296], Reward=-1.0\n",
      "Step 127: State=[-0.27883967 -0.00988236], Reward=-1.0\n",
      "Step 128: State=[-0.28939716 -0.01055749], Reward=-1.0\n",
      "Step 129: State=[-0.30057017 -0.01117301], Reward=-1.0\n",
      "Step 130: State=[-0.31229385 -0.01172368], Reward=-1.0\n",
      "Step 131: State=[-0.32449829 -0.01220444], Reward=-1.0\n",
      "Step 132: State=[-0.33710877 -0.01261047], Reward=-1.0\n",
      "Step 133: State=[-0.35004609 -0.01293732], Reward=-1.0\n",
      "Step 134: State=[-0.36322703 -0.01318094], Reward=-1.0\n",
      "Step 135: State=[-0.3765649  -0.01333787], Reward=-1.0\n",
      "Step 136: State=[-0.3899701 -0.0134052], Reward=-1.0\n",
      "Step 137: State=[-0.40335089 -0.01338079], Reward=-1.0\n",
      "Step 138: State=[-0.41661411 -0.01326322], Reward=-1.0\n",
      "Step 139: State=[-0.429666  -0.0130519], Reward=-1.0\n",
      "Step 140: State=[-0.44241311 -0.0127471 ], Reward=-1.0\n",
      "Step 141: State=[-0.4547631 -0.01235  ], Reward=-1.0\n",
      "Step 142: State=[-0.4666257 -0.0118626], Reward=-1.0\n",
      "Step 143: State=[-0.47791353 -0.01128782], Reward=-1.0\n",
      "Step 144: State=[-0.48854292 -0.01062939], Reward=-1.0\n",
      "Step 145: State=[-0.49843474 -0.00989182], Reward=-1.0\n",
      "Step 146: State=[-0.50751511 -0.00908037], Reward=-1.0\n",
      "Step 147: State=[-0.51571607 -0.00820096], Reward=-1.0\n",
      "Step 148: State=[-0.52297614 -0.00726007], Reward=-1.0\n",
      "Step 149: State=[-0.52924088 -0.00626474], Reward=-1.0\n",
      "Step 150: State=[-0.53446331 -0.00522243], Reward=-1.0\n",
      "Step 151: State=[-0.53860426 -0.00414096], Reward=-1.0\n",
      "Step 152: State=[-0.54163272 -0.00302845], Reward=-1.0\n",
      "Step 153: State=[-0.54352598 -0.00189327], Reward=-1.0\n",
      "Step 154: State=[-0.54426988 -0.0007439 ], Reward=-1.0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step 155: State=[-5.43858848e-01  4.11033963e-04], Reward=-1.0\n",
      "Step 156: State=[-0.54229596  0.00156289], Reward=-1.0\n",
      "Step 157: State=[-0.53959291  0.00270305], Reward=-1.0\n",
      "Step 158: State=[-0.53576995  0.00382296], Reward=-1.0\n",
      "Step 159: State=[-0.53085573  0.00491422], Reward=-1.0\n",
      "Step 160: State=[-0.52488709  0.00596864], Reward=-1.0\n",
      "Step 161: State=[-0.51790879  0.00697831], Reward=-1.0\n",
      "Step 162: State=[-0.50997315  0.00793563], Reward=-1.0\n",
      "Step 163: State=[-0.50113969  0.00883347], Reward=-1.0\n",
      "Step 164: State=[-0.49147453  0.00966515], Reward=-1.0\n",
      "Step 165: State=[-0.48104994  0.01042459], Reward=-1.0\n",
      "Step 166: State=[-0.4699436   0.01110634], Reward=-1.0\n",
      "Step 167: State=[-0.45823793  0.01170567], Reward=-1.0\n",
      "Step 168: State=[-0.44601934  0.01221859], Reward=-1.0\n",
      "Step 169: State=[-0.43337735  0.01264199], Reward=-1.0\n",
      "Step 170: State=[-0.42040379  0.01297356], Reward=-1.0\n",
      "Step 171: State=[-0.40719189  0.0132119 ], Reward=-1.0\n",
      "Step 172: State=[-0.39383541  0.01335649], Reward=-1.0\n",
      "Step 173: State=[-0.38042775  0.01340766], Reward=-1.0\n",
      "Step 174: State=[-0.36706117  0.01336659], Reward=-1.0\n",
      "Step 175: State=[-0.35382593  0.01323523], Reward=-1.0\n",
      "Step 176: State=[-0.34080966  0.01301628], Reward=-1.0\n",
      "Step 177: State=[-0.32809662  0.01271304], Reward=-1.0\n",
      "Step 178: State=[-0.31576721  0.0123294 ], Reward=-1.0\n",
      "Step 179: State=[-0.3038975   0.01186971], Reward=-1.0\n",
      "Step 180: State=[-0.29255881  0.01133869], Reward=-1.0\n",
      "Step 181: State=[-0.28181746  0.01074134], Reward=-1.0\n",
      "Step 182: State=[-0.2717346   0.01008286], Reward=-1.0\n",
      "Step 183: State=[-0.26236604  0.00936856], Reward=-1.0\n",
      "Step 184: State=[-0.25376224  0.0086038 ], Reward=-1.0\n",
      "Step 185: State=[-0.24596832  0.00779393], Reward=-1.0\n",
      "Step 186: State=[-0.23902409  0.00694423], Reward=-1.0\n",
      "Step 187: State=[-0.23296419  0.0060599 ], Reward=-1.0\n",
      "Step 188: State=[-0.22781818  0.00514601], Reward=-1.0\n",
      "Step 189: State=[-0.22361066  0.00420752], Reward=-1.0\n",
      "Step 190: State=[-0.22036141  0.00324926], Reward=-1.0\n",
      "Step 191: State=[-0.21808547  0.00227594], Reward=-1.0\n",
      "Step 192: State=[-0.21679328  0.00129219], Reward=-1.0\n",
      "Step 193: State=[-0.21649073  0.00030255], Reward=-1.0\n",
      "Step 194: State=[-0.21717918 -0.00068845], Reward=-1.0\n",
      "Step 195: State=[-0.21885552 -0.00167634], Reward=-1.0\n",
      "Step 196: State=[-0.22151209 -0.00265657], Reward=-1.0\n",
      "Step 197: State=[-0.22513666 -0.00362457], Reward=-1.0\n",
      "Step 198: State=[-0.22971236 -0.0045757 ], Reward=-1.0\n",
      "Step 199: State=[-0.23521755 -0.00550519], Reward=-1.0\n",
      "Step 200: State=[-0.24162571 -0.00640816], Reward=-1.0\n"
     ]
    }
   ],
   "source": [
    "import gym\n",
    "\n",
    "env = gym.make(\"MountainCar-v0\")\n",
    "env.reset()\n",
    "done = False\n",
    "\n",
    "i = 0\n",
    "while not done:\n",
    "    i += 1\n",
    "    state, reward, done, _ = env.step(2)\n",
    "    env.render()\n",
    "    print(f\"Step {i}: State={state}, Reward={reward}\")\n",
    "    \n",
    "env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Programmed Car\n",
    "\n",
    "This is a car that I hand-programmed.  It uses a simple rule, but solves the problem. The programmed car constantly applies force to one direction or another.  It does not reset.  Whatever direction the car is currently rolling, it applies force in that direction.  Therefore, the car begins to climb a hill, is overpowered, and rolls backward.  However, once it begins to roll backwards force is immediately applied in this new direction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step 1: State=[-0.51431373 -0.00107771], Reward=-1.0\n",
      "Step 2: State=[-0.51646107 -0.00214734], Reward=-1.0\n",
      "Step 3: State=[-0.51966193 -0.00320087], Reward=-1.0\n",
      "Step 4: State=[-0.52389233 -0.00423039], Reward=-1.0\n",
      "Step 5: State=[-0.52912052 -0.00522819], Reward=-1.0\n",
      "Step 6: State=[-0.53530729 -0.00618678], Reward=-1.0\n",
      "Step 7: State=[-0.54240628 -0.00709898], Reward=-1.0\n",
      "Step 8: State=[-0.55036428 -0.007958  ], Reward=-1.0\n",
      "Step 9: State=[-0.55912175 -0.00875748], Reward=-1.0\n",
      "Step 10: State=[-0.56861331 -0.00949156], Reward=-1.0\n",
      "Step 11: State=[-0.57876828 -0.01015497], Reward=-1.0\n",
      "Step 12: State=[-0.58951137 -0.01074309], Reward=-1.0\n",
      "Step 13: State=[-0.60076333 -0.01125196], Reward=-1.0\n",
      "Step 14: State=[-0.61244171 -0.01167838], Reward=-1.0\n",
      "Step 15: State=[-0.62446163 -0.01201992], Reward=-1.0\n",
      "Step 16: State=[-0.63673657 -0.01227494], Reward=-1.0\n",
      "Step 17: State=[-0.64917917 -0.0124426 ], Reward=-1.0\n",
      "Step 18: State=[-0.66170205 -0.01252287], Reward=-1.0\n",
      "Step 19: State=[-0.67421853 -0.01251648], Reward=-1.0\n",
      "Step 20: State=[-0.68664341 -0.01242488], Reward=-1.0\n",
      "Step 21: State=[-0.69889363 -0.01225023], Reward=-1.0\n",
      "Step 22: State=[-0.71088891 -0.01199528], Reward=-1.0\n",
      "Step 23: State=[-0.72255227 -0.01166336], Reward=-1.0\n",
      "Step 24: State=[-0.73381051 -0.01125823], Reward=-1.0\n",
      "Step 25: State=[-0.7445946  -0.01078409], Reward=-1.0\n",
      "Step 26: State=[-0.75484   -0.0102454], Reward=-1.0\n",
      "Step 27: State=[-0.76448689 -0.00964689], Reward=-1.0\n",
      "Step 28: State=[-0.77348032 -0.00899343], Reward=-1.0\n",
      "Step 29: State=[-0.78177031 -0.00828998], Reward=-1.0\n",
      "Step 30: State=[-0.78931187 -0.00754156], Reward=-1.0\n",
      "Step 31: State=[-0.79606502 -0.00675316], Reward=-1.0\n",
      "Step 32: State=[-0.80199476 -0.00592974], Reward=-1.0\n",
      "Step 33: State=[-0.80707094 -0.00507618], Reward=-1.0\n",
      "Step 34: State=[-0.81126824 -0.00419729], Reward=-1.0\n",
      "Step 35: State=[-0.81456603 -0.00329779], Reward=-1.0\n",
      "Step 36: State=[-0.81694832 -0.0023823 ], Reward=-1.0\n",
      "Step 37: State=[-0.81840369 -0.00145537], Reward=-1.0\n",
      "Step 38: State=[-8.18925204e-01 -5.21510941e-04], Reward=-1.0\n",
      "Step 39: State=[-8.18510378e-01  4.14826025e-04], Reward=-1.0\n",
      "Step 40: State=[-0.81516118  0.00334919], Reward=-1.0\n",
      "Step 41: State=[-0.80889363  0.00626755], Reward=-1.0\n",
      "Step 42: State=[-0.7997382   0.00915543], Reward=-1.0\n",
      "Step 43: State=[-0.78774062  0.01199759], Reward=-1.0\n",
      "Step 44: State=[-0.77296289  0.01477773], Reward=-1.0\n",
      "Step 45: State=[-0.75548455  0.01747834], Reward=-1.0\n",
      "Step 46: State=[-0.73540399  0.02008056], Reward=-1.0\n",
      "Step 47: State=[-0.71283965  0.02256434], Reward=-1.0\n",
      "Step 48: State=[-0.68793102  0.02490863], Reward=-1.0\n",
      "Step 49: State=[-0.66083923  0.0270918 ], Reward=-1.0\n",
      "Step 50: State=[-0.63174696  0.02909226], Reward=-1.0\n",
      "Step 51: State=[-0.60085774  0.03088922], Reward=-1.0\n",
      "Step 52: State=[-0.56839425  0.03246349], Reward=-1.0\n",
      "Step 53: State=[-0.53459581  0.03379844], Reward=-1.0\n",
      "Step 54: State=[-0.49971491  0.03488091], Reward=-1.0\n",
      "Step 55: State=[-0.46401297  0.03570193], Reward=-1.0\n",
      "Step 56: State=[-0.42775556  0.03625741], Reward=-1.0\n",
      "Step 57: State=[-0.39120711  0.03654845], Reward=-1.0\n",
      "Step 58: State=[-0.35462569  0.03658142], Reward=-1.0\n",
      "Step 59: State=[-0.31825799  0.0363677 ], Reward=-1.0\n",
      "Step 60: State=[-0.28233478  0.03592322], Reward=-1.0\n",
      "Step 61: State=[-0.24706714  0.03526764], Reward=-1.0\n",
      "Step 62: State=[-0.21264364  0.0344235 ], Reward=-1.0\n",
      "Step 63: State=[-0.17922847  0.03341517], Reward=-1.0\n",
      "Step 64: State=[-0.14696054  0.03226793], Reward=-1.0\n",
      "Step 65: State=[-0.11595355  0.03100699], Reward=-1.0\n",
      "Step 66: State=[-0.08629682  0.02965673], Reward=-1.0\n",
      "Step 67: State=[-0.05805677  0.02824004], Reward=-1.0\n",
      "Step 68: State=[-0.03127891  0.02677787], Reward=-1.0\n",
      "Step 69: State=[-0.00599004  0.02528887], Reward=-1.0\n",
      "Step 70: State=[0.01779923 0.02378927], Reward=-1.0\n",
      "Step 71: State=[0.04009206 0.02229283], Reward=-1.0\n",
      "Step 72: State=[0.06090295 0.02081089], Reward=-1.0\n",
      "Step 73: State=[0.08025546 0.01935251], Reward=-1.0\n",
      "Step 74: State=[0.09818008 0.01792462], Reward=-1.0\n",
      "Step 75: State=[0.11471236 0.01653228], Reward=-1.0\n",
      "Step 76: State=[0.12989122 0.01517886], Reward=-1.0\n",
      "Step 77: State=[0.14375749 0.01386628], Reward=-1.0\n",
      "Step 78: State=[0.15635269 0.01259519], Reward=-1.0\n",
      "Step 79: State=[0.16771789 0.01136521], Reward=-1.0\n",
      "Step 80: State=[0.17789293 0.01017504], Reward=-1.0\n",
      "Step 81: State=[0.18691562 0.00902269], Reward=-1.0\n",
      "Step 82: State=[0.19482116 0.00790554], Reward=-1.0\n",
      "Step 83: State=[0.20164168 0.00682052], Reward=-1.0\n",
      "Step 84: State=[0.20740583 0.00576416], Reward=-1.0\n",
      "Step 85: State=[0.21213852 0.00473269], Reward=-1.0\n",
      "Step 86: State=[0.21586063 0.00372211], Reward=-1.0\n",
      "Step 87: State=[0.21858888 0.00272825], Reward=-1.0\n",
      "Step 88: State=[0.22033568 0.0017468 ], Reward=-1.0\n",
      "Step 89: State=[0.22110904 0.00077336], Reward=-1.0\n",
      "Step 90: State=[ 2.20912527e-01 -1.96509754e-04], Reward=-1.0\n",
      "Step 91: State=[ 0.21774524 -0.00316729], Reward=-1.0\n",
      "Step 92: State=[ 0.21159265 -0.00615259], Reward=-1.0\n",
      "Step 93: State=[ 0.20242705 -0.0091656 ], Reward=-1.0\n",
      "Step 94: State=[ 0.19020845 -0.01221861], Reward=-1.0\n",
      "Step 95: State=[ 0.17488593 -0.01532251], Reward=-1.0\n",
      "Step 96: State=[ 0.15639968 -0.01848625], Reward=-1.0\n",
      "Step 97: State=[ 0.1346836  -0.02171608], Reward=-1.0\n",
      "Step 98: State=[ 0.10966883 -0.02501477], Reward=-1.0\n",
      "Step 99: State=[ 0.08128815 -0.02838068], Reward=-1.0\n",
      "Step 100: State=[ 0.04948145 -0.03180671], Reward=-1.0\n",
      "Step 101: State=[ 0.01420223 -0.03527921], Reward=-1.0\n",
      "Step 102: State=[-0.02457471 -0.03877695], Reward=-1.0\n",
      "Step 103: State=[-0.06684487 -0.04227015], Reward=-1.0\n",
      "Step 104: State=[-0.11256492 -0.04572006], Reward=-1.0\n",
      "Step 105: State=[-0.16164378 -0.04907886], Reward=-1.0\n",
      "Step 106: State=[-0.21393441 -0.05229063], Reward=-1.0\n",
      "Step 107: State=[-0.26922758 -0.05529317], Reward=-1.0\n",
      "Step 108: State=[-0.32724868 -0.05802111], Reward=-1.0\n",
      "Step 109: State=[-0.38765872 -0.06041004], Reward=-1.0\n",
      "Step 110: State=[-0.45006028 -0.06240156], Reward=-1.0\n",
      "Step 111: State=[-0.51400891 -0.06394863], Reward=-1.0\n",
      "Step 112: State=[-0.57902946 -0.06502055], Reward=-1.0\n",
      "Step 113: State=[-0.64463619 -0.06560673], Reward=-1.0\n",
      "Step 114: State=[-0.71035496 -0.06571877], Reward=-1.0\n",
      "Step 115: State=[-0.77574519 -0.06539023], Reward=-1.0\n",
      "Step 116: State=[-0.84041959 -0.06467439], Reward=-1.0\n",
      "Step 117: State=[-0.90405977 -0.06364018], Reward=-1.0\n",
      "Step 118: State=[-0.96642693 -0.06236716], Reward=-1.0\n",
      "Step 119: State=[-1.02736712 -0.06094019], Reward=-1.0\n",
      "Step 120: State=[-1.08681173 -0.05944462], Reward=-1.0\n",
      "Step 121: State=[-1.14477398 -0.05796225], Reward=-1.0\n",
      "Step 122: State=[-1.2  0. ], Reward=-1.0\n",
      "Step 123: State=[-1.1987581  0.0012419], Reward=-1.0\n",
      "Step 124: State=[-1.19427021  0.0044879 ], Reward=-1.0\n",
      "Step 125: State=[-1.18652173  0.00774848], Reward=-1.0\n",
      "Step 126: State=[-1.17548846  0.01103326], Reward=-1.0\n",
      "Step 127: State=[-1.16113808  0.01435038], Reward=-1.0\n",
      "Step 128: State=[-1.14343234  0.01770574], Reward=-1.0\n",
      "Step 129: State=[-1.12233007  0.02110228], Reward=-1.0\n",
      "Step 130: State=[-1.09779103  0.02453904], Reward=-1.0\n",
      "Step 131: State=[-1.06978073  0.0280103 ], Reward=-1.0\n",
      "Step 132: State=[-1.03827616  0.03150456], Reward=-1.0\n",
      "Step 133: State=[-1.0032725   0.03500367], Reward=-1.0\n",
      "Step 134: State=[-0.9647905   0.03848199], Reward=-1.0\n",
      "Step 135: State=[-0.92288452  0.04190598], Reward=-1.0\n",
      "Step 136: State=[-0.87765038  0.04523414], Reward=-1.0\n",
      "Step 137: State=[-0.82923273  0.04841765], Reward=-1.0\n",
      "Step 138: State=[-0.77783078  0.05140195], Reward=-1.0\n",
      "Step 139: State=[-0.72370164  0.05412914], Reward=-1.0\n",
      "Step 140: State=[-0.66716026  0.05654138], Reward=-1.0\n",
      "Step 141: State=[-0.60857514  0.05858511], Reward=-1.0\n",
      "Step 142: State=[-0.54835959  0.06021555], Reward=-1.0\n",
      "Step 143: State=[-0.4869585   0.06140109], Reward=-1.0\n",
      "Step 144: State=[-0.42483166  0.06212684], Reward=-1.0\n",
      "Step 145: State=[-0.36243478  0.06239688], Reward=-1.0\n",
      "Step 146: State=[-0.30020009  0.06223469], Reward=-1.0\n",
      "Step 147: State=[-0.23851824  0.06168185], Reward=-1.0\n",
      "Step 148: State=[-0.17772322  0.06079502], Reward=-1.0\n",
      "Step 149: State=[-0.1180812   0.05964202], Reward=-1.0\n",
      "Step 150: State=[-0.05978395  0.05829725], Reward=-1.0\n",
      "Step 151: State=[-0.0029466   0.05683735], Reward=-1.0\n",
      "Step 152: State=[0.05239085 0.05533745], Reward=-1.0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step 153: State=[0.10625911 0.05386826], Reward=-1.0\n",
      "Step 154: State=[0.15875332 0.05249421], Reward=-1.0\n",
      "Step 155: State=[0.21002575 0.05127242], Reward=-1.0\n",
      "Step 156: State=[0.26027822 0.05025247], Reward=-1.0\n",
      "Step 157: State=[0.30975487 0.04947665], Reward=-1.0\n",
      "Step 158: State=[0.35873547 0.0489806 ], Reward=-1.0\n",
      "Step 159: State=[0.40752939 0.04879392], Reward=-1.0\n",
      "Step 160: State=[0.45647027 0.04894088], Reward=-1.0\n",
      "Step 161: State=[0.50591109 0.04944082], Reward=-1.0\n"
     ]
    }
   ],
   "source": [
    "import gym\n",
    "\n",
    "env = gym.make(\"MountainCar-v0\")\n",
    "state = env.reset()\n",
    "done = False\n",
    "\n",
    "i = 0\n",
    "while not done:\n",
    "    i += 1\n",
    "    \n",
    "    if state[1]>0:\n",
    "        action = 2\n",
    "    else:\n",
    "        action = 0\n",
    "    \n",
    "    state, reward, done, _ = env.step(action)\n",
    "    env.render()\n",
    "    print(f\"Step {i}: State={state}, Reward={reward}\")\n",
    "    \n",
    "env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reinforcement Learning\n",
    "\n",
    "![Reinforcement Learning](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/reinforcement.png \"Reinforcement Learning\")\n",
    "\n",
    "\n",
    "### Q-Learning Car\n",
    "\n",
    "We will now use Q-Learning to produce a car that learns to drive itself.  Look out Tesla! \n",
    "\n",
    "Q-Learning works by building a table that provides a lookup table to determine which of several actions should be taken. As we move through a number of training episodes this table is refined."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$ Q^{new}(s_{t},a_{t}) \\leftarrow (1-\\alpha) \\cdot \\underbrace{Q(s_{t},a_{t})}_{\\text{old value}} + \\underbrace{\\alpha}_{\\text{learning rate}} \\cdot  \\overbrace{\\bigg( \\underbrace{r_{t}}_{\\text{reward}} + \\underbrace{\\gamma}_{\\text{discount factor}} \\cdot \\underbrace{\\max_{a}Q(s_{t+1}, a)}_{\\text{estimate of optimal future value}} \\bigg) }^{\\text{learned value}} $"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gym\n",
    "import numpy as np\n",
    "\n",
    "def calc_discrete_state(state):\n",
    "    discrete_state = (state - env.observation_space.low)/buckets\n",
    "    return tuple(discrete_state.astype(np.int))  \n",
    "\n",
    "def run_game(q_table, render, should_update):\n",
    "    done = False\n",
    "    discrete_state = calc_discrete_state(env.reset())\n",
    "    success = False\n",
    "    \n",
    "    while not done:\n",
    "        # Exploit or explore\n",
    "        if np.random.random() > epsilon:\n",
    "            # Exploit - use q-table to take current best action (and probably refine)\n",
    "            action = np.argmax(q_table[discrete_state])\n",
    "        else:\n",
    "            # Explore - t\n",
    "            action = np.random.randint(0, env.action_space.n)\n",
    "            \n",
    "        # Run simulation step\n",
    "        new_state, reward, done, _ = env.step(action)\n",
    "        \n",
    "        # \n",
    "        new_state_disc = calc_discrete_state(new_state)\n",
    "\n",
    "        # \n",
    "        if new_state[0] >= env.goal_position:\n",
    "            success = True\n",
    "          \n",
    "        # Update q-table\n",
    "        if should_update:\n",
    "            max_future_q = np.max(q_table[new_state_disc])\n",
    "            current_q = q_table[discrete_state + (action,)]\n",
    "            new_q = (1 - LEARNING_RATE) * current_q + LEARNING_RATE * (reward + DISCOUNT * max_future_q)\n",
    "            q_table[discrete_state + (action,)] = new_q\n",
    "\n",
    "        discrete_state = new_state_disc\n",
    "        \n",
    "        if render:\n",
    "            env.render()\n",
    "            \n",
    "    return success\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "LEARNING_RATE = 0.1\n",
    "DISCOUNT = 0.95\n",
    "EPISODES = 10000\n",
    "SHOW_EVERY = 1000\n",
    "\n",
    "DISCRETE_GRID_SIZE = [10, 10]\n",
    "START_EPSILON_DECAYING = 1\n",
    "END_EPSILON_DECAYING = EPISODES//2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "env = gym.make(\"MountainCar-v0\")\n",
    "\n",
    "epsilon = 1  \n",
    "epsilon_change = epsilon/(END_EPSILON_DECAYING - START_EPSILON_DECAYING)\n",
    "buckets = (env.observation_space.high - env.observation_space.low)/DISCRETE_GRID_SIZE\n",
    "q_table = np.random.uniform(low=-3, high=0, size=(DISCRETE_GRID_SIZE + [env.action_space.n]))\n",
    "success = False\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Current episode: 1000, success: 0 (0.0)\n",
      "Current episode: 2000, success: 0 (0.0)\n",
      "Current episode: 3000, success: 1 (0.001)\n",
      "Current episode: 4000, success: 51 (0.051)\n",
      "Current episode: 5000, success: 371 (0.371)\n",
      "Current episode: 6000, success: 543 (0.543)\n",
      "Current episode: 7000, success: 500 (0.5)\n",
      "Current episode: 8000, success: 472 (0.472)\n",
      "Current episode: 9000, success: 558 (0.558)\n",
      "Current episode: 10000, success: 864 (0.864)\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "episode = 0\n",
    "success_count = 0\n",
    "\n",
    "while episode<EPISODES:\n",
    "    episode+=1\n",
    "    done = False\n",
    "\n",
    "    if episode % SHOW_EVERY == 0:\n",
    "        print(f\"Current episode: {episode}, success: {success_count} ({float(success_count)/SHOW_EVERY})\")\n",
    "        success = run_game(q_table, True, False)\n",
    "        success_count = 0\n",
    "    else:\n",
    "        success = run_game(q_table, False, True)\n",
    "        \n",
    "    if success:\n",
    "        success_count += 1\n",
    "\n",
    "    # Move epsilon towards its ending value, if it still needs to move\n",
    "    if END_EPSILON_DECAYING >= episode >= START_EPSILON_DECAYING:\n",
    "        epsilon -= epsilon_change\n",
    "\n",
    "print(success)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "run_game(q_table, True, False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.DataFrame(q_table.argmax(axis=2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>v-0</th>\n",
       "      <th>v-1</th>\n",
       "      <th>v-2</th>\n",
       "      <th>v-3</th>\n",
       "      <th>v-4</th>\n",
       "      <th>v-5</th>\n",
       "      <th>v-6</th>\n",
       "      <th>v-7</th>\n",
       "      <th>v-8</th>\n",
       "      <th>v-9</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>p-0</th>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>p-1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>p-2</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>p-3</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>p-4</th>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>p-5</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>p-6</th>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>p-7</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>p-8</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>p-9</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     v-0  v-1  v-2  v-3  v-4  v-5  v-6  v-7  v-8  v-9\n",
       "p-0    2    0    2    2    2    2    2    2    1    2\n",
       "p-1    1    1    0    0    1    2    1    2    2    0\n",
       "p-2    1    1    0    0    0    2    2    2    2    2\n",
       "p-3    1    1    0    0    0    2    0    2    2    2\n",
       "p-4    2    0    0    0    0    0    2    2    0    2\n",
       "p-5    1    1    0    0    1    0    1    2    2    2\n",
       "p-6    0    2    1    1    0    1    0    2    2    0\n",
       "p-7    2    1    0    0    1    0    2    1    2    0\n",
       "p-8    1    1    0    0    0    0    0    2    0    2\n",
       "p-9    1    2    1    2    0    2    2    2    2    1"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.columns = [f'v-{x}' for x in range(DISCRETE_GRID_SIZE[0])]\n",
    "df.index = [f'p-{x}' for x in range(DISCRETE_GRID_SIZE[1])]\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.argmax(q_table[(2,0)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.6 (tensorflow)",
   "language": "python",
   "name": "rga"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}