Copy_of_unit4.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Ryukijano/Deep-Reinforcement-Learning-and-Hugging-Face/blob/master/Copy_of_unit4.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CjRWziAVU2lZ"
      },
      "source": [
        "# Unit 4: Code your first Deep Reinforcement Learning Algorithm with PyTorch: Reinforce. And test its robustness 💪\n",
        "\n",
        "<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/unit6/thumbnail.png\" alt=\"thumbnail\"/>\n",
        "\n",
        "\n",
        "In this notebook, you'll code your first Deep Reinforcement Learning algorithm from scratch: Reinforce (also called Monte Carlo Policy Gradient).\n",
        "\n",
        "Reinforce is a *Policy-based method*: a Deep Reinforcement Learning algorithm that tries **to optimize the policy directly without using an action-value function**.\n",
        "\n",
        "More precisely, Reinforce is a *Policy-gradient method*, a subclass of *Policy-based methods* that aims **to optimize the policy directly by estimating the weights of the optimal policy using gradient ascent**.\n",
        "\n",
        "To test its robustness, we're going to train it in 2 different simple environments:\n",
        "- Cartpole-v1\n",
        "- PixelcopterEnv\n",
        "\n",
        "⬇️ Here is an example of what **you will achieve at the end of this notebook.** ⬇️"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s4rBom2sbo7S"
      },
      "source": [
        "  <img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/unit6/envs.gif\" alt=\"Environments\"/>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BPLwsPajb1f8"
      },
      "source": [
        "### 🎮 Environments: \n",
        "\n",
        "- [CartPole-v1](https://www.gymlibrary.dev/environments/classic_control/cart_pole/)\n",
        "- [PixelCopter](https://pygame-learning-environment.readthedocs.io/en/latest/user/games/pixelcopter.html)\n",
        "\n",
        "### 📚 RL-Library: \n",
        "\n",
        "- Python\n",
        "- PyTorch\n",
        "\n",
        "\n",
        "We're constantly trying to improve our tutorials, so **if you find some issues in this notebook**, please [open an issue on the GitHub Repo](https://github.com/huggingface/deep-rl-class/issues)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L_WSo0VUV99t"
      },
      "source": [
        "## Objectives of this notebook 🏆\n",
        "At the end of the notebook, you will:\n",
        "- Be able to **code from scratch a Reinforce algorithm using PyTorch.**\n",
        "- Be able to **test the robustness of your agent using simple environments.**\n",
        "- Be able to **push your trained agent to the Hub** with a nice video replay and an evaluation score 🔥."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lEPrZg2eWa4R"
      },
      "source": [
        "## This notebook is from the Deep Reinforcement Learning Course\n",
        "<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/deep-rl-course-illustration.jpg\" alt=\"Deep RL Course illustration\"/>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6p5HnEefISCB"
      },
      "source": [
        "In this free course, you will:\n",
        "\n",
        "- 📖 Study Deep Reinforcement Learning in **theory and practice**.\n",
        "- 🧑‍💻 Learn to **use famous Deep RL libraries** such as Stable Baselines3, RL Baselines3 Zoo, CleanRL and Sample Factory 2.0.\n",
        "- 🤖 Train **agents in unique environments** \n",
        "\n",
        "And more check 📚 the syllabus 👉 https://simoninithomas.github.io/deep-rl-course\n",
        "\n",
        "Don’t forget to **<a href=\"http://eepurl.com/ic5ZUD\">sign up to the course</a>** (we are collecting your email to be able to **send you the links when each Unit is published and give you information about the challenges and updates).**\n",
        "\n",
        "\n",
        "The best way to keep in touch is to join our discord server to exchange with the community and with us 👉🏻 https://discord.gg/ydHrjt3WP5"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mjY-eq3eWh9O"
      },
      "source": [
        "## Prerequisites 🏗️\n",
        "Before diving into the notebook, you need to:\n",
        "\n",
        "🔲 📚 [Study Policy Gradients by reading Unit 4](https://huggingface.co/deep-rl-course/unit4/introduction)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Bsh4ZAamchSl"
      },
      "source": [
        "# Let's code Reinforce algorithm from scratch 🔥\n",
        "\n",
        "\n",
        "To validate this hands-on for the certification process, you need to push your trained models to the Hub.\n",
        "\n",
        "- Get a result of >= 350 for `Cartpole-v1`.\n",
        "- Get a result of >= 5 for `PixelCopter`.\n",
        "\n",
        "To find your result, go to the leaderboard and find your model, **the result = mean_reward - std of reward**. **If you don't see your model on the leaderboard, go at the bottom of the leaderboard page and click on the refresh button**.\n",
        "\n",
        "For more information about the certification process, check this section 👉 https://huggingface.co/deep-rl-course/en/unit0/introduction#certification-process\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JoTC9o2SczNn"
      },
      "source": [
        "## An advice 💡\n",
        "It's better to run this colab in a copy on your Google Drive, so that **if it timeouts** you still have the saved notebook on your Google Drive and do not need to fill everything from scratch.\n",
        "\n",
        "To do that you can either do `Ctrl + S` or `File > Save a copy in Google Drive.`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PU4FVzaoM6fC"
      },
      "source": [
        "## Set the GPU 💪\n",
        "- To **accelerate the agent's training, we'll use a GPU**. To do that, go to `Runtime > Change Runtime type`\n",
        "\n",
        "<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/gpu-step1.jpg\" alt=\"GPU Step 1\">"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KV0NyFdQM9ZG"
      },
      "source": [
        "- `Hardware Accelerator > GPU`\n",
        "\n",
        "<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/gpu-step2.jpg\" alt=\"GPU Step 2\">"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bTpYcVZVMzUI"
      },
      "source": [
        "## Create a virtual display 🖥\n",
        "\n",
        "During the notebook, we'll need to generate a replay video. To do so, with colab, **we need to have a virtual screen to be able to render the environment** (and thus record the frames). \n",
        "\n",
        "Hence the following cell will install the librairies and create and run a virtual screen 🖥"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jV6wjQ7Be7p5"
      },
      "outputs": [],
      "source": [
        "%%capture\n",
        "!apt install python-opengl\n",
        "!apt install ffmpeg\n",
        "!apt install xvfb\n",
        "!pip install pyvirtualdisplay\n",
        "!pip install pyglet==1.5.1"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Sr-Nuyb1dBm0",
        "outputId": "f84d6db2-4156-4839-8687-23b51c747c20"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<pyvirtualdisplay.display.Display at 0x7f607e7c6910>"
            ]
          },
          "metadata": {},
          "execution_count": 2
        }
      ],
      "source": [
        "# Virtual display\n",
        "from pyvirtualdisplay import Display\n",
        "\n",
        "virtual_display = Display(visible=0, size=(1400, 900))\n",
        "virtual_display.start()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tjrLfPFIW8XK"
      },
      "source": [
        "## Install the dependencies 🔽\n",
        "The first step is to install the dependencies. We’ll install multiple ones:\n",
        "\n",
        "- `gym`\n",
        "- `gym-games`: Extra gym environments made with PyGame.\n",
        "- `huggingface_hub`: 🤗 works as a central place where anyone can share and explore models and datasets. It has versioning, metrics, visualizations, and other features that will allow you to easily collaborate with others.\n",
        "\n",
        "You can see here all the Reinforce models available 👉 https://huggingface.co/models?other=reinforce\n",
        "\n",
        "And you can find all the Deep Reinforcement Learning models here 👉 https://huggingface.co/models?pipeline_tag=reinforcement-learning\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "e8ZVi-uydpgL",
        "outputId": "e91c2c8f-e154-4bd6-b560-100ff74b2e48"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting git+https://github.com/ntasfi/PyGame-Learning-Environment.git (from -r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit4/requirements-unit4.txt (line 2))\n",
            "  Cloning https://github.com/ntasfi/PyGame-Learning-Environment.git to /tmp/pip-req-build-8b8o59fo\n",
            "  Running command git clone --filter=blob:none --quiet https://github.com/ntasfi/PyGame-Learning-Environment.git /tmp/pip-req-build-8b8o59fo\n",
            "  Resolved https://github.com/ntasfi/PyGame-Learning-Environment.git to commit 3dbe79dc0c35559bb441b9359948aabf9bb3d331\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting git+https://github.com/qlan3/gym-games.git (from -r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit4/requirements-unit4.txt (line 3))\n",
            "  Cloning https://github.com/qlan3/gym-games.git to /tmp/pip-req-build-b1e7l7w6\n",
            "  Running command git clone --filter=blob:none --quiet https://github.com/qlan3/gym-games.git /tmp/pip-req-build-b1e7l7w6\n",
            "  Resolved https://github.com/qlan3/gym-games.git to commit fe4cc2d4e3de06c7e6593a10e4b01d3b1c5a99d6\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: gym in /usr/local/lib/python3.8/dist-packages (from -r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit4/requirements-unit4.txt (line 1)) (0.25.2)\n",
            "Collecting huggingface_hub\n",
            "  Downloading huggingface_hub-0.12.1-py3-none-any.whl (190 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m190.3/190.3 KB\u001b[0m \u001b[31m13.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting imageio-ffmpeg\n",
            "  Downloading imageio_ffmpeg-0.4.8-py3-none-manylinux2010_x86_64.whl (26.9 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m26.9/26.9 MB\u001b[0m \u001b[31m48.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: pyyaml==6.0 in /usr/local/lib/python3.8/dist-packages (from -r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit4/requirements-unit4.txt (line 6)) (6.0)\n",
            "Requirement already satisfied: numpy>=1.18.0 in /usr/local/lib/python3.8/dist-packages (from gym->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit4/requirements-unit4.txt (line 1)) (1.22.4)\n",
            "Requirement already satisfied: gym-notices>=0.0.4 in /usr/local/lib/python3.8/dist-packages (from gym->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit4/requirements-unit4.txt (line 1)) (0.0.8)\n",
            "Requirement already satisfied: cloudpickle>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from gym->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit4/requirements-unit4.txt (line 1)) (2.2.1)\n",
            "Requirement already satisfied: importlib-metadata>=4.8.0 in /usr/local/lib/python3.8/dist-packages (from gym->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit4/requirements-unit4.txt (line 1)) (6.0.0)\n",
            "Requirement already satisfied: Pillow in /usr/local/lib/python3.8/dist-packages (from ple==0.0.1->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit4/requirements-unit4.txt (line 2)) (7.1.2)\n",
            "Collecting MinAtar>=1.0.10\n",
            "  Downloading MinAtar-1.0.12-py3-none-any.whl (16 kB)\n",
            "Requirement already satisfied: setuptools>=41.0.1 in /usr/local/lib/python3.8/dist-packages (from gym-games==1.0.4->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit4/requirements-unit4.txt (line 3)) (57.4.0)\n",
            "Collecting pygame>=1.9.6\n",
            "  Downloading pygame-2.1.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (13.7 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m13.7/13.7 MB\u001b[0m \u001b[31m98.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: requests in /usr/local/lib/python3.8/dist-packages (from huggingface_hub->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit4/requirements-unit4.txt (line 4)) (2.25.1)\n",
            "Requirement already satisfied: tqdm>=4.42.1 in /usr/local/lib/python3.8/dist-packages (from huggingface_hub->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit4/requirements-unit4.txt (line 4)) (4.64.1)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.8/dist-packages (from huggingface_hub->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit4/requirements-unit4.txt (line 4)) (3.9.0)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.8/dist-packages (from huggingface_hub->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit4/requirements-unit4.txt (line 4)) (4.5.0)\n",
            "Requirement already satisfied: packaging>=20.9 in /usr/local/lib/python3.8/dist-packages (from huggingface_hub->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit4/requirements-unit4.txt (line 4)) (23.0)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.8/dist-packages (from importlib-metadata>=4.8.0->gym->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit4/requirements-unit4.txt (line 1)) (3.13.0)\n",
            "Requirement already satisfied: cycler>=0.10.0 in /usr/local/lib/python3.8/dist-packages (from MinAtar>=1.0.10->gym-games==1.0.4->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit4/requirements-unit4.txt (line 3)) (0.11.0)\n",
            "Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.8/dist-packages (from MinAtar>=1.0.10->gym-games==1.0.4->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit4/requirements-unit4.txt (line 3)) (3.0.9)\n",
            "Requirement already satisfied: seaborn>=0.9.0 in /usr/local/lib/python3.8/dist-packages (from MinAtar>=1.0.10->gym-games==1.0.4->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit4/requirements-unit4.txt (line 3)) (0.11.2)\n",
            "Requirement already satisfied: pandas>=0.24.2 in /usr/local/lib/python3.8/dist-packages (from MinAtar>=1.0.10->gym-games==1.0.4->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit4/requirements-unit4.txt (line 3)) (1.3.5)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.8/dist-packages (from MinAtar>=1.0.10->gym-games==1.0.4->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit4/requirements-unit4.txt (line 3)) (1.4.4)\n",
            "Requirement already satisfied: scipy>=1.2.1 in /usr/local/lib/python3.8/dist-packages (from MinAtar>=1.0.10->gym-games==1.0.4->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit4/requirements-unit4.txt (line 3)) (1.7.3)\n",
            "Requirement already satisfied: matplotlib>=3.0.3 in /usr/local/lib/python3.8/dist-packages (from MinAtar>=1.0.10->gym-games==1.0.4->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit4/requirements-unit4.txt (line 3)) (3.2.2)\n",
            "Requirement already satisfied: six>=1.12.0 in /usr/local/lib/python3.8/dist-packages (from MinAtar>=1.0.10->gym-games==1.0.4->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit4/requirements-unit4.txt (line 3)) (1.15.0)\n",
            "Requirement already satisfied: pytz>=2018.9 in /usr/local/lib/python3.8/dist-packages (from MinAtar>=1.0.10->gym-games==1.0.4->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit4/requirements-unit4.txt (line 3)) (2022.7.1)\n",
            "Requirement already satisfied: python-dateutil>=2.8.0 in /usr/local/lib/python3.8/dist-packages (from MinAtar>=1.0.10->gym-games==1.0.4->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit4/requirements-unit4.txt (line 3)) (2.8.2)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.8/dist-packages (from requests->huggingface_hub->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit4/requirements-unit4.txt (line 4)) (2022.12.7)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.8/dist-packages (from requests->huggingface_hub->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit4/requirements-unit4.txt (line 4)) (2.10)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.8/dist-packages (from requests->huggingface_hub->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit4/requirements-unit4.txt (line 4)) (1.24.3)\n",
            "Requirement already satisfied: chardet<5,>=3.0.2 in /usr/local/lib/python3.8/dist-packages (from requests->huggingface_hub->-r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit4/requirements-unit4.txt (line 4)) (4.0.0)\n",
            "Building wheels for collected packages: ple, gym-games\n",
            "  Building wheel for ple (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for ple: filename=ple-0.0.1-py3-none-any.whl size=50789 sha256=ba6c5168d08484c9652e09e4ed580b6f877139646c66e9cb99d630e7fa5bd7d6\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-hny7ym2g/wheels/25/91/ce/6ff81989b50d0c0fbdea57938f1d3f27ff0f9087817925900c\n",
            "  Building wheel for gym-games (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for gym-games: filename=gym_games-1.0.4-py3-none-any.whl size=17325 sha256=c65086a7ae87a74784440fb8cf618d8990b7833d5e6d343e0da015d9d986c1c8\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-hny7ym2g/wheels/7f/f5/b6/cf9b96eb6b799509d9f396fdc91fb01adc4492ad72ff34d38f\n",
            "Successfully built ple gym-games\n",
            "Installing collected packages: pygame, ple, imageio-ffmpeg, huggingface_hub, MinAtar, gym-games\n",
            "Successfully installed MinAtar-1.0.12 gym-games-1.0.4 huggingface_hub-0.12.1 imageio-ffmpeg-0.4.8 ple-0.0.1 pygame-2.1.3\n"
          ]
        }
      ],
      "source": [
        "!pip install -r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit4/requirements-unit4.txt"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AAHAq6RZW3rn"
      },
      "source": [
        "## Import the packages 📦\n",
        "In addition to import the installed libraries, we also import:\n",
        "\n",
        "- `imageio`: A library that will help us to generate a replay video\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "V8oadoJSWp7C"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "\n",
        "from collections import deque\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "%matplotlib inline\n",
        "\n",
        "# PyTorch\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.nn.functional as F\n",
        "import torch.optim as optim\n",
        "from torch.distributions import Categorical\n",
        "\n",
        "# Gym\n",
        "import gym\n",
        "import gym_pygame\n",
        "\n",
        "# Hugging Face Hub\n",
        "from huggingface_hub import notebook_login # To log to our Hugging Face account to be able to upload models to the Hub.\n",
        "import imageio"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RfxJYdMeeVgv"
      },
      "source": [
        "## Check if we have a GPU\n",
        "\n",
        "- Let's check if we have a GPU\n",
        "- If it's the case you should see `device:cuda0`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "kaJu5FeZxXGY"
      },
      "outputs": [],
      "source": [
        "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GV8qIQ0bIz-c"
      },
      "source": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "U5TNYa14aRav",
        "outputId": "dddc934a-851d-4240-9ac6-a19320955025"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "cuda:0\n"
          ]
        }
      ],
      "source": [
        "print(device)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PBPecCtBL_pZ"
      },
      "source": [
        "We're now ready to implement our Reinforce algorithm 🔥"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8KEyKYo2ZSC-"
      },
      "source": [
        "# First agent: Playing CartPole-v1 🤖"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "haLArKURMyuF"
      },
      "source": [
        "## Create the CartPole environment and understand how it works\n",
        "### [The environment 🎮](https://www.gymlibrary.dev/environments/classic_control/cart_pole/)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AH_TaLKFXo_8"
      },
      "source": [
        "### Why do we use a simple environment like CartPole-v1?\n",
        "As explained in [Reinforcement Learning Tips and Tricks](https://stable-baselines3.readthedocs.io/en/master/guide/rl_tips.html), when you implement your agent from scratch you need **to be sure that it works correctly and find bugs with easy environments before going deeper**. Since finding bugs will be much easier in simple environments.\n",
        "\n",
        "\n",
        "> Try to have some “sign of life” on toy problems\n",
        "\n",
        "\n",
        "> Validate the implementation by making it run on harder and harder envs (you can compare results against the RL zoo). You usually need to run hyperparameter optimization for that step.\n",
        "___\n",
        "### The CartPole-v1 environment\n",
        "\n",
        "> A pole is attached by an un-actuated joint to a cart, which moves along a frictionless track. The pendulum is placed upright on the cart and the goal is to balance the pole by applying forces in the left and right direction on the cart.\n",
        "\n",
        "\n",
        "\n",
        "So, we start with CartPole-v1. The goal is to push the cart left or right **so that the pole stays in the equilibrium.**\n",
        "\n",
        "The episode ends if:\n",
        "- The pole Angle is greater than ±12°\n",
        "- Cart Position is greater than ±2.4\n",
        "- Episode length is greater than 500\n",
        "\n",
        "We get a reward 💰 of +1 every timestep the Pole stays in the equilibrium."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "POOOk15_K6KA",
        "outputId": "ddaeae01-3e9c-4299-ccbf-c5696d11f795"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.8/dist-packages/gym/core.py:317: DeprecationWarning: \u001b[33mWARN: Initializing wrapper in old step API which returns one bool instead of two. It is recommended to set `new_step_api=True` to use new step API. This will be the default behaviour in future.\u001b[0m\n",
            "  deprecation(\n",
            "/usr/local/lib/python3.8/dist-packages/gym/wrappers/step_api_compatibility.py:39: DeprecationWarning: \u001b[33mWARN: Initializing environment in old step API which returns one bool instead of two. It is recommended to set `new_step_api=True` to use new step API. This will be the default behaviour in future.\u001b[0m\n",
            "  deprecation(\n"
          ]
        }
      ],
      "source": [
        "env_id = \"CartPole-v1\"\n",
        "# Create the env\n",
        "env = gym.make(env_id)\n",
        "\n",
        "# Create the evaluation env\n",
        "eval_env = gym.make(env_id)\n",
        "\n",
        "# Get the state space and action space\n",
        "s_size = env.observation_space.shape[0]\n",
        "a_size = env.action_space.n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "FMLFrjiBNLYJ",
        "outputId": "180a5340-2d04-4e29-bb2f-60e3e5fb6186"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "_____OBSERVATION SPACE_____ \n",
            "\n",
            "The State Space is:  4\n",
            "Sample observation [-4.1889806e+00  1.6700559e+38  2.6293755e-01  1.8220632e+38]\n"
          ]
        }
      ],
      "source": [
        "print(\"_____OBSERVATION SPACE_____ \\n\")\n",
        "print(\"The State Space is: \", s_size)\n",
        "print(\"Sample observation\", env.observation_space.sample()) # Get a random observation"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Lu6t4sRNNWkN",
        "outputId": "8f3696fe-537c-42bf-f36e-3ffacd237296"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            " _____ACTION SPACE_____ \n",
            "\n",
            "The Action Space is:  2\n",
            "Action Space Sample 0\n"
          ]
        }
      ],
      "source": [
        "print(\"\\n _____ACTION SPACE_____ \\n\")\n",
        "print(\"The Action Space is: \", a_size)\n",
        "print(\"Action Space Sample\", env.action_space.sample()) # Take a random action"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7SJMJj3WaFOz"
      },
      "source": [
        "## Let's build the Reinforce Architecture\n",
        "This implementation is based on two implementations:\n",
        "- [PyTorch official Reinforcement Learning example](https://github.com/pytorch/examples/blob/main/reinforcement_learning/reinforce.py)\n",
        "- [Udacity Reinforce](https://github.com/udacity/deep-reinforcement-learning/blob/master/reinforce/REINFORCE.ipynb)\n",
        "- [Improvement of the integration by Chris1nexus](https://github.com/huggingface/deep-rl-class/pull/95)\n",
        "\n",
        "<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/unit6/reinforce.png\" alt=\"Reinforce\"/>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "49kogtxBODX8"
      },
      "source": [
        "So we want:\n",
        "- Two fully connected layers (fc1 and fc2).\n",
        "- Using ReLU as activation function of fc1\n",
        "- Using Softmax to output a probability distribution over actions"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "w2LHcHhVZvPZ"
      },
      "outputs": [],
      "source": [
        "class Policy(nn.Module):\n",
        "    def __init__(self, s_size, a_size, h_size):\n",
        "        super(Policy, self).__init__()\n",
        "        self.fc1 = nn.Linear(s_size, h_size)\n",
        "        self.fc2 = nn.Linear(h_size, a_size)\n",
        "        # Create two fully connected layers\n",
        "\n",
        "\n",
        "\n",
        "    def forward(self, x):\n",
        "        x = F.relu(self.fc1(x))\n",
        "        x = self.fc2(x)\n",
        "        return F.softmax(x, dim=1)\n",
        "        # Define the forward pass\n",
        "        # state goes to fc1 then we apply ReLU activation function\n",
        "\n",
        "        # fc1 outputs goes to fc2\n",
        "\n",
        "        # We output the softmax\n",
        "    \n",
        "    def act(self, state):\n",
        "        \"\"\"\n",
        "        Given a state, take action\n",
        "        \"\"\"\n",
        "        state = torch.from_numpy(state).float().unsqueeze(0).to(device)\n",
        "        probs = self.forward(state).cpu()\n",
        "        m = Categorical(probs)\n",
        "        action = m.sample(m)\n",
        "        return action.item(), m.log_prob(action)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rOMrdwSYOWSC"
      },
      "source": [
        "### Solution"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jGdhRSVrOV4K"
      },
      "outputs": [],
      "source": [
        "class Policy(nn.Module):\n",
        "    def __init__(self, s_size, a_size, h_size):\n",
        "        super(Policy, self).__init__()\n",
        "        self.fc1 = nn.Linear(s_size, h_size)\n",
        "        self.fc2 = nn.Linear(h_size, a_size)\n",
        "\n",
        "    def forward(self, x):\n",
        "        x = F.relu(self.fc1(x))\n",
        "        x = self.fc2(x)\n",
        "        return F.softmax(x, dim=1)\n",
        "    \n",
        "    def act(self, state):\n",
        "        state = torch.from_numpy(state).float().unsqueeze(0).to(device)\n",
        "        probs = self.forward(state).cpu()\n",
        "        m = Categorical(probs)\n",
        "        action = np.argmax(m)\n",
        "        return action.item(), m.log_prob(action)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZTGWL4g2eM5B"
      },
      "source": [
        "I make a mistake, can you guess where?\n",
        "\n",
        "- To find out let's make a forward pass:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 328
        },
        "id": "lwnqGBCNePor",
        "outputId": "787848fb-1515-40bf-a916-566d52acb295"
      },
      "outputs": [
        {
          "output_type": "error",
          "ename": "ValueError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-12-f0c25c1a2bbb>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0mdebug_policy\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mPolicy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ms_size\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0ma_size\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m64\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mto\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdevice\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mdebug_policy\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mact\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0menv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreset\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;32m<ipython-input-11-9cba28d0a248>\u001b[0m in \u001b[0;36mact\u001b[0;34m(self, state)\u001b[0m\n\u001b[1;32m     15\u001b[0m         \u001b[0mm\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mCategorical\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mprobs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     16\u001b[0m         \u001b[0maction\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0margmax\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mm\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 17\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0maction\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mitem\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mm\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlog_prob\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0maction\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;32m/usr/local/lib/python3.8/dist-packages/torch/distributions/categorical.py\u001b[0m in \u001b[0;36mlog_prob\u001b[0;34m(self, value)\u001b[0m\n\u001b[1;32m    121\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mlog_prob\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    122\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_validate_args\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 123\u001b[0;31m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_validate_sample\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvalue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    124\u001b[0m         \u001b[0mvalue\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mvalue\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlong\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0munsqueeze\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    125\u001b[0m         \u001b[0mvalue\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlog_pmf\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbroadcast_tensors\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvalue\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlogits\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.8/dist-packages/torch/distributions/distribution.py\u001b[0m in \u001b[0;36m_validate_sample\u001b[0;34m(self, value)\u001b[0m\n\u001b[1;32m    269\u001b[0m         \"\"\"\n\u001b[1;32m    270\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvalue\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTensor\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 271\u001b[0;31m             \u001b[0;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'The value argument to log_prob must be a Tensor'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    272\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    273\u001b[0m         \u001b[0mevent_dim_start\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvalue\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_event_shape\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mValueError\u001b[0m: The value argument to log_prob must be a Tensor"
          ]
        }
      ],
      "source": [
        "debug_policy = Policy(s_size, a_size, 64).to(device)\n",
        "debug_policy.act(env.reset())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "14UYkoxCPaor"
      },
      "source": [
        "- Here we see that the error says `ValueError: The value argument to log_prob must be a Tensor`\n",
        "\n",
        "- It means that `action` in `m.log_prob(action)` must be a Tensor **but it's not.**\n",
        "\n",
        "- Do you know why? Check the act function and try to see why it does not work. \n",
        "\n",
        "Advice 💡: Something is wrong in this implementation. Remember that we act function **we want to sample an action from the probability distribution over actions**.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gfGJNZBUP7Vn"
      },
      "source": [
        "### (Real) Solution"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Ho_UHf49N9i4"
      },
      "outputs": [],
      "source": [
        "class Policy(nn.Module):\n",
        "    def __init__(self, s_size, a_size, h_size):\n",
        "        super(Policy, self).__init__()\n",
        "        self.fc1 = nn.Linear(s_size, h_size)\n",
        "        self.fc2 = nn.Linear(h_size, a_size)\n",
        "\n",
        "    def forward(self, x):\n",
        "        x = F.relu(self.fc1(x))\n",
        "        x = self.fc2(x)\n",
        "        return F.softmax(x, dim=1)\n",
        "    \n",
        "    def act(self, state):\n",
        "        state = torch.from_numpy(state).float().unsqueeze(0).to(device)\n",
        "        probs = self.forward(state).cpu()\n",
        "        m = Categorical(probs)\n",
        "        action = m.sample()\n",
        "        return action.item(), m.log_prob(action)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rgJWQFU_eUYw"
      },
      "source": [
        "By using CartPole, it was easier to debug since **we know that the bug comes from our integration and not from our simple environment**."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c-20i7Pk0l1T"
      },
      "source": [
        "- Since **we want to sample an action from the probability distribution over actions**, we can't use `action = np.argmax(m)` since it will always output the action that have the highest probability.\n",
        "\n",
        "- We need to replace with `action = m.sample()` that will sample an action from the probability distribution P(.|s)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4MXoqetzfIoW"
      },
      "source": [
        "### Let's build the Reinforce Training Algorithm\n",
        "This is the Reinforce algorithm pseudocode:\n",
        "\n",
        "<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/unit6/pg_pseudocode.png\" alt=\"Policy gradient pseudocode\"/>\n",
        "  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QmcXG-9i2Qu2"
      },
      "source": [
        "- When we calculate the return Gt (line 6) we see that we calculate the sum of discounted rewards **starting at timestep t**.\n",
        "\n",
        "- Why? Because our policy should only **reinforce actions on the basis of the consequences**: so rewards obtained before taking an action are useless (since they were not because of the action), **only the ones that come after the action matters**.\n",
        "\n",
        "- Before coding this you should read this section [don't let the past distract you](https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html#don-t-let-the-past-distract-you) that explains why we use reward-to-go policy gradient.\n",
        "\n",
        "We use an interesting technique coded by [Chris1nexus](https://github.com/Chris1nexus) to **compute the return at each timestep efficiently**. The comments explained the procedure. Don't hesitate also [to check the PR explanation](https://github.com/huggingface/deep-rl-class/pull/95)\n",
        "But overall the idea is to **compute the return at each timestep efficiently**."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O554nUGPpcoq"
      },
      "source": [
        "The second question you may ask is **why do we minimize the loss**? You talked about Gradient Ascent not Gradient Descent?\n",
        "\n",
        "- We want to maximize our utility function $J(\\theta)$ but in PyTorch like in Tensorflow it's better to **minimize an objective function.**\n",
        "    - So let's say we want to reinforce action 3 at a certain timestep. Before training this action P is 0.25.\n",
        "    - So we want to modify $\\theta$ such that $\\pi_\\theta(a_3|s; \\theta) > 0.25$\n",
        "    - Because all P must sum to 1, max $\\pi_\\theta(a_3|s; \\theta)$ will **minimize other action probability.**\n",
        "    - So we should tell PyTorch **to min $1 - \\pi_\\theta(a_3|s; \\theta)$.**\n",
        "    - This loss function approaches 0 as $\\pi_\\theta(a_3|s; \\theta)$ nears 1.\n",
        "    - So we are encouraging the gradient to max $\\pi_\\theta(a_3|s; \\theta)$\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "iOdv8Q9NfLK7"
      },
      "outputs": [],
      "source": [
        "def reinforce(policy, optimizer, n_training_episodes, max_t, gamma, print_every):\n",
        "    # Help us to calculate the score during the training\n",
        "    scores_deque = deque(maxlen=100)\n",
        "    scores = []\n",
        "    # Line 3 of pseudocode\n",
        "    for i_episode in range(1, n_training_episodes+1):\n",
        "        saved_log_probs = []\n",
        "        rewards = []\n",
        "        state = env.reset()\n",
        "        # Line 4 of pseudocode\n",
        "        for t in range(max_t):\n",
        "            action, log_prob = policy.act(state)\n",
        "            saved_log_probs.append(log_prob)\n",
        "            state, reward, done, _ = env.step(action)\n",
        "            rewards.append(reward)\n",
        "            if done:\n",
        "                break \n",
        "        scores_deque.append(sum(rewards))\n",
        "        scores.append(sum(rewards))\n",
        "        \n",
        "        # Line 6 of pseudocode: calculate the return\n",
        "        returns = deque(maxlen=max_t) \n",
        "        n_steps = len(rewards) \n",
        "        # Compute the discounted returns at each timestep,\n",
        "        # as the sum of the gamma-discounted return at time t (G_t) + the reward at time t\n",
        "        \n",
        "        # In O(N) time, where N is the number of time steps\n",
        "        # (this definition of the discounted return G_t follows the definition of this quantity \n",
        "        # shown at page 44 of Sutton&Barto 2017 2nd draft)\n",
        "        # G_t = r_(t+1) + r_(t+2) + ...\n",
        "        \n",
        "        # Given this formulation, the returns at each timestep t can be computed \n",
        "        # by re-using the computed future returns G_(t+1) to compute the current return G_t\n",
        "        # G_t = r_(t+1) + gamma*G_(t+1)\n",
        "        # G_(t-1) = r_t + gamma* G_t\n",
        "        # (this follows a dynamic programming approach, with which we memorize solutions in order \n",
        "        # to avoid computing them multiple times)\n",
        "        \n",
        "        # This is correct since the above is equivalent to (see also page 46 of Sutton&Barto 2017 2nd draft)\n",
        "        # G_(t-1) = r_t + gamma*r_(t+1) + gamma*gamma*r_(t+2) + ...\n",
        "        \n",
        "        \n",
        "        ## Given the above, we calculate the returns at timestep t as: \n",
        "        #               gamma[t] * return[t] + reward[t]\n",
        "        #\n",
        "        ## We compute this starting from the last timestep to the first, in order\n",
        "        ## to employ the formula presented above and avoid redundant computations that would be needed \n",
        "        ## if we were to do it from first to last.\n",
        "        \n",
        "        ## Hence, the queue \"returns\" will hold the returns in chronological order, from t=0 to t=n_steps\n",
        "        ## thanks to the appendleft() function which allows to append to the position 0 in constant time O(1)\n",
        "        ## a normal python list would instead require O(N) to do this.\n",
        "        for t in range(n_steps)[::-1]:\n",
        "            disc_return_t = (returns[0] if len(returns)>0 else 0)\n",
        "            returns.appendleft(gamma * disc_return_t + rewards[t]    ) # TODO: complete here        \n",
        "       \n",
        "        ## standardization of the returns is employed to make training more stable\n",
        "        eps = np.finfo(np.float32).eps.item()\n",
        "        \n",
        "        ## eps is the smallest representable float, which is \n",
        "        # added to the standard deviation of the returns to avoid numerical instabilities\n",
        "        returns = torch.tensor(returns)\n",
        "        returns = (returns - returns.mean()) / (returns.std() + eps)\n",
        "        \n",
        "        # Line 7:\n",
        "        policy_loss = []\n",
        "        for log_prob, disc_return in zip(saved_log_probs, returns):\n",
        "            policy_loss.append(-log_prob * disc_return)\n",
        "        policy_loss = torch.cat(policy_loss).sum()\n",
        "        \n",
        "        # Line 8: PyTorch prefers gradient descent \n",
        "        optimizer.zero_grad()\n",
        "        policy_loss.backward()\n",
        "        optimizer.step()\n",
        "        \n",
        "        if i_episode % print_every == 0:\n",
        "            print('Episode {}\\tAverage Score: {:.2f}'.format(i_episode, np.mean(scores_deque)))\n",
        "        \n",
        "    return scores"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YB0Cxrw1StrP"
      },
      "source": [
        "#### Solution"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NCNvyElRStWG"
      },
      "outputs": [],
      "source": [
        "def reinforce(policy, optimizer, n_training_episodes, max_t, gamma, print_every):\n",
        "    # Help us to calculate the score during the training\n",
        "    scores_deque = deque(maxlen=100)\n",
        "    scores = []\n",
        "    # Line 3 of pseudocode\n",
        "    for i_episode in range(1, n_training_episodes+1):\n",
        "        saved_log_probs = []\n",
        "        rewards = []\n",
        "        state = env.reset()\n",
        "        # Line 4 of pseudocode\n",
        "        for t in range(max_t):\n",
        "            action, log_prob = policy.act(state)\n",
        "            saved_log_probs.append(log_prob)\n",
        "            state, reward, done, _ = env.step(action)\n",
        "            rewards.append(reward)\n",
        "            if done:\n",
        "                break \n",
        "        scores_deque.append(sum(rewards))\n",
        "        scores.append(sum(rewards))\n",
        "        \n",
        "        # Line 6 of pseudocode: calculate the return\n",
        "        returns = deque(maxlen=max_t) \n",
        "        n_steps = len(rewards) \n",
        "        # Compute the discounted returns at each timestep,\n",
        "        # as \n",
        "        #      the sum of the gamma-discounted return at time t (G_t) + the reward at time t\n",
        "        #\n",
        "        # In O(N) time, where N is the number of time steps\n",
        "        # (this definition of the discounted return G_t follows the definition of this quantity \n",
        "        # shown at page 44 of Sutton&Barto 2017 2nd draft)\n",
        "        # G_t = r_(t+1) + r_(t+2) + ...\n",
        "        \n",
        "        # Given this formulation, the returns at each timestep t can be computed \n",
        "        # by re-using the computed future returns G_(t+1) to compute the current return G_t\n",
        "        # G_t = r_(t+1) + gamma*G_(t+1)\n",
        "        # G_(t-1) = r_t + gamma* G_t\n",
        "        # (this follows a dynamic programming approach, with which we memorize solutions in order \n",
        "        # to avoid computing them multiple times)\n",
        "        \n",
        "        # This is correct since the above is equivalent to (see also page 46 of Sutton&Barto 2017 2nd draft)\n",
        "        # G_(t-1) = r_t + gamma*r_(t+1) + gamma*gamma*r_(t+2) + ...\n",
        "        \n",
        "        \n",
        "        ## Given the above, we calculate the returns at timestep t as: \n",
        "        #               gamma[t] * return[t] + reward[t]\n",
        "        #\n",
        "        ## We compute this starting from the last timestep to the first, in order\n",
        "        ## to employ the formula presented above and avoid redundant computations that would be needed \n",
        "        ## if we were to do it from first to last.\n",
        "        \n",
        "        ## Hence, the queue \"returns\" will hold the returns in chronological order, from t=0 to t=n_steps\n",
        "        ## thanks to the appendleft() function which allows to append to the position 0 in constant time O(1)\n",
        "        ## a normal python list would instead require O(N) to do this.\n",
        "        for t in range(n_steps)[::-1]:\n",
        "            disc_return_t = (returns[0] if len(returns)>0 else 0)\n",
        "            returns.appendleft( gamma*disc_return_t + rewards[t]   )    \n",
        "            \n",
        "        ## standardization of the returns is employed to make training more stable\n",
        "        eps = np.finfo(np.float32).eps.item()\n",
        "        ## eps is the smallest representable float, which is \n",
        "        # added to the standard deviation of the returns to avoid numerical instabilities        \n",
        "        returns = torch.tensor(returns)\n",
        "        returns = (returns - returns.mean()) / (returns.std() + eps)\n",
        "        \n",
        "        # Line 7:\n",
        "        policy_loss = []\n",
        "        for log_prob, disc_return in zip(saved_log_probs, returns):\n",
        "            policy_loss.append(-log_prob * disc_return)\n",
        "        policy_loss = torch.cat(policy_loss).sum()\n",
        "        \n",
        "        # Line 8: PyTorch prefers gradient descent \n",
        "        optimizer.zero_grad()\n",
        "        policy_loss.backward()\n",
        "        optimizer.step()\n",
        "        \n",
        "        if i_episode % print_every == 0:\n",
        "            print('Episode {}\\tAverage Score: {:.2f}'.format(i_episode, np.mean(scores_deque)))\n",
        "        \n",
        "    return scores"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RIWhQyJjfpEt"
      },
      "source": [
        "##  Train it\n",
        "- We're now ready to train our agent.\n",
        "- But first, we define a variable containing all the training hyperparameters.\n",
        "- You can change the training parameters (and should 😉)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "utRe1NgtVBYF"
      },
      "outputs": [],
      "source": [
        "cartpole_hyperparameters = {\n",
        "    \"h_size\": 64,\n",
        "    \"n_training_episodes\": 1000,\n",
        "    \"n_evaluation_episodes\": 10,\n",
        "    \"max_t\": 1000,\n",
        "    \"gamma\": 0.9,\n",
        "    \"lr\": 1e-2,\n",
        "    \"env_id\": env_id,\n",
        "    \"state_space\": s_size,\n",
        "    \"action_space\": a_size,\n",
        "}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "D3lWyVXBVfl6"
      },
      "outputs": [],
      "source": [
        "# Create policy and place it to the device\n",
        "cartpole_policy = Policy(cartpole_hyperparameters[\"state_space\"], cartpole_hyperparameters[\"action_space\"], cartpole_hyperparameters[\"h_size\"]).to(device)\n",
        "cartpole_optimizer = optim.Adam(cartpole_policy.parameters(), lr=cartpole_hyperparameters[\"lr\"])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uGf-hQCnfouB",
        "outputId": "92095fdb-51fc-4be6-d10e-06b08de5e474"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Episode 100\tAverage Score: 70.96\n",
            "Episode 200\tAverage Score: 258.98\n",
            "Episode 300\tAverage Score: 98.08\n",
            "Episode 400\tAverage Score: 135.79\n",
            "Episode 500\tAverage Score: 190.91\n",
            "Episode 600\tAverage Score: 205.76\n",
            "Episode 700\tAverage Score: 498.72\n",
            "Episode 800\tAverage Score: 500.00\n",
            "Episode 900\tAverage Score: 260.37\n",
            "Episode 1000\tAverage Score: 491.48\n"
          ]
        }
      ],
      "source": [
        "scores = reinforce(cartpole_policy,\n",
        "                   cartpole_optimizer,\n",
        "                   cartpole_hyperparameters[\"n_training_episodes\"], \n",
        "                   cartpole_hyperparameters[\"max_t\"],\n",
        "                   cartpole_hyperparameters[\"gamma\"], \n",
        "                   100)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Qajj2kXqhB3g"
      },
      "source": [
        "## Define evaluation method 📝\n",
        "- Here we define the evaluation method that we're going to use to test our Reinforce agent."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3FamHmxyhBEU"
      },
      "outputs": [],
      "source": [
        "def evaluate_agent(env, max_steps, n_eval_episodes, policy):\n",
        "  \"\"\"\n",
        "  Evaluate the agent for ``n_eval_episodes`` episodes and returns average reward and std of reward.\n",
        "  :param env: The evaluation environment\n",
        "  :param n_eval_episodes: Number of episode to evaluate the agent\n",
        "  :param policy: The Reinforce agent\n",
        "  \"\"\"\n",
        "  episode_rewards = []\n",
        "  for episode in range(n_eval_episodes):\n",
        "    state = env.reset()\n",
        "    step = 0\n",
        "    done = False\n",
        "    total_rewards_ep = 0\n",
        "    \n",
        "    for step in range(max_steps):\n",
        "      action, _ = policy.act(state)\n",
        "      new_state, reward, done, info = env.step(action)\n",
        "      total_rewards_ep += reward\n",
        "        \n",
        "      if done:\n",
        "        break\n",
        "      state = new_state\n",
        "    episode_rewards.append(total_rewards_ep)\n",
        "  mean_reward = np.mean(episode_rewards)\n",
        "  std_reward = np.std(episode_rewards)\n",
        "\n",
        "  return mean_reward, std_reward"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xdH2QCrLTrlT"
      },
      "source": [
        "## Evaluate our agent 📈"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 217
        },
        "id": "ohGSXDyHh0xx",
        "outputId": "6b6c39a0-2023-45a4-e1e5-718e628b8b4b"
      },
      "outputs": [
        {
          "ename": "NameError",
          "evalue": "ignored",
          "output_type": "error",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-30-4e6329a50d81>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m evaluate_agent(eval_env, \n\u001b[0;32m----> 2\u001b[0;31m                \u001b[0mcartpole_hyperparameters\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"max_t\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      3\u001b[0m                \u001b[0mcartpole_hyperparameters\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"n_evaluation_episodes\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m                cartpole_policy)\n",
            "\u001b[0;31mNameError\u001b[0m: name 'cartpole_hyperparameters' is not defined"
          ]
        }
      ],
      "source": [
        "evaluate_agent(eval_env, \n",
        "               cartpole_hyperparameters[\"max_t\"], \n",
        "               cartpole_hyperparameters[\"n_evaluation_episodes\"],\n",
        "               cartpole_policy)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7CoeLkQ7TpO8"
      },
      "source": [
        "### Publish our trained model on the Hub 🔥\n",
        "Now that we saw we got good results after the training, we can publish our trained model on the hub 🤗 with one line of code.\n",
        "\n",
        "Here's an example of a Model Card:\n",
        "\n",
        "<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/unit6/modelcard.png\"/>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Jmhs1k-cftIq"
      },
      "source": [
        "### Push to the Hub\n",
        "#### Do not modify this code"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "LIVsvlW_8tcw"
      },
      "outputs": [],
      "source": [
        "from huggingface_hub import HfApi, snapshot_download\n",
        "from huggingface_hub.repocard import metadata_eval_result, metadata_save\n",
        "\n",
        "from pathlib import Path\n",
        "import datetime\n",
        "import json\n",
        "import imageio\n",
        "\n",
        "import tempfile\n",
        "\n",
        "import os"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Lo4JH45if81z"
      },
      "outputs": [],
      "source": [
        "def record_video(env, policy, out_directory, fps=30):\n",
        "  \"\"\"\n",
        "  Generate a replay video of the agent\n",
        "  :param env\n",
        "  :param Qtable: Qtable of our agent\n",
        "  :param out_directory\n",
        "  :param fps: how many frame per seconds (with taxi-v3 and frozenlake-v1 we use 1)\n",
        "  \"\"\"\n",
        "  images = []  \n",
        "  done = False\n",
        "  state = env.reset()\n",
        "  img = env.render(mode='rgb_array')\n",
        "  images.append(img)\n",
        "  while not done:\n",
        "    # Take the action (index) that have the maximum expected future reward given that state\n",
        "    action, _ = policy.act(state)\n",
        "    state, reward, done, info = env.step(action) # We directly put next_state = state for recording logic\n",
        "    img = env.render(mode='rgb_array')\n",
        "    images.append(img)\n",
        "  imageio.mimsave(out_directory, [np.array(img) for i, img in enumerate(images)], fps=fps)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_TPdq47D7_f_"
      },
      "outputs": [],
      "source": [
        "def push_to_hub(repo_id, \n",
        "                model,\n",
        "                hyperparameters,\n",
        "                eval_env,\n",
        "                video_fps=30\n",
        "                ):\n",
        "  \"\"\"\n",
        "  Evaluate, Generate a video and Upload a model to Hugging Face Hub.\n",
        "  This method does the complete pipeline:\n",
        "  - It evaluates the model\n",
        "  - It generates the model card\n",
        "  - It generates a replay video of the agent\n",
        "  - It pushes everything to the Hub\n",
        "\n",
        "  :param repo_id: repo_id: id of the model repository from the Hugging Face Hub\n",
        "  :param model: the pytorch model we want to save\n",
        "  :param hyperparameters: training hyperparameters\n",
        "  :param eval_env: evaluation environment\n",
        "  :param video_fps: how many frame per seconds to record our video replay \n",
        "  \"\"\"\n",
        "\n",
        "  _, repo_name = repo_id.split(\"/\")\n",
        "  api = HfApi()\n",
        "  \n",
        "  # Step 1: Create the repo\n",
        "  repo_url = api.create_repo(\n",
        "        repo_id=repo_id,\n",
        "        exist_ok=True,\n",
        "  )\n",
        "\n",
        "  with tempfile.TemporaryDirectory() as tmpdirname:\n",
        "    local_directory = Path(tmpdirname)\n",
        "  \n",
        "    # Step 2: Save the model\n",
        "    torch.save(model, local_directory / \"model.pt\")\n",
        "\n",
        "    # Step 3: Save the hyperparameters to JSON\n",
        "    with open(local_directory / \"hyperparameters.json\", \"w\") as outfile:\n",
        "      json.dump(hyperparameters, outfile)\n",
        "    \n",
        "    # Step 4: Evaluate the model and build JSON\n",
        "    mean_reward, std_reward = evaluate_agent(eval_env, \n",
        "                                            hyperparameters[\"max_t\"],\n",
        "                                            hyperparameters[\"n_evaluation_episodes\"], \n",
        "                                            model)\n",
        "    # Get datetime\n",
        "    eval_datetime = datetime.datetime.now()\n",
        "    eval_form_datetime = eval_datetime.isoformat()\n",
        "\n",
        "    evaluate_data = {\n",
        "          \"env_id\": hyperparameters[\"env_id\"], \n",
        "          \"mean_reward\": mean_reward,\n",
        "          \"n_evaluation_episodes\": hyperparameters[\"n_evaluation_episodes\"],\n",
        "          \"eval_datetime\": eval_form_datetime,\n",
        "    }\n",
        "\n",
        "    # Write a JSON file\n",
        "    with open(local_directory / \"results.json\", \"w\") as outfile:\n",
        "        json.dump(evaluate_data, outfile)\n",
        "\n",
        "    # Step 5: Create the model card\n",
        "    env_name = hyperparameters[\"env_id\"]\n",
        "    \n",
        "    metadata = {}\n",
        "    metadata[\"tags\"] = [\n",
        "          env_name,\n",
        "          \"reinforce\",\n",
        "          \"reinforcement-learning\",\n",
        "          \"custom-implementation\",\n",
        "          \"deep-rl-class\"\n",
        "      ]\n",
        "\n",
        "    # Add metrics\n",
        "    eval = metadata_eval_result(\n",
        "        model_pretty_name=repo_name,\n",
        "        task_pretty_name=\"reinforcement-learning\",\n",
        "        task_id=\"reinforcement-learning\",\n",
        "        metrics_pretty_name=\"mean_reward\",\n",
        "        metrics_id=\"mean_reward\",\n",
        "        metrics_value=f\"{mean_reward:.2f} +/- {std_reward:.2f}\",\n",
        "        dataset_pretty_name=env_name,\n",
        "        dataset_id=env_name,\n",
        "      )\n",
        "\n",
        "    # Merges both dictionaries\n",
        "    metadata = {**metadata, **eval}\n",
        "\n",
        "    model_card = f\"\"\"\n",
        "  # **Reinforce** Agent playing **{env_id}**\n",
        "  This is a trained model of a **Reinforce** agent playing **{env_id}** .\n",
        "  To learn to use this model and train yours check Unit 4 of the Deep Reinforcement Learning Course: https://huggingface.co/deep-rl-course/unit4/introduction\n",
        "  \"\"\"\n",
        "\n",
        "    readme_path = local_directory / \"README.md\"\n",
        "    readme = \"\"\n",
        "    if readme_path.exists():\n",
        "        with readme_path.open(\"r\", encoding=\"utf8\") as f:\n",
        "          readme = f.read()\n",
        "    else:\n",
        "      readme = model_card\n",
        "\n",
        "    with readme_path.open(\"w\", encoding=\"utf-8\") as f:\n",
        "      f.write(readme)\n",
        "\n",
        "    # Save our metrics to Readme metadata\n",
        "    metadata_save(readme_path, metadata)\n",
        "\n",
        "    # Step 6: Record a video\n",
        "    video_path =  local_directory / \"replay.mp4\"\n",
        "    record_video(env, model, video_path, video_fps)\n",
        "\n",
        "    # Step 7. Push everything to the Hub\n",
        "    api.upload_folder(\n",
        "          repo_id=repo_id,\n",
        "          folder_path=local_directory,\n",
        "          path_in_repo=\".\",\n",
        "    )\n",
        "\n",
        "    print(f\"Your model is pushed to the Hub. You can view your model here: {repo_url}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w17w8CxzoURM"
      },
      "source": [
        "### .\n",
        "\n",
        "By using `push_to_hub` **you evaluate, record a replay, generate a model card of your agent and push it to the Hub**.\n",
        "\n",
        "This way:\n",
        "- You can **showcase our work** 🔥\n",
        "- You can **visualize your agent playing** 👀\n",
        "- You can **share with the community an agent that others can use** 💾\n",
        "- You can **access a leaderboard 🏆 to see how well your agent is performing compared to your classmates** 👉 https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cWnFC0iZooTw"
      },
      "source": [
        "To be able to share your model with the community there are three more steps to follow:\n",
        "\n",
        "1️⃣ (If it's not already done) create an account to HF ➡ https://huggingface.co/join\n",
        "\n",
        "2️⃣ Sign in and then, you need to store your authentication token from the Hugging Face website.\n",
        "- Create a new token (https://huggingface.co/settings/tokens) **with write role**\n",
        "\n",
        "\n",
        "<img src=\"https://huggingface.co/datasets/huggingface-deep-rl-course/course-images/resolve/main/en/notebooks/create-token.jpg\" alt=\"Create HF Token\">\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 331,
          "referenced_widgets": [
            "4ae384cc0d4342febb5631e387d39d26",
            "4116b0e57cb74fcd8389ef189f07df2c",
            "1ec0e63f09dc4421b2a8c586bc4afb20",
            "7388dbe9640244a4a41d247943d3dfe4",
            "62e2030bd7e845e5846a2e2debddf545",
            "fda396519ae94fbb8f44caa2d275054d",
            "66b14d0f62f34f17ab7c2e97e24dba5a",
            "083a463baab34e28bf2f7de4f57a021b",
            "4feb15f0899a4dd1b0bbdfdc71f949ef",
            "279be38d6f514307b47be5c0e8b62d9c",
            "891fdc2c0f4f42bca1a7ca2f4596af8d",
            "ff4575f7cabf4f2399521a17b3a35fef",
            "c99c3757612c4bb586850dce41ea68d4",
            "947518c1a8e34004bde5b48490ce0a55",
            "bd3ef37fd5494aad9327081ea03a30c2",
            "2cc4f55956134a83a866ec3363b7b1c3",
            "6f13178da385404d83ea00779f9c7d41"
          ]
        },
        "id": "QB5nIcxR8paT",
        "outputId": "84545c4e-2c39-4945-8092-cc02c09f8a81"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Token is valid.\n",
            "Your token has been saved in your configured git credential helpers (store).\n",
            "Your token has been saved to /root/.cache/huggingface/token\n",
            "Login successful\n"
          ]
        }
      ],
      "source": [
        "notebook_login()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GyWc1x3-o3xG"
      },
      "source": [
        "If you don't want to use a Google Colab or a Jupyter Notebook, you need to use this command instead: `huggingface-cli login` (or `login`)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F-D-zhbRoeOm"
      },
      "source": [
        "3️⃣ We're now ready to push our trained agent to the 🤗 Hub 🔥 using `package_to_hub()` function"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 188,
          "referenced_widgets": [
            "7bedf0974ee64d3190f0554a247cfe10",
            "8e4a100e53e24d38966ba53b48e2dd95",
            "3aab122bb14f4468a51d5ddebd47b09c",
            "d2e827af1afa4bb9bffad58f97da09c2",
            "8ebb8fe869ae43ecb186c2411fbfe4c5",
            "b6a2b27c0309484aba3d4d89b55efbde",
            "ca0efeea2bc64d53a97c581613765707",
            "b9e01ea472444827a30b9fd2698106f5",
            "0528f2ac4256491d98ee1c2ab2aa7850",
            "2c5cd55508b24739967b0cd4121f96d7",
            "d5a51687ecb341ee87fc345eed6e498b",
            "596451d013c644e7b8d1dd9d589e4d84",
            "0cdea780ba2e469c89d24ecb2cbe8b23",
            "b17c6bfac1ef40f79d3630148b2df8b1",
            "de27f63695ac40a6993058729d95cba0",
            "d6cb5a48471e4d5fb93229b10672ffa2",
            "00ec48b5d37a4f41a82db8ac8787174e",
            "008a7c260e6b46b38d8ddfc69615bb06",
            "b8ebf0194d0e45d49a105e322d75cb47",
            "3df3360090804c7da4ebee660f14bfdf",
            "b1dc2bb4acba49479ac83d17b7a3b790",
            "0e7874013c314b42bf42e474c5978bcb"
          ]
        },
        "id": "UNwkTS65Uq3Q",
        "outputId": "39d20720-a524-4c9b-ca2d-a17f69422901"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.8/dist-packages/gym/core.py:43: DeprecationWarning: \u001b[33mWARN: The argument mode in render method is deprecated; use render_mode during environment initialization instead.\n",
            "See here for more information: https://www.gymlibrary.ml/content/api/\u001b[0m\n",
            "  deprecation(\n",
            "WARNING:imageio_ffmpeg:IMAGEIO FFMPEG_WRITER WARNING: input image is not divisible by macro_block_size=16, resizing from (600, 400) to (608, 400) to ensure video compatibility with most codecs and players. To prevent resizing, make your input image divisible by the macro_block_size or set the macro_block_size to 1 (risking incompatibility).\n"
          ]
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "7bedf0974ee64d3190f0554a247cfe10",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Upload 1 LFS files:   0%|          | 0/1 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "596451d013c644e7b8d1dd9d589e4d84",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "model.pt:   0%|          | 0.00/3.92k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Your model is pushed to the Hub. You can view your model here: https://huggingface.co/Ryukijano/Reinforce_pole_1\n"
          ]
        }
      ],
      "source": [
        "repo_id = \"Ryukijano/Reinforce_pole_1\" #TODO Define your repo id {username/Reinforce-{model-id}}\n",
        "push_to_hub(repo_id,\n",
        "                cartpole_policy, # The model we want to save\n",
        "                cartpole_hyperparameters, # Hyperparameters\n",
        "                eval_env, # Evaluation environment\n",
        "                video_fps=30\n",
        "                )"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jrnuKH1gYZSz"
      },
      "source": [
        "Now that we try the robustness of our implementation, let's try a more complex environment: PixelCopter 🚁\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JNLVmKKVKA6j"
      },
      "source": [
        "## Second agent: PixelCopter 🚁\n",
        "\n",
        "### Study the PixelCopter environment 👀\n",
        "- [The Environment documentation](https://pygame-learning-environment.readthedocs.io/en/latest/user/games/pixelcopter.html)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JBSc8mlfyin3",
        "outputId": "257cc829-c845-4b43-8453-69fa46d8ddc7"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "couldn't import doomish\n",
            "Couldn't import doom\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.8/dist-packages/gym/core.py:317: DeprecationWarning: \u001b[33mWARN: Initializing wrapper in old step API which returns one bool instead of two. It is recommended to set `new_step_api=True` to use new step API. This will be the default behaviour in future.\u001b[0m\n",
            "  deprecation(\n",
            "/usr/local/lib/python3.8/dist-packages/gym/wrappers/step_api_compatibility.py:39: DeprecationWarning: \u001b[33mWARN: Initializing environment in old step API which returns one bool instead of two. It is recommended to set `new_step_api=True` to use new step API. This will be the default behaviour in future.\u001b[0m\n",
            "  deprecation(\n"
          ]
        }
      ],
      "source": [
        "env_id = \"Pixelcopter-PLE-v0\"\n",
        "env = gym.make(env_id)\n",
        "eval_env = gym.make(env_id)\n",
        "s_size = env.observation_space.shape[0]\n",
        "a_size = env.action_space.n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "L5u_zAHsKBy7",
        "outputId": "f453fc8c-aad2-4b99-b4a5-d45d7954b5a8"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "_____OBSERVATION SPACE_____ \n",
            "\n",
            "The State Space is:  7\n",
            "Sample observation [-0.00546064 -0.65576017  0.12191187 -0.05948507 -1.1834902  -0.30833802\n",
            "  1.4818546 ]\n"
          ]
        }
      ],
      "source": [
        "print(\"_____OBSERVATION SPACE_____ \\n\")\n",
        "print(\"The State Space is: \", s_size)\n",
        "print(\"Sample observation\", env.observation_space.sample()) # Get a random observation"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "D7yJM9YXKNbq",
        "outputId": "d0573e9a-0ce1-4147-d98a-abd18b82e764"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            " _____ACTION SPACE_____ \n",
            "\n",
            "The Action Space is:  2\n",
            "Action Space Sample 0\n"
          ]
        }
      ],
      "source": [
        "print(\"\\n _____ACTION SPACE_____ \\n\")\n",
        "print(\"The Action Space is: \", a_size)\n",
        "print(\"Action Space Sample\", env.action_space.sample()) # Take a random action"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NNWvlyvzalXr"
      },
      "source": [
        "The observation space (7) 👀:\n",
        "- player y position\n",
        "- player velocity\n",
        "- player distance to floor\n",
        "- player distance to ceiling\n",
        "- next block x distance to player\n",
        "- next blocks top y location\n",
        "- next blocks bottom y location\n",
        "\n",
        "The action space(2) 🎮:\n",
        "- Up\n",
        "- Down\n",
        "\n",
        "The reward function 💰: \n",
        "- For each vertical block it passes through it gains a positive reward of +1. Each time a terminal state reached it receives a negative reward of -1."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aV1466QP8crz"
      },
      "source": [
        "### Define the new Policy 🧠\n",
        "- We need to have a deeper neural network since the environment is more complex"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "I1eBkCiX2X_S"
      },
      "outputs": [],
      "source": [
        "class Policy(nn.Module):\n",
        "    def __init__(self, s_size, a_size, h_size):\n",
        "        super(Policy, self).__init__()\n",
        "        # Define the three layers here\n",
        "\n",
        "    def forward(self, x):\n",
        "        # Define the forward process here\n",
        "        return F.softmax(x, dim=1)\n",
        "    \n",
        "    def act(self, state):\n",
        "        state = torch.from_numpy(state).float().unsqueeze(0).to(device)\n",
        "        probs = self.forward(state).cpu()\n",
        "        m = Categorical(probs)\n",
        "        action = m.sample()\n",
        "        return action.item(), m.log_prob(action)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "47iuAFqV8Ws-"
      },
      "source": [
        "#### Solution"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wrNuVcHC8Xu7"
      },
      "outputs": [],
      "source": [
        "class Policy(nn.Module):\n",
        "    def __init__(self, s_size, a_size, h_size):\n",
        "        super(Policy, self).__init__()\n",
        "        self.fc1 = nn.Linear(s_size, h_size)\n",
        "        self.fc2 = nn.Linear(h_size, h_size*2)\n",
        "        self.fc3 = nn.Linear(h_size*2, a_size)\n",
        "\n",
        "    def forward(self, x):\n",
        "        x = F.relu(self.fc1(x))\n",
        "        x = F.relu(self.fc2(x))\n",
        "        x = self.fc3(x)\n",
        "        return F.softmax(x, dim=1)\n",
        "    \n",
        "    def act(self, state):\n",
        "        state = torch.from_numpy(state).float().unsqueeze(0).to(device)\n",
        "        probs = self.forward(state).cpu()\n",
        "        m = Categorical(probs)\n",
        "        action = m.sample()\n",
        "        return action.item(), m.log_prob(action)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SM1QiGCSbBkM"
      },
      "source": [
        "### Define the hyperparameters ⚙️\n",
        "- Because this environment is more complex.\n",
        "- Especially for the hidden size, we need more neurons."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "y0uujOR_ypB6"
      },
      "outputs": [],
      "source": [
        "pixelcopter_hyperparameters = {\n",
        "    \"h_size\": 64,\n",
        "    \"n_training_episodes\": 10000,\n",
        "    \"n_evaluation_episodes\": 10,\n",
        "    \"max_t\": 10000,\n",
        "    \"gamma\": 0.99,\n",
        "    \"lr\": 0.0001,\n",
        "    \"env_id\": env_id,\n",
        "    \"state_space\": s_size,\n",
        "    \"action_space\": a_size,\n",
        "}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wyvXTJWm9GJG"
      },
      "source": [
        "###  Train it\n",
        "- We're now ready to train our agent 🔥."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7mM2P_ckysFE"
      },
      "outputs": [],
      "source": [
        "# Create policy and place it to the device\n",
        "#torch.manual_seed(50)\n",
        "pixelcopter_policy = Policy(pixelcopter_hyperparameters[\"state_space\"], pixelcopter_hyperparameters[\"action_space\"], pixelcopter_hyperparameters[\"h_size\"]).to(device)\n",
        "pixelcopter_optimizer = optim.Adam(pixelcopter_policy.parameters(), lr=pixelcopter_hyperparameters[\"lr\"])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "v1HEqP-fy-Rf",
        "outputId": "cdff3a0f-6938-4855-e144-b662a45f9fa7"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Episode 1000\tAverage Score: 4.62\n",
            "Episode 2000\tAverage Score: 6.62\n",
            "Episode 3000\tAverage Score: 9.53\n",
            "Episode 4000\tAverage Score: 11.15\n",
            "Episode 5000\tAverage Score: 14.18\n",
            "Episode 6000\tAverage Score: 12.35\n",
            "Episode 7000\tAverage Score: 16.17\n",
            "Episode 8000\tAverage Score: 14.01\n",
            "Episode 9000\tAverage Score: 14.28\n",
            "Episode 10000\tAverage Score: 12.97\n"
          ]
        }
      ],
      "source": [
        "scores = reinforce(pixelcopter_policy,\n",
        "                   pixelcopter_optimizer,\n",
        "                   pixelcopter_hyperparameters[\"n_training_episodes\"], \n",
        "                   pixelcopter_hyperparameters[\"max_t\"],\n",
        "                   pixelcopter_hyperparameters[\"gamma\"], \n",
        "                   1000)"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install optuna"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nmubj-P0dyib",
        "outputId": "4b39c746-335c-40e3-fd38-577ba80c514c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting optuna\n",
            "  Downloading optuna-3.1.0-py3-none-any.whl (365 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m365.3/365.3 KB\u001b[0m \u001b[31m25.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting alembic>=1.5.0\n",
            "  Downloading alembic-1.9.4-py3-none-any.whl (210 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m210.5/210.5 KB\u001b[0m \u001b[31m26.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: PyYAML in /usr/local/lib/python3.8/dist-packages (from optuna) (6.0)\n",
            "Requirement already satisfied: sqlalchemy>=1.3.0 in /usr/local/lib/python3.8/dist-packages (from optuna) (1.4.46)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.8/dist-packages (from optuna) (4.64.1)\n",
            "Collecting cmaes>=0.9.1\n",
            "  Downloading cmaes-0.9.1-py3-none-any.whl (21 kB)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.8/dist-packages (from optuna) (23.0)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.8/dist-packages (from optuna) (1.22.4)\n",
            "Collecting colorlog\n",
            "  Downloading colorlog-6.7.0-py2.py3-none-any.whl (11 kB)\n",
            "Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.8/dist-packages (from alembic>=1.5.0->optuna) (6.0.0)\n",
            "Collecting Mako\n",
            "  Downloading Mako-1.2.4-py3-none-any.whl (78 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m78.7/78.7 KB\u001b[0m \u001b[31m7.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: importlib-resources in /usr/local/lib/python3.8/dist-packages (from alembic>=1.5.0->optuna) (5.10.2)\n",
            "Requirement already satisfied: greenlet!=0.4.17 in /usr/local/lib/python3.8/dist-packages (from sqlalchemy>=1.3.0->optuna) (2.0.2)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.8/dist-packages (from importlib-metadata->alembic>=1.5.0->optuna) (3.13.0)\n",
            "Requirement already satisfied: MarkupSafe>=0.9.2 in /usr/local/lib/python3.8/dist-packages (from Mako->alembic>=1.5.0->optuna) (2.0.1)\n",
            "Installing collected packages: Mako, colorlog, cmaes, alembic, optuna\n",
            "Successfully installed Mako-1.2.4 alembic-1.9.4 cmaes-0.9.1 colorlog-6.7.0 optuna-3.1.0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import optuna\n",
        "from optuna.pruners import MedianPruner\n",
        "from optuna.samplers import TPESampler\n",
        "from optuna.visualization import plot_optimization_history, plot_param_importances\n",
        "\n",
        "def objective(trial):\n",
        "    # Define the hyperparameters to tune\n",
        "    h_size = trial.suggest_int(\"h_size\", 64, 128)\n",
        "    lr = trial.suggest_loguniform(\"lr\", 1e-5, 1e-3)\n",
        "    gamma = trial.suggest_loguniform(\"gamma\", 0.9, 0.999)\n",
        "\n",
        "    # Set up the environment and policy\n",
        "    env = gym.make(pixelcopter_hyperparameters[\"env_id\"])\n",
        "    s_size = pixelcopter_hyperparameters[\"state_space\"]\n",
        "    a_size = pixelcopter_hyperparameters[\"action_space\"]\n",
        "    policy = Policy(s_size, a_size, h_size).to(device)\n",
        "    optimizer = optim.Adam(policy.parameters(), lr=lr)\n",
        "\n",
        "    # Train the policy\n",
        "    scores = reinforce(policy, optimizer, pixelcopter_hyperparameters[\"n_training_episodes\"], pixelcopter_hyperparameters[\"max_t\"], gamma, print_every=1000)\n",
        "\n",
        "    # Return the average score over the last 2 episodes\n",
        "    return np.mean(scores[-100:])\n",
        "\n",
        "study = optuna.create_study(direction=\"maximize\")\n",
        "study.optimize(objective, n_trials=2)\n",
        "\n",
        "print(\"Best trial:\")\n",
        "trial = study.best_trial\n",
        "print(\"  Value: \", trial.value)\n",
        "print(\"  Params: \")\n",
        "for key, value in trial.params.items():\n",
        "    print(\"    {}: {}\".format(key, value))\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "MSHid64ic76D",
        "outputId": "b12fe392-b770-4d43-c35a-1e439ce56930"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\u001b[32m[I 2023-02-23 04:11:22,423]\u001b[0m A new study created in memory with name: no-name-b7fc242c-7e6e-487b-a12f-95a97f5011e6\u001b[0m\n",
            "<ipython-input-25-1678b08ce58e>:9: FutureWarning: suggest_loguniform has been deprecated in v3.0.0. This feature will be removed in v6.0.0. See https://github.com/optuna/optuna/releases/tag/v3.0.0. Use :func:`~optuna.trial.Trial.suggest_float` instead.\n",
            "  lr = trial.suggest_loguniform(\"lr\", 1e-5, 1e-3)\n",
            "<ipython-input-25-1678b08ce58e>:10: FutureWarning: suggest_loguniform has been deprecated in v3.0.0. This feature will be removed in v6.0.0. See https://github.com/optuna/optuna/releases/tag/v3.0.0. Use :func:`~optuna.trial.Trial.suggest_float` instead.\n",
            "  gamma = trial.suggest_loguniform(\"gamma\", 0.9, 0.999)\n",
            "/usr/local/lib/python3.8/dist-packages/gym/core.py:317: DeprecationWarning: \u001b[33mWARN: Initializing wrapper in old step API which returns one bool instead of two. It is recommended to set `new_step_api=True` to use new step API. This will be the default behaviour in future.\u001b[0m\n",
            "  deprecation(\n",
            "/usr/local/lib/python3.8/dist-packages/gym/wrappers/step_api_compatibility.py:39: DeprecationWarning: \u001b[33mWARN: Initializing environment in old step API which returns one bool instead of two. It is recommended to set `new_step_api=True` to use new step API. This will be the default behaviour in future.\u001b[0m\n",
            "  deprecation(\n",
            "/usr/local/lib/python3.8/dist-packages/gym/utils/passive_env_checker.py:174: UserWarning: \u001b[33mWARN: Future gym versions will require that `Env.reset` can be passed a `seed` instead of using `Env.seed` for resetting the environment random number generator.\u001b[0m\n",
            "  logger.warn(\n",
            "/usr/local/lib/python3.8/dist-packages/gym/utils/passive_env_checker.py:190: UserWarning: \u001b[33mWARN: Future gym versions will require that `Env.reset` can be passed `return_info` to return information from the environment resetting.\u001b[0m\n",
            "  logger.warn(\n",
            "/usr/local/lib/python3.8/dist-packages/gym/utils/passive_env_checker.py:195: UserWarning: \u001b[33mWARN: Future gym versions will require that `Env.reset` can be passed `options` to allow the environment initialisation to be passed additional information.\u001b[0m\n",
            "  logger.warn(\n",
            "/usr/local/lib/python3.8/dist-packages/gym/utils/passive_env_checker.py:141: UserWarning: \u001b[33mWARN: The obs returned by the `reset()` method was expecting numpy array dtype to be float32, actual type: float64\u001b[0m\n",
            "  logger.warn(\n",
            "/usr/local/lib/python3.8/dist-packages/gym/utils/passive_env_checker.py:165: UserWarning: \u001b[33mWARN: The obs returned by the `reset()` method is not within the observation space.\u001b[0m\n",
            "  logger.warn(f\"{pre} is not within the observation space.\")\n",
            "/usr/local/lib/python3.8/dist-packages/gym/utils/passive_env_checker.py:227: DeprecationWarning: \u001b[33mWARN: Core environment is written in old step API which returns one bool instead of two. It is recommended to rewrite the environment with new step API. \u001b[0m\n",
            "  logger.deprecation(\n",
            "/usr/local/lib/python3.8/dist-packages/gym/utils/passive_env_checker.py:141: UserWarning: \u001b[33mWARN: The obs returned by the `step()` method was expecting numpy array dtype to be float32, actual type: float64\u001b[0m\n",
            "  logger.warn(\n",
            "/usr/local/lib/python3.8/dist-packages/gym/utils/passive_env_checker.py:165: UserWarning: \u001b[33mWARN: The obs returned by the `step()` method is not within the observation space.\u001b[0m\n",
            "  logger.warn(f\"{pre} is not within the observation space.\")\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Episode 1000\tAverage Score: 0.29\n",
            "Episode 2000\tAverage Score: 2.64\n",
            "Episode 3000\tAverage Score: 5.04\n",
            "Episode 4000\tAverage Score: 4.66\n",
            "Episode 5000\tAverage Score: 6.23\n",
            "Episode 6000\tAverage Score: 6.59\n",
            "Episode 7000\tAverage Score: 6.29\n",
            "Episode 8000\tAverage Score: 8.59\n",
            "Episode 9000\tAverage Score: 10.01\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\u001b[32m[I 2023-02-23 04:28:38,091]\u001b[0m Trial 0 finished with value: 10.35 and parameters: {'h_size': 34, 'lr': 2.9816222949232044e-05, 'gamma': 0.9697359279035952}. Best is trial 0 with value: 10.35.\u001b[0m\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Episode 10000\tAverage Score: 10.35\n",
            "Episode 1000\tAverage Score: 1.35\n",
            "Episode 2000\tAverage Score: 6.72\n",
            "Episode 3000\tAverage Score: 9.04\n",
            "Episode 4000\tAverage Score: 12.62\n",
            "Episode 5000\tAverage Score: 9.99\n",
            "Episode 6000\tAverage Score: -2.53\n",
            "Episode 7000\tAverage Score: -2.61\n",
            "Episode 8000\tAverage Score: -2.61\n",
            "Episode 9000\tAverage Score: 4.03\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\u001b[32m[I 2023-02-23 04:42:38,567]\u001b[0m Trial 1 finished with value: 11.91 and parameters: {'h_size': 230, 'lr': 4.650381214198234e-05, 'gamma': 0.9717474272921067}. Best is trial 1 with value: 11.91.\u001b[0m\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Episode 10000\tAverage Score: 11.91\n",
            "Episode 1000\tAverage Score: 1.61\n",
            "Episode 2000\tAverage Score: 4.72\n",
            "Episode 3000\tAverage Score: 4.30\n",
            "Episode 4000\tAverage Score: 6.20\n",
            "Episode 5000\tAverage Score: 4.87\n",
            "Episode 6000\tAverage Score: 7.78\n",
            "Episode 7000\tAverage Score: 7.84\n",
            "Episode 8000\tAverage Score: 7.33\n",
            "Episode 9000\tAverage Score: 7.86\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\u001b[32m[I 2023-02-23 04:59:37,064]\u001b[0m Trial 2 finished with value: 6.96 and parameters: {'h_size': 113, 'lr': 1.3831898390329787e-05, 'gamma': 0.9055345654117689}. Best is trial 1 with value: 11.91.\u001b[0m\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Episode 10000\tAverage Score: 6.96\n",
            "Episode 1000\tAverage Score: -2.69\n",
            "Episode 2000\tAverage Score: -2.56\n",
            "Episode 3000\tAverage Score: -2.74\n",
            "Episode 4000\tAverage Score: -2.65\n",
            "Episode 5000\tAverage Score: -2.69\n",
            "Episode 6000\tAverage Score: -2.66\n",
            "Episode 7000\tAverage Score: -5.00\n",
            "Episode 8000\tAverage Score: -5.00\n",
            "Episode 9000\tAverage Score: -5.00\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\u001b[32m[I 2023-02-23 05:03:33,304]\u001b[0m Trial 3 finished with value: -5.0 and parameters: {'h_size': 191, 'lr': 0.0001226776784208966, 'gamma': 0.9977447208070642}. Best is trial 1 with value: 11.91.\u001b[0m\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Episode 10000\tAverage Score: -5.00\n",
            "Episode 1000\tAverage Score: -5.00\n",
            "Episode 2000\tAverage Score: -5.00\n",
            "Episode 3000\tAverage Score: -5.00\n",
            "Episode 4000\tAverage Score: -5.00\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\u001b[33m[W 2023-02-23 05:04:07,906]\u001b[0m Trial 4 failed with parameters: {'h_size': 254, 'lr': 0.00038341772769750034, 'gamma': 0.9283904812387193} because of the following error: KeyboardInterrupt().\u001b[0m\n",
            "Traceback (most recent call last):\n",
            "  File \"/usr/local/lib/python3.8/dist-packages/optuna/study/_optimize.py\", line 200, in _run_trial\n",
            "    value_or_values = func(trial)\n",
            "  File \"<ipython-input-25-1678b08ce58e>\", line 20, in objective\n",
            "    scores = reinforce(policy, optimizer, pixelcopter_hyperparameters[\"n_training_episodes\"], pixelcopter_hyperparameters[\"max_t\"], gamma, print_every=1000)\n",
            "  File \"<ipython-input-8-d0467f546b2b>\", line 14, in reinforce\n",
            "    state, reward, done, _ = env.step(action)\n",
            "  File \"/usr/local/lib/python3.8/dist-packages/gym/wrappers/order_enforcing.py\", line 37, in step\n",
            "    return self.env.step(action)\n",
            "  File \"/usr/local/lib/python3.8/dist-packages/gym/wrappers/step_api_compatibility.py\", line 52, in step\n",
            "    step_returns = self.env.step(action)\n",
            "  File \"/usr/local/lib/python3.8/dist-packages/gym/wrappers/env_checker.py\", line 39, in step\n",
            "    return self.env.step(action)\n",
            "  File \"/usr/local/lib/python3.8/dist-packages/gym_pygame/envs/base.py\", line 44, in step\n",
            "    reward = self.gameOb.act(self.action_set[action])\n",
            "  File \"/usr/local/lib/python3.8/dist-packages/ple/ple.py\", line 376, in act\n",
            "    return sum(self._oneStepAct(action) for i in range(self.frame_skip))\n",
            "  File \"/usr/local/lib/python3.8/dist-packages/ple/ple.py\", line 376, in <genexpr>\n",
            "    return sum(self._oneStepAct(action) for i in range(self.frame_skip))\n",
            "  File \"/usr/local/lib/python3.8/dist-packages/ple/ple.py\", line 398, in _oneStepAct\n",
            "    self.game.step(time_elapsed)\n",
            "  File \"/usr/local/lib/python3.8/dist-packages/ple/games/pixelcopter.py\", line 326, in step\n",
            "    self.block_group.draw(self.screen)\n",
            "  File \"/usr/local/lib/python3.8/dist-packages/pygame/sprite.py\", line 551, in draw\n",
            "    zip(sprites, surface.blits((spr.image, spr.rect) for spr in sprites))\n",
            "  File \"/usr/local/lib/python3.8/dist-packages/pygame/sprite.py\", line 551, in <genexpr>\n",
            "    zip(sprites, surface.blits((spr.image, spr.rect) for spr in sprites))\n",
            "KeyboardInterrupt\n",
            "\u001b[33m[W 2023-02-23 05:04:07,912]\u001b[0m Trial 4 failed with value None.\u001b[0m\n"
          ]
        },
        {
          "output_type": "error",
          "ename": "KeyboardInterrupt",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-25-1678b08ce58e>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m     24\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     25\u001b[0m \u001b[0mstudy\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0moptuna\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcreate_study\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdirection\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"maximize\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 26\u001b[0;31m \u001b[0mstudy\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moptimize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mobjective\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mn_trials\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m100\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     27\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     28\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Best trial:\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.8/dist-packages/optuna/study/study.py\u001b[0m in \u001b[0;36moptimize\u001b[0;34m(self, func, n_trials, timeout, n_jobs, catch, callbacks, gc_after_trial, show_progress_bar)\u001b[0m\n\u001b[1;32m    423\u001b[0m         \"\"\"\n\u001b[1;32m    424\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 425\u001b[0;31m         _optimize(\n\u001b[0m\u001b[1;32m    426\u001b[0m             \u001b[0mstudy\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    427\u001b[0m             \u001b[0mfunc\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mfunc\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.8/dist-packages/optuna/study/_optimize.py\u001b[0m in \u001b[0;36m_optimize\u001b[0;34m(study, func, n_trials, timeout, n_jobs, catch, callbacks, gc_after_trial, show_progress_bar)\u001b[0m\n\u001b[1;32m     64\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     65\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mn_jobs\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 66\u001b[0;31m             _optimize_sequential(\n\u001b[0m\u001b[1;32m     67\u001b[0m                 \u001b[0mstudy\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     68\u001b[0m                 \u001b[0mfunc\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.8/dist-packages/optuna/study/_optimize.py\u001b[0m in \u001b[0;36m_optimize_sequential\u001b[0;34m(study, func, n_trials, timeout, catch, callbacks, gc_after_trial, reseed_sampler_rng, time_start, progress_bar)\u001b[0m\n\u001b[1;32m    161\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    162\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 163\u001b[0;31m             \u001b[0mfrozen_trial\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_run_trial\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstudy\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcatch\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    164\u001b[0m         \u001b[0;32mfinally\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    165\u001b[0m             \u001b[0;31m# The following line mitigates memory problems that can be occurred in some\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.8/dist-packages/optuna/study/_optimize.py\u001b[0m in \u001b[0;36m_run_trial\u001b[0;34m(study, func, catch)\u001b[0m\n\u001b[1;32m    249\u001b[0m         \u001b[0;32mand\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfunc_err\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcatch\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    250\u001b[0m     ):\n\u001b[0;32m--> 251\u001b[0;31m         \u001b[0;32mraise\u001b[0m \u001b[0mfunc_err\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    252\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0mfrozen_trial\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    253\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.8/dist-packages/optuna/study/_optimize.py\u001b[0m in \u001b[0;36m_run_trial\u001b[0;34m(study, func, catch)\u001b[0m\n\u001b[1;32m    198\u001b[0m     \u001b[0;32mwith\u001b[0m \u001b[0mget_heartbeat_thread\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtrial\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_trial_id\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstudy\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_storage\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    199\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 200\u001b[0;31m             \u001b[0mvalue_or_values\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtrial\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    201\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mexceptions\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTrialPruned\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    202\u001b[0m             \u001b[0;31m# TODO(mamu): Handle multi-objective cases.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m<ipython-input-25-1678b08ce58e>\u001b[0m in \u001b[0;36mobjective\u001b[0;34m(trial)\u001b[0m\n\u001b[1;32m     18\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     19\u001b[0m     \u001b[0;31m# Train the policy\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 20\u001b[0;31m     \u001b[0mscores\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mreinforce\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpolicy\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moptimizer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpixelcopter_hyperparameters\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"n_training_episodes\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpixelcopter_hyperparameters\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"max_t\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgamma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mprint_every\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m1000\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     21\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     22\u001b[0m     \u001b[0;31m# Return the average score over the last 100 episodes\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m<ipython-input-8-d0467f546b2b>\u001b[0m in \u001b[0;36mreinforce\u001b[0;34m(policy, optimizer, n_training_episodes, max_t, gamma, print_every)\u001b[0m\n\u001b[1;32m     12\u001b[0m             \u001b[0maction\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlog_prob\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpolicy\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mact\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstate\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     13\u001b[0m             \u001b[0msaved_log_probs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlog_prob\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 14\u001b[0;31m             \u001b[0mstate\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mreward\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0m_\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0menv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstep\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0maction\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     15\u001b[0m             \u001b[0mrewards\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mreward\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     16\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mdone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.8/dist-packages/gym/wrappers/order_enforcing.py\u001b[0m in \u001b[0;36mstep\u001b[0;34m(self, action)\u001b[0m\n\u001b[1;32m     35\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_has_reset\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     36\u001b[0m             \u001b[0;32mraise\u001b[0m \u001b[0mResetNeeded\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Cannot call env.step() before calling env.reset()\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 37\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0menv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstep\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0maction\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     38\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     39\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mreset\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.8/dist-packages/gym/wrappers/step_api_compatibility.py\u001b[0m in \u001b[0;36mstep\u001b[0;34m(self, action)\u001b[0m\n\u001b[1;32m     50\u001b[0m             \u001b[0;34m(\u001b[0m\u001b[0mobservation\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mreward\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mterminated\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtruncated\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minfo\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mobservation\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mreward\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minfo\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     51\u001b[0m         \"\"\"\n\u001b[0;32m---> 52\u001b[0;31m         \u001b[0mstep_returns\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0menv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstep\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0maction\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     53\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnew_step_api\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     54\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mstep_to_new_api\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstep_returns\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.8/dist-packages/gym/wrappers/env_checker.py\u001b[0m in \u001b[0;36mstep\u001b[0;34m(self, action)\u001b[0m\n\u001b[1;32m     37\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0menv_step_passive_checker\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0menv\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0maction\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     38\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 39\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0menv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstep\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0maction\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     40\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     41\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mreset\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.8/dist-packages/gym_pygame/envs/base.py\u001b[0m in \u001b[0;36mstep\u001b[0;34m(self, action)\u001b[0m\n\u001b[1;32m     42\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     43\u001b[0m   \u001b[0;32mdef\u001b[0m \u001b[0mstep\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0maction\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 44\u001b[0;31m     \u001b[0mreward\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgameOb\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mact\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0maction_set\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0maction\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     45\u001b[0m     \u001b[0mdone\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgameOb\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgame_over\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     46\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgameOb\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgetGameState\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mreward\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.8/dist-packages/ple/ple.py\u001b[0m in \u001b[0;36mact\u001b[0;34m(self, action)\u001b[0m\n\u001b[1;32m    374\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    375\u001b[0m         \"\"\"\n\u001b[0;32m--> 376\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0msum\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_oneStepAct\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0maction\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mframe_skip\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    377\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    378\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_draw_frame\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.8/dist-packages/ple/ple.py\u001b[0m in \u001b[0;36m<genexpr>\u001b[0;34m(.0)\u001b[0m\n\u001b[1;32m    374\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    375\u001b[0m         \"\"\"\n\u001b[0;32m--> 376\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0msum\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_oneStepAct\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0maction\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mframe_skip\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    377\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    378\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_draw_frame\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.8/dist-packages/ple/ple.py\u001b[0m in \u001b[0;36m_oneStepAct\u001b[0;34m(self, action)\u001b[0m\n\u001b[1;32m    396\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnum_steps\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    397\u001b[0m             \u001b[0mtime_elapsed\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_tick\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 398\u001b[0;31m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgame\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstep\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtime_elapsed\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    399\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_draw_frame\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    400\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.8/dist-packages/ple/games/pixelcopter.py\u001b[0m in \u001b[0;36mstep\u001b[0;34m(self, dt)\u001b[0m\n\u001b[1;32m    324\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    325\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mplayer_group\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdraw\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mscreen\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 326\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mblock_group\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdraw\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mscreen\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    327\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mterrain_group\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdraw\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mscreen\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    328\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.8/dist-packages/pygame/sprite.py\u001b[0m in \u001b[0;36mdraw\u001b[0;34m(self, surface)\u001b[0m\n\u001b[1;32m    549\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mhasattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msurface\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"blits\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    550\u001b[0m             self.spritedict.update(\n\u001b[0;32m--> 551\u001b[0;31m                 \u001b[0mzip\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msprites\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msurface\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mblits\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mspr\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mimage\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mspr\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrect\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mspr\u001b[0m \u001b[0;32min\u001b[0m \u001b[0msprites\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    552\u001b[0m             )\n\u001b[1;32m    553\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.8/dist-packages/pygame/sprite.py\u001b[0m in \u001b[0;36m<genexpr>\u001b[0;34m(.0)\u001b[0m\n\u001b[1;32m    549\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mhasattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msurface\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"blits\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    550\u001b[0m             self.spritedict.update(\n\u001b[0;32m--> 551\u001b[0;31m                 \u001b[0mzip\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msprites\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msurface\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mblits\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mspr\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mimage\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mspr\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrect\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mspr\u001b[0m \u001b[0;32min\u001b[0m \u001b[0msprites\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    552\u001b[0m             )\n\u001b[1;32m    553\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "fig1 = plot_optimization_history(study)\n",
        "fig2 = plot_param_importances(study)\n",
        "fig1.show()\n",
        "fig2.show()"
      ],
      "metadata": {
        "id": "Wpy48N0UoMIv"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8kwFQ-Ip85BE"
      },
      "source": [
        "### Publish our trained model on the Hub 🔥"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 150,
          "referenced_widgets": [
            "482c6d46e9684b6a8e365ce7e620a4b4",
            "dded70be99e84b2bbeb2208795e91d37",
            "76593252164b41eeb3fbcb87f09dacc8",
            "542d316e5e484db195b7e5e0e6f51e5c",
            "a5c64fb73e3c47bbb42d72f629a912ca",
            "d6c8270715ef4fd488f31555dfd9cf7f",
            "2e5ebc5bb30c408788ac8516684842c4",
            "6e4392b9a8d24932951e18d205c85594",
            "e07ee055ae5342f598f5b86c7974443a",
            "2e39d34993e84ad9b9d5e3c8ae74e56a",
            "0ed330c70d5d4e93944b3bbbff0ec590",
            "892ec9b90a7d476b8aa4e1b84446287b",
            "32049002598a4001930c87b770d23d0e",
            "2fbaea942b084650a3225c386e16c332",
            "48f13e4905724c0e91027fd23412f6f1",
            "4df2af8d10e24f02bac063e117bf6fdc",
            "905fd54b03c7419a9813e5f581ba917d",
            "4cf2c39d7686476da1d9ecde72aaa73d",
            "ba3c78a02e3745a3b93532bacc1f1b36",
            "6457123fbc724e23857135df5e1c22f3",
            "a8070a26ec6a401a9626f1fd41e508b5",
            "e2323d3f589642ecbf99e1f800033e6a"
          ]
        },
        "id": "6PtB7LRbTKWK",
        "outputId": "245dca01-48cc-43fb-eda5-f1ffa82d714d"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.8/dist-packages/gym/core.py:43: DeprecationWarning: \u001b[33mWARN: The argument mode in render method is deprecated; use render_mode during environment initialization instead.\n",
            "See here for more information: https://www.gymlibrary.ml/content/api/\u001b[0m\n",
            "  deprecation(\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Upload 1 LFS files:   0%|          | 0/1 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "482c6d46e9684b6a8e365ce7e620a4b4"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "model.pt:   0%|          | 0.00/39.0k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "892ec9b90a7d476b8aa4e1b84446287b"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Your model is pushed to the Hub. You can view your model here: https://huggingface.co/Ryukijano/Reinforce_pixel_copter_normal\n"
          ]
        }
      ],
      "source": [
        "repo_id = \"Ryukijano/Reinforce_pixel_copter_normal\" #TODO Define your repo id {username/Reinforce-{model-id}}\n",
        "push_to_hub(repo_id,\n",
        "                pixelcopter_policy, # The model we want to save\n",
        "                pixelcopter_hyperparameters, # Hyperparameters\n",
        "                eval_env, # Evaluation environment\n",
        "                video_fps=30\n",
        "                )"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7VDcJ29FcOyb"
      },
      "source": [
        "## Some additional challenges 🏆\n",
        "The best way to learn **is to try things on your own**! As you saw, the current agent is not doing great. As a first suggestion, you can train for more steps. But also trying to find better parameters.\n",
        "\n",
        "In the [Leaderboard](https://huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard) you will find your agents. Can you get to the top?\n",
        "\n",
        "Here are some ideas to achieve so:\n",
        "* Train more steps\n",
        "* Try different hyperparameters by looking at what your classmates have done 👉 https://huggingface.co/models?other=reinforce\n",
        "* **Push your new trained model** on the Hub 🔥\n",
        "* **Improving the implementation for more complex environments** (for instance, what about changing the network to a Convolutional Neural Network to handle\n",
        "frames as observation)?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x62pP0PHdA-y"
      },
      "source": [
        "________________________________________________________________________\n",
        "\n",
        "**Congrats on finishing this unit**! There was a lot of information.\n",
        "And congrats on finishing the tutorial. You've just coded your first Deep Reinforcement Learning agent from scratch using PyTorch and shared it on the Hub 🥳.\n",
        "\n",
        "Don't hesitate to iterate on this unit **by improving the implementation for more complex environments** (for instance, what about changing the network to a Convolutional Neural Network to handle\n",
        "frames as observation)?\n",
        "\n",
        "In the next unit, **we're going to learn more about Unity MLAgents**, by training agents in Unity environments. This way, you will be ready to participate in the **AI vs AI challenges where you'll train your agents\n",
        "to compete against other agents in a snowball fight and a soccer game.**\n",
        "\n",
        "Sounds fun? See you next time!\n",
        "\n",
        "Finally, we would love **to hear what you think of the course and how we can improve it**. If you have some feedback then, please 👉  [fill this form](https://forms.gle/BzKXWzLAGZESGNaE9)\n",
        "\n",
        "See you in Unit 5! 🔥\n",
        "\n",
        "### Keep Learning, stay awesome 🤗\n",
        "\n"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [
        "BPLwsPajb1f8",
        "L_WSo0VUV99t",
        "mjY-eq3eWh9O",
        "JoTC9o2SczNn",
        "x62pP0PHdA-y"
      ],
      "provenance": [],
      "include_colab_link": true
    },
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.10"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "008a7c260e6b46b38d8ddfc69615bb06": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "00ec48b5d37a4f41a82db8ac8787174e": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "0528f2ac4256491d98ee1c2ab2aa7850": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "0cdea780ba2e469c89d24ecb2cbe8b23": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_00ec48b5d37a4f41a82db8ac8787174e",
            "placeholder": "​",
            "style": "IPY_MODEL_008a7c260e6b46b38d8ddfc69615bb06",
            "value": "model.pt: 100%"
          }
        },
        "0e7874013c314b42bf42e474c5978bcb": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "2c5cd55508b24739967b0cd4121f96d7": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "3aab122bb14f4468a51d5ddebd47b09c": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_b9e01ea472444827a30b9fd2698106f5",
            "max": 1,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_0528f2ac4256491d98ee1c2ab2aa7850",
            "value": 1
          }
        },
        "3df3360090804c7da4ebee660f14bfdf": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "596451d013c644e7b8d1dd9d589e4d84": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_0cdea780ba2e469c89d24ecb2cbe8b23",
              "IPY_MODEL_b17c6bfac1ef40f79d3630148b2df8b1",
              "IPY_MODEL_de27f63695ac40a6993058729d95cba0"
            ],
            "layout": "IPY_MODEL_d6cb5a48471e4d5fb93229b10672ffa2"
          }
        },
        "7bedf0974ee64d3190f0554a247cfe10": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_8e4a100e53e24d38966ba53b48e2dd95",
              "IPY_MODEL_3aab122bb14f4468a51d5ddebd47b09c",
              "IPY_MODEL_d2e827af1afa4bb9bffad58f97da09c2"
            ],
            "layout": "IPY_MODEL_8ebb8fe869ae43ecb186c2411fbfe4c5"
          }
        },
        "8e4a100e53e24d38966ba53b48e2dd95": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_b6a2b27c0309484aba3d4d89b55efbde",
            "placeholder": "​",
            "style": "IPY_MODEL_ca0efeea2bc64d53a97c581613765707",
            "value": "Upload 1 LFS files: 100%"
          }
        },
        "8ebb8fe869ae43ecb186c2411fbfe4c5": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b17c6bfac1ef40f79d3630148b2df8b1": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_b8ebf0194d0e45d49a105e322d75cb47",
            "max": 3923,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_3df3360090804c7da4ebee660f14bfdf",
            "value": 3923
          }
        },
        "b1dc2bb4acba49479ac83d17b7a3b790": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b6a2b27c0309484aba3d4d89b55efbde": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b8ebf0194d0e45d49a105e322d75cb47": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b9e01ea472444827a30b9fd2698106f5": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ca0efeea2bc64d53a97c581613765707": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "d2e827af1afa4bb9bffad58f97da09c2": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_2c5cd55508b24739967b0cd4121f96d7",
            "placeholder": "​",
            "style": "IPY_MODEL_d5a51687ecb341ee87fc345eed6e498b",
            "value": " 1/1 [00:02&lt;00:00,  2.11s/it]"
          }
        },
        "d5a51687ecb341ee87fc345eed6e498b": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "d6cb5a48471e4d5fb93229b10672ffa2": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "de27f63695ac40a6993058729d95cba0": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_b1dc2bb4acba49479ac83d17b7a3b790",
            "placeholder": "​",
            "style": "IPY_MODEL_0e7874013c314b42bf42e474c5978bcb",
            "value": " 3.92k/3.92k [00:02&lt;00:00, 5.11kB/s]"
          }
        },
        "4ae384cc0d4342febb5631e387d39d26": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "VBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "VBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "VBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_4116b0e57cb74fcd8389ef189f07df2c",
              "IPY_MODEL_1ec0e63f09dc4421b2a8c586bc4afb20",
              "IPY_MODEL_7388dbe9640244a4a41d247943d3dfe4",
              "IPY_MODEL_62e2030bd7e845e5846a2e2debddf545",
              "IPY_MODEL_fda396519ae94fbb8f44caa2d275054d"
            ],
            "layout": "IPY_MODEL_66b14d0f62f34f17ab7c2e97e24dba5a"
          }
        },
        "4116b0e57cb74fcd8389ef189f07df2c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_083a463baab34e28bf2f7de4f57a021b",
            "placeholder": "​",
            "style": "IPY_MODEL_4feb15f0899a4dd1b0bbdfdc71f949ef",
            "value": "<center> <img\nsrc=https://huggingface.co/front/assets/huggingface_logo-noborder.svg\nalt='Hugging Face'> <br> Copy a token from <a\nhref=\"https://huggingface.co/settings/tokens\" target=\"_blank\">your Hugging Face\ntokens page</a> and paste it below. <br> Immediately click login after copying\nyour token or it might be stored in plain text in this notebook file. </center>"
          }
        },
        "1ec0e63f09dc4421b2a8c586bc4afb20": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "PasswordModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "PasswordModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "PasswordView",
            "continuous_update": true,
            "description": "Token:",
            "description_tooltip": null,
            "disabled": false,
            "layout": "IPY_MODEL_279be38d6f514307b47be5c0e8b62d9c",
            "placeholder": "​",
            "style": "IPY_MODEL_891fdc2c0f4f42bca1a7ca2f4596af8d",
            "value": ""
          }
        },
        "7388dbe9640244a4a41d247943d3dfe4": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "CheckboxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "CheckboxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "CheckboxView",
            "description": "Add token as git credential?",
            "description_tooltip": null,
            "disabled": false,
            "indent": true,
            "layout": "IPY_MODEL_ff4575f7cabf4f2399521a17b3a35fef",
            "style": "IPY_MODEL_c99c3757612c4bb586850dce41ea68d4",
            "value": true
          }
        },
        "62e2030bd7e845e5846a2e2debddf545": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ButtonModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ButtonModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ButtonView",
            "button_style": "",
            "description": "Login",
            "disabled": false,
            "icon": "",
            "layout": "IPY_MODEL_947518c1a8e34004bde5b48490ce0a55",
            "style": "IPY_MODEL_bd3ef37fd5494aad9327081ea03a30c2",
            "tooltip": ""
          }
        },
        "fda396519ae94fbb8f44caa2d275054d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_2cc4f55956134a83a866ec3363b7b1c3",
            "placeholder": "​",
            "style": "IPY_MODEL_6f13178da385404d83ea00779f9c7d41",
            "value": "\n<b>Pro Tip:</b> If you don't already have one, you can create a dedicated\n'notebooks' token with 'write' access, that you can then easily reuse for all\nnotebooks. </center>"
          }
        },
        "66b14d0f62f34f17ab7c2e97e24dba5a": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": "center",
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": "flex",
            "flex": null,
            "flex_flow": "column",
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": "50%"
          }
        },
        "083a463baab34e28bf2f7de4f57a021b": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "4feb15f0899a4dd1b0bbdfdc71f949ef": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "279be38d6f514307b47be5c0e8b62d9c": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "891fdc2c0f4f42bca1a7ca2f4596af8d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "ff4575f7cabf4f2399521a17b3a35fef": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "c99c3757612c4bb586850dce41ea68d4": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "947518c1a8e34004bde5b48490ce0a55": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "bd3ef37fd5494aad9327081ea03a30c2": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ButtonStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ButtonStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "button_color": null,
            "font_weight": ""
          }
        },
        "2cc4f55956134a83a866ec3363b7b1c3": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "6f13178da385404d83ea00779f9c7d41": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "482c6d46e9684b6a8e365ce7e620a4b4": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_dded70be99e84b2bbeb2208795e91d37",
              "IPY_MODEL_76593252164b41eeb3fbcb87f09dacc8",
              "IPY_MODEL_542d316e5e484db195b7e5e0e6f51e5c"
            ],
            "layout": "IPY_MODEL_a5c64fb73e3c47bbb42d72f629a912ca"
          }
        },
        "dded70be99e84b2bbeb2208795e91d37": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_d6c8270715ef4fd488f31555dfd9cf7f",
            "placeholder": "​",
            "style": "IPY_MODEL_2e5ebc5bb30c408788ac8516684842c4",
            "value": "Upload 1 LFS files: 100%"
          }
        },
        "76593252164b41eeb3fbcb87f09dacc8": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_6e4392b9a8d24932951e18d205c85594",
            "max": 1,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_e07ee055ae5342f598f5b86c7974443a",
            "value": 1
          }
        },
        "542d316e5e484db195b7e5e0e6f51e5c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_2e39d34993e84ad9b9d5e3c8ae74e56a",
            "placeholder": "​",
            "style": "IPY_MODEL_0ed330c70d5d4e93944b3bbbff0ec590",
            "value": " 1/1 [00:03&lt;00:00,  3.56s/it]"
          }
        },
        "a5c64fb73e3c47bbb42d72f629a912ca": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d6c8270715ef4fd488f31555dfd9cf7f": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2e5ebc5bb30c408788ac8516684842c4": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "6e4392b9a8d24932951e18d205c85594": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e07ee055ae5342f598f5b86c7974443a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "2e39d34993e84ad9b9d5e3c8ae74e56a": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "0ed330c70d5d4e93944b3bbbff0ec590": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "892ec9b90a7d476b8aa4e1b84446287b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_32049002598a4001930c87b770d23d0e",
              "IPY_MODEL_2fbaea942b084650a3225c386e16c332",
              "IPY_MODEL_48f13e4905724c0e91027fd23412f6f1"
            ],
            "layout": "IPY_MODEL_4df2af8d10e24f02bac063e117bf6fdc"
          }
        },
        "32049002598a4001930c87b770d23d0e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_905fd54b03c7419a9813e5f581ba917d",
            "placeholder": "​",
            "style": "IPY_MODEL_4cf2c39d7686476da1d9ecde72aaa73d",
            "value": "model.pt: 100%"
          }
        },
        "2fbaea942b084650a3225c386e16c332": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_ba3c78a02e3745a3b93532bacc1f1b36",
            "max": 38983,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_6457123fbc724e23857135df5e1c22f3",
            "value": 38983
          }
        },
        "48f13e4905724c0e91027fd23412f6f1": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_a8070a26ec6a401a9626f1fd41e508b5",
            "placeholder": "​",
            "style": "IPY_MODEL_e2323d3f589642ecbf99e1f800033e6a",
            "value": " 39.0k/39.0k [00:03&lt;00:00, 10.9kB/s]"
          }
        },
        "4df2af8d10e24f02bac063e117bf6fdc": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "905fd54b03c7419a9813e5f581ba917d": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "4cf2c39d7686476da1d9ecde72aaa73d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "ba3c78a02e3745a3b93532bacc1f1b36": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "6457123fbc724e23857135df5e1c22f3": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "a8070a26ec6a401a9626f1fd41e508b5": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e2323d3f589642ecbf99e1f800033e6a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}