3_6_classifying_newswires(김창우).ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.5"
    },
    "colab": {
      "name": "3_6_classifying_newswires(김창우).ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/kcw0331/Deeplearning/blob/main/3_6_classifying_newswires(%EA%B9%80%EC%B0%BD%EC%9A%B0).ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ndwccgcKB9d0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 36
        },
        "outputId": "f8687c07-b40a-4c7a-f78f-ead23c527695"
      },
      "source": [
        "import tensorflow\n",
        "tensorflow.keras.__version__"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'2.4.0'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 1
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dfpKpct8B9d5"
      },
      "source": [
        "# Classifying newswires: a multi-class classification example\n",
        "\n",
        "This notebook contains the code samples found in Chapter 3, Section 5 of [Deep Learning with Python](https://www.manning.com/books/deep-learning-with-python?a_aid=keras&a_bid=76564dff). Note that the original text features far more content, in particular further explanations and figures: in this notebook, you will only find source code and related comments.\n",
        "\n",
        "----\n",
        "\n",
        "In the previous section we saw how to classify vector inputs into two mutually exclusive classes using a densely-connected neural network. \n",
        "But what happens when you have more than two classes? \n",
        "\n",
        "In this section, we will build a network to classify Reuters newswires into 46 different mutually-exclusive topics. Since we have many \n",
        "classes, this problem is an instance of \"multi-class classification\", and since each data point should be classified into only one \n",
        "category, the problem is more specifically an instance of \"single-label, multi-class classification\". If each data point could have \n",
        "belonged to multiple categories (in our case, topics) then we would be facing a \"multi-label, multi-class classification\" problem."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yVnqn7QXB9d5"
      },
      "source": [
        "## The Reuters dataset\n",
        "\n",
        "\n",
        "We will be working with the _Reuters dataset_, a set of short newswires and their topics, published by Reuters in 1986. It's a very simple, \n",
        "widely used toy dataset for text classification. There are 46 different topics; some topics are more represented than others, but each \n",
        "topic has at least 10 examples in the training set.\n",
        "\n",
        "Like IMDB and MNIST, the Reuters dataset comes packaged as part of Keras. Let's take a look right away:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TFRxtIOHB9d6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b13e4847-5927-4f29-a08c-80a875c194ad"
      },
      "source": [
        "from tensorflow.keras.datasets import reuters\n",
        "\n",
        "(train_data, train_labels), (test_data, test_labels) = reuters.load_data(num_words=10000)   #num_word는 똑같이 10000개의 빈도수가 높은 것을 뽑아준다."
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/reuters.npz\n",
            "2113536/2110848 [==============================] - 0s 0us/step\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/datasets/reuters.py:148: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray\n",
            "  x_train, y_train = np.array(xs[:idx]), np.array(labels[:idx])\n",
            "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/datasets/reuters.py:149: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray\n",
            "  x_test, y_test = np.array(xs[idx:]), np.array(labels[idx:])\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jETIau2qB9d6"
      },
      "source": [
        "\n",
        "Like with the IMDB dataset, the argument `num_words=10000` restricts the data to the 10,000 most frequently occurring words found in the \n",
        "data.\n",
        "\n",
        "We have 8,982 training examples and 2,246 test examples:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UUGwhA1ZB9d6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "30d3a615-1aaf-44dc-f8c1-60cee2a791f3"
      },
      "source": [
        "len(train_data)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "8982"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sYb6Y_cMT-D9"
      },
      "source": [
        "- len(train_data)를 사용해서 개수를 확인할 수 있다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qile2-wMB9d6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "aee895c5-4e92-4aee-baf5-907be2c46996"
      },
      "source": [
        "len(test_data)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "2246"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hajtlIkNB9d7"
      },
      "source": [
        "As with the IMDB reviews, each example is a list of integers (word indices):"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QHItTKGhB9d7",
        "outputId": "2940a451-535c-4ca6-99c7-3d5b04f1b526"
      },
      "source": [
        "train_data[10]   #각각의 데이터는 sequence로 나타내진다."
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[1,\n",
              " 245,\n",
              " 273,\n",
              " 207,\n",
              " 156,\n",
              " 53,\n",
              " 74,\n",
              " 160,\n",
              " 26,\n",
              " 14,\n",
              " 46,\n",
              " 296,\n",
              " 26,\n",
              " 39,\n",
              " 74,\n",
              " 2979,\n",
              " 3554,\n",
              " 14,\n",
              " 46,\n",
              " 4689,\n",
              " 4329,\n",
              " 86,\n",
              " 61,\n",
              " 3499,\n",
              " 4795,\n",
              " 14,\n",
              " 61,\n",
              " 451,\n",
              " 4329,\n",
              " 17,\n",
              " 12]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0KxWCOyJB9d7"
      },
      "source": [
        "Here's how you can decode it back to words, in case you are curious:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aktNeRTJB9d7",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "86ac6512-925f-4cbc-fe85-c607d1cfae63"
      },
      "source": [
        "word_index = reuters.get_word_index()\n",
        "reverse_word_index = dict([(value, key) for (key, value) in word_index.items()])\n",
        "# Note that our indices were offset by 3\n",
        "# because 0, 1 and 2 are reserved indices for \"padding\", \"start of sequence\", and \"unknown\".\n",
        "decoded_newswire = ' '.join([reverse_word_index.get(i - 3, '?') for i in train_data[0]])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/reuters_word_index.json\n",
            "557056/550378 [==============================] - 0s 0us/step\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JXPJUgOsB9d8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 73
        },
        "outputId": "229ea44f-a2f1-421d-ab28-21d36ab62d94"
      },
      "source": [
        "decoded_newswire"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'? ? ? said as a result of its december acquisition of space co it expects earnings per share in 1987 of 1 15 to 1 30 dlrs per share up from 70 cts in 1986 the company said pretax net should rise to nine to 10 mln dlrs from six mln dlrs in 1986 and rental operation revenues to 19 to 22 mln dlrs from 12 5 mln dlrs it said cash flow per share this year should be 2 50 to three dlrs reuter 3'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QmYil2fZUYkk"
      },
      "source": [
        "- 여기에서 보이는 ? ? ?는 10000개 안에 포함이 되지 않는 애들을 이렇게 생성해준다."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fVqWZ5zaB9d8"
      },
      "source": [
        "The label associated with an example is an integer between 0 and 45: a topic index."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "157lg3wIB9d8",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f6edc41f-e943-432d-a69d-0e71097ad6e1"
      },
      "source": [
        "train_labels[10]   #train_labels에서 10번째 것을 확인한 결과이다."
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "3"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gjm4bjfKB9d8"
      },
      "source": [
        "## Preparing the data\n",
        "\n",
        "We can vectorize the data with the exact same code as in our previous example:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WCksRu83B9d8"
      },
      "source": [
        "import numpy as np\n",
        "\n",
        "def vectorize_sequences(sequences, dimension=10000):\n",
        "    results = np.zeros((len(sequences), dimension))\n",
        "    for i, sequence in enumerate(sequences):\n",
        "        results[i, sequence] = 1.\n",
        "    return results\n",
        "\n",
        "# Our vectorized training data\n",
        "x_train = vectorize_sequences(train_data)\n",
        "# Our vectorized test data\n",
        "x_test = vectorize_sequences(test_data)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0X8mL-luUr3c"
      },
      "source": [
        "- 이 부분은 metrix를 만들어 주는 부분인데 movie review와 같은 것을 알 수 있다."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TKS2d8UNB9d9"
      },
      "source": [
        "\n",
        "To vectorize the labels, there are two possibilities: we could just cast the label list as an integer tensor, or we could use a \"one-hot\" \n",
        "encoding. One-hot encoding is a widely used format for categorical data, also called \"categorical encoding\". \n",
        "For a more detailed explanation of one-hot encoding, you can refer to Chapter 6, Section 1. \n",
        "In our case, one-hot encoding of our labels consists in embedding each label as an all-zero vector with a 1 in the place of the label index, e.g.:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Xw6BmzMIVFsD",
        "outputId": "8ccdf97c-c132-49fc-f3e4-3f318dd9915b"
      },
      "source": [
        "train_labels[:5]  #이렇게 하면 train_labels을 볼 수 있고 이러한게 46개 있다는 것이다. 그리고 이것을 one_hot인코딩으로 바꾸고 싶은 것이다."
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([3, 4, 3, 4, 4])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vcfdXO4YB9d9"
      },
      "source": [
        "def to_one_hot(labels, dimension=46):\n",
        "    results = np.zeros((len(labels), dimension))   #이것을 사용해서 우리가 필요한 공간을 잡아 놓는다. 이런식으로 코딩을 해야 파이썬이나 R에서 속도를 더 빠르게 만들 수 있다. 이건 메모리를 잡아놓는 방식이다.\n",
        "    for i, label in enumerate(labels):  #labels를 enumerate를 돌면서  \n",
        "        results[i, label] = 1.   #results라는 메트릭스에 i번째 label번째에다가 즉 단어가 나온 부분에 1.로 바꾼 다는 것이다.\n",
        "    return results  #그리고 여기에서는 results를 반환해준다.\n",
        "\n",
        "# Our vectorized training labels\n",
        "one_hot_train_labels = to_one_hot(train_labels)\n",
        "# Our vectorized test labels\n",
        "one_hot_test_labels = to_one_hot(test_labels)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8CZ_WAlGU2Iz"
      },
      "source": [
        "- 여기에서 movie review와 다른점은 one_hot 인코딩을 해주었다는 것이 다르다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6cOF7KReWMX0",
        "outputId": "d22c63d4-28c0-4e67-ad72-8dfaaf81edaa"
      },
      "source": [
        "one_hot_train_labels[:5]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
              "       [0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
              "       [0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
              "       [0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
              "       [0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vvd8o6A0WZAc"
      },
      "source": [
        "- 그럼 one_hot 인코딩 46개를 이런식으로 메트릭스로 표현을 해준다.\n",
        "- 단어가 들어가는 부분은 1로 표현이 되고, 단어가 안들어가는 부분은 0으로 표현이 된다. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MzSJcLC6B9d9"
      },
      "source": [
        "Note that there is a built-in way to do this in Keras, which you have already seen in action in our MNIST example:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Un0TplTvB9d9"
      },
      "source": [
        "from tensorflow.keras.utils import to_categorical   \n",
        "\n",
        "one_hot_train_labels = to_categorical(train_labels)\n",
        "one_hot_test_labels = to_categorical(test_labels)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SGRwMbmIW2Fa"
      },
      "source": [
        "- 여기는 keras.utils 부분에 to_categorical함수로 구현이 되어 있다. \n",
        "- 그래서 우리가 새로 짤 필요없이 실행해서 쓰면된다. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4OTOb2U6B9d9"
      },
      "source": [
        "## Building our network\n",
        "\n",
        "\n",
        "This topic classification problem looks very similar to our previous movie review classification problem: in both cases, we are trying to \n",
        "classify short snippets of text. There is however a new constraint here: the number of output classes has gone from 2 to 46, i.e. the \n",
        "dimensionality of the output space is much larger. \n",
        "\n",
        "In a stack of `Dense` layers like what we were using, each layer can only access information present in the output of the previous layer. \n",
        "If one layer drops some information relevant to the classification problem, this information can never be recovered by later layers: each \n",
        "layer can potentially become an \"information bottleneck\". In our previous example, we were using 16-dimensional intermediate layers, but a \n",
        "16-dimensional space may be too limited to learn to separate 46 different classes: such small layers may act as information bottlenecks, \n",
        "permanently dropping relevant information.\n",
        "\n",
        "For this reason we will use larger layers. Let's go with 64 units:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aSN1AcMmB9d-"
      },
      "source": [
        "from tensorflow.keras import models\n",
        "from tensorflow.keras import layers\n",
        "\n",
        "model = models.Sequential()\n",
        "model.add(layers.Dense(64, activation='relu', input_shape=(10000,)))\n",
        "model.add(layers.Dense(64, activation='relu'))\n",
        "model.add(layers.Dense(46, activation='softmax'))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kBeiWSg0XI0p"
      },
      "source": [
        "- 여기는 Network를 만드는 부분이다. \n",
        "- Sequential로 만들어서 구성을 하고, 한 층 한층 layer를 선언해준다.\n",
        "- 첫번째 layer는 64, 두번째 layer는 64, 세번째layers는 46으로 구성되어 있다.\n",
        "- 앞에서 movie review와의 차이점은 마지막에 46개가 와야되서 Dense를 46으로 잡아주고, 그리고 movie review에서는 sigmoid를 썼었는데, 여기에서는 softmax를 \n",
        "쓰고 있다. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JyGDkKodB9d-"
      },
      "source": [
        "\n",
        "There are two other things you should note about this architecture:\n",
        "\n",
        "* We are ending the network with a `Dense` layer of size 46. This means that for each input sample, our network will output a \n",
        "46-dimensional vector. Each entry in this vector (each dimension) will encode a different output class.\n",
        "* The last layer uses a `softmax` activation. You have already seen this pattern in the MNIST example. It means that the network will \n",
        "output a _probability distribution_ over the 46 different output classes, i.e. for every input sample, the network will produce a \n",
        "46-dimensional output vector where `output[i]` is the probability that the sample belongs to class `i`. The 46 scores will sum to 1.\n",
        "\n",
        "The best loss function to use in this case is `categorical_crossentropy`. It measures the distance between two probability distributions: \n",
        "in our case, between the probability distribution output by our network, and the true distribution of the labels. By minimizing the \n",
        "distance between these two distributions, we train our network to output something as close as possible to the true labels."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bNvhq5CXB9d-"
      },
      "source": [
        "model.compile(optimizer='rmsprop',\n",
        "              loss='categorical_crossentropy',\n",
        "              metrics=['accuracy'])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iEWS7Sp-X9cj"
      },
      "source": [
        "- compile할 때는 loss를 categorical_crossentropy를 써줘야 한다. \n",
        "- multiplication classification할 때는 categorical_crossentropy를 써줘야 한다. 그래야 maximum likelihood estimate를 구할 수 있다. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W-PhMsJPB9d_"
      },
      "source": [
        "## Validating our approach\n",
        "\n",
        "Let's set apart 1,000 samples in our training data to use as a validation set:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pLkN9TZrYjXQ",
        "outputId": "1c5d88dd-b9d7-4129-e2cd-470dea0dad23"
      },
      "source": [
        "len(x_train)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "8982"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vvoD-lQRYmvq"
      },
      "source": [
        "- 여기에서는 x_train이 8982개 있는 것을 알 수 있다. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PlZcc8m_B9d_"
      },
      "source": [
        "x_val = x_train[:1000]\n",
        "partial_x_train = x_train[1000:]\n",
        "\n",
        "y_val = one_hot_train_labels[:1000]\n",
        "partial_y_train = one_hot_train_labels[1000:]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y_ZktszRYcbO"
      },
      "source": [
        "- 여기에서는 varidation은 1000개를 빼내었다. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IPJL-DtNB9d_"
      },
      "source": [
        "Now let's train our network for 20 epochs:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ejPY9evSB9d_",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "fbdf3c87-e69e-4ce4-8369-bfe8ddd2137b"
      },
      "source": [
        "history = model.fit(partial_x_train,\n",
        "                    partial_y_train,\n",
        "                    epochs=20,\n",
        "                    batch_size=512,\n",
        "                    validation_data=(x_val, y_val))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/20\n",
            "16/16 [==============================] - 2s 62ms/step - loss: 3.1083 - accuracy: 0.3563 - val_loss: 1.7245 - val_accuracy: 0.6350\n",
            "Epoch 2/20\n",
            "16/16 [==============================] - 1s 48ms/step - loss: 1.5072 - accuracy: 0.6910 - val_loss: 1.3046 - val_accuracy: 0.7160\n",
            "Epoch 3/20\n",
            "16/16 [==============================] - 1s 46ms/step - loss: 1.1064 - accuracy: 0.7665 - val_loss: 1.1185 - val_accuracy: 0.7700\n",
            "Epoch 4/20\n",
            "16/16 [==============================] - 1s 46ms/step - loss: 0.8573 - accuracy: 0.8181 - val_loss: 1.0073 - val_accuracy: 0.7880\n",
            "Epoch 5/20\n",
            "16/16 [==============================] - 1s 47ms/step - loss: 0.6752 - accuracy: 0.8561 - val_loss: 0.9486 - val_accuracy: 0.8070\n",
            "Epoch 6/20\n",
            "16/16 [==============================] - 1s 46ms/step - loss: 0.5424 - accuracy: 0.8855 - val_loss: 0.9004 - val_accuracy: 0.8090\n",
            "Epoch 7/20\n",
            "16/16 [==============================] - 1s 46ms/step - loss: 0.4288 - accuracy: 0.9132 - val_loss: 0.8681 - val_accuracy: 0.8180\n",
            "Epoch 8/20\n",
            "16/16 [==============================] - 1s 45ms/step - loss: 0.3398 - accuracy: 0.9339 - val_loss: 0.8823 - val_accuracy: 0.8120\n",
            "Epoch 9/20\n",
            "16/16 [==============================] - 1s 46ms/step - loss: 0.2852 - accuracy: 0.9395 - val_loss: 0.8767 - val_accuracy: 0.8150\n",
            "Epoch 10/20\n",
            "16/16 [==============================] - 1s 47ms/step - loss: 0.2202 - accuracy: 0.9525 - val_loss: 0.8638 - val_accuracy: 0.8190\n",
            "Epoch 11/20\n",
            "16/16 [==============================] - 1s 46ms/step - loss: 0.2102 - accuracy: 0.9495 - val_loss: 0.9006 - val_accuracy: 0.8100\n",
            "Epoch 12/20\n",
            "16/16 [==============================] - 1s 47ms/step - loss: 0.1735 - accuracy: 0.9564 - val_loss: 0.9088 - val_accuracy: 0.8160\n",
            "Epoch 13/20\n",
            "16/16 [==============================] - 1s 46ms/step - loss: 0.1577 - accuracy: 0.9552 - val_loss: 0.9046 - val_accuracy: 0.8220\n",
            "Epoch 14/20\n",
            "16/16 [==============================] - 1s 47ms/step - loss: 0.1289 - accuracy: 0.9610 - val_loss: 0.9742 - val_accuracy: 0.7990\n",
            "Epoch 15/20\n",
            "16/16 [==============================] - 1s 47ms/step - loss: 0.1348 - accuracy: 0.9571 - val_loss: 0.9422 - val_accuracy: 0.8190\n",
            "Epoch 16/20\n",
            "16/16 [==============================] - 1s 47ms/step - loss: 0.1283 - accuracy: 0.9560 - val_loss: 1.0238 - val_accuracy: 0.7980\n",
            "Epoch 17/20\n",
            "16/16 [==============================] - 1s 47ms/step - loss: 0.1105 - accuracy: 0.9609 - val_loss: 1.0257 - val_accuracy: 0.8050\n",
            "Epoch 18/20\n",
            "16/16 [==============================] - 1s 46ms/step - loss: 0.1066 - accuracy: 0.9620 - val_loss: 1.0145 - val_accuracy: 0.8100\n",
            "Epoch 19/20\n",
            "16/16 [==============================] - 1s 47ms/step - loss: 0.1049 - accuracy: 0.9608 - val_loss: 1.0843 - val_accuracy: 0.7990\n",
            "Epoch 20/20\n",
            "16/16 [==============================] - 1s 46ms/step - loss: 0.0998 - accuracy: 0.9612 - val_loss: 1.0609 - val_accuracy: 0.8030\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a2YqEGpcY3mg"
      },
      "source": [
        "- 여기에서는 history를 사용해서 모형을 적합시켜 보고"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bvU9vEF1B9d_"
      },
      "source": [
        "Let's display its loss and accuracy curves:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mz0r31cLY-6_"
      },
      "source": [
        "- 이렇게 모형을 만든 다음에는 시각화를 해준다. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9tE1YKPPB9eA",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "05501326-fb85-4de5-aec6-387415218a56"
      },
      "source": [
        "history.history.keys()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "dict_keys(['loss', 'accuracy', 'val_loss', 'val_accuracy'])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 21
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "H0WSBurwB9eA",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 295
        },
        "outputId": "484aeed6-c7bd-43f9-ceeb-beae3e4397bc"
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "loss = history.history['loss']\n",
        "val_loss = history.history['val_loss']\n",
        "\n",
        "epochs = range(1, len(loss) + 1)\n",
        "\n",
        "plt.plot(epochs, loss, 'bo', label='Training loss')\n",
        "plt.plot(epochs, val_loss, 'b', label='Validation loss')\n",
        "plt.title('Training and validation loss')\n",
        "plt.xlabel('Epochs')\n",
        "plt.ylabel('Loss')\n",
        "plt.legend()\n",
        "\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deZgU5bXH8e9hkR1kU5FtMEFxYR9ARQzG5EaBiLsQrkpwg8Q9aojcKDEhm1yvl0SNuGuIqDHhouISFARjXAARQTCgghLRIAgMsjgD5/7x1jDNMD3Tw0x190z/Ps/TT1dXV1WfrumpU+9Sb5m7IyIiuatOpgMQEZHMUiIQEclxSgQiIjlOiUBEJMcpEYiI5DglAhGRHKdEINXKzJ41swure9lMMrPVZvatGLbrZvb1aPoPZvbTVJbdj88ZZWYv7G+c5Wx3sJmtre7tSvrVy3QAknlmtjXhZWNgJ7Aren2Zu09LdVvufmocy9Z27j62OrZjZnnAh0B9dy+Ktj0NSPlvKLlHiUBw96bF02a2GrjY3WeXXs7M6hUfXESk9lDVkCRVXPQ3sx+b2afAA2bW0syeNrP1ZvZFNN0hYZ25ZnZxND3azF4xs8nRsh+a2an7uWwXM5tnZgVmNtvM7jCzPyaJO5UYf25mf4+294KZtUl4/3wzW2NmG8xsQjn7Z4CZfWpmdRPmnWFmS6Lp/mb2DzPbZGbrzOz3ZnZAkm09aGa/SHh9fbTOJ2Y2ptSyQ83sLTPbYmYfm9nEhLfnRc+bzGyrmR1XvG8T1j/ezN40s83R8/Gp7pvymNmR0fqbzGyZmZ2W8N4QM3s32ua/zOy6aH6b6O+zycw2mtl8M9NxKc20w6UihwCtgM7ApYTfzAPR607AduD35aw/AHgPaAP8FrjPzGw/lv0T8AbQGpgInF/OZ6YS4/eA7wMHAQcAxQemo4C7ou0fGn1eB8rg7q8DXwLfLLXdP0XTu4Brou9zHHAy8INy4iaK4ZQonm8DXYHS7RNfAhcABwJDgXFmdnr03onR84Hu3tTd/1Fq262AZ4Ap0Xe7DXjGzFqX+g777JsKYq4PPAW8EK13BTDNzI6IFrmPUM3YDDgGeCma/yNgLdAWOBi4EdC4N2mmRCAV2Q3c7O473X27u29w9yfdfZu7FwCTgG+Us/4ad7/H3XcBDwHtCP/wKS9rZp2AfsBN7v6Vu78CzEz2gSnG+IC7/9PdtwOPA72i+WcDT7v7PHffCfw02gfJPAqMBDCzZsCQaB7uvtDdX3P3IndfDdxdRhxlOTeKb6m7f0lIfInfb667v+Puu919SfR5qWwXQuJY6e6PRHE9CqwAvpuwTLJ9U55jgabAr6O/0UvA00T7BigEjjKz5u7+hbsvSpjfDujs7oXuPt81AFraKRFIRda7+47iF2bW2MzujqpOthCqIg5MrB4p5dPiCXffFk02reSyhwIbE+YBfJws4BRj/DRheltCTIcmbjs6EG9I9lmEs/8zzawBcCawyN3XRHEcHlV7fBrF8UtC6aAie8UArCn1/QaY2Zyo6mszMDbF7RZve02peWuA9gmvk+2bCmN298SkmbjdswhJco2ZvWxmx0XzbwVWAS+Y2QdmNj61ryHVSYlAKlL67OxHwBHAAHdvTklVRLLqnuqwDmhlZo0T5nUsZ/mqxLgucdvRZ7ZOtrC7v0s44J3K3tVCEKqYVgBdozhu3J8YCNVbif5EKBF1dPcWwB8StlvR2fQnhCqzRJ2Af6UQV0Xb7Viqfn/Pdt39TXcfTqg2mkEoaeDuBe7+I3c/DDgNuNbMTq5iLFJJSgRSWc0Ide6bovrmm+P+wOgMewEw0cwOiM4mv1vOKlWJ8c/AMDM7IWrYvYWK/0/+BFxFSDhPlIpjC7DVzLoB41KM4XFgtJkdFSWi0vE3I5SQdphZf0ICKraeUJV1WJJtzwION7PvmVk9MzsPOIpQjVMVrxNKDzeYWX0zG0z4G02P/majzKyFuxcS9sluADMbZmZfj9qCNhPaVcqripMYKBFIZd0ONAI+B14DnkvT544iNLhuAH4BPEa43qEs+x2juy8Dfkg4uK8DviA0ZpanuI7+JXf/PGH+dYSDdAFwTxRzKjE8G32HlwjVJi+VWuQHwC1mVgDcRHR2Ha27jdAm8veoJ86xpba9ARhGKDVtAG4AhpWKu9Lc/SvCgf9Uwn6/E7jA3VdEi5wPrI6qyMYS/p4QGsNnA1uBfwB3uvucqsQilWdql5GayMweA1a4e+wlEpHaTiUCqRHMrJ+Zfc3M6kTdK4cT6ppFpIp0ZbHUFIcAfyE03K4Fxrn7W5kNSaR2UNWQiEiOU9WQiEiOq3FVQ23atPG8vLxMhyEiUqMsXLjwc3dvW9Z7NS4R5OXlsWDBgkyHISJSo5hZ6SvK91DVkIhIjlMiEBHJcUoEIiI5rsa1EYhI+hUWFrJ27Vp27NhR8cKSUQ0bNqRDhw7Ur18/5XWUCESkQmvXrqVZs2bk5eWR/L5CkmnuzoYNG1i7di1dunRJeb2cqBqaNg3y8qBOnfA8TbfxFqmUHTt20Lp1ayWBLGdmtG7dutIlt1pfIpg2DS69FLZFtzRZsya8Bhg1Kvl6IrI3JYGaYX/+TrW+RDBhQkkSKLZtW5gvIiI5kAg++qhy80Uk+2zYsIFevXrRq1cvDjnkENq3b7/n9VdffVXuugsWLODKK6+s8DOOP/74aol17ty5DBs2rFq2lS61PhF0Kn2Tvwrmi0jVVXe7XOvWrVm8eDGLFy9m7NixXHPNNXteH3DAARQVFSVdNz8/nylTplT4Ga+++mrVgqzBan0imDQJGjfee17jxmG+iFS/4na5NWvAvaRdrro7aYwePZqxY8cyYMAAbrjhBt544w2OO+44evfuzfHHH897770H7H2GPnHiRMaMGcPgwYM57LDD9koQTZs23bP84MGDOfvss+nWrRujRo2ieJTmWbNm0a1bN/r27cuVV15Z4Zn/xo0bOf300+nRowfHHnssS5YsAeDll1/eU6Lp3bs3BQUFrFu3jhNPPJFevXpxzDHHMH/+/OrdYeWo9Y3FxQ3CEyaE6qBOnUISUEOxSDzKa5er7v+7tWvX8uqrr1K3bl22bNnC/PnzqVevHrNnz+bGG2/kySef3GedFStWMGfOHAoKCjjiiCMYN27cPn3u33rrLZYtW8ahhx7KwIED+fvf/05+fj6XXXYZ8+bNo0uXLowcObLC+G6++WZ69+7NjBkzeOmll7jgggtYvHgxkydP5o477mDgwIFs3bqVhg0bMnXqVL7zne8wYcIEdu3axbbSOzFGtT4RQPjx6cAvkh7pbJc755xzqFu3LgCbN2/mwgsvZOXKlZgZhYWFZa4zdOhQGjRoQIMGDTjooIP47LPP6NChw17L9O/ff8+8Xr16sXr1apo2bcphhx22p3/+yJEjmTp1arnxvfLKK3uS0Te/+U02bNjAli1bGDhwINdeey2jRo3izDPPpEOHDvTr148xY8ZQWFjI6aefTq9evaq0byojtqohM+toZnPM7F0zW2ZmV5WxzGAz22xmi6PHTXHFIyLpkc52uSZNmuyZ/ulPf8pJJ53E0qVLeeqpp5L2pW/QoMGe6bp165bZvpDKMlUxfvx47r33XrZv387AgQNZsWIFJ554IvPmzaN9+/aMHj2ahx9+uFo/szxxthEUAT9y96OAY4EfmtlRZSw33917RY9bYoxHRNIgU+1ymzdvpn379gA8+OCD1b79I444gg8++IDVq1cD8Nhjj1W4zqBBg5gWNY7MnTuXNm3a0Lx5c95//326d+/Oj3/8Y/r168eKFStYs2YNBx98MJdccgkXX3wxixYtqvbvkExsicDd17n7omi6AFgOtI/r80QkO4waBVOnQufOYBaep06Nv3r2hhtu4Cc/+Qm9e/eu9jN4gEaNGnHnnXdyyimn0LdvX5o1a0aLFi3KXWfixIksXLiQHj16MH78eB566CEAbr/9do455hh69OhB/fr1OfXUU5k7dy49e/akd+/ePPbYY1x11T6VKLFJyz2LzSwPmAcc4+5bEuYPBp4k3Iz8E+A6d19W3rby8/NdN6YRSa/ly5dz5JFHZjqMjNu6dStNmzbF3fnhD39I165dueaaazId1j7K+nuZ2UJ3zy9r+di7j5pZU8LB/urEJBBZBHR2957A74AZSbZxqZktMLMF69evjzdgEZEk7rnnHnr16sXRRx/N5s2bueyyyzIdUrWItURgZvWBp4Hn3f22FJZfDeS7++fJllGJQCT9VCKoWbKmRGBh5KP7gOXJkoCZHRIth5n1j+LZEFdMIiKyrzivIxgInA+8Y2aLo3k3Ap0A3P0PwNnAODMrArYDIzwdjRYiIrJHbInA3V8Byh0P1d1/D/w+rhhERKRitX6sIRERKZ8SgYhkvZNOOonnn39+r3m3334748aNS7rO4MGDKe5YMmTIEDZt2rTPMhMnTmTy5MnlfvaMGTN4991397y+6aabmD17dmXCL1M2DVetRCAiWW/kyJFMnz59r3nTp09PaeA3CKOGHnjggfv12aUTwS233MK3vvWt/dpWtlIiEJGsd/bZZ/PMM8/suQnN6tWr+eSTTxg0aBDjxo0jPz+fo48+mptvvrnM9fPy8vj889ArfdKkSRx++OGccMIJe4aqhnCNQL9+/ejZsydnnXUW27Zt49VXX2XmzJlcf/319OrVi/fff5/Ro0fz5z//GYAXX3yR3r170717d8aMGcPOnTv3fN7NN99Mnz596N69OytWrCj3+2V6uOqcGH1URKrP1VfD4sUVL1cZvXrB7bcnf79Vq1b079+fZ599luHDhzN9+nTOPfdczIxJkybRqlUrdu3axcknn8ySJUvo0aNHmdtZuHAh06dPZ/HixRQVFdGnTx/69u0LwJlnnskll1wCwH/9139x3333ccUVV3DaaacxbNgwzj777L22tWPHDkaPHs2LL77I4YcfzgUXXMBdd93F1VdfDUCbNm1YtGgRd955J5MnT+bee+9N+v0yPVy1SgQiUiMkVg8lVgs9/vjj9OnTh969e7Ns2bK9qnFKmz9/PmeccQaNGzemefPmnHbaaXveW7p0KYMGDaJ79+5MmzaNZcvKHe2G9957jy5dunD44YcDcOGFFzJv3rw975955pkA9O3bd89Adcm88sornH/++UDZw1VPmTKFTZs2Ua9ePfr168cDDzzAxIkTeeedd2jWrFm5206FSgQiUinlnbnHafjw4VxzzTUsWrSIbdu20bdvXz788EMmT57Mm2++ScuWLRk9enTS4acrMnr0aGbMmEHPnj158MEHmTt3bpXiLR7KuirDWI8fP56hQ4cya9YsBg4cyPPPP79nuOpnnnmG0aNHc+2113LBBRdUKVaVCESkRmjatCknnXQSY8aM2VMa2LJlC02aNKFFixZ89tlnPPvss+Vu48QTT2TGjBls376dgoICnnrqqT3vFRQU0K5dOwoLC/cMHQ3QrFkzCgoK9tnWEUccwerVq1m1ahUAjzzyCN/4xjf267tlerhqlQhEpMYYOXIkZ5xxxp4qouJhm7t160bHjh0ZOHBguev36dOH8847j549e3LQQQfRr1+/Pe/9/Oc/Z8CAAbRt25YBAwbsOfiPGDGCSy65hClTpuxpJAZo2LAhDzzwAOeccw5FRUX069ePsWPH7tf3Kr6Xco8ePWjcuPFew1XPmTOHOnXqcPTRR3Pqqacyffp0br31VurXr0/Tpk2r5QY2aRmGujpp0DmR9NOgczVL1gw6JyIiNYMSgYhIjlMiEJGU1LRq5Fy1P38nJQIRqVDDhg3ZsGGDkkGWc3c2bNhAw4YNK7Weeg2JSIU6dOjA2rVr0a1is1/Dhg3p0KFDpdZRIhCRCtWvX58uXbpkOgyJiaqGRERynBKBiEiOUyIQEclxSgQiIjlOiUBEJMcpEYiI5DglAhGRHKdEICKS45QIRERynBKBiEiOUyIQEclxSgQiIjlOiUBEJMcpEYiI5DglAhGRHBdbIjCzjmY2x8zeNbNlZnZVGcuYmU0xs1VmtsTM+sQVj4iIlC3OG9MUAT9y90Vm1gxYaGZ/c/d3E5Y5FegaPQYAd0XPIiKSJrGVCNx9nbsviqYLgOVA+1KLDQce9uA14EAzaxdXTCIisq+0tBGYWR7QG3i91FvtgY8TXq9l32SBmV1qZgvMbIHumSoiUr1iTwRm1hR4Erja3bfszzbcfaq757t7ftu2bas3QBGRHBdrIjCz+oQkMM3d/1LGIv8COia87hDNExGRNImz15AB9wHL3f22JIvNBC6Ieg8dC2x293VxxSQiIvuKs9fQQOB84B0zWxzNuxHoBODufwBmAUOAVcA24PsxxiMiImWILRG4+yuAVbCMAz+MKwYREamYriwWEclxSgQiIjlOiUBEJMcpEYiI5DglAhGRHKdEICKS45QIRERynBKBiEiOUyIQEclxSgQiIjlOiUBEJMcpEYiI5DglAhGRHKdEICKS45QIRERyXM4kgjfegO9+F778MtORiIhkl5xJBIWF8PTTMHVqpiMREckuOZMIBg6EwYNh8mTYuTPT0YiIZI+cSQQAEybAJ5/Agw9mOhIRkeyRU4ng5JOhf3/4zW+gqCjT0YiIZIecSgRmoVTw4Yfw6KOZjkZEJDvkVCIAGDYMuneHX/0Kdu/OdDQiIpmXc4mgTh248UZYvhz++tdMRyMiknk5lwgAzjkHunaFSZPAPdPRiIhkVk4mgrp1Yfx4eOsteO65TEcjIpJZOZkIAP7zP6FjR/jFL1QqEJHclrOJ4IAD4IYb4NVX4eWXMx2NiEjm5GwiALjoIjj44NBWICKSq3I6ETRqBD/6EcyeHQalExHJRTmdCADGjoWWLVUqEJHclfOJoFkzuOoqmDkT3nkn09GIiKRfbInAzO43s3+b2dIk7w82s81mtjh63BRXLBW54gpo2hR++ctMRSAikjlxlggeBE6pYJn57t4retwSYyzlatUKfvADePxxWLkyU1GIiGRGbInA3ecBG+PafnW79trQpfTXv850JCIi6ZXpNoLjzOxtM3vWzI5OtpCZXWpmC8xswfr162MJ5OCD4eKL4eGH4aOPYvkIEZGslMlEsAjo7O49gd8BM5It6O5T3T3f3fPbtm0bW0DXXx+eb701to8QEck6GUsE7r7F3bdG07OA+mbWJlPxAHTqBBdcAPfcA59+mslIRETSJ6VEYGZNzKxONH24mZ1mZvWr8sFmdoiZWTTdP4plQ1W2WR3Gjw83ur/ttpJ506ZBXl4YwjovL7wWEaktUi0RzAMamll74AXgfEKvoKTM7FHgH8ARZrbWzC4ys7FmNjZa5GxgqZm9DUwBRrhnfvi3rl3hvPPgrrtg48Zw0L/0UlizJgxOt2ZNeK1kICK1haVy7DWzRe7ex8yuABq5+2/NbLG794o/xL3l5+f7ggULYv2MpUvDXcxuvjnc6H7Nmn2X6dwZVq+ONQwRkWpjZgvdPb+s9+qlvg07DhgFXBTNq1sdwWWjY46B4cNhyhT44ouyl1HPIhGpLVKtGroa+AnwV3dfZmaHAXPiCyvzJkwISeDAA8t+v1On9MYjIhKXlEoE7v4y8DJA1Gj8ubtfGWdgmdavH3z722FU0kaNYPv2kvcaN9YgdSJSe6Taa+hPZtbczJoAS4F3zez6eEPLvAkTYPPmcI/jzp3BLDxPnQqjRmU6OhGR6pFqG8FR7r7FzEYBzwLjgYVArb706sQTYeBAmDMHVq0KQ1CIiNQ2qbYR1I+uGzgdmOnuhUDGu3rGzSyUCj7+GP74x0xHIyISj1QTwd3AaqAJMM/MOgNb4goqm5xyCvTpA7/6FRQVZToaEZHql1IicPcp7t7e3Yd4sAY4KebYskJxqWDVKnjiiUxHIyJS/VJtLG5hZrcVjwBqZv9NKB3khNNPh6OOCjeu2b0709GIiFSvVKuG7gcKgHOjxxbggbiCyjZ16oRSwdKlYahqVRGJSG2Saq+hr7n7WQmvf2Zmi+MIKFuNHAn//Cf87GdhDKLp06Fhw0xHJSJSdamWCLab2QnFL8xsILC9nOVrHTOYOBF+9zv4v/8LjcibN2c6KhGRqku1RDAWeNjMWkSvvwAujCek7Hb55eEexxdeCCedBM89BwcdlOmoRET2X6q9ht6O7iTWA+jh7r2Bb8YaWRb73vdg5kxYsQJOOEGjkIpIzVapO5RFdxUrvn7g2hjiqTFOPRVmz4b168PVx8uWZToiEZH9U5VbVVq1RVFDHX88zJsXblgzaBD84x+ZjkhEpPKqkghq/RATqejeHf7+99Bu8K1vwfPPZzoiEZHKKTcRmFmBmW0p41EAHJqmGLNely4hGXTtCt/9buhaKiJSU5Tba8jdm6UrkJru4IPh5ZfhtNNCY/IXX8C4cZmOSkSkYlWpGpJSWrQI3UmHDYMf/AB+/vPQfiAiks2UCKpZo0bwl7+E6wxuugmuvlrjE4lIdkv1gjKphHr14P77oXVruO022LABHngA6tfPdGQiIvtSIohJnToweTK0bQs/+UloM3jiiXC/YxGRbKKqoRiZwfjx4R7Hzz0H3/42fPBBpqMSEdmbEkEaXHIJPPYYLF4M3brBNdeE6iIRkWygRJAmZ58NK1eGRuQpU+BrX4Nbb4UdOzIdmYjkOrURpMG0aeHGNh99BJ06waRJMH8+3HAD3HFHeD1yZGhXEJHaZfXqcJHpX/8aqos7dgzHgU6d9p5u2za8nwnmNayje35+vi9YsCDTYaRs2jS49FLYtq1kXuPGod3gkEPg+uvhrbegT5/QuHxSTtwJWqR2W7cudA559FF47bUwb8AAaNYsnBB+/DFsL3VHlwYNQmJIlig6doSmTfc/JjNb6O75Zb6nRBCvvDxYs2bf+Z07hzOF3bvhT38qKTEMHQq/+Q0cfXS6IxWRqti4EZ58Mpz9z50b/rd79gyl/fPOC8eCYu6hnfDjj8P/fXFySHz+5JN9r0G64YZwfNgfSgQZVKdO2VcXm+39R96xI7Qd/PKXUFAAY8bALbdAu3bpi1VEKqegINyxcPr0MOBkUVEYc2zkSBgxAo48cv+3XVQUkkFicsjPh5NP3r/tZSQRmNn9wDDg3+5+TBnvG/C/wBBgGzDa3RdVtN2alggqKhGU9vnn8ItfwJ13hgvQrrsuVB9VpUgoUhvs3AlffhlOoHbt2r/nxo2hefMwHEyzZlC3buXj2L4dnn02VPs8/XQ4ievYMRz4R46EXr0yV9dfnkwlghOBrcDDSRLBEOAKQiIYAPyvuw+oaLs1LRGU10YwalTy9d5/P1yI9sQTYUC7n/0MLrooXLUskivc4dVX4e674fHHQzKoTk2ahKTQvHnJI/F14nSDBvDii6HRt6Ag3KL2nHPCwf+447K/s0d5iSC2w4q7zzOzvHIWGU5IEg68ZmYHmlk7d18XV0yZUHywL91rqLwkAKF76eOPh4am666DsWPhf/4njGw6dCj07p39PzyR/bV5MzzySEgAS5eGA/GYMeE6nDp1wpl8ec/J5m3bBlu2hMfmzSXTia/XrSuZLijYu2q3RYuSg//gwbXnxCzWNoIoETydpETwNPBrd38lev0i8GN3L/d0v6aVCKqDO8yYAb/9Lbz+enh9yCHhdplDhoQrllu0yHSUIlXjDm++GQ7+06eHg3Z+fjgJGjEinL2n2+7doTqqOCl06RJKBjVRRkoE1cnMLgUuBejUqVOGo0k/MzjjjPD4979Do9Qzz4Qi6gMPhLOSE04ISWHo0NBAlY11lFKzbdwY2qoOOKB6t1tQEHrO3X136ErdpEkoMV92GfTtW72fVVl16oS2hGa1/M4smSwR3A3MdfdHo9fvAYMrqhrKxRJBMkVF4T7Js2aFx5IlYX5eXkgKQ4aE6xI00J3sr88+C+1U06eHu/DVrRuqLbt1CyccRx4Zprt1q3yp9K23wsF/2jTYujV0tbzsspAEmjeP5/vksox1H60gEQwFLqeksXiKu/evaJtKBMl9/HHozfDMMzB7dihaN2wYksHQoSExdOmS6Sgl223cGEqbjz4Kc+aE6pEePeCss+Crr2D5clixIgyZUlhYsl67dnsnh+Lpdu1KSqhffhnG3br7bnjjjfD7HDEiJIABA1SSjVOmeg09CgwG2gCfATcD9QHc/Q9R99HfA6cQuo9+v6L2AVAiSNXOneHWmbNmhcSwalWY36kTDBpU8lA1kkConpk5s6Q/fGEhfP3rJf3hjzpq33UKC+HDD0NiKE4OxdMFBSXLNW8eEkPHjuEEZfPm8LsbOxbOPx9atkzf98xluqBMWLkyDIU9f354fPppmN+6dWhfKE4MvXvrBjq5org//PTpoT/89u0l/eFHjAi/hf05SXAPPW8Sk8OKFaFL9PHHhwRwwgk6AUk3JQLZi3v4pyxOCvPnl5QYmjSBY48tSQzHHqs2htqksDCclT/6aOiJVtwf/txzw8G/JvSHl/2jRCAVWrdu78SwZElIGPXqhZ4bgwaFs7ju3UNjtA4WqSs+Q162LFSLfPllaL8pfk6cLm/e9u1hv9evH3ruJD6nMq+wMFT7bNwIBx4Y6vxHjKhd/eElOSUCqbRNm8IVncWJ4c03Q0MhhAa+I44oaQwsfnTtWnP7WFeXnTvh3Xfh7bdDMn377fAo70ZEDRqEUleTJuE5cTpxXqNGIal89VU4qCd7Tvbe7t0hoY8YAd/5TvV3A5XspkQgVbZ9e+ju9+67JQ2Cy5fvPV5S3bpw2GH7Jogjjyy7H/bu3eFCnY0bU3ts2hQ+o0GD5I+GDct/r0WLcDbcsuXez5VtF3EP7SyJB/slS0JdeFFRWKZRIzjmmNAtsmfPMN2mzd4H+EaNdDYu6aFEUMOVdWObioaoSJdt2+C99/ZNECtXlhwQAdq3D71QduwoObB/8cW+w+wmatYMWrUqebRoEQYP27mz7MeOHfvOS/Xn3aRJSAhlJYni58aNw/cqPvCvX1+yfseOoYtl8UG/Z8/wffdnUDOROCgR1GD7O2hdphUWhgbpxOTw4YfhgJt4cC9+tG699+uWLavee8k9JKPiJLFjR6ij37QpPL74ouzn0vM2by7ZZoMGe5/l9+gRHq1aVS1WkbgpESJt+d4AAAuySURBVNRglR3GWqrfrl2hCmvr1nBxlKpypCaq8WMN5bKPPqrcfKl+deuGEooufJLaSp0As1yyMfZycOw9EYmJEkGWmzRp3wu6GjcO80VEqoMSQZYbNSo0DHfuHC7J79w5+xuKRaRmURtBDTBqlA78IhIflQhERHKcEoGISI5TIhARyXFKBCIiOU6JQEQkxykRiIjkOCWCHDBtWsnNZPLywmsRkWK6jqCWKz166Zo14TXo2gQRCVQiqOUmTNh7CGsIrydMyEw8IpJ9lAhqOY1eKiIVUSKo5TR6qYhURImgltPopSJSESWCWk6jl4pIRdRrKAdo9FIRKY9KBCIiOU6JQEQkxykRiIjkOCUCSYmGqRCpvdRYLBXSMBUitVusJQIzO8XM3jOzVWY2voz3R5vZejNbHD0ujjMe2T8apkKkdoutRGBmdYE7gG8Da4E3zWymu79batHH3P3yuOKQqtMwFSK1W5wlgv7AKnf/wN2/AqYDw2P8PImJhqkQqd3iTATtgY8TXq+N5pV2lpktMbM/m1nHsjZkZpea2QIzW7B+/fo4YpVyaJgKkdot072GngLy3L0H8DfgobIWcvep7p7v7vlt27ZNa4CiYSpEars4ew39C0g8w+8QzdvD3TckvLwX+G2M8UgVaJgKkdorzhLBm0BXM+tiZgcAI4CZiQuYWbuEl6cBy2OMRzJI1yGIZK/YSgTuXmRmlwPPA3WB+919mZndAixw95nAlWZ2GlAEbARGxxWPZI6uQxDJbubumY6hUvLz833BggWZDkMqIS8vHPxL69wZVq9OdzQiucnMFrp7flnvZbqxWHKArkMQyW5KBBI7XYcgkt2UCCR2ug5BJLspEUjsdB2CSHZTIpC0GDUqNAzv3h2eK5sE1P1UJD4ahlqynrqfisRLJQLJehoGWyReSgSS9dT9VCReSgSS9aqj+6naGESSUyKQrFfV7qfFbQxr1oB7SRuDkoFIoEQgWa+q3U/VxiBSPo01JLVenTqhJFCaWejOKpILNNaQ5DS1MYiUT4lAaj21MYiUT4lAar1saGNQiUKymdoIRCpQ1TaG0ldGQyiRaLwlSSe1EYhUQVXbGFSikGynRCBSgaq2MVT1ymi1UUjclAhEKlDVNgaVKCTbKRGIpKAqw2jXhhKFEkntpkQgErOaXqJQIskB7l6jHn379nWRXPLHP7o3buweDsPh0bhxmJ8Ks73XLX6YpbZ+585lr9+5c3riL95G584h5s6dK7duNqyfDYAFnuS4mvEDe2UfSgSSi6pyIKrqgbymJ5JMr1+8jUwnIiUCkRxW1QNZTU8kmV4/GxKRuxKBSM6ryhllTU8kmV4/04moWHmJQI3FIjmgKr2eqtrYXdVeU1VtLM/0+lXt9ZWOO/QpEYhIhWpyIsn0+plORClJVlTI1oeqhkRyT6YbWzNZtZaONgINOiciErNp08J1Gx99FM7kJ02qXKmqqutD+YPOKRGIiOQAjT4qIiJJxZoIzOwUM3vPzFaZ2fgy3m9gZo9F779uZnlxxiMiIvuKLRGYWV3gDuBU4ChgpJkdVWqxi4Av3P3rwP8Av4krHhERKVucJYL+wCp3/8DdvwKmA8NLLTMceCia/jNwsplZjDGJiEgpcSaC9sDHCa/XRvPKXMbdi4DNQOvSGzKzS81sgZktWL9+fUzhiojkpnqZDiAV7j4VmApgZuvNbE2GQ0qmDfB5poMoR7bHB9kfo+KrGsVXNVWJr3OyN+JMBP8COia87hDNK2uZtWZWD2gBbChvo+7etjqDrE5mtiBZ96xskO3xQfbHqPiqRvFVTVzxxVk19CbQ1cy6mNkBwAhgZqllZgIXRtNnAy95TbuwQUSkhoutRODuRWZ2OfA8UBe4392XmdkthEudZwL3AY+Y2SpgIyFZiIhIGsXaRuDus4BZpebdlDC9AzgnzhjSbGqmA6hAtscH2R+j4qsaxVc1scRX44aYEBGR6qUhJkREcpwSgYhIjlMiqCQz62hmc8zsXTNbZmZXlbHMYDPbbGaLo8dNZW0rxhhXm9k70WfvM1SrBVOiMZ6WmFmfNMZ2RMJ+WWxmW8zs6lLLpH3/mdn9ZvZvM1uaMK+Vmf3NzFZGzy2TrHthtMxKM7uwrGViiu9WM1sR/Q3/amYHJlm33N9DjPFNNLN/JfwdhyRZt9wxyWKM77GE2Fab2eIk68a6/5IdU9L6+0t2owI9yn4A7YA+0XQz4J/AUaWWGQw8ncEYVwNtynl/CPAsYMCxwOsZirMu8CnQOdP7DzgR6AMsTZj3W2B8ND0e+E0Z67UCPoieW0bTLdMU338A9aLp35QVXyq/hxjjmwhcl8Jv4H3gMOAA4O3S/09xxVfq/f8GbsrE/kt2TEnn708lgkpy93XuviiaLgCWs+/QGdluOPCwB68BB5pZuwzEcTLwvrtn/Epxd59H6MKcKHEsrIeA08tY9TvA39x9o7t/AfwNOCUd8bn7Cx6GZgF4jXDRZkYk2X+pSGVMsiorL75ofLNzgUer+3NTUc4xJW2/PyWCKoiGze4NvF7G28eZ2dtm9qyZHZ3WwMCBF8xsoZldWsb7qYwDlQ4jSP7Pl8n9V+xgd18XTX8KHFzGMtmyL8cQSnllqej3EKfLo6qr+5NUbWTD/hsEfObuK5O8n7b9V+qYkrbfnxLBfjKzpsCTwNXuvqXU24sI1R09gd8BM9Ic3gnu3ocwBPgPzezENH9+haKrzU8Dnijj7Uzvv314KIdnZV9rM5sAFAHTkiySqd/DXcDXgF7AOkL1SzYaSfmlgbTsv/KOKXH//pQI9oOZ1Sf8waa5+19Kv+/uW9x9azQ9C6hvZm3SFZ+7/yt6/jfwV0LxO1Eq40DF7VRgkbt/VvqNTO+/BJ8VV5lFz/8uY5mM7kszGw0MA0ZFB4t9pPB7iIW7f+buu9x9N3BPks/N9P6rB5wJPJZsmXTsvyTHlLT9/pQIKimqT7wPWO7utyVZ5pBoOcysP2E/lzuYXjXG18TMmhVPExoUl5ZabCZwQdR76Fhgc0IRNF2SnoVlcv+VkjgW1oXA/5WxzPPAf5hZy6jq4z+iebEzs1OAG4DT3H1bkmVS+T3EFV9iu9MZST43lTHJ4vQtYIW7ry3rzXTsv3KOKen7/cXVEl5bH8AJhCLaEmBx9BgCjAXGRstcDiwj9IB4DTg+jfEdFn3u21EME6L5ifEZ4e5x7wPvAPlp3odNCAf2FgnzMrr/CElpHVBIqGe9iHBvjBeBlcBsoFW0bD5wb8K6Y4BV0eP7aYxvFaF+uPh3+Ido2UOBWeX9HtIU3yPR72sJ4aDWrnR80eshhJ4y76czvmj+g8W/u4Rl07r/yjmmpO33pyEmRERynKqGRERynBKBiEiOUyIQEclxSgQiIjlOiUBEJMcpEYhEzGyX7T0yarWNhGlmeYkjX4pkk1hvVSlSw2x3916ZDkIk3VQiEKlANB79b6Mx6d8ws69H8/PM7KVoULUXzaxTNP9gC/cHeDt6HB9tqq6Z3RONOf+CmTWKlr8yGot+iZlNz9DXlBymRCBSolGpqqHzEt7b7O7dgd8Dt0fzfgc85O49CAO+TYnmTwFe9jBoXh/CFakAXYE73P1oYBNwVjR/PNA72s7YuL6cSDK6slgkYmZb3b1pGfNXA9909w+iwcE+dffWZvY5YdiEwmj+OndvY2brgQ7uvjNhG3mEceO7Rq9/DNR391+Y2XPAVsIoqzM8GnBPJF1UIhBJjSeZroydCdO7KGmjG0oY+6kP8GY0IqZI2igRiKTmvITnf0TTrxJGywQYBcyPpl8ExgGYWV0za5Fso2ZWB+jo7nOAHwMtgH1KJSJx0pmHSIlGtvcNzJ9z9+IupC3NbAnhrH5kNO8K4AEzux5YD3w/mn8VMNXMLiKc+Y8jjHxZlrrAH6NkYcAUd99Ubd9IJAVqIxCpQNRGkO/un2c6FpE4qGpIRCTHqUQgIpLjVCIQEclxSgQiIjlOiUBEJMcpEYiI5DglAhGRHPf/ZX9pNB2rQX0AAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "snyrSC-4ZFUJ"
      },
      "source": [
        "- 여기에서도 validatin loss가 줄어들다가 조금씩 올라가는 것을 볼 수 있다.\n",
        "- 그리고 training loss는 계속 줄어드는 것을 볼 수 있다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Dvc6QcrpB9eB",
        "outputId": "39d49d44-9494-4685-8801-ab3110dc1e28"
      },
      "source": [
        "plt.clf()   # clear figure\n",
        "\n",
        "acc = history.history['accuracy']\n",
        "val_acc = history.history['val_accuracy']\n",
        "\n",
        "plt.plot(epochs, acc, 'bo', label='Training acc')\n",
        "plt.plot(epochs, val_acc, 'b', label='Validation acc')\n",
        "plt.title('Training and validation accuracy')\n",
        "plt.xlabel('Epochs')\n",
        "plt.ylabel('Loss')\n",
        "plt.legend()\n",
        "\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XmcFNW5//HPwy6LbIMbyOISNwIII+gNKmhCcAMDuBBc0aBGNBrNDS6JxiV63aJGf17R6BUZJUSvCl5xQyIaNwZlEVxAQTOCCMi+Dz6/P05N0zQ9Mz1LLzPzfb9e/ZrqqlPVT1fX1FPnVNUpc3dEREQA6mU7ABERyR1KCiIiEqOkICIiMUoKIiISo6QgIiIxSgoiIhKjpCC7MLP6ZrbezDpWZ9lsMrMDzKzar782s5+a2eK495+Z2dGplK3EZz1qZtdWdn6RVDTIdgBSdWa2Pu5tU2ALsD16f5G7F1Rkee6+HWhe3WXrAnc/qDqWY2YXAme5e7+4ZV9YHcsWKYuSQi3g7rGdcnQkeqG7v15aeTNr4O7FmYhNpDzaHnOLmo/qADO7xcz+bmZPm9k64CwzO8rM3jOz1Wa21MzuN7OGUfkGZuZm1jl6Pz6aPsXM1pnZu2bWpaJlo+knmNnnZrbGzP5qZv8ys/NKiTuVGC8ys4VmtsrM7o+bt76Z/cXMVprZF8DAMtbP9WY2IWHcg2Z2TzR8oZl9En2fL6Kj+NKWVWRm/aLhpmb2ZBTbPKBXks/9MlruPDMbFI3/MfAAcHTUNLcibt3eGDf/xdF3X2lmz5vZ3qmsm4qs55J4zOx1M/vezL41s/+M+5w/ROtkrZkVmtk+yZrqzOztkt85Wp/To8/5HrjezA40s2nRd1kRrbeWcfN3ir7j8mj6fWbWJIr5kLhye5vZRjNrW9r3lXK4u1616AUsBn6aMO4WYCtwCuFAYDfgCKAPoba4H/A5MDoq3wBwoHP0fjywAsgHGgJ/B8ZXouwewDpgcDTtt8A24LxSvksqMb4AtAQ6A9+XfHdgNDAP6AC0BaaHzT3p5+wHrAeaxS37OyA/en9KVMaA44BNQLdo2k+BxXHLKgL6RcN3Af8EWgOdgPkJZU8H9o5+k19GMewZTbsQ+GdCnOOBG6PhAVGMPYAmwP8D3khl3VRwPbcElgG/ARoDuwO9o2nXALOBA6Pv0ANoAxyQuK6Bt0t+5+i7FQOXAPUJ2+OPgOOBRtF28i/grrjv83G0PptF5X8STRsL3Br3OVcBz2X7/7Amv7IegF7V/IOWnhTeKGe+q4F/RMPJdvT/HVd2EPBxJcqOBN6Km2bAUkpJCinGeGTc9P8Fro6GpxOa0UqmnZi4o0pY9nvAL6PhE4DPyyj7InBpNFxWUvg6/rcAfh1fNslyPwZOiobLSwpPAH+Om7Y74TxSh/LWTQXX89lAYSnlviiJN2F8Kknhy3JiGAbMiIaPBr4F6icp9xNgEWDR+1nAkOr+v6pLLzUf1R3/jn9jZgeb2f9FzQFrgZuAvDLm/zZueCNln1wurew+8XF4+C8uKm0hKcaY0mcBX5URL8BTwPBo+JdA7OS8mZ1sZu9HzSerCUfpZa2rEnuXFYOZnWdms6MmkNXAwSkuF8L3iy3P3dcCq4D2cWVS+s3KWc/7AgtLiWFfQmKojMTtcS8zm2hm30Qx/E9CDIs9XNSwE3f/F6HW0dfMugIdgf+rZEyCzinUJYmXYz5MODI9wN13B/5IOHJPp6WEI1kAzMzYeSeWqCoxLiXsTEqUd8ns34GfmlkHQvPWU1GMuwHPALcRmnZaAa+mGMe3pcVgZvsBDxGaUNpGy/00brnlXT67hNAkVbK8FoRmqm9SiCtRWev538D+pcxX2rQNUUxN48btlVAm8fv9F+GquR9HMZyXEEMnM6tfShzjgLMItZqJ7r6llHKSAiWFuqsFsAbYEJ2ouygDn/ki0NPMTjGzBoR26nZpinEicIWZtY9OOv6+rMLuvozQxPE48Jm7L4gmNSa0cy8HtpvZyYS271RjuNbMWlm4j2N03LTmhB3jckJ+vJBQUyixDOgQf8I3wdPABWbWzcwaE5LWW+5eas2rDGWt50lARzMbbWaNzGx3M+sdTXsUuMXM9regh5m1ISTDbwkXNNQ3s1HEJbAyYtgArDGzfQlNWCXeBVYCf7Zw8n43M/tJ3PQnCc1NvyQkCKkCJYW66yrgXMKJ34cJR8ppFe14zwDuIfyT7w98RDhCrO4YHwKmAnOBGYSj/fI8RThH8FRczKuBK4HnCCdrhxGSWypuINRYFgNTiNthufsc4H7gg6jMwcD7cfO+BiwAlplZfDNQyfwvE5p5novm7wiMSDGuRKWuZ3dfA/wMGEo4sf05cGw0+U7gecJ6Xks46dskahb8FXAt4aKDAxK+WzI3AL0JyWkS8GxcDMXAycAhhFrD14TfoWT6YsLvvNXd36ngd5cEJSdnRDIuag5YAgxz97eyHY/UXGY2jnDy+sZsx1LT6eY1ySgzG0hoDthMuKSxmHC0LFIp0fmZwcCPsx1LbaDmI8m0vsCXhGaFgcCpOjEolWVmtxHulfizu3+d7XhqAzUfiYhIjGoKIiISU+POKeTl5Xnnzp2zHYaISI0yc+bMFe5e1iXgQA1MCp07d6awsDDbYYiI1ChmVt5d/YCaj0REJI6SgoiIxCgpiIhIjJKCiIjEKCmIiEiMkoKISI4rKIDOnaFevfC3oKC8OSpPSUFEcl5Vd4o1ef6CAhg1Cr76CtzD31Gj0pgYsv3ot4q+evXq5SJSMePHu3fq5G4W/o4fX3PmHz/evWlT97BLDK+mTVNfRk2fv1OnnecteXXqlNr8JSjlsaqJr6zv5Cv6UlIQqZhs79SyvVOs6fObJZ/fLLX5SygpiOSIqh5lV3UZ2d6pZXunWNPnz3RNQecURMqR7fbgqi7j61I6lC5tfK7N37GUp2uXNr62zX/rrdC06c7jmjYN49MilcyRSy/VFCSTst30UR3LqOnzZ7v5KtvzlyyjqrVN1HwkUnXZbvqojmVke6eWCzvFmj5/dVBSEKkGudAeXB3LyPZOLRd2inWdkoJIJJsnaavrKLmqyxBJNSnoRLPUalU9SVvVk3wjRsDYsdCpE5iFv2PHhvGpqo5liKSqxj2jOT8/3/WQHUlV584hESTq1AkWL05tGQUFcN114WqZjh1DQtAOWWoaM5vp7vnllVNNQXJeVS4JrerlkBASwOLF8MMP4a8SgtRmSgqS06ra/FPVa8RF6holBclp110HGzfuPG7jxjA+FRm/8UekhlNSkJxW1eYfnaQVqZgG2Q5ApCwdOyY/UVyR5p8RI5QERFKlmoLkNDX/iGSWkoLkNDX/iGSWkoKkXVWfWqVLQkUyR+cUJK1KLiktuYKo5JJS0M5dJBeppiBpVdVLSkUks5QUJK2q445iEckcJQVJK91RLFKzpDUpmNlAM/vMzBaa2Zgk0zuZ2VQzm2Nm/zSzDumMRzJPl5SK1CxpSwpmVh94EDgBOBQYbmaHJhS7Cxjn7t2Am4Db0hWPZIcuKRWpWdJ59VFvYKG7fwlgZhOAwcD8uDKHAldGw9OA59MYj2SJ7igWqTnS2XzUHvh33PuiaFy82cDQaPgXQAsza5u4IDMbZWaFZla4fPnytAQrIiLpTQqWZFziE32uBo41s4+AY4FvgOJdZnIf6+757p7frl276o9UylTVm89EpOZIZ/NREbBv3PsOwJL4Au6+BBgCYGbNgaHuviaNMUkF6eYzkbolnTWFGcCBZtbFzBoBZwKT4guYWZ6ZlcRwDfBYGuORStDNZyJ1S9qSgrsXA6OBV4BPgInuPs/MbjKzQVGxfsBnZvY5sCegCxVzjG4+E6lb0tr3kbu/BLyUMO6PccPPAM+kMwapmup4noGI1By6o1nKpJvPROoWJQUpk24+E6lb1HW2lEs3n4nUHaopiIhIjJKCiIjEKCmIiEiMkoKIiMQoKYiISIySQh2gDu1EJFW6JLWWU4d2IlIRqinUcurQTkQqQkmhllOHdiJSEUoKtVxpHdepQzsRSUZJoZZTh3YiUhFKCrWcOrQTkYrQ1Ud1gDq0E5FUqaYgIiIxSgoiIhKjpCAiIjFKCiIiEqOkICIiMUoKIiISo6RQA6iXUxHJFN2nkOPUy6mIZJJqCjlOvZxKXbZmDTz5JAwaBG3bwn/+J2zZku2oajfVFHKcejmtudxh1Sr45htYsiQMd+oEBx0EbdpkO7rctWYNTJoEEyfCq6/C1q3QoQMceSTceSe8/DKMHw/dumU70tpJSSHHdewYmoySjc9V7rBuXdgRLlmyY6e4ZQuce27YMdZ069fv+H7x3zFxuLSj2ry8kBwOPjj8LRnu0gUaNqxcTCVJaNGi8Fq8eMfwokVhZ/unP8GvflXpr502q1eHRPCPf+xIBPvuC5deCqedBn36hHNq//d/cMEFkJ8PN98MV18N9etnO/raxdw92zFUSH5+vhcWFmY7jIxJPKcAoZfTbHVqt2kTLF2afAcYP27DhuTzN2gAZ58N11wDBx6YubjXrYNp08LOfOvWsLOu6N+NG+Hbb8N3XLt2189o1gzat4d99gmv+OF99oFWrcKO+rPPdrw+/RS++27HMho0gP333zVhHHRQSCTr15e+01+8eNe4WrUKiaZLl/A5b78dtqf774fGjdO4wlMQnwheeQW2bQuJYNgwOP106N07JIJEK1bAxRfDs89C377wxBOw336Zjz+TVq+G6dOhe/fKH1SZ2Ux3zy+3nJJC7isoCOcQvv461BBuvTVzCWHlynB0NmkS/POf4X2ixo133QEmvt9773CkeuedIaFt3Qpnnhm+16GHpi/+uXPhoYdCc8O6dWWXbdQovBo33vE3frhJE9hrr12/W8n3bdGicjGuXr0jQcQni4ULw3oq0bTprueXmjYNO/zOnXfs/EtenTuHpFBi+3b4wx/gtttCU8wzz4S4M6kkEZQ0DZUkgtNOC6/SEkEi9/Cbjh4NP/wAf/lLqEGYpf87ZML69SGBv/FGOJj58MPwPe+6C666qnLLVFKQSvviC3jhhfB6++2wMe6zDwwcGI5iE3eIrVtX7J9x2TK4+274f/8v7OSGDIHrr4cePaon/s2bw1HkQw/Bv/4VduZnnAHnnReSU/yOvuRvw4a5t0MpLg5NhyXJ4uuvQ1KKTwDt2lU87mefDeuiWbNwlH700emIfmezZsENN8CUKSERdOwYagQVSQTJfP11+C7TpsEpp8Ajj8Cee1Zr6BmxaRO8++6OJPDBB+H3b9gwJPDjjoP+/cNwZWt4qSYF3L1GvXr16uVSvbZvd3/3XfdrrnE/9FD3cBzm3q2b+/XXu8+Y4f7DD9X/ucuXh+Xvvnv4vFNOcX///covb+FC99/9zj0vLyzvwAPd777bfeXK6ou5tpg3z/1HP3Jv0MD9/vvT8/u6u69Y4X7xxe716oXf5be/dX/vver9vO3b3f/yF/fGjcNn/O//Vt+y02XLFve33nK/6Sb3fv1C7OBev757nz7hf/HVV903bKi+zwQKPYV9bNZ38hV9KSlUj40b3SdPdr/wQvc999yxQR53nPt997l/+WXmYlm1yv3mm93btAlx/Oxn7tOnpzbvtm3uzz/v/vOf7/gOQ4a4v/Za2FlI6VavDokY3M8+O2wT1aW42P3BB91btw6/yeWXh985nebNc+/ZM3yfc88N3y9XbN4ckuHtt7sPGODetGmI08z98MPdr7rK/cUX3desSV8MOZEUgIHAZ8BCYEyS6R2BacBHwBzgxPKWqaRQed995/744+6nnrpjo2zRwv30090LCty//z678a1b537HHe577BFiO+aYsHNPdlT5zTfhKKtDh1C2fXv3P/3Jvago83HXZNu3h/VWsnNatKjqy5w+3b179/C79O/vPndu1ZeZqi1bQu2zXj33jh3dp03L3GeX2LrVfdYs90cecb/oopCoGjbcUQM/7DD3yy4LNZpM1mKznhSA+sAXwH5AI2A2cGhCmbHAJdHwocDi8parpFBxS5a4X3BB+EeBsCP99a/dX3klHMHkmg0bQm2lffsQb58+4Shq+3b31193Hzo0NHtAOOp67rlQY5DKe/FF95Yt3du2DYm4MoqK3IcPD7/Lvvu6T5yYvmap8rz7bmg+hNBktWlTej6nuNh9/nz3cePCjv6oo9ybNNmRAFq2dD/+ePcxY9yfecb922/TE0cqciEpHAW8Evf+GuCahDIPA7+PK/9OectVUkjdhg2hWaZZs3Ck8pvfuM+cmb1/1IravNn9v//bvVOnsKW2ahX+tmnjfvXV7gsWZDvC2uXzz8NRbL16ocaW6nayebP7bbeF7axxY/c//KF628Ira/36cPBTcnT+4YdVW94PP4TzVhMmhOaeY491b958RwJo1sz96KNDEnrqqbB95tL/Wi4khWHAo3HvzwYeSCizNzAXKAJWAb1KWdYooBAo7NixY9pWWm2xfbv7k0/uaFoZOjRszDXV1q2h2ev008MRWbqO+iQ04Q0bFrab004L78vy4ovuBxwQyg8e7P7FF5mJsyKmTHHfe++Q7Fq2rPyrpMkVQvLr08f90kvd/+d/3D/+ONQaclmqSSFtl6Sa2WnAz939wuj92UBvd78srsxvCZfF3m1mRwF/A7q6+w+lLVeXpJbtrbfgt7+FwsJw1+c992TmkkOpPdzD/STXXAOHHALPPw8HHLBzmYUL4Yorwj0sBx0E990HP/95duJNxfffhxjXrKn8MurVC+sjPx+6dq38nefZkuolqens5qII2DfufQdgSUKZCwgno3H3d82sCZAHfIdUyBdfwO9/H65B79AhdCL2y19W/vpvqbvMQsdzhx8ebjDMzw83UJ50Urip6s9/DveZNGoUksfll4fhXNamTejiQ8qXzl3GDOBAM+tiZo2AM4FJCWW+Bo4HMLNDgCbA8jTGVOusXg2/+124K3jKFLjppnCj01lnKSFI1fzsZ6HG2aVLuDFs1KjQ9cZtt4Vk8fnnoe+hXE8IUjFpqym4e7GZjQZeIVyJ9Ji7zzOzmwhtW5OAq4BHzOxKwIHzPF3tWbXMtm2hu4gbbghV4/PPDx2E7bNPtiOT2qRLl3BX+EUXhbuFe/YMXVT8x39kOzJJF3VzkQHV2XeRO7z0UjhC+/TTcOv7PfdUXxcRIsm4w7x5oU1dvZLWTKmeU1ADQ5qV9HL61VfhH6vkyWmVeaTm3LkwYACcfHLo3OyFF2DqVCUEST+zcHJVCaH2U1JIs+p4ctrq1aE3yB49YObMcBXFxx+Hp1HlWiduIlKz6SE7aVaVJ6e5w9NPh0tMly+HX/86XEGhp3aJSLqoppBmpT0hrbwnp332Wbj6Y8SIUPaDD+Cvf1VCEJH0UlJIs1tvDQ9Cide0aRifzKZN8Mc/hufPFhbCgw+GftZ79Up/rCIiSgppNmJEuHS0U6fQ/t+pU+mP0nz55XAy7+abw8NHPv00NBnp5J6IZIrOKWTAiBFlX4L6zTdw5ZXhKVg/+lG4oui44zIXn4hICdUUsqi4GO69N9wlOnlyqCHMmaOEICLZo5pClrz3HlxySXh27QknwAMPwH77ZTsqEanrVFPIsFWr4OKLQzcBy5fDM8+EniaVEEQkFygpZIh76Ln0oIPg0UdDt8OffAJDh+oGNBHJHWo+yoDt20PXFo89BkceCa+9Bt27ZzsqEZFdqaaQZsXFcN55ISFcf33ocVIJQURylWoKabRtW3iuwcSJ4Wa1a6/NdkQiImVTUkiTLVvCg0iefx7uuguuuirbEYmIlE9JIQ02bw4nkF96Ce6/Hy67rPx5RERygZJCNdu4EU49FV5/HR5+OJxgFhGpKZQUqtH69eFZtm++GU4sn3detiMSEakYJYVqsnYtnHhiuFN5/Hj45S+zHZGISMUpKVSDVatg4ED48EOYMAGGDct2RCIilZPSfQpmtr+ZNY6G+5nZ5WbWKr2h1QwrV8Lxx8NHH4UuK5QQRKQmS/XmtWeB7WZ2APA3oAvwVNqiqiG++w7694f58+GFF2Dw4GxHJCJSNakmhR/cvRj4BXCvu18J7J2+sHLf0qXQrx8sXAgvvhh6OhURqelSPaewzcyGA+cCp0TjGqYnpNxXVBSeebBkCUyZAscem+2IRESqR6o1hfOBo4Bb3X2RmXUBxqcvrNy1eDEccwwsWwavvqqEICK1S0o1BXefD1wOYGatgRbufns6A8tFX3wRaghr14ab0444ItsRiYhUr1SvPvqnme1uZm2A2cDjZnZPekPLLYsWhRrChg3wxhtKCCJSO6XafNTS3dcCQ4DH3b0X8NP0hZV7br8dVq+GadPg8MOzHY2ISHqkmhQamNnewOnAi2mMJydt3gx//3vo5O7HP852NCIi6ZNqUrgJeAX4wt1nmNl+wIL0hZVbJk2CNWvgnHOyHYmISHqleqL5H8A/4t5/CQxNV1C5Ztw46NAh3KgmIlKbpXqiuYOZPWdm35nZMjN71sw6pDu4XLBsWbgXYe1aaNgQOneGgoJsRyUikh6pNh89DkwC9gHaA5OjcWUys4Fm9pmZLTSzMUmm/8XMZkWvz81sdUWCz4SrroIffghJwR2++io8I0GJQURqI3P38guZzXL3HuWNS5heH/gc+BlQBMwAhkf3PCQrfxlwuLuPLCuW/Px8LywsLDfm6tKoUXjWcqJOncKNbCIiNYGZzXT3/PLKpVpTWGFmZ5lZ/eh1FrCynHl6Awvd/Ut33wpMAMrqMm448HSK8WTEnDnJEwLA119nNhYRkUxINSmMJFyO+i2wFBhG6PqiLO2Bf8e9L4rG7cLMOhF6Xn2jlOmjzKzQzAqXL1+eYshVN25c6dM6dsxYGCIiGZNSUnD3r919kLu3c/c93P1Uwo1sZbFkiyql7JnAM+6+vZTPH+vu+e6e365du1RCrrLi4nDeoFcvaNp052lNm8Ktt2YkDBGRjEq1ppDMb8uZXgTsG/e+A7CklLJnkmNNR6+/Dt9+C9ddB2PHhnMIZuHv2LEwYkS2IxQRqX5VeRxnsppAvBnAgVGPqt8Qdvy7PLnYzA4CWgPvViGWavfEE9CmTXjucuPGSgIiUjdUpaZQ5mVL0UN5RhPuhP4EmOju88zsJjMbFFd0ODDBU7kMKkPWrIHnn4fhw0NCEBGpK8qsKZjZOpLv/A3YrbyFu/tLwEsJ4/6Y8P7GcqPMsGeeCf0dqVsLEalrykwK7t4iU4HkkieegIMOUvfYIlL3VKX5qFb68kt46y0499xwYllEpC5RUkjw5JMhGejEsojURUoKcdzDDWv9++vmNBGpm5QU4rzzTmg+OvfcbEciIpIdSgpxnngi3K08pLx7tUVEaiklhcimTTBxYnjkZvPm2Y5GRCQ7lBQikyeHm9bUdCQidZmSQuSJJ8IjN/v1y3YkIiLZo6RA6PjulVfg7LOhfv1sRyMikj1KCsDTT8P27SEpiIjUZUoKhKajI46AQw7JdiQiItlV55PC7NnhpRPMIiJKCjz5JDRsCGecke1IRESyr04nheJiGD8eTjoJ8vKyHY2ISPbV6aTw2muwbJmajkREStTppDBu3I5HboqISB1OCvGP3GzUKNvRiIjkhjqbFP7xj/DITTUdiYjsUGeTwrhxcPDBkJ+f7UhERHJHnUwKJY/cPOccPXJTRCRenUwKJY/cPOusbEciIpJb6lxSKHnk5nHHwb77ZjsaEZHcUueSwr/+FZqPzjkn25GIiOSeOpcUxo2DZs30yE0RkWTqVFLQIzdFRMpWp5LCpEnhpjU1HYmIJFenksK4ceHkcv/+2Y5ERCQ31ZmkUPLIzbPOgnp15luLiFRMndk9PvVUeOSmmo5ERErXINsBZMqJJ4Yb1g4+ONuRiIjkrjqTFA4+WAlBRKQ8aW0+MrOBZvaZmS00szGllDndzOab2Twzeyqd8YiISNnSVlMws/rAg8DPgCJghplNcvf5cWUOBK4BfuLuq8xsj3TFIyIi5UtnTaE3sNDdv3T3rcAEYHBCmV8BD7r7KgB3/y6N8YiISDnSmRTaA/+Oe18UjYv3I+BHZvYvM3vPzAYmW5CZjTKzQjMrXL58eZrCFRGRdCaFZE8q8IT3DYADgX7AcOBRM2u1y0zuY909393z27VrV+2BiohIkM6kUATEd07dAViSpMwL7r7N3RcBnxGShIiIZEE6k8IM4EAz62JmjYAzgUkJZZ4H+gOYWR6hOenLNMYkIiJlSFtScPdiYDTwCvAJMNHd55nZTWY2KCr2CrDSzOYD04DfufvKdMUkIiJlM/fEZv7clp+f74WFhdkOQ0SkRjGzme6eX165OtP3kYiIlE9JQUREYpQUREQkRklBRERilBRERCRGSUFERGKUFEREJEZJQUREYpQUREQkRklBRERilBRERCRGSUFERGKUFEREJEZJQUREYpQUREQkRklBRERilBRERCSmQbYDEJGaYdu2bRQVFbF58+ZshyJlaNKkCR06dKBhw4aVml9JQURSUlRURIsWLejcuTNmlu1wJAl3Z+XKlRQVFdGlS5dKLUPNRyKSks2bN9O2bVslhBxmZrRt27ZKtTklBRFJmRJC7qvqb6SkICIiMUoKIpIWBQXQuTPUqxf+FhRUbXkrV66kR48e9OjRg7322ov27dvH3m/dujWlZZx//vl89tlnZZZ58MEHKahqsDWYTjSLSLUrKIBRo2DjxvD+q6/Ce4ARIyq3zLZt2zJr1iwAbrzxRpo3b87VV1+9Uxl3x92pVy/58e7jjz9e7udceumllQuwllBNQUSq3XXX7UgIJTZuDOOr28KFC+natSsXX3wxPXv2ZOnSpYwaNYr8/HwOO+wwbrrppljZvn37MmvWLIqLi2nVqhVjxoyhe/fuHHXUUXz33XcAXH/99dx7772x8mPGjKF3794cdNBBvPPOOwBs2LCBoUOH0r17d4YPH05+fn4sYcW74YYbOOKII2LxuTsAn3/+Occddxzdu3enZ8+eLF68GIA///nP/PjHP6Z79+5cl46VlQIlBRGpdl9/XbHxVTV//nwuuOACPvroI9q3b8/tt99OYWEhs2fP5rXXXmP+/Pm7zLNmzRqOPfZYZs+ezVFHHcVjjz2WdNnuzgcffMDO+xaYAAAPhUlEQVSdd94ZSzB//etf2WuvvZg9ezZjxozho48+Sjrvb37zG2bMmMHcuXNZs2YNL7/8MgDDhw/nyiuvZPbs2bzzzjvsscceTJ48mSlTpvDBBx8we/ZsrrrqqmpaOxWjpCAi1a5jx4qNr6r999+fI444Ivb+6aefpmfPnvTs2ZNPPvkkaVLYbbfdOOGEEwDo1atX7Gg90ZAhQ3Yp8/bbb3PmmWcC0L17dw477LCk806dOpXevXvTvXt33nzzTebNm8eqVatYsWIFp5xyChBuNmvatCmvv/46I0eOZLfddgOgTZs2FV8R1UBJQUSq3a23QtOmO49r2jSMT4dmzZrFhhcsWMB9993HG2+8wZw5cxg4cGDS6/YbNWoUG65fvz7FxcVJl924ceNdypQ0A5Vl48aNjB49mueee445c+YwcuTIWBzJLht195y45FdJQUSq3YgRMHYsdOoEZuHv2LGVP8lcEWvXrqVFixbsvvvuLF26lFdeeaXaP6Nv375MnDgRgLlz5yatiWzatIl69eqRl5fHunXrePbZZwFo3bo1eXl5TJ48GQg3BW7cuJEBAwbwt7/9jU2bNgHw/fffV3vcqdDVRyKSFiNGZCYJJOrZsyeHHnooXbt2Zb/99uMnP/lJtX/GZZddxjnnnEO3bt3o2bMnXbt2pWXLljuVadu2Leeeey5du3alU6dO9OnTJzatoKCAiy66iOuuu45GjRrx7LPPcvLJJzN79mzy8/Np2LAhp5xyCjfffHO1x14eS6UalEvy8/O9sLAw22GI1DmffPIJhxxySLbDyAnFxcUUFxfTpEkTFixYwIABA1iwYAENGuTGcXay38rMZrp7fnnz5sY3EBGpQdavX8/xxx9PcXEx7s7DDz+cMwmhqtL6LcxsIHAfUB941N1vT5h+HnAn8E006gF3fzSdMYmIVFWrVq2YOXNmtsNIi7QlBTOrDzwI/AwoAmaY2SR3Tzwj83d3H52uOEREJHXpvPqoN7DQ3b90963ABGBwGj9PRESqKJ1JoT3w77j3RdG4REPNbI6ZPWNm+yZbkJmNMrNCMytcvnx5OmIVERHSmxSS3YWReKnTZKCzu3cDXgeeSLYgdx/r7vnunt+uXbtqDlNEREqkMykUAfFH/h2AJfEF3H2lu2+J3j4C9EpjPCJSg/Xr12+XG9Huvfdefv3rX5c5X/PmzQFYsmQJw4YNK3XZ5V3qfu+997Ixrpe/E088kdWrV6cSeo2SzqQwAzjQzLqYWSPgTGBSfAEz2zvu7SDgkzTGIyI12PDhw5kwYcJO4yZMmMDw4cNTmn+fffbhmWeeqfTnJyaFl156iVatWlV6ebkqbVcfuXuxmY0GXiFckvqYu88zs5uAQnefBFxuZoOAYuB74Lx0xSMi1eeKKyBJT9FV0qMHRD1WJzVs2DCuv/56tmzZQuPGjVm8eDFLliyhb9++rF+/nsGDB7Nq1Sq2bdvGLbfcwuDBO1/XsnjxYk4++WQ+/vhjNm3axPnnn8/8+fM55JBDYl1LAFxyySXMmDGDTZs2MWzYMP70pz9x//33s2TJEvr3709eXh7Tpk2jc+fOFBYWkpeXxz333BPrZfXCCy/kiiuuYPHixZxwwgn07duXd955h/bt2/PCCy/EOrwrMXnyZG655Ra2bt1K27ZtKSgoYM8992T9+vVcdtllFBYWYmbccMMNDB06lJdffplrr72W7du3k5eXx9SpU6vvRyDN9ym4+0vASwnj/hg3fA1wTTpjEJHaoW3btvTu3ZuXX36ZwYMHM2HCBM444wzMjCZNmvDcc8+x++67s2LFCo488kgGDRpUagdzDz30EE2bNmXOnDnMmTOHnj17xqbdeuuttGnThu3bt3P88cczZ84cLr/8cu655x6mTZtGXl7eTsuaOXMmjz/+OO+//z7uTp8+fTj22GNp3bo1CxYs4Omnn+aRRx7h9NNP59lnn+Wss87aaf6+ffvy3nvvYWY8+uij3HHHHdx9993cfPPNtGzZkrlz5wKwatUqli9fzq9+9SumT59Oly5d0tI/Uu24BU9EMqqsI/p0KmlCKkkKJUfn7s61117L9OnTqVevHt988w3Lli1jr732Srqc6dOnc/nllwPQrVs3unXrFps2ceJExo4dS3FxMUuXLmX+/Pk7TU/09ttv84tf/CLWU+uQIUN46623GDRoEF26dKFHjx5A6d1zFxUVccYZZ7B06VK2bt1Kly5dAHj99dd3ai5r3bo1kydP5phjjomVSUf32nWil9TqflasiGTHqaeeytSpU/nwww/ZtGlT7Ai/oKCA5cuXM3PmTGbNmsWee+6ZtLvseMlqEYsWLeKuu+5i6tSpzJkzh5NOOqnc5ZTVf1xJt9tQevfcl112GaNHj2bu3Lk8/PDDsc9L1pV2JrrXrvVJoeRZsV99Be47nhWrxCBS8zRv3px+/foxcuTInU4wr1mzhj322IOGDRsybdo0vvrqqzKXc8wxx1AQ7QQ+/vhj5syZA4Rut5s1a0bLli1ZtmwZU6ZMic3TokUL1q1bl3RZzz//PBs3bmTDhg0899xzHH300Sl/pzVr1tC+fbiF64kndlyVP2DAAB544IHY+1WrVnHUUUfx5ptvsmjRIiA93WvX+qSQyWfFikj6DR8+nNmzZ8eefAYwYsQICgsLyc/Pp6CggIMPPrjMZVxyySWsX7+ebt26cccdd9C7d28gPEXt8MMP57DDDmPkyJE7dbs9atQoTjjhBPr377/Tsnr27Ml5551H79696dOnDxdeeCGHH354yt/nxhtv5LTTTuPoo4/e6XzF9ddfz6pVq+jatSvdu3dn2rRptGvXjrFjxzJkyBC6d+/OGWeckfLnpKrWd51dr16oISQygx9+qMbARGo5dZ1dc1Sl6+xaX1PI9LNiRURqslqfFDL9rFgRkZqs1ieFbD4rVqS2qWnNzXVRVX+jOnGfQraeFStSmzRp0oSVK1fStm3btF8WKZXj7qxcuZImTZpUehl1IimISNV16NCBoqIi1H19bmvSpAkdOnSo9PxKCiKSkoYNG8bupJXaq9afUxARkdQpKYiISIySgoiIxNS4O5rNbDlQdscm2ZMHrMh2EGVQfFWT6/FB7seo+KqmKvF1cvdyn2dc45JCLjOzwlRuI88WxVc1uR4f5H6Miq9qMhGfmo9ERCRGSUFERGKUFKrX2GwHUA7FVzW5Hh/kfoyKr2rSHp/OKYiISIxqCiIiEqOkICIiMUoKFWRm+5rZNDP7xMzmmdlvkpTpZ2ZrzGxW9PpjhmNcbGZzo8/e5TF1FtxvZgvNbI6Z9cxgbAfFrZdZZrbWzK5IKJPx9Wdmj5nZd2b2cdy4Nmb2mpktiP62LmXec6MyC8zs3AzFdqeZfRr9fs+ZWatS5i1zW0hzjDea2Tdxv+OJpcw70Mw+i7bHMRmM7+9xsS02s1mlzJvWdVjaPiVr25+761WBF7A30DMabgF8DhyaUKYf8GIWY1wM5JUx/URgCmDAkcD7WYqzPvAt4aaarK4/4BigJ/Bx3Lg7gDHR8Bjgv5LM1wb4MvrbOhpunYHYBgANouH/ShZbKttCmmO8Ebg6hW3gC2A/oBEwO/H/KV3xJUy/G/hjNtZhafuUbG1/qilUkLsvdfcPo+F1wCdA++xGVWGDgXEevAe0MrO9sxDH8cAX7p71O9TdfTrwfcLowcAT0fATwKlJZv058Jq7f+/uq4DXgIHpjs3dX3X34ujte0Dl+0quBqWsv1T0Bha6+5fuvhWYQFjv1aqs+Cw8HOJ04Onq/txUlLFPycr2p6RQBWbWGTgceD/J5KPMbLaZTTGzwzIaGDjwqpnNNLNRSaa3B/4d976I7CS2Myn9HzGb66/Enu6+FMI/LrBHkjK5sC5HEmp+yZS3LaTb6KiJ67FSmj9yYf0dDSxz9wWlTM/YOkzYp2Rl+1NSqCQzaw48C1zh7msTJn9IaBLpDvwVeD7D4f3E3XsCJwCXmtkxCdOTPTYro9cmm1kjYBDwjySTs73+KiKr69LMrgOKgYJSipS3LaTTQ8D+QA9gKaGJJlHWt0VgOGXXEjKyDsvZp5Q6W5JxVVp/SgqVYGYNCT9egbv/b+J0d1/r7uuj4ZeAhmaWl6n43H1J9Pc74DlCFT1eEbBv3PsOwJLMRBdzAvChuy9LnJDt9RdnWUmzWvT3uyRlsrYuo5OKJwMjPGpgTpTCtpA27r7M3be7+w/AI6V8dla3RTNrAAwB/l5amUysw1L2KVnZ/pQUKihqf/wb8Im731NKmb2icphZb8J6Xpmh+JqZWYuSYcIJyY8Tik0CzomuQjoSWFNSTc2gUo/Osrn+EkwCSq7mOBd4IUmZV4ABZtY6ah4ZEI1LKzMbCPweGOTuG0spk8q2kM4Y489T/aKUz54BHGhmXaLa45mE9Z4pPwU+dfeiZBMzsQ7L2KdkZ/tL1xn12voC+hKqZ3OAWdHrROBi4OKozGhgHuFKiveA/8hgfPtFnzs7iuG6aHx8fAY8SLjqYy6Qn+F12JSwk28ZNy6r64+QoJYC2whHXxcAbYGpwILob5uobD7waNy8I4GF0ev8DMW2kNCWXLIN/ndUdh/gpbK2hQyuvyej7WsOYQe3d2KM0fsTCVfcfJGuGJPFF43/n5LtLq5sRtdhGfuUrGx/6uZCRERi1HwkIiIxSgoiIhKjpCAiIjFKCiIiEqOkICIiMUoKIhEz22479+BabT12mlnn+B46RXJVg2wHIJJDNrl7j2wHIZJNqimIlCPqT/+/zOyD6HVANL6TmU2NOnybamYdo/F7WnjGwezo9R/Rouqb2SNRn/mvmtluUfnLzWx+tJwJWfqaIoCSgki83RKaj86Im7bW3XsDDwD3RuMeIHRB3o3QId390fj7gTc9dOjXk3AnLMCBwIPufhiwGhgajR8DHB4t5+J0fTmRVOiOZpGIma139+ZJxi8GjnP3L6OOy75197ZmtoLQdcO2aPxSd88zs+VAB3ffEreMzoR+7w+M3v8eaOjut5jZy8B6Qm+wz3vUGaBINqimIJIaL2W4tDLJbIkb3s6Oc3onEfqi6gXMjHruFMkKJQWR1JwR9/fdaPgdQq+eACOAt6PhqcAlAGZW38x2L22hZlYP2NfdpwH/CbQCdqmtiGSKjkhEdtjNdn54+8vuXnJZamMze59wIDU8Gnc58JiZ/Q5YDpwfjf8NMNbMLiDUCC4h9NCZTH1gvJm1JPRe+xd3X11t30ikgnROQaQc0TmFfHdfke1YRNJNzUciIhKjmoKIiMSopiAiIjFKCiIiEqOkICIiMUoKIiISo6QgIiIx/x/n59jII+XEDQAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BAmSHjZoZYbt"
      },
      "source": [
        "- 여기에서는 Training과 validation accuracy를 그려주었다."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QLcecojXB9eC"
      },
      "source": [
        "It seems that the network starts overfitting after 8 epochs. Let's train a new network from scratch for 8 epochs, then let's evaluate it on \n",
        "the test set:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8RxOlDC7B9eC",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a465a175-661b-4521-9b8d-a3645548b890"
      },
      "source": [
        "model = models.Sequential()\n",
        "model.add(layers.Dense(64, activation='relu', input_shape=(10000,)))\n",
        "model.add(layers.Dense(64, activation='relu'))\n",
        "model.add(layers.Dense(46, activation='softmax'))\n",
        "\n",
        "model.compile(optimizer='rmsprop',\n",
        "              loss='categorical_crossentropy',\n",
        "              metrics=['accuracy'])\n",
        "model.fit(partial_x_train,\n",
        "          partial_y_train,\n",
        "          epochs=8,\n",
        "          batch_size=512,\n",
        "          validation_data=(x_val, y_val))\n",
        "results = model.evaluate(x_test, one_hot_test_labels)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/8\n",
            "16/16 [==============================] - 1s 56ms/step - loss: 3.0732 - accuracy: 0.4234 - val_loss: 1.6363 - val_accuracy: 0.6480\n",
            "Epoch 2/8\n",
            "16/16 [==============================] - 1s 47ms/step - loss: 1.4050 - accuracy: 0.7055 - val_loss: 1.2684 - val_accuracy: 0.7260\n",
            "Epoch 3/8\n",
            "16/16 [==============================] - 1s 46ms/step - loss: 1.0352 - accuracy: 0.7729 - val_loss: 1.1238 - val_accuracy: 0.7530\n",
            "Epoch 4/8\n",
            "16/16 [==============================] - 1s 47ms/step - loss: 0.7882 - accuracy: 0.8379 - val_loss: 1.0324 - val_accuracy: 0.7790\n",
            "Epoch 5/8\n",
            "16/16 [==============================] - 1s 46ms/step - loss: 0.6390 - accuracy: 0.8668 - val_loss: 0.9729 - val_accuracy: 0.7910\n",
            "Epoch 6/8\n",
            "16/16 [==============================] - 1s 47ms/step - loss: 0.4935 - accuracy: 0.8985 - val_loss: 0.9274 - val_accuracy: 0.8070\n",
            "Epoch 7/8\n",
            "16/16 [==============================] - 1s 46ms/step - loss: 0.4073 - accuracy: 0.9168 - val_loss: 0.9153 - val_accuracy: 0.8050\n",
            "Epoch 8/8\n",
            "16/16 [==============================] - 1s 46ms/step - loss: 0.3225 - accuracy: 0.9310 - val_loss: 0.9178 - val_accuracy: 0.8040\n",
            "71/71 [==============================] - 0s 3ms/step - loss: 1.0027 - accuracy: 0.7805\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0UrXWkRmZjAD"
      },
      "source": [
        "- 여기에서는 epochs를 8번 사용해서 구해주었다. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JYKgPWQZB9eC",
        "outputId": "0444aefb-f6b2-425d-89a0-457762564001"
      },
      "source": [
        "results"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[0.9629409742058119, 0.7885129]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HBNbcwSIZpf9"
      },
      "source": [
        "- 결과는 78%정도 나온것을 볼 수 있다. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JhpjxnEgB9eC"
      },
      "source": [
        "\n",
        "Our approach reaches an accuracy of ~78%. With a balanced binary classification problem, the accuracy reached by a purely random classifier \n",
        "would be 50%, but in our case it is closer to 19%, so our results seem pretty good, at least when compared to a random baseline:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "P0uyjI7WB9eD",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9c2af92b-2f2e-452d-9872-41e18d0a3554"
      },
      "source": [
        "import copy\n",
        "\n",
        "test_labels_copy = copy.copy(test_labels)\n",
        "np.random.shuffle(test_labels_copy)\n",
        "float(np.sum(np.array(test_labels) == np.array(test_labels_copy))) / len(test_labels)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.18210151380231523"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bv6zcaAOB9eD"
      },
      "source": [
        "## Generating predictions on new data\n",
        "\n",
        "We can verify that the `predict` method of our model instance returns a probability distribution over all 46 topics. Let's generate topic \n",
        "predictions for all of the test data:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N_wEz-sqZxwW"
      },
      "source": [
        "- 여기에서는 새로운것을 predictions하는 것이 나와 있다. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "liruj5k4B9eD"
      },
      "source": [
        "predictions = model.predict(x_test)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jIlzv9tiZ5dm",
        "outputId": "8ed30d24-7fa5-4c41-9d36-fdb82ec7c7e7"
      },
      "source": [
        "predictions[:5]  #아래보이는 것처럼 46개씩 나오게 되며, 거기에서 제일 높은 값이 predictions되는 값이다."
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[3.07086120e-05, 3.31066985e-04, 2.90786829e-05, 8.38530242e-01,\n",
              "        1.45413607e-01, 4.48656538e-05, 9.36699962e-06, 4.45744081e-05,\n",
              "        2.67458451e-03, 1.57969727e-04, 4.63383403e-05, 8.45032337e-04,\n",
              "        5.53703401e-04, 3.82761209e-04, 7.77720288e-06, 8.53238307e-05,\n",
              "        5.60246350e-04, 7.74193832e-05, 4.58032173e-06, 9.06090252e-04,\n",
              "        5.77513035e-03, 1.80793591e-04, 1.21367608e-04, 3.68963840e-04,\n",
              "        9.46226100e-06, 7.59524410e-05, 2.78104417e-05, 2.41364760e-05,\n",
              "        1.51079648e-05, 2.68188422e-04, 6.09311392e-04, 1.03616367e-04,\n",
              "        4.56750195e-06, 4.17206356e-05, 3.76470853e-05, 1.81813466e-05,\n",
              "        5.01699396e-04, 3.28038072e-06, 2.80787772e-05, 7.06108462e-04,\n",
              "        1.55706493e-05, 1.75470195e-05, 1.53679375e-05, 2.83323810e-04,\n",
              "        3.27402972e-06, 8.48169748e-06],\n",
              "       [2.00526416e-02, 4.18223917e-01, 1.39713287e-01, 7.50251493e-05,\n",
              "        1.70211634e-03, 7.60607596e-04, 1.04532787e-03, 2.67872419e-05,\n",
              "        8.17682958e-05, 2.87714927e-03, 3.39429677e-01, 2.66664370e-04,\n",
              "        1.46572432e-03, 9.69102792e-03, 6.48196600e-03, 5.00094437e-04,\n",
              "        2.39921757e-03, 2.47092365e-04, 1.03122846e-03, 1.04989973e-04,\n",
              "        3.85822525e-04, 1.46925240e-03, 1.52981898e-04, 1.41575641e-03,\n",
              "        2.22526118e-03, 3.58198828e-04, 8.68697462e-05, 4.83525306e-04,\n",
              "        1.79097205e-02, 2.68038770e-04, 1.65934107e-05, 5.01086516e-03,\n",
              "        6.82038721e-03, 7.11133737e-07, 3.51233804e-03, 1.90079008e-05,\n",
              "        9.80616664e-04, 2.60892487e-03, 3.75768659e-03, 2.11761100e-03,\n",
              "        5.21254726e-04, 3.38758761e-03, 4.29836327e-05, 2.55551655e-04,\n",
              "        4.66685924e-06, 1.15121084e-05],\n",
              "       [9.27731209e-03, 7.31362700e-01, 5.60576376e-03, 8.34049133e-04,\n",
              "        1.83726859e-03, 7.58086983e-03, 5.95864560e-03, 1.70678919e-04,\n",
              "        1.42546464e-03, 1.32086931e-03, 5.82606765e-03, 5.49187243e-04,\n",
              "        2.12882319e-03, 3.80275175e-02, 3.52787203e-03, 1.36132399e-03,\n",
              "        2.28462256e-02, 2.25912165e-02, 7.02262158e-04, 2.09455888e-04,\n",
              "        7.59399962e-03, 3.72826890e-03, 7.83380703e-04, 9.67269484e-03,\n",
              "        2.83595780e-03, 3.85750202e-04, 3.49353417e-04, 2.41026958e-03,\n",
              "        3.18268873e-02, 3.17664002e-03, 2.59974564e-04, 3.56145650e-02,\n",
              "        3.90246860e-03, 2.59091088e-04, 9.19264928e-03, 1.22507292e-04,\n",
              "        3.66997556e-03, 3.64130805e-03, 4.66665626e-03, 2.82161380e-03,\n",
              "        1.06240204e-03, 5.56326378e-03, 6.78009645e-04, 1.92058750e-03,\n",
              "        2.64367409e-04, 4.53813147e-04],\n",
              "       [3.20777239e-04, 1.15133030e-03, 3.74434603e-05, 1.59536853e-01,\n",
              "        6.42575264e-01, 9.05161360e-05, 1.15429761e-03, 2.50506164e-05,\n",
              "        1.13539933e-03, 4.06933046e-04, 4.82362011e-05, 2.22345334e-04,\n",
              "        7.99158006e-05, 3.80099937e-03, 2.70878772e-05, 1.60814641e-04,\n",
              "        1.67322561e-01, 5.77874761e-03, 4.96063922e-06, 8.44350783e-04,\n",
              "        6.11158507e-03, 5.92923432e-04, 1.01440295e-04, 1.37024932e-03,\n",
              "        4.79561786e-05, 8.49499775e-04, 8.14864034e-05, 3.86104599e-04,\n",
              "        4.00449317e-05, 2.16874207e-04, 3.57090066e-05, 6.91506080e-04,\n",
              "        2.72555062e-05, 2.50930688e-03, 5.19717105e-05, 1.79063154e-05,\n",
              "        1.95081651e-04, 1.03863509e-04, 5.10492355e-05, 8.61983455e-04,\n",
              "        1.00122983e-04, 3.82578903e-04, 8.84762776e-05, 2.45854171e-04,\n",
              "        6.08760138e-06, 1.09268723e-04],\n",
              "       [1.21948266e-04, 1.10386615e-03, 3.92951843e-05, 1.29355132e-04,\n",
              "        8.72491291e-05, 1.84318051e-04, 1.28906497e-04, 5.39482153e-06,\n",
              "        3.69189860e-04, 5.94578887e-05, 1.37380336e-03, 2.72960642e-06,\n",
              "        8.05961554e-06, 9.73410547e-01, 1.93576008e-04, 8.43585804e-06,\n",
              "        1.72178932e-02, 4.98807931e-04, 2.49672721e-05, 1.39519470e-04,\n",
              "        7.53864064e-04, 3.91442627e-05, 1.23288301e-05, 3.58778809e-04,\n",
              "        7.37184246e-06, 6.84385523e-05, 2.55298310e-05, 3.41139821e-05,\n",
              "        1.90605133e-05, 9.24113465e-06, 1.30541430e-05, 7.38311384e-04,\n",
              "        2.81693850e-04, 4.45380647e-05, 1.40800141e-03, 1.06362677e-05,\n",
              "        1.34419251e-05, 1.71042866e-05, 2.59619064e-05, 4.71736530e-05,\n",
              "        7.88079051e-04, 1.12841582e-04, 1.30052094e-05, 2.98865871e-05,\n",
              "        2.57634298e-07, 2.08805432e-05]], dtype=float32)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 27
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QSKCmC7lB9eD"
      },
      "source": [
        "Each entry in `predictions` is a vector of length 46:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OpQRXKX3B9eD",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b530bad9-f9b7-486b-e625-882103f0d5fe"
      },
      "source": [
        "predictions[0].shape"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(46,)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 28
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z7lB7ThoB9eD"
      },
      "source": [
        "The coefficients in this vector sum to 1:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5WYYEtCcB9eE",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b6e23a81-e231-44ce-f631-4fb3c22fd8da"
      },
      "source": [
        "np.sum(predictions[0])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.99999994"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 32
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O84qFwXDB9eE"
      },
      "source": [
        "The largest entry is the predicted class, i.e. the class with the highest probability:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zM97Jx8MB9eE",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "828ea8ce-0afa-4bf5-b89a-2e1825d9a689"
      },
      "source": [
        "np.argmax(predictions[0])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "3"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 29
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DA--J6sVaRk5"
      },
      "source": [
        "- np.argmax이걸 사용해서 그 값의 최대가 되는 위치를 표현해준다."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GNR1u63LB9eE"
      },
      "source": [
        "## A different way to handle the labels and the loss\n",
        "\n",
        "We mentioned earlier that another way to encode the labels would be to cast them as an integer tensor, like such:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hxxSP_DOB9eE"
      },
      "source": [
        "y_train = np.array(train_labels)\n",
        "y_test = np.array(test_labels)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HM0-Cbo7ac6p"
      },
      "source": [
        "- 만약에 labels을 위와 같은 코드로 정의하게 되었으면, 컴파일 할 때 loss='sparse_categorical_crossentropy'이것을 아래 코드에 쳐줘야 더미변수를 인코딩 안해도(one_hot 인코딩)을 안해도 one_hot 인코딩 한 것 처럼 쓸 수 있다.  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YoOl2wtLB9eE"
      },
      "source": [
        "\n",
        "The only thing it would change is the choice of the loss function. Our previous loss, `categorical_crossentropy`, expects the labels to \n",
        "follow a categorical encoding. With integer labels, we should use `sparse_categorical_crossentropy`:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZH7aDc8NB9eE"
      },
      "source": [
        "model.compile(optimizer='rmsprop', loss='sparse_categorical_crossentropy', metrics=['acc'])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SjJ1FMq4B9eF"
      },
      "source": [
        "This new loss function is still mathematically the same as `categorical_crossentropy`; it just has a different interface."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ff4StY6lB9eF"
      },
      "source": [
        "## On the importance of having sufficiently large intermediate layers\n",
        "\n",
        "\n",
        "We mentioned earlier that since our final outputs were 46-dimensional, we should avoid intermediate layers with much less than 46 hidden \n",
        "units. Now let's try to see what happens when we introduce an information bottleneck by having intermediate layers significantly less than \n",
        "46-dimensional, e.g. 4-dimensional."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZHUVxUaOB9eF",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0a6f32d3-665b-4064-e01a-127de6ea383f"
      },
      "source": [
        "model = models.Sequential()\n",
        "model.add(layers.Dense(64, activation='relu', input_shape=(10000,)))\n",
        "model.add(layers.Dense(4, activation='relu'))    #여기는 중간에 4로 조금 두었다가 46으로 두게 되면, 정보의 손실이 발생하여 성능이 떨어진다는 얘기이다.\n",
        "model.add(layers.Dense(46, activation='softmax'))\n",
        "\n",
        "model.compile(optimizer='rmsprop',\n",
        "              loss='categorical_crossentropy',\n",
        "              metrics=['accuracy'])\n",
        "model.fit(partial_x_train,\n",
        "          partial_y_train,\n",
        "          epochs=20,\n",
        "          batch_size=128,\n",
        "          validation_data=(x_val, y_val))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/20\n",
            "63/63 [==============================] - 2s 19ms/step - loss: 3.2721 - accuracy: 0.2410 - val_loss: 1.9532 - val_accuracy: 0.5450\n",
            "Epoch 2/20\n",
            "63/63 [==============================] - 1s 16ms/step - loss: 1.7267 - accuracy: 0.5451 - val_loss: 1.5676 - val_accuracy: 0.5570\n",
            "Epoch 3/20\n",
            "63/63 [==============================] - 1s 17ms/step - loss: 1.3759 - accuracy: 0.5791 - val_loss: 1.4183 - val_accuracy: 0.6060\n",
            "Epoch 4/20\n",
            "63/63 [==============================] - 1s 16ms/step - loss: 1.1792 - accuracy: 0.6441 - val_loss: 1.3339 - val_accuracy: 0.6630\n",
            "Epoch 5/20\n",
            "63/63 [==============================] - 1s 16ms/step - loss: 1.0569 - accuracy: 0.7011 - val_loss: 1.3011 - val_accuracy: 0.7020\n",
            "Epoch 6/20\n",
            "63/63 [==============================] - 1s 16ms/step - loss: 0.8934 - accuracy: 0.7729 - val_loss: 1.3084 - val_accuracy: 0.7090\n",
            "Epoch 7/20\n",
            "63/63 [==============================] - 1s 17ms/step - loss: 0.8574 - accuracy: 0.7867 - val_loss: 1.2717 - val_accuracy: 0.7250\n",
            "Epoch 8/20\n",
            "63/63 [==============================] - 1s 17ms/step - loss: 0.7396 - accuracy: 0.8077 - val_loss: 1.2745 - val_accuracy: 0.7290\n",
            "Epoch 9/20\n",
            "63/63 [==============================] - 1s 17ms/step - loss: 0.7221 - accuracy: 0.8154 - val_loss: 1.3247 - val_accuracy: 0.7180\n",
            "Epoch 10/20\n",
            "63/63 [==============================] - 1s 17ms/step - loss: 0.6404 - accuracy: 0.8246 - val_loss: 1.3483 - val_accuracy: 0.7180\n",
            "Epoch 11/20\n",
            "63/63 [==============================] - 1s 17ms/step - loss: 0.5877 - accuracy: 0.8360 - val_loss: 1.3824 - val_accuracy: 0.7140\n",
            "Epoch 12/20\n",
            "63/63 [==============================] - 1s 16ms/step - loss: 0.5689 - accuracy: 0.8379 - val_loss: 1.4146 - val_accuracy: 0.7180\n",
            "Epoch 13/20\n",
            "63/63 [==============================] - 1s 17ms/step - loss: 0.5150 - accuracy: 0.8569 - val_loss: 1.4550 - val_accuracy: 0.7190\n",
            "Epoch 14/20\n",
            "63/63 [==============================] - 1s 17ms/step - loss: 0.4933 - accuracy: 0.8579 - val_loss: 1.4861 - val_accuracy: 0.7240\n",
            "Epoch 15/20\n",
            "63/63 [==============================] - 1s 16ms/step - loss: 0.4632 - accuracy: 0.8706 - val_loss: 1.5800 - val_accuracy: 0.7120\n",
            "Epoch 16/20\n",
            "63/63 [==============================] - 1s 16ms/step - loss: 0.4482 - accuracy: 0.8726 - val_loss: 1.6111 - val_accuracy: 0.7230\n",
            "Epoch 17/20\n",
            "63/63 [==============================] - 1s 17ms/step - loss: 0.3950 - accuracy: 0.8907 - val_loss: 1.6593 - val_accuracy: 0.7190\n",
            "Epoch 18/20\n",
            "63/63 [==============================] - 1s 17ms/step - loss: 0.3970 - accuracy: 0.8981 - val_loss: 1.6529 - val_accuracy: 0.7290\n",
            "Epoch 19/20\n",
            "63/63 [==============================] - 1s 17ms/step - loss: 0.3848 - accuracy: 0.8957 - val_loss: 1.7373 - val_accuracy: 0.7230\n",
            "Epoch 20/20\n",
            "63/63 [==============================] - 1s 17ms/step - loss: 0.3770 - accuracy: 0.8964 - val_loss: 1.7812 - val_accuracy: 0.7090\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tensorflow.python.keras.callbacks.History at 0x7f7e7ce70d10>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 33
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p58WIK3GbHyb"
      },
      "source": [
        "- 여기는 중간에 4로 조금 두었다가 46으로 두게 되면, 정보의 손실이 발생하여 성능이 떨어진다는 얘기이다.\n",
        "- 그래서 여기는 성능이 8%정도 떨어졌다고 나오게 된다. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2VHBh2fGB9eF"
      },
      "source": [
        "\n",
        "Our network now seems to peak at ~71% test accuracy, a 8% absolute drop. This drop is mostly due to the fact that we are now trying to \n",
        "compress a lot of information (enough information to recover the separation hyperplanes of 46 classes) into an intermediate space that is \n",
        "too low-dimensional. The network is able to cram _most_ of the necessary information into these 8-dimensional representations, but not all \n",
        "of it."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ts9SWFxLB9eF"
      },
      "source": [
        "## Further experiments\n",
        "\n",
        "* Try using larger or smaller layers: 32 units, 128 units...\n",
        "* We were using two hidden layers. Now try to use a single hidden layer, or three hidden layers."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Dr8QNPXxB9eF"
      },
      "source": [
        "## Wrapping up\n",
        "\n",
        "\n",
        "Here's what you should take away from this example:\n",
        "\n",
        "* If you are trying to classify data points between N classes, your network should end with a `Dense` layer of size N.\n",
        "* In a single-label, multi-class classification problem, your network should end with a `softmax` activation, so that it will output a \n",
        "probability distribution over the N output classes.\n",
        "* _Categorical crossentropy_ is almost always the loss function you should use for such problems. It minimizes the distance between the \n",
        "probability distributions output by the network, and the true distribution of the targets.\n",
        "* There are two ways to handle labels in multi-class classification:\n",
        "    ** Encoding the labels via \"categorical encoding\" (also known as \"one-hot encoding\") and using `categorical_crossentropy` as your loss \n",
        "function.\n",
        "    ** Encoding the labels as integers and using the `sparse_categorical_crossentropy` loss function.\n",
        "* If you need to classify data into a large number of categories, then you should avoid creating information bottlenecks in your network by having \n",
        "intermediate layers that are too small."
      ]
    }
  ]
}