ALBERT_Mongolian_text_classification.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "ALBERT_Mongolian_text_classification.ipynb",
      "provenance": [],
      "collapsed_sections": [
        "caMvdHeDJGOx"
      ],
      "toc_visible": true,
      "authorship_tag": "ABX9TyP+wzP47RaMgq0+kEp17Vpk",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "TPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/bayartsogt-ya/albert-mongolian/blob/master/ALBERT_Mongolian_text_classification.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D7GgMGQ_b3HU",
        "colab_type": "text"
      },
      "source": [
        "# Text Classification using ALBERT-Mongolian\n",
        "\n",
        "Сайн байна уу?\n",
        "\n",
        "Энэхүү notebook нь [`ALBERT-Mongolian`](https://huggingface.co/bayartsogt/albert-mongolian) ашиглан хэрхэн `TF Keras` дээр текст ангилал хийхийг харуулах болно.\n",
        "\n",
        "Notebook-ийн төгсгөлд та eduge.mn сайтын мэдээний өгөгдлийг **~90%** таах чадвартай модельтэй болох ба энэхүү үр дүн нь ердөө эхлэл ба цааш сайжрах бүрэн боломжтой юм.\n",
        "\n",
        "Notebook-ийн дараалал:\n",
        "\n",
        "1. [Бэлтгэл ажил](#scrollTo=caMvdHeDJGOx)\n",
        "2. [Өгөгдөл](#scrollTo=lUXesMU1c5Sq)\n",
        "3. [Tokenize хийх](#scrollTo=PJsBXhkvc7L5)\n",
        "4. [Модель ба сургалт](#scrollTo=1LEnvmjDdNlZ)\n",
        "5. [Үр дүн](#scrollTo=FmJPTVHI4E6g)\n",
        "\n",
        "```\n",
        "used lang: python 3.6\n",
        "TF version: 2.1\n",
        "author: Bayartsogt Yadamsuren\n",
        "```\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "caMvdHeDJGOx",
        "colab_type": "text"
      },
      "source": [
        "## Бэлтгэл ажил"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vS3d81vOdX5F",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "673c2933-fcb4-45ab-b190-0406b02670af"
      },
      "source": [
        "! pip install -q transformers\n",
        "! pip install -q tensorflow==2.1.0 # 2.2 дээр тест хийгээгүй болно!"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\u001b[K     |████████████████████████████████| 778kB 3.3MB/s \n",
            "\u001b[K     |████████████████████████████████| 3.0MB 11.6MB/s \n",
            "\u001b[K     |████████████████████████████████| 890kB 17.4MB/s \n",
            "\u001b[K     |████████████████████████████████| 1.1MB 19.3MB/s \n",
            "\u001b[?25h  Building wheel for sacremoses (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting tensorflow==2.1.0\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/85/d4/c0cd1057b331bc38b65478302114194bd8e1b9c2bbc06e300935c0e93d90/tensorflow-2.1.0-cp36-cp36m-manylinux2010_x86_64.whl (421.8MB)\n",
            "\u001b[K     |████████████████████████████████| 421.8MB 38kB/s \n",
            "\u001b[?25hRequirement already satisfied: numpy<2.0,>=1.16.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.1.0) (1.18.5)\n",
            "Requirement already satisfied: wheel>=0.26; python_version >= \"3\" in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.1.0) (0.34.2)\n",
            "Requirement already satisfied: six>=1.12.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.1.0) (1.15.0)\n",
            "Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.1.0) (3.3.0)\n",
            "Requirement already satisfied: protobuf>=3.8.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.1.0) (3.12.2)\n",
            "Requirement already satisfied: absl-py>=0.7.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.1.0) (0.9.0)\n",
            "Requirement already satisfied: keras-preprocessing>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.1.0) (1.1.2)\n",
            "Requirement already satisfied: astor>=0.6.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.1.0) (0.8.1)\n",
            "Requirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.1.0) (1.30.0)\n",
            "Requirement already satisfied: scipy==1.4.1; python_version >= \"3\" in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.1.0) (1.4.1)\n",
            "Collecting tensorboard<2.2.0,>=2.1.0\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/d9/41/bbf49b61370e4f4d245d4c6051dfb6db80cec672605c91b1652ac8cc3d38/tensorboard-2.1.1-py3-none-any.whl (3.8MB)\n",
            "\u001b[K     |████████████████████████████████| 3.9MB 4.5MB/s \n",
            "\u001b[?25hCollecting tensorflow-estimator<2.2.0,>=2.1.0rc0\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/18/90/b77c328a1304437ab1310b463e533fa7689f4bfc41549593056d812fab8e/tensorflow_estimator-2.1.0-py2.py3-none-any.whl (448kB)\n",
            "\u001b[K     |████████████████████████████████| 450kB 25.8MB/s \n",
            "\u001b[?25hRequirement already satisfied: wrapt>=1.11.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.1.0) (1.12.1)\n",
            "Requirement already satisfied: google-pasta>=0.1.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.1.0) (0.2.0)\n",
            "Requirement already satisfied: keras-applications>=1.0.8 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.1.0) (1.0.8)\n",
            "Collecting gast==0.2.2\n",
            "  Downloading https://files.pythonhosted.org/packages/4e/35/11749bf99b2d4e3cceb4d55ca22590b0d7c2c62b9de38ac4a4a7f4687421/gast-0.2.2.tar.gz\n",
            "Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.1.0) (1.1.0)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from protobuf>=3.8.0->tensorflow==2.1.0) (49.1.0)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.6/dist-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (3.2.2)\n",
            "Requirement already satisfied: google-auth<2,>=1.6.3 in /usr/local/lib/python3.6/dist-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (1.17.2)\n",
            "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.6/dist-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (0.4.1)\n",
            "Requirement already satisfied: requests<3,>=2.21.0 in /usr/local/lib/python3.6/dist-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (2.23.0)\n",
            "Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.6/dist-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (1.0.1)\n",
            "Requirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from keras-applications>=1.0.8->tensorflow==2.1.0) (2.10.0)\n",
            "Requirement already satisfied: importlib-metadata; python_version < \"3.8\" in /usr/local/lib/python3.6/dist-packages (from markdown>=2.6.8->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (1.7.0)\n",
            "Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (0.2.8)\n",
            "Requirement already satisfied: rsa<5,>=3.1.4; python_version >= \"3\" in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (4.6)\n",
            "Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (4.1.1)\n",
            "Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.6/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (1.3.0)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests<3,>=2.21.0->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (2020.6.20)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests<3,>=2.21.0->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (2.10)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests<3,>=2.21.0->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (3.0.4)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests<3,>=2.21.0->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (1.24.3)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.6/dist-packages (from importlib-metadata; python_version < \"3.8\"->markdown>=2.6.8->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (3.1.0)\n",
            "Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.6/dist-packages (from pyasn1-modules>=0.2.1->google-auth<2,>=1.6.3->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (0.4.8)\n",
            "Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.6/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.2.0,>=2.1.0->tensorflow==2.1.0) (3.1.0)\n",
            "Building wheels for collected packages: gast\n",
            "  Building wheel for gast (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for gast: filename=gast-0.2.2-cp36-none-any.whl size=7540 sha256=1627a30eb70b0869832dc923f358a951a47c0ed01954a9ce410e72be46149072\n",
            "  Stored in directory: /root/.cache/pip/wheels/5c/2e/7e/a1d4d4fcebe6c381f378ce7743a3ced3699feb89bcfbdadadd\n",
            "Successfully built gast\n",
            "\u001b[31mERROR: tensorflow-probability 0.10.0 has requirement gast>=0.3.2, but you'll have gast 0.2.2 which is incompatible.\u001b[0m\n",
            "Installing collected packages: tensorboard, tensorflow-estimator, gast, tensorflow\n",
            "  Found existing installation: tensorboard 2.2.2\n",
            "    Uninstalling tensorboard-2.2.2:\n",
            "      Successfully uninstalled tensorboard-2.2.2\n",
            "  Found existing installation: tensorflow-estimator 2.2.0\n",
            "    Uninstalling tensorflow-estimator-2.2.0:\n",
            "      Successfully uninstalled tensorflow-estimator-2.2.0\n",
            "  Found existing installation: gast 0.3.3\n",
            "    Uninstalling gast-0.3.3:\n",
            "      Successfully uninstalled gast-0.3.3\n",
            "  Found existing installation: tensorflow 2.2.0\n",
            "    Uninstalling tensorflow-2.2.0:\n",
            "      Successfully uninstalled tensorflow-2.2.0\n",
            "Successfully installed gast-0.2.2 tensorboard-2.1.1 tensorflow-2.1.0 tensorflow-estimator-2.1.0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4sEvCYNUroi2",
        "colab_type": "text"
      },
      "source": [
        "Хэрэгтэй сангуудаа import хийж оруулж ирье!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LLKVip7ieGTe",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 88
        },
        "outputId": "4a3d4fbb-fa6e-4431-f870-cb4acd191344"
      },
      "source": [
        "import gc\n",
        "import logging\n",
        "\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "\n",
        "from time import time\n",
        "from tqdm import tqdm\n",
        "\n",
        "import seaborn as sns\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "import tensorflow as tf\n",
        "from transformers import TFAlbertModel, AlbertTokenizer\n",
        "\n",
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "print(tf.__version__)"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/statsmodels/tools/_testing.py:19: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.\n",
            "  import pandas.util.testing as tm\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "2.1.0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YtjSkSZ7JARX",
        "colab_type": "text"
      },
      "source": [
        "### TPU-г ачааллая"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CoD-WO4hUiRG",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 748
        },
        "outputId": "0dc2bc97-22c8-49b1-8668-8386b46a2dc8"
      },
      "source": [
        "# Detect hardware, return appropriate distribution strategy\n",
        "try:\n",
        "    # TPU detection. No parameters necessary if TPU_NAME environment \n",
        "    # variable is set. On Kaggle this is always the case.\n",
        "    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()  \n",
        "    print('Running on TPU ', tpu.master())\n",
        "except ValueError:\n",
        "    tpu = None\n",
        "\n",
        "if tpu:\n",
        "    tf.config.experimental_connect_to_cluster(tpu)\n",
        "    tf.tpu.experimental.initialize_tpu_system(tpu)\n",
        "    strategy = tf.distribute.experimental.TPUStrategy(tpu)\n",
        "else:\n",
        "    # default distribution strategy in Tensorflow. Works on CPU and single GPU.\n",
        "    strategy = tf.distribute.get_strategy() \n",
        "\n",
        "print(\"REPLICAS: \", strategy.num_replicas_in_sync)"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "INFO:absl:Entering into master device scope: /job:worker/replica:0/task:0/device:CPU:0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Running on TPU  grpc://10.81.81.218:8470\n",
            "INFO:tensorflow:Initializing the TPU system: 10.81.81.218:8470\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Initializing the TPU system: 10.81.81.218:8470\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Clearing out eager caches\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Clearing out eager caches\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Finished initializing TPU system.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Finished initializing TPU system.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Found TPU system:\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Found TPU system:\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Num TPU Cores: 8\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Num TPU Cores: 8\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Num TPU Workers: 1\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Num TPU Workers: 1\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Num TPU Cores Per Worker: 8\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Num TPU Cores Per Worker: 8\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:localhost/replica:0/task:0/device:CPU:0, CPU, 0, 0)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:localhost/replica:0/task:0/device:CPU:0, CPU, 0, 0)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:localhost/replica:0/task:0/device:XLA_CPU:0, XLA_CPU, 0, 0)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:localhost/replica:0/task:0/device:XLA_CPU:0, XLA_CPU, 0, 0)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:CPU:0, CPU, 0, 0)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:CPU:0, CPU, 0, 0)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:0, TPU, 0, 0)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:0, TPU, 0, 0)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:1, TPU, 0, 0)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:1, TPU, 0, 0)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:2, TPU, 0, 0)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:2, TPU, 0, 0)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:3, TPU, 0, 0)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:3, TPU, 0, 0)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:4, TPU, 0, 0)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:4, TPU, 0, 0)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:5, TPU, 0, 0)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:5, TPU, 0, 0)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:6, TPU, 0, 0)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:6, TPU, 0, 0)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:7, TPU, 0, 0)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:7, TPU, 0, 0)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU_SYSTEM:0, TPU_SYSTEM, 0, 0)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU_SYSTEM:0, TPU_SYSTEM, 0, 0)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:XLA_CPU:0, XLA_CPU, 0, 0)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:XLA_CPU:0, XLA_CPU, 0, 0)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "REPLICAS:  8\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lUXesMU1c5Sq",
        "colab_type": "text"
      },
      "source": [
        "## Өгөгдөл\n",
        "Энэ удаагийн сургалтыг **Эдүгэ** сайтын дата дээр туршиж үзэцгээе.\n",
        "\n",
        "Мөн туршилтын датаг [tugstugi/mongolian-nlp](https://github.com/tugstugi/mongolian-nlp) репогоос татаж авсан болно."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "m0wSf3SKqmE-",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 289
        },
        "outputId": "8f156a5e-c4d8-44fe-c5e0-2890292ca8e0"
      },
      "source": [
        "# downloading data\n",
        "! wget https://github.com/tugstugi/mongolian-nlp/raw/master/datasets/eduge.csv.gz\n",
        "! gunzip eduge.csv.gz"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2020-07-26 09:59:28--  https://github.com/tugstugi/mongolian-nlp/raw/master/datasets/eduge.csv.gz\n",
            "Resolving github.com (github.com)... 140.82.112.4\n",
            "Connecting to github.com (github.com)|140.82.112.4|:443... connected.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: https://raw.githubusercontent.com/tugstugi/mongolian-nlp/master/datasets/eduge.csv.gz [following]\n",
            "--2020-07-26 09:59:28--  https://raw.githubusercontent.com/tugstugi/mongolian-nlp/master/datasets/eduge.csv.gz\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 73951267 (71M) [application/octet-stream]\n",
            "Saving to: ‘eduge.csv.gz’\n",
            "\n",
            "eduge.csv.gz        100%[===================>]  70.53M  82.2MB/s    in 0.9s    \n",
            "\n",
            "2020-07-26 09:59:30 (82.2 MB/s) - ‘eduge.csv.gz’ saved [73951267/73951267]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N1XzqBYzJRMV",
        "colab_type": "text"
      },
      "source": [
        "Энэхүү ажил нь eduge датаг 9 ангилалд хуваах ба хэрвээ та хүсвэл 2 ангилалд буюу binary classification асуудалд ашигласан ч болно"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_cY0Gc4uq3OR",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 111
        },
        "outputId": "e39c6d89-43c9-49e3-a568-169535319a04"
      },
      "source": [
        "# eduge_data = pd.read_csv(\"./eduge.csv\", nrows=7500) # binary classification болгохыг хүсвэл энэ мөрийг ажиллуулна\n",
        "eduge_data = pd.read_csv(\"./eduge.csv\")\n",
        "eduge_data.head(2)"
      ],
      "execution_count": 155,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>news</th>\n",
              "      <th>label</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Киноны кадраас «Юрийн галавын үе» кино проката...</td>\n",
              "      <td>урлаг соёл</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>“Universal” кино компани Жадд Апатоугийн найру...</td>\n",
              "      <td>урлаг соёл</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                news       label\n",
              "0  Киноны кадраас «Юрийн галавын үе» кино проката...  урлаг соёл\n",
              "1  “Universal” кино компани Жадд Апатоугийн найру...  урлаг соёл"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 155
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WXSy42U0sMHr",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 313
        },
        "outputId": "f782761a-e677-4ba5-8147-4f80f5eba7c8"
      },
      "source": [
        "print(\"Энэхүү дата доторх *label* утгууд нь:\", eduge_data[\" label\"].unique())\n",
        "sns.countplot(data=eduge_data, y=\" label\");"
      ],
      "execution_count": 70,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Энэхүү дата доторх *label* утгууд нь: ['урлаг соёл' 'эдийн засаг' 'эрүүл мэнд' 'хууль' 'улс төр' 'спорт'\n",
            " 'технологи' 'боловсрол' 'байгал орчин']\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAcYAAAEGCAYAAAD/ijWdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deZhV1Znv8e8PHJgUB4wCxkCMQxwxFsaBGKRNYrxGTdoxTmBsblrj0Immk2vaoIk3V+1MSKISoyKiMZKoae2OM4gahUJGB1qcEpW0ojigooDv/WOvkmV5quoAderssn6f5zlP7b3W2mu/exd1Xtba+5ytiMDMzMwK3eodgJmZWZk4MZqZmWWcGM3MzDJOjGZmZhknRjMzs8w69Q7AVunXr18MGjSo3mGYmXUqM2fOXBwRm7VXf06MJTJo0CAaGxvrHYaZWaci6dn27M9TqWZmZhmPGEvksedeZvezrq53GGarZeZFx9c7BLN25RGjmZlZxonRzMws48RoZmaWcWI0MzPLODGamZllnBjbgaTLJe0gabKkAfWOx8zM1pw/rtEOIuKktHhYXQMxM7O1VqoRo6TzJJ2RrZ8v6XRJwyXdK+lWSQskXSqpW9ZupaTZkhZKuiWVfUXSQ5JmSbpT0uapfIyk51P72ZI+lMwkHS9prqQ5kiamskGS7k7ld0naKpVvJukPkmak1z5ZP/m+lkpqqN3ZMzOz9lCqxAhcARwPkBLfUcA1qW4P4FRgB2Br4GupXXfgzYgYApyU9XUfsGdE7Ab8DvhuVvfziBiSXpPzACTtCPwAGBERuwKnp6qLgQkRsQswCRibyn+Z+hsK/CNwedZdd+CnKbaK3/UmabSkRkmNK956o80TZGZmtVWqqdSIeEbSy5J2AzYHZkXEy5IApkfEUwCSrgOGAZOBnsCyCt1tCVwvqT+wHvB0lWGMAG6IiMUppldS+V6kZAxMBC5My/sDO6QYATaU1CcilqbYFrVxzOOB8QC9txgcVcZoZmY1UqrEmFwOjAS2oBhBNmmeNJrWBwAvVOjnYuBnEfEnScOBMe0a5SrdKEamlZLzAIqRq5mZdRJlm0oFuBE4ABgK3JaV7yFpcJpiPZJVCecI4P4K/fQFnk/LJ6zG/u8GDpe0KYCkTVL5AxRTuwDHANPS8u0UU7yk9kPSz37A54CHVmPfZmZWZ6VLjBHxLnAP8PuIWJlVzQDGAY9RTIveKOk0YB/g3ApdjQFukDQTWLwa+38EOB+YKmkO8LNUdSowStJc4DhWXXs8DWhIN+U8Cnwzld8HjImIVqdSzcysXBRRrstaaUT4MHB4RDyRyoYDZ0bEQfWMrdZ6bzE4tj+uUo43Ky8/XcPqTdLMiGi3u/5LNWKUtAOwELirKSmamZl1pFLdfBMRjwKfrFA+BZjS0fGYmVnXU6oRo5mZWb05MZqZmWVKNZXa1X16y01p9I0MZmZ15RGjmZlZxonRzMws48RoZmaWcWI0MzPL+OabEnl30SP89byd6x2Gmdn7tjpnXr1D6HAeMZqZmWWcGM3MzDJOjGZmZhknRjMzs4wTo5mZWcaJ0czMLPORSIyS9pP0F0kPStqv3vGYmVnn9ZH4HGNE3APsVe84zMys86vLiFHSsZJmptc4SetldbdIWihptqR3JfVL5c9ky9dImp+WR0oal5a3k7RC0mEVtukn6ZkKsewiqVHSLEkzJG2fys9J6/MljZekVP4pSXdKmiPpYUlbS+oj6a60Pk/SIantIElvp2OZLenqGp5WMzNrB/WaSr0hInaPiN2BRcAZWV134MSIGAK80HxDSTsDO7XQ74+Ax1YnkIiYGxENEbEbcAdwQqoaFxFDI2InoCdwUCqfBPwqInYF9k7xLwO+GhGfAfYDftqUSIEnI2JIen3omVKSRqfE3PjKmytXJ3QzM6uBukylRsQ7ku4E+gE9gOeBC1N1H+CVVjb/MfBD4Py8UFIDRaKf2az9PZJWUiTciiQdCPwqtflCKt5P0neBXsAmwCOSpgADI+LGdBzL0vbrAv9X0r7Ae8BAYPNWjuF9ETEeGA+wy8CeUc02ZmZWO3W7+SYi9k+jwm82q/oEFUaKyd7AUmBOhbofAf9WoXy/tJ8Wb8qJiP+MiMHAJcDBknoAvwYOi4idgd9QJPCWHANsBuye9vU/bbQ3M7OSqtc1xi1U6A6cQjGFiaS9gL9GREsjxjHAORXKPw8siojVmkZN++ybTXsuo5imbUpqiyX1AQ4DiIg3gOckHZq2XV9SL6Av8GJELE93xX5ideMwM7NyqNeI8R+AecB8YAnwc0kDgP8Ctm26WQUYAFyUbfdQRDxZob9tKJLmmhgBzJU0DzgC+ElEvEoxSpwP3AbMyNofB5wmaS7wALAFxXXHhtTH8cDjaxiLmZnVmSLKcVlL0iBgTESMbFY+OSIOq0dMHW2XgT3jlv/9qXqHYWb2vs7w2ClJMyOiob36K9MH/F+iuMbX3M87OhAzM+u6SvMB/4h4E3ioQvn9dQjHzMy6qDKNGM3MzOquNCNGg/X678hW5zTWOwwzsy7NI0YzM7OME6OZmVnGidHMzCzjxGhmZpbxzTcl8viLj7PPxfvUOwwrsftP9aeXzGrNI0YzM7OME6OZmVnGidHMzCzjxGhmZpZxYjQzM8s4MZqZmWWcGM3MzDJOjGZmZplSJEZJx0qamV7jJK2XyqdI+qWk2ZLmS9pDUjdJ/y1p09Smu6SnJH1C0pOS1knlG0l6OlufImlB6mtphRiGSwpJB6T1jSW9LWmMpM0kPSjpYUl3SdohtRkpaVzWxzhJI7P1+ZIebWmfZmZWPqVIjMANEbF7ROwOLALOyOp6RcQQ4GTgioh4D7gW+Hqq/yLQGBHPAtOAL6fyo1O/K9J6d+Do1FdLHgaOT8tfB+ak5VeAYRHxGeCfgaskqYrj6g4c0No+JY2W1CipcfnS5VV0aWZmtVSKxBgR70i6U9Js4DjgS1n1danNvcCGkjYCrgROSPUj0zrAb4FRafmErBygJ7CsjVAWAetL2gQ4GPhT2vdK4OQU3++BnYAt0zZHphHhbODIZv31oUiqLYqI8RHREBEN6/ZZt43wzMys1kqRGAEiYv80svpm86oPN41ngVckDQcagNtTxTRgW0n7pvXHsu36UyS+tlwLXAAsAN7Ndjo2IoakGF/M2l+flV/fVCipB9AjIjyFambWiZQiMUraQoXuwCnAHVn1kanNMOC1iHgtlf8WuIZiunRl1n4iRXK7Kut/GPBqRCypIpz/AHYDrsi275lGqkg6BHgLeK6Nfr4K3FbF/szMrERKkRiBfwDmAfOBJcDPs7plkmYBlwLfyMpvAjagSIS5CcDmwO8AJA0FxgInVhNIRLybpjZnZ8W9gTslzQX+FTgyIpqPZN8nqYEicQ/Ppll7SjqvmhjMzKx+1Mr7e91JmgKcGRGNLdQ/AzRExOJm5Usjok/tI6wsTfEOj4gxWVkfYFxEjGxpuz5b9Yldz9q15vFZ5+XHTpl9mKSZEdHQXv35eYy18SiwuFnZMuCSOsRiZmarodSJMSKGt1E/qIXyuo0W0/5f5IM36JA+NvJQfSIyM7NqleUao5mZWSmUesTY1Wz/se19DcnMrM48YjQzM8s4MZqZmWWcGM3MzDJOjGZmZhnffFMibyxYwNR9P1/vMMxK4/P3Tq13CNYFecRoZmaWcWI0MzPLODGamZllnBjNzMwyToxmZmYZJ0YzM7OME6OZmVnGidHMzCzTZROjpKGS5krqIam3pEck/VHSoVmbSZIOkXSvpCFZ+X2Sdk3LYyQ9L2m2pKWSGiQNkjS/HsdlZmZrp8smxoiYAfwJ+DFwIXAN8EtgJICkvsDewK3Ab7PybYEeETEnddUd+GlEDAEaVzcOSaMlNUpqfG358rU5JDMzawddNjEm5wFfABqACyNiKrCNpM2Ao4E/RMQK4AbgIEnrAicCV2V99ASWVeh76zSKnC3p7JYCiIjxEdEQEQ191123fY7KzMzWWFf/rtRNgT7AukAP4E3gauBY4ChgFEBEvCXpDuAQ4Ahg96yPAcB9Ffp+MiKGSOoFzJY0OSIW1OxIzMysXXT1EeNlwL8Bk4ALUtlVwBkAEfFo1vZyYCwwIyKWAEjqB3wOeKiVfbwNvEWRfM3MrOS67IhR0vHA8oi4VlJ34AFJIyLibkmPATfl7SNipqTXgSuz4vuAMRGxqMIuBku6j2Kq9d6I8M04ZmadQJdNjBFxNcW0KRGxEvgsQJr63Aa4Lm8vaQDFCPv2rI/tm/U5PFvtXYu4zcystrr6VOoHSNofeAy4OCJey8qPp5guPTsi3qtXfGZmVntddsRYSUTcCXyiQvn7o0szM/to84jRzMws48RoZmaW8VRqiWyw3XZ8/t6p9Q7DzKxL84jRzMws48RoZmaWaXEqVdK3W9swIn7W/uGYmZnVV2vXGDfosCgMgBefe41x3/mPeodhZtZhvvXTr9Q7hA9pMTFGxLkdGYiZmVkZtHmNUdK2ku5qevCupF0k/aD2oZmZmXW8am6++Q3wfWA5QETMpXgkk5mZ2UdONYmxV0RMb1a2ohbBmJmZ1Vs1iXGxpK2BAJB0GFDpMUtmZmadXjXffHMKMB7YXtLzwNPAMTWNyszMrE7aTIwR8RSwv6TeQLeIeKP2YZmZmdVHNXelbippLDANmCLpl5I2rX1oZmZmHa+aa4y/A14C/hE4LC1fX8ug1pak8ySdka2fL+n0tDxc0muSZkv6u6QzU/lQSQ9ImiNpuqQNmvU5KW3ziqSn0/I3JfWQdKWkmWm7/VL7kZJuljRF0hOSftiR58DMzNZMNdcY+0fEj7L1H0s6slYBtZMrgD8Cv5DUjeLjJXukuu7A1Ig4WNIYAEnrUST7IyNihqQNgbfzDiPimNT2KuCWiJic1r8DPBURoyR9DLhTUtO+9gB2At4CZki6NSIa834ljQZGA2y8wWbteArMzGxNVDNivF3SUZK6pdcRwG21DmxtRMQzwMuSdgO+CMyKiJdTdU9gWbNNtgMWRcSMtP3rEVHtR1KGAcdJmg3cDgwCtk11d0TEyxHxNkWiHlYh1vER0RARDX169a36GM3MrDZa+xLxNyg+oiHgDOCaVNUNWAqcWfPo1s7lwEhgC4oRZJMBwAvtvK/TIuLPAJIezMqjWbvm62ZmVjItjhgjYoOI2DD97BYR66RXt4jYsCODXEM3AgcAQ0kjXEndga8B9zdruwDoL2loareBpGof4jwNOCltNwT4eOoP4AuSNpHUEzi0wn7NzKxkqnrzl7QxsA3Qo6ksIu6tVVDtISLelXQP8GpErEzFE4EngD9UaHskcHFKYm8D+1OMjNvya2BnSY9SfG3esRHxjiSA6WlfWwLXNL++aGZm5dNmYpR0EnA6xZv7bGBP4C/AiNqGtnbSTTd7Aoc3lUXE1/M2ETEmW56R2rcqIkY2W18GjGqh+XMRcWjVQZuZWd1Vc/PN6RTTkc9GxH7AbsCrNY1qLUnaAVgI3BURT9Q7HjMz6zyqmUpdFhHLJCFp/Yh4XNJ2NY9sLUTEo8An6xzDVcBV9YzBzMxWXzWJ8TlJGwE3AXdIWgI8W9uwzMzM6qOa70r9alock25m6Qv8uaZRmZmZ1Ulrn2PcpELxvPSzD/BKTSLqwj62ZV++9dOv1DsMM7MurbUR40xWfcC/SdN6UOdreGZmZrXQYmKMiMEdGYiZmVkZVPNxDTMzsy7DidHMzCxT7feBWgdY9PSTnH/sYfUOw8ysQ519zeR6h/ABHjGamZllnBjNzMwyToxmZmYZJ0YzM7OME6OZmVnGidHMzCzjxGhmZpbp0olR0vGS5kqaI2mipKskPS1pdnq9LWlQavttSfPT64xUNkjS45ImSXpM0mRJvSQdmbZfKOm1tPyf9TxWMzOrTpdNjJJ2BH4AjIiIXYHTU9VZETEkIoYAT6a2uwOjgM8CewL/JGm31H474NcR8WngdeDkiLg+bX8SMC31d2ALcYyW1Cip8c1l79ToaM3MrFpdNjECI4AbImIxQES09hitYcCNEfFmRCwF/gh8LtX9LSLuT8vXpLZVi4jxEdEQEQ29e6y/ekdgZmbtrisnxvYSbaybmVkn0pUT493A4ZI2hRYfzNxkGnBoun7YG/hqKgPYStJeafnrwH21CtjMzGqvy36JeEQ8Iul8YKqklcCsVto+LOkqYHoqujwiZqUbcxYAp0i6AngUuKSmgZuZWU112cQIEBETgAmt1O+ULf8M+FmFZisi4tgWtp8CTFm7KM3MrCN15alUMzOzD+nSI8a1FRHPADu11c7MzDoPjxjNzMwyToxmZmYZT6WWSP/BW3P2NZPrHYaZWZfmEaOZmVnGidHMzCzjxGhmZpZxYjQzM8v45psSWbboDR47/+56h2El8OmzR9Q7BLMuyyNGMzOzjBOjmZlZxonRzMws48RoZmaWcWI0MzPLODGamZllnBjNzMwypU2Mki6SNFvS3yU9n5bPk3SWpBmS5ko6N7UdmtZ7SOot6RFJO0kaLumWrM8zJY1Jy0MkPZi2u1HSxlm7KZIWpH0uTWU9JF0paZ6kWZL2S+UjJb2U2s6WdJqkQZLmp/p1JT0laVwHnj4zM1tDpU2MEXFWRAwBLgV+npbvA7YB9gCGALtL2jciZgB/An4MXAhcExHz29jF1cC/RsQuwDzgh1ldd+DotM8mpxRhxc7A0cAEST1S3fURMSS9xjbbz2hgaUtBSBotqVFS4ytvvtpGyGZmVmulTYwt+GJ6zQIeBranSJQA5wFfABookmOTzzWN5oB/AZDUF9goIqamNhOAfbNtegLLmu17GHANQEQ8DjwLbNtasJJ6A6OAX7fUJiLGR0RDRDRs0nuj1rozM7MO0Nm+Ek7ATyLisgp1mwJ9gHWBHsCbqXxaRBwExVRqatOW/sCitQ+X04HxwLvt0JeZmXWAzjZivA04UVIfAEkDJX0s1V0G/BswCbigtU4i4jVgiaTPpaLjgKmpz2HAqxGxpNlm04BjUpttga2ABa3spi9wKHBFdYdmZmZl0KlGjBFxu6RPA3+RBMW1u2MlHQAsj4hrJXUHHpA0Anivle5OAC6V1At4ChglaSgwFjixQvtfA5dImgesAEZGxDspjkq2BM6MiBWttDEzs5JRRNQ7Bkt2Grhd3HDyJfUOw0rAT9cwq56kmRHR0F79dbapVDMzs5pyYjQzM8s4MZqZmWWcGM3MzDKd6q7Uj7oe/TfwTRdmZnXmEaOZmVnGidHMzCzjxGhmZpZxYjQzM8v45psSeeGFFxgzZky9w7A2+Hdk9tHmEaOZmVnGidHMzCzjxGhmZpZxYjQzM8s4MZqZmWVKnRglbSVpoqTpkuZL6lfvmMzM7KOttB/XkNQDuA44G5gafqKymZl1gDKPGEcAPYFxwDxJFwBIOlrSvDSCvCDfQNJKSbMlLZR0SyobJOluSXMl3SVpq1R+laSnUz9zJe2Uyj8l6U5JcyQ9LGlrScMl3SvpVkkLJF0qqduaxGNmZuVW5sS4GTAQ2A8YAgyV9HXgAoqk2VR2KICk7sCbETEEOCnr52JgQkTsAkwCxmZ1Z0XETsC9qU9Sm19FxK7A3sCiVL4HcCqwA7A18DVJA9YgHjMzK7EyJ0YBt0XESxGxgiJhnQVMaVa2b2rfE1hWoZ+9gGvT8kRgWFZ3kaQngIOBGyRtAAyMiBsBImJZRLyV2k6PiKciYiXFFO8wYOgaxPPBg5RGS2qU1PjWW2+11dzMzGqszInx9dVsPwB4YTW3OSsitgHOA85to23za5xtXfOsKp6IGB8RDRHR0KtXr7aam5lZjZU5Mc4ERkjql6YljwZ+AXy+WdnU1P4I4P4K/TwAHJWWjwGmVWjzOtAvIt4AnsumQ9eX1JSt9pA0OF1bPBK4D5i+BvGYmVmJlfau1Ih4VtIYiut/K4FbI2KCpHeBeyimWm+NiJslnQbsA5xQoatTgSslnQW8BIzK6i6S9AOK0V/TdcDjgMsknQcsBw5P5TMobgT6VNr/jRHxnqTvrWY8ZmZWYvKnINomaThwZkQcVMv9DBgwIEaPHl3LXVg78NM1zMpF0syIaGiv/so8lWpmZtbhSjuVWiYRMQWYUucwzMysA3jEaGZmlnFiNDMzy/jmmxJpaGiIxsbGeodhZtap+OYbMzOzGnJiNDMzyzgxmpmZZZwYzczMMv4cY4ksWfIYv79hj3qHYTVyxOHT6x2CmVXBI0YzM7OME6OZmVnGidHMzCzjxGhmZpZxYjQzM8s4MZqZmWVqlhglbSVpoqTpkuZL6lfFNldLapQ0sVZxmZmZtaYmn2OU1AO4DjgbmBpVflN5RBxfi3jMzMyqVasR4wigJzAOmCfpgqYKSZekUeEjks7Nyp9pGlVKukbS/LQ8UtJLkman12mp/CZJM1M/o/OdS1qZ2i6UdEvz4CRtkrafK+lBSbuk8jFplPsXSU9I+qcshnHZ9kvTz+FN/ac+X5V0ZlqfIqmh+TZmZlZutfrmm82AgcBOwBLgdkmHRsRNwNkR8Yqk7sBdknaJiLlNG0raOW2Xuz4ivtWs7MTUT09ghqQ/RMTLqd83I2KIpOHAmRXiOxeYFRGHShoBXA0MSXW7AHsCvYFZkm4F3gPUxjF/H/hrG20+JCX10QD9+q23upubmVk7q9WIUcBtEfFSRKwAJgH7projJD0MzAJ2BHZotu2PgR9WsY/TJM0BHgQ+DmyTynsCy9rYdhgwESAi7gY2lbRhqrs5It6OiMXAPcAewHPArpIqni9JAymS6Y3NqiY1jXRTXB8SEeMjoiEiGjbc0N/QZ2ZWb7V6J369UqGkwRQjuKERsUTSVUCPrMnewFJgTmudp5Hg/sBeEfGWpClZPwOAF9Yi9ubXQwOYAiygmBZeXmGbHwI/oog/d0xENKaYPZVqZtYJ1GrEOBMYIalfmto8GpgKbAi8CbwmaXPgy822GwOcU0X/fYElKSluTzFaa3IEcH8b208DjoH3k+ziiGhK5odI6iFpU2A4MCMi3ouIb0TEjhExpFlfWwODIuL2KuI2M7OSq8mIMSKelTQGuBdYCdwaETcDSJoFPA78jQ8nsIci4klJg9rYxZ+Bb0p6jGIk92Dq+zRgH+CENrYfA1whaS7wVrP2cymmUPsBP4qItkaf2wOj2mhjZmadhKr8JEWXkJL50oj493rsf+ute8dP/t+O9di1dQA/dsqsNiTNjIiGtltWx998Y2ZmlvFtkJmIGFPvGMzMrL48YjQzM8t4xFgiG2/8aV+HMjOrM48YzczMMk6MZmZmGX9co0QkvUHxuczOqh+wuN5BrKHOHDt07vgde/105vjz2D8REZu1V8e+xlguC9rzszgdTVJjZ42/M8cOnTt+x14/nTn+WsbuqVQzM7OME6OZmVnGibFcxtc7gLXUmePvzLFD547fsddPZ46/ZrH75hszM7OMR4xmZmYZJ0YzM7OME2NJSDpA0gJJCyV9r97xAEj6uKR7JD0q6RFJp6fyTSTdIemJ9HPjVC5JY9MxzJX0mayvE1L7JyS19bzM9jyG7pJmSbolrQ+W9FCK8XpJ66Xy9dP6wlQ/KOvj+6l8gaQvdWDsG0maLOlxSY9J2quznHtJ/5L+zcyXdF16+Hdpz72kKyS9KGl+VtZu51rS7pLmpW3GSlKNY78o/buZK+lGSRtldRXPaUvvQS393moZf1b3HUkhqV9a75hzHxF+1fkFdAeeBD4JrAfMAXYoQVz9gc+k5Q2A/wZ2AC4EvpfKvwdckJYPBP4LELAnxYOnATYBnko/N07LG3fQMXwbuBa4Ja3/HjgqLV8K/HNaPhm4NC0fBVyflndIv4/1gcHp99S9g2KfAJyUltcDNuoM5x4YCDwN9MzO+cgyn3tgX+AzwPysrN3ONTA9tVXa9ss1jv2LwDpp+YIs9ornlFbeg1r6vdUy/lT+ceA24FmgX0ee+5r/cftV1T+MvYDbsvXvA9+vd1wV4rwZ+ALFt/P0T2X9Kb6YAOAy4Ois/YJUfzRwWVb+gXY1jHdL4C5gBHBL+sNYnL1hvH/e0x/gXml5ndROzX8Xebsax96XIrmoWXnpzz1FYvxbepNaJ537L5X93AOD+GByaZdzneoez8o/0K4WsTer+yowKS1XPKe08B7U2t9MreMHJgO7As+wKjF2yLn3VGo5NL2RNHkulZVGmt7aDXgI2DwiFqWqvwObp+WWjqNex/cL4LvAe2l9U+DViFhRIY73Y0z1r6X29Yp9MPAScKWKqeDLJfWmE5z7iHge+Hfgr8AiinM5k85z7pu017kemJabl3eUEylGSrD6sbf2N1Mzkg4Bno+IOc2qOuTcOzFamyT1Af4AnBERr+d1Ufw3rHSf+ZF0EPBiRMysdyxraB2K6aVLImI34E2K6bz3lfjcbwwcQpHcBwC9gQPqGtRaKuu5bouks4EVwKR6x1ItSb2A/wOcU68YnBjL4XmK+fQmW6ayupO0LkVSnBQRf0zF/yOpf6rvD7yYyls6jnoc3z7AwZKeAX5HMZ36S2AjSU3fEZzH8X6Mqb4v8HKdYofif7bPRcRDaX0yRaLsDOd+f+DpiHgpIpYDf6T4fXSWc9+kvc7182m5eXlNSRoJHAQckxI7bcRYqfxlWv691crWFP+pmpP+frcEHpa0RStxtu+5r9V8vV+rNb++DsXF4sGsuvC9YwniEnA18Itm5RfxwZsSLkzL/4sPXhifnso3obhetnF6PQ1s0oHHMZxVN9/cwAdvJDg5LZ/CB28A+X1a3pEP3qzwFB138800YLu0PCad99Kfe+CzwCNArxTPBODUsp97PnyNsd3ONR++AeTAGsd+APAosFmzdhXPKa28B7X0e6tl/M3qnmHVNcYOOfc1/+P2q+p/GAdS3PX5JHB2veNJMQ2jmD6aC8xOrwMprjvcBTwB3Jn9AxTwq3QM84CGrK8TgYXpNaqDj2M4qxLjJ9MfysL0B79+Ku+R1hem+k9m25+djmkB7Xg3YRVxDwEa0/m/Kf3Bd4pzD5wLPA7MByamN+LSnnvgOorrocspRuvfaM9zDTSkc/EkMI5mN1XVIPaFFNfcmv5uL23rnNLCe1BLv7daxt+s/hlWJcYOOff+SjgzM7OMr1ZLMIsAAAGOSURBVDGamZllnBjNzMwyToxmZmYZJ0YzM7OME6OZmVnGidHM2pWkM9K3l5h1Sv64hpm1q/RtJQ0RsbjesZitCY8YzbogScen59nNkTRR0iBJd6eyuyRtldpdJemwbLul6edwSVO06nmRk9Kz8k6j+H7UeyTdU5+jM1s767TdxMw+SiTtCPwA2DsiFkvahOJr2yZExARJJwJjgUPb6Go3iq8YewG4H9gnIsZK+jawn0eM1ll5xGjW9YwAbmhKXBHxCsVz9q5N9RMpvg6wLdMj4rmIeI/ia8cG1SBWsw7nxGhmrVlBep+Q1I3iC6abvJMtr8QzUPYR4cRo1vXcDRwuaVOANJX6AMWTLQCOoXiyBxRf4Lx7Wj4YWLeK/t8ANmivYM06mv+HZ9bFRMQjks4HpkpaCcyieCzUlZLOAl4CRqXmvwFuljQH+DPFA5PbMh74s6QXImK/9j8Cs9ryxzXMzMwynko1MzPLODGamZllnBjNzMwyToxmZmYZJ0YzM7OME6OZmVnGidHMzCzz/wGAeHFjh3hz3wAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lk5WHt_RTNRW",
        "colab_type": "text"
      },
      "source": [
        "Дээрх histogram-аас харахад `спорт, улс төр, эдийн засаг` зэрэг ангилалд 12'000-аас их дата байгаа нь харагдаж байна."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pTv1FxjiwhlG",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 71
        },
        "outputId": "e7c6f15b-882c-48d1-e2f3-3d771d6f77a0"
      },
      "source": [
        "id2label = {i:label for i, label in enumerate(eduge_data[\" label\"].unique())}\n",
        "label2id = {label:i for i, label in enumerate(eduge_data[\" label\"].unique())}\n",
        "print(id2label)\n",
        "print(label2id)"
      ],
      "execution_count": 71,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "{0: 'урлаг соёл', 1: 'эдийн засаг', 2: 'эрүүл мэнд', 3: 'хууль', 4: 'улс төр', 5: 'спорт', 6: 'технологи', 7: 'боловсрол', 8: 'байгал орчин'}\n",
            "{'урлаг соёл': 0, 'эдийн засаг': 1, 'эрүүл мэнд': 2, 'хууль': 3, 'улс төр': 4, 'спорт': 5, 'технологи': 6, 'боловсрол': 7, 'байгал орчин': 8}\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3OEWQlWYw-F0",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 111
        },
        "outputId": "8d7d9f40-089d-4d30-ca98-bd546e68e7df"
      },
      "source": [
        "eduge_data[\" label\"] = eduge_data[\" label\"].apply(lambda x: label2id[x])\n",
        "eduge_data.head(2)"
      ],
      "execution_count": 72,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>news</th>\n",
              "      <th>label</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Киноны кадраас «Юрийн галавын үе» кино проката...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>“Universal” кино компани Жадд Апатоугийн найру...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                news   label\n",
              "0  Киноны кадраас «Юрийн галавын үе» кино проката...       0\n",
              "1  “Universal” кино компани Жадд Апатоугийн найру...       0"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 72
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Fwl5Q97EJ9Go",
        "colab_type": "text"
      },
      "source": [
        "Датаныхаа 20%-ийг сургасан моделио тест хийхэд ашиглахаар үлдээе!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "V0bXXrwktQWZ",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "dfbbeb48-2669-439e-d6fb-39dc62d33d97"
      },
      "source": [
        "# data split\n",
        "X_train, X_test, y_train, y_test = train_test_split(eduge_data[\"news\"], eduge_data[\" label\"], test_size=0.2, random_state=42)\n",
        "print(X_train.shape, X_test.shape, y_train.shape, y_test.shape)"
      ],
      "execution_count": 122,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(60528,) (15133,) (60528,) (15133,)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PJsBXhkvc7L5",
        "colab_type": "text"
      },
      "source": [
        "## Tokenize хийх"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Dv1SOSMmKKje",
        "colab_type": "text"
      },
      "source": [
        "ALBERT-Mongolian-ыг анх сургахдаа хамгийн уртдаа 512 урттай оролт ордог байхаар сургасан ба энэхүү туршилтад ч тэр setting-ээ дагаад MAX_LEN-г 512 оор тохируулж өгье. Ингэснээр 512 оос урт оролт орж ирвэл tokenize хийгээд эхний 512 тэмдэгтийг л цааш модельруу явуулах юм."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tAhaVpoBeudW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "MAX_LEN = 512"
      ],
      "execution_count": 131,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YgyiDuiAKdrd",
        "colab_type": "text"
      },
      "source": [
        "Доор ALBERT-Mongolian-ийг сургахад ашигласан моделийг оруулж ирж байна."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NKRU7tI0c7mD",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# read tokenizer\n",
        "tokenizer = AlbertTokenizer.from_pretrained(\"bayartsogt/albert-mongolian\")"
      ],
      "execution_count": 75,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hWjhSR5Rz8Jg",
        "colab_type": "text"
      },
      "source": [
        "Transformers суурьтай моделийн оролтын хэлбэрийн доор харуулж байна."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AAwnCkhVy1d-",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        },
        "outputId": "1f2ee67f-97f0-477c-a637-29efdb8f3662"
      },
      "source": [
        "# Жишээ + тайлбар\n",
        "texts = [\"Шинэ Монгол ч сайн сургууль шүү\"]\n",
        "texts_encoded = tokenizer.batch_encode_plus(texts, max_length=MAX_LEN, padding=True, truncation=True, return_tensors=\"np\")\n",
        "print(\"Жишээ:\")\n",
        "print([tokenizer.cls_token] + tokenizer.tokenize(texts[0]) + [tokenizer.eos_token] + [tokenizer.pad_token])\n",
        "print(texts_encoded[\"input_ids\"][0])\n",
        "print(texts_encoded[\"token_type_ids\"][0])\n",
        "print(texts_encoded[\"attention_mask\"][0])\n",
        "\n",
        "# sharvaa ахын блогийг энд тавьчихвал амар юм байна!"
      ],
      "execution_count": 76,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Жишээ:\n",
            "['[CLS]', '▁шинэ', '▁монгол', '▁ч', '▁са', 'ин', '▁сургууль', '▁шүү', '[SEP]', '<pad>']\n",
            "[    2   161   396    19 12331   848   479   164     3]\n",
            "[0 0 0 0 0 0 0 0 0]\n",
            "[1 1 1 1 1 1 1 1 1]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CzLRouw5Kmbc",
        "colab_type": "text"
      },
      "source": [
        "`[CLS]` тэмдэгт хамгийн урд нэмэгдсэн байгаа. Сургалтыг явцад ALBERT Mongolian-ий бүх гаралтыг бус энэхүү тэмдэгтийг илэрхийлэх векторыг энэхүү оролтыг илэрхүүлнэ гэж үзээд цааш явах юм.\n",
        "\n",
        "Доорх хүснэгтийн дагуу: \n",
        "* `input_ids` гэдэг нь тухайн token-ий index-ийг заана.\n",
        "* `token_type` гэдэг нь `[SEP]` урд байна уу хойно байн уу гэдгийг заана. Манай тохиолдолд байнга `0` байна\n",
        "* `attention_mask` гэдэг нь ямар нэг үгийг масклах үгүйг заана. Манай тохиолдолд ямар ч үг масклах буюу нуух шаардагагүй тул бүгд `1` байна.\n",
        "\n",
        "| |[CLS]|▁шинэ|▁монгол|▁ч|▁са|ин|▁сургууль|▁шүү|[SEP]|\n",
        "|---|---|---|---|---|---|---|---|---|---|\n",
        "|input_ids|2|161|396|19|12331|848|479|164|3|\n",
        "|token_type_ids|0|0|0|0|0|0|0|0|0|\n",
        "|attention_mask|1|1|1|1|1|1|1|1|1|\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Dtncx1uEh0A8",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        },
        "outputId": "63b31c31-04a2-4e74-dc59-26c8dab939e3"
      },
      "source": [
        "%%time\n",
        "X_train_encoded = tokenizer.batch_encode_plus(list(X_train), max_length=MAX_LEN, padding=True, truncation=True, return_tensors=\"np\")\n",
        "X_test_encoded  = tokenizer.batch_encode_plus(list(X_test), max_length=MAX_LEN, padding=True, truncation=True, return_tensors=\"np\")\n",
        "\n",
        "print(\"size:\", X_train_encoded[\"input_ids\"].shape)\n",
        "print(\"size:\", X_test_encoded[\"input_ids\"].shape)"
      ],
      "execution_count": 77,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "size: (60528, 512)\n",
            "size: (15133, 512)\n",
            "CPU times: user 4min 48s, sys: 12.9 s, total: 5min 1s\n",
            "Wall time: 5min 2s\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1LEnvmjDdNlZ",
        "colab_type": "text"
      },
      "source": [
        "## Модель ба сургалт"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "v1RJLI22y4fC",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "a25e05cc-1fb2-4f23-a8a5-8b9f89a92e88"
      },
      "source": [
        "print(\"нийт ангилалын тоо:\", len(label2id))"
      ],
      "execution_count": 123,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "нийт ангилалын тоо: 9\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xdfg2VQAUCHP",
        "colab_type": "text"
      },
      "source": [
        "Сонгосон архитектур нь:\n",
        "\n",
        "`INPUT(512) => ALBERT-MONGOLIAN(768) => DENSE_LAYER(9)`\n",
        "\n",
        "ба ALBERT-Mongolian-ийг transformers-санг ашиглан дуудаж оруулж ирсэн байна."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KBVAt0kydOAJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def build_model():\n",
        "\n",
        "    with strategy.scope(): # TPU RAM дээр модель оо үүсгэх үед ашиглана\n",
        "        ids = tf.keras.layers.Input((MAX_LEN,), dtype=tf.int32)\n",
        "        att = tf.keras.layers.Input((MAX_LEN,), dtype=tf.int32)\n",
        "        tok = tf.keras.layers.Input((MAX_LEN,), dtype=tf.int32)\n",
        "\n",
        "        bert_model = TFAlbertModel.from_pretrained('bayartsogt/albert-mongolian')\n",
        "        \n",
        "        x = bert_model(ids, attention_mask=att, token_type_ids=tok)[1] # getting class token value\n",
        "        x = tf.keras.layers.Dropout(0.2)(x)\n",
        "        \n",
        "        if len(label2id) == 2:\n",
        "            x = tf.keras.layers.Dense(1, activation=\"sigmoid\")(x)\n",
        "        else:\n",
        "            x = tf.keras.layers.Dense(len(label2id), activation=\"softmax\")(x)\n",
        "        \n",
        "        model = tf.keras.models.Model(inputs=[ids, att, tok], outputs=[x])\n",
        "\n",
        "        optimizer = tf.keras.optimizers.Adam(learning_rate=1e-5)\n",
        "        loss_name = \"binary_crossentropy\" if len(label2id) == 2 else \"sparse_categorical_crossentropy\"\n",
        "\n",
        "        print(\"Loss:\", loss_name)\n",
        "        model.compile(loss=loss_name, \n",
        "                      optimizer=optimizer, \n",
        "                      metrics=[\"accuracy\"])\n",
        "\n",
        "    return model"
      ],
      "execution_count": 145,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aX3tryX-g8YQ",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 615
        },
        "outputId": "31d9fc05-12e2-4189-d5e1-3f227e5e4cc4"
      },
      "source": [
        "tf.keras.backend.clear_session()\n",
        "tf.tpu.experimental.initialize_tpu_system(tpu)\n",
        "\n",
        "model = build_model()\n",
        "model.summary()"
      ],
      "execution_count": 146,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:TPU system 10.81.81.218:8470 has already been initialized. Reinitializing the TPU can cause previously created variables on TPU to be lost.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:TPU system 10.81.81.218:8470 has already been initialized. Reinitializing the TPU can cause previously created variables on TPU to be lost.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Initializing the TPU system: 10.81.81.218:8470\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Initializing the TPU system: 10.81.81.218:8470\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Clearing out eager caches\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Clearing out eager caches\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Finished initializing TPU system.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Finished initializing TPU system.\n",
            "WARNING:transformers.modeling_tf_utils:Some weights of the model checkpoint at bayartsogt/albert-mongolian were not used when initializing TFAlbertModel: ['predictions']\n",
            "- This IS expected if you are initializing TFAlbertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).\n",
            "- This IS NOT expected if you are initializing TFAlbertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "WARNING:transformers.modeling_tf_utils:All the weights of TFAlbertModel were initialized from the model checkpoint at bayartsogt/albert-mongolian.\n",
            "If your task is similar to the task the model of the ckeckpoint was trained on, you can already use TFAlbertModel for predictions without further training.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Loss: sparse_categorical_crossentropy\n",
            "Model: \"model\"\n",
            "__________________________________________________________________________________________________\n",
            "Layer (type)                    Output Shape         Param #     Connected to                     \n",
            "==================================================================================================\n",
            "input_1 (InputLayer)            [(None, 512)]        0                                            \n",
            "__________________________________________________________________________________________________\n",
            "input_2 (InputLayer)            [(None, 512)]        0                                            \n",
            "__________________________________________________________________________________________________\n",
            "input_3 (InputLayer)            [(None, 512)]        0                                            \n",
            "__________________________________________________________________________________________________\n",
            "tf_albert_model (TFAlbertModel) ((None, 512, 768), ( 11683584    input_1[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "dropout_3 (Dropout)             (None, 768)          0           tf_albert_model[0][1]            \n",
            "__________________________________________________________________________________________________\n",
            "dense (Dense)                   (None, 9)            6921        dropout_3[0][0]                  \n",
            "==================================================================================================\n",
            "Total params: 11,690,505\n",
            "Trainable params: 11,690,505\n",
            "Non-trainable params: 0\n",
            "__________________________________________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x0woh6YxUsnQ",
        "colab_type": "text"
      },
      "source": [
        "### Сургалт\n",
        "Дээрх моделио ашиглан сургалт хийе.\n",
        "Энэхүү хэсэгт та early stopping, learning rate scheduler зэрэг арга ашиглан үр дүнг сайжруулах бүрэн боломжтой юм.\n",
        "\n",
        "Сургалт хэрхэн явж байгаа хянахын тулд сургалтад бэлдсэн датаныхаа 80%-ийг train-нд 20%-ийг validation хийхэд ашиглая."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0j9o_Zxd1526",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "BATCH_SIZE = 8 * strategy.num_replicas_in_sync\n",
        "EPOCHS = 5\n",
        "VAL_SPLIT = 0.2\n",
        "\n",
        "STEPS_PER_EPOCH = int(X_train_encoded[\"input_ids\"].shape[0] * (1-VAL_SPLIT) / BATCH_SIZE)\n",
        "\n",
        "history = model.fit(x=[X_train_encoded[\"input_ids\"], X_train_encoded[\"attention_mask\"], X_train_encoded[\"token_type_ids\"]], \n",
        "                    y=[np.array(y_train)], \n",
        "                    steps_per_epoch=STEPS_PER_EPOCH,\n",
        "                    epochs=EPOCHS,\n",
        "                    batch_size=BATCH_SIZE,\n",
        "                    validation_split=VAL_SPLIT,\n",
        "                    verbose=1)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hT2PF3GOp-nN",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 267
        },
        "outputId": "1b723760-4516-4041-ea94-eb5dcf6f1fb1"
      },
      "source": [
        "plt.plot(history.history[\"accuracy\"])\n",
        "plt.plot(history.history[\"val_accuracy\"])\n",
        "plt.show()"
      ],
      "execution_count": 148,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD6CAYAAACvZ4z8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3de3Bc53nf8e8DgABIEAABApRIgLjIJiXeJJFcSq4Vq75UCiN7olymCaU4kVJPNO1YSuok7didjONRmsaTyTRxJxrHisvm0sQaV8mknIxaxa3tutM6KRYkbV4syhSFBQGKEsBdACSI+z7945wFFiAuS2LJXZz9fWZ29uy57L67JH/n5XvOeY65OyIiEl1lhW6AiIjcXgp6EZGIU9CLiEScgl5EJOIU9CIiEaegFxGJuBWD3syOmtl7ZnZ6ieVmZv/BzM6b2ffN7EDWsmfM7Ifh45l8NlxERHJjK51Hb2aPAteAP3P3vYssfwJ4AXgCeBj4krs/bGaNQByIAQ50AwfdPbXc5zU1NXlHR8ctfBURkdLV3d096O7Niy2rWGljd/+OmXUss8qTBDsBB/7ezDaZ2Vbgw8A33D0JYGbfAA4DX1vu8zo6OojH4ys1S0REsphZYqll+RijbwEuZr3uC+ctNX+xBj5nZnEziw8MDOShSSIiklEUB2Pd/WV3j7l7rLl50f95iIjILcpH0PcD27Net4bzlpovIiJ3UD6C/hjwC+HZNx8Aht39HeB14HEzazCzBuDxcJ6IiNxBKx6MNbOvERxYbTKzPuA3gXUA7v5HwGsEZ9ycB64DvxguS5rZbwFd4Vu9mDkwKyIid04uZ908tcJyBz69xLKjwNFba5qIiORDURyMFRGR22fFHr2IiKyeuzM+lWZkfIqRsanweTrr9TQNGyp5+uG2vH+2gl5EJAfuzsR0ejakh8duDOqlAjwzf2pm+UoE+9s2KehFRPJlaibNxeR13h4c5dLw+JIhfTVr/uRMetn3rKooo279OuqqK6hbv45NGypp21wz+7queh116yvC53Xz5tdWV1C9rvy2fFcFvYhE1kzauTQ0Rs+VUd4enHv0DI5yMTXGTHp+D7uyPAzqMIzr169je8P6FUM6M/92BfVqKehFZE1zd967OjEb4NmBnkheZ3J6rhe+fl05nU017NlWzyfu30ZHUw2dTTW0Nqynfn3xBvVqKehFZE1IjU7y9pVR3h4YpefKKBfCYO8ZHGV0cmZ2vcryMto2b6CzqYaP3LeFzqYaOjbXcE9zDVtqqzCzAn6LwlDQi0jRuDYxPa9X3jMYBvqVUYauT82uV2awvXEDHZtrONTRSGfYM+9sqmHbpvWUl5VemC9HQS8id1xydJJT/cP84J0R3h4YDXrqg6MMXJ2Yt97W+mo6m2p4Yt9W7gl75p3NNWxv2EBlhS4DypWCXkRuq4GrE5zuH+Z0/zCnwudLw+Ozy5s2VtKxuYYP72ymo6kmCPQw1NdXRnPM/E5T0ItI3rw7Mj4v0E/3j3B5ZC7UO5tqONjRyLMtdezdVs+ebfXUb1hXwBaXBgW9iNw0d+fyyDin+sJAvzTCqf7h2aEXM7inqYYP3NPI3pZ69rbUs2dbHbXVCvVCUNCLyLLcnf6hsdkeeqa3fmV0EggOjL5/y0Y+tKOJvdvq2ddaz66tdWysUrwUC/1JiMgsd+dicozTl7KHX4ZJhWe8lJcZO7Zs5CP3bWFf2FPftbWWDZWKkmKmPx2REpVOO73J63OBfinosQ+PBaFeUWbsvKuWx3ffzd7WevZuq2PX1rrIXlQUZQp6kRJx5doEx3uHON6b4mTvEKcvDXN1fBoILjK69+5anti3lb0tdexrqefeu2upqlCoR4GCXiSCpmfSvHH5Kid6U7PhnrhyHQh66ru21vHjD2ybHX7ZeVetzkuPMAW9SAQMXpvgRBjoxxMpvt83zNhUUBagubaKA22beOqhNg60NbCvpV7np5cYBb3IGpPprWdC/XjvEL3Jud767m11/Oyh7exv28SBtgZaG9aXZH0XmaOgFylyg9cmOJ5IceLi0JK99Z97uI0D7UFvXQdLZSEFvUgRWam3vke9dbkFCnqRAsr01jMHTE+pty63QU5Bb2aHgS8B5cBX3f2LC5a3A0eBZiAJfNLd+8JlM8CpcNVed//xPLVdZE3Jtbd+oL2BA22baNmk3rrkx4pBb2blwEvAY0Af0GVmx9z9bNZqvwf8mbv/qZl9FPgd4OfDZWPu/mCe2y1S9K5PTnOyd4iunhTxRJLjidTsDTLUW5c7KZce/UPAeXe/AGBmrwBPAtlBvxv41XD6W8Df5LORImvBwNUJuhPJINh7kpy+NMJM2jGDe++q5acOtBLraOBge4N663JH5RL0LcDFrNd9wMML1vke8FMEwzs/CdSa2WZ3vwJUm1kcmAa+6O437ATM7DngOYC2trab/hIid5q7c2FwlHjPXLD3hBckVVWU8eD2TfyLf/w+DnY0cKCtgfr1qtoohZOvg7G/DvyhmT0LfAfoBzI3cWx3934zuwf4ppmdcve3sjd295eBlwFisdj827KLFIHJ6TRnLg0T70nR1ZMknkiRDKs3NmxYR6yjkacfbiPW0cjebfW6ylSKSi5B3w9sz3rdGs6b5e6XCHr0mNlG4KfdfShc1h8+XzCzbwP7gXlBL1JsRsanONE7FPbYk5y8OMT4VBqA9s0b+Mi9WzjU0UCso5H3NddoGEaKWi5B3wXsMLNOgoA/AjydvYKZNQFJd08DnyM4AwczawCuu/tEuM4jwO/msf0iefHO8NjsEExXT4o3Lo/gHpTl3bOtjqcfaudQRwMHOxrYUltd6OaK3JQVg97dp83seeB1gtMrj7r7GTN7EYi7+zHgw8DvmJkTDN18Otx8F/AVM0sDZQRj9Gdv+BCROyiddn743rVgCCYM9v6hMQA2VJZzoK2BX/nYDg51NPLg9k3U6AYassaZe3ENicdiMY/H44VuhkTI+NQM3+8bJp5IEg977SNhed7m2qpgCKa9kUMdjezaWktFucbXZe0xs253jy22TF0ViZzMaY7xnhTdvSlO9w8zNRN0aN6/ZSMfv3/rbLBvb9RpjhJ9CnpZ0zLDMPFEku4w2DN11ysrynigtZ5P/cg9HGwPzl9vrKkscItF7jwFvawp1yenOXlxaDbUjydSs8MwTRsrOdjewCcfbudAewN7W+p0hyQRFPRS5C4Pj8+OrR/vTXEmvNoUYOddG/n4/duIhb319s0bNAwjsggFvRSNmbTzxuURuhOpYHw9MXc2TPW6rKtN28OrTTfoalORXCjopWCuTUxzojc121s/0TvEtYlgGOauuipi7Y186kc6OdjewO5tdazT2TAit0RBL3eEu9M/NDavt/7G5RHSDmUG995dx0/ubyEW1obRDTVE8kdBL7fFyPgUZ/pHOHNpmBPhwdPLI+MA1FSWs7+tgRc+uoNYRwMPbt9EbbWGYURuFwW9rNrw2BRn+oc51T/M6UsjnO4f5u3B0dnlLZvW81Bn42xv/b67dVGSyJ2koJebMnR9ktP9I0Go9w9z+tLw7HnrEIT63pY6fvpAC3tb6tnbUk/TxqoCtlhEFPSypOToJKczPfXwuS81Nrt8e+N69m6r52di29kXhrouSBIpPgp6AYKbVJ/qH+Z0X9BLP90/MntqIwSleR/Yvomfe7g9DPU6Nm1QqIusBQr6EvTe1fGgh94XDMGcuTTMO8Pjs8s7m2o40N7AMx9sZ++2eva01OsOSSJrmII+wtydd0cmbhh+ee/qBABmQag/1NnIvpZ69myrZ09LHXU6A0YkUhT0ETI+NcPp/uHgXPVEipMXhxjICvX3NW/kkfc3sbelnn0t9ezeVsdG1VoXiTz9K1/DBq9N0J1IzT5O9Q0zORPc7q5j8wY+9P4m9rUGob5ra51uoCFSovQvf41Ip523Bq4Rn72yNElPphxveRl7W+p49pGO2TowzbU6pVFEAgr6IjU2OcPJi0Mc7w3uiHS8d4jhsSkAGmuCcrxHHmoj1t7A3pZ6qtepHK+ILE5BXyTeHRkPbnOXSHI8EZTjnU7P3RXpx/bePXvzjM6mGtWBEZGcKegLIFOO93h40DTeM78c7wOtm3ju0XtmSwbofHURWQ0F/R2wXDneLbVVxDoa+Gc/0klM5XjXJvfwMQOennuks19nLS+rgIoqKK+C8nXBKVEit1FOQW9mh4EvAeXAV939iwuWtwNHgWYgCXzS3fvCZc8AvxGu+m/d/U/z1PaidWlojK6eZDgUk+JcWI7XDO4Ly/FmhmGWLcc7PQmT14LHxDWYHIXJq1nT2csWrncNpsIrW60sDBOb/2xlN86DZZbZ0u+13OdgQcARBuLC58Xm4XMBma/tZgM3PT+UZwP5JpdlluOr+wtTXgUV1VBRGU5XBq/LK8MdQvh6drpqbkcxu81y2y8xr7wi2OlYefBclnmumHs9+2cqa5m5L/+X1MzKgTeBx4A+oAt4yt3PZq3zX4C/dfc/NbOPAr/o7j9vZo1AHIgR/GvoBg66e2qpz4vFYh6Px1f5tQrnRM8Av/yVv6WW62xeN8m+5gr2NJWxo8Foq0lTnR6bC+XJUZi4mjWdFeaTozAzmduHWhlU1kJlDVRtDJ4rN8K69cHyTEjlMzDxMN+WWrZgu5x2HMwtW3L9W91BWRBomc+zsqwgK8taFi7PedmC5WVlyy/Dgp3D9DjMTAQ785kJmA4fM5Ph9Pjc9Mxk8Hp23ckbt/f0rf6VXVl2+Fv54juEefPKl9mBLDZvXbDTKa+8iel1wbbllTcxHW6XmS4rj9ROzMy63T222LJcevQPAefd/UL4Zq8ATwJns9bZDfxqOP0t4G/C6R8FvuHuyXDbbwCHga/d7JcoOhPXYPBNGPxh+HwOBn/I/YPn+d9V03PrXQkf2corgyCu3BgGc/iovXtueja0swM8e5uauWXr1kfqL6zcgpnpBTuHHHYe6emsx0z4WDDPZxass3A6a54v8R7pmeAzF3vfmams56ngeWYq7OSs8n9KK7IFO4DKuR1BeWW4U6mYmy7Pns5sV7n4tsu95w3bZk1X1ULjPXn/prkEfQtwMet1H/DwgnW+B/wUwfDOTwK1ZrZ5iW1bFn6AmT0HPAfQ1taWa9tvP3e49m4Q5APnskL9TRjpn1vPyoM/nKad/LeJB3hjcgu//uTDiwRz+LpCB1clz8oroHxjoVuRX+mZG3cA6XAnMDOdw3RmJ7LU9NTcDi8zf3Ze1nRm/uR1mBnKWn/BNtntu1UtMfil/5m/3zCUr4Oxvw78oZk9C3wH6Admct3Y3V8GXoZg6CZPbcrdzDSk3p4L8YE353rrE8Nz61VuhKYd0PGh4Ln5XmjaCQ2dUFHJTNr53It/xyfu3wa79t3xryESKZkhIaoL3ZKb4774zmPeTiXcKWXvMNJTQY/+Nsgl6PuB7VmvW8N5s9z9EkGPHjPbCPy0uw+ZWT/w4QXbfnsV7V2diavze+WZUE9eCH7kjNqtQZDf/zNBkGdCvXbrskMkb757lavj08TaG+7AlxGRomQ2NyRTJHIJ+i5gh5l1EgT8EeDp7BXMrAlIunsa+BzBGTgArwP/zswyyfd4uPz2yQy3DJy7MdCvXspqdDjc0nwv3PcENIW986b3Q3X9LX10PBEcY451KOhFpHisGPTuPm1mzxOEdjlw1N3PmNmLQNzdjxH02n/HzJxg6ObT4bZJM/stgp0FwIuZA7N5d/UyvPJ0ONwyMje/sjbokXc+Cs07wzC/Fxo68j5W3t2TpGljFW2NG/L6viIiq5HTGL27vwa8tmDe57OmXwVeXWLbo8z18G+f9Y3B+Nb9PxuOne8IQn2F4ZZ8iidSxNobVJ5ARIpKdK6MraiEX/ivBfv4y8Pj9KXGePaDHQVrg4jIYnStfZ7EE8GIVKyjscAtERGZT0GfJ/GeFNXrytizra7QTRERmUdBnyfdiRQPtG5SQTIRKTpKpTwYnZjm7DsjOq1SRIqSgj4PvndxiJm0a3xeRIqSgj4P4okUZnCgTT16ESk+Cvo86OpJsnNLLfXri+eSZxGRDAX9Ks2knRO9QxzU+LyIFCkF/Sqdu3yVaxMqZCYixUtBv0rd4YVSh3QgVkSKlIJ+leKJFFtqq2htWF/opoiILEpBv0rxnhSxDhUyE5HipaBfhXeGx+gfGuNgu4ZtRKR4KehXId4T3mhEB2JFpIgp6FehO5Fi/bpydquQmYgUMQX9KsQTSR7crkJmIlLclFC36NrENGcvqZCZiBQ/Bf0tOtk7RNrhoMbnRaTIKehvUTyRDAqZKehFpMgp6G9RdyLFvXfVUletQmYiUtwU9LcgU8hM4/MishbkFPRmdtjMzpnZeTP77CLL28zsW2Z2wsy+b2ZPhPM7zGzMzE6Gjz/K9xcohDcuj4SFzHShlIgUv4qVVjCzcuAl4DGgD+gys2PufjZrtd8Avu7uXzaz3cBrQEe47C13fzC/zS6szIVSOhArImtBLj36h4Dz7n7B3SeBV4AnF6zjQOaqoXrgUv6aWHziiRR31amQmYisDbkEfQtwMet1Xzgv2xeAT5pZH0Fv/oWsZZ3hkM7/MrMPLfYBZvacmcXNLD4wMJB76wukuydJrL1RhcxEZE3I18HYp4A/cfdW4Angz82sDHgHaHP3/cCvAn9pZjfUC3D3l9095u6x5ubmPDXp9rg0NMal4XEdiBWRNSOXoO8Htme9bg3nZfsU8HUAd/8uUA00ufuEu18J53cDbwE7V9voQoonMoXMdCBWRNaGXIK+C9hhZp1mVgkcAY4tWKcX+BiAme0iCPoBM2sOD+ZiZvcAO4AL+Wp8IcR7kmyoLGfX1tpCN0VEJCcrnnXj7tNm9jzwOlAOHHX3M2b2IhB392PArwF/bGafITgw+6y7u5k9CrxoZlNAGvjn7p68bd/mDoj3pHhw+yYqVMhMRNaIFYMewN1fIzjImj3v81nTZ4FHFtnur4C/WmUbi8a1iWneuDzC8x95f6GbIiKSM3VLb8KJ3hRph5huBC4ia4iC/ibEe1KUGexv21TopoiI5ExBfxO6EynuvbuOWhUyE5E1REGfo+mZNMd7U7o/rIisOQr6HL1x+SrXJ2d0oZSIrDkK+hzFe4KzQnUgVkTWGgV9juKJFFvrq2nZpEJmIrK2KOhz4O7Ee1IqSywia5KCPgf9Q2NcHhnXgVgRWZMU9DnozhQy0/i8iKxBCvocxHtS1FSWc9/dKmQmImuPgj4H8USK/W0NKmQmImuSkmsFI+NTnLs8ogOxIrJmKehXcKJ3KCxkpqAXkbVJQb+C7p5kWMhMQS8ia5OCfgXxRIr77q5jY1VOpftFRIqOgn4Z0zNpTl4c4pCGbURkDVPQL+MH7wSFzA7q/HkRWcMU9MuIJ8JCZjrjRkTWMAX9MuI9KbbVV7NNhcxEZA1T0C/B3Yknkhq2EZE1L6egN7PDZnbOzM6b2WcXWd5mZt8ysxNm9n0zeyJr2efC7c6Z2Y/ms/G3U19qjHdHJnQgVkTWvBXPGTSzcuAl4DGgD+gys2PufjZrtd8Avu7uXzaz3cBrQEc4fQTYA2wD/oeZ7XT3mXx/kXzLFDLTFbEistbl0qN/CDjv7hfcfRJ4BXhywToO1IXT9cClcPpJ4BV3n3D3t4Hz4fsVvXgiycaqCu67u27llUVEilguQd8CXMx63RfOy/YF4JNm1kfQm3/hJrbFzJ4zs7iZxQcGBnJs+u0V70mxv20T5WVW6KaIiKxKvg7GPgX8ibu3Ak8Af25mOb+3u7/s7jF3jzU3N+epSbdueGyKc+9e1bCNiERCLtf19wPbs163hvOyfQo4DODu3zWzaqApx22LzoneFO5wSGfciEgE5NLr7gJ2mFmnmVUSHFw9tmCdXuBjAGa2C6gGBsL1jphZlZl1AjuA/5evxt8u3YkU5WXGg9s3FbopIiKrtmKP3t2nzex54HWgHDjq7mfM7EUg7u7HgF8D/tjMPkNwYPZZd3fgjJl9HTgLTAOfXgtn3MR7UuzaWkuNCpmJSATklGTu/hrBQdbseZ/Pmj4LPLLEtr8N/PYq2nhHTc2kOXExxZFDbYVuiohIXujK2AXOXhphfCqtA7EiEhkK+gXi4YVSuqOUiESFgn6B7kSSlk3r2VqvQmYiEg0K+izuTrwnpd68iESKgj5LX2qM965OqP68iESKgj5LV09wo5GD7bpQSkSiQ0GfJZ5IUVtVwb131xa6KSIieaOgz9Ldk2J/e4MKmYlIpCjoQ8NjU7z53lWNz4tI5CjoQ8fDQmYKehGJGgV9KN6TDAqZtamQmYhEi4I+FO9JsXtrHRsqVchMRKJFQU9QyOx7fUO6UEpEIklBD5wJC5nFdP68iESQgp5gfB5UyExEoklBTzA+39qwnrvqqgvdFBGRvCv5oHd34omUTqsUkcgq+aDvTV5n8NoEMd0IXEQiquSDPt6jG42ISLQp6BMpaqsr2LlFhcxEJJpKPui7E0kOtDVQpkJmIhJRJR30Q9cnefPdazoQKyKRllPQm9lhMztnZufN7LOLLP99MzsZPt40s6GsZTNZy47ls/Grdbw3Mz6vA7EiEl0rFnYxs3LgJeAxoA/oMrNj7n42s467fyZr/ReA/VlvMebuD+avyfkT70lRUWY8uF2FzEQkunLp0T8EnHf3C+4+CbwCPLnM+k8BX8tH4263eCLFnm11rK8sL3RTRERum1yCvgW4mPW6L5x3AzNrBzqBb2bNrjazuJn9vZn9xBLbPReuEx8YGMix6aszOZ3mexeHdH9YEYm8fB+MPQK86u4zWfPa3T0GPA38gZm9b+FG7v6yu8fcPdbc3JznJi3u9KVhJqbTOn9eRCIvl6DvB7ZnvW4N5y3mCAuGbdy9P3y+AHyb+eP3BdOduVBKZ9yISMTlEvRdwA4z6zSzSoIwv+HsGTO7D2gAvps1r8HMqsLpJuAR4OzCbQshnkjS1riBLSpkJiIRt+JZN+4+bWbPA68D5cBRdz9jZi8CcXfPhP4R4BV396zNdwFfMbM0wU7li9ln6xSKu9OdSPHojjszTCQiUkg53TfP3V8DXlsw7/MLXn9hke3+L7BvFe27LRJXrjN4bZKDGp8XkRJQklfGdmVuNKIzbkSkBJRk0HcnUtRVV7Bjy8ZCN0VE5LYryaCPJ1IcbFchMxEpDSUX9EPXJzn/3jXVtxGRklFyQd+dCM6fP6jz50WkRJRc0McTQSGzB1pVyExESkPpBX1Pkj0t9SpkJiIlo6SCfmJ6hu/1DXNIwzYiUkJKKuhP948wqUJmIlJiSirouxPBhVIqTSwipaSkgj7ek6J98waaa6sK3RQRkTumZII+U8hMp1WKSKkpmaB/e3CUK6OTHNKFUiJSYkom6OMJ3WhEREpTyQR9d0+K+vXreF+zCpmJSGkpmaCPJ5IqZCYiJakkgj45OslbA6M6ECsiJakkgj5TyEwHYkWkFJVE0McTSdaVG/e31he6KSIid1xJBH13T4q9LfVUr1MhMxEpPZEP+onpGb7fP6zTKkWkZOUU9GZ22MzOmdl5M/vsIst/38xOho83zWwoa9kzZvbD8PFMPhufi9P9w0xOp1XfRkRKVsVKK5hZOfAS8BjQB3SZ2TF3P5tZx90/k7X+C8D+cLoR+E0gBjjQHW6byuu3WEZXT3ihlCpWikiJyqVH/xBw3t0vuPsk8Arw5DLrPwV8LZz+UeAb7p4Mw/0bwOHVNPhmxXtSdDbV0LRRhcxEpDTlEvQtwMWs133hvBuYWTvQCXzzZrY1s+fMLG5m8YGBgVzanRN353ivCpmJSGnL98HYI8Cr7j5zMxu5+8vuHnP3WHNzc94ac2FwlOTopA7EikhJyyXo+4HtWa9bw3mLOcLcsM3Nbpt33RqfFxHJKei7gB1m1mlmlQRhfmzhSmZ2H9AAfDdr9uvA42bWYGYNwOPhvDuiqydJwwYVMhOR0rbiWTfuPm1mzxMEdDlw1N3PmNmLQNzdM6F/BHjF3T1r26SZ/RbBzgLgRXdP5vcrLC1zoxEzFTITkdK1YtADuPtrwGsL5n1+wesvLLHtUeDoLbbvll25NsGFwVH+aWz7yiuLiERYZK+MzRQy0/i8iJS6SAd9ZXkZ+1pUyExESltkgz6eSLG3pU6FzESk5EUy6MenZjjVN6z68yIiRDToT/UPMzmT1hWxIiJENOjj4YVSCnoRkYgGfXciyT1NNWxWITMRkegFfTrtsxdKiYhIBIP+wuA1UtendCBWRCQUuaCfHZ/XhVIiIkAUgz6RorGmknuaagrdFBGRohC5oO9OpDjQpkJmIiIZkQr6wWsTvD04qvo2IiJZIhX0mfH5Qwp6EZFZkQr67kSSyooy9qqQmYjIrEgFfTyR4v6WeqoqVMhMRCQjMkE/PjXD6f5hnVYpIrJAZIL+6vg0T+zbyqM7mgvdFBGRopLTrQTXgubaKr50ZH+hmyEiUnQi06MXEZHFKehFRCJOQS8iEnE5Bb2ZHTazc2Z23sw+u8Q6P2NmZ83sjJn9Zdb8GTM7GT6O5avhIiKSmxUPxppZOfAS8BjQB3SZ2TF3P5u1zg7gc8Aj7p4ysy1ZbzHm7g/mud0iIpKjXHr0DwHn3f2Cu08CrwBPLljnl4CX3D0F4O7v5beZIiJyq3IJ+hbgYtbrvnBetp3ATjP7P2b292Z2OGtZtZnFw/k/sdgHmNlz4TrxgYGBm/oCIiKyvHydR18B7AA+DLQC3zGzfe4+BLS7e7+Z3QN808xOuftb2Ru7+8vAywCxWMzz1CYRESG3oO8Htme9bg3nZesD/sHdp4C3zexNguDvcvd+AHe/YGbfBvYDb7GE7u7uQTNL5P4VbtAEDK5i+yjRbzGffo/59HvMicJv0b7UAnNfvgNtZhXAm8DHCAK+C3ja3c9krXMYeMrdnzGzJuAE8CCQBq67+0Q4/7vAk9kHcvPNzOLuHrtd77+W6LeYT7/HfPo95kT9t1ixR+/u02b2PPA6UA4cdfczZvYiEHf3Y+Gyx83sLDAD/Ct3v2JmHwS+YmZpguMBX7ydIS8iIjdasUe/1kR9z3wz9FvMp99jPv0ec6L+W0TxytiXC92AIqLfYj79HvPp95gT6d8icj16ERGZL+Wi5mEAAAJXSURBVIo9ehERyaKgFxGJuMgEfS6F10qFmW03s29lFZn7lUK3qdDMrNzMTpjZ3xa6LYVmZpvM7FUze8PMfmBm/6jQbSokM/tM+O/ktJl9zcyqC92mfItE0GcVXvsxYDfwlJntLmyrCmoa+DV33w18APh0if8eAL8C/KDQjSgSXwL+u7vfBzxACf8uZtYC/DIQc/e9BKeQHylsq/IvEkFPboXXSoa7v+Pux8PpqwT/kBfWJyoZZtYKfBz4aqHbUmhmVg88CvxHAHefDEuVlLIKYH14cegG4FKB25N3UQn6XAqvlSQz6yAoO/EPhW1JQf0B8K8JrtQudZ3AAPCfwqGsr5pZTaEbVShhiZbfA3qBd4Bhd/+7wrYq/6IS9LIIM9sI/BXwL919pNDtKQQz+wTwnrt3F7otRaICOAB82d33A6NAyR7TMrMGgv/9dwLbgBoz+2RhW5V/UQn6XAqvlRQzW0cQ8n/h7n9d6PYU0CPAj5tZD8GQ3kfN7D8XtkkF1Qf0uXvmf3ivEgR/qfonwNvuPhAWZfxr4IMFblPeRSXou4AdZtZpZpUEB1NK9raFZmYEY7A/cPd/X+j2FJK7f87dW929g+DvxTfdPXI9tly5+2XgopndG876GFDK9ad6gQ+Y2Ybw383HiODB6XzVoy+opQqvFbhZhfQI8PPAKTM7Gc77N+7+WgHbJMXjBeAvwk7RBeAXC9yegnH3fzCzV4HjBGernSCC5RBUAkFEJOKiMnQjIiJLUNCLiEScgl5EJOIU9CIiEaegFxGJOAW9iEjEKehFRCLu/wNM+yht7mf6QQAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f_0DQdSkNlyE",
        "colab_type": "text"
      },
      "source": [
        "Дээрх графикаас харахад манай модель дөнгөж эхлээд л overfit болсон харагдаж байна.\n",
        "\n",
        "Энд байга код нь ойлгоход амар байлгах үүднээс overfit-ийг багасгах ямар нэг арга хэрэглээгүй болно.\n",
        "\n",
        "Та өөрийн drive дээрээ хуулаад дуртайгаараа тоглох боломжтой байгаа."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZUuG1rQx35YN",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# хадгалах\n",
        "file_path = \"albert-mongolian-finetuned.hdf5\"\n",
        "model.save_weights(file_path)"
      ],
      "execution_count": 149,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5H9OXMwT5C_M",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "3df5e754-7374-4d89-f2e5-e05936a1396f"
      },
      "source": [
        "del model\n",
        "gc.collect()"
      ],
      "execution_count": 150,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "30219"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 150
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FmJPTVHI4E6g",
        "colab_type": "text"
      },
      "source": [
        "## Үр дүн\n",
        "\n",
        "Одоо сургасан моделоо ашиглаад тест дата дээр хэрхэн таамаглал хийж байгааг харцгаая!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9yDU8gfc4Emz",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 139
        },
        "outputId": "3acb1f24-cbfe-4ddf-d70f-99b0a3f823e5"
      },
      "source": [
        "trained_model = build_model()\n",
        "trained_model.load_weights(file_path)"
      ],
      "execution_count": 151,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:transformers.modeling_tf_utils:Some weights of the model checkpoint at bayartsogt/albert-mongolian were not used when initializing TFAlbertModel: ['predictions']\n",
            "- This IS expected if you are initializing TFAlbertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).\n",
            "- This IS NOT expected if you are initializing TFAlbertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "WARNING:transformers.modeling_tf_utils:All the weights of TFAlbertModel were initialized from the model checkpoint at bayartsogt/albert-mongolian.\n",
            "If your task is similar to the task the model of the ckeckpoint was trained on, you can already use TFAlbertModel for predictions without further training.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Loss: sparse_categorical_crossentropy\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dv-C9Ef23mva",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "85ae8d81-df24-4915-863d-af3bdd8e0f24"
      },
      "source": [
        "prediction = trained_model.predict(x=[X_test_encoded[\"input_ids\"], X_test_encoded[\"attention_mask\"], X_test_encoded[\"token_type_ids\"]], batch_size=16, verbose=1)"
      ],
      "execution_count": 152,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "15133/15133 [==============================] - 68s 4ms/sample\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zLT72SJB5T6j",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 289
        },
        "outputId": "f024719c-0f39-4b5f-a8b6-a2dcfd50986e"
      },
      "source": [
        "# prediction\n",
        "from sklearn.metrics import classification_report\n",
        "\n",
        "if len(label2id) == 2:\n",
        "    threshold = 0.5\n",
        "    y_pred = np.where(prediction>threshold, 1, 0)\n",
        "else:\n",
        "    y_pred = np.argmax(prediction, 1)\n",
        "print(classification_report(np.array(y_test), y_pred))"
      ],
      "execution_count": 153,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.93      0.94      0.94      1457\n",
            "           1       0.83      0.88      0.86      2509\n",
            "           2       0.89      0.92      0.90      1159\n",
            "           3       0.89      0.87      0.88      1651\n",
            "           4       0.92      0.85      0.89      2647\n",
            "           5       0.98      0.98      0.98      2736\n",
            "           6       0.88      0.93      0.91      1102\n",
            "           7       0.80      0.80      0.80       873\n",
            "           8       0.85      0.83      0.84       999\n",
            "\n",
            "    accuracy                           0.90     15133\n",
            "   macro avg       0.89      0.89      0.89     15133\n",
            "weighted avg       0.90      0.90      0.90     15133\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JlUNJXGVOA9s",
        "colab_type": "text"
      },
      "source": [
        "Модель ~90%-ийн нарийвчлалтай ажиллаж байна. Энэхүү үр дүн нь бага зэргийн өөрчлөлтөөр сайжрах бүрэн боломжтой юм.\n",
        "\n",
        "Одоогоор хамгийн сайн F1 score той байгаа ангилалууд нь `спорт, урлаг соёл, технологи` байгаа бол хамгийн муу нь `боловсрол, байгаль орчин, эдийн засаг` байна.\n",
        "\n",
        "Энэхүү notebook-ээр та бүхэнд albert-mongolian-ийг хэрхэн fine-tuning хийх буюу өөр төрлийн асуудалд ашиглах талаар харууллаа.\n",
        "\n",
        "Танд ямар асуух, лавлах зүйл байгаа бол доорх хаягаар холбогдох боломжтой юм.\n",
        "\n",
        "<h1 style=\"text-align:center\">Анхаарал тавьсан баярлалаа! Enjoy!<h1>\n",
        "\n",
        "|contact|url|\n",
        "|--|--|\n",
        "|E-mail  |mailto:bayartsogt@mn.data-artist.com|\n",
        "|Github  |https://github.com/bayartsogt-ya|\n",
        "|Website |https://bayartsogt-ya.github.io|\n",
        "|LinkedIn|https://www.linkedin.com/in/bayartsogt-yadamsuren-b419b217b/|\n"
      ]
    }
  ]
}