Assignment/8_Seq2Seq/hw8_seq2seq.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "hw8_seq2seq.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WdkAyb0Rs6bW",
        "colab_type": "text"
      },
      "source": [
        "#**Homework 8 - Sequence-to-sequence**\n",
        "\n",
        "若有任何問題，歡迎來信至助教信箱 ntu-ml-2020spring-ta@googlegroups.com"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HJhsqjBlIiP0",
        "colab_type": "text"
      },
      "source": [
        "# Sequence-to-Sequence 介紹\n",
        "- 大多數常見的 **sequence-to-sequence (seq2seq) model** 為 **encoder-decoder model**，主要由兩個部分組成，分別是 **Encoder** 和 **Decoder**，而這兩個部分則大多使用 **recurrent neural network (RNN)** 來實作，主要是用來解決輸入和輸出的長度不一樣的情況\n",
        "- **Encoder** 是將**一連串**的輸入，如文字、影片、聲音訊號等，編碼為**單個向量**，這單個向量可以想像為是整個輸入的抽象表示，包含了整個輸入的資訊\n",
        "- **Decoder** 是將 Encoder 輸出的單個向量逐步解碼，**一次輸出一個結果**，直到將最後目標輸出被產生出來為止，每次輸出會影響下一次的輸出，一般會在開頭加入 \"< BOS >\" 來表示開始解碼，會在結尾輸出 \"< EOS >\" 來表示輸出結束\n",
        "\n",
        "\n",
        "![seq2seq](https://i.imgur.com/0zeDyuI.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jLpvURxG_xPa",
        "colab_type": "text"
      },
      "source": [
        "# 作業介紹\n",
        "- 英文翻譯中文\n",
        "  - 輸入： 一句英文 （e.g.\t\ttom is a student .） \n",
        "  - 輸出： 中文翻譯 （e.g. \t\t湯姆 是 個 學生 。）\n",
        "\n",
        "- TODO\n",
        "  - Teachering Forcing 的功用: 嘗試不用 Teachering Forcing 做訓練\n",
        "  - 實作 Attention Mechanism\n",
        "  - 實作 Beam Search\n",
        "  - 實作 Schedule Sampling"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yfFk6r-PujeH",
        "colab_type": "text"
      },
      "source": [
        "# 資料下載"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ww4-VJoJqE-_",
        "colab_type": "code",
        "outputId": "5114604a-0e82-458b-b811-bfe5fe31657b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 442
        }
      },
      "source": [
        "!gdown --id '1r4px0i-NcrnXy1-tkBsIwvYwbWnxAhcg' --output data.tar.gz\n",
        "!tar -zxvf data.tar.gz\n",
        "!mkdir ckpt\n",
        "!ls"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading...\n",
            "From: https://drive.google.com/uc?id=1r4px0i-NcrnXy1-tkBsIwvYwbWnxAhcg\n",
            "To: /content/data.tar.gz\n",
            "5.83MB [00:00, 27.2MB/s]\n",
            "cmn-eng/\n",
            "cmn-eng/int2word_cn.json\n",
            "cmn-eng/int2word_en.json\n",
            "cmn-eng/preprocess/\n",
            "cmn-eng/preprocess/build_dataset.py\n",
            "cmn-eng/preprocess/build_dictionary.sh\n",
            "cmn-eng/preprocess/cmn.txt\n",
            "cmn-eng/preprocess/cn.txt\n",
            "cmn-eng/preprocess/dict.txt.big\n",
            "cmn-eng/preprocess/dict.txt.small\n",
            "cmn-eng/preprocess/en.txt\n",
            "cmn-eng/preprocess/en_code.txt\n",
            "cmn-eng/preprocess/en_refine.txt\n",
            "cmn-eng/preprocess/en_vocab.txt\n",
            "cmn-eng/preprocess/tokenizer.py\n",
            "cmn-eng/testing.txt\n",
            "cmn-eng/training.txt\n",
            "cmn-eng/validation.txt\n",
            "cmn-eng/word2int_cn.json\n",
            "cmn-eng/word2int_en.json\n",
            "ckpt  cmn-eng  data.tar.gz  sample_data\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BX2vLCiYuq6a",
        "colab_type": "text"
      },
      "source": [
        "# 下載和引入需要的 libraries"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KKShudDCiySL",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%%capture\n",
        "!pip3 install --user nltk"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "skJCdcXBi5b5",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%%capture\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.optim as optim\n",
        "import torch.nn.functional as F\n",
        "import torch.utils.data as data\n",
        "import torch.utils.data.sampler as sampler\n",
        "import torchvision\n",
        "from torchvision import datasets, transforms\n",
        "\n",
        "import numpy as np\n",
        "import sys\n",
        "import os\n",
        "import random\n",
        "import json\n",
        "\n",
        "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\") # 判斷是用 CPU 還是 GPU 執行運算\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U-Irl7IluxMT",
        "colab_type": "text"
      },
      "source": [
        "# 資料結構"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VJb3zc-zwkd6",
        "colab_type": "text"
      },
      "source": [
        "## 定義資料的轉換\n",
        "- 將不同長度的答案拓展到相同長度，以便訓練模型"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "42Iggb94rKcI",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import numpy as np\n",
        "\n",
        "class LabelTransform(object):\n",
        "  def __init__(self, size, pad):\n",
        "    self.size = size\n",
        "    self.pad = pad\n",
        "\n",
        "  def __call__(self, label):\n",
        "    label = np.pad(label, (0, (self.size - label.shape[0])), mode='constant', constant_values=self.pad)\n",
        "    return label\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "42zjTpver4Sk",
        "colab_type": "text"
      },
      "source": [
        "## 定義 Dataset\n",
        "- Data (出自manythings 的 cmn-eng):\n",
        "  - 訓練資料：18000句\n",
        "  - 檢驗資料：  500句\n",
        "  - 測試資料： 2636句\n",
        "\n",
        "- 資料預處理:\n",
        "  - 英文：\n",
        "    - 用 subword-nmt 套件將word轉為subword\n",
        "    - 建立字典：取出標籤中出現頻率高於定值的subword\n",
        "  - 中文：\n",
        "    - 用 jieba 將中文句子斷詞\n",
        "    - 建立字典：取出標籤中出現頻率高於定值的詞\n",
        "  - 特殊字元： < PAD >, < BOS >, < EOS >, < UNK > \n",
        "    - < PAD >  ：無意義，將句子拓展到相同長度\n",
        "    - < BOS >  ：Begin of sentence, 開始字元\n",
        "    - < EOS >  ：End of sentence, 結尾字元\n",
        "    - < UNK > ：單字沒有出現在字典裡的字\n",
        "  - 將字典裡每個 subword (詞) 用一個整數表示，分為英文和中文的字典，方便之後轉為 one-hot vector   \n",
        "\n",
        "- 處理後的檔案:\n",
        "  - 字典：\n",
        "    - int2word_*.json: 將整數轉為文字\n",
        "    ![int2word_en.json](https://i.imgur.com/31E4MdZ.png)\n",
        "    - word2int_*.json: 將文字轉為整數\n",
        "    ![word2int_en.json](https://i.imgur.com/9vI4AS1.png)\n",
        "    - $*$ 分為英文（en）和中文（cn）\n",
        "  \n",
        "  - 訓練資料:\n",
        "    - 不同語言的句子用 TAB ('\\t') 分開\n",
        "    - 字跟字之間用空白分開\n",
        "    ![data](https://i.imgur.com/nSH1fH4.png)\n",
        "\n",
        "\n",
        "- 在將答案傳出去前，在答案開頭加入 \"< BOS >\" 符號，並於答案結尾加入 \"< EOS >\" 符號"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8uypors6r4zD",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import re\n",
        "import json\n",
        "\n",
        "class EN2CNDataset(data.Dataset):\n",
        "  def __init__(self, root, max_output_len, set_name):\n",
        "    self.root = root\n",
        "\n",
        "    self.word2int_cn, self.int2word_cn = self.get_dictionary('cn')\n",
        "    self.word2int_en, self.int2word_en = self.get_dictionary('en')\n",
        "\n",
        "    # 載入資料\n",
        "    self.data = []\n",
        "    with open(os.path.join(self.root, f'{set_name}.txt'), \"r\") as f:\n",
        "      for line in f:\n",
        "        self.data.append(line)\n",
        "    print (f'{set_name} dataset size: {len(self.data)}')\n",
        "\n",
        "    self.cn_vocab_size = len(self.word2int_cn)\n",
        "    self.en_vocab_size = len(self.word2int_en)\n",
        "    self.transform = LabelTransform(max_output_len, self.word2int_en['<PAD>'])\n",
        "\n",
        "  def get_dictionary(self, language):\n",
        "    # 載入字典\n",
        "    with open(os.path.join(self.root, f'word2int_{language}.json'), \"r\") as f:\n",
        "      word2int = json.load(f)\n",
        "    with open(os.path.join(self.root, f'int2word_{language}.json'), \"r\") as f:\n",
        "      int2word = json.load(f)\n",
        "    return word2int, int2word\n",
        "\n",
        "  def __len__(self):\n",
        "    return len(self.data)\n",
        "\n",
        "  def __getitem__(self, Index):\n",
        "    # 先將中英文分開\n",
        "    sentences = self.data[Index]\n",
        "    sentences = re.split('[\\t\\n]', sentences)\n",
        "    sentences = list(filter(None, sentences))\n",
        "    #print (sentences)\n",
        "    assert len(sentences) == 2\n",
        "\n",
        "    # 預備特殊字元\n",
        "    BOS = self.word2int_en['<BOS>']\n",
        "    EOS = self.word2int_en['<EOS>']\n",
        "    UNK = self.word2int_en['<UNK>']\n",
        "\n",
        "    # 在開頭添加 <BOS>，在結尾添加 <EOS> ，不在字典的 subword (詞) 用 <UNK> 取代\n",
        "    en, cn = [BOS], [BOS]\n",
        "    # 將句子拆解為 subword 並轉為整數\n",
        "    sentence = re.split(' ', sentences[0])\n",
        "    sentence = list(filter(None, sentence))\n",
        "    #print (f'en: {sentence}')\n",
        "    for word in sentence:\n",
        "      en.append(self.word2int_en.get(word, UNK))\n",
        "    en.append(EOS)\n",
        "\n",
        "    # 將句子拆解為單詞並轉為整數\n",
        "    # e.g. < BOS >, we, are, friends, < EOS > --> 1, 28, 29, 205, 2\n",
        "    sentence = re.split(' ', sentences[1])\n",
        "    sentence = list(filter(None, sentence))\n",
        "    #print (f'cn: {sentence}')\n",
        "    for word in sentence:\n",
        "      cn.append(self.word2int_cn.get(word, UNK))\n",
        "    cn.append(EOS)\n",
        "\n",
        "    en, cn = np.asarray(en), np.asarray(cn)\n",
        "\n",
        "    # 用 <PAD> 將句子補到相同長度\n",
        "    en, cn = self.transform(en), self.transform(cn)\n",
        "    en, cn = torch.LongTensor(en), torch.LongTensor(cn)\n",
        "\n",
        "    return en, cn\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WJIsNSRCyNA9",
        "colab_type": "text"
      },
      "source": [
        "# 模型架構"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3p1ZcU5HyO73",
        "colab_type": "text"
      },
      "source": [
        "## Encoder\n",
        "- seq2seq模型的編碼器為RNN。 對於每個輸入，，**Encoder** 會輸出**一個向量**和**一個隱藏狀態(hidden state)**，並將隱藏狀態用於下一個輸入，換句話說，**Encoder** 會逐步讀取輸入序列，並輸出單個矢量（最終隱藏狀態）\n",
        "- 參數:\n",
        "  - en_vocab_size 是英文字典的大小，也就是英文的 subword 的個數\n",
        "  - emb_dim 是 embedding 的維度，主要將 one-hot vector 的單詞向量壓縮到指定的維度，主要是為了降維和濃縮資訊的功用，可以使用預先訓練好的 word embedding，如 Glove 和 word2vector\n",
        "  - hid_dim 是 RNN 輸出和隱藏狀態的維度\n",
        "  - n_layers 是 RNN 要疊多少層\n",
        "  - dropout 是決定有多少的機率會將某個節點變為 0，主要是為了防止 overfitting ，一般來說是在訓練時使用，測試時則不使用\n",
        "- Encoder 的輸入和輸出:\n",
        "  - 輸入: \n",
        "    - 英文的整數序列 e.g. 1, 28, 29, 205, 2\n",
        "  - 輸出: \n",
        "    - outputs: 最上層 RNN 全部的輸出，可以用 Attention 再進行處理\n",
        "    - hidden: 每層最後的隱藏狀態，將傳遞到 Decoder 進行解碼\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "l6uNIKqvgb9J",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class Encoder(nn.Module):\n",
        "  def __init__(self, en_vocab_size, emb_dim, hid_dim, n_layers, dropout):\n",
        "    super().__init__()\n",
        "    self.embedding = nn.Embedding(en_vocab_size, emb_dim)\n",
        "    self.hid_dim = hid_dim\n",
        "    self.n_layers = n_layers\n",
        "    self.rnn = nn.GRU(emb_dim, hid_dim, n_layers, dropout=dropout, batch_first=True, bidirectional=True)\n",
        "    self.dropout = nn.Dropout(dropout)\n",
        "\n",
        "  def forward(self, input):\n",
        "    # input = [batch size, sequence len, vocab size]\n",
        "    embedding = self.embedding(input)\n",
        "    outputs, hidden = self.rnn(self.dropout(embedding))\n",
        "    # outputs = [batch size, sequence len, hid dim * directions]\n",
        "    # hidden =  [num_layers * directions, batch size  , hid dim]\n",
        "    # outputs 是最上層RNN的輸出\n",
        "        \n",
        "    return outputs, hidden\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5maDs7wpyRE3",
        "colab_type": "text"
      },
      "source": [
        "## Decoder\n",
        "- **Decoder** 是另一個 RNN，在最簡單的 seq2seq decoder 中，僅使用 **Encoder** 每一層最後的隱藏狀態來進行解碼，而這最後的隱藏狀態有時被稱為 “content vector”，因為可以想像它對整個前文序列進行編碼， 此 “content vector” 用作 **Decoder** 的**初始**隱藏狀態， 而 **Encoder** 的輸出通常用於 Attention Mechanism\n",
        "- 參數\n",
        "  - en_vocab_size 是英文字典的大小，也就是英文的 subword 的個數\n",
        "  - emb_dim 是 embedding 的維度，是用來將 one-hot vector 的單詞向量壓縮到指定的維度，主要是為了降維和濃縮資訊的功用，可以使用預先訓練好的 word embedding，如 Glove 和 word2vector\n",
        "  - hid_dim 是 RNN 輸出和隱藏狀態的維度\n",
        "  - output_dim 是最終輸出的維度，一般來說是將 hid_dim 轉到 one-hot vector 的單詞向量\n",
        "  - n_layers 是 RNN 要疊多少層\n",
        "  - dropout 是決定有多少的機率會將某個節點變為0，主要是為了防止 overfitting ，一般來說是在訓練時使用，測試時則不用\n",
        "  - isatt 是來決定是否使用 Attention Mechanism\n",
        "\n",
        "- Decoder 的輸入和輸出:\n",
        "  - 輸入:\n",
        "    - 前一次解碼出來的單詞的整數表示\n",
        "  - 輸出:\n",
        "    - hidden: 根據輸入和前一次的隱藏狀態，現在的隱藏狀態更新的結果\n",
        "    - output: 每個字有多少機率是這次解碼的結果"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QTXRCkoug3ut",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class Decoder(nn.Module):\n",
        "  def __init__(self, cn_vocab_size, emb_dim, hid_dim, n_layers, dropout, isatt):\n",
        "    super().__init__()\n",
        "    self.cn_vocab_size = cn_vocab_size\n",
        "    self.hid_dim = hid_dim * 2\n",
        "    self.n_layers = n_layers\n",
        "    self.embedding = nn.Embedding(cn_vocab_size, config.emb_dim)\n",
        "    self.isatt = isatt\n",
        "    self.attention = Attention(hid_dim)\n",
        "    # 如果使用 Attention Mechanism 會使得輸入維度變化，請在這裡修改\n",
        "    # e.g. Attention 接在輸入後面會使得維度變化，所以輸入維度改為\n",
        "    # self.input_dim = emb_dim + hid_dim * 2 if isatt else emb_dim\n",
        "    self.input_dim = emb_dim\n",
        "    self.rnn = nn.GRU(self.input_dim, self.hid_dim, self.n_layers, dropout = dropout, batch_first=True)\n",
        "    self.embedding2vocab1 = nn.Linear(self.hid_dim, self.hid_dim * 2)\n",
        "    self.embedding2vocab2 = nn.Linear(self.hid_dim * 2, self.hid_dim * 4)\n",
        "    self.embedding2vocab3 = nn.Linear(self.hid_dim * 4, self.cn_vocab_size)\n",
        "    self.dropout = nn.Dropout(dropout)\n",
        "\n",
        "  def forward(self, input, hidden, encoder_outputs):\n",
        "    # input = [batch size, vocab size]\n",
        "    # hidden = [batch size, n layers * directions, hid dim]\n",
        "    # Decoder 只會是單向，所以 directions=1\n",
        "    input = input.unsqueeze(1)\n",
        "    embedded = self.dropout(self.embedding(input))\n",
        "    # embedded = [batch size, 1, emb dim]\n",
        "    if self.isatt:\n",
        "      attn = self.attention(encoder_outputs, hidden)\n",
        "      # TODO: 在這裡決定如何使用 Attention，e.g. 相加 或是 接在後面， 請注意維度變化\n",
        "    output, hidden = self.rnn(embedded, hidden)\n",
        "    # output = [batch size, 1, hid dim]\n",
        "    # hidden = [num_layers, batch size, hid dim]\n",
        "\n",
        "    # 將 RNN 的輸出轉為每個詞出現的機率\n",
        "    output = self.embedding2vocab1(output.squeeze(1))\n",
        "    output = self.embedding2vocab2(output)\n",
        "    prediction = self.embedding2vocab3(output)\n",
        "    # prediction = [batch size, vocab size]\n",
        "    return prediction, hidden\n",
        "\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "faboaaSJAcpE",
        "colab_type": "text"
      },
      "source": [
        "## Attention\n",
        "- 當輸入過長，或是單獨靠 “content vector” 無法取得整個輸入的意思時，用 Attention Mechanism 來提供 **Decoder** 更多的資訊\n",
        "- 主要是根據現在 **Decoder hidden state** ，去計算在 **Encoder outputs** 中，那些與其有較高的關係，根據關系的數值來決定該傳給 **Decoder** 那些額外資訊 \n",
        "- 常見 Attention 的實作是用 Neural Network / Dot Product 來算 **Decoder hidden state** 和 **Encoder outputs** 之間的關係，再對所有算出來的數值做 **softmax** ，最後根據過完 **softmax** 的值對 **Encoder outputs** 做 **weight sum**\n",
        "\n",
        "- TODO:\n",
        "實作 Attention Mechanism"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "imGefKRIAfBW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class Attention(nn.Module):\n",
        "  def __init__(self, hid_dim):\n",
        "    super(Attention, self).__init__()\n",
        "    self.hid_dim = hid_dim\n",
        "  \n",
        "  def forward(self, encoder_outputs, decoder_hidden):\n",
        "    # encoder_outputs = [batch size, sequence len, hid dim * directions]\n",
        "    # decoder_hidden = [num_layers, batch size, hid dim]\n",
        "    # 一般來說是取 Encoder 最後一層的 hidden state 來做 attention\n",
        "    ########\n",
        "    # TODO #\n",
        "    ########\n",
        "    attention=None\n",
        "    \n",
        "    return attention\n",
        "    "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uJjxd56yySun",
        "colab_type": "text"
      },
      "source": [
        "## Seq2Seq\n",
        "- 由 **Encoder** 和 **Decoder** 組成\n",
        "- 接收輸入並傳給 **Encoder** \n",
        "- 將 **Encoder** 的輸出傳給 **Decoder**\n",
        "- 不斷地將 **Decoder** 的輸出傳回 **Decoder** ，進行解碼  \n",
        "- 當解碼完成後，將 **Decoder** 的輸出傳回 "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jjko57senVKG",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class Seq2Seq(nn.Module):\n",
        "  def __init__(self, encoder, decoder, device):\n",
        "    super().__init__()\n",
        "    self.encoder = encoder\n",
        "    self.decoder = decoder\n",
        "    self.device = device\n",
        "    assert encoder.n_layers == decoder.n_layers, \\\n",
        "            \"Encoder and decoder must have equal number of layers!\"\n",
        "            \n",
        "  def forward(self, input, target, teacher_forcing_ratio):\n",
        "    # input  = [batch size, input len, vocab size]\n",
        "    # target = [batch size, target len, vocab size]\n",
        "    # teacher_forcing_ratio 是有多少機率使用正確答案來訓練\n",
        "    batch_size = target.shape[0]\n",
        "    target_len = target.shape[1]\n",
        "    vocab_size = self.decoder.cn_vocab_size\n",
        "\n",
        "    # 準備一個儲存空間來儲存輸出\n",
        "    outputs = torch.zeros(batch_size, target_len, vocab_size).to(self.device)\n",
        "    # 將輸入放入 Encoder\n",
        "    encoder_outputs, hidden = self.encoder(input)\n",
        "    # Encoder 最後的隱藏層(hidden state) 用來初始化 Decoder\n",
        "    # encoder_outputs 主要是使用在 Attention\n",
        "    # 因為 Encoder 是雙向的RNN，所以需要將同一層兩個方向的 hidden state 接在一起\n",
        "    # hidden =  [num_layers * directions, batch size  , hid dim]  --> [num_layers, directions, batch size  , hid dim]\n",
        "    hidden = hidden.view(self.encoder.n_layers, 2, batch_size, -1)\n",
        "    hidden = torch.cat((hidden[:, -2, :, :], hidden[:, -1, :, :]), dim=2)\n",
        "    # 取的 <BOS> token\n",
        "    input = target[:, 0]\n",
        "    preds = []\n",
        "    for t in range(1, target_len):\n",
        "      output, hidden = self.decoder(input, hidden, encoder_outputs)\n",
        "      outputs[:, t] = output\n",
        "      # 決定是否用正確答案來做訓練\n",
        "      teacher_force = random.random() <= teacher_forcing_ratio\n",
        "      # 取出機率最大的單詞\n",
        "      top1 = output.argmax(1)\n",
        "      # 如果是 teacher force 則用正解訓練，反之用自己預測的單詞做預測\n",
        "      input = target[:, t] if teacher_force and t < target_len else top1\n",
        "      preds.append(top1.unsqueeze(1))\n",
        "    preds = torch.cat(preds, 1)\n",
        "    return outputs, preds\n",
        "\n",
        "  def inference(self, input, target):\n",
        "    ########\n",
        "    # TODO #\n",
        "    ########\n",
        "    # 在這裡實施 Beam Search\n",
        "    # 此函式的 batch size = 1  \n",
        "    # input  = [batch size, input len, vocab size]\n",
        "    # target = [batch size, target len, vocab size]\n",
        "    batch_size = input.shape[0]\n",
        "    input_len = input.shape[1]        # 取得最大字數\n",
        "    vocab_size = self.decoder.cn_vocab_size\n",
        "\n",
        "    # 準備一個儲存空間來儲存輸出\n",
        "    outputs = torch.zeros(batch_size, input_len, vocab_size).to(self.device)\n",
        "    # 將輸入放入 Encoder\n",
        "    encoder_outputs, hidden = self.encoder(input)\n",
        "    # Encoder 最後的隱藏層(hidden state) 用來初始化 Decoder\n",
        "    # encoder_outputs 主要是使用在 Attention\n",
        "    # 因為 Encoder 是雙向的RNN，所以需要將同一層兩個方向的 hidden state 接在一起\n",
        "    # hidden =  [num_layers * directions, batch size  , hid dim]  --> [num_layers, directions, batch size  , hid dim]\n",
        "    hidden = hidden.view(self.encoder.n_layers, 2, batch_size, -1)\n",
        "    hidden = torch.cat((hidden[:, -2, :, :], hidden[:, -1, :, :]), dim=2)\n",
        "    # 取的 <BOS> token\n",
        "    input = target[:, 0]\n",
        "    preds = []\n",
        "    for t in range(1, input_len):\n",
        "      output, hidden = self.decoder(input, hidden, encoder_outputs)\n",
        "      # 將預測結果存起來\n",
        "      outputs[:, t] = output\n",
        "      # 取出機率最大的單詞\n",
        "      top1 = output.argmax(1)\n",
        "      input = top1\n",
        "      preds.append(top1.unsqueeze(1))\n",
        "    preds = torch.cat(preds, 1)\n",
        "    return outputs, preds\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eCOIUTOT59aL",
        "colab_type": "text"
      },
      "source": [
        "# utils\n",
        "- 基本操作:\n",
        "  - 儲存模型\n",
        "  - 載入模型\n",
        "  - 建構模型\n",
        "  - 將一連串的數字還原回句子\n",
        "  - 計算 BLEU score\n",
        "  - 迭代 dataloader\n",
        "  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vMBoCSH5MBLN",
        "colab_type": "text"
      },
      "source": [
        "## 儲存模型"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uCZuQrWiMGmH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def save_model(model, optimizer, store_model_path, step):\n",
        "  torch.save(model.state_dict(), f'{store_model_path}/model_{step}.ckpt')\n",
        "  return"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Nde98xvAMxAd",
        "colab_type": "text"
      },
      "source": [
        "## 載入模型"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FGzZ2Yp2MxK-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def load_model(model, load_model_path):\n",
        "  print(f'Load model from {load_model_path}')\n",
        "  model.load_state_dict(torch.load(f'{load_model_path}.ckpt'))\n",
        "  return model"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eoz6awEcOIAz",
        "colab_type": "text"
      },
      "source": [
        "## 建構模型"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TvWqv_JlOOix",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def build_model(config, en_vocab_size, cn_vocab_size):\n",
        "  # 建構模型\n",
        "  encoder = Encoder(en_vocab_size, config.emb_dim, config.hid_dim, config.n_layers, config.dropout)\n",
        "  decoder = Decoder(cn_vocab_size, config.emb_dim, config.hid_dim, config.n_layers, config.dropout, config.attention)\n",
        "  model = Seq2Seq(encoder, decoder, device)\n",
        "  print(model)\n",
        "  # 建構 optimizer\n",
        "  optimizer = torch.optim.Adam(model.parameters(), lr=config.learning_rate)\n",
        "  print(optimizer)\n",
        "  if config.load_model:\n",
        "    model = load_model(model, config.load_model_path)\n",
        "  model = model.to(device)\n",
        "\n",
        "  return model, optimizer\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qEu4V_axXCF9",
        "colab_type": "text"
      },
      "source": [
        "## 數字轉句子"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ekAoI7L5mlHq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def tokens2sentence(outputs, int2word):\n",
        "  sentences = []\n",
        "  for tokens in outputs:\n",
        "    sentence = []\n",
        "    for token in tokens:\n",
        "      word = int2word[str(int(token))]\n",
        "      if word == '<EOS>':\n",
        "        break\n",
        "      sentence.append(word)\n",
        "    sentences.append(sentence)\n",
        "  \n",
        "  return sentences\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Kwpb4LDFWplj",
        "colab_type": "text"
      },
      "source": [
        "## 計算 BLEU score"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WBRNZDsMo0hF",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import nltk\n",
        "from nltk.translate.bleu_score import sentence_bleu\n",
        "from nltk.translate.bleu_score import SmoothingFunction\n",
        "\n",
        "def computebleu(sentences, targets):\n",
        "  score = 0 \n",
        "  assert (len(sentences) == len(targets))\n",
        "\n",
        "  def cut_token(sentence):\n",
        "    tmp = []\n",
        "    for token in sentence:\n",
        "      if token == '<UNK>' or token.isdigit() or len(bytes(token[0], encoding='utf-8')) == 1:\n",
        "        tmp.append(token)\n",
        "      else:\n",
        "        tmp += [word for word in token]\n",
        "    return tmp \n",
        "\n",
        "  for sentence, target in zip(sentences, targets):\n",
        "    sentence = cut_token(sentence)\n",
        "    target = cut_token(target)\n",
        "    score += sentence_bleu([target], sentence, weights=(1, 0, 0, 0))                                                                                          \n",
        "  \n",
        "  return score\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-vijuXbMeJrn",
        "colab_type": "text"
      },
      "source": [
        "##迭代 dataloader"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iIZ44EdfeJ3i",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def infinite_iter(data_loader):\n",
        "  it = iter(data_loader)\n",
        "  while True:\n",
        "    try:\n",
        "      ret = next(it)\n",
        "      yield ret\n",
        "    except StopIteration:\n",
        "      it = iter(data_loader)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aIfx7oj5fAjP",
        "colab_type": "text"
      },
      "source": [
        "## schedule_sampling"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AHuxM8m-fArz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "########\n",
        "# TODO #\n",
        "########\n",
        "\n",
        "# 請在這裡直接 return 0 來取消 Teacher Forcing\n",
        "# 請在這裡實作 schedule_sampling 的策略\n",
        "\n",
        "def schedule_sampling():\n",
        "    return 1"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bBHAhX5e5xm4",
        "colab_type": "text"
      },
      "source": [
        "# 訓練步驟"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_q9Co3vuGWfu",
        "colab_type": "text"
      },
      "source": [
        "## 訓練\n",
        "- 訓練階段"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TBnE9GbiO8Ob",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def train(model, optimizer, train_iter, loss_function, total_steps, summary_steps, train_dataset):\n",
        "  model.train()\n",
        "  model.zero_grad()\n",
        "  losses = []\n",
        "  loss_sum = 0.0\n",
        "  for step in range(summary_steps):\n",
        "    sources, targets = next(train_iter)\n",
        "    sources, targets = sources.to(device), targets.to(device)\n",
        "    outputs, preds = model(sources, targets, schedule_sampling())\n",
        "    # targets 的第一個 token 是 <BOS> 所以忽略\n",
        "    outputs = outputs[:, 1:].reshape(-1, outputs.size(2))\n",
        "    targets = targets[:, 1:].reshape(-1)\n",
        "    loss = loss_function(outputs, targets)\n",
        "    \n",
        "    optimizer.zero_grad()\n",
        "    loss.backward()\n",
        "    grad_norm = torch.nn.utils.clip_grad_norm_(model.parameters(), 1)\n",
        "    optimizer.step()\n",
        "\n",
        "    loss_sum += loss.item()\n",
        "    if (step + 1) % 5 == 0:\n",
        "      loss_sum = loss_sum / 5\n",
        "      print (\"\\r\", \"train [{}] loss: {:.3f}, Perplexity: {:.3f}      \".format(total_steps + step + 1, loss_sum, np.exp(loss_sum)), end=\" \")\n",
        "      losses.append(loss_sum)\n",
        "      loss_sum = 0.0\n",
        "\n",
        "  return model, optimizer, losses\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6cuHVXxAfHrA",
        "colab_type": "text"
      },
      "source": [
        "## 檢驗/測試\n",
        "- 防止訓練發生overfitting"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZBp-n3FrfOCe",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def test(model, dataloader, loss_function):\n",
        "  model.eval()\n",
        "  loss_sum, bleu_score= 0.0, 0.0\n",
        "  n = 0\n",
        "  result = []\n",
        "  for sources, targets in dataloader:\n",
        "    sources, targets = sources.to(device), targets.to(device)\n",
        "    batch_size = sources.size(0)\n",
        "    outputs, preds = model.inference(sources, targets)\n",
        "    # targets 的第一個 token 是 <BOS> 所以忽略\n",
        "    outputs = outputs[:, 1:].reshape(-1, outputs.size(2))\n",
        "    targets = targets[:, 1:].reshape(-1)\n",
        "\n",
        "    loss = loss_function(outputs, targets)\n",
        "    loss_sum += loss.item()\n",
        "\n",
        "    # 將預測結果轉為文字\n",
        "    targets = targets.view(sources.size(0), -1)\n",
        "    preds = tokens2sentence(preds, dataloader.dataset.int2word_cn)\n",
        "    sources = tokens2sentence(sources, dataloader.dataset.int2word_en)\n",
        "    targets = tokens2sentence(targets, dataloader.dataset.int2word_cn)\n",
        "    for source, pred, target in zip(sources, preds, targets):\n",
        "      result.append((source, pred, target))\n",
        "    # 計算 Bleu Score\n",
        "    bleu_score += computebleu(preds, targets)\n",
        "\n",
        "    n += batch_size\n",
        "\n",
        "  return loss_sum / len(dataloader), bleu_score / n, result\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dFEYmPAlx_SX",
        "colab_type": "text"
      },
      "source": [
        "## 訓練流程\n",
        "- 先訓練，再檢驗"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cJ54vDP2yC2S",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def train_process(config):\n",
        "  # 準備訓練資料\n",
        "  train_dataset = EN2CNDataset(config.data_path, config.max_output_len, 'training')\n",
        "  train_loader = data.DataLoader(train_dataset, batch_size=config.batch_size, shuffle=True)\n",
        "  train_iter = infinite_iter(train_loader)\n",
        "  # 準備檢驗資料\n",
        "  val_dataset = EN2CNDataset(config.data_path, config.max_output_len, 'validation')\n",
        "  val_loader = data.DataLoader(val_dataset, batch_size=1)\n",
        "  # 建構模型\n",
        "  model, optimizer = build_model(config, train_dataset.en_vocab_size, train_dataset.cn_vocab_size)\n",
        "  loss_function = nn.CrossEntropyLoss(ignore_index=0)\n",
        "\n",
        "  train_losses, val_losses, bleu_scores = [], [], []\n",
        "  total_steps = 0\n",
        "  while (total_steps < config.num_steps):\n",
        "    # 訓練模型\n",
        "    model, optimizer, loss = train(model, optimizer, train_iter, loss_function, total_steps, config.summary_steps, train_dataset)\n",
        "    train_losses += loss\n",
        "    # 檢驗模型\n",
        "    val_loss, bleu_score, result = test(model, val_loader, loss_function)\n",
        "    val_losses.append(val_loss)\n",
        "    bleu_scores.append(bleu_score)\n",
        "\n",
        "    total_steps += config.summary_steps\n",
        "    print (\"\\r\", \"val [{}] loss: {:.3f}, Perplexity: {:.3f}, blue score: {:.3f}       \".format(total_steps, val_loss, np.exp(val_loss), bleu_score))\n",
        "    \n",
        "    # 儲存模型和結果\n",
        "    if total_steps % config.store_steps == 0 or total_steps >= config.num_steps:\n",
        "      save_model(model, optimizer, config.store_model_path, total_steps)\n",
        "      with open(f'{config.store_model_path}/output_{total_steps}.txt', 'w') as f:\n",
        "        for line in result:\n",
        "          print (line, file=f)\n",
        "    \n",
        "  return train_losses, val_losses, bleu_scores\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QZvo0u9SofwW",
        "colab_type": "text"
      },
      "source": [
        "## 測試流程"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tvpiHCM-ogNh",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def test_process(config):\n",
        "  # 準備測試資料\n",
        "  test_dataset = EN2CNDataset(config.data_path, config.max_output_len, 'testing')\n",
        "  test_loader = data.DataLoader(test_dataset, batch_size=1)\n",
        "  # 建構模型\n",
        "  model, optimizer = build_model(config, test_dataset.en_vocab_size, test_dataset.cn_vocab_size)\n",
        "  print (\"Finish build model\")\n",
        "  loss_function = nn.CrossEntropyLoss(ignore_index=0)\n",
        "  model.eval()\n",
        "  # 測試模型\n",
        "  test_loss, bleu_score, result = test(model, test_loader, loss_function)\n",
        "  # 儲存結果\n",
        "  with open(f'./test_output.txt', 'w') as f:\n",
        "    for line in result:\n",
        "      print (line, file=f)\n",
        "\n",
        "  return test_loss, bleu_score\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7PbfgB3n9eoT",
        "colab_type": "text"
      },
      "source": [
        "# Config\n",
        "- 實驗的參數設定表"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3kWSZ4w39gzj",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class configurations(object):\n",
        "  def __init__(self):\n",
        "    self.batch_size = 60\n",
        "    self.emb_dim = 256\n",
        "    self.hid_dim = 512\n",
        "    self.n_layers = 3\n",
        "    self.dropout = 0.5\n",
        "    self.learning_rate = 0.00005\n",
        "    self.max_output_len = 50              # 最後輸出句子的最大長度\n",
        "    self.num_steps = 12000                # 總訓練次數\n",
        "    self.store_steps = 300                # 訓練多少次後須儲存模型\n",
        "    self.summary_steps = 300              # 訓練多少次後須檢驗是否有overfitting\n",
        "    self.load_model = False               # 是否需載入模型\n",
        "    self.store_model_path = \"./ckpt\"      # 儲存模型的位置\n",
        "    self.load_model_path = None           # 載入模型的位置 e.g. \"./ckpt/model_{step}\" \n",
        "    self.data_path = \"./cmn-eng\"          # 資料存放的位置\n",
        "    self.attention = False                # 是否使用 Attention Mechanism\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w464f4KOLUh6",
        "colab_type": "text"
      },
      "source": [
        "#Main Function\n",
        "- 讀入參數\n",
        "- 進行訓練或是推論"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qlwNx0z6vu_S",
        "colab_type": "text"
      },
      "source": [
        "## train"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AJwVkorvLaSh",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "if __name__ == '__main__':\n",
        "  config = configurations()\n",
        "  print ('config:\\n', vars(config))\n",
        "  train_losses, val_losses, bleu_scores = train_process(config)\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EjDNDiwZvyCn",
        "colab_type": "text"
      },
      "source": [
        "## test"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "N00ZAkL7ieGP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# 在執行 Test 之前，請先行至 config 設定所要載入的模型位置\n",
        "if __name__ == '__main__':\n",
        "  config = configurations()\n",
        "  print ('config:\\n', vars(config))\n",
        "  test_loss, bleu_score = test_process(config)\n",
        "  print (f'test loss: {test_loss}, bleu_score: {bleu_score}')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JyhX44s2wS1I",
        "colab_type": "text"
      },
      "source": [
        "# 圖形化訓練過程"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nNnx0AcDxiJz",
        "colab_type": "text"
      },
      "source": [
        "## 以圖表呈現 訓練 的 loss 變化趨勢"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EahLuZ2X8zdF",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "plt.figure()\n",
        "plt.plot(train_losses)\n",
        "plt.xlabel('次數')\n",
        "plt.ylabel('loss')\n",
        "plt.title('train loss')\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WbEi0MMGxl-v",
        "colab_type": "text"
      },
      "source": [
        "## 以圖表呈現 檢驗 的 loss 變化趨勢"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tJEqy0aBxA3E",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "plt.figure()\n",
        "plt.plot(val_losses)\n",
        "plt.xlabel('次數')\n",
        "plt.ylabel('loss')\n",
        "plt.title('validation loss')\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C5SJkPbgxp0m",
        "colab_type": "text"
      },
      "source": [
        "## BLEU score"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jhQ3Z7NPxBB8",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "plt.figure()\n",
        "plt.plot(bleu_scores)\n",
        "plt.xlabel('次數')\n",
        "plt.ylabel('BLEU score')\n",
        "plt.title('BLEU score')\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}