bert-GRU-reviews.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "设置随机种子，已保证在相同的网络结构、超参数下，相同的输入具有相同的输出效果，可参考pytorch对应的官方文档：https://pytorch.org/docs/stable/notes/randomness.html\n",
    "\n",
    "第一，控制随机性来源\n",
    "\n",
    "第二，配置pytorch来避免某些操作使用非确定行算法，以保证在相同的输入下，多次调用这些操作产生相同的结果；"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import random\n",
    "import numpy as np\n",
    "\n",
    "SEED = 1234\n",
    "\n",
    "random.seed(SEED)\n",
    "np.random.seed(SEED)\n",
    "torch.manual_seed(SEED)\n",
    "torch.backends.cudnn.deterministic = True\n",
    "# torch.use_deterministic_algorithms(True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "从 transformers 库中导入相应的 Tokenizer；官方文档参考：https://huggingface.co/docs/transformers/main_classes/tokenizer\n",
    "\n",
    "Tokenizers 负责为模型准备输入，主要有以下方法：\n",
    "1. token化，即分词；\n",
    "2. token 和 Vocabulary 的映射；\n",
    "3. 特殊标记，如掩码、句首等；"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import BertTokenizer\n",
    "\n",
    "tokenizer = BertTokenizer.from_pretrained('bert-base-chinese')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "21128"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(tokenizer.vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "将字符串转换为 token"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['真', '没', '搞', '明', '白', '京', '东', '券', '是', '怎', '么', '回', '事', '买', '没', '有', '券', '买', '完', '有', '券']\n"
     ]
    }
   ],
   "source": [
    "tokens = tokenizer.tokenize('真 没 搞 明白 京东 券是 怎么回事 买 没有 券买 完 有券 ')\n",
    "\n",
    "print(tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "tokens to indices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[4696, 3766, 3018, 3209, 4635, 776, 691, 1171, 3221, 2582, 720, 1726, 752, 743, 3766, 3300, 1171, 743, 2130, 3300, 1171]\n"
     ]
    }
   ],
   "source": [
    "indexes = tokenizer.convert_tokens_to_ids(tokens)\n",
    "print(indexes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "获取特殊字符，BERT模型中，输入序列的第一个token为 '<CLS>'，序列结束为'<SEP>'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "101 102 0 100\n"
     ]
    }
   ],
   "source": [
    "init_token_idx = tokenizer.cls_token_id\n",
    "eos_token_idx = tokenizer.sep_token_id\n",
    "pad_token_idx = tokenizer.pad_token_id\n",
    "unk_token_idx = tokenizer.unk_token_id\n",
    "\n",
    "print(init_token_idx, eos_token_idx, pad_token_idx, unk_token_idx)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "输入的最大长度，即token的数量"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "512\n"
     ]
    }
   ],
   "source": [
    "max_input_length = tokenizer.max_model_input_sizes['bert-base-chinese']\n",
    "\n",
    "print(max_input_length)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "token化的函数，即分词函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tokenize_and_cut(sentence):\n",
    "    tokens = tokenizer.tokenize(sentence)\n",
    "    tokens = tokens[: max_input_length - 2]\n",
    "    return tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 利用 torchtext 对数据进行预处理，"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchtext.legacy import data\n",
    "\n",
    "TEXT = data.Field(batch_first=True,\n",
    "                use_vocab=False,\n",
    "                tokenize=tokenize_and_cut,\n",
    "                preprocessing=tokenizer.convert_tokens_to_ids,\n",
    "                init_token = init_token_idx,\n",
    "                eos_token=eos_token_idx,\n",
    "                pad_token=pad_token_idx,\n",
    "                unk_token=unk_token_idx)\n",
    "\n",
    "LABEL = data.LabelField()   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "defaultdict(None, {'2': 0, '0': 1, '1': 2})\n"
     ]
    }
   ],
   "source": [
    "fields = [('label', LABEL), ('comment_processed', TEXT)]\n",
    "\n",
    "train_Dataset, val_Dataset, test_Dataset = data.TabularDataset.splits(\n",
    "    path='/workspace/vscode/works/研一上学期任务/data',\n",
    "    format='csv',\n",
    "    train='train_data.csv',\n",
    "    validation='valid_data.csv',\n",
    "    test='test_data.csv',\n",
    "    skip_header=True,\n",
    "    fields=fields)\n",
    "\n",
    "LABEL.build_vocab(train_Dataset)\n",
    "print(LABEL.vocab.stoi)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'label': '1', 'comment_processed': [4692, 6629, 3341, 679, 1008, 3173, 3322, 1690, 7027, 7481, 2523, 1914, 6763, 816]}\n",
      "['看', '起', '来', '不', '像', '新', '机', '器', '里', '面', '很', '多', '软', '件']\n"
     ]
    }
   ],
   "source": [
    "print(vars(train_Dataset.examples[6]))\n",
    "\n",
    "tokens = tokenizer.convert_ids_to_tokens(vars(train_Dataset.examples[6])['comment_processed'])\n",
    "print(tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of train_data: 45595\n",
      "Number of valid_data: 14249\n",
      "Number of test_data: 11399\n"
     ]
    }
   ],
   "source": [
    "print(f\"Number of train_data: {len(train_Dataset)}\")\n",
    "print(f\"Number of valid_data: {len(val_Dataset)}\")\n",
    "print(f\"Number of test_data: {len(test_Dataset)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'label': '1', 'comment_processed': [4692, 6629, 3341, 679, 1008, 3173, 3322, 1690, 7027, 7481, 2523, 1914, 6763, 816]}\n"
     ]
    }
   ],
   "source": [
    "print(vars(train_Dataset.examples[6]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['看', '起', '来', '不', '像', '新', '机', '器', '里', '面', '很', '多', '软', '件']\n"
     ]
    }
   ],
   "source": [
    "tokens = tokenizer.convert_ids_to_tokens(vars(train_Dataset.examples[6])['comment_processed'])\n",
    "\n",
    "print(tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 128\n",
    "device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')\n",
    "\n",
    "train_iterator, valid_iterator, test_iterator = data.BucketIterator.splits((train_Dataset, val_Dataset, test_Dataset),\n",
    "                                                                           batch_size=batch_size,\n",
    "                                                                           sort=False,\n",
    "                                                                           device=device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([0, 2, 1, 0, 1, 1, 1, 0, 1, 1, 1, 0, 0, 2, 2, 2, 0, 0, 2, 0, 0, 0, 2, 2,\n",
       "        2, 1, 0, 2, 1, 2, 2, 1, 1, 1, 2, 1, 0, 2, 0, 0, 1, 0, 2, 1, 1, 0, 1, 0,\n",
       "        0, 0, 2, 1, 1, 0, 2, 2, 1, 2, 0, 0, 2, 0, 0, 1, 1, 2, 1, 1, 0, 0, 1, 1,\n",
       "        1, 2, 1, 2, 1, 1, 2, 0, 1, 1, 1, 2, 0, 1, 0, 1, 0, 1, 2, 0, 0, 1, 1, 1,\n",
       "        2, 1, 1, 1, 0, 0, 0, 1, 1, 1, 2, 1, 1, 1, 1, 2, 2, 1, 0, 2, 1, 2, 1, 1,\n",
       "        2, 1, 0, 2, 2, 1, 1, 1], device='cuda:0')"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "next(iter(train_iterator)).label"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 101,  782, 4495,  ...,    0,    0,    0],\n",
       "        [ 101, 1927, 3307,  ...,    0,    0,    0],\n",
       "        [ 101, 2797, 3322,  ...,    0,    0,    0],\n",
       "        ...,\n",
       "        [ 101, 1041, 4510,  ...,    0,    0,    0],\n",
       "        [ 101, 3219, 1921,  ...,    0,    0,    0],\n",
       "        [ 101, 4212, 4685,  ...,    0,    0,    0]], device='cuda:0')"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "next(iter(train_iterator)).comment_processed "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "[torchtext.legacy.data.batch.Batch of size 128]\n",
       "\t[.label]:[torch.cuda.LongTensor of size 128 (GPU 0)]\n",
       "\t[.comment_processed]:[torch.cuda.LongTensor of size 128x233 (GPU 0)]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "next(iter(train_iterator))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "导入预训练模型，"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at bert-base-chinese were not used when initializing BertModel: ['cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.weight', 'cls.seq_relationship.bias', 'cls.seq_relationship.weight']\n",
      "- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
     ]
    }
   ],
   "source": [
    "from transformers import BertTokenizer, BertModel\n",
    "\n",
    "bert = BertModel.from_pretrained('bert-base-chinese')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 定义模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn as nn\n",
    "\n",
    "class BERTGRUSentiment(nn.Module):\n",
    "    def __init__(self, bert, hidden_dim, nums_output, n_layers, bidirectional, dropout):\n",
    "        super(BERTGRUSentiment, self).__init__()\n",
    "        self.bert = bert\n",
    "        embedding_dim = bert.config.to_dict()['hidden_size']\n",
    "        self.rnn = nn.GRU(embedding_dim,\n",
    "                            hidden_dim,\n",
    "                            num_layers=n_layers, \n",
    "                            bidirectional=bidirectional,\n",
    "                            batch_first=True,\n",
    "                            dropout = 0 if n_layers < 2 else dropout)\n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "        self.output = nn.Linear(hidden_dim * 2 if bidirectional else hidden_dim, nums_output)\n",
    "\n",
    "    def forward(self, text):\n",
    "        # text的大小： (batch size, sentence len)\n",
    "        # with torch.no_grad():\n",
    "        embeded = self.bert(text)[0]\n",
    "        \n",
    "        # embeded的大小：(batch size, sentence len, embeded dim)\n",
    "        _, hidden = self.rnn(embeded)\n",
    "\n",
    "        # hidden的大小：(nums_layer * bidirectional, batch size, embeded dim)\n",
    "        if self.rnn.bidirectional:\n",
    "            hidden = self.dropout(torch.cat((hidden[-2,:,:], hidden[-1,:,:]), dim=1))\n",
    "        else:\n",
    "            hidden = self.dropout(hidden[-1,:,:])\n",
    "        \n",
    "        # output的大小：(batch size, nums_output)\n",
    "        output = self.output(hidden)\n",
    "\n",
    "        return output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 设置超参数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "HIDDEN_DIM = 512\n",
    "NUMS_OUTPUT = 3\n",
    "N_LAYERS = 2\n",
    "BIDIRECTIONAL = True\n",
    "DROPOUT = 0.25\n",
    "lr = 0.0005\n",
    "\n",
    "model = BERTGRUSentiment(bert, HIDDEN_DIM, NUMS_OUTPUT, N_LAYERS, BIDIRECTIONAL, DROPOUT)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The model has 110,933,763 trainable parameters\n"
     ]
    }
   ],
   "source": [
    "def count_parameters(model):\n",
    "    return sum(p.numel() for p in model.parameters() if p.requires_grad)\n",
    "\n",
    "print(f'The model has {count_parameters(model):,} trainable parameters')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.optim as optim\n",
    "\n",
    "\n",
    "optimizer = optim.Adam(model.parameters(), lr=lr)\n",
    "# optimizer = optim.Adagrad(model.parameters())\n",
    "# criterion = nn.BCEWithLogitsLoss()\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "model = model.to(device)\n",
    "criterion = criterion.to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "def accuracy(pred, y):\n",
    "    correct = (pred.argmax(dim=1) == y).float()\n",
    "    return correct.sum() / len(correct)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train(model, iterator, optimizer, criterion):\n",
    "    epoch_loss = 0\n",
    "    epoch_acc = 0\n",
    "\n",
    "    model.train()\n",
    "\n",
    "    for batch in iterator:\n",
    "        optimizer.zero_grad()\n",
    "        preds = model(batch.comment_processed).squeeze(1)\n",
    "        loss = criterion(preds, batch.label)\n",
    "        acc = accuracy(preds, batch.label)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        epoch_loss += loss.item()\n",
    "        epoch_acc += acc.item()\n",
    "\n",
    "    return epoch_loss / len(iterator), epoch_acc / len(iterator)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate(model, iterator, criterion):\n",
    "    epoch_loss = 0\n",
    "    epoch_acc = 0\n",
    "\n",
    "    model.eval()\n",
    "\n",
    "    with torch.no_grad():\n",
    "        for batch in iterator:\n",
    "            preds = model(batch.comment_processed).squeeze(1)\n",
    "            loss = criterion(preds, batch.label)\n",
    "            acc = accuracy(preds, batch.label)\n",
    "            epoch_loss += loss.item()\n",
    "            epoch_acc += acc.item()\n",
    "    \n",
    "    return epoch_loss / len(iterator), epoch_acc / len(iterator)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "def epoch_time(start_time, end_time):\n",
    "    elapsed_time = end_time - start_time\n",
    "    elapsed_mins = int(elapsed_time / 60)\n",
    "    elapsed_secs = int(elapsed_time - (elapsed_mins * 60))\n",
    "    return elapsed_mins, elapsed_secs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "ename": "RuntimeError",
     "evalue": "CUDA out of memory. Tried to allocate 734.00 MiB (GPU 0; 44.56 GiB total capacity; 39.56 GiB already allocated; 496.31 MiB free; 42.08 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mRuntimeError\u001b[0m                              Traceback (most recent call last)",
      "\u001b[0;32m/tmp/ipykernel_2104022/704896903.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      6\u001b[0m     \u001b[0mstart_time\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtime\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtime\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      7\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 8\u001b[0;31m     \u001b[0mtrain_loss\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtrain_acc\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtrain\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtrain_iterator\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moptimizer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcriterion\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      9\u001b[0m     \u001b[0mvalid_loss\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalid_acc\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mevaluate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalid_iterator\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcriterion\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     10\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/tmp/ipykernel_2104022/845781069.py\u001b[0m in \u001b[0;36mtrain\u001b[0;34m(model, iterator, optimizer, criterion)\u001b[0m\n\u001b[1;32m      7\u001b[0m     \u001b[0;32mfor\u001b[0m \u001b[0mbatch\u001b[0m \u001b[0;32min\u001b[0m \u001b[0miterator\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      8\u001b[0m         \u001b[0moptimizer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mzero_grad\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 9\u001b[0;31m         \u001b[0mpreds\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbatch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcomment_processed\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msqueeze\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     10\u001b[0m         \u001b[0mloss\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcriterion\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpreds\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlabel\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     11\u001b[0m         \u001b[0macc\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0maccuracy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpreds\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlabel\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/envs/lh/lib/python3.9/site-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m   1100\u001b[0m         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\n\u001b[1;32m   1101\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1102\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1103\u001b[0m         \u001b[0;31m# Do not call functions when jit is used\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1104\u001b[0m         \u001b[0mfull_backward_hooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/tmp/ipykernel_2104022/2543720275.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, text)\u001b[0m\n\u001b[1;32m     21\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     22\u001b[0m         \u001b[0;31m# embeded的大小：(batch size, sentence len, embeded dim)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 23\u001b[0;31m         \u001b[0m_\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhidden\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrnn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0membeded\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     24\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     25\u001b[0m         \u001b[0;31m# hidden的大小：(nums_layer * bidirectional, batch size, embeded dim)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/envs/lh/lib/python3.9/site-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m   1100\u001b[0m         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\n\u001b[1;32m   1101\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1102\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1103\u001b[0m         \u001b[0;31m# Do not call functions when jit is used\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1104\u001b[0m         \u001b[0mfull_backward_hooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/envs/lh/lib/python3.9/site-packages/torch/nn/modules/rnn.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, input, hx)\u001b[0m\n\u001b[1;32m    847\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcheck_forward_args\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch_sizes\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    848\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mbatch_sizes\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 849\u001b[0;31m             result = _VF.gru(input, hx, self._flat_weights, self.bias, self.num_layers,\n\u001b[0m\u001b[1;32m    850\u001b[0m                              self.dropout, self.training, self.bidirectional, self.batch_first)\n\u001b[1;32m    851\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mRuntimeError\u001b[0m: CUDA out of memory. Tried to allocate 734.00 MiB (GPU 0; 44.56 GiB total capacity; 39.56 GiB already allocated; 496.31 MiB free; 42.08 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF"
     ]
    }
   ],
   "source": [
    "NUM_EPOCHS = 5\n",
    "\n",
    "best_valid_loss = float('inf')\n",
    "\n",
    "for epoch in range(NUM_EPOCHS):\n",
    "    start_time = time.time()\n",
    "    \n",
    "    train_loss, train_acc = train(model, train_iterator, optimizer, criterion)\n",
    "    valid_loss, valid_acc = evaluate(model, valid_iterator, criterion)\n",
    "\n",
    "    end_time = time.time()\n",
    "\n",
    "    epoch_mins, epoch_secs = epoch_time(start_time, end_time)\n",
    "\n",
    "    if valid_loss < best_valid_loss:\n",
    "        best_valid_loss = valid_loss\n",
    "        torch.save(model.state_dict(), \"bert-GRU-Reviews-Sentiment.pt\")\n",
    "    \n",
    "    print(f'Epoch: {epoch+1:02} | Epoch Time: {epoch_mins}m {epoch_secs}s')\n",
    "    print(f'\\tTrain Loss: {train_loss:.3f} | Train Acc: {train_acc*100:.2f}%')\n",
    "    print(f'\\t Val. Loss: {valid_loss:.3f} |  Val. Acc: {valid_acc*100:.2f}%')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test Loss: 0.548 | Test Acc: 75.43%\n"
     ]
    }
   ],
   "source": [
    "model.load_state_dict(torch.load('bert-GRU-Reviews-Sentiment.pt'))\n",
    "\n",
    "test_loss, test_acc = evaluate(model, test_iterator, criterion)\n",
    "\n",
    "print(f'Test Loss: {test_loss:.3f} | Test Acc: {test_acc*100:.2f}%')"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "e0db6a32edbaa27d77e434f2bc0720d584d57ec005a522e6f96234b9b09a03ca"
  },
  "kernelspec": {
   "display_name": "Python 3.9.7 64-bit ('lh': conda)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}