SMP2018_EDA_and_Baseline_Model(Keras).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# $$SMP2018中文人机对话技术评测（ECDT）$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. 下面是一个完整的针对 [SMP2018中文人机对话技术评测（ECDT）](http://smp2018.cips-smp.org/ecdt_index.html) 的实验，由该实验训练的基线模型能达到评测排行榜的前三的水平。\n",
    "2. 通过本实验，可以掌握处理自然语言文本数据的一般方法。\n",
    "3. 推荐自己修改此文件，达到更好的实验效果，比如改变以下几个超参数 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "# 词嵌入的维度\n",
    "embedding_word_dims = 32\n",
    "# 批次大小\n",
    "batch_size = 30\n",
    "# 周期\n",
    "epochs = 20\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 本实验还可以改进的地方举例 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. 预处理阶段使用其它的分词工具\n",
    "2. 采用字符向量和词向量结合的方式\n",
    "3. 使用预先训练好的词向量\n",
    "4. 改变模型结构\n",
    "5. 改变模型超参数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 导入依赖库"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import collections\n",
    "import jieba\n",
    "from keras.preprocessing.text import Tokenizer\n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Embedding, LSTM, Dense\n",
    "from keras.utils import to_categorical,plot_model\n",
    "from keras.callbacks import TensorBoard, Callback\n",
    "\n",
    "from sklearn.metrics import classification_report\n",
    "\n",
    "import requests \n",
    "\n",
    "import time\n",
    "\n",
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 辅助函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras import backend as K\n",
    "\n",
    "# 计算 F1 值的函数\n",
    "def f1(y_true, y_pred):\n",
    "    def recall(y_true, y_pred):\n",
    "        \"\"\"Recall metric.\n",
    "\n",
    "        Only computes a batch-wise average of recall.\n",
    "\n",
    "        Computes the recall, a metric for multi-label classification of\n",
    "        how many relevant items are selected.\n",
    "        \"\"\"\n",
    "        true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))\n",
    "        possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))\n",
    "        recall = true_positives / (possible_positives + K.epsilon())\n",
    "        return recall\n",
    "\n",
    "    def precision(y_true, y_pred):\n",
    "        \"\"\"Precision metric.\n",
    "\n",
    "        Only computes a batch-wise average of precision.\n",
    "\n",
    "        Computes the precision, a metric for multi-label classification of\n",
    "        how many selected items are relevant.\n",
    "        \"\"\"\n",
    "        true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))\n",
    "        predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))\n",
    "        precision = true_positives / (predicted_positives + K.epsilon())\n",
    "        return precision\n",
    "    precision = precision(y_true, y_pred)\n",
    "    recall = recall(y_true, y_pred)\n",
    "    return 2*((precision*recall)/(precision+recall+K.epsilon()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 获取自定义时间格式的字符串\n",
    "def get_customization_time():\n",
    "    # return '2018_10_10_18_11_45' 年月日时分秒\n",
    "    time_tuple = time.localtime(time.time())\n",
    "    customization_time = \"{}_{}_{}_{}_{}_{}\".format(time_tuple[0], time_tuple[1], time_tuple[2], time_tuple[3], time_tuple[4], time_tuple[5])\n",
    "    return customization_time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 准备数据"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## [下载SMP2018官方数据](https://worksheets.codalab.org/worksheets/0x27203f932f8341b79841d50ce0fd684f/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "raw_train_data_url = \"https://worksheets.codalab.org/rest/bundles/0x0161fd2fb40d4dd48541c2643d04b0b8/contents/blob/\"\n",
    "raw_test_data_url = \"https://worksheets.codalab.org/rest/bundles/0x1f96bc12222641209ad057e762910252/contents/blob/\"\n",
    "\n",
    "# 如果不存在 SMP2018 数据，则下载\n",
    "if (not os.path.exists('./data/train.json')) or (not os.path.exists('./data/dev.json')):\n",
    "    raw_train = requests.get(raw_train_data_url) \n",
    "    raw_test = requests.get(raw_test_data_url) \n",
    "    if not os.path.exists('./data'):\n",
    "        os.makedirs('./data')\n",
    "    with open(\"./data/train.json\", \"wb\") as code:\n",
    "         code.write(raw_train.content)\n",
    "    with open(\"./data/dev.json\", \"wb\") as code:\n",
    "         code.write(raw_test.content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_json_data(path):\n",
    "    # read data\n",
    "    data_df = pd.read_json(path)\n",
    "    # change row and colunm\n",
    "    data_df = data_df.transpose()\n",
    "    # change colunm order\n",
    "    data_df = data_df[['query', 'label']]\n",
    "    return data_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data_df = get_json_data(path=\"data/train.json\")\n",
    "\n",
    "test_data_df = get_json_data(path=\"data/dev.json\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>query</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>今天东莞天气如何</td>\n",
       "      <td>weather</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>从观音桥到重庆市图书馆怎么走</td>\n",
       "      <td>map</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>鸭蛋怎么腌？</td>\n",
       "      <td>cookbook</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>怎么治疗牛皮癣</td>\n",
       "      <td>health</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>唠什么</td>\n",
       "      <td>chat</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            query     label\n",
       "0        今天东莞天气如何   weather\n",
       "1  从观音桥到重庆市图书馆怎么走       map\n",
       "2          鸭蛋怎么腌？  cookbook\n",
       "3         怎么治疗牛皮癣    health\n",
       "4             唠什么      chat"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## [结巴分词](https://github.com/fxsjy/jieba)示例，下面将使用结巴分词对原数据进行处理"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Building prefix dict from the default dictionary ...\n",
      "Loading model from cache /tmp/jieba.cache\n",
      "Loading model cost 1.022 seconds.\n",
      "Prefix dict has been built succesfully.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['他', '来到', '了', '网易', '杭研', '大厦']\n"
     ]
    }
   ],
   "source": [
    "seg_list = jieba.cut(\"他来到了网易杭研大厦\")  # 默认是精确模式\n",
    "print(list(seg_list))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 序列化"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def use_jieba_cut(a_sentence):\n",
    "    return list(jieba.cut(a_sentence))\n",
    "\n",
    "train_data_df['cut_query'] = train_data_df['query'].apply(use_jieba_cut)\n",
    "test_data_df['cut_query'] = test_data_df['query'].apply(use_jieba_cut)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>query</th>\n",
       "      <th>label</th>\n",
       "      <th>cut_query</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>今天东莞天气如何</td>\n",
       "      <td>weather</td>\n",
       "      <td>[今天, 东莞, 天气, 如何]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>从观音桥到重庆市图书馆怎么走</td>\n",
       "      <td>map</td>\n",
       "      <td>[从, 观音桥, 到, 重庆市, 图书馆, 怎么, 走]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>鸭蛋怎么腌？</td>\n",
       "      <td>cookbook</td>\n",
       "      <td>[鸭蛋, 怎么, 腌, ？]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>怎么治疗牛皮癣</td>\n",
       "      <td>health</td>\n",
       "      <td>[怎么, 治疗, 牛皮癣]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>唠什么</td>\n",
       "      <td>chat</td>\n",
       "      <td>[唠, 什么]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>阳澄湖大闸蟹的做法。</td>\n",
       "      <td>cookbook</td>\n",
       "      <td>[阳澄湖, 大闸蟹, 的, 做法, 。]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>昆山大润发在哪里</td>\n",
       "      <td>map</td>\n",
       "      <td>[昆山, 大润发, 在, 哪里]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>红烧肉怎么做？嗯？</td>\n",
       "      <td>cookbook</td>\n",
       "      <td>[红烧肉, 怎么, 做, ？, 嗯, ？]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>南京到厦门的火车票</td>\n",
       "      <td>train</td>\n",
       "      <td>[南京, 到, 厦门, 的, 火车票]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>6的平方</td>\n",
       "      <td>calc</td>\n",
       "      <td>[6, 的, 平方]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            query     label                     cut_query\n",
       "0        今天东莞天气如何   weather              [今天, 东莞, 天气, 如何]\n",
       "1  从观音桥到重庆市图书馆怎么走       map  [从, 观音桥, 到, 重庆市, 图书馆, 怎么, 走]\n",
       "2          鸭蛋怎么腌？  cookbook                [鸭蛋, 怎么, 腌, ？]\n",
       "3         怎么治疗牛皮癣    health                 [怎么, 治疗, 牛皮癣]\n",
       "4             唠什么      chat                       [唠, 什么]\n",
       "5      阳澄湖大闸蟹的做法。  cookbook          [阳澄湖, 大闸蟹, 的, 做法, 。]\n",
       "6        昆山大润发在哪里       map              [昆山, 大润发, 在, 哪里]\n",
       "7       红烧肉怎么做？嗯？  cookbook         [红烧肉, 怎么, 做, ？, 嗯, ？]\n",
       "8       南京到厦门的火车票     train           [南京, 到, 厦门, 的, 火车票]\n",
       "9            6的平方      calc                    [6, 的, 平方]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data_df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 处理特征"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = Tokenizer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer.fit_on_texts(train_data_df['cut_query'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2883"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "max_features = len(tokenizer.index_word)\n",
    "\n",
    "len(tokenizer.index_word)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train = tokenizer.texts_to_sequences(train_data_df['cut_query'])\n",
    "\n",
    "x_test = tokenizer.texts_to_sequences(test_data_df['cut_query'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "max_cut_query_lenth = 26"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train = pad_sequences(x_train, max_cut_query_lenth)\n",
    "\n",
    "x_test = pad_sequences(x_test, max_cut_query_lenth)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2299, 26)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(770, 26)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 处理标签"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "label_tokenizer = Tokenizer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "label_tokenizer.fit_on_texts(train_data_df['label'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "label_numbers = len(label_tokenizer.word_counts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "NUM_CLASSES = len(label_tokenizer.word_counts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "OrderedDict([('weather', 66),\n",
       "             ('map', 68),\n",
       "             ('cookbook', 269),\n",
       "             ('health', 55),\n",
       "             ('chat', 455),\n",
       "             ('train', 70),\n",
       "             ('calc', 24),\n",
       "             ('translation', 61),\n",
       "             ('music', 66),\n",
       "             ('tvchannel', 71),\n",
       "             ('poetry', 102),\n",
       "             ('telephone', 63),\n",
       "             ('stock', 71),\n",
       "             ('radio', 24),\n",
       "             ('contacts', 30),\n",
       "             ('lottery', 24),\n",
       "             ('website', 54),\n",
       "             ('video', 182),\n",
       "             ('news', 58),\n",
       "             ('bus', 24),\n",
       "             ('app', 53),\n",
       "             ('flight', 62),\n",
       "             ('epg', 107),\n",
       "             ('message', 63),\n",
       "             ('match', 24),\n",
       "             ('schedule', 29),\n",
       "             ('novel', 24),\n",
       "             ('riddle', 34),\n",
       "             ('email', 24),\n",
       "             ('datetime', 18),\n",
       "             ('cinemas', 24)])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "label_tokenizer.word_counts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train = label_tokenizer.texts_to_sequences(train_data_df['label'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[10], [9], [2], [17], [1], [2], [9], [2], [8], [23]]"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train = [[y[0]-1] for y in y_train]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[9], [8], [1], [16], [0], [1], [8], [1], [7], [22]]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2299, 31)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train = to_categorical(y_train, label_numbers)\n",
    "y_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(770, 31)"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_test = label_tokenizer.texts_to_sequences(test_data_df['label'])\n",
    "y_test = [y[0]-1 for y in y_test]\n",
    "y_test = to_categorical(y_test, label_numbers)\n",
    "y_test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_test[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 设计模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_SMP2018_lstm_model(max_features, max_cut_query_lenth, label_numbers):\n",
    "    model = Sequential()\n",
    "    model.add(Embedding(input_dim=max_features+1, output_dim=32, input_length=max_cut_query_lenth))\n",
    "    model.add(LSTM(units=64, dropout=0.2, recurrent_dropout=0.2))\n",
    "    model.add(Dense(label_numbers, activation='softmax'))\n",
    "    # try using different optimizers and different optimizer configs\n",
    "    model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='adam',\n",
    "              metrics=[f1])\n",
    "\n",
    "    plot_model(model, to_file='SMP2018_lstm_model.png', show_shapes=True)\n",
    "    \n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 训练模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "if 'max_features'  not in  dir():\n",
    "    max_features = 2888\n",
    "    print('not find max_features variable, use default max_features values:\\t{}'.format(max_features))\n",
    "if 'max_cut_query_lenth'  not in  dir():\n",
    "    max_cut_query_lenth = 26\n",
    "    print('not find max_cut_query_lenth, use default max_features values:\\t{}'.format(max_cut_query_lenth))\n",
    "if 'label_numbers'  not in  dir():\n",
    "    label_numbers = 31\n",
    "    print('not find label_numbers, use default max_features values:\\t{}'.format(label_numbers))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = create_SMP2018_lstm_model(max_features, max_cut_query_lenth, label_numbers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 20\n",
    "epochs = 30"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(2299, 26) (2299, 31)\n"
     ]
    }
   ],
   "source": [
    "print(x_train.shape, y_train.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(770, 26) (770, 31)\n"
     ]
    }
   ],
   "source": [
    "print(x_test.shape, y_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train...\n",
      "Epoch 1/30\n",
      "2299/2299 [==============================] - 16s 7ms/step - loss: 3.0916 - f1: 0.0000e+00\n",
      "Epoch 2/30\n",
      "2299/2299 [==============================] - 14s 6ms/step - loss: 2.6594 - f1: 0.1409\n",
      "Epoch 3/30\n",
      "2299/2299 [==============================] - 13s 6ms/step - loss: 2.0817 - f1: 0.4055\n",
      "Epoch 4/30\n",
      "2299/2299 [==============================] - 14s 6ms/step - loss: 1.6032 - f1: 0.4689\n",
      "Epoch 5/30\n",
      "2299/2299 [==============================] - 14s 6ms/step - loss: 1.1318 - f1: 0.6176\n",
      "Epoch 6/30\n",
      "2299/2299 [==============================] - 14s 6ms/step - loss: 0.8090 - f1: 0.7399\n",
      "Epoch 7/30\n",
      "2299/2299 [==============================] - 14s 6ms/step - loss: 0.5704 - f1: 0.8298\n",
      "Epoch 8/30\n",
      "2299/2299 [==============================] - 14s 6ms/step - loss: 0.4051 - f1: 0.8879\n",
      "Epoch 9/30\n",
      "2299/2299 [==============================] - 14s 6ms/step - loss: 0.3002 - f1: 0.9280\n",
      "Epoch 10/30\n",
      "2299/2299 [==============================] - 14s 6ms/step - loss: 0.2317 - f1: 0.9467\n",
      "Epoch 11/30\n",
      "2299/2299 [==============================] - 14s 6ms/step - loss: 0.1755 - f1: 0.9678\n",
      "Epoch 12/30\n",
      "2299/2299 [==============================] - 14s 6ms/step - loss: 0.1391 - f1: 0.9758\n",
      "Epoch 13/30\n",
      "2299/2299 [==============================] - 14s 6ms/step - loss: 0.1131 - f1: 0.9800\n",
      "Epoch 14/30\n",
      "2299/2299 [==============================] - 14s 6ms/step - loss: 0.0883 - f1: 0.9861\n",
      "Epoch 15/30\n",
      "2299/2299 [==============================] - 14s 6ms/step - loss: 0.0725 - f1: 0.9894\n",
      "Epoch 16/30\n",
      "2299/2299 [==============================] - 14s 6ms/step - loss: 0.0615 - f1: 0.9929\n",
      "Epoch 17/30\n",
      "2299/2299 [==============================] - 14s 6ms/step - loss: 0.0507 - f1: 0.9945\n",
      "Epoch 18/30\n",
      "2299/2299 [==============================] - 14s 6ms/step - loss: 0.0455 - f1: 0.9963\n",
      "Epoch 19/30\n",
      "2299/2299 [==============================] - 14s 6ms/step - loss: 0.0398 - f1: 0.9960\n",
      "Epoch 20/30\n",
      "2299/2299 [==============================] - 14s 6ms/step - loss: 0.0313 - f1: 0.9978\n",
      "Epoch 21/30\n",
      "2299/2299 [==============================] - 14s 6ms/step - loss: 0.0266 - f1: 0.9984\n",
      "Epoch 22/30\n",
      "2299/2299 [==============================] - 14s 6ms/step - loss: 0.0279 - f1: 0.9965\n",
      "Epoch 23/30\n",
      "2299/2299 [==============================] - 14s 6ms/step - loss: 0.0250 - f1: 0.9976\n",
      "Epoch 24/30\n",
      "2299/2299 [==============================] - 14s 6ms/step - loss: 0.0219 - f1: 0.9982\n",
      "Epoch 25/30\n",
      "2299/2299 [==============================] - 14s 6ms/step - loss: 0.0195 - f1: 0.9982\n",
      "Epoch 26/30\n",
      "2299/2299 [==============================] - 14s 6ms/step - loss: 0.0179 - f1: 0.9989\n",
      "Epoch 27/30\n",
      "2299/2299 [==============================] - 14s 6ms/step - loss: 0.0177 - f1: 0.9974\n",
      "Epoch 28/30\n",
      "2299/2299 [==============================] - 14s 6ms/step - loss: 0.0139 - f1: 0.9987\n",
      "Epoch 29/30\n",
      "2299/2299 [==============================] - 14s 6ms/step - loss: 0.0139 - f1: 0.9989\n",
      "Epoch 30/30\n",
      "2299/2299 [==============================] - 14s 6ms/step - loss: 0.0129 - f1: 0.9987\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f84e87c5f28>"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('Train...')\n",
    "model.fit(x_train, y_train,\n",
    "          batch_size=batch_size,\n",
    "          epochs=epochs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 评估模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "770/770 [==============================] - 1s 1ms/step\n",
      "Test score: 0.6803552009068526\n",
      "Test f1: 0.8464262740952628\n"
     ]
    }
   ],
   "source": [
    "score = model.evaluate(x_test, y_test,\n",
    "                       batch_size=batch_size, verbose=1)\n",
    "\n",
    "print('Test score:', score[0])\n",
    "print('Test f1:', score[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_hat_test = model.predict(x_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(770, 31)\n"
     ]
    }
   ],
   "source": [
    "print(y_hat_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 将 one-hot 张量转换成对应的整数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = np.argmax(y_hat_test, axis=1).tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_true = np.argmax(y_test, axis=1).tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 查看多分类的 准确率、召回率、F1 值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.78      0.93      0.85       154\n",
      "           1       0.92      0.97      0.95        89\n",
      "           2       0.67      0.62      0.64        60\n",
      "           3       0.83      0.83      0.83        36\n",
      "           4       0.79      1.00      0.88        34\n",
      "           5       0.83      0.65      0.73        23\n",
      "           6       1.00      0.83      0.91        24\n",
      "           7       1.00      1.00      1.00        24\n",
      "           8       0.68      0.65      0.67        23\n",
      "           9       0.90      0.86      0.88        22\n",
      "          10       0.85      0.50      0.63        22\n",
      "          11       0.88      1.00      0.93        21\n",
      "          12       1.00      0.90      0.95        21\n",
      "          13       0.91      0.95      0.93        21\n",
      "          14       1.00      0.95      0.98        21\n",
      "          15       0.79      0.95      0.86        20\n",
      "          16       0.90      0.47      0.62        19\n",
      "          17       0.79      0.61      0.69        18\n",
      "          18       0.63      0.67      0.65        18\n",
      "          19       0.90      0.82      0.86        11\n",
      "          20       1.00      0.70      0.82        10\n",
      "          21       1.00      0.67      0.80         9\n",
      "          22       1.00      0.88      0.93         8\n",
      "          23       1.00      0.62      0.77         8\n",
      "          24       1.00      1.00      1.00         8\n",
      "          25       1.00      0.88      0.93         8\n",
      "          26       0.88      0.88      0.88         8\n",
      "          27       0.86      0.75      0.80         8\n",
      "          28       1.00      1.00      1.00         8\n",
      "          29       0.75      0.75      0.75         8\n",
      "          30       0.75      1.00      0.86         6\n",
      "\n",
      "   micro avg       0.84      0.84      0.84       770\n",
      "   macro avg       0.88      0.82      0.84       770\n",
      "weighted avg       0.85      0.84      0.84       770\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(classification_report(y_true, y_pred))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}