Recommendation System_0.1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Movie Recommendation system\n",
    "This is a group project of ISSS 610 Applied Machine Learning.  \n",
    "\n",
    "Group Member: GENG Minghong, HUANG Lu, Manish Agarwal, TAO Xinru  \n",
    "\n",
    "Project Timeline:\n",
    "\n",
    "Time|Jobs\n",
    "-----|-----\n",
    "2020-03-21|Data Cleaning\n",
    "2020-03-23|Exploring\n",
    "**2020-03-30**|**Presentation**\n",
    "\n",
    "# 0. Introduction \n",
    "The data of this project can be accessed here:  \n",
    "https://smu.sharepoint.com/teams/ISSS610AppliedMachineLearning/Shared%20Documents/Forms/AllItems.aspx?viewid=1b247e4e%2D6a34%2D4c70%2D8abc%2D06cee2cccbea  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Processing Data\n",
    "## 1.1 Load Packages and Import the Data\n",
    "The packages we used in this packages are:\n",
    "- Pandas\n",
    "- NumPy\n",
    "- ...\n",
    "- (To be fill)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "links= pd.read_csv(r'data\\themoviesdataset\\links.csv')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.2 Data Cleaning "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics.pairwise import (linear_kernel, \n",
    "                                      cosine_similarity)\n",
    "from ast import literal_eval\n",
    "import datetime"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### 1.2.1 keywords.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "keywords= pd.read_csv(r'data\\themoviesdataset\\keywords.csv')\n",
    "keywords['keywordsTr'] = keywords['keywords'].fillna('[]').apply(literal_eval).apply(lambda x: [i['name'] for i in x] if isinstance(x, list) else [])\n",
    "keywordsTr=keywords.drop(['keywords'],axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tmdb_id</th>\n",
       "      <th>keywordsTr</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>862</td>\n",
       "      <td>[jealousy, toy, boy, friendship, friends, riva...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>8844</td>\n",
       "      <td>[board game, disappearance, based on children'...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>15602</td>\n",
       "      <td>[fishing, best friend, duringcreditsstinger, o...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>31357</td>\n",
       "      <td>[based on novel, interracial relationship, sin...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>11862</td>\n",
       "      <td>[baby, midlife crisis, confidence, aging, daug...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  tmdb_id                                         keywordsTr\n",
       "0     862  [jealousy, toy, boy, friendship, friends, riva...\n",
       "1    8844  [board game, disappearance, based on children'...\n",
       "2   15602  [fishing, best friend, duringcreditsstinger, o...\n",
       "3   31357  [based on novel, interracial relationship, sin...\n",
       "4   11862  [baby, midlife crisis, confidence, aging, daug..."
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keywordsTr.rename(columns = {'id':'tmdb_id'},inplace=True)\n",
    "keywordsTr['tmdb_id'] = keywordsTr['tmdb_id'].apply(str)\n",
    "\n",
    "keywordsTr.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "keywordsTr.to_csv('keywordsTr.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2.2  rating.csv\n",
    "We want to transform the time stamp into readable time format.   Reference:https://blog.csdn.net/weixin_43790560/article/details/88412005"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "rating= pd.read_csv(r'data\\themoviesdataset\\ratings.csv')\n",
    "#rating['timestampTr']=pd.to_datetime(rating['timestamp'],unit='s')\n",
    "ratingTr=rating[['userId','movieId','rating','timestamp']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ratingTr.to_csv('ratingTr.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>movieId</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>110</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1425941529</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>147</td>\n",
       "      <td>4.5</td>\n",
       "      <td>1425942435</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>858</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1425941523</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>1221</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1425941546</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1246</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1425941556</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userId  movieId  rating   timestamp\n",
       "0       1      110     1.0  1425941529\n",
       "1       1      147     4.5  1425942435\n",
       "2       1      858     5.0  1425941523\n",
       "3       1     1221     5.0  1425941546\n",
       "4       1     1246     5.0  1425941556"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratingTr.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Then we need to decide what to keep in `timestampTr`. Day? Week? Month?**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2.3 credits.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from ast import literal_eval"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>cast</th>\n",
       "      <th>crew</th>\n",
       "      <th>id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>[{'cast_id': 14, 'character': 'Woody (voice)',...</td>\n",
       "      <td>[{'credit_id': '52fe4284c3a36847f8024f49', 'de...</td>\n",
       "      <td>862</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>[{'cast_id': 1, 'character': 'Alan Parrish', '...</td>\n",
       "      <td>[{'credit_id': '52fe44bfc3a36847f80a7cd1', 'de...</td>\n",
       "      <td>8844</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                cast  \\\n",
       "0  [{'cast_id': 14, 'character': 'Woody (voice)',...   \n",
       "1  [{'cast_id': 1, 'character': 'Alan Parrish', '...   \n",
       "\n",
       "                                                crew    id  \n",
       "0  [{'credit_id': '52fe4284c3a36847f8024f49', 'de...   862  \n",
       "1  [{'credit_id': '52fe44bfc3a36847f80a7cd1', 'de...  8844  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "credits = pd.read_csv(r'data\\themoviesdataset\\credits.csv')\n",
    "credits.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "credits['cast'] = credits['cast'].apply(literal_eval)\n",
    "credits['crew'] = credits['crew'].apply(literal_eval)\n",
    "credits['cast_size'] = credits['cast'].apply(lambda x: len(x))\n",
    "credits['crew_size'] = credits['crew'].apply(lambda x: len(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Rename the `id` column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "credits.rename(columns = {'id':'tmdb_id'},inplace=True)\n",
    "credits['tmdb_id'] = credits['tmdb_id'].apply(str)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>cast</th>\n",
       "      <th>crew</th>\n",
       "      <th>tmdb_id</th>\n",
       "      <th>cast_size</th>\n",
       "      <th>crew_size</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>[{'cast_id': 14, 'character': 'Woody (voice)',...</td>\n",
       "      <td>[{'credit_id': '52fe4284c3a36847f8024f49', 'de...</td>\n",
       "      <td>862</td>\n",
       "      <td>13</td>\n",
       "      <td>106</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>[{'cast_id': 1, 'character': 'Alan Parrish', '...</td>\n",
       "      <td>[{'credit_id': '52fe44bfc3a36847f80a7cd1', 'de...</td>\n",
       "      <td>8844</td>\n",
       "      <td>26</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                cast  \\\n",
       "0  [{'cast_id': 14, 'character': 'Woody (voice)',...   \n",
       "1  [{'cast_id': 1, 'character': 'Alan Parrish', '...   \n",
       "\n",
       "                                                crew tmdb_id  cast_size  \\\n",
       "0  [{'credit_id': '52fe4284c3a36847f8024f49', 'de...     862         13   \n",
       "1  [{'credit_id': '52fe44bfc3a36847f80a7cd1', 'de...    8844         26   \n",
       "\n",
       "   crew_size  \n",
       "0        106  \n",
       "1         16  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "credits.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define a function to get the director name\n",
    "def get_director(x):\n",
    "    for i in x:\n",
    "        if i['job'] == 'Director':\n",
    "            return i['name']\n",
    "    return np.nan"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "credits['director'] = credits['crew'].apply(get_director)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "credits['cast'] = credits['cast'].apply(lambda x: [i['name'] for i in x] if isinstance(x, list) else [])\n",
    "credits['cast'] = credits['cast'].apply(lambda x: x[:3] if len(x) >=3 else x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>cast</th>\n",
       "      <th>crew</th>\n",
       "      <th>tmdb_id</th>\n",
       "      <th>cast_size</th>\n",
       "      <th>crew_size</th>\n",
       "      <th>director</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>[Tom Hanks, Tim Allen, Don Rickles]</td>\n",
       "      <td>[{'credit_id': '52fe4284c3a36847f8024f49', 'de...</td>\n",
       "      <td>862</td>\n",
       "      <td>13</td>\n",
       "      <td>106</td>\n",
       "      <td>John Lasseter</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>[Robin Williams, Jonathan Hyde, Kirsten Dunst]</td>\n",
       "      <td>[{'credit_id': '52fe44bfc3a36847f80a7cd1', 'de...</td>\n",
       "      <td>8844</td>\n",
       "      <td>26</td>\n",
       "      <td>16</td>\n",
       "      <td>Joe Johnston</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                             cast  \\\n",
       "0             [Tom Hanks, Tim Allen, Don Rickles]   \n",
       "1  [Robin Williams, Jonathan Hyde, Kirsten Dunst]   \n",
       "\n",
       "                                                crew tmdb_id  cast_size  \\\n",
       "0  [{'credit_id': '52fe4284c3a36847f8024f49', 'de...     862         13   \n",
       "1  [{'credit_id': '52fe44bfc3a36847f80a7cd1', 'de...    8844         26   \n",
       "\n",
       "   crew_size       director  \n",
       "0        106  John Lasseter  \n",
       "1         16   Joe Johnston  "
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "creditsTr = credits\n",
    "creditsTr.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.2.4 movies_metadata.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>adult</th>\n",
       "      <th>belongs_to_collection</th>\n",
       "      <th>budget</th>\n",
       "      <th>genres</th>\n",
       "      <th>homepage</th>\n",
       "      <th>id</th>\n",
       "      <th>imdb_id</th>\n",
       "      <th>original_language</th>\n",
       "      <th>original_title</th>\n",
       "      <th>overview</th>\n",
       "      <th>...</th>\n",
       "      <th>release_date</th>\n",
       "      <th>revenue</th>\n",
       "      <th>runtime</th>\n",
       "      <th>spoken_languages</th>\n",
       "      <th>status</th>\n",
       "      <th>tagline</th>\n",
       "      <th>title</th>\n",
       "      <th>video</th>\n",
       "      <th>vote_average</th>\n",
       "      <th>vote_count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "      <td>{'id': 10194, 'name': 'Toy Story Collection', ...</td>\n",
       "      <td>30000000</td>\n",
       "      <td>[{'id': 16, 'name': 'Animation'}, {'id': 35, '...</td>\n",
       "      <td>http://toystory.disney.com/toy-story</td>\n",
       "      <td>862</td>\n",
       "      <td>tt0114709</td>\n",
       "      <td>en</td>\n",
       "      <td>Toy Story</td>\n",
       "      <td>Led by Woody, Andy's toys live happily in his ...</td>\n",
       "      <td>...</td>\n",
       "      <td>1995-10-30</td>\n",
       "      <td>373554033.0</td>\n",
       "      <td>81.0</td>\n",
       "      <td>[{'iso_639_1': 'en', 'name': 'English'}]</td>\n",
       "      <td>Released</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Toy Story</td>\n",
       "      <td>False</td>\n",
       "      <td>7.7</td>\n",
       "      <td>5415.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>65000000</td>\n",
       "      <td>[{'id': 12, 'name': 'Adventure'}, {'id': 14, '...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8844</td>\n",
       "      <td>tt0113497</td>\n",
       "      <td>en</td>\n",
       "      <td>Jumanji</td>\n",
       "      <td>When siblings Judy and Peter discover an encha...</td>\n",
       "      <td>...</td>\n",
       "      <td>1995-12-15</td>\n",
       "      <td>262797249.0</td>\n",
       "      <td>104.0</td>\n",
       "      <td>[{'iso_639_1': 'en', 'name': 'English'}, {'iso...</td>\n",
       "      <td>Released</td>\n",
       "      <td>Roll the dice and unleash the excitement!</td>\n",
       "      <td>Jumanji</td>\n",
       "      <td>False</td>\n",
       "      <td>6.9</td>\n",
       "      <td>2413.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2 rows × 24 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   adult                              belongs_to_collection    budget  \\\n",
       "0  False  {'id': 10194, 'name': 'Toy Story Collection', ...  30000000   \n",
       "1  False                                                NaN  65000000   \n",
       "\n",
       "                                              genres  \\\n",
       "0  [{'id': 16, 'name': 'Animation'}, {'id': 35, '...   \n",
       "1  [{'id': 12, 'name': 'Adventure'}, {'id': 14, '...   \n",
       "\n",
       "                               homepage    id    imdb_id original_language  \\\n",
       "0  http://toystory.disney.com/toy-story   862  tt0114709                en   \n",
       "1                                   NaN  8844  tt0113497                en   \n",
       "\n",
       "  original_title                                           overview  ...  \\\n",
       "0      Toy Story  Led by Woody, Andy's toys live happily in his ...  ...   \n",
       "1        Jumanji  When siblings Judy and Peter discover an encha...  ...   \n",
       "\n",
       "  release_date      revenue runtime  \\\n",
       "0   1995-10-30  373554033.0    81.0   \n",
       "1   1995-12-15  262797249.0   104.0   \n",
       "\n",
       "                                    spoken_languages    status  \\\n",
       "0           [{'iso_639_1': 'en', 'name': 'English'}]  Released   \n",
       "1  [{'iso_639_1': 'en', 'name': 'English'}, {'iso...  Released   \n",
       "\n",
       "                                     tagline      title  video vote_average  \\\n",
       "0                                        NaN  Toy Story  False          7.7   \n",
       "1  Roll the dice and unleash the excitement!    Jumanji  False          6.9   \n",
       "\n",
       "  vote_count  \n",
       "0     5415.0  \n",
       "1     2413.0  \n",
       "\n",
       "[2 rows x 24 columns]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movies_metadata= pd.read_csv(r'data\\themoviesdataset\\movies_metadata.csv',low_memory=False)\n",
    "movies_metadata.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the origin dataset, 24 columns are provided. We only need 15 of them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(45466, 15)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>genres</th>\n",
       "      <th>id</th>\n",
       "      <th>imdb_id</th>\n",
       "      <th>overview</th>\n",
       "      <th>popularity</th>\n",
       "      <th>production_companies</th>\n",
       "      <th>production_countries</th>\n",
       "      <th>release_date</th>\n",
       "      <th>revenue</th>\n",
       "      <th>runtime</th>\n",
       "      <th>spoken_languages</th>\n",
       "      <th>status</th>\n",
       "      <th>title</th>\n",
       "      <th>vote_average</th>\n",
       "      <th>vote_count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>[{'id': 16, 'name': 'Animation'}, {'id': 35, '...</td>\n",
       "      <td>862</td>\n",
       "      <td>tt0114709</td>\n",
       "      <td>Led by Woody, Andy's toys live happily in his ...</td>\n",
       "      <td>21.946943</td>\n",
       "      <td>[{'name': 'Pixar Animation Studios', 'id': 3}]</td>\n",
       "      <td>[{'iso_3166_1': 'US', 'name': 'United States o...</td>\n",
       "      <td>1995-10-30</td>\n",
       "      <td>373554033.0</td>\n",
       "      <td>81.0</td>\n",
       "      <td>[{'iso_639_1': 'en', 'name': 'English'}]</td>\n",
       "      <td>Released</td>\n",
       "      <td>Toy Story</td>\n",
       "      <td>7.7</td>\n",
       "      <td>5415.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>[{'id': 12, 'name': 'Adventure'}, {'id': 14, '...</td>\n",
       "      <td>8844</td>\n",
       "      <td>tt0113497</td>\n",
       "      <td>When siblings Judy and Peter discover an encha...</td>\n",
       "      <td>17.015539</td>\n",
       "      <td>[{'name': 'TriStar Pictures', 'id': 559}, {'na...</td>\n",
       "      <td>[{'iso_3166_1': 'US', 'name': 'United States o...</td>\n",
       "      <td>1995-12-15</td>\n",
       "      <td>262797249.0</td>\n",
       "      <td>104.0</td>\n",
       "      <td>[{'iso_639_1': 'en', 'name': 'English'}, {'iso...</td>\n",
       "      <td>Released</td>\n",
       "      <td>Jumanji</td>\n",
       "      <td>6.9</td>\n",
       "      <td>2413.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              genres    id    imdb_id  \\\n",
       "0  [{'id': 16, 'name': 'Animation'}, {'id': 35, '...   862  tt0114709   \n",
       "1  [{'id': 12, 'name': 'Adventure'}, {'id': 14, '...  8844  tt0113497   \n",
       "\n",
       "                                            overview popularity  \\\n",
       "0  Led by Woody, Andy's toys live happily in his ...  21.946943   \n",
       "1  When siblings Judy and Peter discover an encha...  17.015539   \n",
       "\n",
       "                                production_companies  \\\n",
       "0     [{'name': 'Pixar Animation Studios', 'id': 3}]   \n",
       "1  [{'name': 'TriStar Pictures', 'id': 559}, {'na...   \n",
       "\n",
       "                                production_countries release_date  \\\n",
       "0  [{'iso_3166_1': 'US', 'name': 'United States o...   1995-10-30   \n",
       "1  [{'iso_3166_1': 'US', 'name': 'United States o...   1995-12-15   \n",
       "\n",
       "       revenue  runtime                                   spoken_languages  \\\n",
       "0  373554033.0     81.0           [{'iso_639_1': 'en', 'name': 'English'}]   \n",
       "1  262797249.0    104.0  [{'iso_639_1': 'en', 'name': 'English'}, {'iso...   \n",
       "\n",
       "     status      title  vote_average  vote_count  \n",
       "0  Released  Toy Story           7.7      5415.0  \n",
       "1  Released    Jumanji           6.9      2413.0  "
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movies_metadataTr = movies_metadata[['genres','id','imdb_id','overview','popularity',\n",
    "                                 'production_companies','production_countries','release_date',\n",
    "                                 'revenue','runtime','spoken_languages','status',\n",
    "                                 'title','vote_average','vote_count']]\n",
    "print(movies_metadataTr.shape)\n",
    "movies_metadataTr.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Rename the column id to tmdb_id\n",
    "movies_metadataTr.rename(columns = {'id':'tmdb_id'},inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>genres</th>\n",
       "      <th>tmdb_id</th>\n",
       "      <th>imdb_id</th>\n",
       "      <th>overview</th>\n",
       "      <th>popularity</th>\n",
       "      <th>production_companies</th>\n",
       "      <th>production_countries</th>\n",
       "      <th>release_date</th>\n",
       "      <th>revenue</th>\n",
       "      <th>runtime</th>\n",
       "      <th>spoken_languages</th>\n",
       "      <th>status</th>\n",
       "      <th>title</th>\n",
       "      <th>vote_average</th>\n",
       "      <th>vote_count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>[{'id': 16, 'name': 'Animation'}, {'id': 35, '...</td>\n",
       "      <td>862</td>\n",
       "      <td>tt0114709</td>\n",
       "      <td>Led by Woody, Andy's toys live happily in his ...</td>\n",
       "      <td>21.946943</td>\n",
       "      <td>[{'name': 'Pixar Animation Studios', 'id': 3}]</td>\n",
       "      <td>[{'iso_3166_1': 'US', 'name': 'United States o...</td>\n",
       "      <td>1995-10-30</td>\n",
       "      <td>373554033.0</td>\n",
       "      <td>81.0</td>\n",
       "      <td>[{'iso_639_1': 'en', 'name': 'English'}]</td>\n",
       "      <td>Released</td>\n",
       "      <td>Toy Story</td>\n",
       "      <td>7.7</td>\n",
       "      <td>5415.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>[{'id': 12, 'name': 'Adventure'}, {'id': 14, '...</td>\n",
       "      <td>8844</td>\n",
       "      <td>tt0113497</td>\n",
       "      <td>When siblings Judy and Peter discover an encha...</td>\n",
       "      <td>17.015539</td>\n",
       "      <td>[{'name': 'TriStar Pictures', 'id': 559}, {'na...</td>\n",
       "      <td>[{'iso_3166_1': 'US', 'name': 'United States o...</td>\n",
       "      <td>1995-12-15</td>\n",
       "      <td>262797249.0</td>\n",
       "      <td>104.0</td>\n",
       "      <td>[{'iso_639_1': 'en', 'name': 'English'}, {'iso...</td>\n",
       "      <td>Released</td>\n",
       "      <td>Jumanji</td>\n",
       "      <td>6.9</td>\n",
       "      <td>2413.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              genres tmdb_id    imdb_id  \\\n",
       "0  [{'id': 16, 'name': 'Animation'}, {'id': 35, '...     862  tt0114709   \n",
       "1  [{'id': 12, 'name': 'Adventure'}, {'id': 14, '...    8844  tt0113497   \n",
       "\n",
       "                                            overview popularity  \\\n",
       "0  Led by Woody, Andy's toys live happily in his ...  21.946943   \n",
       "1  When siblings Judy and Peter discover an encha...  17.015539   \n",
       "\n",
       "                                production_companies  \\\n",
       "0     [{'name': 'Pixar Animation Studios', 'id': 3}]   \n",
       "1  [{'name': 'TriStar Pictures', 'id': 559}, {'na...   \n",
       "\n",
       "                                production_countries release_date  \\\n",
       "0  [{'iso_3166_1': 'US', 'name': 'United States o...   1995-10-30   \n",
       "1  [{'iso_3166_1': 'US', 'name': 'United States o...   1995-12-15   \n",
       "\n",
       "       revenue  runtime                                   spoken_languages  \\\n",
       "0  373554033.0     81.0           [{'iso_639_1': 'en', 'name': 'English'}]   \n",
       "1  262797249.0    104.0  [{'iso_639_1': 'en', 'name': 'English'}, {'iso...   \n",
       "\n",
       "     status      title  vote_average  vote_count  \n",
       "0  Released  Toy Story           7.7      5415.0  \n",
       "1  Released    Jumanji           6.9      2413.0  "
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movies_metadataTr.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.3 Merge the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3.1 Movie"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Merge the `movies_metadataTr`, `creditsTr`, and `keywordsTr.csv`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Merge the movies_metadata.csv, credits.csv, and keywords.csv\n",
    "metadataTr_creditsTr = pd.merge(movies_metadataTr, \n",
    "                   creditsTr, \n",
    "                   how = 'left', \n",
    "                   on = ['tmdb_id'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "metadataTr_creditsTr_keywordsTr = pd.merge(metadataTr_creditsTr, \n",
    "                                           keywordsTr, \n",
    "                                           how = 'left', \n",
    "                                           on = ['tmdb_id'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>genres</th>\n",
       "      <th>tmdb_id</th>\n",
       "      <th>imdb_id</th>\n",
       "      <th>overview</th>\n",
       "      <th>popularity</th>\n",
       "      <th>production_companies</th>\n",
       "      <th>production_countries</th>\n",
       "      <th>release_date</th>\n",
       "      <th>revenue</th>\n",
       "      <th>runtime</th>\n",
       "      <th>...</th>\n",
       "      <th>status</th>\n",
       "      <th>title</th>\n",
       "      <th>vote_average</th>\n",
       "      <th>vote_count</th>\n",
       "      <th>cast</th>\n",
       "      <th>crew</th>\n",
       "      <th>cast_size</th>\n",
       "      <th>crew_size</th>\n",
       "      <th>director</th>\n",
       "      <th>keywordsTr</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>[{'id': 16, 'name': 'Animation'}, {'id': 35, '...</td>\n",
       "      <td>862</td>\n",
       "      <td>tt0114709</td>\n",
       "      <td>Led by Woody, Andy's toys live happily in his ...</td>\n",
       "      <td>21.946943</td>\n",
       "      <td>[{'name': 'Pixar Animation Studios', 'id': 3}]</td>\n",
       "      <td>[{'iso_3166_1': 'US', 'name': 'United States o...</td>\n",
       "      <td>1995-10-30</td>\n",
       "      <td>373554033.0</td>\n",
       "      <td>81.0</td>\n",
       "      <td>...</td>\n",
       "      <td>Released</td>\n",
       "      <td>Toy Story</td>\n",
       "      <td>7.7</td>\n",
       "      <td>5415.0</td>\n",
       "      <td>[Tom Hanks, Tim Allen, Don Rickles]</td>\n",
       "      <td>[{'credit_id': '52fe4284c3a36847f8024f49', 'de...</td>\n",
       "      <td>13.0</td>\n",
       "      <td>106.0</td>\n",
       "      <td>John Lasseter</td>\n",
       "      <td>[jealousy, toy, boy, friendship, friends, riva...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>[{'id': 12, 'name': 'Adventure'}, {'id': 14, '...</td>\n",
       "      <td>8844</td>\n",
       "      <td>tt0113497</td>\n",
       "      <td>When siblings Judy and Peter discover an encha...</td>\n",
       "      <td>17.015539</td>\n",
       "      <td>[{'name': 'TriStar Pictures', 'id': 559}, {'na...</td>\n",
       "      <td>[{'iso_3166_1': 'US', 'name': 'United States o...</td>\n",
       "      <td>1995-12-15</td>\n",
       "      <td>262797249.0</td>\n",
       "      <td>104.0</td>\n",
       "      <td>...</td>\n",
       "      <td>Released</td>\n",
       "      <td>Jumanji</td>\n",
       "      <td>6.9</td>\n",
       "      <td>2413.0</td>\n",
       "      <td>[Robin Williams, Jonathan Hyde, Kirsten Dunst]</td>\n",
       "      <td>[{'credit_id': '52fe44bfc3a36847f80a7cd1', 'de...</td>\n",
       "      <td>26.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>Joe Johnston</td>\n",
       "      <td>[board game, disappearance, based on children'...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2 rows × 21 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              genres tmdb_id    imdb_id  \\\n",
       "0  [{'id': 16, 'name': 'Animation'}, {'id': 35, '...     862  tt0114709   \n",
       "1  [{'id': 12, 'name': 'Adventure'}, {'id': 14, '...    8844  tt0113497   \n",
       "\n",
       "                                            overview popularity  \\\n",
       "0  Led by Woody, Andy's toys live happily in his ...  21.946943   \n",
       "1  When siblings Judy and Peter discover an encha...  17.015539   \n",
       "\n",
       "                                production_companies  \\\n",
       "0     [{'name': 'Pixar Animation Studios', 'id': 3}]   \n",
       "1  [{'name': 'TriStar Pictures', 'id': 559}, {'na...   \n",
       "\n",
       "                                production_countries release_date  \\\n",
       "0  [{'iso_3166_1': 'US', 'name': 'United States o...   1995-10-30   \n",
       "1  [{'iso_3166_1': 'US', 'name': 'United States o...   1995-12-15   \n",
       "\n",
       "       revenue  runtime  ...    status      title vote_average  vote_count  \\\n",
       "0  373554033.0     81.0  ...  Released  Toy Story          7.7      5415.0   \n",
       "1  262797249.0    104.0  ...  Released    Jumanji          6.9      2413.0   \n",
       "\n",
       "                                             cast  \\\n",
       "0             [Tom Hanks, Tim Allen, Don Rickles]   \n",
       "1  [Robin Williams, Jonathan Hyde, Kirsten Dunst]   \n",
       "\n",
       "                                                crew cast_size  crew_size  \\\n",
       "0  [{'credit_id': '52fe4284c3a36847f8024f49', 'de...      13.0      106.0   \n",
       "1  [{'credit_id': '52fe44bfc3a36847f80a7cd1', 'de...      26.0       16.0   \n",
       "\n",
       "        director                                         keywordsTr  \n",
       "0  John Lasseter  [jealousy, toy, boy, friendship, friends, riva...  \n",
       "1   Joe Johnston  [board game, disappearance, based on children'...  \n",
       "\n",
       "[2 rows x 21 columns]"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metadataTr_creditsTr_keywordsTr.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Select rows where the column status == 'Released'\n",
    "data_movies = metadataTr_creditsTr_keywordsTr[metadataTr_creditsTr_keywordsTr['status'] == 'Released']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fill NA by the space\n",
    "data_movies = cleaned_data.fillna(' ')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>genres</th>\n",
       "      <th>tmdb_id</th>\n",
       "      <th>imdb_id</th>\n",
       "      <th>overview</th>\n",
       "      <th>popularity</th>\n",
       "      <th>production_companies</th>\n",
       "      <th>production_countries</th>\n",
       "      <th>release_date</th>\n",
       "      <th>revenue</th>\n",
       "      <th>runtime</th>\n",
       "      <th>...</th>\n",
       "      <th>status</th>\n",
       "      <th>title</th>\n",
       "      <th>vote_average</th>\n",
       "      <th>vote_count</th>\n",
       "      <th>cast</th>\n",
       "      <th>crew</th>\n",
       "      <th>cast_size</th>\n",
       "      <th>crew_size</th>\n",
       "      <th>director</th>\n",
       "      <th>keywordsTr</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>[{'id': 16, 'name': 'Animation'}, {'id': 35, '...</td>\n",
       "      <td>862</td>\n",
       "      <td>tt0114709</td>\n",
       "      <td>Led by Woody, Andy's toys live happily in his ...</td>\n",
       "      <td>21.946943</td>\n",
       "      <td>[{'name': 'Pixar Animation Studios', 'id': 3}]</td>\n",
       "      <td>[{'iso_3166_1': 'US', 'name': 'United States o...</td>\n",
       "      <td>1995-10-30</td>\n",
       "      <td>373554033.0</td>\n",
       "      <td>81</td>\n",
       "      <td>...</td>\n",
       "      <td>Released</td>\n",
       "      <td>Toy Story</td>\n",
       "      <td>7.7</td>\n",
       "      <td>5415.0</td>\n",
       "      <td>[Tom Hanks, Tim Allen, Don Rickles]</td>\n",
       "      <td>[{'credit_id': '52fe4284c3a36847f8024f49', 'de...</td>\n",
       "      <td>13</td>\n",
       "      <td>106</td>\n",
       "      <td>John Lasseter</td>\n",
       "      <td>[jealousy, toy, boy, friendship, friends, riva...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>[{'id': 12, 'name': 'Adventure'}, {'id': 14, '...</td>\n",
       "      <td>8844</td>\n",
       "      <td>tt0113497</td>\n",
       "      <td>When siblings Judy and Peter discover an encha...</td>\n",
       "      <td>17.015539</td>\n",
       "      <td>[{'name': 'TriStar Pictures', 'id': 559}, {'na...</td>\n",
       "      <td>[{'iso_3166_1': 'US', 'name': 'United States o...</td>\n",
       "      <td>1995-12-15</td>\n",
       "      <td>262797249.0</td>\n",
       "      <td>104</td>\n",
       "      <td>...</td>\n",
       "      <td>Released</td>\n",
       "      <td>Jumanji</td>\n",
       "      <td>6.9</td>\n",
       "      <td>2413.0</td>\n",
       "      <td>[Robin Williams, Jonathan Hyde, Kirsten Dunst]</td>\n",
       "      <td>[{'credit_id': '52fe44bfc3a36847f80a7cd1', 'de...</td>\n",
       "      <td>26</td>\n",
       "      <td>16</td>\n",
       "      <td>Joe Johnston</td>\n",
       "      <td>[board game, disappearance, based on children'...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2 rows × 21 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              genres tmdb_id    imdb_id  \\\n",
       "0  [{'id': 16, 'name': 'Animation'}, {'id': 35, '...     862  tt0114709   \n",
       "1  [{'id': 12, 'name': 'Adventure'}, {'id': 14, '...    8844  tt0113497   \n",
       "\n",
       "                                            overview popularity  \\\n",
       "0  Led by Woody, Andy's toys live happily in his ...  21.946943   \n",
       "1  When siblings Judy and Peter discover an encha...  17.015539   \n",
       "\n",
       "                                production_companies  \\\n",
       "0     [{'name': 'Pixar Animation Studios', 'id': 3}]   \n",
       "1  [{'name': 'TriStar Pictures', 'id': 559}, {'na...   \n",
       "\n",
       "                                production_countries release_date  \\\n",
       "0  [{'iso_3166_1': 'US', 'name': 'United States o...   1995-10-30   \n",
       "1  [{'iso_3166_1': 'US', 'name': 'United States o...   1995-12-15   \n",
       "\n",
       "       revenue runtime  ...    status      title vote_average  vote_count  \\\n",
       "0  373554033.0      81  ...  Released  Toy Story          7.7      5415.0   \n",
       "1  262797249.0     104  ...  Released    Jumanji          6.9      2413.0   \n",
       "\n",
       "                                             cast  \\\n",
       "0             [Tom Hanks, Tim Allen, Don Rickles]   \n",
       "1  [Robin Williams, Jonathan Hyde, Kirsten Dunst]   \n",
       "\n",
       "                                                crew cast_size crew_size  \\\n",
       "0  [{'credit_id': '52fe4284c3a36847f8024f49', 'de...        13       106   \n",
       "1  [{'credit_id': '52fe44bfc3a36847f80a7cd1', 'de...        26        16   \n",
       "\n",
       "        director                                         keywordsTr  \n",
       "0  John Lasseter  [jealousy, toy, boy, friendship, friends, riva...  \n",
       "1   Joe Johnston  [board game, disappearance, based on children'...  \n",
       "\n",
       "[2 rows x 21 columns]"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_movies.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Genre, overview, Keywords"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3.2 User"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "tags = pd.read_csv('data/ml-25m/tags.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1093360, 4)"
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tags.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>movieId</th>\n",
       "      <th>tag</th>\n",
       "      <th>timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>1732</td>\n",
       "      <td>dark comedy</td>\n",
       "      <td>1573943598</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>1732</td>\n",
       "      <td>great dialogue</td>\n",
       "      <td>1573943604</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>7569</td>\n",
       "      <td>so bad it's good</td>\n",
       "      <td>1573943455</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>44665</td>\n",
       "      <td>unreliable narrators</td>\n",
       "      <td>1573943619</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>4</td>\n",
       "      <td>115569</td>\n",
       "      <td>tense</td>\n",
       "      <td>1573943077</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>4</td>\n",
       "      <td>115713</td>\n",
       "      <td>artificial intelligence</td>\n",
       "      <td>1573942979</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>4</td>\n",
       "      <td>115713</td>\n",
       "      <td>philosophical</td>\n",
       "      <td>1573943033</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>4</td>\n",
       "      <td>115713</td>\n",
       "      <td>tense</td>\n",
       "      <td>1573943042</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>4</td>\n",
       "      <td>148426</td>\n",
       "      <td>so bad it's good</td>\n",
       "      <td>1573942965</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>4</td>\n",
       "      <td>164909</td>\n",
       "      <td>cliche</td>\n",
       "      <td>1573943721</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>4</td>\n",
       "      <td>164909</td>\n",
       "      <td>musical</td>\n",
       "      <td>1573943714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>4</td>\n",
       "      <td>168250</td>\n",
       "      <td>horror</td>\n",
       "      <td>1573945163</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>4</td>\n",
       "      <td>168250</td>\n",
       "      <td>unpredictable</td>\n",
       "      <td>1573945171</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    userId  movieId                      tag   timestamp\n",
       "2        4     1732              dark comedy  1573943598\n",
       "3        4     1732           great dialogue  1573943604\n",
       "4        4     7569         so bad it's good  1573943455\n",
       "5        4    44665     unreliable narrators  1573943619\n",
       "6        4   115569                    tense  1573943077\n",
       "7        4   115713  artificial intelligence  1573942979\n",
       "8        4   115713            philosophical  1573943033\n",
       "9        4   115713                    tense  1573943042\n",
       "10       4   148426         so bad it's good  1573942965\n",
       "11       4   164909                   cliche  1573943721\n",
       "12       4   164909                  musical  1573943714\n",
       "13       4   168250                   horror  1573945163\n",
       "14       4   168250            unpredictable  1573945171"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tags[tags['userId']==4]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>movieId</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>59</td>\n",
       "      <td>4</td>\n",
       "      <td>223</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1042668576</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>60</td>\n",
       "      <td>4</td>\n",
       "      <td>415</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1042667925</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>61</td>\n",
       "      <td>4</td>\n",
       "      <td>648</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1042674800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>62</td>\n",
       "      <td>4</td>\n",
       "      <td>1097</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1042667925</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>63</td>\n",
       "      <td>4</td>\n",
       "      <td>1197</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1042667956</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>116</td>\n",
       "      <td>4</td>\n",
       "      <td>4226</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1042674412</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>117</td>\n",
       "      <td>4</td>\n",
       "      <td>4238</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1042674819</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>118</td>\n",
       "      <td>4</td>\n",
       "      <td>4848</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1042674425</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>119</td>\n",
       "      <td>4</td>\n",
       "      <td>4896</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1042667845</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>120</td>\n",
       "      <td>4</td>\n",
       "      <td>5679</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1042674579</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>62 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     userId  movieId  rating   timestamp\n",
       "59        4      223     4.0  1042668576\n",
       "60        4      415     4.0  1042667925\n",
       "61        4      648     4.0  1042674800\n",
       "62        4     1097     5.0  1042667925\n",
       "63        4     1197     4.0  1042667956\n",
       "..      ...      ...     ...         ...\n",
       "116       4     4226     3.0  1042674412\n",
       "117       4     4238     4.0  1042674819\n",
       "118       4     4848     3.0  1042674425\n",
       "119       4     4896     4.0  1042667845\n",
       "120       4     5679     3.0  1042674579\n",
       "\n",
       "[62 rows x 4 columns]"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratingTr[ratingTr['userId']==4]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_user = pd.merge(ratingTr, tags, how = 'left', on = ['userId','movieId','timestamp'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Series([], Name: tag, dtype: int64)"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_user['tag'].value_counts()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:root] *",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}