Movie_recommendation_engine.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Build movie recommendation system"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://raw.githubusercontent.com/jalajthanaki/Movie_recommendation_engine/master/img/3_1.gif\" \n",
    "align=\"middle\" alt=\"Figure 3_1\" data-canonical-src=\"\" style=\"width:40%;height:40%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "1. Import our dependencies\n",
    "2. Load dataset\n",
    "3. Understand dataset\n",
    "4. Data Wrangling (Already done)\n",
    "5. Pre-processing steps (We will perform the pre-processing steps as and when needed)\n",
    "6. Build recommendation system\n",
    "   6.1 Simple recommendation system\n",
    "        6.1.1 Implement model\n",
    "        6.1.2 Evaluate Result\n",
    "    6.2 Content based recommendation system\n",
    "        6.2.1 Implement model\n",
    "        6.2.2 Evaluate Result\n",
    "    6.3 CF based recommendation system\n",
    "        6.3.1 Implement model\n",
    "        6.3.2 Evaluate Result\n",
    "    6.4 Hybrid recommendation system\n",
    "        6.4.1 Implement model\n",
    "        6.4.2 Evaluate Result\n",
    "```   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Import libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import ast \n",
    "from scipy import stats\n",
    "from ast import literal_eval\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer\n",
    "from sklearn.metrics.pairwise import linear_kernel, cosine_similarity\n",
    "from nltk.stem.snowball import SnowballStemmer\n",
    "from nltk.stem.wordnet import WordNetLemmatizer\n",
    "from nltk.corpus import wordnet\n",
    "from surprise import Reader, Dataset, SVD, evaluate\n",
    "\n",
    "import warnings; warnings.simplefilter('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Load dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have MovieLens datasets.\n",
    "\n",
    "**The Full Dataset:** Consists of 26,000,000 ratings and 750,000 tag applications applied to 45,000 movies by 270,000 users. Includes tag genome data with 12 million relevance scores across 1,100 tags.\n",
    "\n",
    "**The Small Dataset:** Comprises of 100,000 ratings and 1,300 tag applications applied to 9,000 movies by 700 users.\n",
    "\n",
    "We will build our Simple Recommender using movies from the Full Dataset "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "credits = pd.read_csv('./input_data/movie_dataset/credits.csv')\n",
    "keywords = pd.read_csv('./input_data/movie_dataset/keywords.csv')\n",
    "links_small = pd.read_csv('./input_data/movie_dataset/links_small.csv')\n",
    "md = pd.read_csv('./input_data/movie_dataset/movies_metadata.csv')\n",
    "ratings = pd.read_csv('./input_data/movie_dataset/ratings_small.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Understand dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Credits dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>cast</th>\n",
       "      <th>crew</th>\n",
       "      <th>id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[{'cast_id': 14, 'character': 'Woody (voice)',...</td>\n",
       "      <td>[{'credit_id': '52fe4284c3a36847f8024f49', 'de...</td>\n",
       "      <td>862</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[{'cast_id': 1, 'character': 'Alan Parrish', '...</td>\n",
       "      <td>[{'credit_id': '52fe44bfc3a36847f80a7cd1', 'de...</td>\n",
       "      <td>8844</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[{'cast_id': 2, 'character': 'Max Goldman', 'c...</td>\n",
       "      <td>[{'credit_id': '52fe466a9251416c75077a89', 'de...</td>\n",
       "      <td>15602</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>[{'cast_id': 1, 'character': \"Savannah 'Vannah...</td>\n",
       "      <td>[{'credit_id': '52fe44779251416c91011acb', 'de...</td>\n",
       "      <td>31357</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[{'cast_id': 1, 'character': 'George Banks', '...</td>\n",
       "      <td>[{'credit_id': '52fe44959251416c75039ed7', 'de...</td>\n",
       "      <td>11862</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                cast  \\\n",
       "0  [{'cast_id': 14, 'character': 'Woody (voice)',...   \n",
       "1  [{'cast_id': 1, 'character': 'Alan Parrish', '...   \n",
       "2  [{'cast_id': 2, 'character': 'Max Goldman', 'c...   \n",
       "3  [{'cast_id': 1, 'character': \"Savannah 'Vannah...   \n",
       "4  [{'cast_id': 1, 'character': 'George Banks', '...   \n",
       "\n",
       "                                                crew     id  \n",
       "0  [{'credit_id': '52fe4284c3a36847f8024f49', 'de...    862  \n",
       "1  [{'credit_id': '52fe44bfc3a36847f80a7cd1', 'de...   8844  \n",
       "2  [{'credit_id': '52fe466a9251416c75077a89', 'de...  15602  \n",
       "3  [{'credit_id': '52fe44779251416c91011acb', 'de...  31357  \n",
       "4  [{'credit_id': '52fe44959251416c75039ed7', 'de...  11862  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "credits.head()\n",
    "#credits.iloc[0:3]\n",
    "#credits['cast'].iloc[0:3]\n",
    "#credits.iloc[:,0:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['cast', 'crew', 'id'], dtype='object')"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "credits.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* **cast:** Information about casting. Name of actor, gender and it's character name in movie\n",
    "* **crew:** Information about crew members. Like who directed the movie, editor of the movie and so on. \n",
    "* **id:** It's movie ID given by TMDb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(45476, 3)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "credits.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 45476 entries, 0 to 45475\n",
      "Data columns (total 3 columns):\n",
      "cast    45476 non-null object\n",
      "crew    45476 non-null object\n",
      "id      45476 non-null int64\n",
      "dtypes: int64(1), object(2)\n",
      "memory usage: 1.0+ MB\n"
     ]
    }
   ],
   "source": [
    "credits.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Keywords dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>keywords</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>862</td>\n",
       "      <td>[{'id': 931, 'name': 'jealousy'}, {'id': 4290,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>8844</td>\n",
       "      <td>[{'id': 10090, 'name': 'board game'}, {'id': 1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>15602</td>\n",
       "      <td>[{'id': 1495, 'name': 'fishing'}, {'id': 12392...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>31357</td>\n",
       "      <td>[{'id': 818, 'name': 'based on novel'}, {'id':...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>11862</td>\n",
       "      <td>[{'id': 1009, 'name': 'baby'}, {'id': 1599, 'n...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      id                                           keywords\n",
       "0    862  [{'id': 931, 'name': 'jealousy'}, {'id': 4290,...\n",
       "1   8844  [{'id': 10090, 'name': 'board game'}, {'id': 1...\n",
       "2  15602  [{'id': 1495, 'name': 'fishing'}, {'id': 12392...\n",
       "3  31357  [{'id': 818, 'name': 'based on novel'}, {'id':...\n",
       "4  11862  [{'id': 1009, 'name': 'baby'}, {'id': 1599, 'n..."
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keywords.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['id', 'keywords'], dtype='object')"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keywords.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* **id:** It's movie ID given by TMDb\n",
    "* **Keywords:** Tags/keywords for the movie. It list of tags/keywords "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(46419, 2)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keywords.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 46419 entries, 0 to 46418\n",
      "Data columns (total 2 columns):\n",
      "id          46419 non-null int64\n",
      "keywords    46419 non-null object\n",
      "dtypes: int64(1), object(1)\n",
      "memory usage: 725.4+ KB\n"
     ]
    }
   ],
   "source": [
    "keywords.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Link dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>movieId</th>\n",
       "      <th>imdbId</th>\n",
       "      <th>tmdbId</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>114709</td>\n",
       "      <td>862.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>113497</td>\n",
       "      <td>8844.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>113228</td>\n",
       "      <td>15602.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>114885</td>\n",
       "      <td>31357.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>113041</td>\n",
       "      <td>11862.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   movieId  imdbId   tmdbId\n",
       "0        1  114709    862.0\n",
       "1        2  113497   8844.0\n",
       "2        3  113228  15602.0\n",
       "3        4  114885  31357.0\n",
       "4        5  113041  11862.0"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "links_small.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['movieId', 'imdbId', 'tmdbId'], dtype='object')"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "links_small.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* **movieId:** It's serial number for movie\n",
    "* **imdbId:** Movie id given on IMDb platform\n",
    "* **tmdbId**: Movie id given on TMDb platform"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(9125, 3)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "links_small.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 9125 entries, 0 to 9124\n",
      "Data columns (total 3 columns):\n",
      "movieId    9125 non-null int64\n",
      "imdbId     9125 non-null int64\n",
      "tmdbId     9112 non-null float64\n",
      "dtypes: float64(1), int64(2)\n",
      "memory usage: 213.9 KB\n"
     ]
    }
   ],
   "source": [
    "links_small.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Metadata dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>adult</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>belongs_to_collection</th>\n",
       "      <td>{'id': 10194, 'name': 'Toy Story Collection', ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>{'id': 119050, 'name': 'Grumpy Old Men Collect...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>budget</th>\n",
       "      <td>30000000</td>\n",
       "      <td>65000000</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>genres</th>\n",
       "      <td>[{'id': 16, 'name': 'Animation'}, {'id': 35, '...</td>\n",
       "      <td>[{'id': 12, 'name': 'Adventure'}, {'id': 14, '...</td>\n",
       "      <td>[{'id': 10749, 'name': 'Romance'}, {'id': 35, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>homepage</th>\n",
       "      <td>http://toystory.disney.com/toy-story</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <td>862</td>\n",
       "      <td>8844</td>\n",
       "      <td>15602</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>imdb_id</th>\n",
       "      <td>tt0114709</td>\n",
       "      <td>tt0113497</td>\n",
       "      <td>tt0113228</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>original_language</th>\n",
       "      <td>en</td>\n",
       "      <td>en</td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>original_title</th>\n",
       "      <td>Toy Story</td>\n",
       "      <td>Jumanji</td>\n",
       "      <td>Grumpier Old Men</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>overview</th>\n",
       "      <td>Led by Woody, Andy's toys live happily in his ...</td>\n",
       "      <td>When siblings Judy and Peter discover an encha...</td>\n",
       "      <td>A family wedding reignites the ancient feud be...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>popularity</th>\n",
       "      <td>21.9469</td>\n",
       "      <td>17.0155</td>\n",
       "      <td>11.7129</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>poster_path</th>\n",
       "      <td>/rhIRbceoE9lR4veEXuwCC2wARtG.jpg</td>\n",
       "      <td>/vzmL6fP7aPKNKPRTFnZmiUfciyV.jpg</td>\n",
       "      <td>/6ksm1sjKMFLbO7UY2i6G1ju9SML.jpg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>production_companies</th>\n",
       "      <td>[{'name': 'Pixar Animation Studios', 'id': 3}]</td>\n",
       "      <td>[{'name': 'TriStar Pictures', 'id': 559}, {'na...</td>\n",
       "      <td>[{'name': 'Warner Bros.', 'id': 6194}, {'name'...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>production_countries</th>\n",
       "      <td>[{'iso_3166_1': 'US', 'name': 'United States o...</td>\n",
       "      <td>[{'iso_3166_1': 'US', 'name': 'United States o...</td>\n",
       "      <td>[{'iso_3166_1': 'US', 'name': 'United States o...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>release_date</th>\n",
       "      <td>1995-10-30</td>\n",
       "      <td>1995-12-15</td>\n",
       "      <td>1995-12-22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>revenue</th>\n",
       "      <td>3.73554e+08</td>\n",
       "      <td>2.62797e+08</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>runtime</th>\n",
       "      <td>81</td>\n",
       "      <td>104</td>\n",
       "      <td>101</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>spoken_languages</th>\n",
       "      <td>[{'iso_639_1': 'en', 'name': 'English'}]</td>\n",
       "      <td>[{'iso_639_1': 'en', 'name': 'English'}, {'iso...</td>\n",
       "      <td>[{'iso_639_1': 'en', 'name': 'English'}]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>status</th>\n",
       "      <td>Released</td>\n",
       "      <td>Released</td>\n",
       "      <td>Released</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>tagline</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Roll the dice and unleash the excitement!</td>\n",
       "      <td>Still Yelling. Still Fighting. Still Ready for...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>title</th>\n",
       "      <td>Toy Story</td>\n",
       "      <td>Jumanji</td>\n",
       "      <td>Grumpier Old Men</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>video</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>vote_average</th>\n",
       "      <td>7.7</td>\n",
       "      <td>6.9</td>\n",
       "      <td>6.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>vote_count</th>\n",
       "      <td>5415</td>\n",
       "      <td>2413</td>\n",
       "      <td>92</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                       0  \\\n",
       "adult                                                              False   \n",
       "belongs_to_collection  {'id': 10194, 'name': 'Toy Story Collection', ...   \n",
       "budget                                                          30000000   \n",
       "genres                 [{'id': 16, 'name': 'Animation'}, {'id': 35, '...   \n",
       "homepage                            http://toystory.disney.com/toy-story   \n",
       "id                                                                   862   \n",
       "imdb_id                                                        tt0114709   \n",
       "original_language                                                     en   \n",
       "original_title                                                 Toy Story   \n",
       "overview               Led by Woody, Andy's toys live happily in his ...   \n",
       "popularity                                                       21.9469   \n",
       "poster_path                             /rhIRbceoE9lR4veEXuwCC2wARtG.jpg   \n",
       "production_companies      [{'name': 'Pixar Animation Studios', 'id': 3}]   \n",
       "production_countries   [{'iso_3166_1': 'US', 'name': 'United States o...   \n",
       "release_date                                                  1995-10-30   \n",
       "revenue                                                      3.73554e+08   \n",
       "runtime                                                               81   \n",
       "spoken_languages                [{'iso_639_1': 'en', 'name': 'English'}]   \n",
       "status                                                          Released   \n",
       "tagline                                                              NaN   \n",
       "title                                                          Toy Story   \n",
       "video                                                              False   \n",
       "vote_average                                                         7.7   \n",
       "vote_count                                                          5415   \n",
       "\n",
       "                                                                       1  \\\n",
       "adult                                                              False   \n",
       "belongs_to_collection                                                NaN   \n",
       "budget                                                          65000000   \n",
       "genres                 [{'id': 12, 'name': 'Adventure'}, {'id': 14, '...   \n",
       "homepage                                                             NaN   \n",
       "id                                                                  8844   \n",
       "imdb_id                                                        tt0113497   \n",
       "original_language                                                     en   \n",
       "original_title                                                   Jumanji   \n",
       "overview               When siblings Judy and Peter discover an encha...   \n",
       "popularity                                                       17.0155   \n",
       "poster_path                             /vzmL6fP7aPKNKPRTFnZmiUfciyV.jpg   \n",
       "production_companies   [{'name': 'TriStar Pictures', 'id': 559}, {'na...   \n",
       "production_countries   [{'iso_3166_1': 'US', 'name': 'United States o...   \n",
       "release_date                                                  1995-12-15   \n",
       "revenue                                                      2.62797e+08   \n",
       "runtime                                                              104   \n",
       "spoken_languages       [{'iso_639_1': 'en', 'name': 'English'}, {'iso...   \n",
       "status                                                          Released   \n",
       "tagline                        Roll the dice and unleash the excitement!   \n",
       "title                                                            Jumanji   \n",
       "video                                                              False   \n",
       "vote_average                                                         6.9   \n",
       "vote_count                                                          2413   \n",
       "\n",
       "                                                                       2  \n",
       "adult                                                              False  \n",
       "belongs_to_collection  {'id': 119050, 'name': 'Grumpy Old Men Collect...  \n",
       "budget                                                                 0  \n",
       "genres                 [{'id': 10749, 'name': 'Romance'}, {'id': 35, ...  \n",
       "homepage                                                             NaN  \n",
       "id                                                                 15602  \n",
       "imdb_id                                                        tt0113228  \n",
       "original_language                                                     en  \n",
       "original_title                                          Grumpier Old Men  \n",
       "overview               A family wedding reignites the ancient feud be...  \n",
       "popularity                                                       11.7129  \n",
       "poster_path                             /6ksm1sjKMFLbO7UY2i6G1ju9SML.jpg  \n",
       "production_companies   [{'name': 'Warner Bros.', 'id': 6194}, {'name'...  \n",
       "production_countries   [{'iso_3166_1': 'US', 'name': 'United States o...  \n",
       "release_date                                                  1995-12-22  \n",
       "revenue                                                                0  \n",
       "runtime                                                              101  \n",
       "spoken_languages                [{'iso_639_1': 'en', 'name': 'English'}]  \n",
       "status                                                          Released  \n",
       "tagline                Still Yelling. Still Fighting. Still Ready for...  \n",
       "title                                                   Grumpier Old Men  \n",
       "video                                                              False  \n",
       "vote_average                                                         6.5  \n",
       "vote_count                                                            92  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "md.iloc[0:3].transpose()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['adult', 'belongs_to_collection', 'budget', 'genres', 'homepage', 'id',\n",
       "       'imdb_id', 'original_language', 'original_title', 'overview',\n",
       "       'popularity', 'poster_path', 'production_companies',\n",
       "       'production_countries', 'release_date', 'revenue', 'runtime',\n",
       "       'spoken_languages', 'status', 'tagline', 'title', 'video',\n",
       "       'vote_average', 'vote_count'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "md.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Features**\n",
    "\n",
    "* **adult:** Indicates if the movie is X-Rated or Adult.\n",
    "* **belongs_to_collection:** A stringified dictionary that gives information on the movie series the particular film belongs to.\n",
    "* **budget:** The budget of the movie in dollars.\n",
    "* **genres:** A stringified list of dictionaries that list out all the genres associated with the movie.\n",
    "* **homepage:** The Official Homepage of the move.\n",
    "* **id:** The ID of the movie.\n",
    "* **imdb_id:** The IMDB ID of the movie.\n",
    "* **original_language:** The language in which the movie was originally shot in.\n",
    "* **original_title:** The original title of the movie.\n",
    "* **overview:** A brief blurb of the movie.\n",
    "* **popularity:** The Popularity Score assigned by TMDB.\n",
    "* **poster_path:** The URL of the poster image.\n",
    "* **production_companies:** A stringified list of production companies involved with the making of the movie.\n",
    "* **production_countries:** A stringified list of countries where the movie was shot/produced in.\n",
    "* **release_date:** Theatrical Release Date of the movie.\n",
    "* **revenue:** The total revenue of the movie in dollars.\n",
    "* **runtime:** The runtime of the movie in minutes.\n",
    "* **spoken_languages:** A stringified list of spoken languages in the film.\n",
    "* **status:** The status of the movie (Released, To Be Released, Announced, etc.)\n",
    "* **tagline:** The tagline of the movie.\n",
    "* **title:** The Official Title of the movie.\n",
    "* **video:** Indicates if there is a video present of the movie with TMDB.\n",
    "* **vote_average:** The average rating of the movie.\n",
    "* **vote_count:** The number of votes by users, as counted by TMDB."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(45466, 24)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "md.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 45466 entries, 0 to 45465\n",
      "Data columns (total 24 columns):\n",
      "adult                    45466 non-null object\n",
      "belongs_to_collection    4494 non-null object\n",
      "budget                   45466 non-null object\n",
      "genres                   45466 non-null object\n",
      "homepage                 7782 non-null object\n",
      "id                       45466 non-null object\n",
      "imdb_id                  45449 non-null object\n",
      "original_language        45455 non-null object\n",
      "original_title           45466 non-null object\n",
      "overview                 44512 non-null object\n",
      "popularity               45461 non-null object\n",
      "poster_path              45080 non-null object\n",
      "production_companies     45463 non-null object\n",
      "production_countries     45463 non-null object\n",
      "release_date             45379 non-null object\n",
      "revenue                  45460 non-null float64\n",
      "runtime                  45203 non-null float64\n",
      "spoken_languages         45460 non-null object\n",
      "status                   45379 non-null object\n",
      "tagline                  20412 non-null object\n",
      "title                    45460 non-null object\n",
      "video                    45460 non-null object\n",
      "vote_average             45460 non-null float64\n",
      "vote_count               45460 non-null float64\n",
      "dtypes: float64(4), object(20)\n",
      "memory usage: 8.3+ MB\n"
     ]
    }
   ],
   "source": [
    "md.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Ratings dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>movieId</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>31</td>\n",
       "      <td>2.5</td>\n",
       "      <td>1260759144</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1029</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1260759179</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>1061</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1260759182</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>1129</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1260759185</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>1172</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1260759205</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userId  movieId  rating   timestamp\n",
       "0       1       31     2.5  1260759144\n",
       "1       1     1029     3.0  1260759179\n",
       "2       1     1061     3.0  1260759182\n",
       "3       1     1129     2.0  1260759185\n",
       "4       1     1172     4.0  1260759205"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratings.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['userId', 'movieId', 'rating', 'timestamp'], dtype='object')"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratings.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* **userId:** It is id for User\n",
    "* **movieId:** It is TMDb movie id.\n",
    "* **rating:** Rating given for the particular movie by specific user\n",
    "* **timestamp:** Time stamp when rating has been given by user"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(100004, 4)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratings.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 100004 entries, 0 to 100003\n",
      "Data columns (total 4 columns):\n",
      "userId       100004 non-null int64\n",
      "movieId      100004 non-null int64\n",
      "rating       100004 non-null float64\n",
      "timestamp    100004 non-null int64\n",
      "dtypes: float64(1), int64(3)\n",
      "memory usage: 3.1 MB\n"
     ]
    }
   ],
   "source": [
    "ratings.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Data Wrangling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* The Movie database [TMDb](https://www.themoviedb.org/movie/269149-zootopia?language=en)\n",
    "* Already converted data from json to csv format"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Pre-processing \n",
    "\n",
    "* We will perform pre-processing as and when needed throughout the "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Build recommendation system"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.1. Simple recommendation system"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Approach: **\n",
    "\n",
    "* The Simple Recommender offers __generalized recommendations__ to every user __based on movie popularity and (sometimes) genre__. \n",
    "\n",
    "* The __basic idea__ behind this recommender is that __movies that are more popular and more critically acclaimed will have a higher probability of being liked by the average audience.__ \n",
    "\n",
    "* This model __does not give personalized recommendations__ based on the user.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**What we are actually doing: **\n",
    "\n",
    "* The implementation of this model is extremely trivial. \n",
    "* All we have to do is __sort our movies based on ratings and popularity__ and display the top movies of our list. \n",
    "* As an added step, we can __pass in a genre argument to get the top movies of a particular genre.__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I will build our overall Top 250 Chart and will define a function to build charts for a particular genre. Let's begin!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "md['genres'] = md['genres'].fillna('[]').apply(literal_eval).apply(lambda x: [i[\n",
    "    'name'] for i in x] if isinstance(x, list) else [])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* I use the TMDB Ratings to come up with our Top Movies Chart. \n",
    "* I will use IMDB's weighted rating formula to construct my chart.\n",
    "* Mathematically, it is represented as follows:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "$\\large Weighted\\; Rating (WR) = (\\frac{v}{v + m} . R) + (\\frac{m}{v + m} . C)$\n",
    "```\n",
    "where,\n",
    "    v is the number of votes for the movie\n",
    "    m is the minimum votes required to be listed in the chart\n",
    "    R is the average rating of the movie\n",
    "    C is the mean vote across the whole report\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5.244896612406511"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# this is V\n",
    "vote_counts = md[md['vote_count'].notnull()]['vote_count'].astype('int')\n",
    "\n",
    "# this is R\n",
    "vote_averages = md[md['vote_average'].notnull()]['vote_average'].astype('int')\n",
    "\n",
    "# this is C\n",
    "C = vote_averages.mean()\n",
    "C"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* The next step, we need to determine an appropriate value for `m`, the minimum votes required to be listed in the chart. \n",
    "\n",
    "* We will use 95th percentile as our cutoff. In other words, for a movie to feature in the charts, it must have more votes than at least 95% of the movies in the list.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "434.0"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m = vote_counts.quantile(0.95)\n",
    "m"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Pre-processing step for getting year from date by splliting it using '-'\n",
    "\n",
    "md['year'] = pd.to_datetime(md['release_date'], errors='coerce').apply(\n",
    "    lambda x: str(x).split('-')[0] if x != np.nan else np.nan)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2274, 6)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "qualified = md[(md['vote_count'] >= m) & \n",
    "               (md['vote_count'].notnull()) & \n",
    "               (md['vote_average'].notnull())][['title', \n",
    "                                                'year', \n",
    "                                                'vote_count', \n",
    "                                                'vote_average', \n",
    "                                                'popularity', \n",
    "                                                'genres']]\n",
    "\n",
    "qualified['vote_count'] = qualified['vote_count'].astype('int')\n",
    "qualified['vote_average'] = qualified['vote_average'].astype('int')\n",
    "qualified.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Therefore, to qualify to be considered for the chart, a movie has to have at least __434 votes__ on TMDB. \n",
    "* We also see that the __average rating__ for __a movie on TMDB__ is __5.244 on a scale of 10__. \n",
    "* Here, only __2274 movies__ are qualify to be on our chart."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "def weighted_rating(x):\n",
    "    v = x['vote_count']\n",
    "    R = x['vote_average']\n",
    "    return (v/(v+m) * R) + (m/(m+v) * C)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "qualified['wr'] = qualified.apply(weighted_rating, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "qualified = qualified.sort_values('wr', ascending=False).head(250)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Top Movies**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>year</th>\n",
       "      <th>vote_count</th>\n",
       "      <th>vote_average</th>\n",
       "      <th>popularity</th>\n",
       "      <th>genres</th>\n",
       "      <th>wr</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>15480</th>\n",
       "      <td>Inception</td>\n",
       "      <td>2010</td>\n",
       "      <td>14075</td>\n",
       "      <td>8</td>\n",
       "      <td>29.1081</td>\n",
       "      <td>[Action, Thriller, Science Fiction, Mystery, A...</td>\n",
       "      <td>7.917588</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12481</th>\n",
       "      <td>The Dark Knight</td>\n",
       "      <td>2008</td>\n",
       "      <td>12269</td>\n",
       "      <td>8</td>\n",
       "      <td>123.167</td>\n",
       "      <td>[Drama, Action, Crime, Thriller]</td>\n",
       "      <td>7.905871</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22879</th>\n",
       "      <td>Interstellar</td>\n",
       "      <td>2014</td>\n",
       "      <td>11187</td>\n",
       "      <td>8</td>\n",
       "      <td>32.2135</td>\n",
       "      <td>[Adventure, Drama, Science Fiction]</td>\n",
       "      <td>7.897107</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2843</th>\n",
       "      <td>Fight Club</td>\n",
       "      <td>1999</td>\n",
       "      <td>9678</td>\n",
       "      <td>8</td>\n",
       "      <td>63.8696</td>\n",
       "      <td>[Drama]</td>\n",
       "      <td>7.881753</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4863</th>\n",
       "      <td>The Lord of the Rings: The Fellowship of the Ring</td>\n",
       "      <td>2001</td>\n",
       "      <td>8892</td>\n",
       "      <td>8</td>\n",
       "      <td>32.0707</td>\n",
       "      <td>[Adventure, Fantasy, Action]</td>\n",
       "      <td>7.871787</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>292</th>\n",
       "      <td>Pulp Fiction</td>\n",
       "      <td>1994</td>\n",
       "      <td>8670</td>\n",
       "      <td>8</td>\n",
       "      <td>140.95</td>\n",
       "      <td>[Thriller, Crime]</td>\n",
       "      <td>7.868660</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>314</th>\n",
       "      <td>The Shawshank Redemption</td>\n",
       "      <td>1994</td>\n",
       "      <td>8358</td>\n",
       "      <td>8</td>\n",
       "      <td>51.6454</td>\n",
       "      <td>[Drama, Crime]</td>\n",
       "      <td>7.864000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7000</th>\n",
       "      <td>The Lord of the Rings: The Return of the King</td>\n",
       "      <td>2003</td>\n",
       "      <td>8226</td>\n",
       "      <td>8</td>\n",
       "      <td>29.3244</td>\n",
       "      <td>[Adventure, Fantasy, Action]</td>\n",
       "      <td>7.861927</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>351</th>\n",
       "      <td>Forrest Gump</td>\n",
       "      <td>1994</td>\n",
       "      <td>8147</td>\n",
       "      <td>8</td>\n",
       "      <td>48.3072</td>\n",
       "      <td>[Comedy, Drama, Romance]</td>\n",
       "      <td>7.860656</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5814</th>\n",
       "      <td>The Lord of the Rings: The Two Towers</td>\n",
       "      <td>2002</td>\n",
       "      <td>7641</td>\n",
       "      <td>8</td>\n",
       "      <td>29.4235</td>\n",
       "      <td>[Adventure, Fantasy, Action]</td>\n",
       "      <td>7.851924</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>256</th>\n",
       "      <td>Star Wars</td>\n",
       "      <td>1977</td>\n",
       "      <td>6778</td>\n",
       "      <td>8</td>\n",
       "      <td>42.1497</td>\n",
       "      <td>[Adventure, Action, Science Fiction]</td>\n",
       "      <td>7.834205</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1225</th>\n",
       "      <td>Back to the Future</td>\n",
       "      <td>1985</td>\n",
       "      <td>6239</td>\n",
       "      <td>8</td>\n",
       "      <td>25.7785</td>\n",
       "      <td>[Adventure, Comedy, Science Fiction, Family]</td>\n",
       "      <td>7.820813</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>834</th>\n",
       "      <td>The Godfather</td>\n",
       "      <td>1972</td>\n",
       "      <td>6024</td>\n",
       "      <td>8</td>\n",
       "      <td>41.1093</td>\n",
       "      <td>[Drama, Crime]</td>\n",
       "      <td>7.814847</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1154</th>\n",
       "      <td>The Empire Strikes Back</td>\n",
       "      <td>1980</td>\n",
       "      <td>5998</td>\n",
       "      <td>8</td>\n",
       "      <td>19.471</td>\n",
       "      <td>[Adventure, Action, Science Fiction]</td>\n",
       "      <td>7.814099</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>Se7en</td>\n",
       "      <td>1995</td>\n",
       "      <td>5915</td>\n",
       "      <td>8</td>\n",
       "      <td>18.4574</td>\n",
       "      <td>[Crime, Mystery, Thriller]</td>\n",
       "      <td>7.811669</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                   title  year  vote_count  \\\n",
       "15480                                          Inception  2010       14075   \n",
       "12481                                    The Dark Knight  2008       12269   \n",
       "22879                                       Interstellar  2014       11187   \n",
       "2843                                          Fight Club  1999        9678   \n",
       "4863   The Lord of the Rings: The Fellowship of the Ring  2001        8892   \n",
       "292                                         Pulp Fiction  1994        8670   \n",
       "314                             The Shawshank Redemption  1994        8358   \n",
       "7000       The Lord of the Rings: The Return of the King  2003        8226   \n",
       "351                                         Forrest Gump  1994        8147   \n",
       "5814               The Lord of the Rings: The Two Towers  2002        7641   \n",
       "256                                            Star Wars  1977        6778   \n",
       "1225                                  Back to the Future  1985        6239   \n",
       "834                                        The Godfather  1972        6024   \n",
       "1154                             The Empire Strikes Back  1980        5998   \n",
       "46                                                 Se7en  1995        5915   \n",
       "\n",
       "       vote_average popularity  \\\n",
       "15480             8    29.1081   \n",
       "12481             8    123.167   \n",
       "22879             8    32.2135   \n",
       "2843              8    63.8696   \n",
       "4863              8    32.0707   \n",
       "292               8     140.95   \n",
       "314               8    51.6454   \n",
       "7000              8    29.3244   \n",
       "351               8    48.3072   \n",
       "5814              8    29.4235   \n",
       "256               8    42.1497   \n",
       "1225              8    25.7785   \n",
       "834               8    41.1093   \n",
       "1154              8     19.471   \n",
       "46                8    18.4574   \n",
       "\n",
       "                                                  genres        wr  \n",
       "15480  [Action, Thriller, Science Fiction, Mystery, A...  7.917588  \n",
       "12481                   [Drama, Action, Crime, Thriller]  7.905871  \n",
       "22879                [Adventure, Drama, Science Fiction]  7.897107  \n",
       "2843                                             [Drama]  7.881753  \n",
       "4863                        [Adventure, Fantasy, Action]  7.871787  \n",
       "292                                    [Thriller, Crime]  7.868660  \n",
       "314                                       [Drama, Crime]  7.864000  \n",
       "7000                        [Adventure, Fantasy, Action]  7.861927  \n",
       "351                             [Comedy, Drama, Romance]  7.860656  \n",
       "5814                        [Adventure, Fantasy, Action]  7.851924  \n",
       "256                 [Adventure, Action, Science Fiction]  7.834205  \n",
       "1225        [Adventure, Comedy, Science Fiction, Family]  7.820813  \n",
       "834                                       [Drama, Crime]  7.814847  \n",
       "1154                [Adventure, Action, Science Fiction]  7.814099  \n",
       "46                            [Crime, Mystery, Thriller]  7.811669  "
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "qualified.head(15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* We see that three Christopher Nolan Films, __Inception__, __The Dark Knight__ and __Interstellar__ occur at the very top of our chart. \n",
    "* The chart also indicates a strong bias of TMDB Users towards particular genres and directors."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Let us now construct our __function that builds charts for particular genres.__\n",
    "\n",
    "* For this, we __relax__ our __default conditions to the 85th percentile instead of 95.__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>0</th>\n",
       "      <th>0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>adult</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>belongs_to_collection</th>\n",
       "      <td>{'id': 10194, 'name': 'Toy Story Collection', ...</td>\n",
       "      <td>{'id': 10194, 'name': 'Toy Story Collection', ...</td>\n",
       "      <td>{'id': 10194, 'name': 'Toy Story Collection', ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>budget</th>\n",
       "      <td>30000000</td>\n",
       "      <td>30000000</td>\n",
       "      <td>30000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>homepage</th>\n",
       "      <td>http://toystory.disney.com/toy-story</td>\n",
       "      <td>http://toystory.disney.com/toy-story</td>\n",
       "      <td>http://toystory.disney.com/toy-story</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <td>862</td>\n",
       "      <td>862</td>\n",
       "      <td>862</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>imdb_id</th>\n",
       "      <td>tt0114709</td>\n",
       "      <td>tt0114709</td>\n",
       "      <td>tt0114709</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>original_language</th>\n",
       "      <td>en</td>\n",
       "      <td>en</td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>original_title</th>\n",
       "      <td>Toy Story</td>\n",
       "      <td>Toy Story</td>\n",
       "      <td>Toy Story</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>overview</th>\n",
       "      <td>Led by Woody, Andy's toys live happily in his ...</td>\n",
       "      <td>Led by Woody, Andy's toys live happily in his ...</td>\n",
       "      <td>Led by Woody, Andy's toys live happily in his ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>popularity</th>\n",
       "      <td>21.9469</td>\n",
       "      <td>21.9469</td>\n",
       "      <td>21.9469</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>poster_path</th>\n",
       "      <td>/rhIRbceoE9lR4veEXuwCC2wARtG.jpg</td>\n",
       "      <td>/rhIRbceoE9lR4veEXuwCC2wARtG.jpg</td>\n",
       "      <td>/rhIRbceoE9lR4veEXuwCC2wARtG.jpg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>production_companies</th>\n",
       "      <td>[{'name': 'Pixar Animation Studios', 'id': 3}]</td>\n",
       "      <td>[{'name': 'Pixar Animation Studios', 'id': 3}]</td>\n",
       "      <td>[{'name': 'Pixar Animation Studios', 'id': 3}]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>production_countries</th>\n",
       "      <td>[{'iso_3166_1': 'US', 'name': 'United States o...</td>\n",
       "      <td>[{'iso_3166_1': 'US', 'name': 'United States o...</td>\n",
       "      <td>[{'iso_3166_1': 'US', 'name': 'United States o...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>release_date</th>\n",
       "      <td>1995-10-30</td>\n",
       "      <td>1995-10-30</td>\n",
       "      <td>1995-10-30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>revenue</th>\n",
       "      <td>3.73554e+08</td>\n",
       "      <td>3.73554e+08</td>\n",
       "      <td>3.73554e+08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>runtime</th>\n",
       "      <td>81</td>\n",
       "      <td>81</td>\n",
       "      <td>81</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>spoken_languages</th>\n",
       "      <td>[{'iso_639_1': 'en', 'name': 'English'}]</td>\n",
       "      <td>[{'iso_639_1': 'en', 'name': 'English'}]</td>\n",
       "      <td>[{'iso_639_1': 'en', 'name': 'English'}]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>status</th>\n",
       "      <td>Released</td>\n",
       "      <td>Released</td>\n",
       "      <td>Released</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>tagline</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>title</th>\n",
       "      <td>Toy Story</td>\n",
       "      <td>Toy Story</td>\n",
       "      <td>Toy Story</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>video</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>vote_average</th>\n",
       "      <td>7.7</td>\n",
       "      <td>7.7</td>\n",
       "      <td>7.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>vote_count</th>\n",
       "      <td>5415</td>\n",
       "      <td>5415</td>\n",
       "      <td>5415</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>year</th>\n",
       "      <td>1995</td>\n",
       "      <td>1995</td>\n",
       "      <td>1995</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>genre</th>\n",
       "      <td>Animation</td>\n",
       "      <td>Comedy</td>\n",
       "      <td>Family</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                       0  \\\n",
       "adult                                                              False   \n",
       "belongs_to_collection  {'id': 10194, 'name': 'Toy Story Collection', ...   \n",
       "budget                                                          30000000   \n",
       "homepage                            http://toystory.disney.com/toy-story   \n",
       "id                                                                   862   \n",
       "imdb_id                                                        tt0114709   \n",
       "original_language                                                     en   \n",
       "original_title                                                 Toy Story   \n",
       "overview               Led by Woody, Andy's toys live happily in his ...   \n",
       "popularity                                                       21.9469   \n",
       "poster_path                             /rhIRbceoE9lR4veEXuwCC2wARtG.jpg   \n",
       "production_companies      [{'name': 'Pixar Animation Studios', 'id': 3}]   \n",
       "production_countries   [{'iso_3166_1': 'US', 'name': 'United States o...   \n",
       "release_date                                                  1995-10-30   \n",
       "revenue                                                      3.73554e+08   \n",
       "runtime                                                               81   \n",
       "spoken_languages                [{'iso_639_1': 'en', 'name': 'English'}]   \n",
       "status                                                          Released   \n",
       "tagline                                                              NaN   \n",
       "title                                                          Toy Story   \n",
       "video                                                              False   \n",
       "vote_average                                                         7.7   \n",
       "vote_count                                                          5415   \n",
       "year                                                                1995   \n",
       "genre                                                          Animation   \n",
       "\n",
       "                                                                       0  \\\n",
       "adult                                                              False   \n",
       "belongs_to_collection  {'id': 10194, 'name': 'Toy Story Collection', ...   \n",
       "budget                                                          30000000   \n",
       "homepage                            http://toystory.disney.com/toy-story   \n",
       "id                                                                   862   \n",
       "imdb_id                                                        tt0114709   \n",
       "original_language                                                     en   \n",
       "original_title                                                 Toy Story   \n",
       "overview               Led by Woody, Andy's toys live happily in his ...   \n",
       "popularity                                                       21.9469   \n",
       "poster_path                             /rhIRbceoE9lR4veEXuwCC2wARtG.jpg   \n",
       "production_companies      [{'name': 'Pixar Animation Studios', 'id': 3}]   \n",
       "production_countries   [{'iso_3166_1': 'US', 'name': 'United States o...   \n",
       "release_date                                                  1995-10-30   \n",
       "revenue                                                      3.73554e+08   \n",
       "runtime                                                               81   \n",
       "spoken_languages                [{'iso_639_1': 'en', 'name': 'English'}]   \n",
       "status                                                          Released   \n",
       "tagline                                                              NaN   \n",
       "title                                                          Toy Story   \n",
       "video                                                              False   \n",
       "vote_average                                                         7.7   \n",
       "vote_count                                                          5415   \n",
       "year                                                                1995   \n",
       "genre                                                             Comedy   \n",
       "\n",
       "                                                                       0  \n",
       "adult                                                              False  \n",
       "belongs_to_collection  {'id': 10194, 'name': 'Toy Story Collection', ...  \n",
       "budget                                                          30000000  \n",
       "homepage                            http://toystory.disney.com/toy-story  \n",
       "id                                                                   862  \n",
       "imdb_id                                                        tt0114709  \n",
       "original_language                                                     en  \n",
       "original_title                                                 Toy Story  \n",
       "overview               Led by Woody, Andy's toys live happily in his ...  \n",
       "popularity                                                       21.9469  \n",
       "poster_path                             /rhIRbceoE9lR4veEXuwCC2wARtG.jpg  \n",
       "production_companies      [{'name': 'Pixar Animation Studios', 'id': 3}]  \n",
       "production_countries   [{'iso_3166_1': 'US', 'name': 'United States o...  \n",
       "release_date                                                  1995-10-30  \n",
       "revenue                                                      3.73554e+08  \n",
       "runtime                                                               81  \n",
       "spoken_languages                [{'iso_639_1': 'en', 'name': 'English'}]  \n",
       "status                                                          Released  \n",
       "tagline                                                              NaN  \n",
       "title                                                          Toy Story  \n",
       "video                                                              False  \n",
       "vote_average                                                         7.7  \n",
       "vote_count                                                          5415  \n",
       "year                                                                1995  \n",
       "genre                                                             Family  "
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'''\n",
    ">>> s\n",
    "     a   b\n",
    "one  1.  2.\n",
    "two  3.  4.\n",
    "\n",
    ">>> s.stack()\n",
    "one a    1\n",
    "    b    2\n",
    "two a    3\n",
    "    b    4\n",
    "'''\n",
    "s = md.apply(lambda x: pd.Series(x['genres']),axis=1).stack().reset_index(level=1, drop=True)\n",
    "s.name = 'genre'\n",
    "gen_md = md.drop('genres', axis=1).join(s)\n",
    "gen_md.head(3).transpose()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_chart(genre, percentile=0.85):\n",
    "    df = gen_md[gen_md['genre'] == genre]\n",
    "    vote_counts = df[df['vote_count'].notnull()]['vote_count'].astype('int')\n",
    "    vote_averages = df[df['vote_average'].notnull()]['vote_average'].astype('int')\n",
    "    C = vote_averages.mean()\n",
    "    m = vote_counts.quantile(percentile)\n",
    "    \n",
    "    qualified = df[(df['vote_count'] >= m) & (df['vote_count'].notnull()) & \n",
    "                   (df['vote_average'].notnull())][['title', 'year', 'vote_count', 'vote_average', 'popularity']]\n",
    "    qualified['vote_count'] = qualified['vote_count'].astype('int')\n",
    "    qualified['vote_average'] = qualified['vote_average'].astype('int')\n",
    "    \n",
    "    qualified['wr'] = qualified.apply(lambda x: \n",
    "                        (x['vote_count']/(x['vote_count']+m) * x['vote_average']) + (m/(m+x['vote_count']) * C),\n",
    "                        axis=1)\n",
    "    qualified = qualified.sort_values('wr', ascending=False).head(250)\n",
    "    \n",
    "    return qualified"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us see our method in action by displaying the __Top 15 Romance Movies__ (Romance almost didn't feature at all in our Generic Top Chart despite being one of the most popular movie genres).\n",
    "\n",
    "\n",
    "**Top 15 Romantic Movies**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>year</th>\n",
       "      <th>vote_count</th>\n",
       "      <th>vote_average</th>\n",
       "      <th>popularity</th>\n",
       "      <th>wr</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>10309</th>\n",
       "      <td>Dilwale Dulhania Le Jayenge</td>\n",
       "      <td>1995</td>\n",
       "      <td>661</td>\n",
       "      <td>9</td>\n",
       "      <td>34.457</td>\n",
       "      <td>8.565285</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>351</th>\n",
       "      <td>Forrest Gump</td>\n",
       "      <td>1994</td>\n",
       "      <td>8147</td>\n",
       "      <td>8</td>\n",
       "      <td>48.3072</td>\n",
       "      <td>7.971357</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>876</th>\n",
       "      <td>Vertigo</td>\n",
       "      <td>1958</td>\n",
       "      <td>1162</td>\n",
       "      <td>8</td>\n",
       "      <td>18.2082</td>\n",
       "      <td>7.811667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40251</th>\n",
       "      <td>Your Name.</td>\n",
       "      <td>2016</td>\n",
       "      <td>1030</td>\n",
       "      <td>8</td>\n",
       "      <td>34.461252</td>\n",
       "      <td>7.789489</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>883</th>\n",
       "      <td>Some Like It Hot</td>\n",
       "      <td>1959</td>\n",
       "      <td>835</td>\n",
       "      <td>8</td>\n",
       "      <td>11.8451</td>\n",
       "      <td>7.745154</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1132</th>\n",
       "      <td>Cinema Paradiso</td>\n",
       "      <td>1988</td>\n",
       "      <td>834</td>\n",
       "      <td>8</td>\n",
       "      <td>14.177</td>\n",
       "      <td>7.744878</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19901</th>\n",
       "      <td>Paperman</td>\n",
       "      <td>2012</td>\n",
       "      <td>734</td>\n",
       "      <td>8</td>\n",
       "      <td>7.19863</td>\n",
       "      <td>7.713951</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37863</th>\n",
       "      <td>Sing Street</td>\n",
       "      <td>2016</td>\n",
       "      <td>669</td>\n",
       "      <td>8</td>\n",
       "      <td>10.672862</td>\n",
       "      <td>7.689483</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>882</th>\n",
       "      <td>The Apartment</td>\n",
       "      <td>1960</td>\n",
       "      <td>498</td>\n",
       "      <td>8</td>\n",
       "      <td>11.9943</td>\n",
       "      <td>7.599317</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38718</th>\n",
       "      <td>The Handmaiden</td>\n",
       "      <td>2016</td>\n",
       "      <td>453</td>\n",
       "      <td>8</td>\n",
       "      <td>16.727405</td>\n",
       "      <td>7.566166</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3189</th>\n",
       "      <td>City Lights</td>\n",
       "      <td>1931</td>\n",
       "      <td>444</td>\n",
       "      <td>8</td>\n",
       "      <td>10.8915</td>\n",
       "      <td>7.558867</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24886</th>\n",
       "      <td>The Way He Looks</td>\n",
       "      <td>2014</td>\n",
       "      <td>262</td>\n",
       "      <td>8</td>\n",
       "      <td>5.71127</td>\n",
       "      <td>7.331363</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45437</th>\n",
       "      <td>In a Heartbeat</td>\n",
       "      <td>2017</td>\n",
       "      <td>146</td>\n",
       "      <td>8</td>\n",
       "      <td>20.82178</td>\n",
       "      <td>7.003959</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1639</th>\n",
       "      <td>Titanic</td>\n",
       "      <td>1997</td>\n",
       "      <td>7770</td>\n",
       "      <td>7</td>\n",
       "      <td>26.8891</td>\n",
       "      <td>6.981546</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19731</th>\n",
       "      <td>Silver Linings Playbook</td>\n",
       "      <td>2012</td>\n",
       "      <td>4840</td>\n",
       "      <td>7</td>\n",
       "      <td>14.4881</td>\n",
       "      <td>6.970581</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                             title  year  vote_count  vote_average popularity  \\\n",
       "10309  Dilwale Dulhania Le Jayenge  1995         661             9     34.457   \n",
       "351                   Forrest Gump  1994        8147             8    48.3072   \n",
       "876                        Vertigo  1958        1162             8    18.2082   \n",
       "40251                   Your Name.  2016        1030             8  34.461252   \n",
       "883               Some Like It Hot  1959         835             8    11.8451   \n",
       "1132               Cinema Paradiso  1988         834             8     14.177   \n",
       "19901                     Paperman  2012         734             8    7.19863   \n",
       "37863                  Sing Street  2016         669             8  10.672862   \n",
       "882                  The Apartment  1960         498             8    11.9943   \n",
       "38718               The Handmaiden  2016         453             8  16.727405   \n",
       "3189                   City Lights  1931         444             8    10.8915   \n",
       "24886             The Way He Looks  2014         262             8    5.71127   \n",
       "45437               In a Heartbeat  2017         146             8   20.82178   \n",
       "1639                       Titanic  1997        7770             7    26.8891   \n",
       "19731      Silver Linings Playbook  2012        4840             7    14.4881   \n",
       "\n",
       "             wr  \n",
       "10309  8.565285  \n",
       "351    7.971357  \n",
       "876    7.811667  \n",
       "40251  7.789489  \n",
       "883    7.745154  \n",
       "1132   7.744878  \n",
       "19901  7.713951  \n",
       "37863  7.689483  \n",
       "882    7.599317  \n",
       "38718  7.566166  \n",
       "3189   7.558867  \n",
       "24886  7.331363  \n",
       "45437  7.003959  \n",
       "1639   6.981546  \n",
       "19731  6.970581  "
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "build_chart('Romance').head(15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.2 Content based recommendation system"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "links_small = links_small[links_small['tmdbId'].notnull()]['tmdbId'].astype('int')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Pre-processing step\n",
    "\n",
    "def convert_int(x):\n",
    "    try:\n",
    "        return int(x)\n",
    "    except:\n",
    "        return np.nan\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>adult</th>\n",
       "      <th>belongs_to_collection</th>\n",
       "      <th>budget</th>\n",
       "      <th>genres</th>\n",
       "      <th>homepage</th>\n",
       "      <th>id</th>\n",
       "      <th>imdb_id</th>\n",
       "      <th>original_language</th>\n",
       "      <th>original_title</th>\n",
       "      <th>overview</th>\n",
       "      <th>...</th>\n",
       "      <th>revenue</th>\n",
       "      <th>runtime</th>\n",
       "      <th>spoken_languages</th>\n",
       "      <th>status</th>\n",
       "      <th>tagline</th>\n",
       "      <th>title</th>\n",
       "      <th>video</th>\n",
       "      <th>vote_average</th>\n",
       "      <th>vote_count</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>19730</th>\n",
       "      <td>- Written by Ørnås</td>\n",
       "      <td>0.065736</td>\n",
       "      <td>/ff9qCepilowshEtG2GYWwzt2bs4.jpg</td>\n",
       "      <td>[Carousel Productions, Vision View Entertainme...</td>\n",
       "      <td>[{'iso_3166_1': 'CA', 'name': 'Canada'}, {'iso...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>104.0</td>\n",
       "      <td>[{'iso_639_1': 'en', 'name': 'English'}]</td>\n",
       "      <td>Released</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaT</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29503</th>\n",
       "      <td>Rune Balot goes to a casino connected to the ...</td>\n",
       "      <td>1.931659</td>\n",
       "      <td>/zV8bHuSL6WXoD6FWogP9j4x80bL.jpg</td>\n",
       "      <td>[Aniplex, GoHands, BROSTA TV, Mardock Scramble...</td>\n",
       "      <td>[{'iso_3166_1': 'US', 'name': 'United States o...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>68.0</td>\n",
       "      <td>[{'iso_639_1': 'ja', 'name': '日本語'}]</td>\n",
       "      <td>Released</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaT</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35587</th>\n",
       "      <td>Avalanche Sharks tells the story of a bikini ...</td>\n",
       "      <td>2.185485</td>\n",
       "      <td>/zaSf5OG7V8X8gqFvly88zDdRm46.jpg</td>\n",
       "      <td>[Odyssey Media, Pulser Productions, Rogue Stat...</td>\n",
       "      <td>[{'iso_3166_1': 'CA', 'name': 'Canada'}]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>82.0</td>\n",
       "      <td>[{'iso_639_1': 'en', 'name': 'English'}]</td>\n",
       "      <td>Released</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaT</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 25 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                   adult  \\\n",
       "19730                                 - Written by Ørnås   \n",
       "29503   Rune Balot goes to a casino connected to the ...   \n",
       "35587   Avalanche Sharks tells the story of a bikini ...   \n",
       "\n",
       "      belongs_to_collection                            budget  \\\n",
       "19730              0.065736  /ff9qCepilowshEtG2GYWwzt2bs4.jpg   \n",
       "29503              1.931659  /zV8bHuSL6WXoD6FWogP9j4x80bL.jpg   \n",
       "35587              2.185485  /zaSf5OG7V8X8gqFvly88zDdRm46.jpg   \n",
       "\n",
       "                                                  genres  \\\n",
       "19730  [Carousel Productions, Vision View Entertainme...   \n",
       "29503  [Aniplex, GoHands, BROSTA TV, Mardock Scramble...   \n",
       "35587  [Odyssey Media, Pulser Productions, Rogue Stat...   \n",
       "\n",
       "                                                homepage  id imdb_id  \\\n",
       "19730  [{'iso_3166_1': 'CA', 'name': 'Canada'}, {'iso... NaN       0   \n",
       "29503  [{'iso_3166_1': 'US', 'name': 'United States o... NaN       0   \n",
       "35587           [{'iso_3166_1': 'CA', 'name': 'Canada'}] NaN       0   \n",
       "\n",
       "      original_language                            original_title  overview  \\\n",
       "19730             104.0  [{'iso_639_1': 'en', 'name': 'English'}]  Released   \n",
       "29503              68.0      [{'iso_639_1': 'ja', 'name': '日本語'}]  Released   \n",
       "35587              82.0  [{'iso_639_1': 'en', 'name': 'English'}]  Released   \n",
       "\n",
       "      ...  revenue runtime spoken_languages status tagline  title  video  \\\n",
       "19730 ...      NaN     NaN              NaN    NaN     NaN    NaN    NaN   \n",
       "29503 ...      NaN     NaN              NaN    NaN     NaN    NaN    NaN   \n",
       "35587 ...      NaN     NaN              NaN    NaN     NaN    NaN    NaN   \n",
       "\n",
       "      vote_average vote_count year  \n",
       "19730          NaN        NaN  NaT  \n",
       "29503          NaN        NaN  NaT  \n",
       "35587          NaN        NaN  NaT  \n",
       "\n",
       "[3 rows x 25 columns]"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "md['id'] = md['id'].apply(convert_int)\n",
    "md[md['id'].isnull()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "md = md.drop([19730, 29503, 35587])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "md['id'] = md['id'].astype('int')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(9099, 25)"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "smd = md[md['id'].isin(links_small)]\n",
    "smd.shape\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have __9099 movies__ available in our small movies metadata dataset which is 5 times smaller than our original dataset of 45000 movies."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Content based recommendation system : Using movie description and taglines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Let us first try to build a recommender using movie descriptions and taglines. \n",
    "* We do not have a quantitative metric to judge our machine's performance so this will have to be done \n",
    "  qualitatively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "smd['tagline'] = smd['tagline'].fillna('')\n",
    "smd['description'] = smd['overview'] + smd['tagline']\n",
    "smd['description'] = smd['description'].fillna('')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf = TfidfVectorizer(analyzer='word',ngram_range=(1, 2),min_df=0, stop_words='english')\n",
    "tfidf_matrix = tf.fit_transform(smd['description'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(9099, 268124)"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfidf_matrix.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Since we have used the TF-IDF Vectorizer, calculating the Dot Product will directly give us the Cosine Similarity Score. \n",
    "\n",
    "* Therefore, we will use sklearn's linear_kernel instead of cosine_similarities since it is much faster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "# http://scikit-learn.org/stable/modules/metrics.html#linear-kernel\n",
    "cosine_sim = linear_kernel(tfidf_matrix, tfidf_matrix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1.        , 0.00680476, 0.        , ..., 0.        , 0.00344913,\n",
       "       0.        ])"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cosine_sim[0]\n",
    "#cosine_sim.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* We now have a pairwise cosine similarity matrix for all the movies in our dataset. \n",
    "* The next step is to write a function that returns the 30 most similar movies based on the cosine similarity score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "smd = smd.reset_index()\n",
    "titles = smd['title']\n",
    "indices = pd.Series(smd.index, index=smd['title'])\n",
    "#indices.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_recommendations(title):\n",
    "    idx = indices[title]\n",
    "    sim_scores = list(enumerate(cosine_sim[idx]))\n",
    "    sim_scores = sorted(sim_scores, key=lambda x: x[1], reverse=True)\n",
    "    sim_scores = sim_scores[1:31]\n",
    "    movie_indices = [i[0] for i in sim_scores]\n",
    "    return titles.iloc[movie_indices]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* We're all set...!\n",
    "* Let us now try and get the top recommendations for a few movies and see how good the recommendations are."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "973      The Godfather: Part II\n",
       "8387                 The Family\n",
       "3509                       Made\n",
       "4196         Johnny Dangerously\n",
       "29               Shanghai Triad\n",
       "5667                       Fury\n",
       "2412             American Movie\n",
       "1582    The Godfather: Part III\n",
       "4221                    8 Women\n",
       "2159              Summer of Sam\n",
       "Name: title, dtype: object"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_recommendations('The Godfather').head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7931                      The Dark Knight Rises\n",
       "132                              Batman Forever\n",
       "1113                             Batman Returns\n",
       "8227    Batman: The Dark Knight Returns, Part 2\n",
       "7565                 Batman: Under the Red Hood\n",
       "524                                      Batman\n",
       "7901                           Batman: Year One\n",
       "2579               Batman: Mask of the Phantasm\n",
       "2696                                        JFK\n",
       "8165    Batman: The Dark Knight Returns, Part 1\n",
       "Name: title, dtype: object"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_recommendations('The Dark Knight').head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* We see that for The __Dark Knight__, our system is able to identify it as a __Batman film and subsequently recommend other Batman films__ as its top recommendations.\n",
    "\n",
    "* But unfortunately, that is all this system can do at the moment. \n",
    "\n",
    "* This is not of much use to most people as it doesn't take into considerations very important features such as cast, crew, director and genre, which determine the rating and the popularity of a movie. \n",
    "\n",
    "* Someone who liked The Dark Knight probably likes it more because of Nolan and would hate Batman Forever and every other substandard movie in the Batman Franchise."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Therefore, we are going to use much more suggestive metadata than Overview and Tagline. \n",
    "* In the next subsection, we will build a more sophisticated recommender that takes __genre, keywords, cast and crew__ into consideration."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Content based RS : Using movie description, taglines, keywords, cast, director and genres"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* To build our standard metadata based content recommender, we will need to __merge our current dataset with the \n",
    "  crew and the keyword datasets__. \n",
    "* Let us prepare this data as our first step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "keywords['id'] = keywords['id'].astype('int')\n",
    "credits['id'] = credits['id'].astype('int')\n",
    "md['id'] = md['id'].astype('int')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(45463, 25)"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "md.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "md = md.merge(credits, on='id')\n",
    "md = md.merge(keywords, on='id')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(9219, 28)"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "smd = md[md['id'].isin(links_small)]\n",
    "smd.shape\n",
    "\n",
    "# smd = md[md['id'].isin(links_small['tmdbId'])]\n",
    "# smd.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have our cast, crew, genres and credits, all in one dataframe. Let us wrangle this a little more using the following intuitions:\n",
    "\n",
    "**1. Crew:** From the crew, we will only pick the director as our feature since the others don't contribute that much to the feel of the movie.\n",
    "\n",
    "**2. Cast:** Choosing Cast is a little more tricky. Lesser known actors and minor roles do not really affect people's opinion of a movie. Therefore, we must only select the major characters and their respective actors. Arbitrarily we will choose the top 3 actors that appear in the credits list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "smd['cast'] = smd['cast'].apply(literal_eval)\n",
    "smd['crew'] = smd['crew'].apply(literal_eval)\n",
    "smd['keywords'] = smd['keywords'].apply(literal_eval)\n",
    "smd['cast_size'] = smd['cast'].apply(lambda x: len(x))\n",
    "smd['crew_size'] = smd['crew'].apply(lambda x: len(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_director(x):\n",
    "    for i in x:\n",
    "        if i['job'] == 'Director':\n",
    "            return i['name']\n",
    "    return np.nan"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "smd['director'] = smd['crew'].apply(get_director)\n",
    "smd['cast'] = smd['cast'].apply(lambda x: [i['name'] for i in x] if isinstance(x, list) else [])\n",
    "smd['cast'] = smd['cast'].apply(lambda x: x[:3] if len(x) >=3 else x)\n",
    "smd['keywords'] = smd['keywords'].apply(lambda x: [i['name'] for i in x] if isinstance(x, list) else [])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Approach to building the recommender is going to be extremely hacky. \n",
    "\n",
    "* What I plan on doing is creating a metadata dump for every movie which consists of genres, director, main actors and keywords. \n",
    "\n",
    "* I then use a __Count Vectorizer__ to create our __count matrix__\n",
    "\n",
    "* The remaining steps are similar to what we did earlier: we calculate the cosine similarities and return movies that are most similar."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are steps I follow in the preparation of my genres and credits data:\n",
    "\n",
    "1. __Strip Spaces and Convert to Lowercase__ from all our features. This way, our engine will not confuse between __Johnny Depp and Johnny Galecki.__\n",
    "2. __Mention Director 2 times__ to give it __more weight relative to the entire cast.__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "smd['cast'] = smd['cast'].apply(lambda x: [str.lower(i.replace(\" \", \"\")) for i in x])\n",
    "smd['director'] = smd['director'].astype('str').apply(lambda x: str.lower(x.replace(\" \", \"\")))\n",
    "smd['director'] = smd['director'].apply(lambda x: [x,x, x])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Keywords**\n",
    "\n",
    "* We will do a small amount of pre-processing of our keywords before putting them to any use. \n",
    "* we __calculate the frequenct counts of every keyword__ that appears in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "independent film        610\n",
       "woman director          550\n",
       "murder                  399\n",
       "duringcreditsstinger    327\n",
       "based on novel          318\n",
       "Name: keyword, dtype: int64"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s = smd.apply(lambda x: pd.Series(x['keywords']),axis=1).stack().reset_index(level=1, drop=True)\n",
    "s.name = 'keyword'\n",
    "s = s.value_counts()\n",
    "s[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Keywords occur in frequencies ranging from 1 to 610.\n",
    "* We do not have any use for keywords that occur only once. \n",
    "* Therefore, these can be safely removed. \n",
    "* Finally, we will convert every word to its stem so that words such as __Dogs__ and __Dog__ are considered the same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "s = s[s > 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'dog'"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Just an example\n",
    "stemmer = SnowballStemmer('english')\n",
    "stemmer.stem('dogs')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "def filter_keywords(x):\n",
    "    words = []\n",
    "    for i in x:\n",
    "        if i in s:\n",
    "            words.append(i)\n",
    "    return words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "smd['keywords'] = smd['keywords'].apply(filter_keywords)\n",
    "smd['keywords'] = smd['keywords'].apply(lambda x: [stemmer.stem(i) for i in x])\n",
    "smd['keywords'] = smd['keywords'].apply(lambda x: [str.lower(i.replace(\" \", \"\")) for i in x])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "smd['soup'] = smd['keywords'] + smd['cast'] + smd['director'] + smd['genres']\n",
    "smd['soup'] = smd['soup'].apply(lambda x: ' '.join(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "count = CountVectorizer(analyzer='word',ngram_range=(1, 2),min_df=0, stop_words='english')\n",
    "count_matrix = count.fit_transform(smd['soup'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "cosine_sim = cosine_similarity(count_matrix, count_matrix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "smd = smd.reset_index()\n",
    "titles = smd['title']\n",
    "indices = pd.Series(smd.index, index=smd['title'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* We will reuse the get_recommendations function that we had written earlier. \n",
    "* Since our cosine similarity scores have changed, we expect it to give us different (and probably better) results.\n",
    "* Let us check for __The Dark Knight__ again and see what recommendations I get this time around."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6218              Batman Begins\n",
       "6623               The Prestige\n",
       "8031      The Dark Knight Rises\n",
       "2085                  Following\n",
       "7648                  Inception\n",
       "4145                   Insomnia\n",
       "3381                    Memento\n",
       "8613               Interstellar\n",
       "8927    Kidnapping Mr. Heineken\n",
       "5943                   Thursday\n",
       "Name: title, dtype: object"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_recommendations('The Dark Knight').head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* I am much more satisfied with the results I get this time around. The recommendations seem to have recognized other Christopher Nolan movies (due to the high weightage given to director) and put them as top recommendations. \n",
    "* I enjoyed watching __The Dark Knight__ as well as some of the other ones in the list including __Batman Begins, The Prestige and The Dark Knight Rises.__\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Improvment** \n",
    "\n",
    "* We can of course experiment on this engine by trying out different weights for our features (directors, actors, genres), limiting the number of keywords that can be used in the soup, weighing genres based on their frequency, only showing movies with the same languages, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6623             The Prestige\n",
       "6218            Batman Begins\n",
       "4145                 Insomnia\n",
       "2085                Following\n",
       "8613             Interstellar\n",
       "6981          The Dark Knight\n",
       "3381                  Memento\n",
       "8031    The Dark Knight Rises\n",
       "8500                  Don Jon\n",
       "8207                   Looper\n",
       "Name: title, dtype: object"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_recommendations('Inception').head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1329                        The House of Yes\n",
       "3319                         Head Over Heels\n",
       "4763                           Freaky Friday\n",
       "7905                   Mr. Popper's Penguins\n",
       "7332              Ghosts of Girlfriends Past\n",
       "6277                        Just Like Heaven\n",
       "6959               The Spiderwick Chronicles\n",
       "5152    Confessions of a Teenage Drama Queen\n",
       "1637                         The Parent Trap\n",
       "3178                          The Ladies Man\n",
       "Name: title, dtype: object"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_recommendations('Mean Girls').head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5200                               Kill Bill: Vol. 2\n",
       "1381                                    Jackie Brown\n",
       "8905                               The Hateful Eight\n",
       "898                                   Reservoir Dogs\n",
       "4903                               Kill Bill: Vol. 1\n",
       "8310                                Django Unchained\n",
       "7280                            Inglourious Basterds\n",
       "6788                                     Death Proof\n",
       "65                               From Dusk Till Dawn\n",
       "6392    The Cutting Edge: The Magic of Movie Editing\n",
       "Name: title, dtype: object"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_recommendations('Pulp Fiction').head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Add Popularity and Ratings "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* One thing that we notice about our recommendation system is that it recommends movies regardless of ratings and \n",
    "  popularity. It is true that Batman and Robin has a lot of similar characters as compared to The Dark Knight but   \n",
    "  it was a terrible movie that shouldn't be recommended to anyone.\n",
    "\n",
    "* Therefore, we will add a mechanism to remove bad movies and return movies which are popular and have had a good\n",
    "  critical response.\n",
    "\n",
    "* I will take the top 25 movies based on similarity scores and calculate the vote of the __60th percentile__ movie. \n",
    "  Then, using this as the value of $m$, we will calculate the weighted rating of each movie using IMDB's formula \n",
    "  like we did in the Simple Recommender section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "def improved_recommendations(title):\n",
    "    idx = indices[title]\n",
    "    sim_scores = list(enumerate(cosine_sim[idx]))\n",
    "    sim_scores = sorted(sim_scores, key=lambda x: x[1], reverse=True)\n",
    "    sim_scores = sim_scores[1:26]\n",
    "    movie_indices = [i[0] for i in sim_scores]\n",
    "    \n",
    "    movies = smd.iloc[movie_indices][['title', 'vote_count', 'vote_average', 'year']]\n",
    "    vote_counts = movies[movies['vote_count'].notnull()]['vote_count'].astype('int')\n",
    "    vote_averages = movies[movies['vote_average'].notnull()]['vote_average'].astype('int')\n",
    "    C = vote_averages.mean()\n",
    "    m = vote_counts.quantile(0.60)\n",
    "    qualified = movies[(movies['vote_count'] >= m) & (movies['vote_count'].notnull()) & \n",
    "                       (movies['vote_average'].notnull())]\n",
    "    qualified['vote_count'] = qualified['vote_count'].astype('int')\n",
    "    qualified['vote_average'] = qualified['vote_average'].astype('int')\n",
    "    qualified['wr'] = qualified.apply(weighted_rating, axis=1)\n",
    "    qualified = qualified.sort_values('wr', ascending=False).head(10)\n",
    "    return qualified"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>vote_count</th>\n",
       "      <th>vote_average</th>\n",
       "      <th>year</th>\n",
       "      <th>wr</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>7648</th>\n",
       "      <td>Inception</td>\n",
       "      <td>14075</td>\n",
       "      <td>8</td>\n",
       "      <td>2010</td>\n",
       "      <td>7.917588</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8613</th>\n",
       "      <td>Interstellar</td>\n",
       "      <td>11187</td>\n",
       "      <td>8</td>\n",
       "      <td>2014</td>\n",
       "      <td>7.897107</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6623</th>\n",
       "      <td>The Prestige</td>\n",
       "      <td>4510</td>\n",
       "      <td>8</td>\n",
       "      <td>2006</td>\n",
       "      <td>7.758148</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3381</th>\n",
       "      <td>Memento</td>\n",
       "      <td>4168</td>\n",
       "      <td>8</td>\n",
       "      <td>2000</td>\n",
       "      <td>7.740175</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8031</th>\n",
       "      <td>The Dark Knight Rises</td>\n",
       "      <td>9263</td>\n",
       "      <td>7</td>\n",
       "      <td>2012</td>\n",
       "      <td>6.921448</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6218</th>\n",
       "      <td>Batman Begins</td>\n",
       "      <td>7511</td>\n",
       "      <td>7</td>\n",
       "      <td>2005</td>\n",
       "      <td>6.904127</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1532</th>\n",
       "      <td>The French Connection</td>\n",
       "      <td>435</td>\n",
       "      <td>7</td>\n",
       "      <td>1971</td>\n",
       "      <td>6.123458</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4145</th>\n",
       "      <td>Insomnia</td>\n",
       "      <td>1181</td>\n",
       "      <td>6</td>\n",
       "      <td>2002</td>\n",
       "      <td>5.797081</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7133</th>\n",
       "      <td>RockNRolla</td>\n",
       "      <td>851</td>\n",
       "      <td>6</td>\n",
       "      <td>2008</td>\n",
       "      <td>5.744969</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8026</th>\n",
       "      <td>Bullet to the Head</td>\n",
       "      <td>490</td>\n",
       "      <td>5</td>\n",
       "      <td>2013</td>\n",
       "      <td>5.115027</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                      title  vote_count  vote_average  year        wr\n",
       "7648              Inception       14075             8  2010  7.917588\n",
       "8613           Interstellar       11187             8  2014  7.897107\n",
       "6623           The Prestige        4510             8  2006  7.758148\n",
       "3381                Memento        4168             8  2000  7.740175\n",
       "8031  The Dark Knight Rises        9263             7  2012  6.921448\n",
       "6218          Batman Begins        7511             7  2005  6.904127\n",
       "1532  The French Connection         435             7  1971  6.123458\n",
       "4145               Insomnia        1181             6  2002  5.797081\n",
       "7133             RockNRolla         851             6  2008  5.744969\n",
       "8026     Bullet to the Head         490             5  2013  5.115027"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "improved_recommendations('The Dark Knight')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>vote_count</th>\n",
       "      <th>vote_average</th>\n",
       "      <th>year</th>\n",
       "      <th>wr</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>898</th>\n",
       "      <td>Reservoir Dogs</td>\n",
       "      <td>3821</td>\n",
       "      <td>8</td>\n",
       "      <td>1992</td>\n",
       "      <td>7.718986</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8310</th>\n",
       "      <td>Django Unchained</td>\n",
       "      <td>10297</td>\n",
       "      <td>7</td>\n",
       "      <td>2012</td>\n",
       "      <td>6.929017</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7280</th>\n",
       "      <td>Inglourious Basterds</td>\n",
       "      <td>6598</td>\n",
       "      <td>7</td>\n",
       "      <td>2009</td>\n",
       "      <td>6.891679</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4903</th>\n",
       "      <td>Kill Bill: Vol. 1</td>\n",
       "      <td>5091</td>\n",
       "      <td>7</td>\n",
       "      <td>2003</td>\n",
       "      <td>6.862133</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8905</th>\n",
       "      <td>The Hateful Eight</td>\n",
       "      <td>4405</td>\n",
       "      <td>7</td>\n",
       "      <td>2015</td>\n",
       "      <td>6.842588</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5200</th>\n",
       "      <td>Kill Bill: Vol. 2</td>\n",
       "      <td>4061</td>\n",
       "      <td>7</td>\n",
       "      <td>2004</td>\n",
       "      <td>6.830542</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1381</th>\n",
       "      <td>Jackie Brown</td>\n",
       "      <td>1580</td>\n",
       "      <td>7</td>\n",
       "      <td>1997</td>\n",
       "      <td>6.621790</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>65</th>\n",
       "      <td>From Dusk Till Dawn</td>\n",
       "      <td>1644</td>\n",
       "      <td>6</td>\n",
       "      <td>1996</td>\n",
       "      <td>5.842293</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6788</th>\n",
       "      <td>Death Proof</td>\n",
       "      <td>1359</td>\n",
       "      <td>6</td>\n",
       "      <td>2007</td>\n",
       "      <td>5.817225</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4764</th>\n",
       "      <td>S.W.A.T.</td>\n",
       "      <td>780</td>\n",
       "      <td>5</td>\n",
       "      <td>2003</td>\n",
       "      <td>5.087550</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     title  vote_count  vote_average  year        wr\n",
       "898         Reservoir Dogs        3821             8  1992  7.718986\n",
       "8310      Django Unchained       10297             7  2012  6.929017\n",
       "7280  Inglourious Basterds        6598             7  2009  6.891679\n",
       "4903     Kill Bill: Vol. 1        5091             7  2003  6.862133\n",
       "8905     The Hateful Eight        4405             7  2015  6.842588\n",
       "5200     Kill Bill: Vol. 2        4061             7  2004  6.830542\n",
       "1381          Jackie Brown        1580             7  1997  6.621790\n",
       "65     From Dusk Till Dawn        1644             6  1996  5.842293\n",
       "6788           Death Proof        1359             6  2007  5.817225\n",
       "4764              S.W.A.T.         780             5  2003  5.087550"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "improved_recommendations('Pulp Fiction')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Unfortunately, __Batman and Robin__ does not disappear from our recommendation list. \n",
    "* This is probably due to the fact that it is rated a 4, which is only slightly below average on TMDB. \n",
    "* It certainly doesn't deserve a 4 when amazing movies like The Dark Knight Rises has only a 7. \n",
    "* However, there is nothing much we can do about this. Therefore, we will conclude our Content Based Recommender \n",
    "  section here "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.3 CF based recommendation system"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Our content based engine suffers from some severe limitations.**\n",
    "\n",
    "* It is only capable of suggesting movies which are close to a certain movie. That is, it is not capable of capturing tastes and providing recommendations across genres.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Also, the engine that we built is not really personal in that it doesn't capture the personal tastes and biases of a user. Anyone querying our engine for recommendations based on a movie will receive the same recommendations for that movie, regardless of who (s)he is.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Therefore, in this section, we will use Collaborative Filtering to make recommendations to Movie Watchers. Collaborative Filtering is based on the idea that users similar to a me can be used to predict how much I will like a particular product or service those users have used/experienced but I have not.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* I will not be implementing Collaborative Filtering from scratch. Instead, I will use the __[Surprise library](http://surpriselib.com/)__ that used extremely powerful algorithms like __Singular Value Decomposition (SVD) to minimise RMSE (Root Mean Square Error) and give great recommendations__.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Implementation of SVD for surprise library is given on this [link](https://github.com/NicolasHug/Surprise/blob/master/surprise/prediction_algorithms/matrix_factorization.pyx)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [],
   "source": [
    "# surprise reader API to read the dataset\n",
    "reader = Reader()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = Dataset.load_from_df(ratings[['userId', 'movieId', 'rating']], reader)\n",
    "data.split(n_folds=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating RMSE, MAE of algorithm SVD.\n",
      "\n",
      "------------\n",
      "Fold 1\n",
      "RMSE: 0.9061\n",
      "MAE:  0.6978\n",
      "------------\n",
      "Fold 2\n",
      "RMSE: 0.8942\n",
      "MAE:  0.6894\n",
      "------------\n",
      "Fold 3\n",
      "RMSE: 0.8924\n",
      "MAE:  0.6856\n",
      "------------\n",
      "Fold 4\n",
      "RMSE: 0.8983\n",
      "MAE:  0.6902\n",
      "------------\n",
      "Fold 5\n",
      "RMSE: 0.8950\n",
      "MAE:  0.6911\n",
      "------------\n",
      "------------\n",
      "Mean RMSE: 0.8972\n",
      "Mean MAE : 0.6908\n",
      "------------\n",
      "------------\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "CaseInsensitiveDefaultDict(list,\n",
       "                           {'mae': [0.6977986010578162,\n",
       "                             0.6893989097172819,\n",
       "                             0.6855953736686582,\n",
       "                             0.6902436024123828,\n",
       "                             0.6910891113934926],\n",
       "                            'rmse': [0.9061093999472343,\n",
       "                             0.894157877002033,\n",
       "                             0.8924156875301242,\n",
       "                             0.8983103096964073,\n",
       "                             0.8949992619618011]})"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "svd = SVD()\n",
    "evaluate(svd, data, measures=['RMSE', 'MAE'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<surprise.prediction_algorithms.matrix_factorization.SVD at 0x7f4c22ec4d68>"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainset = data.build_full_trainset()\n",
    "svd.train(trainset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>movieId</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>31</td>\n",
       "      <td>2.5</td>\n",
       "      <td>1260759144</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1029</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1260759179</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>1061</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1260759182</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>1129</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1260759185</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>1172</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1260759205</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1</td>\n",
       "      <td>1263</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1260759151</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>1</td>\n",
       "      <td>1287</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1260759187</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1</td>\n",
       "      <td>1293</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1260759148</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>1</td>\n",
       "      <td>1339</td>\n",
       "      <td>3.5</td>\n",
       "      <td>1260759125</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1</td>\n",
       "      <td>1343</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1260759131</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>1</td>\n",
       "      <td>1371</td>\n",
       "      <td>2.5</td>\n",
       "      <td>1260759135</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>1</td>\n",
       "      <td>1405</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1260759203</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>1</td>\n",
       "      <td>1953</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1260759191</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>1</td>\n",
       "      <td>2105</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1260759139</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>1</td>\n",
       "      <td>2150</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1260759194</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>1</td>\n",
       "      <td>2193</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1260759198</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>1</td>\n",
       "      <td>2294</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1260759108</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>1</td>\n",
       "      <td>2455</td>\n",
       "      <td>2.5</td>\n",
       "      <td>1260759113</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>1</td>\n",
       "      <td>2968</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1260759200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>1</td>\n",
       "      <td>3671</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1260759117</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    userId  movieId  rating   timestamp\n",
       "0        1       31     2.5  1260759144\n",
       "1        1     1029     3.0  1260759179\n",
       "2        1     1061     3.0  1260759182\n",
       "3        1     1129     2.0  1260759185\n",
       "4        1     1172     4.0  1260759205\n",
       "5        1     1263     2.0  1260759151\n",
       "6        1     1287     2.0  1260759187\n",
       "7        1     1293     2.0  1260759148\n",
       "8        1     1339     3.5  1260759125\n",
       "9        1     1343     2.0  1260759131\n",
       "10       1     1371     2.5  1260759135\n",
       "11       1     1405     1.0  1260759203\n",
       "12       1     1953     4.0  1260759191\n",
       "13       1     2105     4.0  1260759139\n",
       "14       1     2150     3.0  1260759194\n",
       "15       1     2193     2.0  1260759198\n",
       "16       1     2294     2.0  1260759108\n",
       "17       1     2455     2.5  1260759113\n",
       "18       1     2968     1.0  1260759200\n",
       "19       1     3671     3.0  1260759117"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratings[ratings['userId'] == 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Prediction(uid=1, iid=302, r_ui=None, est=2.7480119426314853, details={'was_impossible': False})"
      ]
     },
     "execution_count": 112,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "svd.predict(1, 302)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* For movie with ID 302, we get an estimated prediction of 2.686. One startling feature of this recommender system is that it doesn't care what the movie is (or what it contains). It works purely on the basis of an assigned movie ID and tries to predict ratings based on how the other users have perceive the movie."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hybrid recommendation system"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* In this section, will try to build a simple hybrid recommender that brings together techniques we have implemented in the content based and collaborative filter based engines. This is how it will work:\n",
    "\n",
    "* **Input:** User ID and the Title of a Movie\n",
    "* **Output:** Similar movies sorted on the basis of expected ratings by that particular user."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [],
   "source": [
    "def convert_int(x):\n",
    "    try:\n",
    "        return int(x)\n",
    "    except:\n",
    "        return np.nan"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [],
   "source": [
    "id_map = pd.read_csv('./input_data/movie_dataset/links_small.csv')[['movieId', 'tmdbId']]\n",
    "id_map['tmdbId'] = id_map['tmdbId'].apply(convert_int)\n",
    "id_map.columns = ['movieId', 'id']\n",
    "id_map = id_map.merge(smd[['title', 'id']], on='id').set_index('title')\n",
    "#id_map = id_map.set_index('tmdbId')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [],
   "source": [
    "indices_map = id_map.set_index('id')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [],
   "source": [
    "def hybrid(userId, title):\n",
    "    idx = indices[title]\n",
    "    tmdbId = id_map.loc[title]['id']\n",
    "    movie_id = id_map.loc[title]['movieId']\n",
    "    sim_scores = list(enumerate(cosine_sim[int(idx)]))\n",
    "    sim_scores = sorted(sim_scores, key=lambda x: x[1], reverse=True)\n",
    "    sim_scores = sim_scores[1:26]\n",
    "    movie_indices = [i[0] for i in sim_scores]\n",
    "    movies = smd.iloc[movie_indices][['title', 'vote_count', 'vote_average', 'release_date', 'id']]\n",
    "    movies['est'] = movies['id'].apply(lambda x: svd.predict(userId, indices_map.loc[x]['movieId']).est)\n",
    "    movies = movies.sort_values('est', ascending=False)\n",
    "    return movies.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>vote_count</th>\n",
       "      <th>vote_average</th>\n",
       "      <th>release_date</th>\n",
       "      <th>id</th>\n",
       "      <th>est</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>974</th>\n",
       "      <td>Aliens</td>\n",
       "      <td>3282.0</td>\n",
       "      <td>7.7</td>\n",
       "      <td>1986-07-18</td>\n",
       "      <td>679</td>\n",
       "      <td>3.343526</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1011</th>\n",
       "      <td>The Terminator</td>\n",
       "      <td>4208.0</td>\n",
       "      <td>7.4</td>\n",
       "      <td>1984-10-26</td>\n",
       "      <td>218</td>\n",
       "      <td>3.165331</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8658</th>\n",
       "      <td>X-Men: Days of Future Past</td>\n",
       "      <td>6155.0</td>\n",
       "      <td>7.5</td>\n",
       "      <td>2014-05-15</td>\n",
       "      <td>127585</td>\n",
       "      <td>3.095831</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>522</th>\n",
       "      <td>Terminator 2: Judgment Day</td>\n",
       "      <td>4274.0</td>\n",
       "      <td>7.7</td>\n",
       "      <td>1991-07-01</td>\n",
       "      <td>280</td>\n",
       "      <td>3.005560</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8401</th>\n",
       "      <td>Star Trek Into Darkness</td>\n",
       "      <td>4479.0</td>\n",
       "      <td>7.4</td>\n",
       "      <td>2013-05-05</td>\n",
       "      <td>54138</td>\n",
       "      <td>2.845466</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>344</th>\n",
       "      <td>True Lies</td>\n",
       "      <td>1138.0</td>\n",
       "      <td>6.8</td>\n",
       "      <td>1994-07-14</td>\n",
       "      <td>36955</td>\n",
       "      <td>2.840542</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014</th>\n",
       "      <td>Fantastic Planet</td>\n",
       "      <td>140.0</td>\n",
       "      <td>7.6</td>\n",
       "      <td>1973-05-01</td>\n",
       "      <td>16306</td>\n",
       "      <td>2.820183</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>922</th>\n",
       "      <td>The Abyss</td>\n",
       "      <td>822.0</td>\n",
       "      <td>7.1</td>\n",
       "      <td>1989-08-09</td>\n",
       "      <td>2756</td>\n",
       "      <td>2.722440</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4347</th>\n",
       "      <td>Piranha Part Two: The Spawning</td>\n",
       "      <td>41.0</td>\n",
       "      <td>3.9</td>\n",
       "      <td>1981-01-01</td>\n",
       "      <td>31646</td>\n",
       "      <td>2.684467</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8231</th>\n",
       "      <td>Side by Side</td>\n",
       "      <td>111.0</td>\n",
       "      <td>7.1</td>\n",
       "      <td>2012-08-19</td>\n",
       "      <td>110354</td>\n",
       "      <td>2.647725</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                               title  vote_count  vote_average release_date  \\\n",
       "974                           Aliens      3282.0           7.7   1986-07-18   \n",
       "1011                  The Terminator      4208.0           7.4   1984-10-26   \n",
       "8658      X-Men: Days of Future Past      6155.0           7.5   2014-05-15   \n",
       "522       Terminator 2: Judgment Day      4274.0           7.7   1991-07-01   \n",
       "8401         Star Trek Into Darkness      4479.0           7.4   2013-05-05   \n",
       "344                        True Lies      1138.0           6.8   1994-07-14   \n",
       "2014                Fantastic Planet       140.0           7.6   1973-05-01   \n",
       "922                        The Abyss       822.0           7.1   1989-08-09   \n",
       "4347  Piranha Part Two: The Spawning        41.0           3.9   1981-01-01   \n",
       "8231                    Side by Side       111.0           7.1   2012-08-19   \n",
       "\n",
       "          id       est  \n",
       "974      679  3.343526  \n",
       "1011     218  3.165331  \n",
       "8658  127585  3.095831  \n",
       "522      280  3.005560  \n",
       "8401   54138  2.845466  \n",
       "344    36955  2.840542  \n",
       "2014   16306  2.820183  \n",
       "922     2756  2.722440  \n",
       "4347   31646  2.684467  \n",
       "8231  110354  2.647725  "
      ]
     },
     "execution_count": 104,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hybrid(1, 'Avatar')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>vote_count</th>\n",
       "      <th>vote_average</th>\n",
       "      <th>release_date</th>\n",
       "      <th>id</th>\n",
       "      <th>est</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1011</th>\n",
       "      <td>The Terminator</td>\n",
       "      <td>4208.0</td>\n",
       "      <td>7.4</td>\n",
       "      <td>1984-10-26</td>\n",
       "      <td>218</td>\n",
       "      <td>3.956651</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>522</th>\n",
       "      <td>Terminator 2: Judgment Day</td>\n",
       "      <td>4274.0</td>\n",
       "      <td>7.7</td>\n",
       "      <td>1991-07-01</td>\n",
       "      <td>280</td>\n",
       "      <td>3.951281</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>974</th>\n",
       "      <td>Aliens</td>\n",
       "      <td>3282.0</td>\n",
       "      <td>7.7</td>\n",
       "      <td>1986-07-18</td>\n",
       "      <td>679</td>\n",
       "      <td>3.902304</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8401</th>\n",
       "      <td>Star Trek Into Darkness</td>\n",
       "      <td>4479.0</td>\n",
       "      <td>7.4</td>\n",
       "      <td>2013-05-05</td>\n",
       "      <td>54138</td>\n",
       "      <td>3.789442</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8658</th>\n",
       "      <td>X-Men: Days of Future Past</td>\n",
       "      <td>6155.0</td>\n",
       "      <td>7.5</td>\n",
       "      <td>2014-05-15</td>\n",
       "      <td>127585</td>\n",
       "      <td>3.785119</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014</th>\n",
       "      <td>Fantastic Planet</td>\n",
       "      <td>140.0</td>\n",
       "      <td>7.6</td>\n",
       "      <td>1973-05-01</td>\n",
       "      <td>16306</td>\n",
       "      <td>3.718719</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8231</th>\n",
       "      <td>Side by Side</td>\n",
       "      <td>111.0</td>\n",
       "      <td>7.1</td>\n",
       "      <td>2012-08-19</td>\n",
       "      <td>110354</td>\n",
       "      <td>3.634942</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>922</th>\n",
       "      <td>The Abyss</td>\n",
       "      <td>822.0</td>\n",
       "      <td>7.1</td>\n",
       "      <td>1989-08-09</td>\n",
       "      <td>2756</td>\n",
       "      <td>3.614626</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>344</th>\n",
       "      <td>True Lies</td>\n",
       "      <td>1138.0</td>\n",
       "      <td>6.8</td>\n",
       "      <td>1994-07-14</td>\n",
       "      <td>36955</td>\n",
       "      <td>3.549296</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5310</th>\n",
       "      <td>Frank Herbert's Dune</td>\n",
       "      <td>114.0</td>\n",
       "      <td>6.7</td>\n",
       "      <td>2000-12-03</td>\n",
       "      <td>876</td>\n",
       "      <td>3.483959</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                           title  vote_count  vote_average release_date  \\\n",
       "1011              The Terminator      4208.0           7.4   1984-10-26   \n",
       "522   Terminator 2: Judgment Day      4274.0           7.7   1991-07-01   \n",
       "974                       Aliens      3282.0           7.7   1986-07-18   \n",
       "8401     Star Trek Into Darkness      4479.0           7.4   2013-05-05   \n",
       "8658  X-Men: Days of Future Past      6155.0           7.5   2014-05-15   \n",
       "2014            Fantastic Planet       140.0           7.6   1973-05-01   \n",
       "8231                Side by Side       111.0           7.1   2012-08-19   \n",
       "922                    The Abyss       822.0           7.1   1989-08-09   \n",
       "344                    True Lies      1138.0           6.8   1994-07-14   \n",
       "5310        Frank Herbert's Dune       114.0           6.7   2000-12-03   \n",
       "\n",
       "          id       est  \n",
       "1011     218  3.956651  \n",
       "522      280  3.951281  \n",
       "974      679  3.902304  \n",
       "8401   54138  3.789442  \n",
       "8658  127585  3.785119  \n",
       "2014   16306  3.718719  \n",
       "8231  110354  3.634942  \n",
       "922     2756  3.614626  \n",
       "344    36955  3.549296  \n",
       "5310     876  3.483959  "
      ]
     },
     "execution_count": 105,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hybrid(5000, 'Avatar')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>vote_count</th>\n",
       "      <th>vote_average</th>\n",
       "      <th>release_date</th>\n",
       "      <th>id</th>\n",
       "      <th>est</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>522</th>\n",
       "      <td>Terminator 2: Judgment Day</td>\n",
       "      <td>4274.0</td>\n",
       "      <td>7.7</td>\n",
       "      <td>1991-07-01</td>\n",
       "      <td>280</td>\n",
       "      <td>3.951281</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>974</th>\n",
       "      <td>Aliens</td>\n",
       "      <td>3282.0</td>\n",
       "      <td>7.7</td>\n",
       "      <td>1986-07-18</td>\n",
       "      <td>679</td>\n",
       "      <td>3.902304</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2972</th>\n",
       "      <td>Mad Max</td>\n",
       "      <td>1235.0</td>\n",
       "      <td>6.6</td>\n",
       "      <td>1979-04-12</td>\n",
       "      <td>9659</td>\n",
       "      <td>3.838765</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7502</th>\n",
       "      <td>The Book of Eli</td>\n",
       "      <td>2207.0</td>\n",
       "      <td>6.6</td>\n",
       "      <td>2010-01-14</td>\n",
       "      <td>20504</td>\n",
       "      <td>3.809446</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4173</th>\n",
       "      <td>Minority Report</td>\n",
       "      <td>2663.0</td>\n",
       "      <td>7.1</td>\n",
       "      <td>2002-06-20</td>\n",
       "      <td>180</td>\n",
       "      <td>3.758776</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6394</th>\n",
       "      <td>District B13</td>\n",
       "      <td>572.0</td>\n",
       "      <td>6.5</td>\n",
       "      <td>2004-11-09</td>\n",
       "      <td>10045</td>\n",
       "      <td>3.703780</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8231</th>\n",
       "      <td>Side by Side</td>\n",
       "      <td>111.0</td>\n",
       "      <td>7.1</td>\n",
       "      <td>2012-08-19</td>\n",
       "      <td>110354</td>\n",
       "      <td>3.634942</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7488</th>\n",
       "      <td>Avatar</td>\n",
       "      <td>12114.0</td>\n",
       "      <td>7.2</td>\n",
       "      <td>2009-12-10</td>\n",
       "      <td>19995</td>\n",
       "      <td>3.615983</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>922</th>\n",
       "      <td>The Abyss</td>\n",
       "      <td>822.0</td>\n",
       "      <td>7.1</td>\n",
       "      <td>1989-08-09</td>\n",
       "      <td>2756</td>\n",
       "      <td>3.614626</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6622</th>\n",
       "      <td>Children of Men</td>\n",
       "      <td>2120.0</td>\n",
       "      <td>7.4</td>\n",
       "      <td>2006-09-22</td>\n",
       "      <td>9693</td>\n",
       "      <td>3.614090</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                           title  vote_count  vote_average release_date  \\\n",
       "522   Terminator 2: Judgment Day      4274.0           7.7   1991-07-01   \n",
       "974                       Aliens      3282.0           7.7   1986-07-18   \n",
       "2972                     Mad Max      1235.0           6.6   1979-04-12   \n",
       "7502             The Book of Eli      2207.0           6.6   2010-01-14   \n",
       "4173             Minority Report      2663.0           7.1   2002-06-20   \n",
       "6394                District B13       572.0           6.5   2004-11-09   \n",
       "8231                Side by Side       111.0           7.1   2012-08-19   \n",
       "7488                      Avatar     12114.0           7.2   2009-12-10   \n",
       "922                    The Abyss       822.0           7.1   1989-08-09   \n",
       "6622             Children of Men      2120.0           7.4   2006-09-22   \n",
       "\n",
       "          id       est  \n",
       "522      280  3.951281  \n",
       "974      679  3.902304  \n",
       "2972    9659  3.838765  \n",
       "7502   20504  3.809446  \n",
       "4173     180  3.758776  \n",
       "6394   10045  3.703780  \n",
       "8231  110354  3.634942  \n",
       "7488   19995  3.615983  \n",
       "922     2756  3.614626  \n",
       "6622    9693  3.614090  "
      ]
     },
     "execution_count": 111,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hybrid(3423, \"The Terminator\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **We are done...!**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://raw.githubusercontent.com/jalajthanaki/Movie_recommendation_engine/master/img/3_2.gif\" \n",
    "align=\"middle\" alt=\"Figure 3_1\" data-canonical-src=\"\" style=\"width:40%;height:40%\">\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}