NLPFinalProject.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nES2XJU-u3RZ"
      },
      "source": [
        "# Introduction\n",
        "\n",
        "This notebook implements some applications of the NLP course at DSSC AA 2021-2022. \n",
        "\n",
        "In particular, I will perform topic modeling using LDA and sentiment analysis for a dataset of book reviews from amazon. The data set was found at:\n",
        "\n",
        "https://www.kaggle.com/datasets/mohamedbakhet/amazon-books-reviews\n",
        "\n",
        "The intended use cases are the following:\n",
        "- Topic modeling: enable publishing companies to understand the most popular topics of books (perhaps by year) in order to increase revenue. It could also be used by Authors to know which are the hottest topics as of late.\n",
        "- Sentiment Analysis: By predicting the sentiment of a review for a book, a company such as amazon can incorporate this knowledge in a recommender system for collaborative filtering techniques. \n",
        "\n",
        "Therefore, we will develop these techniques with these goals in mind. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JjjUBn2xvqXK"
      },
      "source": [
        "### The dataset\n",
        "\n",
        "The dataset contains of two main files:\n",
        "\n",
        "- books_data.csv: information for 200,000 books. Namely, the title, the author name, the summary, the category of the book, etc.\n",
        "- Books_rating.csv: contains information for 3 million book reviews. Namely, the book id, the book title, the user id who rated the book, the review/score, the summary of the review, and the full text of the former. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Import Statements"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ztNEIWEhyY4F",
        "outputId": "a97f6684-e18f-41c9-9c91-d5f4dafe97b9"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "c:\\Users\\andre\\anaconda3\\envs\\nlp\\lib\\site-packages\\tqdm\\auto.py:22: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
            "  from .autonotebook import tqdm as notebook_tqdm\n"
          ]
        }
      ],
      "source": [
        "# import statements for the whole notebook\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "\n",
        "\n",
        "import spacy\n",
        "\n",
        "#for language detection, uncomment\n",
        "#!pip install spacy_fastlang\n",
        "import spacy_fastlang\n",
        "\n",
        "import nltk\n",
        "#nltk.download('all')\n",
        "\n",
        "# collocations\n",
        "from nltk.collocations import BigramCollocationFinder, BigramAssocMeasures\n",
        "from nltk.corpus import stopwords\n",
        "from collections import Counter\n",
        "from tqdm import tqdm\n",
        "\n",
        "from sklearn.metrics import classification_report, f1_score, balanced_accuracy_score, precision_score, recall_score\n",
        "from sklearn.dummy import DummyClassifier\n",
        "from sklearn.model_selection import cross_val_score\n",
        "\n",
        "from transformers import AutoTokenizer,  AutoModelForSequenceClassification, pipeline, DataCollatorWithPadding\n",
        "from transformers import TrainingArguments, Trainer\n",
        "import evaluate\n",
        "\n",
        "import torch \n",
        "\n",
        "import re\n",
        "import pickle\n",
        "\n",
        "import plotly.express as px\n",
        "from plotly.subplots import make_subplots\n",
        "import plotly.graph_objects as go\n",
        "\n",
        "import stylecloud\n",
        "\n",
        "#load spacys english dictionary.\n",
        "nlp = spacy.load(\"en_core_web_sm\")\n",
        "#nlp.add_pipe(\"language_detector\")\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "eqjpgksB3rmf"
      },
      "outputs": [],
      "source": [
        "#Global variable to stop execution in notebook re runs\n",
        "FIRST_RUN = False"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Topic Modeling\n",
        "\n",
        "We focus on the first data set which contains information about the 200,000 books.\n",
        "\n",
        "We briefly recall the idea behind topic modeling: we use LDA to extract the main topics that the books are talking about in their descriptions. \n",
        "\n",
        "LDA is a document generating model which assumes that all documents are generated by sampling a specific topic distribution and sampling the words most relevant to each topic in a proportional manner.\n",
        "This assumption is faulty in some ways since it doesn't account for dependence between consecutive words.\n",
        "\n",
        "We can reverse engineer the process to find the $k$ topics being discussed and, for each topic, the top words (descriptors) associated to it.\n",
        "\n",
        "To choose the optimal value of $k$, we will maximize the UMass and CVscore, while also keeping in mind the principle of Occam's razor.\n",
        "- UMass: based on log probability of word cooccurrences\n",
        "- CVscore: uses normalized PMI and cosine similarity"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 337
        },
        "id": "3SPBtvYJzJDT",
        "outputId": "fa0b6ebd-d712-44ab-97a8-de13fb89674d"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Title</th>\n",
              "      <th>description</th>\n",
              "      <th>authors</th>\n",
              "      <th>image</th>\n",
              "      <th>previewLink</th>\n",
              "      <th>publisher</th>\n",
              "      <th>publishedDate</th>\n",
              "      <th>infoLink</th>\n",
              "      <th>categories</th>\n",
              "      <th>ratingsCount</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Its Only Art If Its Well Hung!</td>\n",
              "      <td>NaN</td>\n",
              "      <td>['Julie Strain']</td>\n",
              "      <td>http://books.google.com/books/content?id=DykPA...</td>\n",
              "      <td>http://books.google.nl/books?id=DykPAAAACAAJ&amp;d...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1996</td>\n",
              "      <td>http://books.google.nl/books?id=DykPAAAACAAJ&amp;d...</td>\n",
              "      <td>['Comics &amp; Graphic Novels']</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Dr. Seuss: American Icon</td>\n",
              "      <td>Philip Nel takes a fascinating look into the k...</td>\n",
              "      <td>['Philip Nel']</td>\n",
              "      <td>http://books.google.com/books/content?id=IjvHQ...</td>\n",
              "      <td>http://books.google.nl/books?id=IjvHQsCn_pgC&amp;p...</td>\n",
              "      <td>A&amp;C Black</td>\n",
              "      <td>2005-01-01</td>\n",
              "      <td>http://books.google.nl/books?id=IjvHQsCn_pgC&amp;d...</td>\n",
              "      <td>['Biography &amp; Autobiography']</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Wonderful Worship in Smaller Churches</td>\n",
              "      <td>This resource includes twelve principles in un...</td>\n",
              "      <td>['David R. Ray']</td>\n",
              "      <td>http://books.google.com/books/content?id=2tsDA...</td>\n",
              "      <td>http://books.google.nl/books?id=2tsDAAAACAAJ&amp;d...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>2000</td>\n",
              "      <td>http://books.google.nl/books?id=2tsDAAAACAAJ&amp;d...</td>\n",
              "      <td>['Religion']</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Whispers of the Wicked Saints</td>\n",
              "      <td>Julia Thomas finds her life spinning out of co...</td>\n",
              "      <td>['Veronica Haddon']</td>\n",
              "      <td>http://books.google.com/books/content?id=aRSIg...</td>\n",
              "      <td>http://books.google.nl/books?id=aRSIgJlq6JwC&amp;d...</td>\n",
              "      <td>iUniverse</td>\n",
              "      <td>2005-02</td>\n",
              "      <td>http://books.google.nl/books?id=aRSIgJlq6JwC&amp;d...</td>\n",
              "      <td>['Fiction']</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Nation Dance: Religion, Identity and Cultural ...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>['Edward Long']</td>\n",
              "      <td>NaN</td>\n",
              "      <td>http://books.google.nl/books?id=399SPgAACAAJ&amp;d...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>2003-03-01</td>\n",
              "      <td>http://books.google.nl/books?id=399SPgAACAAJ&amp;d...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                               Title  \\\n",
              "0                     Its Only Art If Its Well Hung!   \n",
              "1                           Dr. Seuss: American Icon   \n",
              "2              Wonderful Worship in Smaller Churches   \n",
              "3                      Whispers of the Wicked Saints   \n",
              "4  Nation Dance: Religion, Identity and Cultural ...   \n",
              "\n",
              "                                         description              authors  \\\n",
              "0                                                NaN     ['Julie Strain']   \n",
              "1  Philip Nel takes a fascinating look into the k...       ['Philip Nel']   \n",
              "2  This resource includes twelve principles in un...     ['David R. Ray']   \n",
              "3  Julia Thomas finds her life spinning out of co...  ['Veronica Haddon']   \n",
              "4                                                NaN      ['Edward Long']   \n",
              "\n",
              "                                               image  \\\n",
              "0  http://books.google.com/books/content?id=DykPA...   \n",
              "1  http://books.google.com/books/content?id=IjvHQ...   \n",
              "2  http://books.google.com/books/content?id=2tsDA...   \n",
              "3  http://books.google.com/books/content?id=aRSIg...   \n",
              "4                                                NaN   \n",
              "\n",
              "                                         previewLink  publisher publishedDate  \\\n",
              "0  http://books.google.nl/books?id=DykPAAAACAAJ&d...        NaN          1996   \n",
              "1  http://books.google.nl/books?id=IjvHQsCn_pgC&p...  A&C Black    2005-01-01   \n",
              "2  http://books.google.nl/books?id=2tsDAAAACAAJ&d...        NaN          2000   \n",
              "3  http://books.google.nl/books?id=aRSIgJlq6JwC&d...  iUniverse       2005-02   \n",
              "4  http://books.google.nl/books?id=399SPgAACAAJ&d...        NaN    2003-03-01   \n",
              "\n",
              "                                            infoLink  \\\n",
              "0  http://books.google.nl/books?id=DykPAAAACAAJ&d...   \n",
              "1  http://books.google.nl/books?id=IjvHQsCn_pgC&d...   \n",
              "2  http://books.google.nl/books?id=2tsDAAAACAAJ&d...   \n",
              "3  http://books.google.nl/books?id=aRSIgJlq6JwC&d...   \n",
              "4  http://books.google.nl/books?id=399SPgAACAAJ&d...   \n",
              "\n",
              "                      categories  ratingsCount  \n",
              "0    ['Comics & Graphic Novels']           NaN  \n",
              "1  ['Biography & Autobiography']           NaN  \n",
              "2                   ['Religion']           NaN  \n",
              "3                    ['Fiction']           NaN  \n",
              "4                            NaN           NaN  "
            ]
          },
          "execution_count": 3,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "df_books = pd.read_csv(\"./DataSet/books_data.csv\")\n",
        "df_books.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Corpus size:  212404\n",
            "Books without a description:  68442\n",
            "Percentage of books without a description:  0.32222557013992204\n"
          ]
        }
      ],
      "source": [
        "print(\"Corpus size: \", len(df_books))\n",
        "print(\"Books without a description: \", df_books[\"description\"].isna().sum())\n",
        "print(\"Percentage of books without a description: \", df_books[\"description\"].isna().sum()/len(df_books))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "10o2JOi-DqOA"
      },
      "source": [
        "#### Step 0: Exploration\n",
        "\n",
        "We extract the description of each book. However, since also the titles of the books are useful in terms of information, we will join the two columns together and separate them with a period. \n",
        "\n",
        "Furthermore, we will filter non english documents since LDA operates under the assumption that documents are in the same language. To do this, we use **spacy_fastlang** library to detect the language."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "IxtjFvmPyRPV"
      },
      "outputs": [],
      "source": [
        "df_books = df_books.dropna(subset = [\"description\"])\n",
        "df_books[\"new_desc\"] = df_books.loc[:,\"description\"] + \". \" + df_books.loc[:,\"Title\"]\n",
        "newdate = [re.sub(r\"[^0-9].*\",\"\",date ) for date in df_books[\"publishedDate\"].apply(str)]\n",
        "df_books[\"publishedDate\"] = newdate\n",
        "df_books = df_books.drop([\"image\", \"publisher\", \"infoLink\", \"ratingsCount\", \"previewLink\"], axis = 1)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 252
        },
        "id": "UXCVxlMt1IKt",
        "outputId": "22413a55-0f03-4e29-fb90-1be03c3604bd"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Length of corpus of english books:  142149\n"
          ]
        }
      ],
      "source": [
        "if FIRST_RUN:\n",
        "    #check if the summaries are in english. Runtime is approximately 1 hr\n",
        "    lang = [nlp(doc)._.language for doc in df_books[\"new_desc\"]]\n",
        "    df_books[\"lang\"] = lang\n",
        "    df_books_en = df_books[df_books[\"lang\"] == \"en\"]\n",
        "    with open(\"./DataSet/books_en.csv\", \"wb\") as fp:\n",
        "            pickle.dump(df_books_en, fp)\n",
        "else:\n",
        "    with open(\"./DataSet/books_en.csv\", \"rb\") as fp:\n",
        "            df_books_en = pickle.load(fp)\n",
        "book_description_corpus = df_books_en[\"new_desc\"].tolist()\n",
        "\n",
        "print(\"Length of corpus of english books: \", len(book_description_corpus))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6wnN9PvsEWa8"
      },
      "source": [
        "As we can see, the final corpus contains approximatley 150,000 documents in english. Below we show an example:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 105
        },
        "id": "bjX5oD7qdMQg",
        "outputId": "b754e236-5780-4ed1-acd3-09f5bdf9284d"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "'Philip Nel takes a fascinating look into the key aspects of Seuss\\'s career - his poetry, politics, art, marketing, and place in the popular imagination.\" \"Nel argues convincingly that Dr. Seuss is one of the most influential poets in America. His nonsense verse, like that of Lewis Carroll and Edward Lear, has changed language itself, giving us new words like \"nerd.\" And Seuss\\'s famously loopy artistic style - what Nel terms an \"energetic cartoon surrealism\" - has been equally important, inspiring artists like filmmaker Tim Burton and illustrator Lane Smith. --from back cover. Dr. Seuss: American Icon'"
            ]
          },
          "execution_count": 9,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "book_description_corpus[0]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Most frequent categories\n",
        "Having filtered out non english documents, we now analyze the most frequent categories. These give an idea of the concepts which the corpus talks about. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 542
        },
        "id": "mWNsAzTCQx-o",
        "outputId": "4403dd46-459c-4d58-b2c9-8e5021f2a934"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.plotly.v1+json": {
              "config": {
                "plotlyServerURL": "https://plot.ly"
              },
              "data": [
                {
                  "alignmentgroup": "True",
                  "hovertemplate": "variable=categories<br>count=%{x}<br>=%{y}<extra></extra>",
                  "legendgroup": "categories",
                  "marker": {
                    "color": "#636efa",
                    "pattern": {
                      "shape": ""
                    }
                  },
                  "name": "categories",
                  "offsetgroup": "categories",
                  "orientation": "h",
                  "showlegend": true,
                  "textposition": "auto",
                  "type": "bar",
                  "x": [
                    2411,
                    3289,
                    3600,
                    4114,
                    5240,
                    5964,
                    6496,
                    8712,
                    8776,
                    21881
                  ],
                  "xaxis": "x",
                  "y": [
                    "['Science']",
                    "['Juvenile Nonfiction']",
                    "['Social Science']",
                    "['Computers']",
                    "['Business & Economics']",
                    "['Biography & Autobiography']",
                    "['Juvenile Fiction']",
                    "['Religion']",
                    "['History']",
                    "['Fiction']"
                  ],
                  "yaxis": "y"
                }
              ],
              "layout": {
                "barmode": "relative",
                "legend": {
                  "title": {
                    "text": "variable"
                  },
                  "tracegroupgap": 0
                },
                "template": {
                  "data": {
                    "bar": [
                      {
                        "error_x": {
                          "color": "#2a3f5f"
                        },
                        "error_y": {
                          "color": "#2a3f5f"
                        },
                        "marker": {
                          "line": {
                            "color": "#E5ECF6",
                            "width": 0.5
                          },
                          "pattern": {
                            "fillmode": "overlay",
                            "size": 10,
                            "solidity": 0.2
                          }
                        },
                        "type": "bar"
                      }
                    ],
                    "barpolar": [
                      {
                        "marker": {
                          "line": {
                            "color": "#E5ECF6",
                            "width": 0.5
                          },
                          "pattern": {
                            "fillmode": "overlay",
                            "size": 10,
                            "solidity": 0.2
                          }
                        },
                        "type": "barpolar"
                      }
                    ],
                    "carpet": [
                      {
                        "aaxis": {
                          "endlinecolor": "#2a3f5f",
                          "gridcolor": "white",
                          "linecolor": "white",
                          "minorgridcolor": "white",
                          "startlinecolor": "#2a3f5f"
                        },
                        "baxis": {
                          "endlinecolor": "#2a3f5f",
                          "gridcolor": "white",
                          "linecolor": "white",
                          "minorgridcolor": "white",
                          "startlinecolor": "#2a3f5f"
                        },
                        "type": "carpet"
                      }
                    ],
                    "choropleth": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "type": "choropleth"
                      }
                    ],
                    "contour": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "colorscale": [
                          [
                            0,
                            "#0d0887"
                          ],
                          [
                            0.1111111111111111,
                            "#46039f"
                          ],
                          [
                            0.2222222222222222,
                            "#7201a8"
                          ],
                          [
                            0.3333333333333333,
                            "#9c179e"
                          ],
                          [
                            0.4444444444444444,
                            "#bd3786"
                          ],
                          [
                            0.5555555555555556,
                            "#d8576b"
                          ],
                          [
                            0.6666666666666666,
                            "#ed7953"
                          ],
                          [
                            0.7777777777777778,
                            "#fb9f3a"
                          ],
                          [
                            0.8888888888888888,
                            "#fdca26"
                          ],
                          [
                            1,
                            "#f0f921"
                          ]
                        ],
                        "type": "contour"
                      }
                    ],
                    "contourcarpet": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "type": "contourcarpet"
                      }
                    ],
                    "heatmap": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "colorscale": [
                          [
                            0,
                            "#0d0887"
                          ],
                          [
                            0.1111111111111111,
                            "#46039f"
                          ],
                          [
                            0.2222222222222222,
                            "#7201a8"
                          ],
                          [
                            0.3333333333333333,
                            "#9c179e"
                          ],
                          [
                            0.4444444444444444,
                            "#bd3786"
                          ],
                          [
                            0.5555555555555556,
                            "#d8576b"
                          ],
                          [
                            0.6666666666666666,
                            "#ed7953"
                          ],
                          [
                            0.7777777777777778,
                            "#fb9f3a"
                          ],
                          [
                            0.8888888888888888,
                            "#fdca26"
                          ],
                          [
                            1,
                            "#f0f921"
                          ]
                        ],
                        "type": "heatmap"
                      }
                    ],
                    "heatmapgl": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "colorscale": [
                          [
                            0,
                            "#0d0887"
                          ],
                          [
                            0.1111111111111111,
                            "#46039f"
                          ],
                          [
                            0.2222222222222222,
                            "#7201a8"
                          ],
                          [
                            0.3333333333333333,
                            "#9c179e"
                          ],
                          [
                            0.4444444444444444,
                            "#bd3786"
                          ],
                          [
                            0.5555555555555556,
                            "#d8576b"
                          ],
                          [
                            0.6666666666666666,
                            "#ed7953"
                          ],
                          [
                            0.7777777777777778,
                            "#fb9f3a"
                          ],
                          [
                            0.8888888888888888,
                            "#fdca26"
                          ],
                          [
                            1,
                            "#f0f921"
                          ]
                        ],
                        "type": "heatmapgl"
                      }
                    ],
                    "histogram": [
                      {
                        "marker": {
                          "pattern": {
                            "fillmode": "overlay",
                            "size": 10,
                            "solidity": 0.2
                          }
                        },
                        "type": "histogram"
                      }
                    ],
                    "histogram2d": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "colorscale": [
                          [
                            0,
                            "#0d0887"
                          ],
                          [
                            0.1111111111111111,
                            "#46039f"
                          ],
                          [
                            0.2222222222222222,
                            "#7201a8"
                          ],
                          [
                            0.3333333333333333,
                            "#9c179e"
                          ],
                          [
                            0.4444444444444444,
                            "#bd3786"
                          ],
                          [
                            0.5555555555555556,
                            "#d8576b"
                          ],
                          [
                            0.6666666666666666,
                            "#ed7953"
                          ],
                          [
                            0.7777777777777778,
                            "#fb9f3a"
                          ],
                          [
                            0.8888888888888888,
                            "#fdca26"
                          ],
                          [
                            1,
                            "#f0f921"
                          ]
                        ],
                        "type": "histogram2d"
                      }
                    ],
                    "histogram2dcontour": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "colorscale": [
                          [
                            0,
                            "#0d0887"
                          ],
                          [
                            0.1111111111111111,
                            "#46039f"
                          ],
                          [
                            0.2222222222222222,
                            "#7201a8"
                          ],
                          [
                            0.3333333333333333,
                            "#9c179e"
                          ],
                          [
                            0.4444444444444444,
                            "#bd3786"
                          ],
                          [
                            0.5555555555555556,
                            "#d8576b"
                          ],
                          [
                            0.6666666666666666,
                            "#ed7953"
                          ],
                          [
                            0.7777777777777778,
                            "#fb9f3a"
                          ],
                          [
                            0.8888888888888888,
                            "#fdca26"
                          ],
                          [
                            1,
                            "#f0f921"
                          ]
                        ],
                        "type": "histogram2dcontour"
                      }
                    ],
                    "mesh3d": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "type": "mesh3d"
                      }
                    ],
                    "parcoords": [
                      {
                        "line": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "parcoords"
                      }
                    ],
                    "pie": [
                      {
                        "automargin": true,
                        "type": "pie"
                      }
                    ],
                    "scatter": [
                      {
                        "fillpattern": {
                          "fillmode": "overlay",
                          "size": 10,
                          "solidity": 0.2
                        },
                        "type": "scatter"
                      }
                    ],
                    "scatter3d": [
                      {
                        "line": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scatter3d"
                      }
                    ],
                    "scattercarpet": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scattercarpet"
                      }
                    ],
                    "scattergeo": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scattergeo"
                      }
                    ],
                    "scattergl": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scattergl"
                      }
                    ],
                    "scattermapbox": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scattermapbox"
                      }
                    ],
                    "scatterpolar": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scatterpolar"
                      }
                    ],
                    "scatterpolargl": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scatterpolargl"
                      }
                    ],
                    "scatterternary": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scatterternary"
                      }
                    ],
                    "surface": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "colorscale": [
                          [
                            0,
                            "#0d0887"
                          ],
                          [
                            0.1111111111111111,
                            "#46039f"
                          ],
                          [
                            0.2222222222222222,
                            "#7201a8"
                          ],
                          [
                            0.3333333333333333,
                            "#9c179e"
                          ],
                          [
                            0.4444444444444444,
                            "#bd3786"
                          ],
                          [
                            0.5555555555555556,
                            "#d8576b"
                          ],
                          [
                            0.6666666666666666,
                            "#ed7953"
                          ],
                          [
                            0.7777777777777778,
                            "#fb9f3a"
                          ],
                          [
                            0.8888888888888888,
                            "#fdca26"
                          ],
                          [
                            1,
                            "#f0f921"
                          ]
                        ],
                        "type": "surface"
                      }
                    ],
                    "table": [
                      {
                        "cells": {
                          "fill": {
                            "color": "#EBF0F8"
                          },
                          "line": {
                            "color": "white"
                          }
                        },
                        "header": {
                          "fill": {
                            "color": "#C8D4E3"
                          },
                          "line": {
                            "color": "white"
                          }
                        },
                        "type": "table"
                      }
                    ]
                  },
                  "layout": {
                    "annotationdefaults": {
                      "arrowcolor": "#2a3f5f",
                      "arrowhead": 0,
                      "arrowwidth": 1
                    },
                    "autotypenumbers": "strict",
                    "coloraxis": {
                      "colorbar": {
                        "outlinewidth": 0,
                        "ticks": ""
                      }
                    },
                    "colorscale": {
                      "diverging": [
                        [
                          0,
                          "#8e0152"
                        ],
                        [
                          0.1,
                          "#c51b7d"
                        ],
                        [
                          0.2,
                          "#de77ae"
                        ],
                        [
                          0.3,
                          "#f1b6da"
                        ],
                        [
                          0.4,
                          "#fde0ef"
                        ],
                        [
                          0.5,
                          "#f7f7f7"
                        ],
                        [
                          0.6,
                          "#e6f5d0"
                        ],
                        [
                          0.7,
                          "#b8e186"
                        ],
                        [
                          0.8,
                          "#7fbc41"
                        ],
                        [
                          0.9,
                          "#4d9221"
                        ],
                        [
                          1,
                          "#276419"
                        ]
                      ],
                      "sequential": [
                        [
                          0,
                          "#0d0887"
                        ],
                        [
                          0.1111111111111111,
                          "#46039f"
                        ],
                        [
                          0.2222222222222222,
                          "#7201a8"
                        ],
                        [
                          0.3333333333333333,
                          "#9c179e"
                        ],
                        [
                          0.4444444444444444,
                          "#bd3786"
                        ],
                        [
                          0.5555555555555556,
                          "#d8576b"
                        ],
                        [
                          0.6666666666666666,
                          "#ed7953"
                        ],
                        [
                          0.7777777777777778,
                          "#fb9f3a"
                        ],
                        [
                          0.8888888888888888,
                          "#fdca26"
                        ],
                        [
                          1,
                          "#f0f921"
                        ]
                      ],
                      "sequentialminus": [
                        [
                          0,
                          "#0d0887"
                        ],
                        [
                          0.1111111111111111,
                          "#46039f"
                        ],
                        [
                          0.2222222222222222,
                          "#7201a8"
                        ],
                        [
                          0.3333333333333333,
                          "#9c179e"
                        ],
                        [
                          0.4444444444444444,
                          "#bd3786"
                        ],
                        [
                          0.5555555555555556,
                          "#d8576b"
                        ],
                        [
                          0.6666666666666666,
                          "#ed7953"
                        ],
                        [
                          0.7777777777777778,
                          "#fb9f3a"
                        ],
                        [
                          0.8888888888888888,
                          "#fdca26"
                        ],
                        [
                          1,
                          "#f0f921"
                        ]
                      ]
                    },
                    "colorway": [
                      "#636efa",
                      "#EF553B",
                      "#00cc96",
                      "#ab63fa",
                      "#FFA15A",
                      "#19d3f3",
                      "#FF6692",
                      "#B6E880",
                      "#FF97FF",
                      "#FECB52"
                    ],
                    "font": {
                      "color": "#2a3f5f"
                    },
                    "geo": {
                      "bgcolor": "white",
                      "lakecolor": "white",
                      "landcolor": "#E5ECF6",
                      "showlakes": true,
                      "showland": true,
                      "subunitcolor": "white"
                    },
                    "hoverlabel": {
                      "align": "left"
                    },
                    "hovermode": "closest",
                    "mapbox": {
                      "style": "light"
                    },
                    "paper_bgcolor": "white",
                    "plot_bgcolor": "#E5ECF6",
                    "polar": {
                      "angularaxis": {
                        "gridcolor": "white",
                        "linecolor": "white",
                        "ticks": ""
                      },
                      "bgcolor": "#E5ECF6",
                      "radialaxis": {
                        "gridcolor": "white",
                        "linecolor": "white",
                        "ticks": ""
                      }
                    },
                    "scene": {
                      "xaxis": {
                        "backgroundcolor": "#E5ECF6",
                        "gridcolor": "white",
                        "gridwidth": 2,
                        "linecolor": "white",
                        "showbackground": true,
                        "ticks": "",
                        "zerolinecolor": "white"
                      },
                      "yaxis": {
                        "backgroundcolor": "#E5ECF6",
                        "gridcolor": "white",
                        "gridwidth": 2,
                        "linecolor": "white",
                        "showbackground": true,
                        "ticks": "",
                        "zerolinecolor": "white"
                      },
                      "zaxis": {
                        "backgroundcolor": "#E5ECF6",
                        "gridcolor": "white",
                        "gridwidth": 2,
                        "linecolor": "white",
                        "showbackground": true,
                        "ticks": "",
                        "zerolinecolor": "white"
                      }
                    },
                    "shapedefaults": {
                      "line": {
                        "color": "#2a3f5f"
                      }
                    },
                    "ternary": {
                      "aaxis": {
                        "gridcolor": "white",
                        "linecolor": "white",
                        "ticks": ""
                      },
                      "baxis": {
                        "gridcolor": "white",
                        "linecolor": "white",
                        "ticks": ""
                      },
                      "bgcolor": "#E5ECF6",
                      "caxis": {
                        "gridcolor": "white",
                        "linecolor": "white",
                        "ticks": ""
                      }
                    },
                    "title": {
                      "x": 0.05
                    },
                    "xaxis": {
                      "automargin": true,
                      "gridcolor": "white",
                      "linecolor": "white",
                      "ticks": "",
                      "title": {
                        "standoff": 15
                      },
                      "zerolinecolor": "white",
                      "zerolinewidth": 2
                    },
                    "yaxis": {
                      "automargin": true,
                      "gridcolor": "white",
                      "linecolor": "white",
                      "ticks": "",
                      "title": {
                        "standoff": 15
                      },
                      "zerolinecolor": "white",
                      "zerolinewidth": 2
                    }
                  }
                },
                "title": {
                  "text": "Top 10 Categories"
                },
                "xaxis": {
                  "anchor": "y",
                  "domain": [
                    0,
                    1
                  ],
                  "title": {
                    "text": "count"
                  }
                },
                "yaxis": {
                  "anchor": "x",
                  "domain": [
                    0,
                    1
                  ],
                  "title": {
                    "text": ""
                  }
                }
              }
            }
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "cat_num = 10\n",
        "fig = px.bar(df_books_en[\"categories\"].value_counts()[:cat_num][::-1], orientation = \"h\",\n",
        "             title = f\"Top {cat_num} Categories\", labels={\"index\": \"\", \"value\": \"count\"})\n",
        "fig.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "As we can see, the most frequent topic is related to fiction, while other topics such as religion and history are also very popular. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Top 25 Categories:\n",
            "\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "['Cooking']                        2211\n",
              "['Sports & Recreation']            2111\n",
              "['Family & Relationships']         2052\n",
              "['Literary Criticism']             2006\n",
              "['Medical']                        1928\n",
              "['Music']                          1921\n",
              "['Health & Fitness']               1899\n",
              "['Body, Mind & Spirit']            1883\n",
              "['Language Arts & Disciplines']    1876\n",
              "['Political Science']              1841\n",
              "['Art']                            1835\n",
              "['Psychology']                     1777\n",
              "['Philosophy']                     1731\n",
              "['Travel']                         1621\n",
              "['Technology & Engineering']       1493\n",
              "['Self-Help']                      1429\n",
              "['Poetry']                         1348\n",
              "['Foreign Language Study']         1284\n",
              "['Performing Arts']                1262\n",
              "['Crafts & Hobbies']               1260\n",
              "['Reference']                      1116\n",
              "['Mathematics']                    1085\n",
              "['Comics & Graphic Novels']        1081\n",
              "['Nature']                         1080\n",
              "['Architecture']                    864\n",
              "['Transportation']                  859\n",
              "['Law']                             781\n",
              "Name: categories, dtype: int64"
            ]
          },
          "execution_count": 22,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "print(\"Top 25 Categories:\\n\")\n",
        "df_books_en[\"categories\"].value_counts()[11:38]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KxrJpjeX3S4h"
      },
      "source": [
        "#### Step 1: Preprocessing\n",
        "\n",
        "Before using LDA, we must do some aggressive preprocessing in order to reduce variance to a minimum. The proprocessing for LDA has to be much more aggressive than other tasks since we want to capture important words in order to define our topics clearly. \n",
        "\n",
        "For example, if we don't lemmatize or stem, we could find that the top descriptors of a topic are simply all the conjugations of a word. This is not very informative and thus must be avoided at all costs. \n",
        "\n",
        "The preprocessing steps that we will emply are as follows:\n",
        "\n",
        "- tokenization: using spacys tokenizer.\n",
        "- normalization: removing punctuation except for hyphens and apostrophes (since spacys tokenizer is much better at handling this)\n",
        "- lemmatization: reduce each token to it's lemma.\n",
        "- stopword removal: we use spacy's predefined list of stop words.\n",
        "- joint collocations: we introduce the bigrams with PMI greater than or equal to 1.0 (for computational reasons we don't include all of them)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "izQ6ZMrl7o48"
      },
      "outputs": [],
      "source": [
        "def normalize(text):\n",
        "    '''This function removes punctuation, symbols, and numbers. Returns the text into lower case.\n",
        "      Input: a string of text\n",
        "      Output: cleaned a string of text\n",
        "    '''\n",
        "    no_punctuation = re.sub(r'[^\\w^\\s^-^\\']','',text)\n",
        "    no_numbers = re.sub(r'[0-9+]', '0', no_punctuation)\n",
        "    downcase = no_numbers.lower()\n",
        "    return downcase\n",
        "\n",
        "def clean_corpus(corpus, stopword_removal = True):\n",
        "    '''This function preprocesses each document of the corpus by first normalizing it and then keeping only the lemma of the word\n",
        "     if the word is not a stop word.\n",
        "     Input: a list of strings (documents)\n",
        "     Outout: a list of strings (documents) which have been preprocessed.\n",
        "    '''\n",
        "    clean_corpus = [] \n",
        "    if stopword_removal:\n",
        "        for document in corpus:\n",
        "            lowered_text = normalize(document)\n",
        "            clean_corpus.append(' '.join([token.lemma_ for token in nlp(lowered_text) if token.pos_ in ['NOUN', 'VERB', 'ADJ', 'ADV', 'PROPN'] and not token.is_stop]))\n",
        "    else:\n",
        "        for document in corpus:\n",
        "            lowered_text = normalize(document)\n",
        "            clean_corpus.append(' '.join([token.lemma_ for token in nlp(lowered_text) if token.pos_ in ['NOUN', 'VERB', 'ADJ', 'ADV', 'PROPN']]))\n",
        "     \n",
        "    return clean_corpus "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "id": "K_9ZfGFNZ0vw"
      },
      "outputs": [],
      "source": [
        "def collocation_finder(cleaned_corpus, min_freq = 10, thresh = 1):\n",
        "    words = [word for document in cleaned_corpus for word in document.split()]   \n",
        "    finder = BigramCollocationFinder.from_words(words)\n",
        "    bgm = BigramAssocMeasures()\n",
        "    score = bgm.mi_like\n",
        "    finder.apply_freq_filter(min_freq)\n",
        "    collocations = pd.DataFrame(finder.score_ngrams(score), columns = [\"bigram\", \"pmi\"])\n",
        "    return collocations[collocations[\"pmi\"] >= thresh]\n",
        "def replace_words_with_collocations(cleaned_corpus, collocations):\n",
        "    corpus_with_collocations = []\n",
        "    for document in tqdm(cleaned_corpus):\n",
        "        for word1, word2 in collocations:\n",
        "            document = re.sub(f'{word1} {word2}', f'{word1}_{word2}', document)\n",
        "        corpus_with_collocations.append(document)\n",
        "    return corpus_with_collocations\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ISAacyV1hgzC"
      },
      "source": [
        "We can now finally preprocess the corpus by first applying the pipeline and then finding all collocations and adding them to the documents as bigrams. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "PyLlvwNdfTuA"
      },
      "outputs": [],
      "source": [
        "#preprocess corpus and save once\n",
        "if FIRST_RUN:\n",
        "    book_description_corpus_cleaned = clean_corpus(book_description_corpus)\n",
        "\n",
        "    with open(\"./DataSet/book_description_cleaned.csv\", \"wb\") as fp:\n",
        "        pickle.dump(book_description_corpus_cleaned, fp)\n",
        "else:\n",
        "    with open(\"./DataSet/book_description_cleaned.csv\", \"rb\") as fp:\n",
        "        book_description_corpus_cleaned = pickle.load(fp)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 424
        },
        "id": "Y2W8-n1hDqUT",
        "outputId": "82852f57-d2a0-489b-8b32-920b16e751a7"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>bigram</th>\n",
              "      <th>pmi</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>(united, states)</td>\n",
              "      <td>3860.036858</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>(york, times)</td>\n",
              "      <td>1570.857642</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>(new, york)</td>\n",
              "      <td>1257.851052</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>(los, angeles)</td>\n",
              "      <td>742.395737</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>(san, francisco)</td>\n",
              "      <td>388.615226</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1300</th>\n",
              "      <td>(professionally, typeset)</td>\n",
              "      <td>1.007619</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1301</th>\n",
              "      <td>(p, lovecraft)</td>\n",
              "      <td>1.001788</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1302</th>\n",
              "      <td>(british, columbia)</td>\n",
              "      <td>1.001537</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1303</th>\n",
              "      <td>(fort, sumter)</td>\n",
              "      <td>1.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1304</th>\n",
              "      <td>(schiffer, militaryaviation)</td>\n",
              "      <td>1.000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>1305 rows × 2 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "                            bigram          pmi\n",
              "0                 (united, states)  3860.036858\n",
              "1                    (york, times)  1570.857642\n",
              "2                      (new, york)  1257.851052\n",
              "3                   (los, angeles)   742.395737\n",
              "4                 (san, francisco)   388.615226\n",
              "...                            ...          ...\n",
              "1300     (professionally, typeset)     1.007619\n",
              "1301                (p, lovecraft)     1.001788\n",
              "1302           (british, columbia)     1.001537\n",
              "1303                (fort, sumter)     1.000000\n",
              "1304  (schiffer, militaryaviation)     1.000000\n",
              "\n",
              "[1305 rows x 2 columns]"
            ]
          },
          "execution_count": 16,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "collocations = collocation_finder(book_description_corpus_cleaned)\n",
        "collocations"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "As we can see, we have a total of 1305 collocations to add to our corpus. This is computationally intensive since for each document we must search for all the collocations and substitute with the bigram.\n",
        "However, these bigrams help to add information to our descriptors since they free up space for other words rather than occupying two slots. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "xI5WLufUDh9j"
      },
      "outputs": [],
      "source": [
        "if FIRST_RUN:\n",
        "    book_description_corpus_cleaned_collocations = replace_words_with_collocations(book_description_corpus_cleaned, collocations[\"bigram\"])\n",
        "    \n",
        "    with open(\"./DataSet/books_description_corpus_cleaned_with_collocations.csv\", \"wb\") as fp:\n",
        "        pickle.dump(book_description_corpus_cleaned_collocations, fp)\n",
        "else:\n",
        "    with open(\"./DataSet/books_description_corpus_cleaned_with_collocations.csv\", \"rb\") as fp:\n",
        "        book_description_corpus_cleaned_collocations = pickle.load(fp)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {},
      "outputs": [],
      "source": [
        "custom_stop = [\"book\", \"author\", \"write\", \"story\"]\n",
        "if FIRST_RUN:\n",
        "    book_description_corpus_cleaned_collocations_customremoved = [ \" \".join([word for word in doc.split() if word not in custom_stop]) for doc in book_description_corpus_cleaned_collocations]\n",
        "    with open(\"./DataSet/final_LDA.csv\", \"wb\") as fp:\n",
        "        pickle.dump(book_description_corpus_cleaned_collocations_customremoved, fp)\n",
        "else:\n",
        "    with open(\"./DataSet/final_LDA.csv\", \"rb\") as fp:\n",
        "        book_description_corpus_cleaned_collocations = pickle.load(fp)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "After having run the whole preprocessing, we can finally see the results. We show the raw text, the cleaned text without collocations, and the final text with collocations. This can be seen by the appearance of the term \"united_states\" in the final example."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hzjOZWFIuIiT",
        "outputId": "e934ff60-5652-4f10-b8f7-e80bcea3abeb"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "The United States \"Explorer\" map is a classic example of the cartographic excellence National Geographic is known for. This colorful political map clearly shows state boundaries, capitals, major highways, rivers, lakes, and many major cities. Insets show detail of Alaska and Hawaii. The color palette is vibrant and stunning shaded relief provides additional texture and detail. The map is encapsulated in heavy-duty 1.6 mil laminate which makes the paper much more durable and resistant to the swelling and shrinking caused by changes in humidity. Laminated maps can be framed without the need for glass, so the fames can be much lighter and less expensive. \"Map Scale = 1:6,396,000Sheet Size = 32\" x 20.25\"\". Usa Laminated Map \n",
            "\n",
            "united states explorer map classic example cartographic excellence national geographic know colorful political map clearly show state boundary capital major highway river lake major city inset detail alaska hawaii color palette vibrant stunning shaded relief provide additional texture detail map encapsulate heavyduty mil laminate make paper durable resistant swelling shrinking cause change humidity laminate map frame need glass fame light expensive map scale 00000000sheet size usa laminate map \n",
            "\n",
            "united_states explorer map classic example cartographic excellence national geographic know colorful political map clearly show state boundary capital major highway river lake major city inset detail alaska hawaii color palette vibrant stunning shaded relief provide additional texture detail map encapsulate heavyduty mil laminate make paper durable resistant swelling shrinking cause change humidity laminate map frame need glass fame light expensive map scale 00000000sheet size usa laminate map \n",
            "\n"
          ]
        }
      ],
      "source": [
        "i = 29\n",
        "print(book_description_corpus[i], \"\\n\")\n",
        "print(book_description_corpus_cleaned[i], \"\\n\")\n",
        "print(book_description_corpus_cleaned_collocations[i], \"\\n\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NMevdcC_3xX3"
      },
      "source": [
        "#### Word Cloud\n",
        "We now look at a wordcloud to see the most frequent terms in the corpus. This will also serve to give an idea of the topics being discussed.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 30,
      "metadata": {
        "id": "j_nnluWy3064"
      },
      "outputs": [],
      "source": [
        "words = \" \".join(book_description_corpus_cleaned_collocations)\n",
        "wordcloud = stylecloud.gen_stylecloud(\n",
        "                        text = words,\n",
        "                        size = 1028,\n",
        "                        background_color = \"black\",\n",
        "                        output_name = \"./Images/wordcloud.png\",\n",
        "                        icon_name = \"fas fa-book-open\"\n",
        "                        )"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "![image](./Images/wordcloud.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "As we can see, frequent words such as work, life, history, time, love, relationship, are very suggestive of the topics present in the corpus. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dPZnWbiU0ZFM"
      },
      "source": [
        "#### Step 2: Using Gensim's LDA \n",
        "\n",
        "To apply LDA, we use the gensim library which is the most popular library for this task. In particular, we use the LDAmulticore library to pararellize the work and speed the computation. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "Ehyu2L9C0Yjz"
      },
      "outputs": [],
      "source": [
        "from gensim.models import LdaMulticore, TfidfModel, CoherenceModel\n",
        "from gensim.corpora import Dictionary\n",
        "import time \n",
        "import multiprocessing "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "rJEVe0GriKFj"
      },
      "outputs": [],
      "source": [
        "instances = [doc.split() for doc in book_description_corpus_cleaned_collocations]\n",
        "dictionary = Dictionary(instances)\n",
        "dictionary.filter_extremes(no_below = 50, no_above = 0.5)\n",
        "\n",
        "ldacorpus = [dictionary.doc2bow(text) for text in instances]\n",
        "tfidfmodel = TfidfModel(ldacorpus)\n",
        "model_corpus = tfidfmodel[ldacorpus]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0KBcAy53AZVl",
        "outputId": "62c7fe75-2257-45a2-b149-da6ec6efd809"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "resource include principle understand small church worship practice planning worship few people suggestion congregational study wonderful worship small church\n",
            "['resource', 'include', 'principle', 'understand', 'small', 'church', 'worship', 'practice', 'planning', 'worship', 'few', 'people', 'suggestion', 'congregational', 'study', 'wonderful', 'worship', 'small', 'church']\n",
            "[(51, 2), (52, 1), (53, 1), (54, 1), (55, 1), (56, 1), (57, 1), (58, 1), (59, 1), (60, 2), (61, 1), (62, 1), (63, 1), (64, 1), (65, 3)]\n",
            "[(51, 0.31376452057035326), (52, 0.32180463855405556), (53, 0.2693150375467911), (54, 0.0664025240567632), (55, 0.09767008421079369), (56, 0.19392337346329966), (57, 0.12066432336191125), (58, 0.1504597029346263), (59, 0.1314502090225251), (60, 0.28331521221527595), (61, 0.09744878989309225), (62, 0.1936860437731818), (63, 0.11980735412718313), (64, 0.18348738900751174), (65, 0.666431806837864)]\n"
          ]
        }
      ],
      "source": [
        "i = 1 \n",
        "print(book_description_corpus_cleaned_collocations[i])\n",
        "print(instances[i])\n",
        "print(ldacorpus[i])\n",
        "print(model_corpus[i])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B1QHDuazlMja"
      },
      "source": [
        "#### Step 3: Choosing number of topics\n",
        "To choose the number of topics we use the UMass score and CV score as mentioned earlier. In particular, we have to find a number of topics for which these scores have a local maxima. We also keep in mind the principle of Occams razor which states that simpler models are preferred. Therefore, sometimes if we the maximum occurs when there is a lot of topics, it may be preferable to pick another local maxima which is lower but has less topics. This is because we are more interested in explainability rather than maximizing a metric. In fact, at the end of the day, we are interesting in **knowing** which topics are being talked about, not having the best scores. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zTk9n3hAlTOn",
        "outputId": "859d0fb8-e672-4f27-f42a-b8cdc286ba83"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Number of topics: 2, UMass Score: -2.009851077821883, CV Score: 0.291924723457169 \n",
            "\n",
            "Number of topics: 3, UMass Score: -2.0423419197352763, CV Score: 0.3210428082390775 \n",
            "\n",
            "Number of topics: 4, UMass Score: -2.0361261627812417, CV Score: 0.3147624528182584 \n",
            "\n",
            "Number of topics: 5, UMass Score: -2.0652016202076022, CV Score: 0.28941978677360825 \n",
            "\n",
            "Number of topics: 6, UMass Score: -2.1309735738758584, CV Score: 0.30745509663656906 \n",
            "\n",
            "Number of topics: 7, UMass Score: -2.0819305045052485, CV Score: 0.3298985172566807 \n",
            "\n",
            "Number of topics: 8, UMass Score: -2.0710492831339113, CV Score: 0.3148824563295076 \n",
            "\n",
            "Number of topics: 9, UMass Score: -2.090810064113382, CV Score: 0.3302618115973315 \n",
            "\n",
            "Number of topics: 10, UMass Score: -2.0308307642940364, CV Score: 0.3398214271640517 \n",
            "\n",
            "Number of topics: 11, UMass Score: -2.036447362692701, CV Score: 0.3283298407587069 \n",
            "\n",
            "Number of topics: 12, UMass Score: -2.069065092532882, CV Score: 0.33486920756113303 \n",
            "\n",
            "Number of topics: 13, UMass Score: -2.1471898762122574, CV Score: 0.3191794169593257 \n",
            "\n",
            "Number of topics: 14, UMass Score: -2.059963854619938, CV Score: 0.31230142319037224 \n",
            "\n",
            "Number of topics: 15, UMass Score: -2.094732943207696, CV Score: 0.33716512502126966 \n",
            "\n",
            "Number of topics: 16, UMass Score: -2.101444423375238, CV Score: 0.31189354036982 \n",
            "\n",
            "Number of topics: 17, UMass Score: -2.0820379934448754, CV Score: 0.3223987845836904 \n",
            "\n",
            "Number of topics: 18, UMass Score: -2.0915576349754175, CV Score: 0.3145265949911885 \n",
            "\n",
            "Number of topics: 19, UMass Score: -2.1161041734766317, CV Score: 0.31636079747347773 \n",
            "\n",
            "Number of topics: 20, UMass Score: -2.147949755827603, CV Score: 0.3037245432915203 \n",
            "\n"
          ]
        }
      ],
      "source": [
        "coherence_values = []\n",
        "\n",
        "dev_size = 100000\n",
        "eval_size = 40000\n",
        "\n",
        "for num_topics in range(2, 21):\n",
        "    model = LdaMulticore(corpus=model_corpus[:dev_size], \n",
        "                         id2word=dictionary, \n",
        "                         num_topics=num_topics,\n",
        "                         random_state = 2,\n",
        "                         alpha = 0.01,\n",
        "                         eta = 'auto')\n",
        "\n",
        "    coherencemodel_umass = CoherenceModel(model=model, \n",
        "                                          texts=instances[dev_size:dev_size+eval_size], \n",
        "                                          dictionary=dictionary, \n",
        "                                          coherence='u_mass')\n",
        "\n",
        "    coherencemodel_cv = CoherenceModel(model=model, \n",
        "                                       texts=instances[dev_size:dev_size+eval_size], \n",
        "                                       dictionary=dictionary, \n",
        "                                       coherence='c_v')\n",
        "\n",
        "    umass_score = coherencemodel_umass.get_coherence()\n",
        "    cv_score = coherencemodel_cv.get_coherence()\n",
        "    print(f\"Number of topics: {num_topics}, UMass Score: {umass_score}, CV Score: {cv_score} \\n\")\n",
        "    coherence_values.append((num_topics, umass_score, cv_score))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 617
        },
        "id": "p0s8q4NmophY",
        "outputId": "68bd69d5-da7a-486a-ba54-5a663c0cb184"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.plotly.v1+json": {
              "config": {
                "plotlyServerURL": "https://plot.ly"
              },
              "data": [
                {
                  "type": "scatter",
                  "x": [
                    2,
                    3,
                    4,
                    5,
                    6,
                    7,
                    8,
                    9,
                    10,
                    11,
                    12,
                    13,
                    14,
                    15,
                    16,
                    17,
                    18,
                    19,
                    20
                  ],
                  "xaxis": "x",
                  "y": [
                    -2.009851077821883,
                    -2.0423419197352763,
                    -2.0361261627812417,
                    -2.0652016202076022,
                    -2.1309735738758584,
                    -2.0819305045052485,
                    -2.0710492831339113,
                    -2.090810064113382,
                    -2.0308307642940364,
                    -2.036447362692701,
                    -2.069065092532882,
                    -2.1471898762122574,
                    -2.059963854619938,
                    -2.094732943207696,
                    -2.101444423375238,
                    -2.0820379934448754,
                    -2.0915576349754175,
                    -2.1161041734766317,
                    -2.147949755827603
                  ],
                  "yaxis": "y"
                },
                {
                  "type": "scatter",
                  "x": [
                    2,
                    3,
                    4,
                    5,
                    6,
                    7,
                    8,
                    9,
                    10,
                    11,
                    12,
                    13,
                    14,
                    15,
                    16,
                    17,
                    18,
                    19,
                    20
                  ],
                  "xaxis": "x2",
                  "y": [
                    0.291924723457169,
                    0.3210428082390775,
                    0.3147624528182584,
                    0.28941978677360825,
                    0.30745509663656906,
                    0.3298985172566807,
                    0.3148824563295076,
                    0.3302618115973315,
                    0.3398214271640517,
                    0.3283298407587069,
                    0.33486920756113303,
                    0.3191794169593257,
                    0.31230142319037224,
                    0.33716512502126966,
                    0.31189354036982,
                    0.3223987845836904,
                    0.3145265949911885,
                    0.31636079747347773,
                    0.3037245432915203
                  ],
                  "yaxis": "y2"
                }
              ],
              "layout": {
                "annotations": [
                  {
                    "font": {
                      "size": 16
                    },
                    "showarrow": false,
                    "text": "UMass Score",
                    "x": 0.225,
                    "xanchor": "center",
                    "xref": "paper",
                    "y": 1,
                    "yanchor": "bottom",
                    "yref": "paper"
                  },
                  {
                    "font": {
                      "size": 16
                    },
                    "showarrow": false,
                    "text": "CV Score",
                    "x": 0.775,
                    "xanchor": "center",
                    "xref": "paper",
                    "y": 1,
                    "yanchor": "bottom",
                    "yref": "paper"
                  }
                ],
                "height": 600,
                "template": {
                  "data": {
                    "bar": [
                      {
                        "error_x": {
                          "color": "#2a3f5f"
                        },
                        "error_y": {
                          "color": "#2a3f5f"
                        },
                        "marker": {
                          "line": {
                            "color": "#E5ECF6",
                            "width": 0.5
                          },
                          "pattern": {
                            "fillmode": "overlay",
                            "size": 10,
                            "solidity": 0.2
                          }
                        },
                        "type": "bar"
                      }
                    ],
                    "barpolar": [
                      {
                        "marker": {
                          "line": {
                            "color": "#E5ECF6",
                            "width": 0.5
                          },
                          "pattern": {
                            "fillmode": "overlay",
                            "size": 10,
                            "solidity": 0.2
                          }
                        },
                        "type": "barpolar"
                      }
                    ],
                    "carpet": [
                      {
                        "aaxis": {
                          "endlinecolor": "#2a3f5f",
                          "gridcolor": "white",
                          "linecolor": "white",
                          "minorgridcolor": "white",
                          "startlinecolor": "#2a3f5f"
                        },
                        "baxis": {
                          "endlinecolor": "#2a3f5f",
                          "gridcolor": "white",
                          "linecolor": "white",
                          "minorgridcolor": "white",
                          "startlinecolor": "#2a3f5f"
                        },
                        "type": "carpet"
                      }
                    ],
                    "choropleth": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "type": "choropleth"
                      }
                    ],
                    "contour": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "colorscale": [
                          [
                            0,
                            "#0d0887"
                          ],
                          [
                            0.1111111111111111,
                            "#46039f"
                          ],
                          [
                            0.2222222222222222,
                            "#7201a8"
                          ],
                          [
                            0.3333333333333333,
                            "#9c179e"
                          ],
                          [
                            0.4444444444444444,
                            "#bd3786"
                          ],
                          [
                            0.5555555555555556,
                            "#d8576b"
                          ],
                          [
                            0.6666666666666666,
                            "#ed7953"
                          ],
                          [
                            0.7777777777777778,
                            "#fb9f3a"
                          ],
                          [
                            0.8888888888888888,
                            "#fdca26"
                          ],
                          [
                            1,
                            "#f0f921"
                          ]
                        ],
                        "type": "contour"
                      }
                    ],
                    "contourcarpet": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "type": "contourcarpet"
                      }
                    ],
                    "heatmap": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "colorscale": [
                          [
                            0,
                            "#0d0887"
                          ],
                          [
                            0.1111111111111111,
                            "#46039f"
                          ],
                          [
                            0.2222222222222222,
                            "#7201a8"
                          ],
                          [
                            0.3333333333333333,
                            "#9c179e"
                          ],
                          [
                            0.4444444444444444,
                            "#bd3786"
                          ],
                          [
                            0.5555555555555556,
                            "#d8576b"
                          ],
                          [
                            0.6666666666666666,
                            "#ed7953"
                          ],
                          [
                            0.7777777777777778,
                            "#fb9f3a"
                          ],
                          [
                            0.8888888888888888,
                            "#fdca26"
                          ],
                          [
                            1,
                            "#f0f921"
                          ]
                        ],
                        "type": "heatmap"
                      }
                    ],
                    "heatmapgl": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "colorscale": [
                          [
                            0,
                            "#0d0887"
                          ],
                          [
                            0.1111111111111111,
                            "#46039f"
                          ],
                          [
                            0.2222222222222222,
                            "#7201a8"
                          ],
                          [
                            0.3333333333333333,
                            "#9c179e"
                          ],
                          [
                            0.4444444444444444,
                            "#bd3786"
                          ],
                          [
                            0.5555555555555556,
                            "#d8576b"
                          ],
                          [
                            0.6666666666666666,
                            "#ed7953"
                          ],
                          [
                            0.7777777777777778,
                            "#fb9f3a"
                          ],
                          [
                            0.8888888888888888,
                            "#fdca26"
                          ],
                          [
                            1,
                            "#f0f921"
                          ]
                        ],
                        "type": "heatmapgl"
                      }
                    ],
                    "histogram": [
                      {
                        "marker": {
                          "pattern": {
                            "fillmode": "overlay",
                            "size": 10,
                            "solidity": 0.2
                          }
                        },
                        "type": "histogram"
                      }
                    ],
                    "histogram2d": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "colorscale": [
                          [
                            0,
                            "#0d0887"
                          ],
                          [
                            0.1111111111111111,
                            "#46039f"
                          ],
                          [
                            0.2222222222222222,
                            "#7201a8"
                          ],
                          [
                            0.3333333333333333,
                            "#9c179e"
                          ],
                          [
                            0.4444444444444444,
                            "#bd3786"
                          ],
                          [
                            0.5555555555555556,
                            "#d8576b"
                          ],
                          [
                            0.6666666666666666,
                            "#ed7953"
                          ],
                          [
                            0.7777777777777778,
                            "#fb9f3a"
                          ],
                          [
                            0.8888888888888888,
                            "#fdca26"
                          ],
                          [
                            1,
                            "#f0f921"
                          ]
                        ],
                        "type": "histogram2d"
                      }
                    ],
                    "histogram2dcontour": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "colorscale": [
                          [
                            0,
                            "#0d0887"
                          ],
                          [
                            0.1111111111111111,
                            "#46039f"
                          ],
                          [
                            0.2222222222222222,
                            "#7201a8"
                          ],
                          [
                            0.3333333333333333,
                            "#9c179e"
                          ],
                          [
                            0.4444444444444444,
                            "#bd3786"
                          ],
                          [
                            0.5555555555555556,
                            "#d8576b"
                          ],
                          [
                            0.6666666666666666,
                            "#ed7953"
                          ],
                          [
                            0.7777777777777778,
                            "#fb9f3a"
                          ],
                          [
                            0.8888888888888888,
                            "#fdca26"
                          ],
                          [
                            1,
                            "#f0f921"
                          ]
                        ],
                        "type": "histogram2dcontour"
                      }
                    ],
                    "mesh3d": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "type": "mesh3d"
                      }
                    ],
                    "parcoords": [
                      {
                        "line": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "parcoords"
                      }
                    ],
                    "pie": [
                      {
                        "automargin": true,
                        "type": "pie"
                      }
                    ],
                    "scatter": [
                      {
                        "fillpattern": {
                          "fillmode": "overlay",
                          "size": 10,
                          "solidity": 0.2
                        },
                        "type": "scatter"
                      }
                    ],
                    "scatter3d": [
                      {
                        "line": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scatter3d"
                      }
                    ],
                    "scattercarpet": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scattercarpet"
                      }
                    ],
                    "scattergeo": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scattergeo"
                      }
                    ],
                    "scattergl": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scattergl"
                      }
                    ],
                    "scattermapbox": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scattermapbox"
                      }
                    ],
                    "scatterpolar": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scatterpolar"
                      }
                    ],
                    "scatterpolargl": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scatterpolargl"
                      }
                    ],
                    "scatterternary": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scatterternary"
                      }
                    ],
                    "surface": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "colorscale": [
                          [
                            0,
                            "#0d0887"
                          ],
                          [
                            0.1111111111111111,
                            "#46039f"
                          ],
                          [
                            0.2222222222222222,
                            "#7201a8"
                          ],
                          [
                            0.3333333333333333,
                            "#9c179e"
                          ],
                          [
                            0.4444444444444444,
                            "#bd3786"
                          ],
                          [
                            0.5555555555555556,
                            "#d8576b"
                          ],
                          [
                            0.6666666666666666,
                            "#ed7953"
                          ],
                          [
                            0.7777777777777778,
                            "#fb9f3a"
                          ],
                          [
                            0.8888888888888888,
                            "#fdca26"
                          ],
                          [
                            1,
                            "#f0f921"
                          ]
                        ],
                        "type": "surface"
                      }
                    ],
                    "table": [
                      {
                        "cells": {
                          "fill": {
                            "color": "#EBF0F8"
                          },
                          "line": {
                            "color": "white"
                          }
                        },
                        "header": {
                          "fill": {
                            "color": "#C8D4E3"
                          },
                          "line": {
                            "color": "white"
                          }
                        },
                        "type": "table"
                      }
                    ]
                  },
                  "layout": {
                    "annotationdefaults": {
                      "arrowcolor": "#2a3f5f",
                      "arrowhead": 0,
                      "arrowwidth": 1
                    },
                    "autotypenumbers": "strict",
                    "coloraxis": {
                      "colorbar": {
                        "outlinewidth": 0,
                        "ticks": ""
                      }
                    },
                    "colorscale": {
                      "diverging": [
                        [
                          0,
                          "#8e0152"
                        ],
                        [
                          0.1,
                          "#c51b7d"
                        ],
                        [
                          0.2,
                          "#de77ae"
                        ],
                        [
                          0.3,
                          "#f1b6da"
                        ],
                        [
                          0.4,
                          "#fde0ef"
                        ],
                        [
                          0.5,
                          "#f7f7f7"
                        ],
                        [
                          0.6,
                          "#e6f5d0"
                        ],
                        [
                          0.7,
                          "#b8e186"
                        ],
                        [
                          0.8,
                          "#7fbc41"
                        ],
                        [
                          0.9,
                          "#4d9221"
                        ],
                        [
                          1,
                          "#276419"
                        ]
                      ],
                      "sequential": [
                        [
                          0,
                          "#0d0887"
                        ],
                        [
                          0.1111111111111111,
                          "#46039f"
                        ],
                        [
                          0.2222222222222222,
                          "#7201a8"
                        ],
                        [
                          0.3333333333333333,
                          "#9c179e"
                        ],
                        [
                          0.4444444444444444,
                          "#bd3786"
                        ],
                        [
                          0.5555555555555556,
                          "#d8576b"
                        ],
                        [
                          0.6666666666666666,
                          "#ed7953"
                        ],
                        [
                          0.7777777777777778,
                          "#fb9f3a"
                        ],
                        [
                          0.8888888888888888,
                          "#fdca26"
                        ],
                        [
                          1,
                          "#f0f921"
                        ]
                      ],
                      "sequentialminus": [
                        [
                          0,
                          "#0d0887"
                        ],
                        [
                          0.1111111111111111,
                          "#46039f"
                        ],
                        [
                          0.2222222222222222,
                          "#7201a8"
                        ],
                        [
                          0.3333333333333333,
                          "#9c179e"
                        ],
                        [
                          0.4444444444444444,
                          "#bd3786"
                        ],
                        [
                          0.5555555555555556,
                          "#d8576b"
                        ],
                        [
                          0.6666666666666666,
                          "#ed7953"
                        ],
                        [
                          0.7777777777777778,
                          "#fb9f3a"
                        ],
                        [
                          0.8888888888888888,
                          "#fdca26"
                        ],
                        [
                          1,
                          "#f0f921"
                        ]
                      ]
                    },
                    "colorway": [
                      "#636efa",
                      "#EF553B",
                      "#00cc96",
                      "#ab63fa",
                      "#FFA15A",
                      "#19d3f3",
                      "#FF6692",
                      "#B6E880",
                      "#FF97FF",
                      "#FECB52"
                    ],
                    "font": {
                      "color": "#2a3f5f"
                    },
                    "geo": {
                      "bgcolor": "white",
                      "lakecolor": "white",
                      "landcolor": "#E5ECF6",
                      "showlakes": true,
                      "showland": true,
                      "subunitcolor": "white"
                    },
                    "hoverlabel": {
                      "align": "left"
                    },
                    "hovermode": "closest",
                    "mapbox": {
                      "style": "light"
                    },
                    "paper_bgcolor": "white",
                    "plot_bgcolor": "#E5ECF6",
                    "polar": {
                      "angularaxis": {
                        "gridcolor": "white",
                        "linecolor": "white",
                        "ticks": ""
                      },
                      "bgcolor": "#E5ECF6",
                      "radialaxis": {
                        "gridcolor": "white",
                        "linecolor": "white",
                        "ticks": ""
                      }
                    },
                    "scene": {
                      "xaxis": {
                        "backgroundcolor": "#E5ECF6",
                        "gridcolor": "white",
                        "gridwidth": 2,
                        "linecolor": "white",
                        "showbackground": true,
                        "ticks": "",
                        "zerolinecolor": "white"
                      },
                      "yaxis": {
                        "backgroundcolor": "#E5ECF6",
                        "gridcolor": "white",
                        "gridwidth": 2,
                        "linecolor": "white",
                        "showbackground": true,
                        "ticks": "",
                        "zerolinecolor": "white"
                      },
                      "zaxis": {
                        "backgroundcolor": "#E5ECF6",
                        "gridcolor": "white",
                        "gridwidth": 2,
                        "linecolor": "white",
                        "showbackground": true,
                        "ticks": "",
                        "zerolinecolor": "white"
                      }
                    },
                    "shapedefaults": {
                      "line": {
                        "color": "#2a3f5f"
                      }
                    },
                    "ternary": {
                      "aaxis": {
                        "gridcolor": "white",
                        "linecolor": "white",
                        "ticks": ""
                      },
                      "baxis": {
                        "gridcolor": "white",
                        "linecolor": "white",
                        "ticks": ""
                      },
                      "bgcolor": "#E5ECF6",
                      "caxis": {
                        "gridcolor": "white",
                        "linecolor": "white",
                        "ticks": ""
                      }
                    },
                    "title": {
                      "x": 0.05
                    },
                    "xaxis": {
                      "automargin": true,
                      "gridcolor": "white",
                      "linecolor": "white",
                      "ticks": "",
                      "title": {
                        "standoff": 15
                      },
                      "zerolinecolor": "white",
                      "zerolinewidth": 2
                    },
                    "yaxis": {
                      "automargin": true,
                      "gridcolor": "white",
                      "linecolor": "white",
                      "ticks": "",
                      "title": {
                        "standoff": 15
                      },
                      "zerolinecolor": "white",
                      "zerolinewidth": 2
                    }
                  }
                },
                "title": {
                  "text": "Choosing Number of Topics Via UMass and CV scores"
                },
                "width": 800,
                "xaxis": {
                  "anchor": "y",
                  "domain": [
                    0,
                    0.45
                  ],
                  "title": {
                    "text": "Num. Topics"
                  }
                },
                "xaxis2": {
                  "anchor": "y2",
                  "domain": [
                    0.55,
                    1
                  ],
                  "title": {
                    "text": "Num. Topics"
                  }
                },
                "yaxis": {
                  "anchor": "x",
                  "domain": [
                    0,
                    1
                  ],
                  "title": {
                    "text": "Coherence Score"
                  }
                },
                "yaxis2": {
                  "anchor": "x2",
                  "domain": [
                    0,
                    1
                  ]
                }
              }
            }
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "scores = pd.DataFrame(coherence_values, columns=['num_topics', 'UMass', 'CV'])\n",
        "\n",
        "fig = make_subplots(rows=1, cols=2,\n",
        "                    subplot_titles=(\"UMass Score\", \"CV Score\"))\n",
        "\n",
        "fig.add_trace(\n",
        "    go.Scatter(x=scores[\"num_topics\"], y=scores[\"UMass\"]),\n",
        "    row=1, col=1\n",
        ")\n",
        "\n",
        "fig.add_trace(\n",
        "    go.Scatter(x=scores[\"num_topics\"], y=scores[\"CV\"]),\n",
        "    row=1, col=2\n",
        ")\n",
        "\n",
        "fig.update_xaxes(title_text=\"Num. Topics\", row=1, col=1)\n",
        "fig.update_xaxes(title_text=\"Num. Topics\", row=1, col=2)\n",
        "\n",
        "fig.update_yaxes(title_text=\"Coherence Score\", row=1, col=1)\n",
        "\n",
        "\n",
        "fig.update_layout(height=600, width=800, title_text=\"Choosing Number of Topics Via UMass and CV scores\")\n",
        "fig.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7BgdI8cCqSFE"
      },
      "source": [
        "The best maxima occurs at $K=10$ for UMass and CV Score."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bYlZT1qyqaXH"
      },
      "source": [
        "#### Step 4: Running the model\n",
        "\n",
        "Having chosen the number of topics, we run the full model and analyze the results."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 304
        },
        "id": "UlFc_bwLtBHm",
        "outputId": "252704d1-4d48-4be3-eaa2-368351dd03de"
      },
      "outputs": [],
      "source": [
        "if FIRST_RUN: \n",
        "    num_topics = 10\n",
        "    \n",
        "    # find chunksize to make about 200 updates\n",
        "    num_passes = 10\n",
        "    chunk_size = len(model_corpus) * num_passes/200\n",
        "    print(f\"chunk size: {chunk_size}\")\n",
        "    \n",
        "    start = time.time()\n",
        "    print(\"fitting model\", flush=True)\n",
        "    model = LdaMulticore(num_topics=num_topics, # number of topics\n",
        "                         corpus=model_corpus, # what to train on \n",
        "                         id2word=dictionary, # mapping from IDs to words\n",
        "                         workers=min(10, multiprocessing.cpu_count()-1), # choose 10 cores, or whatever computer has\n",
        "                         passes=num_passes, # make this many passes over data\n",
        "                         chunksize=chunk_size, # update after this many instances\n",
        "                         alpha=0.01,\n",
        "                         eta = 'auto',\n",
        "                         random_state = 2\n",
        "                        )\n",
        "        \n",
        "    print(\"done in {}\".format(time.time()-start), flush=True)\n",
        "    with open(\"./Models/model_lda\", \"wb\") as fp:\n",
        "        pickle.dump(model, fp)\n",
        "else:\n",
        "    with open(\"./Models/model_lda\", \"rb\") as fp:\n",
        "        model = pickle.load(fp)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Step 5: Analyzing results\n",
        "\n",
        "We now extract the descriptors for all 10 topics and try to understand the topics which they describe."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "id": "hjqiF5D1ZATg"
      },
      "outputs": [],
      "source": [
        "def get_descriptors(model, num_words = 5):\n",
        "    topic_sep = re.compile(r\"0\\.[0-9]{3}\\*|\\+|\\\"\") # getting rid of useless formatting\n",
        "    model_topics = [(topic_no, re.sub(topic_sep, '', model_topic)) for topic_no, model_topic in\n",
        "                    model.print_topics(num_words=num_words)]\n",
        "    descriptors = {}\n",
        "    for i, m in model_topics:\n",
        "        descriptors[f\"topic {i+1}\"] = m.split()\n",
        "    return pd.DataFrame.from_dict(descriptors, orient = \"index\", columns = [f\"Word {i}\" for i in range(1,num_words+1)])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "id": "x40X1qHiayEF"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Word 1</th>\n",
              "      <th>Word 2</th>\n",
              "      <th>Word 3</th>\n",
              "      <th>Word 4</th>\n",
              "      <th>Word 5</th>\n",
              "      <th>Word 6</th>\n",
              "      <th>Word 7</th>\n",
              "      <th>Word 8</th>\n",
              "      <th>Word 9</th>\n",
              "      <th>Word 10</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>topic 1</th>\n",
              "      <td>recipe</td>\n",
              "      <td>food</td>\n",
              "      <td>guide</td>\n",
              "      <td>cookbook</td>\n",
              "      <td>cook</td>\n",
              "      <td>garden</td>\n",
              "      <td>plant</td>\n",
              "      <td>dish</td>\n",
              "      <td>cooking</td>\n",
              "      <td>eat</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>topic 2</th>\n",
              "      <td>child</td>\n",
              "      <td>life</td>\n",
              "      <td>god</td>\n",
              "      <td>love</td>\n",
              "      <td>parent</td>\n",
              "      <td>help</td>\n",
              "      <td>dog</td>\n",
              "      <td>little</td>\n",
              "      <td>animal</td>\n",
              "      <td>baby</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>topic 3</th>\n",
              "      <td>social</td>\n",
              "      <td>theory</td>\n",
              "      <td>political</td>\n",
              "      <td>study</td>\n",
              "      <td>history</td>\n",
              "      <td>science</td>\n",
              "      <td>economic</td>\n",
              "      <td>culture</td>\n",
              "      <td>human</td>\n",
              "      <td>american</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>topic 4</th>\n",
              "      <td>music</td>\n",
              "      <td>art</td>\n",
              "      <td>film</td>\n",
              "      <td>work</td>\n",
              "      <td>artist</td>\n",
              "      <td>original</td>\n",
              "      <td>include</td>\n",
              "      <td>guide</td>\n",
              "      <td>song</td>\n",
              "      <td>history</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>topic 5</th>\n",
              "      <td>bible</td>\n",
              "      <td>god</td>\n",
              "      <td>church</td>\n",
              "      <td>christian</td>\n",
              "      <td>jesus</td>\n",
              "      <td>testament</td>\n",
              "      <td>biblical</td>\n",
              "      <td>spiritual</td>\n",
              "      <td>study</td>\n",
              "      <td>christ</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>topic 6</th>\n",
              "      <td>student</td>\n",
              "      <td>guide</td>\n",
              "      <td>edition</td>\n",
              "      <td>provide</td>\n",
              "      <td>business</td>\n",
              "      <td>design</td>\n",
              "      <td>language</td>\n",
              "      <td>include</td>\n",
              "      <td>information</td>\n",
              "      <td>system</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>topic 7</th>\n",
              "      <td>love</td>\n",
              "      <td>novel</td>\n",
              "      <td>man</td>\n",
              "      <td>life</td>\n",
              "      <td>find</td>\n",
              "      <td>woman</td>\n",
              "      <td>murder</td>\n",
              "      <td>family</td>\n",
              "      <td>young</td>\n",
              "      <td>new</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>topic 8</th>\n",
              "      <td>game</td>\n",
              "      <td>baseball</td>\n",
              "      <td>sport</td>\n",
              "      <td>guide</td>\n",
              "      <td>player</td>\n",
              "      <td>bird</td>\n",
              "      <td>quilt</td>\n",
              "      <td>include</td>\n",
              "      <td>color</td>\n",
              "      <td>history</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>topic 9</th>\n",
              "      <td>poem</td>\n",
              "      <td>poetry</td>\n",
              "      <td>poet</td>\n",
              "      <td>literature</td>\n",
              "      <td>work</td>\n",
              "      <td>american</td>\n",
              "      <td>history</td>\n",
              "      <td>publish</td>\n",
              "      <td>english</td>\n",
              "      <td>essay</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>topic 10</th>\n",
              "      <td>war</td>\n",
              "      <td>history</td>\n",
              "      <td>american</td>\n",
              "      <td>world</td>\n",
              "      <td>life</td>\n",
              "      <td>man</td>\n",
              "      <td>america</td>\n",
              "      <td>battle</td>\n",
              "      <td>year</td>\n",
              "      <td>new</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "           Word 1    Word 2     Word 3      Word 4    Word 5     Word 6  \\\n",
              "topic 1    recipe      food      guide    cookbook      cook     garden   \n",
              "topic 2     child      life        god        love    parent       help   \n",
              "topic 3    social    theory  political       study   history    science   \n",
              "topic 4     music       art       film        work    artist   original   \n",
              "topic 5     bible       god     church   christian     jesus  testament   \n",
              "topic 6   student     guide    edition     provide  business     design   \n",
              "topic 7      love     novel        man        life      find      woman   \n",
              "topic 8      game  baseball      sport       guide    player       bird   \n",
              "topic 9      poem    poetry       poet  literature      work   american   \n",
              "topic 10      war   history   american       world      life        man   \n",
              "\n",
              "            Word 7     Word 8       Word 9   Word 10  \n",
              "topic 1      plant       dish      cooking       eat  \n",
              "topic 2        dog     little       animal      baby  \n",
              "topic 3   economic    culture        human  american  \n",
              "topic 4    include      guide         song   history  \n",
              "topic 5   biblical  spiritual        study    christ  \n",
              "topic 6   language    include  information    system  \n",
              "topic 7     murder     family        young       new  \n",
              "topic 8      quilt    include        color   history  \n",
              "topic 9    history    publish      english     essay  \n",
              "topic 10   america     battle         year       new  "
            ]
          },
          "execution_count": 22,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "topics = get_descriptors(model, 10)\n",
        "topics"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "26H30tI2_HfC"
      },
      "source": [
        "As we can see, the topics are quite easy to interpret:\n",
        "- topic 1: **cooking**\n",
        "- topic 2: **family**.\n",
        "- topic 3: **socio economic science**.\n",
        "- topic 4: **fine arts**.\n",
        "- topic 5: **religion**.\n",
        "- topic 6: **student help**.\n",
        "- topic 7: **fictional romance**.\n",
        "- topic 8: **sport & outdoors**.\n",
        "- topic 9: **poetry & literature**.\n",
        "- topic 10: **history**.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Topic frequency over the years\n",
        "Finally, lets analyze how the topics change throughout the years.\n",
        "\n",
        "In particular, we will analyze the last 12 years (for graphical and practical reasons)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "id": "NFyUPg_2Blf8"
      },
      "outputs": [],
      "source": [
        "topic_corpus = model[model_corpus]\n",
        "topic = [sorted(topic_corpus[entry], key = lambda x: x[1])[0][0] for entry in range(len(book_description_corpus_cleaned_collocations))]\n",
        "df_books_en[\"topic\"]=topic"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 77,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Title</th>\n",
              "      <th>description</th>\n",
              "      <th>authors</th>\n",
              "      <th>publishedDate</th>\n",
              "      <th>categories</th>\n",
              "      <th>new_desc</th>\n",
              "      <th>lang</th>\n",
              "      <th>topic</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Dr. Seuss: American Icon</td>\n",
              "      <td>Philip Nel takes a fascinating look into the k...</td>\n",
              "      <td>['Philip Nel']</td>\n",
              "      <td>2005</td>\n",
              "      <td>['Biography &amp; Autobiography']</td>\n",
              "      <td>Philip Nel takes a fascinating look into the k...</td>\n",
              "      <td>en</td>\n",
              "      <td>8</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Wonderful Worship in Smaller Churches</td>\n",
              "      <td>This resource includes twelve principles in un...</td>\n",
              "      <td>['David R. Ray']</td>\n",
              "      <td>2000</td>\n",
              "      <td>['Religion']</td>\n",
              "      <td>This resource includes twelve principles in un...</td>\n",
              "      <td>en</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Whispers of the Wicked Saints</td>\n",
              "      <td>Julia Thomas finds her life spinning out of co...</td>\n",
              "      <td>['Veronica Haddon']</td>\n",
              "      <td>2005</td>\n",
              "      <td>['Fiction']</td>\n",
              "      <td>Julia Thomas finds her life spinning out of co...</td>\n",
              "      <td>en</td>\n",
              "      <td>6</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>The Church of Christ: A Biblical Ecclesiology ...</td>\n",
              "      <td>In The Church of Christ: A Biblical Ecclesiolo...</td>\n",
              "      <td>['Everett Ferguson']</td>\n",
              "      <td>1996</td>\n",
              "      <td>['Religion']</td>\n",
              "      <td>In The Church of Christ: A Biblical Ecclesiolo...</td>\n",
              "      <td>en</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>Saint Hyacinth of Poland</td>\n",
              "      <td>The story for children 10 and up of St. Hyacin...</td>\n",
              "      <td>['Mary Fabyan Windeatt']</td>\n",
              "      <td>2009</td>\n",
              "      <td>['Biography &amp; Autobiography']</td>\n",
              "      <td>The story for children 10 and up of St. Hyacin...</td>\n",
              "      <td>en</td>\n",
              "      <td>9</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>212399</th>\n",
              "      <td>The Orphan Of Ellis Island (Time Travel Advent...</td>\n",
              "      <td>During a school trip to Ellis Island, Dominick...</td>\n",
              "      <td>['Elvira Woodruff']</td>\n",
              "      <td>2000</td>\n",
              "      <td>['Juvenile Fiction']</td>\n",
              "      <td>During a school trip to Ellis Island, Dominick...</td>\n",
              "      <td>en</td>\n",
              "      <td>6</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>212400</th>\n",
              "      <td>Red Boots for Christmas</td>\n",
              "      <td>Everyone in the village of Friedensdorf is hap...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1995</td>\n",
              "      <td>['Juvenile Fiction']</td>\n",
              "      <td>Everyone in the village of Friedensdorf is hap...</td>\n",
              "      <td>en</td>\n",
              "      <td>6</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>212401</th>\n",
              "      <td>Mamaw</td>\n",
              "      <td>Give your Mamaw a useful, beautiful and though...</td>\n",
              "      <td>['Wild Wild Cabbage']</td>\n",
              "      <td>2018</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Give your Mamaw a useful, beautiful and though...</td>\n",
              "      <td>en</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>212402</th>\n",
              "      <td>The Autograph Man</td>\n",
              "      <td>Alex-Li Tandem sells autographs. His business ...</td>\n",
              "      <td>['Zadie Smith']</td>\n",
              "      <td>2003</td>\n",
              "      <td>['Fiction']</td>\n",
              "      <td>Alex-Li Tandem sells autographs. His business ...</td>\n",
              "      <td>en</td>\n",
              "      <td>6</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>212403</th>\n",
              "      <td>Student's Solutions Manual for Johnson/Mowry's...</td>\n",
              "      <td>Discover the many ways mathematics is relevant...</td>\n",
              "      <td>['David B. Johnson', 'Thomas A. Mowry']</td>\n",
              "      <td>1998</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Discover the many ways mathematics is relevant...</td>\n",
              "      <td>en</td>\n",
              "      <td>5</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>141745 rows × 8 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                    Title  \\\n",
              "1                                Dr. Seuss: American Icon   \n",
              "2                   Wonderful Worship in Smaller Churches   \n",
              "3                           Whispers of the Wicked Saints   \n",
              "5       The Church of Christ: A Biblical Ecclesiology ...   \n",
              "8                                Saint Hyacinth of Poland   \n",
              "...                                                   ...   \n",
              "212399  The Orphan Of Ellis Island (Time Travel Advent...   \n",
              "212400                            Red Boots for Christmas   \n",
              "212401                                              Mamaw   \n",
              "212402                                  The Autograph Man   \n",
              "212403  Student's Solutions Manual for Johnson/Mowry's...   \n",
              "\n",
              "                                              description  \\\n",
              "1       Philip Nel takes a fascinating look into the k...   \n",
              "2       This resource includes twelve principles in un...   \n",
              "3       Julia Thomas finds her life spinning out of co...   \n",
              "5       In The Church of Christ: A Biblical Ecclesiolo...   \n",
              "8       The story for children 10 and up of St. Hyacin...   \n",
              "...                                                   ...   \n",
              "212399  During a school trip to Ellis Island, Dominick...   \n",
              "212400  Everyone in the village of Friedensdorf is hap...   \n",
              "212401  Give your Mamaw a useful, beautiful and though...   \n",
              "212402  Alex-Li Tandem sells autographs. His business ...   \n",
              "212403  Discover the many ways mathematics is relevant...   \n",
              "\n",
              "                                        authors publishedDate  \\\n",
              "1                                ['Philip Nel']          2005   \n",
              "2                              ['David R. Ray']          2000   \n",
              "3                           ['Veronica Haddon']          2005   \n",
              "5                          ['Everett Ferguson']          1996   \n",
              "8                      ['Mary Fabyan Windeatt']          2009   \n",
              "...                                         ...           ...   \n",
              "212399                      ['Elvira Woodruff']          2000   \n",
              "212400                                      NaN          1995   \n",
              "212401                    ['Wild Wild Cabbage']          2018   \n",
              "212402                          ['Zadie Smith']          2003   \n",
              "212403  ['David B. Johnson', 'Thomas A. Mowry']          1998   \n",
              "\n",
              "                           categories  \\\n",
              "1       ['Biography & Autobiography']   \n",
              "2                        ['Religion']   \n",
              "3                         ['Fiction']   \n",
              "5                        ['Religion']   \n",
              "8       ['Biography & Autobiography']   \n",
              "...                               ...   \n",
              "212399           ['Juvenile Fiction']   \n",
              "212400           ['Juvenile Fiction']   \n",
              "212401                            NaN   \n",
              "212402                    ['Fiction']   \n",
              "212403                            NaN   \n",
              "\n",
              "                                                 new_desc lang  topic  \n",
              "1       Philip Nel takes a fascinating look into the k...   en      8  \n",
              "2       This resource includes twelve principles in un...   en      4  \n",
              "3       Julia Thomas finds her life spinning out of co...   en      6  \n",
              "5       In The Church of Christ: A Biblical Ecclesiolo...   en      4  \n",
              "8       The story for children 10 and up of St. Hyacin...   en      9  \n",
              "...                                                   ...  ...    ...  \n",
              "212399  During a school trip to Ellis Island, Dominick...   en      6  \n",
              "212400  Everyone in the village of Friedensdorf is hap...   en      6  \n",
              "212401  Give your Mamaw a useful, beautiful and though...   en      1  \n",
              "212402  Alex-Li Tandem sells autographs. His business ...   en      6  \n",
              "212403  Discover the many ways mathematics is relevant...   en      5  \n",
              "\n",
              "[141745 rows x 8 columns]"
            ]
          },
          "execution_count": 77,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "df_books_en"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 78,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "2022    1006\n",
              "2021    2645\n",
              "2020    2724\n",
              "2019    2697\n",
              "2018    2766\n",
              "2017    2992\n",
              "2016    3187\n",
              "2015    3540\n",
              "2014    4288\n",
              "2013    5098\n",
              "2012    5218\n",
              "2011    4532\n",
              "2010    3906\n",
              "Name: publishedDate, dtype: int64"
            ]
          },
          "execution_count": 78,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "df_books_en.loc[98662,\"publishedDate\"] = \"2007\" #Wrong Data point, used to be 2030 which is impossible\n",
        "years = df_books_en[\"publishedDate\"].value_counts().sort_index(ascending=False)[:14]\n",
        "years.pop(\"2023\") # strangely enough, we had some points from 2023 (perhaps books that are announced but yet to be published?). These were eliminated\n",
        "years"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 79,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "44599\n"
          ]
        }
      ],
      "source": [
        "print(sum(years.values))\n",
        "years = years.index.tolist()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 95,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Title</th>\n",
              "      <th>description</th>\n",
              "      <th>authors</th>\n",
              "      <th>publishedDate</th>\n",
              "      <th>categories</th>\n",
              "      <th>new_desc</th>\n",
              "      <th>lang</th>\n",
              "      <th>topic</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>Mensa Number Puzzles (Mensa Word Games for Kids)</td>\n",
              "      <td>Acclaimed teacher and puzzler Evelyn B. Christ...</td>\n",
              "      <td>['Evelyn B. Christensen']</td>\n",
              "      <td>2018</td>\n",
              "      <td>['Juvenile Nonfiction']</td>\n",
              "      <td>Acclaimed teacher and puzzler Evelyn B. Christ...</td>\n",
              "      <td>en</td>\n",
              "      <td>5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>Vector Quantization and Signal Compression (Th...</td>\n",
              "      <td>Herb Caen, a popular columnist for the San Fra...</td>\n",
              "      <td>['Allen Gersho', 'Robert M. Gray']</td>\n",
              "      <td>2012</td>\n",
              "      <td>['Technology &amp; Engineering']</td>\n",
              "      <td>Herb Caen, a popular columnist for the San Fra...</td>\n",
              "      <td>en</td>\n",
              "      <td>5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>A husband for Kutani</td>\n",
              "      <td>First published in 1938, this is a collection ...</td>\n",
              "      <td>['Frank Owen']</td>\n",
              "      <td>2018</td>\n",
              "      <td>['History']</td>\n",
              "      <td>First published in 1938, this is a collection ...</td>\n",
              "      <td>en</td>\n",
              "      <td>8</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16</th>\n",
              "      <td>The Ultimate Guide to Law School Admission: In...</td>\n",
              "      <td>This collection brings together a distinguishe...</td>\n",
              "      <td>['Fiona Cownie']</td>\n",
              "      <td>2010</td>\n",
              "      <td>['Law']</td>\n",
              "      <td>This collection brings together a distinguishe...</td>\n",
              "      <td>en</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17</th>\n",
              "      <td>The Repeal of Reticence: A History of America'...</td>\n",
              "      <td>At a time when America's faculties of taste an...</td>\n",
              "      <td>['Rochelle Gurstein']</td>\n",
              "      <td>2016</td>\n",
              "      <td>['Political Science']</td>\n",
              "      <td>At a time when America's faculties of taste an...</td>\n",
              "      <td>en</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                Title  \\\n",
              "12   Mensa Number Puzzles (Mensa Word Games for Kids)   \n",
              "13  Vector Quantization and Signal Compression (Th...   \n",
              "14                               A husband for Kutani   \n",
              "16  The Ultimate Guide to Law School Admission: In...   \n",
              "17  The Repeal of Reticence: A History of America'...   \n",
              "\n",
              "                                          description  \\\n",
              "12  Acclaimed teacher and puzzler Evelyn B. Christ...   \n",
              "13  Herb Caen, a popular columnist for the San Fra...   \n",
              "14  First published in 1938, this is a collection ...   \n",
              "16  This collection brings together a distinguishe...   \n",
              "17  At a time when America's faculties of taste an...   \n",
              "\n",
              "                               authors publishedDate  \\\n",
              "12           ['Evelyn B. Christensen']          2018   \n",
              "13  ['Allen Gersho', 'Robert M. Gray']          2012   \n",
              "14                      ['Frank Owen']          2018   \n",
              "16                    ['Fiona Cownie']          2010   \n",
              "17               ['Rochelle Gurstein']          2016   \n",
              "\n",
              "                      categories  \\\n",
              "12       ['Juvenile Nonfiction']   \n",
              "13  ['Technology & Engineering']   \n",
              "14                   ['History']   \n",
              "16                       ['Law']   \n",
              "17         ['Political Science']   \n",
              "\n",
              "                                             new_desc lang  topic  \n",
              "12  Acclaimed teacher and puzzler Evelyn B. Christ...   en      5  \n",
              "13  Herb Caen, a popular columnist for the San Fra...   en      5  \n",
              "14  First published in 1938, this is a collection ...   en      8  \n",
              "16  This collection brings together a distinguishe...   en      2  \n",
              "17  At a time when America's faculties of taste an...   en      2  "
            ]
          },
          "execution_count": 95,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "df_books_en[\"publishedDate\"].replace(\"\",np.nan, inplace=True)\n",
        "df_books_en = df_books_en.dropna(subset = [\"publishedDate\"])\n",
        "df_years = df_books_en[(df_books_en[\"publishedDate\"].apply(int) > 2009) & (df_books_en[\"publishedDate\"].apply(int) < 2023)]\n",
        "df_years.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 96,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th>topic</th>\n",
              "      <th>0</th>\n",
              "      <th>1</th>\n",
              "      <th>2</th>\n",
              "      <th>3</th>\n",
              "      <th>4</th>\n",
              "      <th>5</th>\n",
              "      <th>6</th>\n",
              "      <th>7</th>\n",
              "      <th>8</th>\n",
              "      <th>9</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>publishedDate</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2010</th>\n",
              "      <td>196</td>\n",
              "      <td>428</td>\n",
              "      <td>318</td>\n",
              "      <td>199</td>\n",
              "      <td>286</td>\n",
              "      <td>602</td>\n",
              "      <td>1076</td>\n",
              "      <td>135</td>\n",
              "      <td>178</td>\n",
              "      <td>488</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2011</th>\n",
              "      <td>252</td>\n",
              "      <td>489</td>\n",
              "      <td>356</td>\n",
              "      <td>216</td>\n",
              "      <td>304</td>\n",
              "      <td>662</td>\n",
              "      <td>1323</td>\n",
              "      <td>126</td>\n",
              "      <td>204</td>\n",
              "      <td>600</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2012</th>\n",
              "      <td>259</td>\n",
              "      <td>481</td>\n",
              "      <td>520</td>\n",
              "      <td>254</td>\n",
              "      <td>345</td>\n",
              "      <td>994</td>\n",
              "      <td>1238</td>\n",
              "      <td>173</td>\n",
              "      <td>247</td>\n",
              "      <td>707</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2013</th>\n",
              "      <td>228</td>\n",
              "      <td>416</td>\n",
              "      <td>609</td>\n",
              "      <td>313</td>\n",
              "      <td>338</td>\n",
              "      <td>903</td>\n",
              "      <td>1122</td>\n",
              "      <td>212</td>\n",
              "      <td>312</td>\n",
              "      <td>645</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2014</th>\n",
              "      <td>188</td>\n",
              "      <td>377</td>\n",
              "      <td>382</td>\n",
              "      <td>226</td>\n",
              "      <td>277</td>\n",
              "      <td>629</td>\n",
              "      <td>1250</td>\n",
              "      <td>106</td>\n",
              "      <td>253</td>\n",
              "      <td>600</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2015</th>\n",
              "      <td>181</td>\n",
              "      <td>321</td>\n",
              "      <td>346</td>\n",
              "      <td>238</td>\n",
              "      <td>240</td>\n",
              "      <td>551</td>\n",
              "      <td>893</td>\n",
              "      <td>104</td>\n",
              "      <td>178</td>\n",
              "      <td>488</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2016</th>\n",
              "      <td>132</td>\n",
              "      <td>313</td>\n",
              "      <td>308</td>\n",
              "      <td>254</td>\n",
              "      <td>224</td>\n",
              "      <td>518</td>\n",
              "      <td>773</td>\n",
              "      <td>97</td>\n",
              "      <td>161</td>\n",
              "      <td>407</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2017</th>\n",
              "      <td>163</td>\n",
              "      <td>308</td>\n",
              "      <td>349</td>\n",
              "      <td>210</td>\n",
              "      <td>207</td>\n",
              "      <td>468</td>\n",
              "      <td>661</td>\n",
              "      <td>88</td>\n",
              "      <td>128</td>\n",
              "      <td>410</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2018</th>\n",
              "      <td>144</td>\n",
              "      <td>269</td>\n",
              "      <td>333</td>\n",
              "      <td>223</td>\n",
              "      <td>187</td>\n",
              "      <td>441</td>\n",
              "      <td>677</td>\n",
              "      <td>75</td>\n",
              "      <td>96</td>\n",
              "      <td>321</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2019</th>\n",
              "      <td>152</td>\n",
              "      <td>338</td>\n",
              "      <td>265</td>\n",
              "      <td>162</td>\n",
              "      <td>162</td>\n",
              "      <td>382</td>\n",
              "      <td>789</td>\n",
              "      <td>62</td>\n",
              "      <td>101</td>\n",
              "      <td>284</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2020</th>\n",
              "      <td>128</td>\n",
              "      <td>366</td>\n",
              "      <td>294</td>\n",
              "      <td>159</td>\n",
              "      <td>190</td>\n",
              "      <td>371</td>\n",
              "      <td>737</td>\n",
              "      <td>67</td>\n",
              "      <td>126</td>\n",
              "      <td>286</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2021</th>\n",
              "      <td>115</td>\n",
              "      <td>327</td>\n",
              "      <td>275</td>\n",
              "      <td>173</td>\n",
              "      <td>161</td>\n",
              "      <td>238</td>\n",
              "      <td>813</td>\n",
              "      <td>54</td>\n",
              "      <td>140</td>\n",
              "      <td>349</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2022</th>\n",
              "      <td>36</td>\n",
              "      <td>109</td>\n",
              "      <td>92</td>\n",
              "      <td>94</td>\n",
              "      <td>58</td>\n",
              "      <td>79</td>\n",
              "      <td>311</td>\n",
              "      <td>25</td>\n",
              "      <td>54</td>\n",
              "      <td>148</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "topic            0    1    2    3    4    5     6    7    8    9\n",
              "publishedDate                                                   \n",
              "2010           196  428  318  199  286  602  1076  135  178  488\n",
              "2011           252  489  356  216  304  662  1323  126  204  600\n",
              "2012           259  481  520  254  345  994  1238  173  247  707\n",
              "2013           228  416  609  313  338  903  1122  212  312  645\n",
              "2014           188  377  382  226  277  629  1250  106  253  600\n",
              "2015           181  321  346  238  240  551   893  104  178  488\n",
              "2016           132  313  308  254  224  518   773   97  161  407\n",
              "2017           163  308  349  210  207  468   661   88  128  410\n",
              "2018           144  269  333  223  187  441   677   75   96  321\n",
              "2019           152  338  265  162  162  382   789   62  101  284\n",
              "2020           128  366  294  159  190  371   737   67  126  286\n",
              "2021           115  327  275  173  161  238   813   54  140  349\n",
              "2022            36  109   92   94   58   79   311   25   54  148"
            ]
          },
          "execution_count": 96,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "df_years = df_years.groupby([pd.Grouper(key = \"publishedDate\"), \"topic\"]).size().reset_index(name = \"count\").pivot(index = \"publishedDate\", columns = \"topic\", values = \"count\")\n",
        "df_years"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 102,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "application/vnd.plotly.v1+json": {
              "config": {
                "plotlyServerURL": "https://plot.ly"
              },
              "data": [
                {
                  "name": "cooking",
                  "type": "scatter",
                  "x": [
                    "2010",
                    "2011",
                    "2012",
                    "2013",
                    "2014",
                    "2015",
                    "2016",
                    "2017",
                    "2018",
                    "2019",
                    "2020",
                    "2021",
                    "2022"
                  ],
                  "y": [
                    196,
                    252,
                    259,
                    228,
                    188,
                    181,
                    132,
                    163,
                    144,
                    152,
                    128,
                    115,
                    36
                  ]
                },
                {
                  "name": "family",
                  "type": "scatter",
                  "x": [
                    "2010",
                    "2011",
                    "2012",
                    "2013",
                    "2014",
                    "2015",
                    "2016",
                    "2017",
                    "2018",
                    "2019",
                    "2020",
                    "2021",
                    "2022"
                  ],
                  "y": [
                    428,
                    489,
                    481,
                    416,
                    377,
                    321,
                    313,
                    308,
                    269,
                    338,
                    366,
                    327,
                    109
                  ]
                },
                {
                  "name": "socio economics",
                  "type": "scatter",
                  "x": [
                    "2010",
                    "2011",
                    "2012",
                    "2013",
                    "2014",
                    "2015",
                    "2016",
                    "2017",
                    "2018",
                    "2019",
                    "2020",
                    "2021",
                    "2022"
                  ],
                  "y": [
                    318,
                    356,
                    520,
                    609,
                    382,
                    346,
                    308,
                    349,
                    333,
                    265,
                    294,
                    275,
                    92
                  ]
                },
                {
                  "name": "fine arts",
                  "type": "scatter",
                  "x": [
                    "2010",
                    "2011",
                    "2012",
                    "2013",
                    "2014",
                    "2015",
                    "2016",
                    "2017",
                    "2018",
                    "2019",
                    "2020",
                    "2021",
                    "2022"
                  ],
                  "y": [
                    199,
                    216,
                    254,
                    313,
                    226,
                    238,
                    254,
                    210,
                    223,
                    162,
                    159,
                    173,
                    94
                  ]
                },
                {
                  "name": "religion",
                  "type": "scatter",
                  "x": [
                    "2010",
                    "2011",
                    "2012",
                    "2013",
                    "2014",
                    "2015",
                    "2016",
                    "2017",
                    "2018",
                    "2019",
                    "2020",
                    "2021",
                    "2022"
                  ],
                  "y": [
                    286,
                    304,
                    345,
                    338,
                    277,
                    240,
                    224,
                    207,
                    187,
                    162,
                    190,
                    161,
                    58
                  ]
                },
                {
                  "name": "student help",
                  "type": "scatter",
                  "x": [
                    "2010",
                    "2011",
                    "2012",
                    "2013",
                    "2014",
                    "2015",
                    "2016",
                    "2017",
                    "2018",
                    "2019",
                    "2020",
                    "2021",
                    "2022"
                  ],
                  "y": [
                    602,
                    662,
                    994,
                    903,
                    629,
                    551,
                    518,
                    468,
                    441,
                    382,
                    371,
                    238,
                    79
                  ]
                },
                {
                  "name": "fictional romance",
                  "type": "scatter",
                  "x": [
                    "2010",
                    "2011",
                    "2012",
                    "2013",
                    "2014",
                    "2015",
                    "2016",
                    "2017",
                    "2018",
                    "2019",
                    "2020",
                    "2021",
                    "2022"
                  ],
                  "y": [
                    1076,
                    1323,
                    1238,
                    1122,
                    1250,
                    893,
                    773,
                    661,
                    677,
                    789,
                    737,
                    813,
                    311
                  ]
                },
                {
                  "name": "sport & outdoors",
                  "type": "scatter",
                  "x": [
                    "2010",
                    "2011",
                    "2012",
                    "2013",
                    "2014",
                    "2015",
                    "2016",
                    "2017",
                    "2018",
                    "2019",
                    "2020",
                    "2021",
                    "2022"
                  ],
                  "y": [
                    135,
                    126,
                    173,
                    212,
                    106,
                    104,
                    97,
                    88,
                    75,
                    62,
                    67,
                    54,
                    25
                  ]
                },
                {
                  "name": "poetry & literature",
                  "type": "scatter",
                  "x": [
                    "2010",
                    "2011",
                    "2012",
                    "2013",
                    "2014",
                    "2015",
                    "2016",
                    "2017",
                    "2018",
                    "2019",
                    "2020",
                    "2021",
                    "2022"
                  ],
                  "y": [
                    178,
                    204,
                    247,
                    312,
                    253,
                    178,
                    161,
                    128,
                    96,
                    101,
                    126,
                    140,
                    54
                  ]
                },
                {
                  "name": "history",
                  "type": "scatter",
                  "x": [
                    "2010",
                    "2011",
                    "2012",
                    "2013",
                    "2014",
                    "2015",
                    "2016",
                    "2017",
                    "2018",
                    "2019",
                    "2020",
                    "2021",
                    "2022"
                  ],
                  "y": [
                    488,
                    600,
                    707,
                    645,
                    600,
                    488,
                    407,
                    410,
                    321,
                    284,
                    286,
                    349,
                    148
                  ]
                }
              ],
              "layout": {
                "barmode": "group",
                "template": {
                  "data": {
                    "bar": [
                      {
                        "error_x": {
                          "color": "#2a3f5f"
                        },
                        "error_y": {
                          "color": "#2a3f5f"
                        },
                        "marker": {
                          "line": {
                            "color": "#E5ECF6",
                            "width": 0.5
                          },
                          "pattern": {
                            "fillmode": "overlay",
                            "size": 10,
                            "solidity": 0.2
                          }
                        },
                        "type": "bar"
                      }
                    ],
                    "barpolar": [
                      {
                        "marker": {
                          "line": {
                            "color": "#E5ECF6",
                            "width": 0.5
                          },
                          "pattern": {
                            "fillmode": "overlay",
                            "size": 10,
                            "solidity": 0.2
                          }
                        },
                        "type": "barpolar"
                      }
                    ],
                    "carpet": [
                      {
                        "aaxis": {
                          "endlinecolor": "#2a3f5f",
                          "gridcolor": "white",
                          "linecolor": "white",
                          "minorgridcolor": "white",
                          "startlinecolor": "#2a3f5f"
                        },
                        "baxis": {
                          "endlinecolor": "#2a3f5f",
                          "gridcolor": "white",
                          "linecolor": "white",
                          "minorgridcolor": "white",
                          "startlinecolor": "#2a3f5f"
                        },
                        "type": "carpet"
                      }
                    ],
                    "choropleth": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "type": "choropleth"
                      }
                    ],
                    "contour": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "colorscale": [
                          [
                            0,
                            "#0d0887"
                          ],
                          [
                            0.1111111111111111,
                            "#46039f"
                          ],
                          [
                            0.2222222222222222,
                            "#7201a8"
                          ],
                          [
                            0.3333333333333333,
                            "#9c179e"
                          ],
                          [
                            0.4444444444444444,
                            "#bd3786"
                          ],
                          [
                            0.5555555555555556,
                            "#d8576b"
                          ],
                          [
                            0.6666666666666666,
                            "#ed7953"
                          ],
                          [
                            0.7777777777777778,
                            "#fb9f3a"
                          ],
                          [
                            0.8888888888888888,
                            "#fdca26"
                          ],
                          [
                            1,
                            "#f0f921"
                          ]
                        ],
                        "type": "contour"
                      }
                    ],
                    "contourcarpet": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "type": "contourcarpet"
                      }
                    ],
                    "heatmap": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "colorscale": [
                          [
                            0,
                            "#0d0887"
                          ],
                          [
                            0.1111111111111111,
                            "#46039f"
                          ],
                          [
                            0.2222222222222222,
                            "#7201a8"
                          ],
                          [
                            0.3333333333333333,
                            "#9c179e"
                          ],
                          [
                            0.4444444444444444,
                            "#bd3786"
                          ],
                          [
                            0.5555555555555556,
                            "#d8576b"
                          ],
                          [
                            0.6666666666666666,
                            "#ed7953"
                          ],
                          [
                            0.7777777777777778,
                            "#fb9f3a"
                          ],
                          [
                            0.8888888888888888,
                            "#fdca26"
                          ],
                          [
                            1,
                            "#f0f921"
                          ]
                        ],
                        "type": "heatmap"
                      }
                    ],
                    "heatmapgl": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "colorscale": [
                          [
                            0,
                            "#0d0887"
                          ],
                          [
                            0.1111111111111111,
                            "#46039f"
                          ],
                          [
                            0.2222222222222222,
                            "#7201a8"
                          ],
                          [
                            0.3333333333333333,
                            "#9c179e"
                          ],
                          [
                            0.4444444444444444,
                            "#bd3786"
                          ],
                          [
                            0.5555555555555556,
                            "#d8576b"
                          ],
                          [
                            0.6666666666666666,
                            "#ed7953"
                          ],
                          [
                            0.7777777777777778,
                            "#fb9f3a"
                          ],
                          [
                            0.8888888888888888,
                            "#fdca26"
                          ],
                          [
                            1,
                            "#f0f921"
                          ]
                        ],
                        "type": "heatmapgl"
                      }
                    ],
                    "histogram": [
                      {
                        "marker": {
                          "pattern": {
                            "fillmode": "overlay",
                            "size": 10,
                            "solidity": 0.2
                          }
                        },
                        "type": "histogram"
                      }
                    ],
                    "histogram2d": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "colorscale": [
                          [
                            0,
                            "#0d0887"
                          ],
                          [
                            0.1111111111111111,
                            "#46039f"
                          ],
                          [
                            0.2222222222222222,
                            "#7201a8"
                          ],
                          [
                            0.3333333333333333,
                            "#9c179e"
                          ],
                          [
                            0.4444444444444444,
                            "#bd3786"
                          ],
                          [
                            0.5555555555555556,
                            "#d8576b"
                          ],
                          [
                            0.6666666666666666,
                            "#ed7953"
                          ],
                          [
                            0.7777777777777778,
                            "#fb9f3a"
                          ],
                          [
                            0.8888888888888888,
                            "#fdca26"
                          ],
                          [
                            1,
                            "#f0f921"
                          ]
                        ],
                        "type": "histogram2d"
                      }
                    ],
                    "histogram2dcontour": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "colorscale": [
                          [
                            0,
                            "#0d0887"
                          ],
                          [
                            0.1111111111111111,
                            "#46039f"
                          ],
                          [
                            0.2222222222222222,
                            "#7201a8"
                          ],
                          [
                            0.3333333333333333,
                            "#9c179e"
                          ],
                          [
                            0.4444444444444444,
                            "#bd3786"
                          ],
                          [
                            0.5555555555555556,
                            "#d8576b"
                          ],
                          [
                            0.6666666666666666,
                            "#ed7953"
                          ],
                          [
                            0.7777777777777778,
                            "#fb9f3a"
                          ],
                          [
                            0.8888888888888888,
                            "#fdca26"
                          ],
                          [
                            1,
                            "#f0f921"
                          ]
                        ],
                        "type": "histogram2dcontour"
                      }
                    ],
                    "mesh3d": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "type": "mesh3d"
                      }
                    ],
                    "parcoords": [
                      {
                        "line": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "parcoords"
                      }
                    ],
                    "pie": [
                      {
                        "automargin": true,
                        "type": "pie"
                      }
                    ],
                    "scatter": [
                      {
                        "fillpattern": {
                          "fillmode": "overlay",
                          "size": 10,
                          "solidity": 0.2
                        },
                        "type": "scatter"
                      }
                    ],
                    "scatter3d": [
                      {
                        "line": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scatter3d"
                      }
                    ],
                    "scattercarpet": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scattercarpet"
                      }
                    ],
                    "scattergeo": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scattergeo"
                      }
                    ],
                    "scattergl": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scattergl"
                      }
                    ],
                    "scattermapbox": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scattermapbox"
                      }
                    ],
                    "scatterpolar": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scatterpolar"
                      }
                    ],
                    "scatterpolargl": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scatterpolargl"
                      }
                    ],
                    "scatterternary": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scatterternary"
                      }
                    ],
                    "surface": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "colorscale": [
                          [
                            0,
                            "#0d0887"
                          ],
                          [
                            0.1111111111111111,
                            "#46039f"
                          ],
                          [
                            0.2222222222222222,
                            "#7201a8"
                          ],
                          [
                            0.3333333333333333,
                            "#9c179e"
                          ],
                          [
                            0.4444444444444444,
                            "#bd3786"
                          ],
                          [
                            0.5555555555555556,
                            "#d8576b"
                          ],
                          [
                            0.6666666666666666,
                            "#ed7953"
                          ],
                          [
                            0.7777777777777778,
                            "#fb9f3a"
                          ],
                          [
                            0.8888888888888888,
                            "#fdca26"
                          ],
                          [
                            1,
                            "#f0f921"
                          ]
                        ],
                        "type": "surface"
                      }
                    ],
                    "table": [
                      {
                        "cells": {
                          "fill": {
                            "color": "#EBF0F8"
                          },
                          "line": {
                            "color": "white"
                          }
                        },
                        "header": {
                          "fill": {
                            "color": "#C8D4E3"
                          },
                          "line": {
                            "color": "white"
                          }
                        },
                        "type": "table"
                      }
                    ]
                  },
                  "layout": {
                    "annotationdefaults": {
                      "arrowcolor": "#2a3f5f",
                      "arrowhead": 0,
                      "arrowwidth": 1
                    },
                    "autotypenumbers": "strict",
                    "coloraxis": {
                      "colorbar": {
                        "outlinewidth": 0,
                        "ticks": ""
                      }
                    },
                    "colorscale": {
                      "diverging": [
                        [
                          0,
                          "#8e0152"
                        ],
                        [
                          0.1,
                          "#c51b7d"
                        ],
                        [
                          0.2,
                          "#de77ae"
                        ],
                        [
                          0.3,
                          "#f1b6da"
                        ],
                        [
                          0.4,
                          "#fde0ef"
                        ],
                        [
                          0.5,
                          "#f7f7f7"
                        ],
                        [
                          0.6,
                          "#e6f5d0"
                        ],
                        [
                          0.7,
                          "#b8e186"
                        ],
                        [
                          0.8,
                          "#7fbc41"
                        ],
                        [
                          0.9,
                          "#4d9221"
                        ],
                        [
                          1,
                          "#276419"
                        ]
                      ],
                      "sequential": [
                        [
                          0,
                          "#0d0887"
                        ],
                        [
                          0.1111111111111111,
                          "#46039f"
                        ],
                        [
                          0.2222222222222222,
                          "#7201a8"
                        ],
                        [
                          0.3333333333333333,
                          "#9c179e"
                        ],
                        [
                          0.4444444444444444,
                          "#bd3786"
                        ],
                        [
                          0.5555555555555556,
                          "#d8576b"
                        ],
                        [
                          0.6666666666666666,
                          "#ed7953"
                        ],
                        [
                          0.7777777777777778,
                          "#fb9f3a"
                        ],
                        [
                          0.8888888888888888,
                          "#fdca26"
                        ],
                        [
                          1,
                          "#f0f921"
                        ]
                      ],
                      "sequentialminus": [
                        [
                          0,
                          "#0d0887"
                        ],
                        [
                          0.1111111111111111,
                          "#46039f"
                        ],
                        [
                          0.2222222222222222,
                          "#7201a8"
                        ],
                        [
                          0.3333333333333333,
                          "#9c179e"
                        ],
                        [
                          0.4444444444444444,
                          "#bd3786"
                        ],
                        [
                          0.5555555555555556,
                          "#d8576b"
                        ],
                        [
                          0.6666666666666666,
                          "#ed7953"
                        ],
                        [
                          0.7777777777777778,
                          "#fb9f3a"
                        ],
                        [
                          0.8888888888888888,
                          "#fdca26"
                        ],
                        [
                          1,
                          "#f0f921"
                        ]
                      ]
                    },
                    "colorway": [
                      "#636efa",
                      "#EF553B",
                      "#00cc96",
                      "#ab63fa",
                      "#FFA15A",
                      "#19d3f3",
                      "#FF6692",
                      "#B6E880",
                      "#FF97FF",
                      "#FECB52"
                    ],
                    "font": {
                      "color": "#2a3f5f"
                    },
                    "geo": {
                      "bgcolor": "white",
                      "lakecolor": "white",
                      "landcolor": "#E5ECF6",
                      "showlakes": true,
                      "showland": true,
                      "subunitcolor": "white"
                    },
                    "hoverlabel": {
                      "align": "left"
                    },
                    "hovermode": "closest",
                    "mapbox": {
                      "style": "light"
                    },
                    "paper_bgcolor": "white",
                    "plot_bgcolor": "#E5ECF6",
                    "polar": {
                      "angularaxis": {
                        "gridcolor": "white",
                        "linecolor": "white",
                        "ticks": ""
                      },
                      "bgcolor": "#E5ECF6",
                      "radialaxis": {
                        "gridcolor": "white",
                        "linecolor": "white",
                        "ticks": ""
                      }
                    },
                    "scene": {
                      "xaxis": {
                        "backgroundcolor": "#E5ECF6",
                        "gridcolor": "white",
                        "gridwidth": 2,
                        "linecolor": "white",
                        "showbackground": true,
                        "ticks": "",
                        "zerolinecolor": "white"
                      },
                      "yaxis": {
                        "backgroundcolor": "#E5ECF6",
                        "gridcolor": "white",
                        "gridwidth": 2,
                        "linecolor": "white",
                        "showbackground": true,
                        "ticks": "",
                        "zerolinecolor": "white"
                      },
                      "zaxis": {
                        "backgroundcolor": "#E5ECF6",
                        "gridcolor": "white",
                        "gridwidth": 2,
                        "linecolor": "white",
                        "showbackground": true,
                        "ticks": "",
                        "zerolinecolor": "white"
                      }
                    },
                    "shapedefaults": {
                      "line": {
                        "color": "#2a3f5f"
                      }
                    },
                    "ternary": {
                      "aaxis": {
                        "gridcolor": "white",
                        "linecolor": "white",
                        "ticks": ""
                      },
                      "baxis": {
                        "gridcolor": "white",
                        "linecolor": "white",
                        "ticks": ""
                      },
                      "bgcolor": "#E5ECF6",
                      "caxis": {
                        "gridcolor": "white",
                        "linecolor": "white",
                        "ticks": ""
                      }
                    },
                    "title": {
                      "x": 0.05
                    },
                    "xaxis": {
                      "automargin": true,
                      "gridcolor": "white",
                      "linecolor": "white",
                      "ticks": "",
                      "title": {
                        "standoff": 15
                      },
                      "zerolinecolor": "white",
                      "zerolinewidth": 2
                    },
                    "yaxis": {
                      "automargin": true,
                      "gridcolor": "white",
                      "linecolor": "white",
                      "ticks": "",
                      "title": {
                        "standoff": 15
                      },
                      "zerolinecolor": "white",
                      "zerolinewidth": 2
                    }
                  }
                },
                "title": {
                  "text": "Topics by Year"
                },
                "xaxis": {
                  "title": {
                    "text": "Year"
                  }
                },
                "yaxis": {
                  "title": {
                    "text": "Quantity"
                  }
                }
              }
            }
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "#df_years = df_years.groupby(df_years.publishedDate)\n",
        "\n",
        "fig = go.Figure(data=[\n",
        "    go.Scatter(name=\"cooking\", x=df_years.index, y=df_years[0]),\n",
        "    go.Scatter(name=\"family\", x=df_years.index, y=df_years[1]),\n",
        "    go.Scatter(name=\"socio economics\", x=df_years.index, y=df_years[2]),\n",
        "    go.Scatter(name=\"fine arts\", x=df_years.index, y=df_years[3]),\n",
        "    go.Scatter(name=\"religion\", x=df_years.index, y=df_years[4]),\n",
        "    go.Scatter(name=\"student help\", x=df_years.index, y=df_years[5]),\n",
        "    go.Scatter(name=\"fictional romance\", x=df_years.index, y=df_years[6]),\n",
        "    go.Scatter(name=\"sport & outdoors\", x=df_years.index, y=df_years[7]),\n",
        "    go.Scatter(name=\"poetry & literature\", x=df_years.index, y=df_years[8]),\n",
        "    go.Scatter(name=\"history\", x=df_years.index, y=df_years[9])\n",
        "\n",
        "])\n",
        "fig.update_layout(title='Topics by Year',\n",
        "                   xaxis_title='Year',\n",
        "                   yaxis_title='Quantity',\n",
        "                   barmode = \"group\")\n",
        "\n",
        "# Change the bar mode\n",
        "#fig.update_layout(barmode='group')\n",
        "fig.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Some notable observations from the bar chart above:\n",
        "- In 2021, the number of books related to **fiction & romance** and **family** has gone up. This could be due to the difficult times due to the pandemic which undoubtedly left lots of people with desires of affection and human contact. \n",
        "link: https://www.frontiersin.org/articles/10.3389/fpsyg.2021.798260/full\n",
        "- There was a noticeable increase in quantity of books related to **student help** in 2012. This could be linked to the boom in popularity of massive open online courses - MOOCs in those years. It could be that the boom in MOOCs cause an increase in the production of texts for helping students in various aspects.  \n",
        "link: https://onlinelearninginsights.wordpress.com/2012/12/21/what-the-heck-happened-in-2012-review-of-the-top-three-events-in-education/\n",
        "- Not many books related to **sport & outdoors** are produced. This could be due to the fact that this is a theme in which people prefer to learn by doing rather than reading. \n",
        "- All the trends are decreasing and the sheer quantity of books are also less. This could be due to lack of data for these years or perhaps it could reflect a general trend of diminishing interest in books due to the internet. \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Sentiment Analysis\n",
        "In this section we will analyze the reviews from various users and perform sentiment analysis. This is a useful application in NLP since it can be used to know the sentiment towards a specific book based on the reviews. This information can be used by publishing companies and authors alike to know which products are liked most and what the public likes/dislikes.\n",
        "\n",
        "In particular we will use the reviews dataset which we saw earlier."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {},
      "outputs": [],
      "source": [
        "df_reviews = pd.read_csv(\"./DataSet/Books_rating.csv\" )"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "The Data Set contains 3.0 million reviews\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Id</th>\n",
              "      <th>Title</th>\n",
              "      <th>Price</th>\n",
              "      <th>User_id</th>\n",
              "      <th>profileName</th>\n",
              "      <th>helpfulness</th>\n",
              "      <th>score</th>\n",
              "      <th>time</th>\n",
              "      <th>summary</th>\n",
              "      <th>text</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1882931173</td>\n",
              "      <td>Its Only Art If Its Well Hung!</td>\n",
              "      <td>NaN</td>\n",
              "      <td>AVCGYZL8FQQTD</td>\n",
              "      <td>Jim of Oz \"jim-of-oz\"</td>\n",
              "      <td>7/7</td>\n",
              "      <td>4.0</td>\n",
              "      <td>940636800</td>\n",
              "      <td>Nice collection of Julie Strain images</td>\n",
              "      <td>This is only for Julie Strain fans. It's a col...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0826414346</td>\n",
              "      <td>Dr. Seuss: American Icon</td>\n",
              "      <td>NaN</td>\n",
              "      <td>A30TK6U7DNS82R</td>\n",
              "      <td>Kevin Killian</td>\n",
              "      <td>10/10</td>\n",
              "      <td>5.0</td>\n",
              "      <td>1095724800</td>\n",
              "      <td>Really Enjoyed It</td>\n",
              "      <td>I don't care much for Dr. Seuss but after read...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0826414346</td>\n",
              "      <td>Dr. Seuss: American Icon</td>\n",
              "      <td>NaN</td>\n",
              "      <td>A3UH4UZ4RSVO82</td>\n",
              "      <td>John Granger</td>\n",
              "      <td>10/11</td>\n",
              "      <td>5.0</td>\n",
              "      <td>1078790400</td>\n",
              "      <td>Essential for every personal and Public Library</td>\n",
              "      <td>If people become the books they read and if \"t...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>0826414346</td>\n",
              "      <td>Dr. Seuss: American Icon</td>\n",
              "      <td>NaN</td>\n",
              "      <td>A2MVUWT453QH61</td>\n",
              "      <td>Roy E. Perry \"amateur philosopher\"</td>\n",
              "      <td>7/7</td>\n",
              "      <td>4.0</td>\n",
              "      <td>1090713600</td>\n",
              "      <td>Phlip Nel gives silly Seuss a serious treatment</td>\n",
              "      <td>Theodore Seuss Geisel (1904-1991), aka &amp;quot;D...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0826414346</td>\n",
              "      <td>Dr. Seuss: American Icon</td>\n",
              "      <td>NaN</td>\n",
              "      <td>A22X4XUPKF66MR</td>\n",
              "      <td>D. H. Richards \"ninthwavestore\"</td>\n",
              "      <td>3/3</td>\n",
              "      <td>4.0</td>\n",
              "      <td>1107993600</td>\n",
              "      <td>Good academic overview</td>\n",
              "      <td>Philip Nel - Dr. Seuss: American IconThis is b...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "           Id                           Title  Price         User_id  \\\n",
              "0  1882931173  Its Only Art If Its Well Hung!    NaN   AVCGYZL8FQQTD   \n",
              "1  0826414346        Dr. Seuss: American Icon    NaN  A30TK6U7DNS82R   \n",
              "2  0826414346        Dr. Seuss: American Icon    NaN  A3UH4UZ4RSVO82   \n",
              "3  0826414346        Dr. Seuss: American Icon    NaN  A2MVUWT453QH61   \n",
              "4  0826414346        Dr. Seuss: American Icon    NaN  A22X4XUPKF66MR   \n",
              "\n",
              "                          profileName helpfulness  score        time  \\\n",
              "0               Jim of Oz \"jim-of-oz\"         7/7    4.0   940636800   \n",
              "1                       Kevin Killian       10/10    5.0  1095724800   \n",
              "2                        John Granger       10/11    5.0  1078790400   \n",
              "3  Roy E. Perry \"amateur philosopher\"         7/7    4.0  1090713600   \n",
              "4     D. H. Richards \"ninthwavestore\"         3/3    4.0  1107993600   \n",
              "\n",
              "                                           summary  \\\n",
              "0           Nice collection of Julie Strain images   \n",
              "1                                Really Enjoyed It   \n",
              "2  Essential for every personal and Public Library   \n",
              "3  Phlip Nel gives silly Seuss a serious treatment   \n",
              "4                           Good academic overview   \n",
              "\n",
              "                                                text  \n",
              "0  This is only for Julie Strain fans. It's a col...  \n",
              "1  I don't care much for Dr. Seuss but after read...  \n",
              "2  If people become the books they read and if \"t...  \n",
              "3  Theodore Seuss Geisel (1904-1991), aka &quot;D...  \n",
              "4  Philip Nel - Dr. Seuss: American IconThis is b...  "
            ]
          },
          "execution_count": 14,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "print(f\"The Data Set contains {len(df_reviews)/1e6} million reviews\")\n",
        "df_reviews = df_reviews.rename(columns = {\"review/helpfulness\":\"helpfulness\", \"review/score\":\"score\", \"review/time\":\"time\", \"review/summary\":\"summary\", \"review/text\":\"text\"})\n",
        "\n",
        "df_reviews.head()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Analysis of Data Set\n",
        " \n",
        "We analyze the distribution of the scores and find the most frequent words for the positive and negative reviews."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "application/vnd.plotly.v1+json": {
              "config": {
                "plotlyServerURL": "https://plot.ly"
              },
              "data": [
                {
                  "alignmentgroup": "True",
                  "hovertemplate": "variable=score<br>count=%{x}<br>score=%{y}<extra></extra>",
                  "legendgroup": "score",
                  "marker": {
                    "color": "#636efa",
                    "pattern": {
                      "shape": ""
                    }
                  },
                  "name": "score",
                  "offsetgroup": "score",
                  "orientation": "h",
                  "showlegend": true,
                  "textposition": "auto",
                  "type": "bar",
                  "x": [
                    1807343,
                    585616,
                    254295,
                    201688,
                    151058
                  ],
                  "xaxis": "x",
                  "y": [
                    5,
                    4,
                    3,
                    1,
                    2
                  ],
                  "yaxis": "y"
                }
              ],
              "layout": {
                "barmode": "relative",
                "legend": {
                  "title": {
                    "text": "variable"
                  },
                  "tracegroupgap": 0
                },
                "template": {
                  "data": {
                    "bar": [
                      {
                        "error_x": {
                          "color": "#2a3f5f"
                        },
                        "error_y": {
                          "color": "#2a3f5f"
                        },
                        "marker": {
                          "line": {
                            "color": "#E5ECF6",
                            "width": 0.5
                          },
                          "pattern": {
                            "fillmode": "overlay",
                            "size": 10,
                            "solidity": 0.2
                          }
                        },
                        "type": "bar"
                      }
                    ],
                    "barpolar": [
                      {
                        "marker": {
                          "line": {
                            "color": "#E5ECF6",
                            "width": 0.5
                          },
                          "pattern": {
                            "fillmode": "overlay",
                            "size": 10,
                            "solidity": 0.2
                          }
                        },
                        "type": "barpolar"
                      }
                    ],
                    "carpet": [
                      {
                        "aaxis": {
                          "endlinecolor": "#2a3f5f",
                          "gridcolor": "white",
                          "linecolor": "white",
                          "minorgridcolor": "white",
                          "startlinecolor": "#2a3f5f"
                        },
                        "baxis": {
                          "endlinecolor": "#2a3f5f",
                          "gridcolor": "white",
                          "linecolor": "white",
                          "minorgridcolor": "white",
                          "startlinecolor": "#2a3f5f"
                        },
                        "type": "carpet"
                      }
                    ],
                    "choropleth": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "type": "choropleth"
                      }
                    ],
                    "contour": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "colorscale": [
                          [
                            0,
                            "#0d0887"
                          ],
                          [
                            0.1111111111111111,
                            "#46039f"
                          ],
                          [
                            0.2222222222222222,
                            "#7201a8"
                          ],
                          [
                            0.3333333333333333,
                            "#9c179e"
                          ],
                          [
                            0.4444444444444444,
                            "#bd3786"
                          ],
                          [
                            0.5555555555555556,
                            "#d8576b"
                          ],
                          [
                            0.6666666666666666,
                            "#ed7953"
                          ],
                          [
                            0.7777777777777778,
                            "#fb9f3a"
                          ],
                          [
                            0.8888888888888888,
                            "#fdca26"
                          ],
                          [
                            1,
                            "#f0f921"
                          ]
                        ],
                        "type": "contour"
                      }
                    ],
                    "contourcarpet": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "type": "contourcarpet"
                      }
                    ],
                    "heatmap": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "colorscale": [
                          [
                            0,
                            "#0d0887"
                          ],
                          [
                            0.1111111111111111,
                            "#46039f"
                          ],
                          [
                            0.2222222222222222,
                            "#7201a8"
                          ],
                          [
                            0.3333333333333333,
                            "#9c179e"
                          ],
                          [
                            0.4444444444444444,
                            "#bd3786"
                          ],
                          [
                            0.5555555555555556,
                            "#d8576b"
                          ],
                          [
                            0.6666666666666666,
                            "#ed7953"
                          ],
                          [
                            0.7777777777777778,
                            "#fb9f3a"
                          ],
                          [
                            0.8888888888888888,
                            "#fdca26"
                          ],
                          [
                            1,
                            "#f0f921"
                          ]
                        ],
                        "type": "heatmap"
                      }
                    ],
                    "heatmapgl": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "colorscale": [
                          [
                            0,
                            "#0d0887"
                          ],
                          [
                            0.1111111111111111,
                            "#46039f"
                          ],
                          [
                            0.2222222222222222,
                            "#7201a8"
                          ],
                          [
                            0.3333333333333333,
                            "#9c179e"
                          ],
                          [
                            0.4444444444444444,
                            "#bd3786"
                          ],
                          [
                            0.5555555555555556,
                            "#d8576b"
                          ],
                          [
                            0.6666666666666666,
                            "#ed7953"
                          ],
                          [
                            0.7777777777777778,
                            "#fb9f3a"
                          ],
                          [
                            0.8888888888888888,
                            "#fdca26"
                          ],
                          [
                            1,
                            "#f0f921"
                          ]
                        ],
                        "type": "heatmapgl"
                      }
                    ],
                    "histogram": [
                      {
                        "marker": {
                          "pattern": {
                            "fillmode": "overlay",
                            "size": 10,
                            "solidity": 0.2
                          }
                        },
                        "type": "histogram"
                      }
                    ],
                    "histogram2d": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "colorscale": [
                          [
                            0,
                            "#0d0887"
                          ],
                          [
                            0.1111111111111111,
                            "#46039f"
                          ],
                          [
                            0.2222222222222222,
                            "#7201a8"
                          ],
                          [
                            0.3333333333333333,
                            "#9c179e"
                          ],
                          [
                            0.4444444444444444,
                            "#bd3786"
                          ],
                          [
                            0.5555555555555556,
                            "#d8576b"
                          ],
                          [
                            0.6666666666666666,
                            "#ed7953"
                          ],
                          [
                            0.7777777777777778,
                            "#fb9f3a"
                          ],
                          [
                            0.8888888888888888,
                            "#fdca26"
                          ],
                          [
                            1,
                            "#f0f921"
                          ]
                        ],
                        "type": "histogram2d"
                      }
                    ],
                    "histogram2dcontour": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "colorscale": [
                          [
                            0,
                            "#0d0887"
                          ],
                          [
                            0.1111111111111111,
                            "#46039f"
                          ],
                          [
                            0.2222222222222222,
                            "#7201a8"
                          ],
                          [
                            0.3333333333333333,
                            "#9c179e"
                          ],
                          [
                            0.4444444444444444,
                            "#bd3786"
                          ],
                          [
                            0.5555555555555556,
                            "#d8576b"
                          ],
                          [
                            0.6666666666666666,
                            "#ed7953"
                          ],
                          [
                            0.7777777777777778,
                            "#fb9f3a"
                          ],
                          [
                            0.8888888888888888,
                            "#fdca26"
                          ],
                          [
                            1,
                            "#f0f921"
                          ]
                        ],
                        "type": "histogram2dcontour"
                      }
                    ],
                    "mesh3d": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "type": "mesh3d"
                      }
                    ],
                    "parcoords": [
                      {
                        "line": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "parcoords"
                      }
                    ],
                    "pie": [
                      {
                        "automargin": true,
                        "type": "pie"
                      }
                    ],
                    "scatter": [
                      {
                        "fillpattern": {
                          "fillmode": "overlay",
                          "size": 10,
                          "solidity": 0.2
                        },
                        "type": "scatter"
                      }
                    ],
                    "scatter3d": [
                      {
                        "line": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scatter3d"
                      }
                    ],
                    "scattercarpet": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scattercarpet"
                      }
                    ],
                    "scattergeo": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scattergeo"
                      }
                    ],
                    "scattergl": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scattergl"
                      }
                    ],
                    "scattermapbox": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scattermapbox"
                      }
                    ],
                    "scatterpolar": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scatterpolar"
                      }
                    ],
                    "scatterpolargl": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scatterpolargl"
                      }
                    ],
                    "scatterternary": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scatterternary"
                      }
                    ],
                    "surface": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "colorscale": [
                          [
                            0,
                            "#0d0887"
                          ],
                          [
                            0.1111111111111111,
                            "#46039f"
                          ],
                          [
                            0.2222222222222222,
                            "#7201a8"
                          ],
                          [
                            0.3333333333333333,
                            "#9c179e"
                          ],
                          [
                            0.4444444444444444,
                            "#bd3786"
                          ],
                          [
                            0.5555555555555556,
                            "#d8576b"
                          ],
                          [
                            0.6666666666666666,
                            "#ed7953"
                          ],
                          [
                            0.7777777777777778,
                            "#fb9f3a"
                          ],
                          [
                            0.8888888888888888,
                            "#fdca26"
                          ],
                          [
                            1,
                            "#f0f921"
                          ]
                        ],
                        "type": "surface"
                      }
                    ],
                    "table": [
                      {
                        "cells": {
                          "fill": {
                            "color": "#EBF0F8"
                          },
                          "line": {
                            "color": "white"
                          }
                        },
                        "header": {
                          "fill": {
                            "color": "#C8D4E3"
                          },
                          "line": {
                            "color": "white"
                          }
                        },
                        "type": "table"
                      }
                    ]
                  },
                  "layout": {
                    "annotationdefaults": {
                      "arrowcolor": "#2a3f5f",
                      "arrowhead": 0,
                      "arrowwidth": 1
                    },
                    "autotypenumbers": "strict",
                    "coloraxis": {
                      "colorbar": {
                        "outlinewidth": 0,
                        "ticks": ""
                      }
                    },
                    "colorscale": {
                      "diverging": [
                        [
                          0,
                          "#8e0152"
                        ],
                        [
                          0.1,
                          "#c51b7d"
                        ],
                        [
                          0.2,
                          "#de77ae"
                        ],
                        [
                          0.3,
                          "#f1b6da"
                        ],
                        [
                          0.4,
                          "#fde0ef"
                        ],
                        [
                          0.5,
                          "#f7f7f7"
                        ],
                        [
                          0.6,
                          "#e6f5d0"
                        ],
                        [
                          0.7,
                          "#b8e186"
                        ],
                        [
                          0.8,
                          "#7fbc41"
                        ],
                        [
                          0.9,
                          "#4d9221"
                        ],
                        [
                          1,
                          "#276419"
                        ]
                      ],
                      "sequential": [
                        [
                          0,
                          "#0d0887"
                        ],
                        [
                          0.1111111111111111,
                          "#46039f"
                        ],
                        [
                          0.2222222222222222,
                          "#7201a8"
                        ],
                        [
                          0.3333333333333333,
                          "#9c179e"
                        ],
                        [
                          0.4444444444444444,
                          "#bd3786"
                        ],
                        [
                          0.5555555555555556,
                          "#d8576b"
                        ],
                        [
                          0.6666666666666666,
                          "#ed7953"
                        ],
                        [
                          0.7777777777777778,
                          "#fb9f3a"
                        ],
                        [
                          0.8888888888888888,
                          "#fdca26"
                        ],
                        [
                          1,
                          "#f0f921"
                        ]
                      ],
                      "sequentialminus": [
                        [
                          0,
                          "#0d0887"
                        ],
                        [
                          0.1111111111111111,
                          "#46039f"
                        ],
                        [
                          0.2222222222222222,
                          "#7201a8"
                        ],
                        [
                          0.3333333333333333,
                          "#9c179e"
                        ],
                        [
                          0.4444444444444444,
                          "#bd3786"
                        ],
                        [
                          0.5555555555555556,
                          "#d8576b"
                        ],
                        [
                          0.6666666666666666,
                          "#ed7953"
                        ],
                        [
                          0.7777777777777778,
                          "#fb9f3a"
                        ],
                        [
                          0.8888888888888888,
                          "#fdca26"
                        ],
                        [
                          1,
                          "#f0f921"
                        ]
                      ]
                    },
                    "colorway": [
                      "#636efa",
                      "#EF553B",
                      "#00cc96",
                      "#ab63fa",
                      "#FFA15A",
                      "#19d3f3",
                      "#FF6692",
                      "#B6E880",
                      "#FF97FF",
                      "#FECB52"
                    ],
                    "font": {
                      "color": "#2a3f5f"
                    },
                    "geo": {
                      "bgcolor": "white",
                      "lakecolor": "white",
                      "landcolor": "#E5ECF6",
                      "showlakes": true,
                      "showland": true,
                      "subunitcolor": "white"
                    },
                    "hoverlabel": {
                      "align": "left"
                    },
                    "hovermode": "closest",
                    "mapbox": {
                      "style": "light"
                    },
                    "paper_bgcolor": "white",
                    "plot_bgcolor": "#E5ECF6",
                    "polar": {
                      "angularaxis": {
                        "gridcolor": "white",
                        "linecolor": "white",
                        "ticks": ""
                      },
                      "bgcolor": "#E5ECF6",
                      "radialaxis": {
                        "gridcolor": "white",
                        "linecolor": "white",
                        "ticks": ""
                      }
                    },
                    "scene": {
                      "xaxis": {
                        "backgroundcolor": "#E5ECF6",
                        "gridcolor": "white",
                        "gridwidth": 2,
                        "linecolor": "white",
                        "showbackground": true,
                        "ticks": "",
                        "zerolinecolor": "white"
                      },
                      "yaxis": {
                        "backgroundcolor": "#E5ECF6",
                        "gridcolor": "white",
                        "gridwidth": 2,
                        "linecolor": "white",
                        "showbackground": true,
                        "ticks": "",
                        "zerolinecolor": "white"
                      },
                      "zaxis": {
                        "backgroundcolor": "#E5ECF6",
                        "gridcolor": "white",
                        "gridwidth": 2,
                        "linecolor": "white",
                        "showbackground": true,
                        "ticks": "",
                        "zerolinecolor": "white"
                      }
                    },
                    "shapedefaults": {
                      "line": {
                        "color": "#2a3f5f"
                      }
                    },
                    "ternary": {
                      "aaxis": {
                        "gridcolor": "white",
                        "linecolor": "white",
                        "ticks": ""
                      },
                      "baxis": {
                        "gridcolor": "white",
                        "linecolor": "white",
                        "ticks": ""
                      },
                      "bgcolor": "#E5ECF6",
                      "caxis": {
                        "gridcolor": "white",
                        "linecolor": "white",
                        "ticks": ""
                      }
                    },
                    "title": {
                      "x": 0.05
                    },
                    "xaxis": {
                      "automargin": true,
                      "gridcolor": "white",
                      "linecolor": "white",
                      "ticks": "",
                      "title": {
                        "standoff": 15
                      },
                      "zerolinecolor": "white",
                      "zerolinewidth": 2
                    },
                    "yaxis": {
                      "automargin": true,
                      "gridcolor": "white",
                      "linecolor": "white",
                      "ticks": "",
                      "title": {
                        "standoff": 15
                      },
                      "zerolinecolor": "white",
                      "zerolinewidth": 2
                    }
                  }
                },
                "title": {
                  "text": "Distribution of scores"
                },
                "xaxis": {
                  "anchor": "y",
                  "domain": [
                    0,
                    1
                  ],
                  "title": {
                    "text": "count"
                  }
                },
                "yaxis": {
                  "anchor": "x",
                  "domain": [
                    0,
                    1
                  ],
                  "title": {
                    "text": "score"
                  }
                }
              }
            }
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "fig = px.bar(df_reviews[\"score\"].value_counts(), orientation = \"h\", title = f\"Distribution of scores\",\n",
        "             labels = {\"index\": \"score\", \"value\":\"count\"})\n",
        "fig.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "As we can see, there is a great imbalance towards 5-star reviews. \n",
        "\n",
        "By looking at the first 50 3-star reviews, we realize that it is difficult to categorize them as positive or negative. Therefore, they will simply be classified as \"neutral\"."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "360556                         study guide for microbiology\n",
              "2951265                                             OK book\n",
              "1933705    The Scarlet Letter -- Wordy, dense, far too long\n",
              "1792488             Another bittersweet slice of reality...\n",
              "1771923           A classic, but hard to place an age group\n",
              "2719754                                        Average book\n",
              "2100828                   Yes, something is definitely awry\n",
              "462543                         stats, stats, and mo' stats!\n",
              "1860971                                 I have not read it.\n",
              "624287            Better than your average cash cow sequel.\n",
              "2545175                              Interesting but slight\n",
              "1475002                     The turning point in the series\n",
              "441864            Not what I was expecting, but still okay.\n",
              "2755116                                         Cigar Guide\n",
              "1792390          Perfection is always a tough act to follow\n",
              "2004466                                   Moving yet Opaque\n",
              "1825311                                            Good one\n",
              "2360625                                          Full House\n",
              "906208       Exciting adventure as well as quality romance.\n",
              "1499827                         Informative, but uneventful\n",
              "Name: summary, dtype: object"
            ]
          },
          "execution_count": 18,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "df_reviews[df_reviews[\"score\"] == 3.0][\"summary\"].sample(20, random_state= 2)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {},
      "outputs": [],
      "source": [
        "df_reviews.loc[df_reviews[\"score\"]>3.0, \"class\"] = \"POSITIVE\"\n",
        "df_reviews.loc[df_reviews[\"score\"]==3.0, \"class\"] = \"NEUTRAL\"\n",
        "df_reviews.loc[df_reviews[\"score\"]<3.0, \"class\"] = \"NEGATIVE\"\n",
        " "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Word Cloud\n",
        "\n",
        "Lets look at the word cloud for positive reviews and negative ones. As usual we have to do some preprocessing in order to reduce variation and catch more conceptual words. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {},
      "outputs": [],
      "source": [
        "positive_reviews = df_reviews[df_reviews[\"class\"]==\"POSITIVE\"].dropna(subset = [\"summary\"]).sample(100000, random_state=2)\n",
        "negative_reviews = df_reviews[df_reviews[\"class\"]==\"NEGATIVE\"].dropna(subset = [\"summary\"]).sample(100000, random_state=2)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {},
      "outputs": [],
      "source": [
        "positive_reviews[\"clean_summary\"] = clean_corpus(positive_reviews[\"summary\"])\n",
        "negative_reviews[\"clean_summary\"] = clean_corpus(negative_reviews[\"summary\"])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {},
      "outputs": [],
      "source": [
        "positive_words = \" \".join(positive_reviews[\"clean_summary\"])\n",
        "wordcloud_positive = stylecloud.gen_stylecloud(\n",
        "                        text = positive_words,\n",
        "                        size = 1028,\n",
        "                        background_color = \"black\",\n",
        "                        output_name = \"./Images/positive_wordcloud.png\",\n",
        "                        icon_name = \"fas fa-comment\"\n",
        "                        )"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "![image](./Images/positive_wordcloud.png)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {},
      "outputs": [],
      "source": [
        "negative_words = \" \".join(negative_reviews[\"clean_summary\"])\n",
        "wordcloud_negative = stylecloud.gen_stylecloud(\n",
        "                        text = negative_words,\n",
        "                        size = 1028,\n",
        "                        background_color = \"black\",\n",
        "                        output_name = \"./Images/negative_wordcloud.png\",\n",
        "                        icon_name = \"fas fa-comment\"\n",
        "                        )"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "![image](./Images/negative_wordcloud.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "In the negative wordcloud, we can immediately observe the presence of some terms that we associate with positive emotions. For example, the word \"good\" and \"great\" are seen in the wordcloud. One possible explanation is that these terms are always associated with negative connotations. For example, they could be preceded by the word \"not\" or \"hardly any\" or any word that negates the positive context of the word. \n",
        "This would be easier to understand if we included the collocations as we did for topic modeling. However, since this is an exploratory analysis on a random subsample of 100 thousand documents, it won't be performed.\n",
        "\n",
        "We can try to get a sense of the correctness of our claim by analyzing the reviews which contain the word \"good\" in it. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "1477650    Notice most of the good reviews on this are by...\n",
              "2073598         Airplane/Beach Read, Not good for much else.\n",
              "2002262                                     not good quality\n",
              "606137              Some good information, but high on guilt\n",
              "546251                             Not really that good.....\n",
              "Name: summary, dtype: object"
            ]
          },
          "execution_count": 15,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "negative_reviews[negative_reviews[\"summary\"].str.contains(\"good\")][\"summary\"].head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "As we can see, our hypothesis seems to be correct. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Bert\n",
        "\n",
        "As we mentioned before, we will be using BERT for our analysis.\n",
        "\n",
        "In particular, we will test:\n",
        "\n",
        "- Multilingual Bert: This model is trained on product reviews in various languages and is the most natural model to handle our data set since it outputs a score from 1-5. In this way, we do not have to do any preprocessing at all. \n",
        "- RoBERTa: This model is trained on 15 different datasets including reviews and tweets and can be fine tuned. However, it is only in english and ouputs a binary classification of positive or negative.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Multilingual Bert (not used in project)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {},
      "outputs": [],
      "source": [
        "model_name = \"nlptown/bert-base-multilingual-uncased-sentiment\"\n",
        "\n",
        "tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
        "model = AutoModelForSequenceClassification.from_pretrained(model_name)\n",
        "\n",
        "classifier = pipeline(\"sentiment-analysis\", model=model, tokenizer=tokenizer)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Id</th>\n",
              "      <th>Title</th>\n",
              "      <th>Price</th>\n",
              "      <th>User_id</th>\n",
              "      <th>profileName</th>\n",
              "      <th>helpfulness</th>\n",
              "      <th>score</th>\n",
              "      <th>time</th>\n",
              "      <th>summary</th>\n",
              "      <th>text</th>\n",
              "      <th>predicted_rating_BERT</th>\n",
              "      <th>lang</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2524966</th>\n",
              "      <td>B000HZ8ZR8</td>\n",
              "      <td>The Garden of Eden, A Novel</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>4/11</td>\n",
              "      <td>2.0</td>\n",
              "      <td>912297600</td>\n",
              "      <td>The Garden of Eden? Hmmmm......</td>\n",
              "      <td>This book starts out great, it romantisizes Eu...</td>\n",
              "      <td>2</td>\n",
              "      <td>en</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>530830</th>\n",
              "      <td>B000HOSLEG</td>\n",
              "      <td>Fairytale</td>\n",
              "      <td>NaN</td>\n",
              "      <td>A2PGVEDRPGGIJ0</td>\n",
              "      <td>coteb</td>\n",
              "      <td>2/2</td>\n",
              "      <td>5.0</td>\n",
              "      <td>1338854400</td>\n",
              "      <td>loved it</td>\n",
              "      <td>I highly recommend if you like Maggie shayne. ...</td>\n",
              "      <td>5</td>\n",
              "      <td>en</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2597389</th>\n",
              "      <td>0754053822</td>\n",
              "      <td>Disgrace: Complete &amp; Unabridged</td>\n",
              "      <td>NaN</td>\n",
              "      <td>A1MRV6VAB1HVJW</td>\n",
              "      <td>Gisele W. Wright</td>\n",
              "      <td>0/4</td>\n",
              "      <td>3.0</td>\n",
              "      <td>1022803200</td>\n",
              "      <td>A hard read</td>\n",
              "      <td>What drew to this novel was the title and the ...</td>\n",
              "      <td>2</td>\n",
              "      <td>en</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>218539</th>\n",
              "      <td>0786222859</td>\n",
              "      <td>Bittersweet Rain</td>\n",
              "      <td>NaN</td>\n",
              "      <td>A21MVX83HPR5VK</td>\n",
              "      <td>Dina_179</td>\n",
              "      <td>17/22</td>\n",
              "      <td>4.0</td>\n",
              "      <td>952128000</td>\n",
              "      <td>Ms. Brown keeps surprising me with her amazing...</td>\n",
              "      <td>Only Sandra Brown will put together the most u...</td>\n",
              "      <td>5</td>\n",
              "      <td>en</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2759866</th>\n",
              "      <td>0821755641</td>\n",
              "      <td>Unforgettable</td>\n",
              "      <td>NaN</td>\n",
              "      <td>A1N967FZREGFG</td>\n",
              "      <td>Tess Capra</td>\n",
              "      <td>3/3</td>\n",
              "      <td>4.0</td>\n",
              "      <td>890524800</td>\n",
              "      <td>Not unforgettable, but pretty darn good.</td>\n",
              "      <td>Perhaps if Zebra spent as much on publicity as...</td>\n",
              "      <td>3</td>\n",
              "      <td>en</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1644252</th>\n",
              "      <td>0395615801</td>\n",
              "      <td>A Nightmare in History: The Holocaust 1933-1945</td>\n",
              "      <td>NaN</td>\n",
              "      <td>A1OJHNAMUW4F4Y</td>\n",
              "      <td>David N. 1st period.</td>\n",
              "      <td>2/5</td>\n",
              "      <td>5.0</td>\n",
              "      <td>1008028800</td>\n",
              "      <td>1st peiod reading -A nightmare in history</td>\n",
              "      <td>Imagne for a second that you are sleeping in y...</td>\n",
              "      <td>5</td>\n",
              "      <td>en</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2816988</th>\n",
              "      <td>1593555563</td>\n",
              "      <td>Stone of Tears (Sword of Truth Series)</td>\n",
              "      <td>167.25</td>\n",
              "      <td>A2JNZ09Y5JYPAK</td>\n",
              "      <td>I hate peas</td>\n",
              "      <td>2/5</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1325462400</td>\n",
              "      <td>Never should have been published</td>\n",
              "      <td>This book is absolutely terrible.If you take o...</td>\n",
              "      <td>1</td>\n",
              "      <td>en</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1088854</th>\n",
              "      <td>B000KIITWS</td>\n",
              "      <td>CONFUCIUS: THE SECULAR AS SACRED.</td>\n",
              "      <td>NaN</td>\n",
              "      <td>A3PBFMN1QU4VWV</td>\n",
              "      <td>John J. Gibbs</td>\n",
              "      <td>0/0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>1318636800</td>\n",
              "      <td>a gem</td>\n",
              "      <td>concise, well-written and constructed (as you ...</td>\n",
              "      <td>5</td>\n",
              "      <td>pt</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>962254</th>\n",
              "      <td>0875968708</td>\n",
              "      <td>You Bet Your Tomatoes: Fun Facts, Tall Tales, ...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>ASFNOYMHYMQNO</td>\n",
              "      <td>GLORIA O'CONNELL</td>\n",
              "      <td>1/1</td>\n",
              "      <td>4.0</td>\n",
              "      <td>1245974400</td>\n",
              "      <td>Entertaining and informative.</td>\n",
              "      <td>I've been gardening for many years (actually, ...</td>\n",
              "      <td>4</td>\n",
              "      <td>en</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2583708</th>\n",
              "      <td>B000H7EO2G</td>\n",
              "      <td>The Lord of the Rings Trilogy 3 Volumes</td>\n",
              "      <td>NaN</td>\n",
              "      <td>A1FIBTUKX80YNU</td>\n",
              "      <td>Brian F. Hudon</td>\n",
              "      <td>8/9</td>\n",
              "      <td>5.0</td>\n",
              "      <td>1148256000</td>\n",
              "      <td>One book to rule them all!</td>\n",
              "      <td>This is an exquisite book to own and to read, ...</td>\n",
              "      <td>5</td>\n",
              "      <td>en</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>300000 rows × 12 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "                 Id                                              Title  \\\n",
              "2524966  B000HZ8ZR8                        The Garden of Eden, A Novel   \n",
              "530830   B000HOSLEG                                          Fairytale   \n",
              "2597389  0754053822                    Disgrace: Complete & Unabridged   \n",
              "218539   0786222859                                   Bittersweet Rain   \n",
              "2759866  0821755641                                      Unforgettable   \n",
              "...             ...                                                ...   \n",
              "1644252  0395615801    A Nightmare in History: The Holocaust 1933-1945   \n",
              "2816988  1593555563             Stone of Tears (Sword of Truth Series)   \n",
              "1088854  B000KIITWS                  CONFUCIUS: THE SECULAR AS SACRED.   \n",
              "962254   0875968708  You Bet Your Tomatoes: Fun Facts, Tall Tales, ...   \n",
              "2583708  B000H7EO2G            The Lord of the Rings Trilogy 3 Volumes   \n",
              "\n",
              "          Price         User_id           profileName helpfulness  score  \\\n",
              "2524966     NaN             NaN                   NaN        4/11    2.0   \n",
              "530830      NaN  A2PGVEDRPGGIJ0                 coteb         2/2    5.0   \n",
              "2597389     NaN  A1MRV6VAB1HVJW      Gisele W. Wright         0/4    3.0   \n",
              "218539      NaN  A21MVX83HPR5VK              Dina_179       17/22    4.0   \n",
              "2759866     NaN   A1N967FZREGFG            Tess Capra         3/3    4.0   \n",
              "...         ...             ...                   ...         ...    ...   \n",
              "1644252     NaN  A1OJHNAMUW4F4Y  David N. 1st period.         2/5    5.0   \n",
              "2816988  167.25  A2JNZ09Y5JYPAK           I hate peas         2/5    1.0   \n",
              "1088854     NaN  A3PBFMN1QU4VWV         John J. Gibbs         0/0    5.0   \n",
              "962254      NaN   ASFNOYMHYMQNO      GLORIA O'CONNELL         1/1    4.0   \n",
              "2583708     NaN  A1FIBTUKX80YNU        Brian F. Hudon         8/9    5.0   \n",
              "\n",
              "               time                                            summary  \\\n",
              "2524966   912297600                    The Garden of Eden? Hmmmm......   \n",
              "530830   1338854400                                           loved it   \n",
              "2597389  1022803200                                        A hard read   \n",
              "218539    952128000  Ms. Brown keeps surprising me with her amazing...   \n",
              "2759866   890524800           Not unforgettable, but pretty darn good.   \n",
              "...             ...                                                ...   \n",
              "1644252  1008028800          1st peiod reading -A nightmare in history   \n",
              "2816988  1325462400                   Never should have been published   \n",
              "1088854  1318636800                                              a gem   \n",
              "962254   1245974400                      Entertaining and informative.   \n",
              "2583708  1148256000                         One book to rule them all!   \n",
              "\n",
              "                                                      text  \\\n",
              "2524966  This book starts out great, it romantisizes Eu...   \n",
              "530830   I highly recommend if you like Maggie shayne. ...   \n",
              "2597389  What drew to this novel was the title and the ...   \n",
              "218539   Only Sandra Brown will put together the most u...   \n",
              "2759866  Perhaps if Zebra spent as much on publicity as...   \n",
              "...                                                    ...   \n",
              "1644252  Imagne for a second that you are sleeping in y...   \n",
              "2816988  This book is absolutely terrible.If you take o...   \n",
              "1088854  concise, well-written and constructed (as you ...   \n",
              "962254   I've been gardening for many years (actually, ...   \n",
              "2583708  This is an exquisite book to own and to read, ...   \n",
              "\n",
              "         predicted_rating_BERT lang  \n",
              "2524966                      2   en  \n",
              "530830                       5   en  \n",
              "2597389                      2   en  \n",
              "218539                       5   en  \n",
              "2759866                      3   en  \n",
              "...                        ...  ...  \n",
              "1644252                      5   en  \n",
              "2816988                      1   en  \n",
              "1088854                      5   pt  \n",
              "962254                       4   en  \n",
              "2583708                      5   en  \n",
              "\n",
              "[300000 rows x 12 columns]"
            ]
          },
          "execution_count": 3,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "if FIRST_RUN:\n",
        "    df_reviews = df_reviews.dropna(subset=[\"summary\"]).sample(300000, random_state=2)\n",
        "    tqdm.pandas()\n",
        "    df_reviews[\"predicted_rating_BERT\"] = df_reviews[\"summary\"].progress_apply(lambda x: int(re.sub(r\"\\D\",\"\",classifier(x)[0]['label'])))\n",
        "    lang = [nlp(doc)._.language for doc in tqdm(df_reviews[\"summary\"])]\n",
        "    df_reviews[\"lang\"] = lang\n",
        "    with open(\"./DataSet/reviews_small_predicted.csv\", \"wb\") as fp:\n",
        "        pickle.dump(df_reviews,fp)\n",
        "else:\n",
        "    with open(\"./DataSet/reviews_small_predicted.csv\", \"rb\") as fp:\n",
        "        df_reviews = pickle.load(fp)\n",
        "df_reviews"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "application/vnd.plotly.v1+json": {
              "config": {
                "plotlyServerURL": "https://plot.ly"
              },
              "data": [
                {
                  "alignmentgroup": "True",
                  "hovertemplate": "variable=score<br>value=%{x}<br>index=%{y}<extra></extra>",
                  "legendgroup": "score",
                  "marker": {
                    "color": "#636efa",
                    "pattern": {
                      "shape": ""
                    }
                  },
                  "name": "score",
                  "offsetgroup": "score",
                  "orientation": "h",
                  "showlegend": true,
                  "textposition": "auto",
                  "type": "bar",
                  "x": [
                    180957,
                    58262,
                    25563,
                    20127,
                    15091
                  ],
                  "xaxis": "x",
                  "y": [
                    5,
                    4,
                    3,
                    1,
                    2
                  ],
                  "yaxis": "y"
                }
              ],
              "layout": {
                "barmode": "relative",
                "legend": {
                  "title": {
                    "text": "variable"
                  },
                  "tracegroupgap": 0
                },
                "template": {
                  "data": {
                    "bar": [
                      {
                        "error_x": {
                          "color": "#2a3f5f"
                        },
                        "error_y": {
                          "color": "#2a3f5f"
                        },
                        "marker": {
                          "line": {
                            "color": "#E5ECF6",
                            "width": 0.5
                          },
                          "pattern": {
                            "fillmode": "overlay",
                            "size": 10,
                            "solidity": 0.2
                          }
                        },
                        "type": "bar"
                      }
                    ],
                    "barpolar": [
                      {
                        "marker": {
                          "line": {
                            "color": "#E5ECF6",
                            "width": 0.5
                          },
                          "pattern": {
                            "fillmode": "overlay",
                            "size": 10,
                            "solidity": 0.2
                          }
                        },
                        "type": "barpolar"
                      }
                    ],
                    "carpet": [
                      {
                        "aaxis": {
                          "endlinecolor": "#2a3f5f",
                          "gridcolor": "white",
                          "linecolor": "white",
                          "minorgridcolor": "white",
                          "startlinecolor": "#2a3f5f"
                        },
                        "baxis": {
                          "endlinecolor": "#2a3f5f",
                          "gridcolor": "white",
                          "linecolor": "white",
                          "minorgridcolor": "white",
                          "startlinecolor": "#2a3f5f"
                        },
                        "type": "carpet"
                      }
                    ],
                    "choropleth": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "type": "choropleth"
                      }
                    ],
                    "contour": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "colorscale": [
                          [
                            0,
                            "#0d0887"
                          ],
                          [
                            0.1111111111111111,
                            "#46039f"
                          ],
                          [
                            0.2222222222222222,
                            "#7201a8"
                          ],
                          [
                            0.3333333333333333,
                            "#9c179e"
                          ],
                          [
                            0.4444444444444444,
                            "#bd3786"
                          ],
                          [
                            0.5555555555555556,
                            "#d8576b"
                          ],
                          [
                            0.6666666666666666,
                            "#ed7953"
                          ],
                          [
                            0.7777777777777778,
                            "#fb9f3a"
                          ],
                          [
                            0.8888888888888888,
                            "#fdca26"
                          ],
                          [
                            1,
                            "#f0f921"
                          ]
                        ],
                        "type": "contour"
                      }
                    ],
                    "contourcarpet": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "type": "contourcarpet"
                      }
                    ],
                    "heatmap": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "colorscale": [
                          [
                            0,
                            "#0d0887"
                          ],
                          [
                            0.1111111111111111,
                            "#46039f"
                          ],
                          [
                            0.2222222222222222,
                            "#7201a8"
                          ],
                          [
                            0.3333333333333333,
                            "#9c179e"
                          ],
                          [
                            0.4444444444444444,
                            "#bd3786"
                          ],
                          [
                            0.5555555555555556,
                            "#d8576b"
                          ],
                          [
                            0.6666666666666666,
                            "#ed7953"
                          ],
                          [
                            0.7777777777777778,
                            "#fb9f3a"
                          ],
                          [
                            0.8888888888888888,
                            "#fdca26"
                          ],
                          [
                            1,
                            "#f0f921"
                          ]
                        ],
                        "type": "heatmap"
                      }
                    ],
                    "heatmapgl": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "colorscale": [
                          [
                            0,
                            "#0d0887"
                          ],
                          [
                            0.1111111111111111,
                            "#46039f"
                          ],
                          [
                            0.2222222222222222,
                            "#7201a8"
                          ],
                          [
                            0.3333333333333333,
                            "#9c179e"
                          ],
                          [
                            0.4444444444444444,
                            "#bd3786"
                          ],
                          [
                            0.5555555555555556,
                            "#d8576b"
                          ],
                          [
                            0.6666666666666666,
                            "#ed7953"
                          ],
                          [
                            0.7777777777777778,
                            "#fb9f3a"
                          ],
                          [
                            0.8888888888888888,
                            "#fdca26"
                          ],
                          [
                            1,
                            "#f0f921"
                          ]
                        ],
                        "type": "heatmapgl"
                      }
                    ],
                    "histogram": [
                      {
                        "marker": {
                          "pattern": {
                            "fillmode": "overlay",
                            "size": 10,
                            "solidity": 0.2
                          }
                        },
                        "type": "histogram"
                      }
                    ],
                    "histogram2d": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "colorscale": [
                          [
                            0,
                            "#0d0887"
                          ],
                          [
                            0.1111111111111111,
                            "#46039f"
                          ],
                          [
                            0.2222222222222222,
                            "#7201a8"
                          ],
                          [
                            0.3333333333333333,
                            "#9c179e"
                          ],
                          [
                            0.4444444444444444,
                            "#bd3786"
                          ],
                          [
                            0.5555555555555556,
                            "#d8576b"
                          ],
                          [
                            0.6666666666666666,
                            "#ed7953"
                          ],
                          [
                            0.7777777777777778,
                            "#fb9f3a"
                          ],
                          [
                            0.8888888888888888,
                            "#fdca26"
                          ],
                          [
                            1,
                            "#f0f921"
                          ]
                        ],
                        "type": "histogram2d"
                      }
                    ],
                    "histogram2dcontour": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "colorscale": [
                          [
                            0,
                            "#0d0887"
                          ],
                          [
                            0.1111111111111111,
                            "#46039f"
                          ],
                          [
                            0.2222222222222222,
                            "#7201a8"
                          ],
                          [
                            0.3333333333333333,
                            "#9c179e"
                          ],
                          [
                            0.4444444444444444,
                            "#bd3786"
                          ],
                          [
                            0.5555555555555556,
                            "#d8576b"
                          ],
                          [
                            0.6666666666666666,
                            "#ed7953"
                          ],
                          [
                            0.7777777777777778,
                            "#fb9f3a"
                          ],
                          [
                            0.8888888888888888,
                            "#fdca26"
                          ],
                          [
                            1,
                            "#f0f921"
                          ]
                        ],
                        "type": "histogram2dcontour"
                      }
                    ],
                    "mesh3d": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "type": "mesh3d"
                      }
                    ],
                    "parcoords": [
                      {
                        "line": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "parcoords"
                      }
                    ],
                    "pie": [
                      {
                        "automargin": true,
                        "type": "pie"
                      }
                    ],
                    "scatter": [
                      {
                        "fillpattern": {
                          "fillmode": "overlay",
                          "size": 10,
                          "solidity": 0.2
                        },
                        "type": "scatter"
                      }
                    ],
                    "scatter3d": [
                      {
                        "line": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scatter3d"
                      }
                    ],
                    "scattercarpet": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scattercarpet"
                      }
                    ],
                    "scattergeo": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scattergeo"
                      }
                    ],
                    "scattergl": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scattergl"
                      }
                    ],
                    "scattermapbox": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scattermapbox"
                      }
                    ],
                    "scatterpolar": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scatterpolar"
                      }
                    ],
                    "scatterpolargl": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scatterpolargl"
                      }
                    ],
                    "scatterternary": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scatterternary"
                      }
                    ],
                    "surface": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "colorscale": [
                          [
                            0,
                            "#0d0887"
                          ],
                          [
                            0.1111111111111111,
                            "#46039f"
                          ],
                          [
                            0.2222222222222222,
                            "#7201a8"
                          ],
                          [
                            0.3333333333333333,
                            "#9c179e"
                          ],
                          [
                            0.4444444444444444,
                            "#bd3786"
                          ],
                          [
                            0.5555555555555556,
                            "#d8576b"
                          ],
                          [
                            0.6666666666666666,
                            "#ed7953"
                          ],
                          [
                            0.7777777777777778,
                            "#fb9f3a"
                          ],
                          [
                            0.8888888888888888,
                            "#fdca26"
                          ],
                          [
                            1,
                            "#f0f921"
                          ]
                        ],
                        "type": "surface"
                      }
                    ],
                    "table": [
                      {
                        "cells": {
                          "fill": {
                            "color": "#EBF0F8"
                          },
                          "line": {
                            "color": "white"
                          }
                        },
                        "header": {
                          "fill": {
                            "color": "#C8D4E3"
                          },
                          "line": {
                            "color": "white"
                          }
                        },
                        "type": "table"
                      }
                    ]
                  },
                  "layout": {
                    "annotationdefaults": {
                      "arrowcolor": "#2a3f5f",
                      "arrowhead": 0,
                      "arrowwidth": 1
                    },
                    "autotypenumbers": "strict",
                    "coloraxis": {
                      "colorbar": {
                        "outlinewidth": 0,
                        "ticks": ""
                      }
                    },
                    "colorscale": {
                      "diverging": [
                        [
                          0,
                          "#8e0152"
                        ],
                        [
                          0.1,
                          "#c51b7d"
                        ],
                        [
                          0.2,
                          "#de77ae"
                        ],
                        [
                          0.3,
                          "#f1b6da"
                        ],
                        [
                          0.4,
                          "#fde0ef"
                        ],
                        [
                          0.5,
                          "#f7f7f7"
                        ],
                        [
                          0.6,
                          "#e6f5d0"
                        ],
                        [
                          0.7,
                          "#b8e186"
                        ],
                        [
                          0.8,
                          "#7fbc41"
                        ],
                        [
                          0.9,
                          "#4d9221"
                        ],
                        [
                          1,
                          "#276419"
                        ]
                      ],
                      "sequential": [
                        [
                          0,
                          "#0d0887"
                        ],
                        [
                          0.1111111111111111,
                          "#46039f"
                        ],
                        [
                          0.2222222222222222,
                          "#7201a8"
                        ],
                        [
                          0.3333333333333333,
                          "#9c179e"
                        ],
                        [
                          0.4444444444444444,
                          "#bd3786"
                        ],
                        [
                          0.5555555555555556,
                          "#d8576b"
                        ],
                        [
                          0.6666666666666666,
                          "#ed7953"
                        ],
                        [
                          0.7777777777777778,
                          "#fb9f3a"
                        ],
                        [
                          0.8888888888888888,
                          "#fdca26"
                        ],
                        [
                          1,
                          "#f0f921"
                        ]
                      ],
                      "sequentialminus": [
                        [
                          0,
                          "#0d0887"
                        ],
                        [
                          0.1111111111111111,
                          "#46039f"
                        ],
                        [
                          0.2222222222222222,
                          "#7201a8"
                        ],
                        [
                          0.3333333333333333,
                          "#9c179e"
                        ],
                        [
                          0.4444444444444444,
                          "#bd3786"
                        ],
                        [
                          0.5555555555555556,
                          "#d8576b"
                        ],
                        [
                          0.6666666666666666,
                          "#ed7953"
                        ],
                        [
                          0.7777777777777778,
                          "#fb9f3a"
                        ],
                        [
                          0.8888888888888888,
                          "#fdca26"
                        ],
                        [
                          1,
                          "#f0f921"
                        ]
                      ]
                    },
                    "colorway": [
                      "#636efa",
                      "#EF553B",
                      "#00cc96",
                      "#ab63fa",
                      "#FFA15A",
                      "#19d3f3",
                      "#FF6692",
                      "#B6E880",
                      "#FF97FF",
                      "#FECB52"
                    ],
                    "font": {
                      "color": "#2a3f5f"
                    },
                    "geo": {
                      "bgcolor": "white",
                      "lakecolor": "white",
                      "landcolor": "#E5ECF6",
                      "showlakes": true,
                      "showland": true,
                      "subunitcolor": "white"
                    },
                    "hoverlabel": {
                      "align": "left"
                    },
                    "hovermode": "closest",
                    "mapbox": {
                      "style": "light"
                    },
                    "paper_bgcolor": "white",
                    "plot_bgcolor": "#E5ECF6",
                    "polar": {
                      "angularaxis": {
                        "gridcolor": "white",
                        "linecolor": "white",
                        "ticks": ""
                      },
                      "bgcolor": "#E5ECF6",
                      "radialaxis": {
                        "gridcolor": "white",
                        "linecolor": "white",
                        "ticks": ""
                      }
                    },
                    "scene": {
                      "xaxis": {
                        "backgroundcolor": "#E5ECF6",
                        "gridcolor": "white",
                        "gridwidth": 2,
                        "linecolor": "white",
                        "showbackground": true,
                        "ticks": "",
                        "zerolinecolor": "white"
                      },
                      "yaxis": {
                        "backgroundcolor": "#E5ECF6",
                        "gridcolor": "white",
                        "gridwidth": 2,
                        "linecolor": "white",
                        "showbackground": true,
                        "ticks": "",
                        "zerolinecolor": "white"
                      },
                      "zaxis": {
                        "backgroundcolor": "#E5ECF6",
                        "gridcolor": "white",
                        "gridwidth": 2,
                        "linecolor": "white",
                        "showbackground": true,
                        "ticks": "",
                        "zerolinecolor": "white"
                      }
                    },
                    "shapedefaults": {
                      "line": {
                        "color": "#2a3f5f"
                      }
                    },
                    "ternary": {
                      "aaxis": {
                        "gridcolor": "white",
                        "linecolor": "white",
                        "ticks": ""
                      },
                      "baxis": {
                        "gridcolor": "white",
                        "linecolor": "white",
                        "ticks": ""
                      },
                      "bgcolor": "#E5ECF6",
                      "caxis": {
                        "gridcolor": "white",
                        "linecolor": "white",
                        "ticks": ""
                      }
                    },
                    "title": {
                      "x": 0.05
                    },
                    "xaxis": {
                      "automargin": true,
                      "gridcolor": "white",
                      "linecolor": "white",
                      "ticks": "",
                      "title": {
                        "standoff": 15
                      },
                      "zerolinecolor": "white",
                      "zerolinewidth": 2
                    },
                    "yaxis": {
                      "automargin": true,
                      "gridcolor": "white",
                      "linecolor": "white",
                      "ticks": "",
                      "title": {
                        "standoff": 15
                      },
                      "zerolinecolor": "white",
                      "zerolinewidth": 2
                    }
                  }
                },
                "title": {
                  "text": "Distribution of scores"
                },
                "xaxis": {
                  "anchor": "y",
                  "domain": [
                    0,
                    1
                  ],
                  "title": {
                    "text": "value"
                  }
                },
                "yaxis": {
                  "anchor": "x",
                  "domain": [
                    0,
                    1
                  ],
                  "title": {
                    "text": "index"
                  }
                }
              }
            }
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "fig = px.bar(df_reviews[\"score\"].value_counts(), orientation = \"h\", title = f\"Distribution of scores\", labels = {\"index\": \"score\", \"value\":\"count\"} )\n",
        "fig.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "         1.0       0.32      0.55      0.41     20127\n",
            "         2.0       0.28      0.27      0.27     15091\n",
            "         3.0       0.30      0.32      0.31     25563\n",
            "         4.0       0.35      0.30      0.32     58262\n",
            "         5.0       0.77      0.74      0.76    180957\n",
            "\n",
            "    accuracy                           0.58    300000\n",
            "   macro avg       0.41      0.44      0.41    300000\n",
            "weighted avg       0.60      0.58      0.59    300000\n",
            "\n"
          ]
        }
      ],
      "source": [
        "print(classification_report(df_reviews[\"score\"], df_reviews[\"predicted_rating_BERT\"]))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The results are not so good. The accuracy is around 58% which is not surprising since the multilingual model has an average that is close to that value. \n",
        "\n",
        "However, in the description of the model, they propose an \"off by 1\" metric. In other terms, we classify a prediction to be accurate if it is off by 1 from the ground label.\n",
        "This metric can make sense in an application where we are not interested in the exact score, but a qualitative measure of positivity or negativity towards a certain book. This could be the case for example, in some applications of collaborative filtering where we are simply interested to know if the user consumed an item positively or negatively. \n",
        "\n",
        "Hence, we also use this metric here as well to evaluate the model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 80,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Id</th>\n",
              "      <th>Title</th>\n",
              "      <th>Price</th>\n",
              "      <th>User_id</th>\n",
              "      <th>profileName</th>\n",
              "      <th>helpfulness</th>\n",
              "      <th>score</th>\n",
              "      <th>time</th>\n",
              "      <th>summary</th>\n",
              "      <th>text</th>\n",
              "      <th>predicted_rating_BERT</th>\n",
              "      <th>off_score_1</th>\n",
              "      <th>off_score_2</th>\n",
              "      <th>off_score_3</th>\n",
              "      <th>off_score_4</th>\n",
              "      <th>off_score_5</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2524966</th>\n",
              "      <td>B000HZ8ZR8</td>\n",
              "      <td>The Garden of Eden, A Novel</td>\n",
              "      <td>0.00</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>4/11</td>\n",
              "      <td>2.0</td>\n",
              "      <td>912297600</td>\n",
              "      <td>The Garden of Eden? Hmmmm......</td>\n",
              "      <td>This book starts out great, it romantisizes Eu...</td>\n",
              "      <td>2</td>\n",
              "      <td>0.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>530830</th>\n",
              "      <td>B000HOSLEG</td>\n",
              "      <td>Fairytale</td>\n",
              "      <td>0.00</td>\n",
              "      <td>A2PGVEDRPGGIJ0</td>\n",
              "      <td>coteb</td>\n",
              "      <td>2/2</td>\n",
              "      <td>5.0</td>\n",
              "      <td>1338854400</td>\n",
              "      <td>loved it</td>\n",
              "      <td>I highly recommend if you like Maggie shayne. ...</td>\n",
              "      <td>5</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>5.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2597389</th>\n",
              "      <td>0754053822</td>\n",
              "      <td>Disgrace: Complete &amp; Unabridged</td>\n",
              "      <td>0.00</td>\n",
              "      <td>A1MRV6VAB1HVJW</td>\n",
              "      <td>Gisele W. Wright</td>\n",
              "      <td>0/4</td>\n",
              "      <td>3.0</td>\n",
              "      <td>1022803200</td>\n",
              "      <td>A hard read</td>\n",
              "      <td>What drew to this novel was the title and the ...</td>\n",
              "      <td>2</td>\n",
              "      <td>0.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>218539</th>\n",
              "      <td>0786222859</td>\n",
              "      <td>Bittersweet Rain</td>\n",
              "      <td>0.00</td>\n",
              "      <td>A21MVX83HPR5VK</td>\n",
              "      <td>Dina_179</td>\n",
              "      <td>17/22</td>\n",
              "      <td>4.0</td>\n",
              "      <td>952128000</td>\n",
              "      <td>Ms. Brown keeps surprising me with her amazing...</td>\n",
              "      <td>Only Sandra Brown will put together the most u...</td>\n",
              "      <td>5</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>5.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2759866</th>\n",
              "      <td>0821755641</td>\n",
              "      <td>Unforgettable</td>\n",
              "      <td>0.00</td>\n",
              "      <td>A1N967FZREGFG</td>\n",
              "      <td>Tess Capra</td>\n",
              "      <td>3/3</td>\n",
              "      <td>4.0</td>\n",
              "      <td>890524800</td>\n",
              "      <td>Not unforgettable, but pretty darn good.</td>\n",
              "      <td>Perhaps if Zebra spent as much on publicity as...</td>\n",
              "      <td>3</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1644252</th>\n",
              "      <td>0395615801</td>\n",
              "      <td>A Nightmare in History: The Holocaust 1933-1945</td>\n",
              "      <td>0.00</td>\n",
              "      <td>A1OJHNAMUW4F4Y</td>\n",
              "      <td>David N. 1st period.</td>\n",
              "      <td>2/5</td>\n",
              "      <td>5.0</td>\n",
              "      <td>1008028800</td>\n",
              "      <td>1st peiod reading -A nightmare in history</td>\n",
              "      <td>Imagne for a second that you are sleeping in y...</td>\n",
              "      <td>5</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>5.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2816988</th>\n",
              "      <td>1593555563</td>\n",
              "      <td>Stone of Tears (Sword of Truth Series)</td>\n",
              "      <td>167.25</td>\n",
              "      <td>A2JNZ09Y5JYPAK</td>\n",
              "      <td>I hate peas</td>\n",
              "      <td>2/5</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1325462400</td>\n",
              "      <td>Never should have been published</td>\n",
              "      <td>This book is absolutely terrible.If you take o...</td>\n",
              "      <td>1</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1088854</th>\n",
              "      <td>B000KIITWS</td>\n",
              "      <td>CONFUCIUS: THE SECULAR AS SACRED.</td>\n",
              "      <td>0.00</td>\n",
              "      <td>A3PBFMN1QU4VWV</td>\n",
              "      <td>John J. Gibbs</td>\n",
              "      <td>0/0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>1318636800</td>\n",
              "      <td>a gem</td>\n",
              "      <td>concise, well-written and constructed (as you ...</td>\n",
              "      <td>5</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>5.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>962254</th>\n",
              "      <td>0875968708</td>\n",
              "      <td>You Bet Your Tomatoes: Fun Facts, Tall Tales, ...</td>\n",
              "      <td>0.00</td>\n",
              "      <td>ASFNOYMHYMQNO</td>\n",
              "      <td>GLORIA O'CONNELL</td>\n",
              "      <td>1/1</td>\n",
              "      <td>4.0</td>\n",
              "      <td>1245974400</td>\n",
              "      <td>Entertaining and informative.</td>\n",
              "      <td>I've been gardening for many years (actually, ...</td>\n",
              "      <td>4</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2583708</th>\n",
              "      <td>B000H7EO2G</td>\n",
              "      <td>The Lord of the Rings Trilogy 3 Volumes</td>\n",
              "      <td>0.00</td>\n",
              "      <td>A1FIBTUKX80YNU</td>\n",
              "      <td>Brian F. Hudon</td>\n",
              "      <td>8/9</td>\n",
              "      <td>5.0</td>\n",
              "      <td>1148256000</td>\n",
              "      <td>One book to rule them all!</td>\n",
              "      <td>This is an exquisite book to own and to read, ...</td>\n",
              "      <td>5</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>5.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>300000 rows × 16 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "                 Id                                              Title  \\\n",
              "2524966  B000HZ8ZR8                        The Garden of Eden, A Novel   \n",
              "530830   B000HOSLEG                                          Fairytale   \n",
              "2597389  0754053822                    Disgrace: Complete & Unabridged   \n",
              "218539   0786222859                                   Bittersweet Rain   \n",
              "2759866  0821755641                                      Unforgettable   \n",
              "...             ...                                                ...   \n",
              "1644252  0395615801    A Nightmare in History: The Holocaust 1933-1945   \n",
              "2816988  1593555563             Stone of Tears (Sword of Truth Series)   \n",
              "1088854  B000KIITWS                  CONFUCIUS: THE SECULAR AS SACRED.   \n",
              "962254   0875968708  You Bet Your Tomatoes: Fun Facts, Tall Tales, ...   \n",
              "2583708  B000H7EO2G            The Lord of the Rings Trilogy 3 Volumes   \n",
              "\n",
              "          Price         User_id           profileName helpfulness  score  \\\n",
              "2524966    0.00               0                     0        4/11    2.0   \n",
              "530830     0.00  A2PGVEDRPGGIJ0                 coteb         2/2    5.0   \n",
              "2597389    0.00  A1MRV6VAB1HVJW      Gisele W. Wright         0/4    3.0   \n",
              "218539     0.00  A21MVX83HPR5VK              Dina_179       17/22    4.0   \n",
              "2759866    0.00   A1N967FZREGFG            Tess Capra         3/3    4.0   \n",
              "...         ...             ...                   ...         ...    ...   \n",
              "1644252    0.00  A1OJHNAMUW4F4Y  David N. 1st period.         2/5    5.0   \n",
              "2816988  167.25  A2JNZ09Y5JYPAK           I hate peas         2/5    1.0   \n",
              "1088854    0.00  A3PBFMN1QU4VWV         John J. Gibbs         0/0    5.0   \n",
              "962254     0.00   ASFNOYMHYMQNO      GLORIA O'CONNELL         1/1    4.0   \n",
              "2583708    0.00  A1FIBTUKX80YNU        Brian F. Hudon         8/9    5.0   \n",
              "\n",
              "               time                                            summary  \\\n",
              "2524966   912297600                    The Garden of Eden? Hmmmm......   \n",
              "530830   1338854400                                           loved it   \n",
              "2597389  1022803200                                        A hard read   \n",
              "218539    952128000  Ms. Brown keeps surprising me with her amazing...   \n",
              "2759866   890524800           Not unforgettable, but pretty darn good.   \n",
              "...             ...                                                ...   \n",
              "1644252  1008028800          1st peiod reading -A nightmare in history   \n",
              "2816988  1325462400                   Never should have been published   \n",
              "1088854  1318636800                                              a gem   \n",
              "962254   1245974400                      Entertaining and informative.   \n",
              "2583708  1148256000                         One book to rule them all!   \n",
              "\n",
              "                                                      text  \\\n",
              "2524966  This book starts out great, it romantisizes Eu...   \n",
              "530830   I highly recommend if you like Maggie shayne. ...   \n",
              "2597389  What drew to this novel was the title and the ...   \n",
              "218539   Only Sandra Brown will put together the most u...   \n",
              "2759866  Perhaps if Zebra spent as much on publicity as...   \n",
              "...                                                    ...   \n",
              "1644252  Imagne for a second that you are sleeping in y...   \n",
              "2816988  This book is absolutely terrible.If you take o...   \n",
              "1088854  concise, well-written and constructed (as you ...   \n",
              "962254   I've been gardening for many years (actually, ...   \n",
              "2583708  This is an exquisite book to own and to read, ...   \n",
              "\n",
              "         predicted_rating_BERT  off_score_1  off_score_2  off_score_3  \\\n",
              "2524966                      2          0.0          2.0          0.0   \n",
              "530830                       5          0.0          0.0          0.0   \n",
              "2597389                      2          0.0          2.0          3.0   \n",
              "218539                       5          0.0          0.0          0.0   \n",
              "2759866                      3          0.0          0.0          3.0   \n",
              "...                        ...          ...          ...          ...   \n",
              "1644252                      5          0.0          0.0          0.0   \n",
              "2816988                      1          1.0          0.0          0.0   \n",
              "1088854                      5          0.0          0.0          0.0   \n",
              "962254                       4          0.0          0.0          0.0   \n",
              "2583708                      5          0.0          0.0          0.0   \n",
              "\n",
              "         off_score_4  off_score_5  \n",
              "2524966          0.0          0.0  \n",
              "530830           0.0          5.0  \n",
              "2597389          0.0          0.0  \n",
              "218539           4.0          5.0  \n",
              "2759866          4.0          0.0  \n",
              "...              ...          ...  \n",
              "1644252          0.0          5.0  \n",
              "2816988          0.0          0.0  \n",
              "1088854          0.0          5.0  \n",
              "962254           4.0          0.0  \n",
              "2583708          0.0          5.0  \n",
              "\n",
              "[300000 rows x 16 columns]"
            ]
          },
          "execution_count": 80,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "for score in range(1,6):\n",
        "    df_reviews.loc[df_reviews[\"score\"] == score, \"off_score_\"+str(score)]= df_reviews[\"predicted_rating_BERT\"].apply(lambda x: score if x<=score+1 and x>=score-1 else 0.0)\n",
        "    df_reviews.loc[df_reviews[\"predicted_rating_BERT\"] == score, \"off_score_\"+str(score)] = score\n",
        "df_reviews.fillna(0, inplace = True)\n",
        "df_reviews"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 85,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Off by one metrics \n",
            "\n",
            "\n",
            "Class 1\n",
            "Precision:\t 0.3755330847839498\n",
            "Recall:\t\t 0.6956327321508422\n",
            "\n",
            "Class 2\n",
            "Precision:\t 0.5294090700870362\n",
            "Recall:\t\t 0.7658206878271817\n",
            "\n",
            "Class 3\n",
            "Precision:\t 0.4673034737455919\n",
            "Recall:\t\t 0.6583343113093142\n",
            "\n",
            "Class 4\n",
            "Precision:\t 0.6173135169401986\n",
            "Recall:\t\t 0.90254368198826\n",
            "\n",
            "Class 5\n",
            "Precision:\t 0.8017643352236925\n",
            "Recall:\t\t 0.878938090264538\n",
            "\n"
          ]
        }
      ],
      "source": [
        "print(\"Off by one metrics \\n\\n\")\n",
        "for score in range(1,6):\n",
        "    tp = (df_reviews[\"score\"] == df_reviews[\"off_score_\"+str(score)]).sum()\n",
        "    fp = ((df_reviews[\"score\"] != score) & (df_reviews[\"off_score_\"+str(score)]==score)).sum()\n",
        "    fn = ((df_reviews[\"score\"] == score) & (df_reviews[\"off_score_\"+str(score)]!=score)).sum()\n",
        "    print(f\"Class {score}\")\n",
        "    print(f\"Precision:\\t {tp/(tp+fp)}\")\n",
        "    print(f\"Recall:\\t\\t {tp/(tp+fn)}\\n\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "As we can see, with this metric, the per class precision and recall is much higher. However, one could object that the off by one metric is unrealiable as it is simply increasing the number of true positives (and keeping the false positives and false negatives the same). For this reason, we shall take this with a grain of salt and prefer to stick to the normal classification report. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## RoBERTa\n",
        "\n",
        "One interesting alternative to the multilingual model, is RoBERTa. This model is trained on a wider variety of data sets and also gives the ability of fine tuning which is a desirable quality. However, the first limitation of this model is that it only handles English. \n",
        "Therefore, we must first keep only the books that are in english using spacy fast lang as before. This was done previously.\n",
        "\n",
        "A quick analysis shows that approximately 15000 of the documents are not in english while a quick inspection showed that this is not so accurate. Therefore, just to be sure, we proceed to reclassify them using the text of the review instead of the summary. In this way, we are not throwing away data which could be useful."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 32,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Documents with score of 3.0: 25546\n",
            "We are left with 273958 documents\n"
          ]
        }
      ],
      "source": [
        "nonenglish = df_reviews[df_reviews[\"lang\"]!=\"en\"][\"text\"]\n",
        "lang = [nlp(doc)._.language for doc in tqdm(nonenglish)]\n",
        "df_reviews.loc[df_reviews[\"lang\"]!=\"en\", \"lang\"] = lang\n",
        "\n",
        "#remove non english docs\n",
        "df_reviews_english = df_reviews[df_reviews[\"lang\"]==\"en\"]\n",
        "\n",
        "len_3 = len( df_reviews_english[df_reviews_english[\"score\"]==3.0])\n",
        "print(f\"Documents with score of 3.0: { len_3}\")\n",
        "\n",
        "#remove docs with score of 3.0\n",
        "df_reviews_english = df_reviews_english[df_reviews_english[\"score\"]!=3.0]\n",
        "\n",
        "print(f\"We are left with {len(df_reviews_english)} documents\")\n",
        "\n",
        "df_reviews_english.loc[df_reviews_english[\"score\"]>3.0, \"class\"] = \"POSITIVE\"\n",
        "df_reviews_english.loc[df_reviews_english[\"score\"]<3.0, \"class\"] = \"NEGATIVE\"\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Now the non english documents have decreased to only 500 documents, which we drop. \n",
        "\n",
        "\n",
        "The next step involves handling another limitation of the model, namely that it only classifies text as positive or negative. As we analyzed previously, we can easily group 1 & 2 stars as negative and 4 & 5 stars as positive, while for 3 stars it is not so clear whether they are one category or the other. This could be harmful for evaluating the models performance as we have no prior idea of the true labels. For this reason, we will drop 3 star reviews when we evaluate the performance. \n",
        "\n",
        "This means dropping 25500 reviews. \n",
        "\n",
        "In the end, we are left with more than 250000 documents, which is acceptable to measure a comparable performance. Unfortunately, the run time required for this is 12 hours, therefore, we choose to consider only a sample of 100 thousand documents, which requires only 5 hours. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 27,
      "metadata": {},
      "outputs": [],
      "source": [
        "model_name = \"siebert/sentiment-roberta-large-english\"\n",
        "\n",
        "tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
        "model = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels = 2)\n",
        "classifier = pipeline(\"sentiment-analysis\", model=model, tokenizer=tokenizer)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {},
      "outputs": [],
      "source": [
        "if FIRST_RUN:\n",
        "    df_reviews_english = df_reviews_english.sample(100000, random_state=2)\n",
        "    tqdm.pandas()\n",
        "    df_reviews_english[\"predicted_rating_RoBERTa\"] = df_reviews_english[\"summary\"].progress_apply(lambda x: str(classifier(x)[0]['label']))\n",
        "    with open(\"./DataSet/reviews_small_roberta.csv\", \"wb\") as fp:\n",
        "        pickle.dump(df_reviews_english,fp)\n",
        "else:\n",
        "    with open(\"./DataSet/reviews_small_roberta.csv\", \"rb\") as fp:\n",
        "        df_reviews_english = pickle.load(fp)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "application/vnd.plotly.v1+json": {
              "config": {
                "plotlyServerURL": "https://plot.ly"
              },
              "data": [
                {
                  "alignmentgroup": "True",
                  "hovertemplate": "variable=class<br>count=%{x}<br>=%{y}<extra></extra>",
                  "legendgroup": "class",
                  "marker": {
                    "color": "#636efa",
                    "pattern": {
                      "shape": ""
                    }
                  },
                  "name": "class",
                  "offsetgroup": "class",
                  "orientation": "h",
                  "showlegend": true,
                  "textposition": "auto",
                  "type": "bar",
                  "x": [
                    87280,
                    12720
                  ],
                  "xaxis": "x",
                  "y": [
                    "POSITIVE",
                    "NEGATIVE"
                  ],
                  "yaxis": "y"
                }
              ],
              "layout": {
                "barmode": "relative",
                "legend": {
                  "title": {
                    "text": "variable"
                  },
                  "tracegroupgap": 0
                },
                "template": {
                  "data": {
                    "bar": [
                      {
                        "error_x": {
                          "color": "#2a3f5f"
                        },
                        "error_y": {
                          "color": "#2a3f5f"
                        },
                        "marker": {
                          "line": {
                            "color": "#E5ECF6",
                            "width": 0.5
                          },
                          "pattern": {
                            "fillmode": "overlay",
                            "size": 10,
                            "solidity": 0.2
                          }
                        },
                        "type": "bar"
                      }
                    ],
                    "barpolar": [
                      {
                        "marker": {
                          "line": {
                            "color": "#E5ECF6",
                            "width": 0.5
                          },
                          "pattern": {
                            "fillmode": "overlay",
                            "size": 10,
                            "solidity": 0.2
                          }
                        },
                        "type": "barpolar"
                      }
                    ],
                    "carpet": [
                      {
                        "aaxis": {
                          "endlinecolor": "#2a3f5f",
                          "gridcolor": "white",
                          "linecolor": "white",
                          "minorgridcolor": "white",
                          "startlinecolor": "#2a3f5f"
                        },
                        "baxis": {
                          "endlinecolor": "#2a3f5f",
                          "gridcolor": "white",
                          "linecolor": "white",
                          "minorgridcolor": "white",
                          "startlinecolor": "#2a3f5f"
                        },
                        "type": "carpet"
                      }
                    ],
                    "choropleth": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "type": "choropleth"
                      }
                    ],
                    "contour": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "colorscale": [
                          [
                            0,
                            "#0d0887"
                          ],
                          [
                            0.1111111111111111,
                            "#46039f"
                          ],
                          [
                            0.2222222222222222,
                            "#7201a8"
                          ],
                          [
                            0.3333333333333333,
                            "#9c179e"
                          ],
                          [
                            0.4444444444444444,
                            "#bd3786"
                          ],
                          [
                            0.5555555555555556,
                            "#d8576b"
                          ],
                          [
                            0.6666666666666666,
                            "#ed7953"
                          ],
                          [
                            0.7777777777777778,
                            "#fb9f3a"
                          ],
                          [
                            0.8888888888888888,
                            "#fdca26"
                          ],
                          [
                            1,
                            "#f0f921"
                          ]
                        ],
                        "type": "contour"
                      }
                    ],
                    "contourcarpet": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "type": "contourcarpet"
                      }
                    ],
                    "heatmap": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "colorscale": [
                          [
                            0,
                            "#0d0887"
                          ],
                          [
                            0.1111111111111111,
                            "#46039f"
                          ],
                          [
                            0.2222222222222222,
                            "#7201a8"
                          ],
                          [
                            0.3333333333333333,
                            "#9c179e"
                          ],
                          [
                            0.4444444444444444,
                            "#bd3786"
                          ],
                          [
                            0.5555555555555556,
                            "#d8576b"
                          ],
                          [
                            0.6666666666666666,
                            "#ed7953"
                          ],
                          [
                            0.7777777777777778,
                            "#fb9f3a"
                          ],
                          [
                            0.8888888888888888,
                            "#fdca26"
                          ],
                          [
                            1,
                            "#f0f921"
                          ]
                        ],
                        "type": "heatmap"
                      }
                    ],
                    "heatmapgl": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "colorscale": [
                          [
                            0,
                            "#0d0887"
                          ],
                          [
                            0.1111111111111111,
                            "#46039f"
                          ],
                          [
                            0.2222222222222222,
                            "#7201a8"
                          ],
                          [
                            0.3333333333333333,
                            "#9c179e"
                          ],
                          [
                            0.4444444444444444,
                            "#bd3786"
                          ],
                          [
                            0.5555555555555556,
                            "#d8576b"
                          ],
                          [
                            0.6666666666666666,
                            "#ed7953"
                          ],
                          [
                            0.7777777777777778,
                            "#fb9f3a"
                          ],
                          [
                            0.8888888888888888,
                            "#fdca26"
                          ],
                          [
                            1,
                            "#f0f921"
                          ]
                        ],
                        "type": "heatmapgl"
                      }
                    ],
                    "histogram": [
                      {
                        "marker": {
                          "pattern": {
                            "fillmode": "overlay",
                            "size": 10,
                            "solidity": 0.2
                          }
                        },
                        "type": "histogram"
                      }
                    ],
                    "histogram2d": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "colorscale": [
                          [
                            0,
                            "#0d0887"
                          ],
                          [
                            0.1111111111111111,
                            "#46039f"
                          ],
                          [
                            0.2222222222222222,
                            "#7201a8"
                          ],
                          [
                            0.3333333333333333,
                            "#9c179e"
                          ],
                          [
                            0.4444444444444444,
                            "#bd3786"
                          ],
                          [
                            0.5555555555555556,
                            "#d8576b"
                          ],
                          [
                            0.6666666666666666,
                            "#ed7953"
                          ],
                          [
                            0.7777777777777778,
                            "#fb9f3a"
                          ],
                          [
                            0.8888888888888888,
                            "#fdca26"
                          ],
                          [
                            1,
                            "#f0f921"
                          ]
                        ],
                        "type": "histogram2d"
                      }
                    ],
                    "histogram2dcontour": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "colorscale": [
                          [
                            0,
                            "#0d0887"
                          ],
                          [
                            0.1111111111111111,
                            "#46039f"
                          ],
                          [
                            0.2222222222222222,
                            "#7201a8"
                          ],
                          [
                            0.3333333333333333,
                            "#9c179e"
                          ],
                          [
                            0.4444444444444444,
                            "#bd3786"
                          ],
                          [
                            0.5555555555555556,
                            "#d8576b"
                          ],
                          [
                            0.6666666666666666,
                            "#ed7953"
                          ],
                          [
                            0.7777777777777778,
                            "#fb9f3a"
                          ],
                          [
                            0.8888888888888888,
                            "#fdca26"
                          ],
                          [
                            1,
                            "#f0f921"
                          ]
                        ],
                        "type": "histogram2dcontour"
                      }
                    ],
                    "mesh3d": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "type": "mesh3d"
                      }
                    ],
                    "parcoords": [
                      {
                        "line": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "parcoords"
                      }
                    ],
                    "pie": [
                      {
                        "automargin": true,
                        "type": "pie"
                      }
                    ],
                    "scatter": [
                      {
                        "fillpattern": {
                          "fillmode": "overlay",
                          "size": 10,
                          "solidity": 0.2
                        },
                        "type": "scatter"
                      }
                    ],
                    "scatter3d": [
                      {
                        "line": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scatter3d"
                      }
                    ],
                    "scattercarpet": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scattercarpet"
                      }
                    ],
                    "scattergeo": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scattergeo"
                      }
                    ],
                    "scattergl": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scattergl"
                      }
                    ],
                    "scattermapbox": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scattermapbox"
                      }
                    ],
                    "scatterpolar": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scatterpolar"
                      }
                    ],
                    "scatterpolargl": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scatterpolargl"
                      }
                    ],
                    "scatterternary": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scatterternary"
                      }
                    ],
                    "surface": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "colorscale": [
                          [
                            0,
                            "#0d0887"
                          ],
                          [
                            0.1111111111111111,
                            "#46039f"
                          ],
                          [
                            0.2222222222222222,
                            "#7201a8"
                          ],
                          [
                            0.3333333333333333,
                            "#9c179e"
                          ],
                          [
                            0.4444444444444444,
                            "#bd3786"
                          ],
                          [
                            0.5555555555555556,
                            "#d8576b"
                          ],
                          [
                            0.6666666666666666,
                            "#ed7953"
                          ],
                          [
                            0.7777777777777778,
                            "#fb9f3a"
                          ],
                          [
                            0.8888888888888888,
                            "#fdca26"
                          ],
                          [
                            1,
                            "#f0f921"
                          ]
                        ],
                        "type": "surface"
                      }
                    ],
                    "table": [
                      {
                        "cells": {
                          "fill": {
                            "color": "#EBF0F8"
                          },
                          "line": {
                            "color": "white"
                          }
                        },
                        "header": {
                          "fill": {
                            "color": "#C8D4E3"
                          },
                          "line": {
                            "color": "white"
                          }
                        },
                        "type": "table"
                      }
                    ]
                  },
                  "layout": {
                    "annotationdefaults": {
                      "arrowcolor": "#2a3f5f",
                      "arrowhead": 0,
                      "arrowwidth": 1
                    },
                    "autotypenumbers": "strict",
                    "coloraxis": {
                      "colorbar": {
                        "outlinewidth": 0,
                        "ticks": ""
                      }
                    },
                    "colorscale": {
                      "diverging": [
                        [
                          0,
                          "#8e0152"
                        ],
                        [
                          0.1,
                          "#c51b7d"
                        ],
                        [
                          0.2,
                          "#de77ae"
                        ],
                        [
                          0.3,
                          "#f1b6da"
                        ],
                        [
                          0.4,
                          "#fde0ef"
                        ],
                        [
                          0.5,
                          "#f7f7f7"
                        ],
                        [
                          0.6,
                          "#e6f5d0"
                        ],
                        [
                          0.7,
                          "#b8e186"
                        ],
                        [
                          0.8,
                          "#7fbc41"
                        ],
                        [
                          0.9,
                          "#4d9221"
                        ],
                        [
                          1,
                          "#276419"
                        ]
                      ],
                      "sequential": [
                        [
                          0,
                          "#0d0887"
                        ],
                        [
                          0.1111111111111111,
                          "#46039f"
                        ],
                        [
                          0.2222222222222222,
                          "#7201a8"
                        ],
                        [
                          0.3333333333333333,
                          "#9c179e"
                        ],
                        [
                          0.4444444444444444,
                          "#bd3786"
                        ],
                        [
                          0.5555555555555556,
                          "#d8576b"
                        ],
                        [
                          0.6666666666666666,
                          "#ed7953"
                        ],
                        [
                          0.7777777777777778,
                          "#fb9f3a"
                        ],
                        [
                          0.8888888888888888,
                          "#fdca26"
                        ],
                        [
                          1,
                          "#f0f921"
                        ]
                      ],
                      "sequentialminus": [
                        [
                          0,
                          "#0d0887"
                        ],
                        [
                          0.1111111111111111,
                          "#46039f"
                        ],
                        [
                          0.2222222222222222,
                          "#7201a8"
                        ],
                        [
                          0.3333333333333333,
                          "#9c179e"
                        ],
                        [
                          0.4444444444444444,
                          "#bd3786"
                        ],
                        [
                          0.5555555555555556,
                          "#d8576b"
                        ],
                        [
                          0.6666666666666666,
                          "#ed7953"
                        ],
                        [
                          0.7777777777777778,
                          "#fb9f3a"
                        ],
                        [
                          0.8888888888888888,
                          "#fdca26"
                        ],
                        [
                          1,
                          "#f0f921"
                        ]
                      ]
                    },
                    "colorway": [
                      "#636efa",
                      "#EF553B",
                      "#00cc96",
                      "#ab63fa",
                      "#FFA15A",
                      "#19d3f3",
                      "#FF6692",
                      "#B6E880",
                      "#FF97FF",
                      "#FECB52"
                    ],
                    "font": {
                      "color": "#2a3f5f"
                    },
                    "geo": {
                      "bgcolor": "white",
                      "lakecolor": "white",
                      "landcolor": "#E5ECF6",
                      "showlakes": true,
                      "showland": true,
                      "subunitcolor": "white"
                    },
                    "hoverlabel": {
                      "align": "left"
                    },
                    "hovermode": "closest",
                    "mapbox": {
                      "style": "light"
                    },
                    "paper_bgcolor": "white",
                    "plot_bgcolor": "#E5ECF6",
                    "polar": {
                      "angularaxis": {
                        "gridcolor": "white",
                        "linecolor": "white",
                        "ticks": ""
                      },
                      "bgcolor": "#E5ECF6",
                      "radialaxis": {
                        "gridcolor": "white",
                        "linecolor": "white",
                        "ticks": ""
                      }
                    },
                    "scene": {
                      "xaxis": {
                        "backgroundcolor": "#E5ECF6",
                        "gridcolor": "white",
                        "gridwidth": 2,
                        "linecolor": "white",
                        "showbackground": true,
                        "ticks": "",
                        "zerolinecolor": "white"
                      },
                      "yaxis": {
                        "backgroundcolor": "#E5ECF6",
                        "gridcolor": "white",
                        "gridwidth": 2,
                        "linecolor": "white",
                        "showbackground": true,
                        "ticks": "",
                        "zerolinecolor": "white"
                      },
                      "zaxis": {
                        "backgroundcolor": "#E5ECF6",
                        "gridcolor": "white",
                        "gridwidth": 2,
                        "linecolor": "white",
                        "showbackground": true,
                        "ticks": "",
                        "zerolinecolor": "white"
                      }
                    },
                    "shapedefaults": {
                      "line": {
                        "color": "#2a3f5f"
                      }
                    },
                    "ternary": {
                      "aaxis": {
                        "gridcolor": "white",
                        "linecolor": "white",
                        "ticks": ""
                      },
                      "baxis": {
                        "gridcolor": "white",
                        "linecolor": "white",
                        "ticks": ""
                      },
                      "bgcolor": "#E5ECF6",
                      "caxis": {
                        "gridcolor": "white",
                        "linecolor": "white",
                        "ticks": ""
                      }
                    },
                    "title": {
                      "x": 0.05
                    },
                    "xaxis": {
                      "automargin": true,
                      "gridcolor": "white",
                      "linecolor": "white",
                      "ticks": "",
                      "title": {
                        "standoff": 15
                      },
                      "zerolinecolor": "white",
                      "zerolinewidth": 2
                    },
                    "yaxis": {
                      "automargin": true,
                      "gridcolor": "white",
                      "linecolor": "white",
                      "ticks": "",
                      "title": {
                        "standoff": 15
                      },
                      "zerolinecolor": "white",
                      "zerolinewidth": 2
                    }
                  }
                },
                "title": {
                  "text": "Distribution of positive vs negative"
                },
                "xaxis": {
                  "anchor": "y",
                  "domain": [
                    0,
                    1
                  ],
                  "title": {
                    "text": "count"
                  }
                },
                "yaxis": {
                  "anchor": "x",
                  "domain": [
                    0,
                    1
                  ],
                  "title": {
                    "text": ""
                  }
                }
              }
            }
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "fig = px.bar(df_reviews_english[\"class\"].value_counts(), orientation = \"h\", title = f\"Distribution of positive vs negative\", labels = {\"index\":\"\", \"value\": \"count\"})\n",
        "fig.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "    NEGATIVE       0.51      0.82      0.63     12720\n",
            "    POSITIVE       0.97      0.89      0.93     87280\n",
            "\n",
            "    accuracy                           0.88    100000\n",
            "   macro avg       0.74      0.85      0.78    100000\n",
            "weighted avg       0.91      0.88      0.89    100000\n",
            "\n"
          ]
        }
      ],
      "source": [
        "print(classification_report(df_reviews_english[\"class\"], df_reviews_english[\"predicted_rating_RoBERTa\"]))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "As we can see, this model performs much better than the previous one. It has high values of recall for both classes and a high precision for the positive class. However, it has a precision of 50% for the negative class, which indicates that half of the time, the model is classifying positive reviews as negative ones. This is largely due to the imbalance of the classes since the missing 10% of the recall in the positive class corresponds to around 8500 documents which are positive but classified as negative. However, the size of this misclassification is pratically as large as the number of negative documents, which will cause its precision to inevitably go down. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Fine Tuning RoBERTa\n",
        "\n",
        "As mentioned before, we can fine tune this model for our own dataset to try to obtain better results.\n",
        "\n",
        "In particular, we follow the approach outlined in:\n",
        "https://huggingface.co/docs/transformers/training"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Step 1: Prepare Data\n",
        "In the tutorial, we see that we must have data in HuggingFace format. This can be achieved by uploading the train,validation,test sets into the HUB of huggingface (account needed). \n",
        "\n",
        "The strategy was as follows: \n",
        "- Drop all columns except class and summary (to store less data)\n",
        "- Convert Positive - 1, Negative - 0\n",
        "- Shuffle Data\n",
        "- Split train, validation, test into 60%, 20%, 20%.\n",
        "- Upoad the results into the HUB.\n",
        "- Load using the appropriate library."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {},
      "outputs": [],
      "source": [
        "df_reviews_english[\"class\"] = df_reviews_english[\"class\"].apply(lambda x: 1 if x == \"POSITIVE\" else 0)\n",
        "df_reviews_english = df_reviews_english.drop([\"Id\", \"Title\", \"score\", \"Price\", \"User_id\", \"profileName\", \"helpfulness\", \"time\", \"text\", \"predicted_rating_BERT\", \"lang\", \"predicted_rating_RoBERTa\"], axis = 1)\n",
        "df_reviews_english.rename(columns = {\"class\":\"labels\",\"summary\":\"text\"},inplace = True)\n",
        "lendf = len(df_reviews_english)\n",
        "train, validation, test = np.split(df_reviews_english.sample(frac = 1, random_state = 2), [int(0.6*lendf), int(0.8*lendf)])\n",
        "train.to_csv(\"./DataSet/train.csv\", index = False)\n",
        "validation.to_csv(\"./DataSet/validation.csv\", index = False)\n",
        "test.to_csv(\"./DataSet/test.csv\", index = False)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {},
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Using custom data configuration theabm--NLPFinalProject-c2fd898ba956fdb1\n",
            "Found cached dataset csv (C:/Users/andre/.cache/huggingface/datasets/theabm___csv/theabm--NLPFinalProject-c2fd898ba956fdb1/0.0.0/6b34fb8fcf56f7c8ba51dc895bfa2bfbe43546f190a60fcf74bb5e8afdcc2317)\n",
            "100%|██████████| 3/3 [00:00<00:00, 460.98it/s]\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "DatasetDict({\n",
              "    train: Dataset({\n",
              "        features: ['text', 'labels'],\n",
              "        num_rows: 60000\n",
              "    })\n",
              "    test: Dataset({\n",
              "        features: ['text', 'labels'],\n",
              "        num_rows: 20000\n",
              "    })\n",
              "    validation: Dataset({\n",
              "        features: ['text', 'labels'],\n",
              "        num_rows: 20000\n",
              "    })\n",
              "})"
            ]
          },
          "execution_count": 10,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "from datasets import load_dataset\n",
        "dataset = load_dataset(\"theabm/NLPFinalProject\")\n",
        "dataset\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "As we can see, the dataset is composed of a DataDict which is itself composed of the splits. We can access the data in the following way:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'text': 'A Perfect Slice of History', 'labels': 1}"
            ]
          },
          "execution_count": 20,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "dataset[\"train\"][0]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Step 2: Tokenizer\n",
        "\n",
        "We use the same tokenizer as above to handle the inputs."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {},
      "outputs": [],
      "source": [
        "model_name = \"siebert/sentiment-roberta-large-english\"\n",
        "\n",
        "tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
        "model = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels = 2)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {},
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Parameter 'function'=<function tokenize_function at 0x000001EDE1AD45E0> of the transform datasets.arrow_dataset.Dataset._map_single couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything. This warning is only showed once. Subsequent hashing failures won't be showed.\n",
            " 98%|█████████▊| 59/60 [00:01<00:00, 39.63ba/s]\n",
            " 95%|█████████▌| 19/20 [00:00<00:00, 36.48ba/s]\n",
            " 95%|█████████▌| 19/20 [00:00<00:00, 49.05ba/s]\n"
          ]
        }
      ],
      "source": [
        "def tokenize_function(examples):\n",
        "    return tokenizer(examples[\"text\"], truncation=True)\n",
        "\n",
        "\n",
        "tokenized_datasets = dataset.map(tokenize_function, batched=True)\n",
        "data_collator = DataCollatorWithPadding(tokenizer)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "Dataset({\n",
              "    features: ['text', 'labels', 'input_ids', 'attention_mask'],\n",
              "    num_rows: 60000\n",
              "})"
            ]
          },
          "execution_count": 6,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "tokenized_datasets[\"train\"]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {},
      "outputs": [],
      "source": [
        "training_args = TrainingArguments(\n",
        "    output_dir=\"./test_trainer\",\n",
        "    evaluation_strategy = \"epoch\",\n",
        "    learning_rate=2e-5,\n",
        "    num_train_epochs=5,\n",
        "    warmup_steps=500,\n",
        "    weight_decay=0.01,\n",
        "    eval_accumulation_steps=100,\n",
        "    per_device_eval_batch_size=32,\n",
        "    per_device_train_batch_size=32,\n",
        "    group_by_length=True,\n",
        "    save_strategy = \"epoch\"\n",
        ")\n",
        "    \n",
        "\n",
        "clf_metrics = evaluate.combine([\"accuracy\", \"f1\", \"precision\", \"recall\"])\n",
        "\n",
        "def compute_metrics(eval_pred):\n",
        "    logits, labels = eval_pred\n",
        "    predictions = np.argmax(logits, axis=-1)\n",
        "    return clf_metrics.compute(predictions=predictions, references=labels)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {},
      "outputs": [],
      "source": [
        "trainer = Trainer(\n",
        "    model = model,\n",
        "    args = training_args,\n",
        "    tokenizer= tokenizer,\n",
        "    train_dataset = tokenized_datasets[\"train\"],\n",
        "    eval_dataset = tokenized_datasets[\"validation\"],\n",
        "    compute_metrics = compute_metrics,\n",
        "    data_collator=data_collator\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "#requires a decent amount of memory in gpu. Settings above were used in Colab and Finished in an hour of training\n",
        "#trainer.train()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "![image](./Images/finetuning.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We fine tuned the model for 5 epochs and saved the model at each of these. As we can see, the training loss is quickly decreasing while the validation loss is increasing a lot towards the end. This indicates we are in a situation of overfitting. For the first epoch the validation loss is at a minimum, so we will use the weights of the model at this epoch. \n",
        "\n",
        "A fine grained analysis could save the model weights every 1000 steps and have a better understanding of the models behaviour. We did not do this for computational reasons. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Results\n",
        "\n",
        "We evaluate the fine tuned model on the unseen test set, in order to compare the performances."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {},
      "outputs": [],
      "source": [
        "path_to_model = \"C:\\\\Users\\\\andre\\Documents\\\\NLP\\\\NLPFinalProject\\\\FineTuningCheckpoint\\\\checkpoint-1875\"\n",
        "model = AutoModelForSequenceClassification.from_pretrained(path_to_model)\n",
        "tokenizer = AutoTokenizer.from_pretrained(path_to_model)\n",
        "ft_clf = pipeline(\"sentiment-analysis\", model=model, tokenizer=tokenizer, device = 0)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from transformers.pipelines.pt_utils import KeyDataset\n",
        "from tqdm.auto import tqdm\n",
        "\n",
        "dataset = load_dataset(\"theabm/NLPFinalProject\", split = \"test\")\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {},
      "outputs": [],
      "source": [
        "if FIRST_RUN:\n",
        "    predicted = []\n",
        "    for out in tqdm(ft_clf(KeyDataset(dataset, \"text\"))):\n",
        "        predicted.append(str(out['label']))\n",
        "    data = {\"trueLabels\": dataset[\"labels\"],\n",
        "            \"predictedLabels\": predicted\n",
        "        }\n",
        "    df = pd.DataFrame(data)\n",
        "    df[\"predictedLabels\"] = df[\"predictedLabels\"].apply(lambda x: 1 if x == \"POSITIVE\" else 0)\n",
        "    df.to_csv(\"./DataSet/ft_predicted_labels.csv\", index = False)\n",
        "else:\n",
        "    predicted = pd.read_csv(\"./DataSet/ft_predicted_labels.csv\")\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 31,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.77      0.70      0.73      2561\n",
            "           1       0.96      0.97      0.96     17439\n",
            "\n",
            "    accuracy                           0.93     20000\n",
            "   macro avg       0.86      0.83      0.85     20000\n",
            "weighted avg       0.93      0.93      0.93     20000\n",
            "\n"
          ]
        }
      ],
      "source": [
        "print(classification_report(predicted[\"trueLabels\"], predicted[\"predictedLabels\"]))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "As we can see, the model has noticeably improved compared to before. We have higher values for precision (51% -> 71%) and f1-score (63% -> 73%) for the negative class, and higher recall (89%->97%), and f1(93%->96%) score for the positive class. Furthermore, the accuracy also increased from 88% to 93%. \n",
        "\n",
        "The precision for the positive class decreased slightly (97% -> 96%) and the recall droped considerably (82% -> 70%).\n",
        "\n",
        "However, the performance metrics indicate and **overall** improvement and are more desirable for an all purpose application. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Baselines\n",
        "\n",
        "We now have to compare these models against a baseline in order to understand if our model makes sense. In particular, we will use the most frequent baseline. \n",
        "\n",
        "We will only compare the fine tuned RoBERTa model and the baseline, since multilingual bert behaved poorly and the proposed application only requires to obtain positive or negative feedback in the most basic form to be incorporated into a recommender system. \n",
        "\n",
        "In particular, we will use the f1 score weighted, micro, and macro which each try to account differently for the imbalance in the classes."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Balanced accuracy baseline: 0.5\n",
            "balanced accuracy RoBERTa: 0.8537024626878578\n",
            "balanced accuracy RoBERTa FT: 0.8333527349272734\n",
            "\n",
            "f1-weighted baseline: 0.8123046048238468\n",
            "f1-weighted RoBERTa: 0.8897823494948959\n",
            "f1-weighted RoBERTa FT: 0.9331138209427435\n",
            "\n",
            "f1-micro baseline: 0.87195\n",
            "f1-micro RoBERTa: 0.87862\n",
            "f1-micro RoBERTa FT: 0.9345\n",
            "\n",
            "f1-macro baseline: 0.4657976975880766\n",
            "f1-macro RoBERTa: 0.7797817179406492\n",
            "f1-macro RoBERTa FT: 0.8471822708849865\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# most frequent\n",
        "\n",
        "# we simply predict a 5 for all documents\n",
        "\n",
        "predicted_most_frequent = np.ones(len(predicted))\n",
        "\n",
        "balanced_acc_baseline = balanced_accuracy_score(predicted[\"trueLabels\"], predicted_most_frequent)\n",
        "balanced_acc_roberta = balanced_accuracy_score(df_reviews_english[\"class\"], df_reviews_english[\"predicted_rating_RoBERTa\"])\n",
        "balanced_acc_roberta_ft = balanced_accuracy_score(predicted[\"trueLabels\"], predicted[\"predictedLabels\"])\n",
        "\n",
        "f1_weighted_baseline = f1_score(predicted[\"trueLabels\"], predicted_most_frequent, average = \"weighted\")\n",
        "f1_weighted_roberta = f1_score(df_reviews_english[\"class\"], df_reviews_english[\"predicted_rating_RoBERTa\"], average = \"weighted\")\n",
        "f1_weighted_roberta_ft = f1_score(predicted[\"trueLabels\"], predicted[\"predictedLabels\"], average = \"weighted\")\n",
        "\n",
        "f1_micro_baseline = f1_score(predicted[\"trueLabels\"], predicted_most_frequent, average = \"micro\")\n",
        "f1_micro_roberta = f1_score(df_reviews_english[\"class\"], df_reviews_english[\"predicted_rating_RoBERTa\"], average = \"micro\")\n",
        "f1_micro_roberta_ft = f1_score(predicted[\"trueLabels\"], predicted[\"predictedLabels\"], average = \"micro\")\n",
        "\n",
        "f1_macro_baseline = f1_score(predicted[\"trueLabels\"], predicted_most_frequent, average = \"macro\")\n",
        "f1_macro_roberta = f1_score(df_reviews_english[\"class\"], df_reviews_english[\"predicted_rating_RoBERTa\"], average = \"macro\")\n",
        "f1_macro_roberta_ft = f1_score(predicted[\"trueLabels\"], predicted[\"predictedLabels\"], average = \"macro\")\n",
        "\n",
        "\n",
        "print(f\"Balanced accuracy baseline: {balanced_acc_baseline}\\nbalanced accuracy RoBERTa: {balanced_acc_roberta}\\nbalanced accuracy RoBERTa FT: {balanced_acc_roberta_ft}\\n\")\n",
        "print(f\"f1-weighted baseline: {f1_weighted_baseline}\\nf1-weighted RoBERTa: {f1_weighted_roberta}\\nf1-weighted RoBERTa FT: {f1_weighted_roberta_ft}\\n\")\n",
        "print(f\"f1-micro baseline: {f1_micro_baseline}\\nf1-micro RoBERTa: {f1_micro_roberta}\\nf1-micro RoBERTa FT: {f1_micro_roberta_ft}\\n\")\n",
        "print(f\"f1-macro baseline: {f1_macro_baseline}\\nf1-macro RoBERTa: {f1_macro_roberta}\\nf1-macro RoBERTa FT: {f1_macro_roberta_ft}\\n\")\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Baseline</th>\n",
              "      <th>RoBERTa</th>\n",
              "      <th>RoBERTa FT</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>balanced accuracy</th>\n",
              "      <td>0.50</td>\n",
              "      <td>0.85</td>\n",
              "      <td>0.83</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>f1-weighted</th>\n",
              "      <td>0.81</td>\n",
              "      <td>0.89</td>\n",
              "      <td>0.93</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>f1-micro</th>\n",
              "      <td>0.87</td>\n",
              "      <td>0.88</td>\n",
              "      <td>0.93</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>f1-macro</th>\n",
              "      <td>0.47</td>\n",
              "      <td>0.78</td>\n",
              "      <td>0.85</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                   Baseline  RoBERTa  RoBERTa FT\n",
              "balanced accuracy      0.50     0.85        0.83\n",
              "f1-weighted            0.81     0.89        0.93\n",
              "f1-micro               0.87     0.88        0.93\n",
              "f1-macro               0.47     0.78        0.85"
            ]
          },
          "execution_count": 10,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "metrics = {\n",
        "    \"balanced accuracy\": [0.5, 0.85, 0.83],\n",
        "    \"f1-weighted\": [0.81, 0.89, 0.93],\n",
        "    \"f1-micro\": [0.87, 0.88, 0.93],\n",
        "    \"f1-macro\": [0.47, 0.78, 0.85]\n",
        "}\n",
        "metrics = pd.DataFrame(metrics, index = [\"Baseline\", \"RoBERTa\", \"RoBERTa FT\"]).transpose()\n",
        "metrics"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "application/vnd.plotly.v1+json": {
              "config": {
                "plotlyServerURL": "https://plot.ly"
              },
              "data": [
                {
                  "marker": {
                    "color": "rgb(27,158,119)"
                  },
                  "name": "Baseline",
                  "orientation": "h",
                  "text": [
                    0.5,
                    0.81,
                    0.87,
                    0.47
                  ],
                  "textposition": "outside",
                  "type": "bar",
                  "x": [
                    0.5,
                    0.81,
                    0.87,
                    0.47
                  ],
                  "y": [
                    "balanced accuracy",
                    "f1-weighted",
                    "f1-micro",
                    "f1-macro"
                  ]
                },
                {
                  "marker": {
                    "color": "rgb(217,95,2)"
                  },
                  "name": "RoBERTa",
                  "orientation": "h",
                  "text": [
                    0.85,
                    0.89,
                    0.88,
                    0.78
                  ],
                  "textposition": "outside",
                  "type": "bar",
                  "x": [
                    0.85,
                    0.89,
                    0.88,
                    0.78
                  ],
                  "y": [
                    "balanced accuracy",
                    "f1-weighted",
                    "f1-micro",
                    "f1-macro"
                  ]
                },
                {
                  "marker": {
                    "color": "rgb(117,112,179)"
                  },
                  "name": "RoBERTa FT",
                  "orientation": "h",
                  "text": [
                    0.83,
                    0.93,
                    0.93,
                    0.85
                  ],
                  "textposition": "outside",
                  "type": "bar",
                  "x": [
                    0.83,
                    0.93,
                    0.93,
                    0.85
                  ],
                  "y": [
                    "balanced accuracy",
                    "f1-weighted",
                    "f1-micro",
                    "f1-macro"
                  ]
                }
              ],
              "layout": {
                "bargap": 0.3,
                "bargroupgap": 0.2,
                "barmode": "group",
                "legend": {
                  "traceorder": "reversed"
                },
                "template": {
                  "data": {
                    "bar": [
                      {
                        "error_x": {
                          "color": "#2a3f5f"
                        },
                        "error_y": {
                          "color": "#2a3f5f"
                        },
                        "marker": {
                          "line": {
                            "color": "#E5ECF6",
                            "width": 0.5
                          },
                          "pattern": {
                            "fillmode": "overlay",
                            "size": 10,
                            "solidity": 0.2
                          }
                        },
                        "type": "bar"
                      }
                    ],
                    "barpolar": [
                      {
                        "marker": {
                          "line": {
                            "color": "#E5ECF6",
                            "width": 0.5
                          },
                          "pattern": {
                            "fillmode": "overlay",
                            "size": 10,
                            "solidity": 0.2
                          }
                        },
                        "type": "barpolar"
                      }
                    ],
                    "carpet": [
                      {
                        "aaxis": {
                          "endlinecolor": "#2a3f5f",
                          "gridcolor": "white",
                          "linecolor": "white",
                          "minorgridcolor": "white",
                          "startlinecolor": "#2a3f5f"
                        },
                        "baxis": {
                          "endlinecolor": "#2a3f5f",
                          "gridcolor": "white",
                          "linecolor": "white",
                          "minorgridcolor": "white",
                          "startlinecolor": "#2a3f5f"
                        },
                        "type": "carpet"
                      }
                    ],
                    "choropleth": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "type": "choropleth"
                      }
                    ],
                    "contour": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "colorscale": [
                          [
                            0,
                            "#0d0887"
                          ],
                          [
                            0.1111111111111111,
                            "#46039f"
                          ],
                          [
                            0.2222222222222222,
                            "#7201a8"
                          ],
                          [
                            0.3333333333333333,
                            "#9c179e"
                          ],
                          [
                            0.4444444444444444,
                            "#bd3786"
                          ],
                          [
                            0.5555555555555556,
                            "#d8576b"
                          ],
                          [
                            0.6666666666666666,
                            "#ed7953"
                          ],
                          [
                            0.7777777777777778,
                            "#fb9f3a"
                          ],
                          [
                            0.8888888888888888,
                            "#fdca26"
                          ],
                          [
                            1,
                            "#f0f921"
                          ]
                        ],
                        "type": "contour"
                      }
                    ],
                    "contourcarpet": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "type": "contourcarpet"
                      }
                    ],
                    "heatmap": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "colorscale": [
                          [
                            0,
                            "#0d0887"
                          ],
                          [
                            0.1111111111111111,
                            "#46039f"
                          ],
                          [
                            0.2222222222222222,
                            "#7201a8"
                          ],
                          [
                            0.3333333333333333,
                            "#9c179e"
                          ],
                          [
                            0.4444444444444444,
                            "#bd3786"
                          ],
                          [
                            0.5555555555555556,
                            "#d8576b"
                          ],
                          [
                            0.6666666666666666,
                            "#ed7953"
                          ],
                          [
                            0.7777777777777778,
                            "#fb9f3a"
                          ],
                          [
                            0.8888888888888888,
                            "#fdca26"
                          ],
                          [
                            1,
                            "#f0f921"
                          ]
                        ],
                        "type": "heatmap"
                      }
                    ],
                    "heatmapgl": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "colorscale": [
                          [
                            0,
                            "#0d0887"
                          ],
                          [
                            0.1111111111111111,
                            "#46039f"
                          ],
                          [
                            0.2222222222222222,
                            "#7201a8"
                          ],
                          [
                            0.3333333333333333,
                            "#9c179e"
                          ],
                          [
                            0.4444444444444444,
                            "#bd3786"
                          ],
                          [
                            0.5555555555555556,
                            "#d8576b"
                          ],
                          [
                            0.6666666666666666,
                            "#ed7953"
                          ],
                          [
                            0.7777777777777778,
                            "#fb9f3a"
                          ],
                          [
                            0.8888888888888888,
                            "#fdca26"
                          ],
                          [
                            1,
                            "#f0f921"
                          ]
                        ],
                        "type": "heatmapgl"
                      }
                    ],
                    "histogram": [
                      {
                        "marker": {
                          "pattern": {
                            "fillmode": "overlay",
                            "size": 10,
                            "solidity": 0.2
                          }
                        },
                        "type": "histogram"
                      }
                    ],
                    "histogram2d": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "colorscale": [
                          [
                            0,
                            "#0d0887"
                          ],
                          [
                            0.1111111111111111,
                            "#46039f"
                          ],
                          [
                            0.2222222222222222,
                            "#7201a8"
                          ],
                          [
                            0.3333333333333333,
                            "#9c179e"
                          ],
                          [
                            0.4444444444444444,
                            "#bd3786"
                          ],
                          [
                            0.5555555555555556,
                            "#d8576b"
                          ],
                          [
                            0.6666666666666666,
                            "#ed7953"
                          ],
                          [
                            0.7777777777777778,
                            "#fb9f3a"
                          ],
                          [
                            0.8888888888888888,
                            "#fdca26"
                          ],
                          [
                            1,
                            "#f0f921"
                          ]
                        ],
                        "type": "histogram2d"
                      }
                    ],
                    "histogram2dcontour": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "colorscale": [
                          [
                            0,
                            "#0d0887"
                          ],
                          [
                            0.1111111111111111,
                            "#46039f"
                          ],
                          [
                            0.2222222222222222,
                            "#7201a8"
                          ],
                          [
                            0.3333333333333333,
                            "#9c179e"
                          ],
                          [
                            0.4444444444444444,
                            "#bd3786"
                          ],
                          [
                            0.5555555555555556,
                            "#d8576b"
                          ],
                          [
                            0.6666666666666666,
                            "#ed7953"
                          ],
                          [
                            0.7777777777777778,
                            "#fb9f3a"
                          ],
                          [
                            0.8888888888888888,
                            "#fdca26"
                          ],
                          [
                            1,
                            "#f0f921"
                          ]
                        ],
                        "type": "histogram2dcontour"
                      }
                    ],
                    "mesh3d": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "type": "mesh3d"
                      }
                    ],
                    "parcoords": [
                      {
                        "line": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "parcoords"
                      }
                    ],
                    "pie": [
                      {
                        "automargin": true,
                        "type": "pie"
                      }
                    ],
                    "scatter": [
                      {
                        "fillpattern": {
                          "fillmode": "overlay",
                          "size": 10,
                          "solidity": 0.2
                        },
                        "type": "scatter"
                      }
                    ],
                    "scatter3d": [
                      {
                        "line": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scatter3d"
                      }
                    ],
                    "scattercarpet": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scattercarpet"
                      }
                    ],
                    "scattergeo": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scattergeo"
                      }
                    ],
                    "scattergl": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scattergl"
                      }
                    ],
                    "scattermapbox": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scattermapbox"
                      }
                    ],
                    "scatterpolar": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scatterpolar"
                      }
                    ],
                    "scatterpolargl": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scatterpolargl"
                      }
                    ],
                    "scatterternary": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scatterternary"
                      }
                    ],
                    "surface": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "colorscale": [
                          [
                            0,
                            "#0d0887"
                          ],
                          [
                            0.1111111111111111,
                            "#46039f"
                          ],
                          [
                            0.2222222222222222,
                            "#7201a8"
                          ],
                          [
                            0.3333333333333333,
                            "#9c179e"
                          ],
                          [
                            0.4444444444444444,
                            "#bd3786"
                          ],
                          [
                            0.5555555555555556,
                            "#d8576b"
                          ],
                          [
                            0.6666666666666666,
                            "#ed7953"
                          ],
                          [
                            0.7777777777777778,
                            "#fb9f3a"
                          ],
                          [
                            0.8888888888888888,
                            "#fdca26"
                          ],
                          [
                            1,
                            "#f0f921"
                          ]
                        ],
                        "type": "surface"
                      }
                    ],
                    "table": [
                      {
                        "cells": {
                          "fill": {
                            "color": "#EBF0F8"
                          },
                          "line": {
                            "color": "white"
                          }
                        },
                        "header": {
                          "fill": {
                            "color": "#C8D4E3"
                          },
                          "line": {
                            "color": "white"
                          }
                        },
                        "type": "table"
                      }
                    ]
                  },
                  "layout": {
                    "annotationdefaults": {
                      "arrowcolor": "#2a3f5f",
                      "arrowhead": 0,
                      "arrowwidth": 1
                    },
                    "autotypenumbers": "strict",
                    "coloraxis": {
                      "colorbar": {
                        "outlinewidth": 0,
                        "ticks": ""
                      }
                    },
                    "colorscale": {
                      "diverging": [
                        [
                          0,
                          "#8e0152"
                        ],
                        [
                          0.1,
                          "#c51b7d"
                        ],
                        [
                          0.2,
                          "#de77ae"
                        ],
                        [
                          0.3,
                          "#f1b6da"
                        ],
                        [
                          0.4,
                          "#fde0ef"
                        ],
                        [
                          0.5,
                          "#f7f7f7"
                        ],
                        [
                          0.6,
                          "#e6f5d0"
                        ],
                        [
                          0.7,
                          "#b8e186"
                        ],
                        [
                          0.8,
                          "#7fbc41"
                        ],
                        [
                          0.9,
                          "#4d9221"
                        ],
                        [
                          1,
                          "#276419"
                        ]
                      ],
                      "sequential": [
                        [
                          0,
                          "#0d0887"
                        ],
                        [
                          0.1111111111111111,
                          "#46039f"
                        ],
                        [
                          0.2222222222222222,
                          "#7201a8"
                        ],
                        [
                          0.3333333333333333,
                          "#9c179e"
                        ],
                        [
                          0.4444444444444444,
                          "#bd3786"
                        ],
                        [
                          0.5555555555555556,
                          "#d8576b"
                        ],
                        [
                          0.6666666666666666,
                          "#ed7953"
                        ],
                        [
                          0.7777777777777778,
                          "#fb9f3a"
                        ],
                        [
                          0.8888888888888888,
                          "#fdca26"
                        ],
                        [
                          1,
                          "#f0f921"
                        ]
                      ],
                      "sequentialminus": [
                        [
                          0,
                          "#0d0887"
                        ],
                        [
                          0.1111111111111111,
                          "#46039f"
                        ],
                        [
                          0.2222222222222222,
                          "#7201a8"
                        ],
                        [
                          0.3333333333333333,
                          "#9c179e"
                        ],
                        [
                          0.4444444444444444,
                          "#bd3786"
                        ],
                        [
                          0.5555555555555556,
                          "#d8576b"
                        ],
                        [
                          0.6666666666666666,
                          "#ed7953"
                        ],
                        [
                          0.7777777777777778,
                          "#fb9f3a"
                        ],
                        [
                          0.8888888888888888,
                          "#fdca26"
                        ],
                        [
                          1,
                          "#f0f921"
                        ]
                      ]
                    },
                    "colorway": [
                      "#636efa",
                      "#EF553B",
                      "#00cc96",
                      "#ab63fa",
                      "#FFA15A",
                      "#19d3f3",
                      "#FF6692",
                      "#B6E880",
                      "#FF97FF",
                      "#FECB52"
                    ],
                    "font": {
                      "color": "#2a3f5f"
                    },
                    "geo": {
                      "bgcolor": "white",
                      "lakecolor": "white",
                      "landcolor": "#E5ECF6",
                      "showlakes": true,
                      "showland": true,
                      "subunitcolor": "white"
                    },
                    "hoverlabel": {
                      "align": "left"
                    },
                    "hovermode": "closest",
                    "mapbox": {
                      "style": "light"
                    },
                    "paper_bgcolor": "white",
                    "plot_bgcolor": "#E5ECF6",
                    "polar": {
                      "angularaxis": {
                        "gridcolor": "white",
                        "linecolor": "white",
                        "ticks": ""
                      },
                      "bgcolor": "#E5ECF6",
                      "radialaxis": {
                        "gridcolor": "white",
                        "linecolor": "white",
                        "ticks": ""
                      }
                    },
                    "scene": {
                      "xaxis": {
                        "backgroundcolor": "#E5ECF6",
                        "gridcolor": "white",
                        "gridwidth": 2,
                        "linecolor": "white",
                        "showbackground": true,
                        "ticks": "",
                        "zerolinecolor": "white"
                      },
                      "yaxis": {
                        "backgroundcolor": "#E5ECF6",
                        "gridcolor": "white",
                        "gridwidth": 2,
                        "linecolor": "white",
                        "showbackground": true,
                        "ticks": "",
                        "zerolinecolor": "white"
                      },
                      "zaxis": {
                        "backgroundcolor": "#E5ECF6",
                        "gridcolor": "white",
                        "gridwidth": 2,
                        "linecolor": "white",
                        "showbackground": true,
                        "ticks": "",
                        "zerolinecolor": "white"
                      }
                    },
                    "shapedefaults": {
                      "line": {
                        "color": "#2a3f5f"
                      }
                    },
                    "ternary": {
                      "aaxis": {
                        "gridcolor": "white",
                        "linecolor": "white",
                        "ticks": ""
                      },
                      "baxis": {
                        "gridcolor": "white",
                        "linecolor": "white",
                        "ticks": ""
                      },
                      "bgcolor": "#E5ECF6",
                      "caxis": {
                        "gridcolor": "white",
                        "linecolor": "white",
                        "ticks": ""
                      }
                    },
                    "title": {
                      "x": 0.05
                    },
                    "xaxis": {
                      "automargin": true,
                      "gridcolor": "white",
                      "linecolor": "white",
                      "ticks": "",
                      "title": {
                        "standoff": 15
                      },
                      "zerolinecolor": "white",
                      "zerolinewidth": 2
                    },
                    "yaxis": {
                      "automargin": true,
                      "gridcolor": "white",
                      "linecolor": "white",
                      "ticks": "",
                      "title": {
                        "standoff": 15
                      },
                      "zerolinecolor": "white",
                      "zerolinewidth": 2
                    }
                  }
                },
                "title": {
                  "text": "Performance comparisons"
                },
                "uniformtext": {
                  "minsize": 50
                },
                "xaxis": {
                  "title": {
                    "text": "score"
                  }
                },
                "yaxis": {
                  "title": {
                    "text": "metric"
                  }
                }
              }
            }
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "fig = go.Figure(data=[\n",
        "    go.Bar(name=\"Baseline\", y=metrics.index, x=metrics[\"Baseline\"],\n",
        "           marker_color = \"rgb(27,158,119)\", orientation = \"h\", text = metrics[\"Baseline\"],\n",
        "           textposition = \"outside\"),\n",
        "    go.Bar(name=\"RoBERTa\", y=metrics.index, x=metrics[\"RoBERTa\"],\n",
        "           marker_color = \"rgb(217,95,2)\", orientation = \"h\", text = metrics[\"RoBERTa\"], textposition = \"outside\"),\n",
        "    go.Bar(name=\"RoBERTa FT\", y=metrics.index, x=metrics[\"RoBERTa FT\"],\n",
        "           marker_color = \"rgb(117,112,179)\", orientation = \"h\", text = metrics[\"RoBERTa FT\"], textposition = \"outside\")\n",
        "])\n",
        "fig.update_layout(title='Performance comparisons',\n",
        "                   xaxis_title='score',\n",
        "                   yaxis_title='metric',\n",
        "                   barmode = \"group\",\n",
        "                   bargap = 0.3,\n",
        "                   bargroupgap = 0.2,\n",
        "                   legend = {\"traceorder\": \"reversed\"},\n",
        "                   uniformtext_minsize = 50)\n",
        "\n",
        "# Change the bar mode\n",
        "#fig.update_layout(barmode='group')\n",
        "fig.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We notice tha the both the model and its fine tuned version performs better than the baseline in all the metrics.\n",
        "\n",
        "In particular, we see that the balanced accuracy for the baseline is reduced to $1/n_{classes}$ which is the accuracy of a random classifier. \n",
        "\n",
        "The models also outperform the weighted and micro f1 scores, which is important due to the high imbalance of the classes. To briefly explain further, the baseline will correctly predict all the occurrences of the positive class, which account for approximately 85% of the dataset. Thus, if we are weighting the metrics by their class weight, this will \"hide\" the null performance on the negative class. However, both models are able to outperform this and correctly predict much more than just the majority of the positive class but also 70% of the negative class.\n",
        "\n",
        "Lastly, we are not surprised that the models outperform the baseline in f1-macro score since this gives equal weights to the classes and the baseline misclassifies all the negative instances. "
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3.9.12 ('nlp')",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.12"
    },
    "vscode": {
      "interpreter": {
        "hash": "f17bbc6fc1a5651d6d3872ee9163e60a2b8331be980fe3739f2e3a57e5088261"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}