Data_prepration.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Analysis of effects of sentiment of tweets on stock prices\n",
        "In this project our aim is to find how sentiments of tweets on twitter effect the fluctuation of valuation of a company.\n",
        "\n",
        "We have chosen 5 companies :-\n",
        "\n",
        "\n",
        "1.   Apple\n",
        "2.   Tesla\n",
        "3.   Google\n",
        "4.   Microsoft\n",
        "5.   Amazon\n",
        "\n"
      ],
      "metadata": {
        "id": "36S6PjV_SHMy"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MXlzQnzB-Xso"
      },
      "source": [
        "### Importing Libraries"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 90,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "q3WyBI1myRlI",
        "outputId": "8752d031-6ff3-4b8f-b236-0e991b27731f"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Requirement already satisfied: varname in /usr/local/lib/python3.8/dist-packages (0.10.0)\n",
            "Requirement already satisfied: executing<2.0,>=1.1 in /usr/local/lib/python3.8/dist-packages (from varname) (1.2.0)\n"
          ]
        }
      ],
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "import plotly.express as px\n",
        "import seaborn as sns\n",
        "!pip install varname\n",
        "from varname import nameof\n",
        "import os"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dVFpXgKn-kLF"
      },
      "source": [
        "### Creating function for sentiment analysis\n",
        "We have used the vader_lexicon as it works well for internet reviews and tweets as it takes into consideration emojis as well which are quite prevalent in communication methods over internet"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 91,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "po95QOoDybwP",
        "outputId": "8d941629-2ff5-417f-a7ad-bfa0ba9908ef"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package vader_lexicon to /root/nltk_data...\n",
            "[nltk_data]   Package vader_lexicon is already up-to-date!\n"
          ]
        }
      ],
      "source": [
        "# For sentiment analysis, we use VADER library\n",
        "import nltk\n",
        "nltk.download('vader_lexicon')\n",
        "from nltk.sentiment.vader import SentimentIntensityAnalyzer\n",
        "\n",
        "sid = SentimentIntensityAnalyzer()\n",
        "\n",
        "# Sentiment Analysis function\n",
        "def sentiment_analysis(tweet):\n",
        "    score = sid.polarity_scores(tweet)[\"compound\"]\n",
        "    return score"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Rzg4kOcq-vsn"
      },
      "source": [
        "### Importing data \n",
        "Converting data from csv files to panda data frame"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "A7urQEh67Xv7",
        "outputId": "fdafb299-ce3b-44dd-b693-acb3e7f3bc82"
      },
      "execution_count": 92,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount(\"/content/drive\", force_remount=True).\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 93,
      "metadata": {
        "id": "w0IwUtnfy1-Y"
      },
      "outputs": [],
      "source": [
        "# changing the directory to read the database\n",
        "\n",
        "# reading the database\n",
        "raw_company_db = pd.read_csv(\"/content/drive/MyDrive/Company.csv\")\n",
        "raw_company_tweet_db = pd.read_csv(\"/content/drive/MyDrive/Company_Tweet.csv\")\n",
        "raw_tweet_db = pd.read_csv(\"/content/drive/MyDrive/Tweet.csv\")\n",
        "raw_companyvalue_db = pd.read_csv(\"/content/drive/MyDrive/CompanyValues.csv\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C6zo4FPI_A36"
      },
      "source": [
        "### Viewing Current Data frames"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 94,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 112
        },
        "id": "RtRvG35D9rpu",
        "outputId": "99fd755d-dad9-42a6-e2b4-33350b40178b"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "  ticker_symbol company_name\n",
              "0          AAPL        apple\n",
              "1          GOOG   Google Inc"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-db67abba-4c6d-4939-99ec-0a20f662efb6\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>ticker_symbol</th>\n",
              "      <th>company_name</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>AAPL</td>\n",
              "      <td>apple</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>GOOG</td>\n",
              "      <td>Google Inc</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-db67abba-4c6d-4939-99ec-0a20f662efb6')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-db67abba-4c6d-4939-99ec-0a20f662efb6 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-db67abba-4c6d-4939-99ec-0a20f662efb6');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 94
        }
      ],
      "source": [
        "raw_company_db.head(2)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 95,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 112
        },
        "id": "ctZOY0lu9u6A",
        "outputId": "417c80e3-38b4-4305-aa7c-2d535c9d1a71"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "             tweet_id ticker_symbol\n",
              "0  550803612197457920          AAPL\n",
              "1  550803610825928706          AAPL"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-c402ea64-3ee5-4d86-9663-1b9359b62389\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>tweet_id</th>\n",
              "      <th>ticker_symbol</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>550803612197457920</td>\n",
              "      <td>AAPL</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>550803610825928706</td>\n",
              "      <td>AAPL</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-c402ea64-3ee5-4d86-9663-1b9359b62389')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-c402ea64-3ee5-4d86-9663-1b9359b62389 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-c402ea64-3ee5-4d86-9663-1b9359b62389');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 95
        }
      ],
      "source": [
        "raw_company_tweet_db.head(2)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 96,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 112
        },
        "id": "pfwiFx3d9zTB",
        "outputId": "b57f90b1-ab37-48f2-9ebc-e1acf82ac9c5"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "             tweet_id           writer   post_date  \\\n",
              "0  550441509175443456  VisualStockRSRC  1420070457   \n",
              "1  550441672312512512      KeralaGuy77  1420070496   \n",
              "\n",
              "                                                body  comment_num  \\\n",
              "0  lx21 made $10,008  on $AAPL -Check it out! htt...            0   \n",
              "1  Insanity of today weirdo massive selling. $aap...            0   \n",
              "\n",
              "   retweet_num  like_num  \n",
              "0            0         1  \n",
              "1            0         0  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-94a09116-bd59-46f3-972b-3fb39c7e6262\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>tweet_id</th>\n",
              "      <th>writer</th>\n",
              "      <th>post_date</th>\n",
              "      <th>body</th>\n",
              "      <th>comment_num</th>\n",
              "      <th>retweet_num</th>\n",
              "      <th>like_num</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>550441509175443456</td>\n",
              "      <td>VisualStockRSRC</td>\n",
              "      <td>1420070457</td>\n",
              "      <td>lx21 made $10,008  on $AAPL -Check it out! htt...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>550441672312512512</td>\n",
              "      <td>KeralaGuy77</td>\n",
              "      <td>1420070496</td>\n",
              "      <td>Insanity of today weirdo massive selling. $aap...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-94a09116-bd59-46f3-972b-3fb39c7e6262')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-94a09116-bd59-46f3-972b-3fb39c7e6262 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-94a09116-bd59-46f3-972b-3fb39c7e6262');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 96
        }
      ],
      "source": [
        "raw_tweet_db.head(2)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 97,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 112
        },
        "id": "sY3v12PX90JR",
        "outputId": "b16fbc4b-7237-402c-f9e0-b5083c8ef356"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "  ticker_symbol    day_date  close_value    volume  open_value  high_value  \\\n",
              "0          AAPL  2020-05-29       317.94  38399530      319.25      321.15   \n",
              "1          AAPL  2020-05-28       318.25  33449100      316.77      323.44   \n",
              "\n",
              "   low_value  \n",
              "0     316.47  \n",
              "1     315.63  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-60f867f8-4c03-430b-b689-a0c694acbde3\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>ticker_symbol</th>\n",
              "      <th>day_date</th>\n",
              "      <th>close_value</th>\n",
              "      <th>volume</th>\n",
              "      <th>open_value</th>\n",
              "      <th>high_value</th>\n",
              "      <th>low_value</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>AAPL</td>\n",
              "      <td>2020-05-29</td>\n",
              "      <td>317.94</td>\n",
              "      <td>38399530</td>\n",
              "      <td>319.25</td>\n",
              "      <td>321.15</td>\n",
              "      <td>316.47</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>AAPL</td>\n",
              "      <td>2020-05-28</td>\n",
              "      <td>318.25</td>\n",
              "      <td>33449100</td>\n",
              "      <td>316.77</td>\n",
              "      <td>323.44</td>\n",
              "      <td>315.63</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-60f867f8-4c03-430b-b689-a0c694acbde3')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-60f867f8-4c03-430b-b689-a0c694acbde3 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-60f867f8-4c03-430b-b689-a0c694acbde3');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 97
        }
      ],
      "source": [
        "raw_companyvalue_db.head(2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5zUQsei9_FPc"
      },
      "source": [
        "## Data Cleaning\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YjR-M2B7EDQe"
      },
      "source": [
        "### Merging the DataFrames to form one single DataFrame"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 98,
      "metadata": {
        "id": "1bT2SbXrDhYU"
      },
      "outputs": [],
      "source": [
        "# Part1: Merge raw_company_tweet_db, raw_company_db\n",
        "raw_company_tweet_db = pd.merge(raw_company_tweet_db, raw_company_db,on='ticker_symbol')\n",
        "\n",
        "# Part2: Merge raw_company_tweet_db, raw_company_db\n",
        "raw_tweet_db = pd.merge(raw_tweet_db, raw_company_tweet_db, on=\"tweet_id\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 99,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 191
        },
        "id": "4JfpI5yqDknc",
        "outputId": "13de8aa3-0ce0-4501-b262-4a83128b9d9d"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "             tweet_id           writer   post_date  \\\n",
              "0  550441509175443456  VisualStockRSRC  1420070457   \n",
              "1  550441672312512512      KeralaGuy77  1420070496   \n",
              "\n",
              "                                                body  comment_num  \\\n",
              "0  lx21 made $10,008  on $AAPL -Check it out! htt...            0   \n",
              "1  Insanity of today weirdo massive selling. $aap...            0   \n",
              "\n",
              "   retweet_num  like_num ticker_symbol company_name  \n",
              "0            0         1          AAPL        apple  \n",
              "1            0         0          AAPL        apple  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-9e6b5f40-5956-4abe-bd9d-ca35a47331eb\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>tweet_id</th>\n",
              "      <th>writer</th>\n",
              "      <th>post_date</th>\n",
              "      <th>body</th>\n",
              "      <th>comment_num</th>\n",
              "      <th>retweet_num</th>\n",
              "      <th>like_num</th>\n",
              "      <th>ticker_symbol</th>\n",
              "      <th>company_name</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>550441509175443456</td>\n",
              "      <td>VisualStockRSRC</td>\n",
              "      <td>1420070457</td>\n",
              "      <td>lx21 made $10,008  on $AAPL -Check it out! htt...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>AAPL</td>\n",
              "      <td>apple</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>550441672312512512</td>\n",
              "      <td>KeralaGuy77</td>\n",
              "      <td>1420070496</td>\n",
              "      <td>Insanity of today weirdo massive selling. $aap...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>AAPL</td>\n",
              "      <td>apple</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-9e6b5f40-5956-4abe-bd9d-ca35a47331eb')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-9e6b5f40-5956-4abe-bd9d-ca35a47331eb button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-9e6b5f40-5956-4abe-bd9d-ca35a47331eb');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 99
        }
      ],
      "source": [
        "# Veiwing the marged Dataframe \n",
        "raw_tweet_db.head(2)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 100,
      "metadata": {
        "id": "WdgBW7VMDmnI"
      },
      "outputs": [],
      "source": [
        "# Saving the names of dataframes so we can use it if required in future\n",
        "raw_tweet_db.name = nameof(raw_tweet_db)\n",
        "raw_companyvalue_db.name = nameof(raw_companyvalue_db)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 101,
      "metadata": {
        "id": "RuiKHhrRDrOd"
      },
      "outputs": [],
      "source": [
        "list_df = [raw_tweet_db, raw_companyvalue_db]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7pdpNeSRDySE"
      },
      "source": [
        "### Missing values check: Let us check if there are any missing values"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 102,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6JZaGwzWDt0_",
        "outputId": "52ae8ae2-b10c-4b51-e06c-0e6266ea193c"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "For DataFrame raw_tweet_db, we have missing values check as\n",
            "tweet_id             0\n",
            "writer           55919\n",
            "post_date            0\n",
            "body                 0\n",
            "comment_num          0\n",
            "retweet_num          0\n",
            "like_num             0\n",
            "ticker_symbol        0\n",
            "company_name         0\n",
            "dtype: int64\n",
            "\n",
            "\n",
            "For DataFrame raw_companyvalue_db, we have missing values check as\n",
            "ticker_symbol    0\n",
            "day_date         0\n",
            "close_value      0\n",
            "volume           0\n",
            "open_value       0\n",
            "high_value       0\n",
            "low_value        0\n",
            "dtype: int64\n",
            "\n",
            "\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[None, None]"
            ]
          },
          "metadata": {},
          "execution_count": 102
        }
      ],
      "source": [
        "[print(f\"For DataFrame {df.name}, we have missing values check as\\n{df.isna().sum()}\\n\\n\") for df in list_df]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 103,
      "metadata": {
        "id": "_ik_VcpmD1WS"
      },
      "outputs": [],
      "source": [
        "raw_tweet_db.writer = raw_tweet_db.writer.fillna('anonymous')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 104,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MjYNvssRD5EH",
        "outputId": "c5b9f6b6-caf7-4ea7-b30d-32adf2b1ba11"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tweet_id         0\n",
              "writer           0\n",
              "post_date        0\n",
              "body             0\n",
              "comment_num      0\n",
              "retweet_num      0\n",
              "like_num         0\n",
              "ticker_symbol    0\n",
              "company_name     0\n",
              "dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 104
        }
      ],
      "source": [
        "# let us check if all null values are replaced\n",
        "raw_tweet_db.isna().sum()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N_OWX4TvEJ_G"
      },
      "source": [
        "## Data Engineering\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8D6slhSeENZj"
      },
      "source": [
        "We will convert:\n",
        "1.   'post_date' column in raw_tweet_db\n",
        "2.   'day_date' column in raw_companyvalue_db\n",
        "\n",
        "to datetime for further processing"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 105,
      "metadata": {
        "id": "l96rT9g1D84v"
      },
      "outputs": [],
      "source": [
        "raw_tweet_db.post_date = pd.to_datetime(raw_tweet_db.post_date, unit=\"s\")\n",
        "raw_companyvalue_db.day_date = pd.to_datetime(raw_companyvalue_db.day_date)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 106,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 112
        },
        "id": "JTcCKFXmEmbs",
        "outputId": "48397599-c036-44f4-aba0-4aee41d26ada"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "  ticker_symbol   day_date  close_value    volume  open_value  high_value  \\\n",
              "0          AAPL 2020-05-29       317.94  38399530      319.25      321.15   \n",
              "1          AAPL 2020-05-28       318.25  33449100      316.77      323.44   \n",
              "\n",
              "   low_value  \n",
              "0     316.47  \n",
              "1     315.63  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-b3c70bbd-3cb6-4041-9f42-2573c3d3a40c\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>ticker_symbol</th>\n",
              "      <th>day_date</th>\n",
              "      <th>close_value</th>\n",
              "      <th>volume</th>\n",
              "      <th>open_value</th>\n",
              "      <th>high_value</th>\n",
              "      <th>low_value</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>AAPL</td>\n",
              "      <td>2020-05-29</td>\n",
              "      <td>317.94</td>\n",
              "      <td>38399530</td>\n",
              "      <td>319.25</td>\n",
              "      <td>321.15</td>\n",
              "      <td>316.47</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>AAPL</td>\n",
              "      <td>2020-05-28</td>\n",
              "      <td>318.25</td>\n",
              "      <td>33449100</td>\n",
              "      <td>316.77</td>\n",
              "      <td>323.44</td>\n",
              "      <td>315.63</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-b3c70bbd-3cb6-4041-9f42-2573c3d3a40c')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-b3c70bbd-3cb6-4041-9f42-2573c3d3a40c button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-b3c70bbd-3cb6-4041-9f42-2573c3d3a40c');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 106
        }
      ],
      "source": [
        "raw_companyvalue_db.head(2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QK1R-J1DE_zb"
      },
      "source": [
        "On viewing the dataset we see that we have tweets available from 1 Jan 2015 abd the last tweet was on 31 Dec 2019 but the stocks are given till May 2020 so we'll drop those values whose corresponding tweets are not available"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 107,
      "metadata": {
        "id": "jvWFJqYaEpCx"
      },
      "outputs": [],
      "source": [
        "raw_companyvalue_db = raw_companyvalue_db[raw_companyvalue_db.day_date < \"2020-01-01\"]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 108,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "4PHkhjklFjG1",
        "outputId": "8cb8f6a8-9180-4003-e65c-b0b03ed55516"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "    ticker_symbol   day_date  close_value    volume  open_value  high_value  \\\n",
              "103          AAPL 2019-12-31       293.65  25247630      289.93      293.68   \n",
              "104          AAPL 2019-12-30       291.52  36059610      289.46      292.69   \n",
              "105          AAPL 2019-12-27       289.80  36592940      291.12      293.97   \n",
              "106          AAPL 2019-12-26       289.91  23334000      284.82      289.98   \n",
              "107          AAPL 2019-12-24       284.27  12119710      284.69      284.89   \n",
              "\n",
              "     low_value  \n",
              "103   289.5200  \n",
              "104   285.2200  \n",
              "105   288.1200  \n",
              "106   284.7000  \n",
              "107   282.9197  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-e9209d50-4c1e-489f-a475-043cbb6589c0\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>ticker_symbol</th>\n",
              "      <th>day_date</th>\n",
              "      <th>close_value</th>\n",
              "      <th>volume</th>\n",
              "      <th>open_value</th>\n",
              "      <th>high_value</th>\n",
              "      <th>low_value</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>103</th>\n",
              "      <td>AAPL</td>\n",
              "      <td>2019-12-31</td>\n",
              "      <td>293.65</td>\n",
              "      <td>25247630</td>\n",
              "      <td>289.93</td>\n",
              "      <td>293.68</td>\n",
              "      <td>289.5200</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>104</th>\n",
              "      <td>AAPL</td>\n",
              "      <td>2019-12-30</td>\n",
              "      <td>291.52</td>\n",
              "      <td>36059610</td>\n",
              "      <td>289.46</td>\n",
              "      <td>292.69</td>\n",
              "      <td>285.2200</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>105</th>\n",
              "      <td>AAPL</td>\n",
              "      <td>2019-12-27</td>\n",
              "      <td>289.80</td>\n",
              "      <td>36592940</td>\n",
              "      <td>291.12</td>\n",
              "      <td>293.97</td>\n",
              "      <td>288.1200</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>106</th>\n",
              "      <td>AAPL</td>\n",
              "      <td>2019-12-26</td>\n",
              "      <td>289.91</td>\n",
              "      <td>23334000</td>\n",
              "      <td>284.82</td>\n",
              "      <td>289.98</td>\n",
              "      <td>284.7000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>107</th>\n",
              "      <td>AAPL</td>\n",
              "      <td>2019-12-24</td>\n",
              "      <td>284.27</td>\n",
              "      <td>12119710</td>\n",
              "      <td>284.69</td>\n",
              "      <td>284.89</td>\n",
              "      <td>282.9197</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-e9209d50-4c1e-489f-a475-043cbb6589c0')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-e9209d50-4c1e-489f-a475-043cbb6589c0 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-e9209d50-4c1e-489f-a475-043cbb6589c0');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 108
        }
      ],
      "source": [
        "raw_companyvalue_db.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aMTOTLRyF5_G"
      },
      "source": [
        "In order to check stock value performance rather than taking in absolute value we take the changes to analyze stocks better"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 109,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9N4NPb6jFm1f",
        "outputId": "3c56e439-a2d2-44b2-8180-571c204bea6e"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "<ipython-input-109-69a1f3a09aa9>:2: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame.\n",
            "Try using .loc[row_indexer,col_indexer] = value instead\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  raw_companyvalue_db['fluctuation'] = raw_companyvalue_db.high_value - raw_companyvalue_db.low_value\n",
            "<ipython-input-109-69a1f3a09aa9>:5: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame.\n",
            "Try using .loc[row_indexer,col_indexer] = value instead\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  raw_companyvalue_db['price_gain'] = raw_companyvalue_db.close_value - raw_companyvalue_db.open_value\n",
            "<ipython-input-109-69a1f3a09aa9>:8: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame.\n",
            "Try using .loc[row_indexer,col_indexer] = value instead\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  raw_companyvalue_db['total_valuation_EOD'] = raw_companyvalue_db.volume * raw_companyvalue_db.close_value\n"
          ]
        }
      ],
      "source": [
        "# Let us add column to the stock price dataframe which shows the max stock price fluctuation\n",
        "raw_companyvalue_db['fluctuation'] = raw_companyvalue_db.high_value - raw_companyvalue_db.low_value\n",
        "\n",
        "# Let us add column to the stock price dataframe which shows the net rise in stock price\n",
        "raw_companyvalue_db['price_gain'] = raw_companyvalue_db.close_value - raw_companyvalue_db.open_value\n",
        "\n",
        "# Let us add column to the stock price dataframe which shows the total valuation at the end of the day\n",
        "raw_companyvalue_db['total_valuation_EOD'] = raw_companyvalue_db.volume * raw_companyvalue_db.close_value"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 110,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 143
        },
        "id": "YbtUdjbIFvKZ",
        "outputId": "e77b9d35-a195-44a8-ac95-9e2e128f7dec"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "    ticker_symbol   day_date  close_value    volume  open_value  high_value  \\\n",
              "103          AAPL 2019-12-31       293.65  25247630      289.93      293.68   \n",
              "104          AAPL 2019-12-30       291.52  36059610      289.46      292.69   \n",
              "105          AAPL 2019-12-27       289.80  36592940      291.12      293.97   \n",
              "\n",
              "     low_value  fluctuation  price_gain  total_valuation_EOD  \n",
              "103     289.52         4.16        3.72         7.413967e+09  \n",
              "104     285.22         7.47        2.06         1.051210e+10  \n",
              "105     288.12         5.85       -1.32         1.060463e+10  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-c570345f-3955-4424-8001-d31091ccfa5a\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>ticker_symbol</th>\n",
              "      <th>day_date</th>\n",
              "      <th>close_value</th>\n",
              "      <th>volume</th>\n",
              "      <th>open_value</th>\n",
              "      <th>high_value</th>\n",
              "      <th>low_value</th>\n",
              "      <th>fluctuation</th>\n",
              "      <th>price_gain</th>\n",
              "      <th>total_valuation_EOD</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>103</th>\n",
              "      <td>AAPL</td>\n",
              "      <td>2019-12-31</td>\n",
              "      <td>293.65</td>\n",
              "      <td>25247630</td>\n",
              "      <td>289.93</td>\n",
              "      <td>293.68</td>\n",
              "      <td>289.52</td>\n",
              "      <td>4.16</td>\n",
              "      <td>3.72</td>\n",
              "      <td>7.413967e+09</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>104</th>\n",
              "      <td>AAPL</td>\n",
              "      <td>2019-12-30</td>\n",
              "      <td>291.52</td>\n",
              "      <td>36059610</td>\n",
              "      <td>289.46</td>\n",
              "      <td>292.69</td>\n",
              "      <td>285.22</td>\n",
              "      <td>7.47</td>\n",
              "      <td>2.06</td>\n",
              "      <td>1.051210e+10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>105</th>\n",
              "      <td>AAPL</td>\n",
              "      <td>2019-12-27</td>\n",
              "      <td>289.80</td>\n",
              "      <td>36592940</td>\n",
              "      <td>291.12</td>\n",
              "      <td>293.97</td>\n",
              "      <td>288.12</td>\n",
              "      <td>5.85</td>\n",
              "      <td>-1.32</td>\n",
              "      <td>1.060463e+10</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-c570345f-3955-4424-8001-d31091ccfa5a')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-c570345f-3955-4424-8001-d31091ccfa5a button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-c570345f-3955-4424-8001-d31091ccfa5a');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 110
        }
      ],
      "source": [
        "raw_companyvalue_db.head(3)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ExVT9vhmGaE5"
      },
      "source": [
        "### Sentiment analysis on the tweets"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 111,
      "metadata": {
        "id": "cS5FGmcmF25y"
      },
      "outputs": [],
      "source": [
        "raw_tweet_db['sentiment'] = raw_tweet_db['body'].apply(lambda x : sentiment_analysis(x))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 112,
      "metadata": {
        "id": "feW5nG5fGkav",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 389
        },
        "outputId": "db610d76-9577-4b23-b949-1ce11f7ce0de"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "             tweet_id           writer           post_date  \\\n",
              "0  550441509175443456  VisualStockRSRC 2015-01-01 00:00:57   \n",
              "1  550441672312512512      KeralaGuy77 2015-01-01 00:01:36   \n",
              "2  550441732014223360      DozenStocks 2015-01-01 00:01:50   \n",
              "3  550442977802207232     ShowDreamCar 2015-01-01 00:06:47   \n",
              "4  550443807834402816     i_Know_First 2015-01-01 00:10:05   \n",
              "\n",
              "                                                body  comment_num  \\\n",
              "0  lx21 made $10,008  on $AAPL -Check it out! htt...            0   \n",
              "1  Insanity of today weirdo massive selling. $aap...            0   \n",
              "2  S&P100 #Stocks Performance $HD $LOW $SBUX $TGT...            0   \n",
              "3  $GM $TSLA: Volkswagen Pushes 2014 Record Recal...            0   \n",
              "4  Swing Trading: Up To 8.91% Return In 14 Days h...            0   \n",
              "\n",
              "   retweet_num  like_num ticker_symbol company_name  sentiment  \n",
              "0            0         1          AAPL        apple     0.0000  \n",
              "1            0         0          AAPL        apple    -0.8271  \n",
              "2            0         0          AMZN   Amazon.com     0.0000  \n",
              "3            0         1          TSLA    Tesla Inc     0.0000  \n",
              "4            0         1          AAPL        apple     0.0000  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-fc368e7b-342a-45c7-abac-33a6d5a2ae6d\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>tweet_id</th>\n",
              "      <th>writer</th>\n",
              "      <th>post_date</th>\n",
              "      <th>body</th>\n",
              "      <th>comment_num</th>\n",
              "      <th>retweet_num</th>\n",
              "      <th>like_num</th>\n",
              "      <th>ticker_symbol</th>\n",
              "      <th>company_name</th>\n",
              "      <th>sentiment</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>550441509175443456</td>\n",
              "      <td>VisualStockRSRC</td>\n",
              "      <td>2015-01-01 00:00:57</td>\n",
              "      <td>lx21 made $10,008  on $AAPL -Check it out! htt...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>AAPL</td>\n",
              "      <td>apple</td>\n",
              "      <td>0.0000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>550441672312512512</td>\n",
              "      <td>KeralaGuy77</td>\n",
              "      <td>2015-01-01 00:01:36</td>\n",
              "      <td>Insanity of today weirdo massive selling. $aap...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>AAPL</td>\n",
              "      <td>apple</td>\n",
              "      <td>-0.8271</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>550441732014223360</td>\n",
              "      <td>DozenStocks</td>\n",
              "      <td>2015-01-01 00:01:50</td>\n",
              "      <td>S&amp;P100 #Stocks Performance $HD $LOW $SBUX $TGT...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>AMZN</td>\n",
              "      <td>Amazon.com</td>\n",
              "      <td>0.0000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>550442977802207232</td>\n",
              "      <td>ShowDreamCar</td>\n",
              "      <td>2015-01-01 00:06:47</td>\n",
              "      <td>$GM $TSLA: Volkswagen Pushes 2014 Record Recal...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>TSLA</td>\n",
              "      <td>Tesla Inc</td>\n",
              "      <td>0.0000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>550443807834402816</td>\n",
              "      <td>i_Know_First</td>\n",
              "      <td>2015-01-01 00:10:05</td>\n",
              "      <td>Swing Trading: Up To 8.91% Return In 14 Days h...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>AAPL</td>\n",
              "      <td>apple</td>\n",
              "      <td>0.0000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-fc368e7b-342a-45c7-abac-33a6d5a2ae6d')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-fc368e7b-342a-45c7-abac-33a6d5a2ae6d button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-fc368e7b-342a-45c7-abac-33a6d5a2ae6d');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 112
        }
      ],
      "source": [
        "raw_tweet_db.head()"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Considering there is a 'comment,  retweet & like' column, we can consider those tweets having the same sentiments\n",
        "# So for counting the total number of tweets, we add a count column telling the trending score of the tweet which will be addition of all these 3 columns\n",
        "# Adding 1 to trend score as tweet itself is one of the contributors to itself\n",
        "\n",
        "raw_tweet_db.insert(7, \"trend_score\", raw_tweet_db.comment_num + raw_tweet_db.retweet_num + raw_tweet_db.like_num + 1)"
      ],
      "metadata": {
        "id": "4FtItj8CBmeY"
      },
      "execution_count": 113,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r7LZ_7QrUukn"
      },
      "source": [
        "Using an anchor to merge two data frames"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 114,
      "metadata": {
        "id": "-xEni47aUu7d",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "33724a63-14c0-4cb4-f3b0-e99277e3204a"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.8/dist-packages/pandas/core/generic.py:5516: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame.\n",
            "Try using .loc[row_indexer,col_indexer] = value instead\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  self[name] = value\n"
          ]
        }
      ],
      "source": [
        "# Creating Anchor Column for raw_tweet_db\n",
        "raw_tweet_db.insert(3, \"date_str\", raw_tweet_db.post_date.astype(\"str\").str.split(\" \"))\n",
        "raw_tweet_db.date_str = [element[0] for element in raw_tweet_db.date_str]\n",
        "raw_tweet_db.insert(0, \"anchor\", raw_tweet_db.date_str + raw_tweet_db.ticker_symbol)\n",
        "\n",
        "# Creating Anchor Column for raw_companyvalue_db\n",
        "raw_companyvalue_db.insert(\n",
        "    2, \"date_str\", raw_companyvalue_db.day_date.astype(\"str\").str.split(\" \")\n",
        ")\n",
        "raw_companyvalue_db.date_str = [element[0] for element in raw_companyvalue_db.date_str]\n",
        "raw_companyvalue_db.insert(\n",
        "    0, \"anchor\", raw_companyvalue_db.date_str + raw_companyvalue_db.ticker_symbol\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 115,
      "metadata": {
        "id": "ZjYyrMg3VC3V"
      },
      "outputs": [],
      "source": [
        "# Merging the two dataframe\n",
        "processed_db = pd.merge(raw_tweet_db, raw_companyvalue_db, on=\"anchor\")\n",
        "\n",
        "# Let us convert he string date column \"date_str_x\" to datetime\n",
        "processed_db.date_str_x = pd.to_datetime(processed_db.date_str_x)\n",
        "\n",
        "# Since we are aiming to see the impact of tweets on stock value (i.e. rise and fall), we can drop \"neutral\" sentiments\n",
        "processed_db = processed_db[processed_db.sentiment != 0]"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "processed_db.head()"
      ],
      "metadata": {
        "id": "1ySHNxVvA1Jh",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 733
        },
        "outputId": "cd5fca40-69e8-49a8-df14-e6f0e4cdb2f7"
      },
      "execution_count": 116,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "            anchor            tweet_id          writer           post_date  \\\n",
              "1   2015-01-01AAPL  550441672312512512     KeralaGuy77 2015-01-01 00:01:36   \n",
              "9   2015-01-01AAPL  550444112328261632          GetAOM 2015-01-01 00:11:17   \n",
              "13  2015-01-01AAPL  550445850170642432  JorelLaraKalel 2015-01-01 00:18:12   \n",
              "14  2015-01-01AAPL  550447574285418497       btcgemini 2015-01-01 00:25:03   \n",
              "15  2015-01-01AAPL  550447850857828352  JorelLaraKalel 2015-01-01 00:26:09   \n",
              "\n",
              "   date_str_x                                               body  comment_num  \\\n",
              "1  2015-01-01  Insanity of today weirdo massive selling. $aap...            0   \n",
              "9  2015-01-01  $UNP $ORCL $QCOM $MSFT $AAPL Top scoring mega ...            0   \n",
              "13 2015-01-01  @CNBC 15 Top #trades for #2015 #FastMoney $TIF...            0   \n",
              "14 2015-01-01  We searched through hundreds of charts and fou...            0   \n",
              "15 2015-01-01  Top 10 searched #stocks of #2014 $AAPL $FB $BA...            0   \n",
              "\n",
              "    retweet_num  like_num  trend_score  ...   day_date  date_str_y  \\\n",
              "1             0         0            1  ... 2015-01-01  2015-01-01   \n",
              "9             0         0            1  ... 2015-01-01  2015-01-01   \n",
              "13            0         2            3  ... 2015-01-01  2015-01-01   \n",
              "14            0         0            1  ... 2015-01-01  2015-01-01   \n",
              "15            0         2            3  ... 2015-01-01  2015-01-01   \n",
              "\n",
              "    close_value    volume open_value high_value  low_value  fluctuation  \\\n",
              "1        110.38  41304780     112.82     113.13     110.21         2.92   \n",
              "9        110.38  41304780     112.82     113.13     110.21         2.92   \n",
              "13       110.38  41304780     112.82     113.13     110.21         2.92   \n",
              "14       110.38  41304780     112.82     113.13     110.21         2.92   \n",
              "15       110.38  41304780     112.82     113.13     110.21         2.92   \n",
              "\n",
              "    price_gain  total_valuation_EOD  \n",
              "1        -2.44         4.559222e+09  \n",
              "9        -2.44         4.559222e+09  \n",
              "13       -2.44         4.559222e+09  \n",
              "14       -2.44         4.559222e+09  \n",
              "15       -2.44         4.559222e+09  \n",
              "\n",
              "[5 rows x 24 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-89ab4ca2-9e69-4612-af1e-0aa71a7effdf\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>anchor</th>\n",
              "      <th>tweet_id</th>\n",
              "      <th>writer</th>\n",
              "      <th>post_date</th>\n",
              "      <th>date_str_x</th>\n",
              "      <th>body</th>\n",
              "      <th>comment_num</th>\n",
              "      <th>retweet_num</th>\n",
              "      <th>like_num</th>\n",
              "      <th>trend_score</th>\n",
              "      <th>...</th>\n",
              "      <th>day_date</th>\n",
              "      <th>date_str_y</th>\n",
              "      <th>close_value</th>\n",
              "      <th>volume</th>\n",
              "      <th>open_value</th>\n",
              "      <th>high_value</th>\n",
              "      <th>low_value</th>\n",
              "      <th>fluctuation</th>\n",
              "      <th>price_gain</th>\n",
              "      <th>total_valuation_EOD</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2015-01-01AAPL</td>\n",
              "      <td>550441672312512512</td>\n",
              "      <td>KeralaGuy77</td>\n",
              "      <td>2015-01-01 00:01:36</td>\n",
              "      <td>2015-01-01</td>\n",
              "      <td>Insanity of today weirdo massive selling. $aap...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>...</td>\n",
              "      <td>2015-01-01</td>\n",
              "      <td>2015-01-01</td>\n",
              "      <td>110.38</td>\n",
              "      <td>41304780</td>\n",
              "      <td>112.82</td>\n",
              "      <td>113.13</td>\n",
              "      <td>110.21</td>\n",
              "      <td>2.92</td>\n",
              "      <td>-2.44</td>\n",
              "      <td>4.559222e+09</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>2015-01-01AAPL</td>\n",
              "      <td>550444112328261632</td>\n",
              "      <td>GetAOM</td>\n",
              "      <td>2015-01-01 00:11:17</td>\n",
              "      <td>2015-01-01</td>\n",
              "      <td>$UNP $ORCL $QCOM $MSFT $AAPL Top scoring mega ...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>...</td>\n",
              "      <td>2015-01-01</td>\n",
              "      <td>2015-01-01</td>\n",
              "      <td>110.38</td>\n",
              "      <td>41304780</td>\n",
              "      <td>112.82</td>\n",
              "      <td>113.13</td>\n",
              "      <td>110.21</td>\n",
              "      <td>2.92</td>\n",
              "      <td>-2.44</td>\n",
              "      <td>4.559222e+09</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>2015-01-01AAPL</td>\n",
              "      <td>550445850170642432</td>\n",
              "      <td>JorelLaraKalel</td>\n",
              "      <td>2015-01-01 00:18:12</td>\n",
              "      <td>2015-01-01</td>\n",
              "      <td>@CNBC 15 Top #trades for #2015 #FastMoney $TIF...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>2</td>\n",
              "      <td>3</td>\n",
              "      <td>...</td>\n",
              "      <td>2015-01-01</td>\n",
              "      <td>2015-01-01</td>\n",
              "      <td>110.38</td>\n",
              "      <td>41304780</td>\n",
              "      <td>112.82</td>\n",
              "      <td>113.13</td>\n",
              "      <td>110.21</td>\n",
              "      <td>2.92</td>\n",
              "      <td>-2.44</td>\n",
              "      <td>4.559222e+09</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>2015-01-01AAPL</td>\n",
              "      <td>550447574285418497</td>\n",
              "      <td>btcgemini</td>\n",
              "      <td>2015-01-01 00:25:03</td>\n",
              "      <td>2015-01-01</td>\n",
              "      <td>We searched through hundreds of charts and fou...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>...</td>\n",
              "      <td>2015-01-01</td>\n",
              "      <td>2015-01-01</td>\n",
              "      <td>110.38</td>\n",
              "      <td>41304780</td>\n",
              "      <td>112.82</td>\n",
              "      <td>113.13</td>\n",
              "      <td>110.21</td>\n",
              "      <td>2.92</td>\n",
              "      <td>-2.44</td>\n",
              "      <td>4.559222e+09</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15</th>\n",
              "      <td>2015-01-01AAPL</td>\n",
              "      <td>550447850857828352</td>\n",
              "      <td>JorelLaraKalel</td>\n",
              "      <td>2015-01-01 00:26:09</td>\n",
              "      <td>2015-01-01</td>\n",
              "      <td>Top 10 searched #stocks of #2014 $AAPL $FB $BA...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>2</td>\n",
              "      <td>3</td>\n",
              "      <td>...</td>\n",
              "      <td>2015-01-01</td>\n",
              "      <td>2015-01-01</td>\n",
              "      <td>110.38</td>\n",
              "      <td>41304780</td>\n",
              "      <td>112.82</td>\n",
              "      <td>113.13</td>\n",
              "      <td>110.21</td>\n",
              "      <td>2.92</td>\n",
              "      <td>-2.44</td>\n",
              "      <td>4.559222e+09</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>5 rows × 24 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-89ab4ca2-9e69-4612-af1e-0aa71a7effdf')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-89ab4ca2-9e69-4612-af1e-0aa71a7effdf button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-89ab4ca2-9e69-4612-af1e-0aa71a7effdf');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 116
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Nj3BhboYVXt5"
      },
      "source": [
        "### Making a clean data frame for further analysis"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 117,
      "metadata": {
        "id": "XxAYn1BvVIpc"
      },
      "outputs": [],
      "source": [
        "# Let us make a clean dataset with only the desired values\n",
        "clean_db = processed_db[\n",
        "    [\n",
        "        \"post_date\",\n",
        "        \"date_str_x\",\n",
        "        \"body\",\n",
        "        \"trend_score\",\n",
        "        \"ticker_symbol_x\",\n",
        "        \"company_name\",\n",
        "        \"sentiment\",\n",
        "        \"close_value\",\n",
        "        \"volume\",\n",
        "        \"open_value\",\n",
        "        \"high_value\",\n",
        "        \"low_value\",\n",
        "        \"fluctuation\",\n",
        "        \"price_gain\",\n",
        "        \"total_valuation_EOD\"\n",
        "    ]\n",
        "]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 118,
      "metadata": {
        "id": "NA_WFL7uVqul",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e156123a-2aaa-4b6b-c52d-834254bca7ba"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "In our dataset, we have total 5 companies, namely\n",
            "apple         766291\n",
            "Tesla Inc     713108\n",
            "Amazon.com    426806\n",
            "Google Inc    357476\n",
            "Microsoft     188850\n",
            "Name: company_name, dtype: int64\n"
          ]
        }
      ],
      "source": [
        "# Let us check how many companies do we have in our dataset\n",
        "print(\n",
        "    f\"In our dataset, we have total {len(clean_db.company_name.value_counts())} companies, namely\\n{clean_db.company_name.value_counts()}\"\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 119,
      "metadata": {
        "id": "jwawSVjtVy7p",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "60d5a397-0350-49e6-9aa9-fdb738090379"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.8/dist-packages/pandas/core/generic.py:6619: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  return self._update_inplace(result)\n"
          ]
        }
      ],
      "source": [
        "# after checking the ticker symbol, we see that Google Inc has 2 ticker symbols: GOOG and GOOGL\n",
        "# Let us change that so all Google Inc tickers have the same symbol as GOOG\n",
        "clean_db.ticker_symbol_x.replace(\"GOOGL\", \"GOOG\", inplace=True)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 120,
      "metadata": {
        "id": "i63flIQ3V5Xg"
      },
      "outputs": [],
      "source": [
        "# Let us make datasets for these 5 companies\n",
        "apple_df = clean_db[clean_db.ticker_symbol_x == \"AAPL\"]\n",
        "tesla_df = clean_db[clean_db.ticker_symbol_x == \"TSLA\"]\n",
        "amazon_df = clean_db[clean_db.ticker_symbol_x == \"AMZN\"]\n",
        "google_df = clean_db[clean_db.ticker_symbol_x == \"GOOG\"]\n",
        "microsoft_df = clean_db[clean_db.ticker_symbol_x == \"MSFT\"]"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "trend_df = apple_df.groupby(by=[\"date_str_x\",\"close_value\"], as_index=False).agg({\"trend_score\":pd.Series.sum})\n",
        "sentiment_df = apple_df.groupby(by=[\"date_str_x\",], as_index=False).agg({\"sentiment\":pd.Series.mean})\n",
        "final_apple = pd.merge(trend_df, sentiment_df, on='date_str_x')\n",
        "final_apple.to_csv('final_apple.csv')\n",
        "final_apple"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 424
        },
        "id": "B-5TUIBs0MvD",
        "outputId": "10fd8405-f6d9-408b-ac55-a5b1ee8fe5d0"
      },
      "execution_count": 121,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "     date_str_x  close_value  trend_score  sentiment\n",
              "0    2015-01-01       110.38         1705   0.275451\n",
              "1    2015-01-02       109.33         1539   0.169319\n",
              "2    2015-01-03       109.33          330   0.392737\n",
              "3    2015-01-04       109.33          801   0.370549\n",
              "4    2015-01-05       106.25         1402   0.248010\n",
              "...         ...          ...          ...        ...\n",
              "1814 2019-12-27       289.80         1170   0.275813\n",
              "1815 2019-12-28       289.80          520   0.300492\n",
              "1816 2019-12-29       289.80         1045   0.315323\n",
              "1817 2019-12-30       291.52         2761   0.333157\n",
              "1818 2019-12-31       293.65         1691   0.281534\n",
              "\n",
              "[1819 rows x 4 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-a9c3e24f-63d2-4ec6-9e18-a19b78a4b53b\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>date_str_x</th>\n",
              "      <th>close_value</th>\n",
              "      <th>trend_score</th>\n",
              "      <th>sentiment</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2015-01-01</td>\n",
              "      <td>110.38</td>\n",
              "      <td>1705</td>\n",
              "      <td>0.275451</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2015-01-02</td>\n",
              "      <td>109.33</td>\n",
              "      <td>1539</td>\n",
              "      <td>0.169319</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2015-01-03</td>\n",
              "      <td>109.33</td>\n",
              "      <td>330</td>\n",
              "      <td>0.392737</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>2015-01-04</td>\n",
              "      <td>109.33</td>\n",
              "      <td>801</td>\n",
              "      <td>0.370549</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>2015-01-05</td>\n",
              "      <td>106.25</td>\n",
              "      <td>1402</td>\n",
              "      <td>0.248010</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1814</th>\n",
              "      <td>2019-12-27</td>\n",
              "      <td>289.80</td>\n",
              "      <td>1170</td>\n",
              "      <td>0.275813</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1815</th>\n",
              "      <td>2019-12-28</td>\n",
              "      <td>289.80</td>\n",
              "      <td>520</td>\n",
              "      <td>0.300492</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1816</th>\n",
              "      <td>2019-12-29</td>\n",
              "      <td>289.80</td>\n",
              "      <td>1045</td>\n",
              "      <td>0.315323</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1817</th>\n",
              "      <td>2019-12-30</td>\n",
              "      <td>291.52</td>\n",
              "      <td>2761</td>\n",
              "      <td>0.333157</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1818</th>\n",
              "      <td>2019-12-31</td>\n",
              "      <td>293.65</td>\n",
              "      <td>1691</td>\n",
              "      <td>0.281534</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>1819 rows × 4 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-a9c3e24f-63d2-4ec6-9e18-a19b78a4b53b')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-a9c3e24f-63d2-4ec6-9e18-a19b78a4b53b button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-a9c3e24f-63d2-4ec6-9e18-a19b78a4b53b');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 121
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "trend_df = google_df.groupby(by=[\"date_str_x\",\"close_value\"], as_index=False).agg({\"trend_score\":pd.Series.sum})\n",
        "sentiment_df = google_df.groupby(by=[\"date_str_x\",], as_index=False).agg({\"sentiment\":pd.Series.mean})\n",
        "final_google = pd.merge(trend_df, sentiment_df, on='date_str_x')\n",
        "final_google.to_csv('final_google.csv')\n",
        "final_google"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 424
        },
        "id": "RzEvkuky012M",
        "outputId": "d011825b-cd8d-41af-b639-bc64b6392680"
      },
      "execution_count": 122,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "     date_str_x  close_value  trend_score  sentiment\n",
              "0    2015-01-01       526.40           64   0.338186\n",
              "1    2015-01-01       530.66           37   0.338186\n",
              "2    2015-01-02       524.81          143   0.270801\n",
              "3    2015-01-02       529.55           77   0.270801\n",
              "4    2015-01-03       524.81          169   0.463743\n",
              "...         ...          ...          ...        ...\n",
              "3646 2019-12-29      1354.64          200   0.368337\n",
              "3647 2019-12-30      1336.14          297   0.394844\n",
              "3648 2019-12-30      1339.71          298   0.394844\n",
              "3649 2019-12-31      1337.02          132   0.294916\n",
              "3650 2019-12-31      1339.39          213   0.294916\n",
              "\n",
              "[3651 rows x 4 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-14f98c49-d31b-4412-8506-d6f44ca255ce\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>date_str_x</th>\n",
              "      <th>close_value</th>\n",
              "      <th>trend_score</th>\n",
              "      <th>sentiment</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2015-01-01</td>\n",
              "      <td>526.40</td>\n",
              "      <td>64</td>\n",
              "      <td>0.338186</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2015-01-01</td>\n",
              "      <td>530.66</td>\n",
              "      <td>37</td>\n",
              "      <td>0.338186</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2015-01-02</td>\n",
              "      <td>524.81</td>\n",
              "      <td>143</td>\n",
              "      <td>0.270801</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>2015-01-02</td>\n",
              "      <td>529.55</td>\n",
              "      <td>77</td>\n",
              "      <td>0.270801</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>2015-01-03</td>\n",
              "      <td>524.81</td>\n",
              "      <td>169</td>\n",
              "      <td>0.463743</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3646</th>\n",
              "      <td>2019-12-29</td>\n",
              "      <td>1354.64</td>\n",
              "      <td>200</td>\n",
              "      <td>0.368337</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3647</th>\n",
              "      <td>2019-12-30</td>\n",
              "      <td>1336.14</td>\n",
              "      <td>297</td>\n",
              "      <td>0.394844</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3648</th>\n",
              "      <td>2019-12-30</td>\n",
              "      <td>1339.71</td>\n",
              "      <td>298</td>\n",
              "      <td>0.394844</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3649</th>\n",
              "      <td>2019-12-31</td>\n",
              "      <td>1337.02</td>\n",
              "      <td>132</td>\n",
              "      <td>0.294916</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3650</th>\n",
              "      <td>2019-12-31</td>\n",
              "      <td>1339.39</td>\n",
              "      <td>213</td>\n",
              "      <td>0.294916</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>3651 rows × 4 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-14f98c49-d31b-4412-8506-d6f44ca255ce')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-14f98c49-d31b-4412-8506-d6f44ca255ce button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-14f98c49-d31b-4412-8506-d6f44ca255ce');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 122
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "trend_df = amazon_df.groupby(by=[\"date_str_x\",\"close_value\"], as_index=False).agg({\"trend_score\":pd.Series.sum})\n",
        "sentiment_df = amazon_df.groupby(by=[\"date_str_x\",], as_index=False).agg({\"sentiment\":pd.Series.mean})\n",
        "final_amazon = pd.merge(trend_df, sentiment_df, on='date_str_x')\n",
        "final_amazon.to_csv('final_amazon.csv')\n",
        "final_amazon"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 424
        },
        "id": "uRiElwP629HV",
        "outputId": "35f273cd-34b0-4a21-cc4d-2614ed7e4f7d"
      },
      "execution_count": 123,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "     date_str_x  close_value  trend_score  sentiment\n",
              "0    2015-01-01       310.35         1007  -0.221957\n",
              "1    2015-01-02       308.52          534   0.184688\n",
              "2    2015-01-03       308.52           61   0.259859\n",
              "3    2015-01-04       308.52           90   0.287238\n",
              "4    2015-01-05       302.19          242   0.318348\n",
              "...         ...          ...          ...        ...\n",
              "1821 2019-12-27      1869.80         3860   0.353117\n",
              "1822 2019-12-28      1869.80          753   0.276010\n",
              "1823 2019-12-29      1869.80          868   0.327622\n",
              "1824 2019-12-30      1846.89         1498   0.355969\n",
              "1825 2019-12-31      1847.84         1067   0.345254\n",
              "\n",
              "[1826 rows x 4 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-c42990e3-29ea-478f-840d-83f4728bb6d5\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>date_str_x</th>\n",
              "      <th>close_value</th>\n",
              "      <th>trend_score</th>\n",
              "      <th>sentiment</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2015-01-01</td>\n",
              "      <td>310.35</td>\n",
              "      <td>1007</td>\n",
              "      <td>-0.221957</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2015-01-02</td>\n",
              "      <td>308.52</td>\n",
              "      <td>534</td>\n",
              "      <td>0.184688</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2015-01-03</td>\n",
              "      <td>308.52</td>\n",
              "      <td>61</td>\n",
              "      <td>0.259859</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>2015-01-04</td>\n",
              "      <td>308.52</td>\n",
              "      <td>90</td>\n",
              "      <td>0.287238</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>2015-01-05</td>\n",
              "      <td>302.19</td>\n",
              "      <td>242</td>\n",
              "      <td>0.318348</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1821</th>\n",
              "      <td>2019-12-27</td>\n",
              "      <td>1869.80</td>\n",
              "      <td>3860</td>\n",
              "      <td>0.353117</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1822</th>\n",
              "      <td>2019-12-28</td>\n",
              "      <td>1869.80</td>\n",
              "      <td>753</td>\n",
              "      <td>0.276010</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1823</th>\n",
              "      <td>2019-12-29</td>\n",
              "      <td>1869.80</td>\n",
              "      <td>868</td>\n",
              "      <td>0.327622</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1824</th>\n",
              "      <td>2019-12-30</td>\n",
              "      <td>1846.89</td>\n",
              "      <td>1498</td>\n",
              "      <td>0.355969</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1825</th>\n",
              "      <td>2019-12-31</td>\n",
              "      <td>1847.84</td>\n",
              "      <td>1067</td>\n",
              "      <td>0.345254</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>1826 rows × 4 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-c42990e3-29ea-478f-840d-83f4728bb6d5')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-c42990e3-29ea-478f-840d-83f4728bb6d5 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-c42990e3-29ea-478f-840d-83f4728bb6d5');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 123
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "trend_df = tesla_df.groupby(by=[\"date_str_x\",\"close_value\"], as_index=False).agg({\"trend_score\":pd.Series.sum})\n",
        "sentiment_df = tesla_df.groupby(by=[\"date_str_x\",], as_index=False).agg({\"sentiment\":pd.Series.mean})\n",
        "final_tesla = pd.merge(trend_df, sentiment_df, on='date_str_x')\n",
        "final_tesla.to_csv('final_tesla.csv')\n",
        "final_tesla"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 424
        },
        "id": "9-V9j5AN3MXB",
        "outputId": "4404491d-7d81-4354-baea-4bc47e559f73"
      },
      "execution_count": 124,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "     date_str_x  close_value  trend_score  sentiment\n",
              "0    2015-01-01       222.41          172   0.248615\n",
              "1    2015-01-02       219.31          246   0.203091\n",
              "2    2015-01-03       219.31           99   0.155294\n",
              "3    2015-01-04       219.31           88   0.335715\n",
              "4    2015-01-05       210.09          249   0.358320\n",
              "...         ...          ...          ...        ...\n",
              "1821 2019-12-27       430.38        13045   0.172203\n",
              "1822 2019-12-28       430.38         5067   0.198432\n",
              "1823 2019-12-29       430.38         5937   0.126686\n",
              "1824 2019-12-30       414.70        13469   0.140709\n",
              "1825 2019-12-31       418.33         8375   0.187276\n",
              "\n",
              "[1826 rows x 4 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-3b3fc1a1-6702-46c7-b621-7d6cdd03e917\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>date_str_x</th>\n",
              "      <th>close_value</th>\n",
              "      <th>trend_score</th>\n",
              "      <th>sentiment</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2015-01-01</td>\n",
              "      <td>222.41</td>\n",
              "      <td>172</td>\n",
              "      <td>0.248615</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2015-01-02</td>\n",
              "      <td>219.31</td>\n",
              "      <td>246</td>\n",
              "      <td>0.203091</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2015-01-03</td>\n",
              "      <td>219.31</td>\n",
              "      <td>99</td>\n",
              "      <td>0.155294</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>2015-01-04</td>\n",
              "      <td>219.31</td>\n",
              "      <td>88</td>\n",
              "      <td>0.335715</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>2015-01-05</td>\n",
              "      <td>210.09</td>\n",
              "      <td>249</td>\n",
              "      <td>0.358320</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1821</th>\n",
              "      <td>2019-12-27</td>\n",
              "      <td>430.38</td>\n",
              "      <td>13045</td>\n",
              "      <td>0.172203</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1822</th>\n",
              "      <td>2019-12-28</td>\n",
              "      <td>430.38</td>\n",
              "      <td>5067</td>\n",
              "      <td>0.198432</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1823</th>\n",
              "      <td>2019-12-29</td>\n",
              "      <td>430.38</td>\n",
              "      <td>5937</td>\n",
              "      <td>0.126686</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1824</th>\n",
              "      <td>2019-12-30</td>\n",
              "      <td>414.70</td>\n",
              "      <td>13469</td>\n",
              "      <td>0.140709</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1825</th>\n",
              "      <td>2019-12-31</td>\n",
              "      <td>418.33</td>\n",
              "      <td>8375</td>\n",
              "      <td>0.187276</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>1826 rows × 4 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-3b3fc1a1-6702-46c7-b621-7d6cdd03e917')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-3b3fc1a1-6702-46c7-b621-7d6cdd03e917 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-3b3fc1a1-6702-46c7-b621-7d6cdd03e917');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 124
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "trend_df = microsoft_df.groupby(by=[\"date_str_x\",\"close_value\"], as_index=False).agg({\"trend_score\":pd.Series.sum})\n",
        "sentiment_df = microsoft_df.groupby(by=[\"date_str_x\",], as_index=False).agg({\"sentiment\":pd.Series.mean})\n",
        "final_microsoft = pd.merge(trend_df, sentiment_df, on='date_str_x')\n",
        "final_microsoft.to_csv('final_microsoft.csv')\n",
        "final_microsoft"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 424
        },
        "id": "fggv-7DW3TgW",
        "outputId": "dcb669f0-8724-44a0-8eba-84f8f666e4e2"
      },
      "execution_count": 125,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "     date_str_x  close_value  trend_score  sentiment\n",
              "0    2015-01-01       46.450           81   0.204437\n",
              "1    2015-01-02       46.760          118   0.265249\n",
              "2    2015-01-03       46.760           58   0.219583\n",
              "3    2015-01-04       46.760           30   0.304124\n",
              "4    2015-01-05       46.325          172   0.240367\n",
              "...         ...          ...          ...        ...\n",
              "1821 2019-12-27      158.960          736   0.303387\n",
              "1822 2019-12-28      158.960          249   0.257665\n",
              "1823 2019-12-29      158.960          601   0.399279\n",
              "1824 2019-12-30      157.590          508   0.407251\n",
              "1825 2019-12-31      157.700          861   0.344254\n",
              "\n",
              "[1826 rows x 4 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-1b99e6c9-8df4-462a-a63f-fe68768e0ff1\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>date_str_x</th>\n",
              "      <th>close_value</th>\n",
              "      <th>trend_score</th>\n",
              "      <th>sentiment</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2015-01-01</td>\n",
              "      <td>46.450</td>\n",
              "      <td>81</td>\n",
              "      <td>0.204437</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2015-01-02</td>\n",
              "      <td>46.760</td>\n",
              "      <td>118</td>\n",
              "      <td>0.265249</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2015-01-03</td>\n",
              "      <td>46.760</td>\n",
              "      <td>58</td>\n",
              "      <td>0.219583</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>2015-01-04</td>\n",
              "      <td>46.760</td>\n",
              "      <td>30</td>\n",
              "      <td>0.304124</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>2015-01-05</td>\n",
              "      <td>46.325</td>\n",
              "      <td>172</td>\n",
              "      <td>0.240367</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1821</th>\n",
              "      <td>2019-12-27</td>\n",
              "      <td>158.960</td>\n",
              "      <td>736</td>\n",
              "      <td>0.303387</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1822</th>\n",
              "      <td>2019-12-28</td>\n",
              "      <td>158.960</td>\n",
              "      <td>249</td>\n",
              "      <td>0.257665</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1823</th>\n",
              "      <td>2019-12-29</td>\n",
              "      <td>158.960</td>\n",
              "      <td>601</td>\n",
              "      <td>0.399279</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1824</th>\n",
              "      <td>2019-12-30</td>\n",
              "      <td>157.590</td>\n",
              "      <td>508</td>\n",
              "      <td>0.407251</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1825</th>\n",
              "      <td>2019-12-31</td>\n",
              "      <td>157.700</td>\n",
              "      <td>861</td>\n",
              "      <td>0.344254</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>1826 rows × 4 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-1b99e6c9-8df4-462a-a63f-fe68768e0ff1')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-1b99e6c9-8df4-462a-a63f-fe68768e0ff1 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-1b99e6c9-8df4-462a-a63f-fe68768e0ff1');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 125
        }
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.6"
    },
    "vscode": {
      "interpreter": {
        "hash": "b6fae8e20247ccfeab698d47ab548faf2c3464c235d66cd81eacce5375a2494f"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}