Twitter_Sentiment_Analysis.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Anshuman-37/Twitter_Sentiment_Analysis/blob/main/Twitter_Sentiment_Analysis.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lBguLkqplQN9"
      },
      "source": [
        "## IDEA- \n",
        "0. Downloading Data and Reading it\n",
        "\n",
        "\n",
        "> Use KAGGLE JSON file to directly donwload data in colab and use it if having any problem search on google how to download kaggle data.\n",
        "\n",
        "> ~~Then read the data in pyspark dataframe~~\n",
        "\n",
        "Done\n",
        "\n",
        "\n",
        "1. Change Data Type \n",
        "\n",
        "> ~~Everything is a string type first convert it to the required type and change the schema~~ \n",
        "\n",
        "~~<em>Answer -  I think I will brute force on this one getting all the column names and changing the schema</em>~~\n",
        "\n",
        "Done\n",
        "\n",
        "2. Data visualization\n",
        "\n",
        "\n",
        "> ~~We need to visualize the data and check for imbalance. To do that we need to concatinate the data~~ \n",
        "\n",
        "~~Answer - Concatinate Biden and Trumps data in a single data frame~~ \n",
        "\n",
        "Done\n",
        "\n",
        "3. Data cleaning\n",
        "\n",
        "> ~~Haven't thought about it in detail but we need to do normal nltk steps to do this~~\n",
        "\n",
        "~~Answer - Used Regular Expression to remove the useless characters in the string~~\n",
        "\n",
        "Done \n",
        "\n",
        "4. Analysis\n",
        "\n",
        "> Think of using a library or we need to do it on our own ? \n",
        "\n",
        "~~Answer - Create a new column and apply text blob on it to get the result.~~\n",
        "\n",
        "> Do we need to use vector assembler and the other shit as well to do this. I am not sure..\n",
        "we need to apply some ML algorithm i was thinking in k-means (something like this: https://medium.com/analytics-vidhya/congressional-tweets-using-sentiment-analysis-to-cluster-members-of-congress-in-pyspark-10afa4d1556e ) but we can do another one.\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "W5uuOaFEFg_x"
      },
      "outputs": [],
      "source": [
        ""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sYCL7wB5Fh0C"
      },
      "source": [
        "## Downloading the data from Kaggle"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "qOYJg_BsDDmW",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f94a6cb7-7eac-4f89-fa34-2ad061e1e7cf"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: kaggle in /usr/local/lib/python3.7/dist-packages (1.5.12)\n",
            "Requirement already satisfied: python-slugify in /usr/local/lib/python3.7/dist-packages (from kaggle) (6.1.2)\n",
            "Requirement already satisfied: six>=1.10 in /usr/local/lib/python3.7/dist-packages (from kaggle) (1.15.0)\n",
            "Requirement already satisfied: certifi in /usr/local/lib/python3.7/dist-packages (from kaggle) (2021.10.8)\n",
            "Requirement already satisfied: python-dateutil in /usr/local/lib/python3.7/dist-packages (from kaggle) (2.8.2)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from kaggle) (4.64.0)\n",
            "Requirement already satisfied: urllib3 in /usr/local/lib/python3.7/dist-packages (from kaggle) (1.24.3)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from kaggle) (2.23.0)\n",
            "Requirement already satisfied: text-unidecode>=1.3 in /usr/local/lib/python3.7/dist-packages (from python-slugify->kaggle) (1.3)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->kaggle) (2.10)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->kaggle) (3.0.4)\n",
            "mkdir: cannot create directory ‘/root/.kaggle’: File exists\n"
          ]
        }
      ],
      "source": [
        "## Prerequisites\n",
        "!pip install kaggle --upgrade\n",
        "!mkdir ~/.kaggle\n",
        "!cp kaggle.json ~/.kaggle/\n",
        "!chmod 600 ~/.kaggle/kaggle.json"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "IRuQFE-cE7bp",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "40ddd875-b8fa-4b60-ad34-8238647ad359"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "us-election-2020-tweets.zip: Skipping, found more recently modified local copy (use --force to force download)\n"
          ]
        }
      ],
      "source": [
        "## Downloading data command\n",
        "!kaggle datasets download -d manchunhui/us-election-2020-tweets"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "GO1eJ1FDF2uP",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "459324b4-770d-4a3f-9f1b-4b4cace396bb"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "mkdir: cannot create directory ‘content/data’: No such file or directory\n",
            "Archive:  us-election-2020-tweets.zip\n",
            "replace data/hashtag_donaldtrump.csv? [y]es, [n]o, [A]ll, [N]one, [r]ename: N\n"
          ]
        }
      ],
      "source": [
        "!mkdir content/data\n",
        "!unzip us-election-2020-tweets.zip -d data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Sz3hB6Dc-GX2"
      },
      "source": [
        "## Download Pyspark"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "V_Cs4IN0-Gfj",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "aaf3306f-8cc4-4190-cf5c-46a95d3e3a4d"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: pyspark in /usr/local/lib/python3.7/dist-packages (3.2.1)\n",
            "Requirement already satisfied: py4j==0.10.9.3 in /usr/local/lib/python3.7/dist-packages (from pyspark) (0.10.9.3)\n"
          ]
        }
      ],
      "source": [
        "!pip install pyspark "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tdwfhe3oFpy-"
      },
      "source": [
        "## Import Header Files"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "Q2PUC_Z2FaEY"
      },
      "outputs": [],
      "source": [
        "## Python \n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sns \n",
        "import re\n",
        "\n",
        "\n",
        "# stopwords, tokenizer, stemmer\n",
        "import nltk  \n",
        "from nltk.corpus import stopwords\n",
        "from nltk.tokenize import word_tokenize\n",
        "from nltk.stem import PorterStemmer\n",
        "from nltk.probability import FreqDist\n",
        "\n",
        "# spell correction, lemmatization\n",
        "from textblob import TextBlob\n",
        "from textblob import Word\n",
        "\n",
        "## Spark\n",
        "import pyspark.sql.functions as F\n",
        "from pyspark.sql import Row\n",
        "from pyspark import SparkContext\n",
        "from pyspark.sql import SparkSession\n",
        "from pyspark.sql.types import StructType, StructField, StringType , FloatType, IntegerType, ByteType\n",
        "from pyspark.sql.functions import lit\n",
        "from pyspark.sql.functions import regexp_replace\n",
        "from pyspark.sql.functions import udf\n",
        "\n",
        "# Create Spark Session\n",
        "spark = SparkSession.builder.master(\"local[1]\").appName('Twitter Sentiment Analysis').getOrCreate()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AYCcYnYLlrdb"
      },
      "source": [
        "## Reading the Data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Zp1PRJSTz9zo"
      },
      "source": [
        "#### Defining the Schema"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "fm82ZCTKz98f"
      },
      "outputs": [],
      "source": [
        "\n",
        "# Created the schema that we need \n",
        "# Fucking Brute Force I dont like it\n",
        "mySchema = StructType([ StructField('created_at', StringType(), True)\\\n",
        "                       ,StructField('tweet_id', FloatType(), True)\\\n",
        "                       ,StructField('tweet',StringType(),True)\\\n",
        "                       ,StructField('likes',FloatType(),True)\\\n",
        "                       ,StructField('retweet_count',FloatType(),True)\\\n",
        "                       ,StructField('source',StringType(),True)\\\n",
        "                       ,StructField('user_id',FloatType(),True)\\\n",
        "                       ,StructField('user_name',StringType(),True)\\\n",
        "                       ,StructField('user_screen_name',StringType(),True)\\\n",
        "                       ,StructField('user_description',StringType(),True)\\\n",
        "                       ,StructField('user_join_date',StringType(),True)\\\n",
        "                       ,StructField('user_followers_count',FloatType(),True)\\\n",
        "                       ,StructField('user_location',StringType(),True)\\\n",
        "                       ,StructField('lat',FloatType(),True)\\\n",
        "                       ,StructField('long',FloatType(),True)\\\n",
        "                       ,StructField('city',StringType(),True)\\\n",
        "                       ,StructField('country',StringType(),True)\\\n",
        "                       ,StructField('continent',StringType(),True)\\\n",
        "                       ,StructField('state',StringType(),True)\\\n",
        "                       ,StructField('state_code',StringType(),True)\\\n",
        "                       ,StructField('collected_at',StringType(),True)\\\n",
        "                       ])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "cpeKGK5IH3rG",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 930
        },
        "outputId": "43647734-6b57-483d-832f-94ffbe0fd604"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Trump DataFrame Schema\n",
            "root\n",
            " |-- created_at: string (nullable = true)\n",
            " |-- tweet_id: float (nullable = true)\n",
            " |-- tweet: string (nullable = true)\n",
            " |-- likes: float (nullable = true)\n",
            " |-- retweet_count: float (nullable = true)\n",
            " |-- source: string (nullable = true)\n",
            " |-- user_id: float (nullable = true)\n",
            " |-- user_name: string (nullable = true)\n",
            " |-- user_screen_name: string (nullable = true)\n",
            " |-- user_description: string (nullable = true)\n",
            " |-- user_join_date: string (nullable = true)\n",
            " |-- user_followers_count: float (nullable = true)\n",
            " |-- user_location: string (nullable = true)\n",
            " |-- lat: float (nullable = true)\n",
            " |-- long: float (nullable = true)\n",
            " |-- city: string (nullable = true)\n",
            " |-- country: string (nullable = true)\n",
            " |-- continent: string (nullable = true)\n",
            " |-- state: string (nullable = true)\n",
            " |-- state_code: string (nullable = true)\n",
            " |-- collected_at: string (nullable = true)\n",
            "\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "DataFrame[created_at: string, tweet_id: float, tweet: string, likes: float, retweet_count: float, source: string, user_id: float, user_name: string, user_screen_name: string, user_description: string, user_join_date: string, user_followers_count: float, user_location: string, lat: float, long: float, city: string, country: string, continent: string, state: string, state_code: string, collected_at: string]"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Biden DataFrame Schema\n",
            "root\n",
            " |-- created_at: string (nullable = true)\n",
            " |-- tweet_id: float (nullable = true)\n",
            " |-- tweet: string (nullable = true)\n",
            " |-- likes: float (nullable = true)\n",
            " |-- retweet_count: float (nullable = true)\n",
            " |-- source: string (nullable = true)\n",
            " |-- user_id: float (nullable = true)\n",
            " |-- user_name: string (nullable = true)\n",
            " |-- user_screen_name: string (nullable = true)\n",
            " |-- user_description: string (nullable = true)\n",
            " |-- user_join_date: string (nullable = true)\n",
            " |-- user_followers_count: float (nullable = true)\n",
            " |-- user_location: string (nullable = true)\n",
            " |-- lat: float (nullable = true)\n",
            " |-- long: float (nullable = true)\n",
            " |-- city: string (nullable = true)\n",
            " |-- country: string (nullable = true)\n",
            " |-- continent: string (nullable = true)\n",
            " |-- state: string (nullable = true)\n",
            " |-- state_code: string (nullable = true)\n",
            " |-- collected_at: string (nullable = true)\n",
            "\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "DataFrame[created_at: string, tweet_id: float, tweet: string, likes: float, retweet_count: float, source: string, user_id: float, user_name: string, user_screen_name: string, user_description: string, user_join_date: string, user_followers_count: float, user_location: string, lat: float, long: float, city: string, country: string, continent: string, state: string, state_code: string, collected_at: string]"
            ]
          },
          "metadata": {}
        }
      ],
      "source": [
        "trump_df = spark.read.option('multiline',True)\\\n",
        "                     .option('header', 'true')\\\n",
        "                     .option('lineterminator',\"\\n\")\\\n",
        "                     .option(\"quote\", \"\\\"\")\\\n",
        "                     .option(\"escape\", \"\\\"\")\\\n",
        "                     .schema(mySchema)\\\n",
        "                     .csv('/content/data/hashtag_donaldtrump.csv')\n",
        "trump_df.cache()\n",
        "#trump_df.unpersist()\n",
        "print('Trump DataFrame Schema')\n",
        "\n",
        "trump_df.printSchema()\n",
        "display(trump_df)\n",
        "\n",
        "biden_df = spark.read.option('multiline',True)\\\n",
        "                     .option('header', 'true')\\\n",
        "                     .option('lineterminator',\"\\n\")\\\n",
        "                     .option(\"quote\", \"\\\"\")\\\n",
        "                     .option(\"escape\", \"\\\"\")\\\n",
        "                     .schema(mySchema)\\\n",
        "                     .csv('/content/data/hashtag_joebiden.csv')\n",
        "\n",
        "biden_df.cache()\n",
        "print('Biden DataFrame Schema')\n",
        "biden_df.printSchema()\n",
        "display(biden_df)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gxHe27iI6B-8"
      },
      "source": [
        "#### Trump Data Stats"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "X4nxWpnQjDzd",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "dcfc5ddd-cbae-448b-fee2-7a7fbbbcb49c"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+-------------------+-------------+--------------------+-----+-------------+-------------------+-------------+--------------------+----------------+--------------------+-------------------+--------------------+--------------------+---------+-----------+-----------------+--------------------+-------------+--------------------+----------+--------------------+\n",
            "|         created_at|     tweet_id|               tweet|likes|retweet_count|             source|      user_id|           user_name|user_screen_name|    user_description|     user_join_date|user_followers_count|       user_location|      lat|       long|             city|             country|    continent|               state|state_code|        collected_at|\n",
            "+-------------------+-------------+--------------------+-----+-------------+-------------------+-------------+--------------------+----------------+--------------------+-------------------+--------------------+--------------------+---------+-----------+-----------------+--------------------+-------------+--------------------+----------+--------------------+\n",
            "|2020-10-15 00:00:01|1.31652925E18|#Elecciones2020 |...|  0.0|          0.0|          TweetDeck| 3.60666528E8|  El Sol Latino News| elsollatinonews|🌐 Noticias de in...|2011-08-23 15:33:45|              1860.0|Philadelphia, PA ...| 25.77427|  -80.19366|             null|United States of ...|North America|             Florida|        FL| 2020-10-21 00:00:00|\n",
            "|2020-10-15 00:00:01|1.31652925E18|Usa 2020, Trump c...| 26.0|          9.0|   Social Mediaset | 3.31617632E8|             Tgcom24| MediasetTgcom24|Profilo ufficiale...|2011-07-08 13:12:20|           1067661.0|                null|     null|       null|             null|                null|         null|                null|      null|2020-10-21 00:00:...|\n",
            "|2020-10-15 00:00:02|1.31652925E18|#Trump: As a stud...|  2.0|          1.0|    Twitter Web App|    8436472.0|              snarke|          snarke|Will mock for foo...|2007-08-26 05:56:11|              1185.0|            Portland| 45.52025|-122.674194|         Portland|United States of ...|North America|              Oregon|        OR|2020-10-21 00:00:...|\n",
            "|2020-10-15 00:00:02|1.31652925E18|2 hours since las...|  0.0|          0.0|      Trumpytweeter| 8.2835557E17|       Trumpytweeter|   trumpytweeter|If he doesn't twe...|2017-02-05 21:32:17|                32.0|                null|     null|       null|             null|                null|         null|                null|      null|2020-10-21 00:00:...|\n",
            "|2020-10-15 00:00:08|1.31652925E18|You get a tie! An...|  4.0|          3.0| Twitter for iPhone|    4.74138E7|Rana Abtar - رنا ...|       Ranaabtar|Washington Corres...|2009-06-15 19:05:35|              5393.0|       Washington DC|38.894993|  -77.03656|       Washington|United States of ...|North America|District of Columbia|        DC|2020-10-21 00:00:...|\n",
            "|2020-10-15 00:00:17|1.31652925E18|@CLady62 Her 15 m...|  2.0|          0.0|Twitter for Android| 1.13841613E9|        Farris Flagg|     FarrisFlagg|#BidenHarris2020 ...|2013-02-01 01:37:38|              2363.0|   Perris,California| 33.78252|-117.228645|             null|United States of ...|North America|          California|        CA|2020-10-21 00:00:...|\n",
            "|2020-10-15 00:00:17|1.31652925E18|@richardmarx Glad...|  0.0|          0.0| Twitter for iPhone| 7.6740187E17|      Michael Wilson|     wilsonfire9|                null|2016-08-21 16:43:51|                75.0|          Powell, TN|     null|       null|             null|                null|         null|                null|      null|2020-10-21 00:00:...|\n",
            "|2020-10-15 00:00:18|1.31652925E18|@DeeviousDenise @...|  0.0|          0.0| Twitter for iPhone| 9.0076109E17|Stacey Gulledge ?...|     sm_gulledge|Patriot, Wife, “S...|2017-08-24 16:45:49|               766.0|           Ohio, USA|40.225357|  -82.68814|             null|United States of ...|North America|                Ohio|        OH|2020-10-21 00:00:...|\n",
            "|2020-10-15 00:00:20|1.31652925E18|One of the single...|  0.0|          0.0|    Twitter Web App|  5.4047686E8|              Jamieo|        jamieo33|Don't know what I...|2012-03-30 00:30:54|               151.0|   Pennsylvania, USA| 40.96999|  -77.72788|             null|United States of ...|North America|        Pennsylvania|        PA|2020-10-21 00:00:...|\n",
            "|2020-10-15 00:00:21|1.31652925E18|#Election2020 #Tr...|  0.0|          0.0|    Twitter Web App|1.30553303E18|        Johnny Quest|   JohnnyQuest22|Independent\\n\\nWh...|2020-09-14 15:45:18|                 8.0|                null|     null|       null|             null|                null|         null|                null|      null|2020-10-21 00:00:...|\n",
            "|2020-10-15 00:00:23|1.31652938E18|\"IS THIS WRONG??!...|  0.0|          0.0|    Twitter Web App| 3.05727904E8|  Dose of Dissonance| Dose_Dissonance|YOUTUBE CHANNEL: ...|2011-05-26 18:09:32|              4622.0|                null|     null|       null|             null|                null|         null|                null|      null|2020-10-21 00:00:...|\n",
            "|2020-10-15 00:00:25|1.31652938E18|In 2020, #NYPost ...|  0.0|          0.0| Twitter for iPhone|  1.9940334E7|Change Illinois |...|  changeillinois|Illinois, home of...|2009-02-02 23:08:28|              1396.0|   Chicago, Illinois| 41.87556|  -87.62442|          Chicago|United States of ...|North America|            Illinois|        IL|2020-10-21 00:00:...|\n",
            "|2020-10-15 00:00:26|1.31652938E18|#Trump #President...|  3.0|          5.0|Twitter for Android|1.24331538E18|        Ron Burgundy|   Anchorman_USA|I'm kind of a Big...|2020-03-26 23:14:28|               496.0|       San Diego, CA|32.717422| -117.16277|        San Diego|United States of ...|North America|          California|        CA|2020-10-21 00:00:...|\n",
            "|2020-10-15 00:00:31|1.31652938E18|@Susan_Hutch @Joe...|  2.0|          0.0|    Twitter Web App|  3.4395616E8|     Sebastian White|       SebWrites|Tortured, literar...|2011-07-28 10:07:32|              2755.0|Edinburgh ~ Heart...|55.953346|  -3.188375|City of Edinburgh|      United Kingdom|       Europe|            Scotland|       SCT|2020-10-21 00:00:...|\n",
            "|2020-10-15 00:00:31|1.31652938E18|►► Tell Politicia...|  0.0|          0.0|      Freebie-Depot| 1.03083216E8|    🆓 Freebie Depot|    FreebieDepot|Free Stuff - No F...|2010-01-08 20:44:21|              6402.0|USA - Land of the...|     null|       null|             null|                null|         null|                null|      null|2020-10-21 00:00:...|\n",
            "|2020-10-15 00:00:42|1.31652938E18|@meiselasb I wond...|  3.0|          1.0|    Twitter Web App| 8.3237765E17|          Mark Rager|     Hervorrager|Diploma-Biologist...|2017-02-16 23:54:29|               828.0|             Germany| 51.08342|  10.423447|             null|             Germany|       Europe|                null|      null|2020-10-21 00:00:...|\n",
            "|2020-10-15 00:00:42|1.31652938E18|#Ukraine @RealDon...|  0.0|          2.0|    Twitter Web App|  3.4395616E8|     Sebastian White|       SebWrites|Tortured, literar...|2011-07-28 10:07:32|              2755.0|Edinburgh ~ Heart...|55.953346|  -3.188375|City of Edinburgh|      United Kingdom|       Europe|            Scotland|       SCT|2020-10-21 00:00:...|\n",
            "|2020-10-15 00:00:44|1.31652938E18|Now Open! Create ...|  0.0|          0.0|     Hootsuite Inc.|  2.1932048E7|     OfficePools.com| officepools_com|Visit https://t.c...|2009-02-25 22:52:43|               967.0|                null|     null|       null|             null|                null|         null|                null|      null|2020-10-21 00:00:...|\n",
            "|2020-10-15 00:00:55|1.31652938E18|#Trump https://t....|  1.0|          1.0|    Twitter Web App|1.08338653E18|              Cheryl|  Cheryl84037124|                null|2019-01-10 15:34:08|                49.0|                null|     null|       null|             null|                null|         null|                null|      null|2020-10-21 00:00:...|\n",
            "|2020-10-15 00:01:08|1.31652952E18|@cnnbrk #Trump ow...|  3.0|          2.0| Twitter for iPhone| 1.94650432E8|           MoClarker|       MoClarker|Media Maven/Scien...|2010-09-24 17:15:43|               101.0|  Santa Monica Beach| 47.00521| -88.962906|             null|United States of ...|North America|            Michigan|        MI|2020-10-21 00:00:...|\n",
            "+-------------------+-------------+--------------------+-----+-------------+-------------------+-------------+--------------------+----------------+--------------------+-------------------+--------------------+--------------------+---------+-----------+-----------------+--------------------+-------------+--------------------+----------+--------------------+\n",
            "only showing top 20 rows\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# Took me 35 seconds to run\n",
        "trump_df.show(20)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "id": "Pp_hw_UklPSG",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6bfd7390-ecfc-41b5-d217-87ac00fa3d78"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+-------+-------------------+--------------------+--------------------+------------------+------------------+--------------+--------------------+--------------------+-------------------+--------------------+-------------------+--------------------+-------------+------------------+------------------+--------------------+-----------+-------------+----------------+----------+--------------------+\n",
            "|summary|         created_at|            tweet_id|               tweet|             likes|     retweet_count|        source|             user_id|           user_name|   user_screen_name|    user_description|     user_join_date|user_followers_count|user_location|               lat|              long|                city|    country|    continent|           state|state_code|        collected_at|\n",
            "+-------+-------------------+--------------------+--------------------+------------------+------------------+--------------+--------------------+--------------------+-------------------+--------------------+-------------------+--------------------+-------------+------------------+------------------+--------------------+-----------+-------------+----------------+----------+--------------------+\n",
            "|  count|             970919|              970919|              970919|            970919|            970919|        970043|              970919|              970903|             970919|              869653|             970919|              970919|       675966|            445719|            445719|              227187|     442748|       442765|          320620|    300425|              970919|\n",
            "|   mean|               null|      1.322493581E18|                null|7.4770109555997974|1.6985000808512347|          null|4.468311179391641...|                 NaN|   8.29344233856E10|6.616298555620129...|               null|  22603.570877694227|     Infinity| 35.69793596221496|-40.36963760259293|                null|       null|         null|            null|      null|                null|\n",
            "| stddev|               null|2.555133051536166E15|                null| 158.0581171022403|40.028419304682146|          null|5.544701551209815E17|                 NaN|5.53691625376839E11|9.835834918063301...|               null|   304215.1701738431|          NaN|18.823129287921375| 67.53175083179354|                null|       null|         null|            null|      null|                null|\n",
            "|    min|2020-10-15 00:00:01|       1.31652925E18|!\\n#DonaldTrump😂...|               0.0|               0.0|101soundboards|               531.0|   B-Smooth Cromwell|           00001Kat|                   !|2006-07-12 23:17:01|                 0.0|           \\t|             -90.0|        -175.20264|        (Old) Ottawa|Afghanistan|       Africa|          Aargau|         A| 2020-10-21 00:00:00|\n",
            "|    25%|               null|       1.32047801E18|                null|               0.0|               0.0|          null|        2.16943712E8|                22.0|           540856.0|                16.0|               null|                77.0|         33.0|          32.70194|         -96.79685|                null|       null|         null|            null|      null|                null|\n",
            "|    50%|               null|       1.32361189E18|                null|               0.0|               0.0|          null|        2.37231411E9|              211.11|          9072021.0|                32.0|               null|               441.0|        616.0|          39.78373|         -74.00601|                null|       null|         null|            null|      null|                null|\n",
            "|    75%|               null|       1.32450469E18|                null|               1.0|               0.0|          null|        1.0780405E18|              2020.0|        1.0382238E7|              1312.0|               null|              2066.0|      23601.0|         46.603355|          6.776314|                null|       null|         null|            null|      null|                null|\n",
            "|    max|2020-11-08 23:59:56|       1.32558895E18|🫂 🤌🏼\\n\\n#الانت...|           74084.0|           20491.0|        禁闻网|       1.32558125E18|🪴🌌🇺🇸 Sheri Ho...|            zzzzzme|   🪐🦸🏻‍♂️☄️🏳️‍🌈|2020-11-08 23:29:53|         1.9115332E7|           🪔|              90.0|         179.04884|مجمع كافوري السكني 1|   Zimbabwe|South America|Łódź Voivodeship|        ZP|2020-11-09 17:57:...|\n",
            "+-------+-------------------+--------------------+--------------------+------------------+------------------+--------------+--------------------+--------------------+-------------------+--------------------+-------------------+--------------------+-------------+------------------+------------------+--------------------+-----------+-------------+----------------+----------+--------------------+\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# Took me 2 minutes to run one command carefull while running\n",
        "trump_df.summary().show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SK96TGda6FnP"
      },
      "source": [
        "#### Biden Data Stats"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "id": "k0BjcKm9Va_3",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "87e402f3-1374-4a49-ae35-1abbd9e1e130"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+-------------------+-------------+--------------------+-----+-------------+------------------+-------------+------------------+----------------+--------------------+-------------------+--------------------+--------------------+---------+----------+----+--------------------+-------------+----------+----------+--------------------+\n",
            "|         created_at|     tweet_id|               tweet|likes|retweet_count|            source|      user_id|         user_name|user_screen_name|    user_description|     user_join_date|user_followers_count|       user_location|      lat|      long|city|             country|    continent|     state|state_code|        collected_at|\n",
            "+-------------------+-------------+--------------------+-----+-------------+------------------+-------------+------------------+----------------+--------------------+-------------------+--------------------+--------------------+---------+----------+----+--------------------+-------------+----------+----------+--------------------+\n",
            "|2020-10-15 00:00:01|1.31652925E18|#Elecciones2020 |...|  0.0|          0.0|         TweetDeck| 3.60666528E8|El Sol Latino News| elsollatinonews|🌐 Noticias de in...|2011-08-23 15:33:45|              1860.0|Philadelphia, PA ...| 25.77427| -80.19366|null|United States of ...|North America|   Florida|        FL| 2020-10-21 00:00:00|\n",
            "|2020-10-15 00:00:18|1.31652925E18|#HunterBiden #Hun...|  0.0|          0.0|  Twitter for iPad|  8.0990445E8|     Cheri A. 🇺🇸|    Biloximeemaw|Locked and loaded...|2012-09-08 01:03:57|              6628.0|                null|     null|      null|null|                null|         null|      null|      null|2020-10-21 00:00:...|\n",
            "|2020-10-15 00:00:20|1.31652925E18|@IslandGirlPRV @B...|  0.0|          0.0|   Twitter Web App|  3.4941824E9|        Flag Waver|     Flag_Wavers|                null|2015-08-30 19:38:21|              1536.0|Golden Valley Ari...|46.304035|-109.17143|null|United States of ...|North America|   Montana|        MT|2020-10-21 00:00:...|\n",
            "|2020-10-15 00:00:21|1.31652925E18|@chrislongview Wa...|  0.0|          0.0|Twitter for iPhone| 8.2425961E17|     Michelle Ferg|   MichelleFerg4|                null|2017-01-25 14:16:17|                27.0|                null|     null|      null|null|                null|         null|      null|      null|2020-10-21 00:00:...|\n",
            "|2020-10-15 00:00:22|1.31652925E18|#censorship #Hunt...|  1.0|          0.0|   Twitter Web App|1.03280694E18|    the Gold State|   theegoldstate|A Silicon Valley ...|2018-08-24 01:49:01|               390.0|     California, USA| 36.70146|  -118.756|null|United States of ...|North America|California|        CA|2020-10-21 00:00:...|\n",
            "+-------------------+-------------+--------------------+-----+-------------+------------------+-------------+------------------+----------------+--------------------+-------------------+--------------------+--------------------+---------+----------+----+--------------------+-------------+----------+----------+--------------------+\n",
            "only showing top 5 rows\n",
            "\n"
          ]
        }
      ],
      "source": [
        "biden_df.show(5)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "id": "X5FlV8Za4XlM",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b1003483-6fd2-4dc9-8319-7caf27de07eb"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+-------+-------------------+--------------------+--------------------+------------------+-----------------+--------------------+--------------------+-------------+--------------------+--------------------+-------------------+--------------------+-------------+------------------+------------------+------------+-----------+-------------+----------------+----------+--------------------+\n",
            "|summary|         created_at|            tweet_id|               tweet|             likes|    retweet_count|              source|             user_id|    user_name|    user_screen_name|    user_description|     user_join_date|user_followers_count|user_location|               lat|              long|        city|    country|    continent|           state|state_code|        collected_at|\n",
            "+-------+-------------------+--------------------+--------------------+------------------+-----------------+--------------------+--------------------+-------------+--------------------+--------------------+-------------------+--------------------+-------------+------------------+------------------+------------+-----------+-------------+----------------+----------+--------------------+\n",
            "|  count|             776886|              776886|              776886|            776886|           776886|              776173|              776886|       776868|              776886|              694880|             776886|              776886|       543095|            355293|            355293|      186872|     353779|       353797|          260195|    244609|              776886|\n",
            "|   mean|               null|1.322843920233239...|                null|10.161163928813236|2.131330722911727|                null|4.532033316628750...|          NaN|2.667938119987417...|6.393044367152595...|               null|  28849.941214026254|     Infinity|35.104202478340454|-41.97966042024943|        null|       null|         null|            null|      null|                null|\n",
            "| stddev|               null|2.585560115570453...|                null| 390.9744227814309|96.66152719420691|                null|5.573483072428794...|          NaN|3.123624233461713E11|8.035932162668051E33|               null|  414108.47934963374|          NaN|17.907910231104253| 67.82372395086975|        null|       null|         null|            null|      null|                null|\n",
            "|    min|2020-10-15 00:00:01|       1.31652925E18|!\\nhttps://t.co/j...|               0.0|              0.0|#pinnalla READ/WRITE|              1081.0| Sille Kjærbo|           0000000ef|                   !|1970-01-01 00:00:00|                 0.0|           \\t|         -79.40631|        -172.12006|(Old) Ottawa|Afghanistan|       Africa|          Aargau|         A| 2020-10-21 00:00:00|\n",
            "|    25%|               null|       1.32086957E18|                null|               0.0|              0.0|                null|        2.25812752E8|         22.0|           1515243.0|                19.0|               null|                74.0|         67.0|         31.816038|         -97.26841|        null|       null|         null|            null|      null|                null|\n",
            "|    50%|               null|       1.32389845E18|                null|               0.0|              0.0|                null|        2.46326938E9|         22.0|           1515243.0|                28.0|               null|               428.0|        757.0|          39.78373|         -74.00601|        null|       null|         null|            null|      null|                null|\n",
            "|    75%|               null|       1.32511506E18|                null|               2.0|              0.0|                null|       1.08810474E18|        718.0|           1515243.0|              1949.0|               null|              2082.0|      20815.0|          43.65348|          5.966069|        null|       null|         null|            null|      null|                null|\n",
            "|    max|2020-11-08 23:59:58|       1.32558895E18|🫂 \"No hay estado...|          165702.0|          63473.0|ツイタマ for Android|       1.32557465E18|         🪔🔱|            zzzzooop|🪐⛓ 𝓁𝓊𝓋𝒾𝓃’ ?...|2020-11-08 23:03:18|         8.2417096E7|           🪔|              90.0|         179.04884|        Łódź|   Zimbabwe|South America|Łódź Voivodeship|        ZP|2020-11-09 18:40:...|\n",
            "+-------+-------------------+--------------------+--------------------+------------------+-----------------+--------------------+--------------------+-------------+--------------------+--------------------+-------------------+--------------------+-------------+------------------+------------------+------------+-----------+-------------+----------------+----------+--------------------+\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# Took me 1 minute 21 seconds to run this command carefull while running\n",
        "biden_df.summary().show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_HSBIeJN6_sM"
      },
      "source": [
        "#### Concatinating Data "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "id": "1sjw5nxc4bRQ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6a630303-1418-4df7-f216-dc446cc96755"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+-------------------+-------------+--------------------+-----+-------------+-------------------+-------------+--------------------+----------------+--------------------+-------------------+--------------------+--------------------+---------+-----------+-----------------+--------------------+-------------+--------------------+----------+--------------------+---------+\n",
            "|         created_at|     tweet_id|               tweet|likes|retweet_count|             source|      user_id|           user_name|user_screen_name|    user_description|     user_join_date|user_followers_count|       user_location|      lat|       long|             city|             country|    continent|               state|state_code|        collected_at|Candidate|\n",
            "+-------------------+-------------+--------------------+-----+-------------+-------------------+-------------+--------------------+----------------+--------------------+-------------------+--------------------+--------------------+---------+-----------+-----------------+--------------------+-------------+--------------------+----------+--------------------+---------+\n",
            "|2020-10-15 00:00:01|1.31652925E18|#Elecciones2020 |...|  0.0|          0.0|          TweetDeck| 3.60666528E8|  El Sol Latino News| elsollatinonews|🌐 Noticias de in...|2011-08-23 15:33:45|              1860.0|Philadelphia, PA ...| 25.77427|  -80.19366|             null|United States of ...|North America|             Florida|        FL| 2020-10-21 00:00:00|    TRUMP|\n",
            "|2020-10-15 00:00:01|1.31652925E18|Usa 2020, Trump c...| 26.0|          9.0|   Social Mediaset | 3.31617632E8|             Tgcom24| MediasetTgcom24|Profilo ufficiale...|2011-07-08 13:12:20|           1067661.0|                null|     null|       null|             null|                null|         null|                null|      null|2020-10-21 00:00:...|    TRUMP|\n",
            "|2020-10-15 00:00:02|1.31652925E18|#Trump: As a stud...|  2.0|          1.0|    Twitter Web App|    8436472.0|              snarke|          snarke|Will mock for foo...|2007-08-26 05:56:11|              1185.0|            Portland| 45.52025|-122.674194|         Portland|United States of ...|North America|              Oregon|        OR|2020-10-21 00:00:...|    TRUMP|\n",
            "|2020-10-15 00:00:02|1.31652925E18|2 hours since las...|  0.0|          0.0|      Trumpytweeter| 8.2835557E17|       Trumpytweeter|   trumpytweeter|If he doesn't twe...|2017-02-05 21:32:17|                32.0|                null|     null|       null|             null|                null|         null|                null|      null|2020-10-21 00:00:...|    TRUMP|\n",
            "|2020-10-15 00:00:08|1.31652925E18|You get a tie! An...|  4.0|          3.0| Twitter for iPhone|    4.74138E7|Rana Abtar - رنا ...|       Ranaabtar|Washington Corres...|2009-06-15 19:05:35|              5393.0|       Washington DC|38.894993|  -77.03656|       Washington|United States of ...|North America|District of Columbia|        DC|2020-10-21 00:00:...|    TRUMP|\n",
            "|2020-10-15 00:00:17|1.31652925E18|@CLady62 Her 15 m...|  2.0|          0.0|Twitter for Android| 1.13841613E9|        Farris Flagg|     FarrisFlagg|#BidenHarris2020 ...|2013-02-01 01:37:38|              2363.0|   Perris,California| 33.78252|-117.228645|             null|United States of ...|North America|          California|        CA|2020-10-21 00:00:...|    TRUMP|\n",
            "|2020-10-15 00:00:17|1.31652925E18|@richardmarx Glad...|  0.0|          0.0| Twitter for iPhone| 7.6740187E17|      Michael Wilson|     wilsonfire9|                null|2016-08-21 16:43:51|                75.0|          Powell, TN|     null|       null|             null|                null|         null|                null|      null|2020-10-21 00:00:...|    TRUMP|\n",
            "|2020-10-15 00:00:18|1.31652925E18|@DeeviousDenise @...|  0.0|          0.0| Twitter for iPhone| 9.0076109E17|Stacey Gulledge ?...|     sm_gulledge|Patriot, Wife, “S...|2017-08-24 16:45:49|               766.0|           Ohio, USA|40.225357|  -82.68814|             null|United States of ...|North America|                Ohio|        OH|2020-10-21 00:00:...|    TRUMP|\n",
            "|2020-10-15 00:00:20|1.31652925E18|One of the single...|  0.0|          0.0|    Twitter Web App|  5.4047686E8|              Jamieo|        jamieo33|Don't know what I...|2012-03-30 00:30:54|               151.0|   Pennsylvania, USA| 40.96999|  -77.72788|             null|United States of ...|North America|        Pennsylvania|        PA|2020-10-21 00:00:...|    TRUMP|\n",
            "|2020-10-15 00:00:21|1.31652925E18|#Election2020 #Tr...|  0.0|          0.0|    Twitter Web App|1.30553303E18|        Johnny Quest|   JohnnyQuest22|Independent\\n\\nWh...|2020-09-14 15:45:18|                 8.0|                null|     null|       null|             null|                null|         null|                null|      null|2020-10-21 00:00:...|    TRUMP|\n",
            "|2020-10-15 00:00:23|1.31652938E18|\"IS THIS WRONG??!...|  0.0|          0.0|    Twitter Web App| 3.05727904E8|  Dose of Dissonance| Dose_Dissonance|YOUTUBE CHANNEL: ...|2011-05-26 18:09:32|              4622.0|                null|     null|       null|             null|                null|         null|                null|      null|2020-10-21 00:00:...|    TRUMP|\n",
            "|2020-10-15 00:00:25|1.31652938E18|In 2020, #NYPost ...|  0.0|          0.0| Twitter for iPhone|  1.9940334E7|Change Illinois |...|  changeillinois|Illinois, home of...|2009-02-02 23:08:28|              1396.0|   Chicago, Illinois| 41.87556|  -87.62442|          Chicago|United States of ...|North America|            Illinois|        IL|2020-10-21 00:00:...|    TRUMP|\n",
            "|2020-10-15 00:00:26|1.31652938E18|#Trump #President...|  3.0|          5.0|Twitter for Android|1.24331538E18|        Ron Burgundy|   Anchorman_USA|I'm kind of a Big...|2020-03-26 23:14:28|               496.0|       San Diego, CA|32.717422| -117.16277|        San Diego|United States of ...|North America|          California|        CA|2020-10-21 00:00:...|    TRUMP|\n",
            "|2020-10-15 00:00:31|1.31652938E18|@Susan_Hutch @Joe...|  2.0|          0.0|    Twitter Web App|  3.4395616E8|     Sebastian White|       SebWrites|Tortured, literar...|2011-07-28 10:07:32|              2755.0|Edinburgh ~ Heart...|55.953346|  -3.188375|City of Edinburgh|      United Kingdom|       Europe|            Scotland|       SCT|2020-10-21 00:00:...|    TRUMP|\n",
            "|2020-10-15 00:00:31|1.31652938E18|►► Tell Politicia...|  0.0|          0.0|      Freebie-Depot| 1.03083216E8|    🆓 Freebie Depot|    FreebieDepot|Free Stuff - No F...|2010-01-08 20:44:21|              6402.0|USA - Land of the...|     null|       null|             null|                null|         null|                null|      null|2020-10-21 00:00:...|    TRUMP|\n",
            "|2020-10-15 00:00:42|1.31652938E18|@meiselasb I wond...|  3.0|          1.0|    Twitter Web App| 8.3237765E17|          Mark Rager|     Hervorrager|Diploma-Biologist...|2017-02-16 23:54:29|               828.0|             Germany| 51.08342|  10.423447|             null|             Germany|       Europe|                null|      null|2020-10-21 00:00:...|    TRUMP|\n",
            "|2020-10-15 00:00:42|1.31652938E18|#Ukraine @RealDon...|  0.0|          2.0|    Twitter Web App|  3.4395616E8|     Sebastian White|       SebWrites|Tortured, literar...|2011-07-28 10:07:32|              2755.0|Edinburgh ~ Heart...|55.953346|  -3.188375|City of Edinburgh|      United Kingdom|       Europe|            Scotland|       SCT|2020-10-21 00:00:...|    TRUMP|\n",
            "|2020-10-15 00:00:44|1.31652938E18|Now Open! Create ...|  0.0|          0.0|     Hootsuite Inc.|  2.1932048E7|     OfficePools.com| officepools_com|Visit https://t.c...|2009-02-25 22:52:43|               967.0|                null|     null|       null|             null|                null|         null|                null|      null|2020-10-21 00:00:...|    TRUMP|\n",
            "|2020-10-15 00:00:55|1.31652938E18|#Trump https://t....|  1.0|          1.0|    Twitter Web App|1.08338653E18|              Cheryl|  Cheryl84037124|                null|2019-01-10 15:34:08|                49.0|                null|     null|       null|             null|                null|         null|                null|      null|2020-10-21 00:00:...|    TRUMP|\n",
            "|2020-10-15 00:01:08|1.31652952E18|@cnnbrk #Trump ow...|  3.0|          2.0| Twitter for iPhone| 1.94650432E8|           MoClarker|       MoClarker|Media Maven/Scien...|2010-09-24 17:15:43|               101.0|  Santa Monica Beach| 47.00521| -88.962906|             null|United States of ...|North America|            Michigan|        MI|2020-10-21 00:00:...|    TRUMP|\n",
            "+-------------------+-------------+--------------------+-----+-------------+-------------------+-------------+--------------------+----------------+--------------------+-------------------+--------------------+--------------------+---------+-----------+-----------------+--------------------+-------------+--------------------+----------+--------------------+---------+\n",
            "only showing top 20 rows\n",
            "\n"
          ]
        }
      ],
      "source": [
        "## Adding a column to the dataframe \n",
        "trump_df = trump_df.withColumn(\"Candidate\", lit('TRUMP'))\n",
        "trump_df.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "id": "KbC4koSq8KCh",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7af1d552-0f2f-442b-d48d-0b139db9e47e"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+-------------------+-------------+--------------------+-----+-------------+-------------------+-------------+--------------------+----------------+--------------------+-------------------+--------------------+--------------------+---------+----------+-------+--------------------+-------------+----------+----------+--------------------+---------+\n",
            "|         created_at|     tweet_id|               tweet|likes|retweet_count|             source|      user_id|           user_name|user_screen_name|    user_description|     user_join_date|user_followers_count|       user_location|      lat|      long|   city|             country|    continent|     state|state_code|        collected_at|Candidate|\n",
            "+-------------------+-------------+--------------------+-----+-------------+-------------------+-------------+--------------------+----------------+--------------------+-------------------+--------------------+--------------------+---------+----------+-------+--------------------+-------------+----------+----------+--------------------+---------+\n",
            "|2020-10-15 00:00:01|1.31652925E18|#Elecciones2020 |...|  0.0|          0.0|          TweetDeck| 3.60666528E8|  El Sol Latino News| elsollatinonews|🌐 Noticias de in...|2011-08-23 15:33:45|              1860.0|Philadelphia, PA ...| 25.77427| -80.19366|   null|United States of ...|North America|   Florida|        FL| 2020-10-21 00:00:00|    BIDEN|\n",
            "|2020-10-15 00:00:18|1.31652925E18|#HunterBiden #Hun...|  0.0|          0.0|   Twitter for iPad|  8.0990445E8|       Cheri A. 🇺🇸|    Biloximeemaw|Locked and loaded...|2012-09-08 01:03:57|              6628.0|                null|     null|      null|   null|                null|         null|      null|      null|2020-10-21 00:00:...|    BIDEN|\n",
            "|2020-10-15 00:00:20|1.31652925E18|@IslandGirlPRV @B...|  0.0|          0.0|    Twitter Web App|  3.4941824E9|          Flag Waver|     Flag_Wavers|                null|2015-08-30 19:38:21|              1536.0|Golden Valley Ari...|46.304035|-109.17143|   null|United States of ...|North America|   Montana|        MT|2020-10-21 00:00:...|    BIDEN|\n",
            "|2020-10-15 00:00:21|1.31652925E18|@chrislongview Wa...|  0.0|          0.0| Twitter for iPhone| 8.2425961E17|       Michelle Ferg|   MichelleFerg4|                null|2017-01-25 14:16:17|                27.0|                null|     null|      null|   null|                null|         null|      null|      null|2020-10-21 00:00:...|    BIDEN|\n",
            "|2020-10-15 00:00:22|1.31652925E18|#censorship #Hunt...|  1.0|          0.0|    Twitter Web App|1.03280694E18|      the Gold State|   theegoldstate|A Silicon Valley ...|2018-08-24 01:49:01|               390.0|     California, USA| 36.70146|  -118.756|   null|United States of ...|North America|California|        CA|2020-10-21 00:00:...|    BIDEN|\n",
            "|2020-10-15 00:00:23|1.31652938E18|\"IS THIS WRONG??!...|  0.0|          0.0|    Twitter Web App| 3.05727904E8|  Dose of Dissonance| Dose_Dissonance|YOUTUBE CHANNEL: ...|2011-05-26 18:09:32|              4622.0|                null|     null|      null|   null|                null|         null|      null|      null|2020-10-21 00:00:...|    BIDEN|\n",
            "|2020-10-15 00:00:25|1.31652938E18|In 2020, #NYPost ...|  0.0|          0.0| Twitter for iPhone|  1.9940334E7|Change Illinois |...|  changeillinois|Illinois, home of...|2009-02-02 23:08:28|              1397.0|   Chicago, Illinois| 41.87556| -87.62442|Chicago|United States of ...|North America|  Illinois|        IL|2020-10-21 00:00:...|    BIDEN|\n",
            "|2020-10-15 00:00:31|1.31652938E18|►► Tell Politicia...|  0.0|          0.0|      Freebie-Depot| 1.03083216E8|    🆓 Freebie Depot|    FreebieDepot|Free Stuff - No F...|2010-01-08 20:44:21|              6402.0|USA - Land of the...|     null|      null|   null|                null|         null|      null|      null|2020-10-21 00:00:...|    BIDEN|\n",
            "|2020-10-15 00:00:36|1.31652938E18|#Biden https://t....|  0.0|          0.0|    Twitter Web App|1.27256844E18|             Kenneth|   Kennethuteliv|                null|2020-06-15 16:35:31|               103.0|                null|     null|      null|   null|                null|         null|      null|      null|2020-10-21 00:00:...|    BIDEN|\n",
            "|2020-10-15 00:00:41|1.31652938E18|Proof  Bidens are...|  1.0|          1.0|Twitter for Android|1.27565848E18|      Jake Rivendale|   JakeRivendale|                null|2020-06-24 05:15:06|                80.0|                null|     null|      null|   null|                null|         null|      null|      null|2020-10-21 00:00:...|    BIDEN|\n",
            "|2020-10-15 00:00:44|1.31652938E18|Now Open! Create ...|  0.0|          0.0|     Hootsuite Inc.|  2.1932048E7|     OfficePools.com| officepools_com|Visit https://t.c...|2009-02-25 22:52:43|               967.0|                null|     null|      null|   null|                null|         null|      null|      null|2020-10-21 00:00:...|    BIDEN|\n",
            "|2020-10-15 00:00:57|1.31652952E18|FBI Allegedly Obt...|  0.0|          0.0|Twitter for Android| 9.6073869E17|           RLCompton|  NewfoundStudio|\"There are things...|2018-02-06 04:55:35|               266.0|       Kentucky, USA|  37.5726| -85.15514|   null|United States of ...|North America|  Kentucky|        KY|2020-10-21 00:00:...|    BIDEN|\n",
            "|2020-10-15 00:01:03|1.31652952E18|#JoeBiden was the...|  0.0|          1.0|Twitter for Android|1.24026891E18|       Lilmissnoname|   TwiggyTulip65|Proud Republican ...|2020-03-18 13:28:56|               190.0|                null|     null|      null|   null|                null|         null|      null|      null|2020-10-21 00:00:...|    BIDEN|\n",
            "|2020-10-15 00:01:15|1.31652952E18|Y’all Just Lockin...|  0.0|          0.0|       ReverbNation|   8.341048E7|🆎➕TÿÇkØ🇧🇸 🐲🐉...|    HipHopMatrix|#ARAWAK (https://...|2009-10-18 18:07:33|              4373.0|              H-Town|     null|      null|   null|                null|         null|      null|      null|2020-10-21 00:00:...|    BIDEN|\n",
            "|2020-10-15 00:01:16|1.31652952E18|@tedcruz @cc125 #...|  1.0|          0.0|    Twitter Web App|1.31004007E18|        Infamous One|   InfamousOne13|\"The company you ...|2020-09-27 02:14:26|               360.0|                null|     null|      null|   null|                null|         null|      null|      null|2020-10-21 00:00:...|    BIDEN|\n",
            "|2020-10-15 00:01:20|1.31652952E18|#IceCube isn’t a ...|  1.0|          1.0| Twitter for iPhone| 2.85131936E8|𝓐𝓵𝓮𝔁𝓲𝓪 𝓜𝓪...|   alexiamariee_|snap: 𝒜𝓁𝑒𝓍𝒾?...|2011-04-20 15:50:33|               582.0|            🇨🇱🇧🇸|     null|      null|   null|                null|         null|      null|      null|2020-10-21 00:00:...|    BIDEN|\n",
            "|2020-10-15 00:01:20|1.31652952E18|BREAKING — Twitte...|  1.0|          1.0| Twitter for iPhone|1.24722332E18|       Insider Paper| TheInsiderPaper|Insider Paper is ...|2020-04-06 18:09:10|              1200.0|                null|     null|      null|   null|                null|         null|      null|      null|2020-10-21 00:00:...|    BIDEN|\n",
            "|2020-10-15 00:01:23|1.31652952E18|Comments on this?...|  0.0|          0.0|    Twitter Web App|  1.0165928E8|         John Ubaldi|   ubaldireports|Just Facts... No ...|2010-01-04 04:34:39|                83.0|      Tampa, Florida| 27.94776| -82.45844|  Tampa|United States of ...|North America|   Florida|        FL|2020-10-21 00:00:...|    BIDEN|\n",
            "|2020-10-15 00:01:23|1.31652952E18|https://t.co/khrZ...|  0.0|          0.0|    Twitter Web App| 9.3317043E17|                 4u2|   whipit_Studio|                null|2017-11-22 03:09:08|                11.0|                null|     null|      null|   null|                null|         null|      null|      null|2020-10-21 00:00:...|    BIDEN|\n",
            "|2020-10-15 00:01:27|1.31652952E18|I’m going to shar...|  0.0|          0.0| Twitter for iPhone|  9.1536928E8|       Sammy McCarty|   SammyTMcCarty|Sammy priorities ...|2012-10-30 19:50:37|               394.0|       Rosemount, MN|     null|      null|   null|                null|         null|      null|      null|2020-10-21 00:00:...|    BIDEN|\n",
            "+-------------------+-------------+--------------------+-----+-------------+-------------------+-------------+--------------------+----------------+--------------------+-------------------+--------------------+--------------------+---------+----------+-------+--------------------+-------------+----------+----------+--------------------+---------+\n",
            "only showing top 20 rows\n",
            "\n"
          ]
        }
      ],
      "source": [
        "biden_df = biden_df.withColumn(\"Candidate\", lit('BIDEN'))\n",
        "biden_df.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "id": "4m1vgbR08YFK",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "fbefd4fe-1062-4c86-f538-4f9452aba94f"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+-------------------+-------------+--------------------+-----+-------------+-------------------+-------------+--------------------+----------------+--------------------+-------------------+--------------------+--------------------+---------+-----------+-----------------+--------------------+-------------+--------------------+----------+--------------------+---------+\n",
            "|         created_at|     tweet_id|               tweet|likes|retweet_count|             source|      user_id|           user_name|user_screen_name|    user_description|     user_join_date|user_followers_count|       user_location|      lat|       long|             city|             country|    continent|               state|state_code|        collected_at|Candidate|\n",
            "+-------------------+-------------+--------------------+-----+-------------+-------------------+-------------+--------------------+----------------+--------------------+-------------------+--------------------+--------------------+---------+-----------+-----------------+--------------------+-------------+--------------------+----------+--------------------+---------+\n",
            "|2020-10-15 00:00:01|1.31652925E18|#Elecciones2020 |...|  0.0|          0.0|          TweetDeck| 3.60666528E8|  El Sol Latino News| elsollatinonews|🌐 Noticias de in...|2011-08-23 15:33:45|              1860.0|Philadelphia, PA ...| 25.77427|  -80.19366|             null|United States of ...|North America|             Florida|        FL| 2020-10-21 00:00:00|    TRUMP|\n",
            "|2020-10-15 00:00:01|1.31652925E18|Usa 2020, Trump c...| 26.0|          9.0|   Social Mediaset | 3.31617632E8|             Tgcom24| MediasetTgcom24|Profilo ufficiale...|2011-07-08 13:12:20|           1067661.0|                null|     null|       null|             null|                null|         null|                null|      null|2020-10-21 00:00:...|    TRUMP|\n",
            "|2020-10-15 00:00:02|1.31652925E18|#Trump: As a stud...|  2.0|          1.0|    Twitter Web App|    8436472.0|              snarke|          snarke|Will mock for foo...|2007-08-26 05:56:11|              1185.0|            Portland| 45.52025|-122.674194|         Portland|United States of ...|North America|              Oregon|        OR|2020-10-21 00:00:...|    TRUMP|\n",
            "|2020-10-15 00:00:02|1.31652925E18|2 hours since las...|  0.0|          0.0|      Trumpytweeter| 8.2835557E17|       Trumpytweeter|   trumpytweeter|If he doesn't twe...|2017-02-05 21:32:17|                32.0|                null|     null|       null|             null|                null|         null|                null|      null|2020-10-21 00:00:...|    TRUMP|\n",
            "|2020-10-15 00:00:08|1.31652925E18|You get a tie! An...|  4.0|          3.0| Twitter for iPhone|    4.74138E7|Rana Abtar - رنا ...|       Ranaabtar|Washington Corres...|2009-06-15 19:05:35|              5393.0|       Washington DC|38.894993|  -77.03656|       Washington|United States of ...|North America|District of Columbia|        DC|2020-10-21 00:00:...|    TRUMP|\n",
            "|2020-10-15 00:00:17|1.31652925E18|@CLady62 Her 15 m...|  2.0|          0.0|Twitter for Android| 1.13841613E9|        Farris Flagg|     FarrisFlagg|#BidenHarris2020 ...|2013-02-01 01:37:38|              2363.0|   Perris,California| 33.78252|-117.228645|             null|United States of ...|North America|          California|        CA|2020-10-21 00:00:...|    TRUMP|\n",
            "|2020-10-15 00:00:17|1.31652925E18|@richardmarx Glad...|  0.0|          0.0| Twitter for iPhone| 7.6740187E17|      Michael Wilson|     wilsonfire9|                null|2016-08-21 16:43:51|                75.0|          Powell, TN|     null|       null|             null|                null|         null|                null|      null|2020-10-21 00:00:...|    TRUMP|\n",
            "|2020-10-15 00:00:18|1.31652925E18|@DeeviousDenise @...|  0.0|          0.0| Twitter for iPhone| 9.0076109E17|Stacey Gulledge ?...|     sm_gulledge|Patriot, Wife, “S...|2017-08-24 16:45:49|               766.0|           Ohio, USA|40.225357|  -82.68814|             null|United States of ...|North America|                Ohio|        OH|2020-10-21 00:00:...|    TRUMP|\n",
            "|2020-10-15 00:00:20|1.31652925E18|One of the single...|  0.0|          0.0|    Twitter Web App|  5.4047686E8|              Jamieo|        jamieo33|Don't know what I...|2012-03-30 00:30:54|               151.0|   Pennsylvania, USA| 40.96999|  -77.72788|             null|United States of ...|North America|        Pennsylvania|        PA|2020-10-21 00:00:...|    TRUMP|\n",
            "|2020-10-15 00:00:21|1.31652925E18|#Election2020 #Tr...|  0.0|          0.0|    Twitter Web App|1.30553303E18|        Johnny Quest|   JohnnyQuest22|Independent\\n\\nWh...|2020-09-14 15:45:18|                 8.0|                null|     null|       null|             null|                null|         null|                null|      null|2020-10-21 00:00:...|    TRUMP|\n",
            "|2020-10-15 00:00:23|1.31652938E18|\"IS THIS WRONG??!...|  0.0|          0.0|    Twitter Web App| 3.05727904E8|  Dose of Dissonance| Dose_Dissonance|YOUTUBE CHANNEL: ...|2011-05-26 18:09:32|              4622.0|                null|     null|       null|             null|                null|         null|                null|      null|2020-10-21 00:00:...|    TRUMP|\n",
            "|2020-10-15 00:00:25|1.31652938E18|In 2020, #NYPost ...|  0.0|          0.0| Twitter for iPhone|  1.9940334E7|Change Illinois |...|  changeillinois|Illinois, home of...|2009-02-02 23:08:28|              1396.0|   Chicago, Illinois| 41.87556|  -87.62442|          Chicago|United States of ...|North America|            Illinois|        IL|2020-10-21 00:00:...|    TRUMP|\n",
            "|2020-10-15 00:00:26|1.31652938E18|#Trump #President...|  3.0|          5.0|Twitter for Android|1.24331538E18|        Ron Burgundy|   Anchorman_USA|I'm kind of a Big...|2020-03-26 23:14:28|               496.0|       San Diego, CA|32.717422| -117.16277|        San Diego|United States of ...|North America|          California|        CA|2020-10-21 00:00:...|    TRUMP|\n",
            "|2020-10-15 00:00:31|1.31652938E18|@Susan_Hutch @Joe...|  2.0|          0.0|    Twitter Web App|  3.4395616E8|     Sebastian White|       SebWrites|Tortured, literar...|2011-07-28 10:07:32|              2755.0|Edinburgh ~ Heart...|55.953346|  -3.188375|City of Edinburgh|      United Kingdom|       Europe|            Scotland|       SCT|2020-10-21 00:00:...|    TRUMP|\n",
            "|2020-10-15 00:00:31|1.31652938E18|►► Tell Politicia...|  0.0|          0.0|      Freebie-Depot| 1.03083216E8|    🆓 Freebie Depot|    FreebieDepot|Free Stuff - No F...|2010-01-08 20:44:21|              6402.0|USA - Land of the...|     null|       null|             null|                null|         null|                null|      null|2020-10-21 00:00:...|    TRUMP|\n",
            "|2020-10-15 00:00:42|1.31652938E18|@meiselasb I wond...|  3.0|          1.0|    Twitter Web App| 8.3237765E17|          Mark Rager|     Hervorrager|Diploma-Biologist...|2017-02-16 23:54:29|               828.0|             Germany| 51.08342|  10.423447|             null|             Germany|       Europe|                null|      null|2020-10-21 00:00:...|    TRUMP|\n",
            "|2020-10-15 00:00:42|1.31652938E18|#Ukraine @RealDon...|  0.0|          2.0|    Twitter Web App|  3.4395616E8|     Sebastian White|       SebWrites|Tortured, literar...|2011-07-28 10:07:32|              2755.0|Edinburgh ~ Heart...|55.953346|  -3.188375|City of Edinburgh|      United Kingdom|       Europe|            Scotland|       SCT|2020-10-21 00:00:...|    TRUMP|\n",
            "|2020-10-15 00:00:44|1.31652938E18|Now Open! Create ...|  0.0|          0.0|     Hootsuite Inc.|  2.1932048E7|     OfficePools.com| officepools_com|Visit https://t.c...|2009-02-25 22:52:43|               967.0|                null|     null|       null|             null|                null|         null|                null|      null|2020-10-21 00:00:...|    TRUMP|\n",
            "|2020-10-15 00:00:55|1.31652938E18|#Trump https://t....|  1.0|          1.0|    Twitter Web App|1.08338653E18|              Cheryl|  Cheryl84037124|                null|2019-01-10 15:34:08|                49.0|                null|     null|       null|             null|                null|         null|                null|      null|2020-10-21 00:00:...|    TRUMP|\n",
            "|2020-10-15 00:01:08|1.31652952E18|@cnnbrk #Trump ow...|  3.0|          2.0| Twitter for iPhone| 1.94650432E8|           MoClarker|       MoClarker|Media Maven/Scien...|2010-09-24 17:15:43|               101.0|  Santa Monica Beach| 47.00521| -88.962906|             null|United States of ...|North America|            Michigan|        MI|2020-10-21 00:00:...|    TRUMP|\n",
            "+-------------------+-------------+--------------------+-----+-------------+-------------------+-------------+--------------------+----------------+--------------------+-------------------+--------------------+--------------------+---------+-----------+-----------------+--------------------+-------------+--------------------+----------+--------------------+---------+\n",
            "only showing top 20 rows\n",
            "\n"
          ]
        }
      ],
      "source": [
        "data = trump_df.union(biden_df)\n",
        "data.cache()\n",
        "data.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kczyT_jCEi7j"
      },
      "source": [
        "## EDA\n",
        "\n",
        "\n",
        "\n",
        "```\n",
        "\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "id": "cLGJDlPgC3j6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "153544ff-dca0-43c6-d3f1-9f15c536ab32"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+---------+--------------+\n",
            "|Candidate|Number of Data|\n",
            "+---------+--------------+\n",
            "|    TRUMP|        970919|\n",
            "|    BIDEN|        776886|\n",
            "+---------+--------------+\n",
            "\n"
          ]
        }
      ],
      "source": [
        "## Checking the data imbalance\n",
        "data.groupBy('Candidate').count().withColumnRenamed(\"count\",\"Number of Data\").show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "id": "H__CBpUBH4AQ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "889ba333-4e49-44e2-a044-915d14e7303e"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+---------+----------------+\n",
            "|Candidate|Number of Tweets|\n",
            "+---------+----------------+\n",
            "|    TRUMP|          970919|\n",
            "|    BIDEN|          776886|\n",
            "+---------+----------------+\n",
            "\n"
          ]
        }
      ],
      "source": [
        "## Comparrsion for the number of tweets\n",
        "data.groupby('Candidate').agg(F.expr('count(tweet)')\\\n",
        "                              .alias('Number of Tweets')).show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "id": "UDHXUZ3iDjmq",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "48618b93-dd37-45f1-c72d-77e105f303d3"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+---------+---------------+\n",
            "|Candidate|Number of Likes|\n",
            "+---------+---------------+\n",
            "|    TRUMP|         970919|\n",
            "|    BIDEN|         776886|\n",
            "+---------+---------------+\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# comparison of likes\n",
        "data.groupby('Candidate').agg(F.expr('count(likes)')\\\n",
        "                              .alias('Number of Likes')).show()\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "id": "EEF0ijQPFQBA",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e148a957-605b-4ce2-9d47-ca3a84fdcd43"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+--------------------+--------------+\n",
            "|             Country|Number of Data|\n",
            "+--------------------+--------------+\n",
            "|United States of ...|        332495|\n",
            "|       United States|         61905|\n",
            "|      United Kingdom|         58051|\n",
            "|               India|         40091|\n",
            "|             Germany|         35379|\n",
            "|              France|         35299|\n",
            "|              Canada|         27805|\n",
            "|               Italy|         20076|\n",
            "|           Australia|         14899|\n",
            "|              Mexico|         10903|\n",
            "|              Turkey|         10368|\n",
            "|     The Netherlands|          9587|\n",
            "|              Brazil|          8735|\n",
            "|            Pakistan|          8597|\n",
            "|               Spain|          7252|\n",
            "|             Ireland|          5452|\n",
            "|         Netherlands|          5279|\n",
            "|            Colombia|          4425|\n",
            "|           Argentina|          4404|\n",
            "|           Venezuela|          4333|\n",
            "+--------------------+--------------+\n",
            "only showing top 20 rows\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# Can someone please remove the null that is being shown also show only top 10 rows \n",
        "data.groupby('Country').count()\\\n",
        ".withColumnRenamed('count','Number of Data')\\\n",
        ".dropna()\\\n",
        ".sort(F.desc('Number of Data')).show()\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "id": "npdv7qvAL42c",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "8ee7d89c-a033-47b6-befc-b1781d3c1b3f"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+---------------+--------------+\n",
            "|        Country|Number of Data|\n",
            "+---------------+--------------+\n",
            "|             US|        394400|\n",
            "| United Kingdom|         58051|\n",
            "|          India|         40091|\n",
            "|        Germany|         35379|\n",
            "|         France|         35299|\n",
            "|         Canada|         27805|\n",
            "|          Italy|         20076|\n",
            "|      Australia|         14899|\n",
            "|         Mexico|         10903|\n",
            "|         Turkey|         10368|\n",
            "|The Netherlands|          9587|\n",
            "|         Brazil|          8735|\n",
            "|       Pakistan|          8597|\n",
            "|          Spain|          7252|\n",
            "|        Ireland|          5452|\n",
            "|    Netherlands|          5279|\n",
            "|       Colombia|          4425|\n",
            "|      Argentina|          4404|\n",
            "|      Venezuela|          4333|\n",
            "|          Chile|          3844|\n",
            "+---------------+--------------+\n",
            "only showing top 20 rows\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# Making USA to US \n",
        "data = data.withColumn('Country', regexp_replace('Country', 'United States of America', 'US'))\n",
        "data = data.withColumn('Country', regexp_replace('Country', 'United States', 'US'))\n",
        "\n",
        "## Make sure by checking the count of the data \n",
        "data.groupby('Country').count()\\\n",
        ".withColumnRenamed('count','Number of Data')\\\n",
        ".dropna()\\\n",
        ".sort(F.desc('Number of Data')).show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C690EzEMmYBS"
      },
      "source": [
        "## Preprocessing the Data(Tweets)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "id": "elXOry3-o7_9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "3e5710fd-d9e8-4c17-fa62-12fc7afca68a"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: nltk in /usr/local/lib/python3.7/dist-packages (3.2.5)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from nltk) (1.15.0)\n",
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Package stopwords is already up-to-date!\n",
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Package punkt is already up-to-date!\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {},
          "execution_count": 20
        }
      ],
      "source": [
        "!pip install nltk\n",
        "nltk.download(\"stopwords\")\n",
        "nltk.download('punkt')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "id": "YXUSrer5nVkr"
      },
      "outputs": [],
      "source": [
        "stop_words = set(stopwords.words('english'))\n",
        "ps = PorterStemmer()\n",
        "\n",
        "def preprocess_tweets(text, stem=False, lemmatize=False):\n",
        "    ## If statement for fault tolerance\n",
        "    if isinstance(text,str):\n",
        "        text = text.lower()\n",
        "        # Remove links, special characters , numbers punctuation etc. \n",
        "        chars_and_links= r'\\d+|http?\\S+|[^A-Za-z0-9]+'\n",
        "        # Cleaned the text\n",
        "        text = re.sub(chars_and_links,' ',text)\n",
        "\n",
        "        filtered_text = []\n",
        "        # Tokenize the text\n",
        "        words = word_tokenize(text) \n",
        "        # Remove stopwords and stem\n",
        "        for word in words:\n",
        "            # Words in stop words or not\n",
        "            if not word in stop_words:\n",
        "                if stem:\n",
        "                    filtered_text.append(ps.stem(word))\n",
        "                elif lemmatize:\n",
        "                    filtered_text.append(Word(word).lemmatize())\n",
        "                else:\n",
        "                    filtered_text.append(word)\n",
        "        # Return the filtered text\n",
        "        return ' '.join(filtered_text)\n",
        "    else:\n",
        "        return text"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "id": "ptY9mMkds79R"
      },
      "outputs": [],
      "source": [
        "Preprocess_UDF = udf(lambda x: preprocess_tweets(x))  \n",
        "\n",
        "tokenized_df = data.withColumn('tokenized_tweet', Preprocess_UDF(F.col('tweet')))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "l-ze-ZJ_APOh"
      },
      "outputs": [],
      "source": [
        "def classify(polarity):\n",
        "    ''' This is UDF function for classification '''\n",
        "    if polarity > 0:\n",
        "        return 'positive'\n",
        "    if polarity == 0:\n",
        "        return 'neutral'\n",
        "    return 'negative'\n",
        "\n",
        "## UDF to calculate polarity\n",
        "get_polarity = udf(lambda text: TextBlob(text).sentiment.polarity)\n",
        "## UDF to calcularte the subjectivity\n",
        "get_subjectivity = udf(lambda text: TextBlob(text).sentiment.subjectivity)\n",
        "## UDF for getting classification \n",
        "classify_sentiment = udf(classify)\n",
        "\n",
        "## Using text blob to get the polarity \n",
        "tokenized_df = tokenized_df.withColumn('polarity', get_polarity(tokenized_df.tokenized_tweet))\\\n",
        "                            .withColumn('subjectivity', get_subjectivity(tokenized_df.tokenized_tweet))\n",
        "\n",
        "sent_analysis = tokenized_df.withColumn('classification', classify_sentiment(tokenized_df.polarity))\n",
        "                                \n",
        "sent_analysis.show()\n",
        "sent_analysis.count()                  "
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Droping useless attributes from the data frame"
      ],
      "metadata": {
        "id": "Ayd1bjpVzO9C"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Droping useless attributes from the main data frame\n",
        "df = sent_analysis.drop(\"tweet_id\",\"created_at\",\"retweet_count\",\"source\", \"user_id\", \"user_name\", \"user_screen_name\",\"user_description\", \"user_join_date\", \"user_followers_count\",\"user_location\", \"lat\", \"long\", \"city\", \"state_code\", \"collected_at\",\"continent\", \"polarity\", \"subjectivity\",\"tokenized_tweet\")"
      ],
      "metadata": {
        "id": "GdaC9beozQBb"
      },
      "execution_count": 24,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Printing the modified dataframe\n",
        "#df.show()\n",
        "df.write.csv('Tweet_Analysis.csv')\n",
        "## Converting to pandas data frame for simplicity\n",
        "# import pandas as pd \n",
        "# pandas_df = df.toPandas()\n",
        "# print(pandas_df)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 389
        },
        "id": "aW1k8CGkzQEs",
        "outputId": "8a7c39be-bc2f-4e8c-c7ca-1fa7dbd93ad1"
      },
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "error",
          "ename": "AnalysisException",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mAnalysisException\u001b[0m                         Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-25-8921daabea1f>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m# Printing the modified dataframe\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0;31m#df.show()\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0mdf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwrite\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcsv\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'Tweet_Analysis.csv'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      4\u001b[0m \u001b[0;31m## Converting to pandas data frame for simplicity\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0;31m# import pandas as pd\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pyspark/sql/readwriter.py\u001b[0m in \u001b[0;36mcsv\u001b[0;34m(self, path, mode, compression, sep, quote, escape, header, nullValue, escapeQuotes, quoteAll, dateFormat, timestampFormat, ignoreLeadingWhiteSpace, ignoreTrailingWhiteSpace, charToEscapeQuoteEscaping, encoding, emptyValue, lineSep)\u001b[0m\n\u001b[1;32m    953\u001b[0m                        \u001b[0mcharToEscapeQuoteEscaping\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcharToEscapeQuoteEscaping\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    954\u001b[0m                        encoding=encoding, emptyValue=emptyValue, lineSep=lineSep)\n\u001b[0;32m--> 955\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_jwrite\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcsv\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpath\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    956\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    957\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0morc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpath\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmode\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpartitionBy\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcompression\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/py4j/java_gateway.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *args)\u001b[0m\n\u001b[1;32m   1320\u001b[0m         \u001b[0manswer\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgateway_client\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msend_command\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcommand\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1321\u001b[0m         return_value = get_return_value(\n\u001b[0;32m-> 1322\u001b[0;31m             answer, self.gateway_client, self.target_id, self.name)\n\u001b[0m\u001b[1;32m   1323\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1324\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mtemp_arg\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mtemp_args\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/pyspark/sql/utils.py\u001b[0m in \u001b[0;36mdeco\u001b[0;34m(*a, **kw)\u001b[0m\n\u001b[1;32m    115\u001b[0m                 \u001b[0;31m# Hide where the exception came from that shows a non-Pythonic\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    116\u001b[0m                 \u001b[0;31m# JVM exception message.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 117\u001b[0;31m                 \u001b[0;32mraise\u001b[0m \u001b[0mconverted\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    118\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    119\u001b[0m                 \u001b[0;32mraise\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mAnalysisException\u001b[0m: path file:/content/Tweet_Analysis.csv already exists."
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# import pandas as pd \n",
        "# pandas_df = pd.read_csv('/content/Tweet_Analysis.csv/part-00001-b60952d2-6944-4377-84ec-ec7b62117f35-c000.csv')\n",
        "# pandas_df.show()\n",
        "import pandas as pd \n",
        "pandas_df = df.toPandas()\n",
        "#print(pandas_df)"
      ],
      "metadata": {
        "id": "yXHJbgtpIBw4"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(pandas_df)"
      ],
      "metadata": {
        "id": "J3QmvQnQRquO"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "pandas_df = pandas_df.groupby(['country','Candidate'])['tweet'].count().reset_index()\n"
      ],
      "metadata": {
        "id": "DYJW7fRmFHJz"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "## Creating a temporary table on Data Frame\n",
        "# df.createOrReplaceTempView(\"Tweet_Analysis\")\n",
        "# # Checking the Sql query\n",
        "# df1 = spark.sql(\"SELECT * from Tweet_Analysis\")\n",
        "# df1.show()"
      ],
      "metadata": {
        "id": "tnGkpfnszQI_"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "## Candidate group by \n",
        "#df2 = spark.sql(\"SELECT Country from Tweet_Analysis group by Candidate \")\n",
        "#df2 = spark.sql(\"SELECT Country from Tweet_Analysis group by Candidate \")\n",
        "\n",
        "# df2 = spark.sql(\"SELECT * from Tweet_Analysis\")\n",
        "# df2.show()\n",
        "#.withColumnRenamed('count','Number of Data')\\\n",
        "#.dropna()\\"
      ],
      "metadata": {
        "id": "l6jQUIP8zQM1"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        ""
      ],
      "metadata": {
        "id": "Bzug0GaKzQYF"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Dalila's Try"
      ],
      "metadata": {
        "id": "aaRnw8YgzJcW"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oeKSDkE6H5VS"
      },
      "outputs": [],
      "source": [
        "@udf(returnType=ByteType())\n",
        "def classify(polarity):\n",
        "    if polarity > 0:\n",
        "      return 1\n",
        "    if polarity == 0:\n",
        "      return 0\n",
        "    return -1\n",
        "\n",
        "@udf(returnType=FloatType())\n",
        "def get_polarity(text):\n",
        "    return TextBlob(text).sentiment.polarity\n",
        "\n",
        "@udf(returnType=FloatType())\n",
        "def get_subjectivity(text):\n",
        "    return TextBlob(text).sentiment.subjectivity\n",
        "\n",
        "tokenized_df = tokenized_df.withColumn('polarity', get_polarity(tokenized_df.tokenized_tweet))\\\n",
        "                            .withColumn('subjectivity', get_subjectivity(tokenized_df.tokenized_tweet))\n",
        "\n",
        "sent_analysis = tokenized_df.withColumn('classification', classify(tokenized_df.polarity))\n",
        "                                \n",
        "sent_analysis.show()\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "sent_analysis.write.parquet(\"tweetscandidate1.parquet\").save()"
      ],
      "metadata": {
        "id": "ebPpmW0og5h7"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "DF=spark.read.parquet(\"tweetscandidate.parquet\")"
      ],
      "metadata": {
        "id": "pw3K1m-Jfnr3"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "sent_analysis.write.csv('tweetscandidate.csv')"
      ],
      "metadata": {
        "id": "2fbz5cpv82l8"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "\n",
        "parDF=spark.read.parquet(\"tweetsclean.parquet\")"
      ],
      "metadata": {
        "id": "Zl5RcHHhBYVb"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "sent_analysis.write.csv('tweets1.csv')"
      ],
      "metadata": {
        "id": "QEQaVqhYehdh"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "parDF.show()\n",
        "parDF.count()"
      ],
      "metadata": {
        "id": "z6n9CCKQZQNU"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "df = sent_analysis.drop(\"created_at\",\"retweet_count\",\"source\", \"user_id\", \"user_name\", \"user_screen_name\",\"user_description\", \"user_join_date\", \"user_followers_count\",\"user_location\", \"lat\", \"long\", \"city\", \"continent\", \"state\", \"state_code\", \"collected_at\")"
      ],
      "metadata": {
        "id": "g0FSVXBP80lL"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        ""
      ],
      "metadata": {
        "id": "GD74e6EV34pU"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df.show()"
      ],
      "metadata": {
        "id": "MEVYHaKV-SDp"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "sent_analysis.groupBy('classification').count().withColumnRenamed(\"count\",\"Number of Data\").show()\n"
      ],
      "metadata": {
        "id": "xS2W16-mDYZJ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        ""
      ],
      "metadata": {
        "id": "qMDvuz3G8vu-"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "eakD-LgLavOl"
      },
      "outputs": [],
      "source": [
        "train, test = sent_analysis.randomSplit([0.7, 0.3])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "F8pWq8-Tt1NS"
      },
      "outputs": [],
      "source": [
        "from pyspark.ml.feature import VectorAssembler, VectorIndexer\n",
        "from pyspark.ml.regression import GBTRegressor\n",
        "\n",
        "vectorAssembler = VectorAssembler(inputCols=['polarity','subjectivity'],\n",
        "                                  outputCol='features')\n",
        "\n",
        "# Takes the \"features\" column and learns to predict \"classification\"\n",
        "gbt = GBTRegressor(labelCol=\"classification\")\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pFSmPQon9rnc"
      },
      "outputs": [],
      "source": [
        "from pyspark.ml.tuning import CrossValidator, ParamGridBuilder\n",
        "from pyspark.ml.evaluation import RegressionEvaluator\n",
        "\n",
        "paramGrid = ParamGridBuilder()\\\n",
        "  .addGrid(gbt.maxDepth, [2, 5])\\\n",
        "  .addGrid(gbt.maxIter, [10, 20])\\\n",
        "  .build()\n",
        "\n",
        "from pyspark.ml.evaluation import RegressionEvaluator\n",
        "\n",
        "# We define an evaluation metric.  This tells CrossValidator how well we are doing by comparing the true labels with predictions.\n",
        "evaluator = RegressionEvaluator(metricName=\"rmse\", labelCol=gbt.getLabelCol(), predictionCol=gbt.getPredictionCol())\n",
        "# Declare the CrossValidator, which runs model tuning for us.\n",
        "cv = CrossValidator(estimator=gbt, evaluator=evaluator, estimatorParamMaps=paramGrid)\n",
        "\n",
        "from pyspark.ml import Pipeline\n",
        "pipeline = Pipeline(stages=[vectorAssembler, cv])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_TGcfQQrghr6"
      },
      "outputs": [],
      "source": [
        "\n",
        "pipelineModel = pipeline.fit(train)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TQm7Ct1UiKoC"
      },
      "outputs": [],
      "source": [
        "predictions = pipelineModel.transform(test)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "aP6qiEEI1tq1"
      },
      "outputs": [],
      "source": [
        "#predictions.select"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ET71D2zUj9CK"
      },
      "outputs": [],
      "source": [
        ""
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "machine_shape": "hm",
      "name": "Twitter_Sentiment_Analysis.ipynb",
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "nbformat": 4,
  "nbformat_minor": 0
}