ETL Pipeline Preparation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ETL Pipeline Preparation\n",
    "Follow the instructions below to help you create your ETL pipeline.\n",
    "### 1. Import libraries and load datasets.\n",
    "- Import Python libraries\n",
    "- Load `messages.csv` into a dataframe and inspect the first few lines.\n",
    "- Load `categories.csv` into a dataframe and inspect the first few lines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 303,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import libraries\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sqlalchemy import create_engine\n",
    "\n",
    "pd.set_option('max_colwidth', 100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 271,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>message</th>\n",
       "      <th>original</th>\n",
       "      <th>genre</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>Weather update - a cold front from Cuba that could pass over Haiti</td>\n",
       "      <td>Un front froid se retrouve sur Cuba ce matin. Il pourrait traverser Haiti demain. Des averses de...</td>\n",
       "      <td>direct</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>7</td>\n",
       "      <td>Is the Hurricane over or is it not over</td>\n",
       "      <td>Cyclone nan fini osinon li pa fini</td>\n",
       "      <td>direct</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8</td>\n",
       "      <td>Looking for someone but no name</td>\n",
       "      <td>Patnm, di Maryani relem pou li banm nouvel li ak timoun yo. Mesi se john jean depi Monben kwochi.</td>\n",
       "      <td>direct</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>9</td>\n",
       "      <td>UN reports Leogane 80-90 destroyed. Only Hospital St. Croix functioning. Needs supplies desperat...</td>\n",
       "      <td>UN reports Leogane 80-90 destroyed. Only Hospital St. Croix functioning. Needs supplies desperat...</td>\n",
       "      <td>direct</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>12</td>\n",
       "      <td>says: west side of Haiti, rest of the country today and tonight</td>\n",
       "      <td>facade ouest d Haiti et le reste du pays aujourd hui et ce soir</td>\n",
       "      <td>direct</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id  \\\n",
       "0   2   \n",
       "1   7   \n",
       "2   8   \n",
       "3   9   \n",
       "4  12   \n",
       "\n",
       "                                                                                               message  \\\n",
       "0                                   Weather update - a cold front from Cuba that could pass over Haiti   \n",
       "1                                                              Is the Hurricane over or is it not over   \n",
       "2                                                                      Looking for someone but no name   \n",
       "3  UN reports Leogane 80-90 destroyed. Only Hospital St. Croix functioning. Needs supplies desperat...   \n",
       "4                                      says: west side of Haiti, rest of the country today and tonight   \n",
       "\n",
       "                                                                                              original  \\\n",
       "0  Un front froid se retrouve sur Cuba ce matin. Il pourrait traverser Haiti demain. Des averses de...   \n",
       "1                                                                   Cyclone nan fini osinon li pa fini   \n",
       "2    Patnm, di Maryani relem pou li banm nouvel li ak timoun yo. Mesi se john jean depi Monben kwochi.   \n",
       "3  UN reports Leogane 80-90 destroyed. Only Hospital St. Croix functioning. Needs supplies desperat...   \n",
       "4                                      facade ouest d Haiti et le reste du pays aujourd hui et ce soir   \n",
       "\n",
       "    genre  \n",
       "0  direct  \n",
       "1  direct  \n",
       "2  direct  \n",
       "3  direct  \n",
       "4  direct  "
      ]
     },
     "execution_count": 271,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# load messages dataset\n",
    "messages = pd.read_csv('messages.csv')\n",
    "messages.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 272,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>categories</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>related-1;request-0;offer-0;aid_related-0;medical_help-0;medical_products-0;search_and_rescue-0;...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>7</td>\n",
       "      <td>related-1;request-0;offer-0;aid_related-1;medical_help-0;medical_products-0;search_and_rescue-0;...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8</td>\n",
       "      <td>related-1;request-0;offer-0;aid_related-0;medical_help-0;medical_products-0;search_and_rescue-0;...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>9</td>\n",
       "      <td>related-1;request-1;offer-0;aid_related-1;medical_help-0;medical_products-1;search_and_rescue-0;...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>12</td>\n",
       "      <td>related-1;request-0;offer-0;aid_related-0;medical_help-0;medical_products-0;search_and_rescue-0;...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id  \\\n",
       "0   2   \n",
       "1   7   \n",
       "2   8   \n",
       "3   9   \n",
       "4  12   \n",
       "\n",
       "                                                                                            categories  \n",
       "0  related-1;request-0;offer-0;aid_related-0;medical_help-0;medical_products-0;search_and_rescue-0;...  \n",
       "1  related-1;request-0;offer-0;aid_related-1;medical_help-0;medical_products-0;search_and_rescue-0;...  \n",
       "2  related-1;request-0;offer-0;aid_related-0;medical_help-0;medical_products-0;search_and_rescue-0;...  \n",
       "3  related-1;request-1;offer-0;aid_related-1;medical_help-0;medical_products-1;search_and_rescue-0;...  \n",
       "4  related-1;request-0;offer-0;aid_related-0;medical_help-0;medical_products-0;search_and_rescue-0;...  "
      ]
     },
     "execution_count": 272,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# load categories dataset\n",
    "categories = pd.read_csv('categories.csv')\n",
    "categories.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 273,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 26248 entries, 0 to 26247\n",
      "Data columns (total 4 columns):\n",
      "id          26248 non-null int64\n",
      "message     26248 non-null object\n",
      "original    10184 non-null object\n",
      "genre       26248 non-null object\n",
      "dtypes: int64(1), object(3)\n",
      "memory usage: 820.3+ KB\n"
     ]
    }
   ],
   "source": [
    "# Get some understanding of datasets\n",
    "messages.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 274,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "id              0\n",
       "message         0\n",
       "original    16064\n",
       "genre           0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 274,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# NA values\n",
    "messages.isna().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 275,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "24779    3\n",
      "7747     2\n",
      "14246    2\n",
      "25512    2\n",
      "17553    2\n",
      "13914    2\n",
      "29119    2\n",
      "14135    2\n",
      "14592    2\n",
      "17919    2\n",
      "18313    2\n",
      "6573     2\n",
      "18925    2\n",
      "14073    2\n",
      "12589    2\n",
      "6327     2\n",
      "27768    2\n",
      "23002    2\n",
      "6492     2\n",
      "15938    2\n",
      "2446     2\n",
      "4956     2\n",
      "1652     2\n",
      "7892     2\n",
      "11503    2\n",
      "15576    2\n",
      "3250     2\n",
      "5776     2\n",
      "6515     2\n",
      "24247    2\n",
      "        ..\n",
      "27384    1\n",
      "25337    1\n",
      "29435    1\n",
      "19196    1\n",
      "8913     1\n",
      "10960    1\n",
      "21183    1\n",
      "23230    1\n",
      "14994    1\n",
      "12947    1\n",
      "2708     1\n",
      "661      1\n",
      "6806     1\n",
      "4759     1\n",
      "27288    1\n",
      "25241    1\n",
      "19100    1\n",
      "17053    1\n",
      "23198    1\n",
      "21151    1\n",
      "15026    1\n",
      "12979    1\n",
      "2740     1\n",
      "6838     1\n",
      "4791     1\n",
      "27320    1\n",
      "25273    1\n",
      "29371    1\n",
      "17085    1\n",
      "2049     1\n",
      "Name: id, Length: 26180, dtype: int64\n",
      "*********************\n",
      "#NAME?                                                                                                                                                                                                                                                                                                                                                                           4\n",
      "Shelter materials (thick polyesters) are being distributed to 18,000 households.                                                                                                                                                                                                                                                                                                 3\n",
      "The regime says the charter will clear the way for democratic elections in two years, but critics say the document will ensure the generals remain dominant in a country that has been ruled by the military for nearly half a century.                                                                                                                                          2\n",
      "Sir or madam, I'm hungry, I cannot stand because I'm very hungry, if you could share the food, it would benefit the population, thanks                                                                                                                                                                                                                                           2\n",
      "Falta tan tan poco.. ¬¨¬¥Simply Red - Holding back the years¬¨¬™. Arena Santiago, de all‚àö¬∞ somos! Night.  http://blip.fm/~n0frf                                                                                                                                                                                                                                               2\n",
      "in my village after collapse of whole infrastructure due to flood the majority of people is homeless  in need of tents which are not available.                                                                                                                                                                                                                                  2\n",
      "Any person who enters or exits at other points will be considered an infiltrator, he added.                                                                                                                                                                                                                                                                                      2\n",
      "Full Circle, my local, collecting supplies and making lunches for #SandyRelief @Full Circle Bar http://t.co/EHLqbhHw                                                                                                                                                                                                                                                             2\n",
      "Tell the President to give the nation over to God.                                                                                                                                                                                                                                                                                                                               2\n",
      "* German Agro Action (GAA) reported that in the 6 weeks up to 15 July it had completed the distribution of ECHO food to almost 14,000 beneficiaries in the Tavildara district, as well as financing a total of 49 Food for Work micro-projects through WFP - the majority concerned with irrigation, canal cleaning and feeder road construction in the Garm district.           2\n",
      "BEIJING, Aug 16 (AFP) - Some 200 people are dead or missing following three days of flash flooding that hit central China's Hunan province, officials said Monday.                                                                                                                                                                                                               2\n",
      "To date, a total of 39 people suffering from Ebola or having had a high risk exposure to the virus, have been medically evacuated to Europe.                                                                                                                                                                                                                                     2\n",
      "As the harsh winter sets in, more people are in danger of dying in the freezing conditions.                                                                                                                                                                                                                                                                                      2\n",
      "I'd like to sign up in the 4636 program please                                                                                                                                                                                                                                                                                                                                   2\n",
      "The slow pace of reconstruction along Japan's devastated north-eastern coastline is contributing to survivors' stress, as there is little clarity on how long they will have to remain in cramped temporary housing.                                                                                                                                                             2\n",
      "Starting in October 1999, severe food insecurity began to be reported in Gode zone, Somali Region, as a result of delayed and erratic Deyr rains (expected from September-December).                                                                                                                                                                                             2\n",
      "In response to North Korea's devastating artillery attack on Yeonpyeong Island on Tuesday, South Korea has suspended aid shipments of food and cement to the North's flood-stricken Shinuiju region.                                                                                                                                                                             2\n",
      "I am in Carefour Feuilles, Dkayet neighborhood. We need potable water because diarrhea is rampant here and is undermining our health.                                                                                                                                                                                                                                            2\n",
      "I can help deliver food to seniors who are trapped in their homes . I speak Mandarin and Cantonese .                                                                                                                                                                                                                                                                             2\n",
      "Mera ghar selab ki waja sy gir giya hy. aur dobara nahin bna sakta main shadeed mooshkil men hoon. mery 3 choty choty bachy aur baqi ghar waly shdeed sardi khaema mein guzarny par majboor hin. khdara ghar banany mein meri mdad kro. plz help me. khalil                                                                                                                      2\n",
      "I am a victim. I need food, water and a radio.                                                                                                                                                                                                                                                                                                                                   2\n",
      "What we saw yesterday is a sign, you need to know what Jesus is stirring up/doing                                                                                                                                                                                                                                                                                                2\n",
      "The last locust infestation in the Central Asian republic was in 1995 and is expected to reoccur every four to five years.                                                                                                                                                                                                                                                       2\n",
      "Additionally, ointment to treat skin infections, cough medicine, iodine and bandages are needed for basic care.                                                                                                                                                                                                                                                                  2\n",
      "As we approach Luabo, I get my first impressions of the camp, dilapidated buildings, most without roofs, or windows and walls collapsing.                                                                                                                                                                                                                                        2\n",
      "please we need water, food and tents, we have 101 chlidren. .. please come rescue                                                                                                                                                                                                                                                                                                2\n",
      "where we can paticipate in the law reorganizations?                                                                                                                                                                                                                                                                                                                              2\n",
      "After the 2011 elections, widespread rioting left hundreds dead.                                                                                                                                                                                                                                                                                                                 2\n",
      "They reported sporadic rains were still flooding air fields, roads and slowing urgent repairs along several key highways.                                                                                                                                                                                                                                                        2\n",
      "jel2  Acte 5: 29 2chr7 Lev11 Ecl9 Ecl4 jos5 ch102kr Es59                                                                                                                                                                                                                                                                                                                         2\n",
      "                                                                                                                                                                                                                                                                                                                                                                                ..\n",
      "As he spoke, the French government said on Wednesday it was diverting the naval helicopter carrier Jeanne d'Arc to Madagascar immediately.                                                                                                                                                                                                                                       1\n",
      "The number of swarms has begun declining in Mauritania as control interventions have intensified and escapee swarms move northward.                                                                                                                                                                                                                                              1\n",
      "I'm sending this message to see what you can do for me.                                                                                                                                                                                                                                                                                                                          1\n",
      "The next step is to transfer these technologies to the farmer/fisher communities affected by the tsunami through the farm women agricultural extension services and the women rural development society.                                                                                                                                                                         1\n",
      "Some of what will be necessary will be to find subsidy resources to allow us to expand our concessional lending capacity because African countries, most of them, benefit from the Fund's concessional resources which can only be made available to the extent that we have subsidy resources to make our general resources more affordable.                                    1\n",
      "What are the authorities saying about the functioning of the schools in the rest of the country.                                                                                                                                                                                                                                                                                 1\n",
      "Hello,how could you help me to find a job,cause I sturdied Technical and operator in agricultural mekanisation.                                                                                                                                                                                                                                                                  1\n",
      "With relief flights suspended for the third straight day due to heavy snow and icy conditions, it is even more critical to reach earthquake survivors.                                                                                                                                                                                                                           1\n",
      "My prayers to everyone affected by the earthquake in Haiti so sad and heartbreaking these people already suffer too much and now this                                                                                                                                                                                                                                            1\n",
      "What's the summary of the earthquakes?                                                                                                                                                                                                                                                                                                                                           1\n",
      "Donors provide greater funding support for infrastructure improvements, especially for the continued rehabilitation of transportation networks throughout the Congo.                                                                                                                                                                                                             1\n",
      "In world weather news today we have snow UK Europe record heatwave Australia terrible earthquake Haiti . Makes me thankful for snow.                                                                                                                                                                                                                                             1\n",
      "The three people include; Mrs. Shallah Mahaindost, formerly Director of an Employment Agency in Northern Zone (Mazar), Mr. Abdurrazzaq Baig, formerly a member of the police academy and Faculty of law and Mr. Mohammed Ashraf Rasuli, who was a Supreme Court Judge, and a constitutional committee member.                                                                    1\n",
      "http bit.ly 8okEsH P Magnitude 7.0 Earthquake Hits Haiti haiti earthquake                                                                                                                                                                                                                                                                                                        1\n",
      "The Myanmar [Burma] Climate Change Watch website mentions the danger of the downburst on its web page but does not mention details of the meso scale phenomena.                                                                                                                                                                                                                  1\n",
      "I think you will do the necessary things for me. .. Note : call for details. .. incomplete request                                                                                                                                                                                                                                                                               1\n",
      "The technical evidence obtained from remote-sensing (e.g. satellite photography) and other sources shows that flooding of the scale in Spring 2007 is very unlikely, because this is a function of the thickness of the snowpack, the suddenness of the rise in temperature, and often also rain which speeds up the melting and run-off process.                                1\n",
      "With funding from Direct Relief FSW has begun the construction of a two-story public health clinic in Murawatte that will offer maternal health care, child health care, malnutrition prevention, health promotion activities for youth and adults, chronic disease screening, mental health care, indigenous medicine clinic, dental clinic, and a library and reading room.    1\n",
      "I congratulate you for the job you are doing for the Haitian community. Keep up the good work, the Haitian people are with you.                                                                                                                                                                                                                                                  1\n",
      "WE WILL CONTINUE TO. TRY TO SOLVE THE PROBLEM. WE HAVE SOME SUPPORT FROM UNICEF                                                                                                                                                                                                                                                                                                  1\n",
      "The Reearth project thoughts go out to the people of #Haiti after the devastating earthquake!                                                                                                                                                                                                                                                                                    1\n",
      "We find no potable water for us to drink. People area of TNH (Tele National D'Haiti)                                                                                                                                                                                                                                                                                             1\n",
      "Over in a shadowy corner lies one last bundle of sorghum.                                                                                                                                                                                                                                                                                                                        1\n",
      "NOTES: This message is not complete. So I can't understand it.                                                                                                                                                                                                                                                                                                                   1\n",
      "However, unseasonably cold weather, with prolonged sub-freezing temperatures and snow storms, are hampering relief efforts.                                                                                                                                                                                                                                                      1\n",
      "However, it has largely relied on local tribesmen under the leadership of Hazrat Ali and others to pursue alleged Al-Qaida supporters.                                                                                                                                                                                                                                           1\n",
      "This is due to severe droughts, poor harvests and rising food prices across the region.                                                                                                                                                                                                                                                                                          1\n",
      "we need food amd water for alot of kids                                                                                                                                                                                                                                                                                                                                          1\n",
      "I want you to make the authorithy hear this message                                                                                                                                                                                                                                                                                                                              1\n",
      "In response to this disaster, between 2006 and 2009 several AKDN agencies and FOCUS will be implementing a programme to enhance the state of disaster resilience and preparedness among coastal communities affected by the tsunami.                                                                                                                                             1\n",
      "Name: message, Length: 26177, dtype: int64\n",
      "*********************\n",
      "Nap fe ou konnen ke apati de jodi a sevis SMS 4636 pou enfomasyon ijan                                                                                                                                                                                          20\n",
      "Un front froid se retrouve sur Cuba ce matin. Il pourrait traverser Haiti demain. Des averses de pluie isolee sont encore prevues sur notre region ce soi                                                                                                       19\n",
      "4636 : Nasyonzini di ou men m retire kounye a tout fatra ak gravwa nan#                                                                                                                                                                                          7\n",
      "hmre vilage kachipul men flood ne bht nuksan kiya he hmra crop,hmre ghr aur hmra rozgar sbkch khtm hogya he mgr govt ne abi tk hmri koi help nhn ki he.                                                                                                          6\n",
      "Enfomasyon sou tranbleman ta a                                                                                                                                                                                                                                   6\n",
      "Apre chak katastwof ki pase nan peyi a, nou tout nou okouran, nasyonal e entenasyonal, kouman ed yo, lajan yo pa janm rive jwenn vre viktim kit se LETA oubye                                                                                                    5\n",
      "4636 : ONG di trete dlow ak akwatab oubyen yon ti kloroks pouw ka bwel. siw met kloroks tann 30 minit. sante piblik ab bay akwatab                                                                                                                               5\n",
      "Si gen tant kap distribiye, fe tout sa nou ka f, poum ka jwenn younn svp.                                                                                                                                                                                        5\n",
      "Chak 1 h 2 tan, mande gras, padon, mizerikod pou Ayiti! Fl ak lafwa, sel Bondye ki ka retire nou nan sa. .. Voye mesaj sa pou 10 lot moun!                                                                                                                       5\n",
      "LID KAP TRAVAY NAN KOMIN LAN POU AVANSMAN TOUT AKINWA YO MALGRE NOU PA GEN MWAYEN MEN NAP FE PREV DE NOU AK TRANSPARAN POU TOUT KONPRANN SA NOU REYALIZE                                                                                                         5\n",
      "Mwen se yon chof kamion mwen bezwen yon djob                                                                                                                                                                                                                     4\n",
      "      a                                                                                                                                                                                                                                                          4\n",
      "J'aimerais savoir quand dbuteront les inscriptions pour le full bright schoollar ship?                                                                                                                                                                           4\n",
      "Infomasyon sou sa m dwe konnen.                                                                                                                                                                                                                                  4\n",
      "Message received. Message recu. Mesaj ou ale                                                                                                                                                                                                                     4\n",
      "zone kafou new york bezwen tant                                                                                                                                                                                                                                  4\n",
      "4636 : Nasyonzini di pouw pa pran malozye lave menw souvan, siw ba y#                                                                                                                                                                                            4\n",
      "Tgen yon tranblement de terre kiti pase nan nord pays d'haiti st an 1842 bulan trt inconnue                                                                                                                                                                      4\n",
      "Mera ghar selab ki waja sy gir giya hy. aur dobara nahin bna sakta main shadeed mooshkil men hoon. mery 3 choty choty bachy aur baqi ghar waly shdeed                                                                                                            4\n",
      "Rlle canne--sucre prolonge(carrefour) Nou bezwen Tant,dlo,manje.                                                                                                                                                                                                 4\n",
      "Logane 9m section citronniers nou bezwen manje                                                                                                                                                                                                                   4\n",
      "Kisa pou yon moun f lw santiw pa menm janw te ye a ou ta vle gen twoub mantal kisa poum f?                                                                                                                                                                       4\n",
      "hmre vilage kachipul men flood ne bht nuksan kiya he hmra crop,hmre ghr aur hmra rozgar sbkch khtm hogya he mgr govt ne abi tk hmri koi help nhn ki he. Location:vilage kachipul distt:kambr shadadkot plz visit u city kachipul                                 4\n",
      "4636 : ONG di si yon granmoun te fe yon tan li pat pran medikamanl, r#                                                                                                                                                                                           4\n",
      "Je vous remercie tant de m'avoir donn la chance de m'exprimer par sms. En effet j'ai grand besoin d'un emploi dans un ONG, veuillez m'aider s'il vous plait 4636. Vos informations seront satisfaisantes pour moi.                                               4\n",
      "Sil vou pl ede mwen avek moun ki viktim yo nan tranbleman de ta a, ki kiti potoprins ki vini nan provinse mwen ede ak ti kob mwen te ginyin kouni a                                                                                                              4\n",
      "hum 2 bhaiyon ny flood sy 1 mounth pehly shadi ki te flood ny hmara sb kch tabha kr dia hmara na ghar ra hy na kch min metric ka studnt hon or bhai fsc ka studnt hy or hmara watan card b ni aya hy bs es wja sy hm ny apni study chor di ab beghar or besa     3\n",
      "Moinsyon victime moinret bonrepos serin mapedormir madame moinyontimoume moinpagenmange poume bayo silvousplait dirigent yowsanous caferpourmoin                                                                                                                 3\n",
      "Cheri l amour est le seul reve qui ne se reve pas                                                                                                                                                                                                                3\n",
      "Penses vous que la reconstruction du pays sera possible? si oui comment?                                                                                                                                                                                         3\n",
      "                                                                                                                                                                                                                                                                ..\n",
      "Bagay ki konsne PAM paske PAM pa itil mwen anyen se menm yo menm kap fe magouy ak peyi a.                                                                                                                                                                        1\n",
      "Moun mon repo 44 pa janm jwenn ed sa yo yap bay nan peyi a.                                                                                                                                                                                                      1\n",
      "Eske map bezwen peye pou sevis sa a? p pwofite felisite nou pou bon travay sa a!                                                                                                                                                                                 1\n",
      "Si nan yon kan pa janm gen dlo pou moun sevi, kisa nou ka fe poun jwenn dlo? Kijan yon moun ka jwenn job nan yon ONG e kib pou li ale? Mesi                                                                                                                      1\n",
      "L, ecole communautaire de otah, ( martissant ) sans l, assistances financiere des org, elle n, ouvrirait sa porte on a 8 professeurs un directeur pedagogique un directeur administratif un censeur et servante.                                                 1\n",
      "Jodia ki enfomasyon ki gen svp                                                                                                                                                                                                                                   1\n",
      "Mwen renmen Digicel ou ban nou tout dnye enfomasyon yo e map mache ak digicel                                                                                                                                                                                    1\n",
      "Mesi anpil pou bon konsey sa yo.                                                                                                                                                                                                                                 1\n",
      "4636 : Mwen ta renmen konnen eske te a ak souke, ancore                                                                                                                                                                                                          1\n",
      "Pa voye mesaj sou telephon mwen anko tanpri                                                                                                                                                                                                                      1\n",
      "KI LABORATWA NAN MOMAN AN KI FONKSYONE SVP.                                                                                                                                                                                                                      1\n",
      "F yon tipalem de replik la sasa vle di                                                                                                                                                                                                                           1\n",
      "Map mande eske polisye yo pat touche nan dram nan? Yo fanmi, yap domi nan seren paske yo pa gen tant. Eske se posib pou jiska prezan polisye yo paka j                                                                                                           1\n",
      "Ap ko aur ap k sare team ko Eid Mubarak ho?Sir Ap logon ko aik Request hai k sir aik bar ap hamar city mein aa kr visit kro ya to apna Nimaind behjo to hamare sub halat ke khaber malom hoge!Sir hazaron k tadad mein logo ko watan card be na mila jin ko      1\n",
      "S. v. p pourvez vous me donner des informatoins sur kros                                                                                                                                                                                                         1\n",
      "Men si kay la pa tonbe se kay tel eske moun ka demi la dann                                                                                                                                                                                                      1\n",
      "Bonm yon ide sou domin kay mwen ki kraze eske map jwenn plasman an menm kote a oubyen m pedil nt                                                                                                                                                                 1\n",
      "Mwen bezwen konen kek ifmasyon                                                                                                                                                                                                                                   1\n",
      "ns avon bez daid a pax vila crematorium mais gate js kisi ns navon rien recu merci                                                                                                                                                                               1\n",
      "former que le seisme du 12 Janvier a engendr des pertes normes en vie humaine et a laiss plus de 2000 sinistres. Par consquent nous sollicitons de                                                                                                               1\n",
      "PAM ap f distribisyon manje men yo di se fanm ki ka pran li men se vf yo ak felen                                                                                                                                                                                1\n",
      "mwen ta renmen nou enfo mem de sa kap pase jeremi                                                                                                                                                                                                                1\n",
      "E pou moun Ndws ki kote yo ka jwen bis pou yo vini                                                                                                                                                                                                               1\n",
      "A Chak moman mande gras, padon, mizerikod pou pap Ayisyen! Fl ak lafwa, sel Bondye ki ka retire nou nan sa. .. Voye mesaj sa pou 10 lot moun                                                                                                                     1\n",
      "Mwen gen anpil pwoblem mwen pa gen annyen poum manje tanpri tanpri mete yon ti kat pou mwen pou mwen ka rele yon moun ltb svp mesi dava                                                                                                                          1\n",
      "Ki danje ki genyen pou elv nan pwoven ki nan lekol an dal beton?                                                                                                                                                                                                 1\n",
      "Il faut que les stations de Radio recommencent avec les missions q                                                                                                                                                                                               1\n",
      "Di nasyon zini pou mwen yo pa vyole m se abi yo fe m, nan ka sa a sil ka ede m li t'ap bon pou mwen. MWEN BEZWEN MANJE                                                                                                                                           1\n",
      "MWEN KONTE SOU SIP NOU. NAN TOUT SANS MESI ANPIL.                                                                                                                                                                                                                1\n",
      "Pa gen problem. Map tou profite fe mesaj la pase nan kan kotem ye a ki se plas Boyer.                                                                                                                                                                            1\n",
      "Name: original, Length: 9630, dtype: int64\n",
      "*********************\n",
      "news      13068\n",
      "direct    10782\n",
      "social     2398\n",
      "Name: genre, dtype: int64\n",
      "*********************\n"
     ]
    }
   ],
   "source": [
    "# Counts of unique values\n",
    "for col in messages.columns:\n",
    "    print(messages[col].value_counts())\n",
    "    print('*********************')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 276,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 26248 entries, 0 to 26247\n",
      "Data columns (total 2 columns):\n",
      "id            26248 non-null int64\n",
      "categories    26248 non-null object\n",
      "dtypes: int64(1), object(1)\n",
      "memory usage: 410.2+ KB\n"
     ]
    }
   ],
   "source": [
    "categories.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 277,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "id            0\n",
       "categories    0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 277,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "categories.isna().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 278,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "24779    3\n",
      "7747     2\n",
      "14246    2\n",
      "25512    2\n",
      "17553    2\n",
      "13914    2\n",
      "29119    2\n",
      "14135    2\n",
      "14592    2\n",
      "17919    2\n",
      "18313    2\n",
      "6573     2\n",
      "18925    2\n",
      "14073    2\n",
      "12589    2\n",
      "6327     2\n",
      "27768    2\n",
      "23002    2\n",
      "6492     2\n",
      "15938    2\n",
      "2446     2\n",
      "4956     2\n",
      "1652     2\n",
      "7892     2\n",
      "11503    2\n",
      "15576    2\n",
      "3250     2\n",
      "5776     2\n",
      "6515     2\n",
      "24247    2\n",
      "        ..\n",
      "27384    1\n",
      "25337    1\n",
      "29435    1\n",
      "19196    1\n",
      "8913     1\n",
      "10960    1\n",
      "21183    1\n",
      "23230    1\n",
      "14994    1\n",
      "12947    1\n",
      "2708     1\n",
      "661      1\n",
      "6806     1\n",
      "4759     1\n",
      "27288    1\n",
      "25241    1\n",
      "19100    1\n",
      "17053    1\n",
      "23198    1\n",
      "21151    1\n",
      "15026    1\n",
      "12979    1\n",
      "2740     1\n",
      "6838     1\n",
      "4791     1\n",
      "27320    1\n",
      "25273    1\n",
      "29371    1\n",
      "17085    1\n",
      "2049     1\n",
      "Name: id, Length: 26180, dtype: int64\n",
      "related-0;request-0;offer-0;aid_related-0;medical_help-0;medical_products-0;search_and_rescue-0;security-0;military-0;child_alone-0;water-0;food-0;shelter-0;clothing-0;money-0;missing_people-0;refugees-0;death-0;other_aid-0;infrastructure_related-0;transport-0;buildings-0;electricity-0;tools-0;hospitals-0;shops-0;aid_centers-0;other_infrastructure-0;weather_related-0;floods-0;storm-0;fire-0;earthquake-0;cold-0;other_weather-0;direct_report-0    6125\n",
      "related-1;request-0;offer-0;aid_related-0;medical_help-0;medical_products-0;search_and_rescue-0;security-0;military-0;child_alone-0;water-0;food-0;shelter-0;clothing-0;money-0;missing_people-0;refugees-0;death-0;other_aid-0;infrastructure_related-0;transport-0;buildings-0;electricity-0;tools-0;hospitals-0;shops-0;aid_centers-0;other_infrastructure-0;weather_related-0;floods-0;storm-0;fire-0;earthquake-0;cold-0;other_weather-0;direct_report-0    5123\n",
      "related-1;request-0;offer-0;aid_related-0;medical_help-0;medical_products-0;search_and_rescue-0;security-0;military-0;child_alone-0;water-0;food-0;shelter-0;clothing-0;money-0;missing_people-0;refugees-0;death-0;other_aid-0;infrastructure_related-0;transport-0;buildings-0;electricity-0;tools-0;hospitals-0;shops-0;aid_centers-0;other_infrastructure-0;weather_related-1;floods-0;storm-0;fire-0;earthquake-1;cold-0;other_weather-0;direct_report-0     725\n",
      "related-1;request-0;offer-0;aid_related-0;medical_help-0;medical_products-0;search_and_rescue-0;security-0;military-0;child_alone-0;water-0;food-0;shelter-0;clothing-0;money-0;missing_people-0;refugees-0;death-0;other_aid-0;infrastructure_related-0;transport-0;buildings-0;electricity-0;tools-0;hospitals-0;shops-0;aid_centers-0;other_infrastructure-0;weather_related-1;floods-0;storm-1;fire-0;earthquake-0;cold-0;other_weather-0;direct_report-0     436\n",
      "related-1;request-1;offer-0;aid_related-1;medical_help-0;medical_products-0;search_and_rescue-0;security-0;military-0;child_alone-0;water-0;food-0;shelter-0;clothing-0;money-0;missing_people-0;refugees-0;death-0;other_aid-1;infrastructure_related-0;transport-0;buildings-0;electricity-0;tools-0;hospitals-0;shops-0;aid_centers-0;other_infrastructure-0;weather_related-0;floods-0;storm-0;fire-0;earthquake-0;cold-0;other_weather-0;direct_report-1     433\n",
      "related-1;request-1;offer-0;aid_related-1;medical_help-0;medical_products-0;search_and_rescue-0;security-0;military-0;child_alone-0;water-0;food-1;shelter-0;clothing-0;money-0;missing_people-0;refugees-0;death-0;other_aid-0;infrastructure_related-0;transport-0;buildings-0;electricity-0;tools-0;hospitals-0;shops-0;aid_centers-0;other_infrastructure-0;weather_related-0;floods-0;storm-0;fire-0;earthquake-0;cold-0;other_weather-0;direct_report-1     357\n",
      "related-1;request-0;offer-0;aid_related-1;medical_help-0;medical_products-0;search_and_rescue-0;security-0;military-1;child_alone-0;water-0;food-0;shelter-0;clothing-0;money-0;missing_people-0;refugees-0;death-0;other_aid-0;infrastructure_related-0;transport-0;buildings-0;electricity-0;tools-0;hospitals-0;shops-0;aid_centers-0;other_infrastructure-0;weather_related-0;floods-0;storm-0;fire-0;earthquake-0;cold-0;other_weather-0;direct_report-0     302\n",
      "related-1;request-0;offer-0;aid_related-1;medical_help-1;medical_products-0;search_and_rescue-0;security-0;military-0;child_alone-0;water-0;food-0;shelter-0;clothing-0;money-0;missing_people-0;refugees-0;death-0;other_aid-0;infrastructure_related-0;transport-0;buildings-0;electricity-0;tools-0;hospitals-0;shops-0;aid_centers-0;other_infrastructure-0;weather_related-0;floods-0;storm-0;fire-0;earthquake-0;cold-0;other_weather-0;direct_report-0     281\n",
      "related-1;request-0;offer-0;aid_related-1;medical_help-0;medical_products-0;search_and_rescue-0;security-0;military-0;child_alone-0;water-0;food-0;shelter-0;clothing-0;money-0;missing_people-0;refugees-0;death-0;other_aid-1;infrastructure_related-0;transport-0;buildings-0;electricity-0;tools-0;hospitals-0;shops-0;aid_centers-0;other_infrastructure-0;weather_related-0;floods-0;storm-0;fire-0;earthquake-0;cold-0;other_weather-0;direct_report-0     252\n",
      "related-1;request-0;offer-0;aid_related-1;medical_help-0;medical_products-0;search_and_rescue-0;security-0;military-0;child_alone-0;water-0;food-1;shelter-0;clothing-0;money-0;missing_people-0;refugees-0;death-0;other_aid-0;infrastructure_related-0;transport-0;buildings-0;electricity-0;tools-0;hospitals-0;shops-0;aid_centers-0;other_infrastructure-0;weather_related-0;floods-0;storm-0;fire-0;earthquake-0;cold-0;other_weather-0;direct_report-0     249\n",
      "related-1;request-1;offer-0;aid_related-0;medical_help-0;medical_products-0;search_and_rescue-0;security-0;military-0;child_alone-0;water-0;food-0;shelter-0;clothing-0;money-0;missing_people-0;refugees-0;death-0;other_aid-0;infrastructure_related-0;transport-0;buildings-0;electricity-0;tools-0;hospitals-0;shops-0;aid_centers-0;other_infrastructure-0;weather_related-0;floods-0;storm-0;fire-0;earthquake-0;cold-0;other_weather-0;direct_report-1     244\n",
      "related-1;request-0;offer-0;aid_related-0;medical_help-0;medical_products-0;search_and_rescue-0;security-0;military-0;child_alone-0;water-0;food-0;shelter-0;clothing-0;money-0;missing_people-0;refugees-0;death-0;other_aid-0;infrastructure_related-0;transport-0;buildings-0;electricity-0;tools-0;hospitals-0;shops-0;aid_centers-0;other_infrastructure-0;weather_related-0;floods-0;storm-0;fire-0;earthquake-0;cold-0;other_weather-0;direct_report-1     196\n",
      "related-1;request-0;offer-0;aid_related-0;medical_help-0;medical_products-0;search_and_rescue-0;security-0;military-0;child_alone-0;water-0;food-0;shelter-0;clothing-0;money-0;missing_people-0;refugees-0;death-0;other_aid-0;infrastructure_related-0;transport-0;buildings-0;electricity-0;tools-0;hospitals-0;shops-0;aid_centers-0;other_infrastructure-0;weather_related-1;floods-1;storm-0;fire-0;earthquake-0;cold-0;other_weather-0;direct_report-0     196\n",
      "related-1;request-0;offer-0;aid_related-0;medical_help-0;medical_products-0;search_and_rescue-0;security-0;military-0;child_alone-0;water-0;food-0;shelter-0;clothing-0;money-0;missing_people-0;refugees-0;death-0;other_aid-0;infrastructure_related-0;transport-0;buildings-0;electricity-0;tools-0;hospitals-0;shops-0;aid_centers-0;other_infrastructure-0;weather_related-1;floods-0;storm-1;fire-0;earthquake-0;cold-0;other_weather-0;direct_report-1     194\n",
      "related-2;request-0;offer-0;aid_related-0;medical_help-0;medical_products-0;search_and_rescue-0;security-0;military-0;child_alone-0;water-0;food-0;shelter-0;clothing-0;money-0;missing_people-0;refugees-0;death-0;other_aid-0;infrastructure_related-0;transport-0;buildings-0;electricity-0;tools-0;hospitals-0;shops-0;aid_centers-0;other_infrastructure-0;weather_related-0;floods-0;storm-0;fire-0;earthquake-0;cold-0;other_weather-0;direct_report-0     193\n",
      "related-1;request-1;offer-0;aid_related-1;medical_help-0;medical_products-0;search_and_rescue-0;security-0;military-0;child_alone-0;water-0;food-0;shelter-1;clothing-0;money-0;missing_people-0;refugees-0;death-0;other_aid-0;infrastructure_related-0;transport-0;buildings-0;electricity-0;tools-0;hospitals-0;shops-0;aid_centers-0;other_infrastructure-0;weather_related-0;floods-0;storm-0;fire-0;earthquake-0;cold-0;other_weather-0;direct_report-1     172\n",
      "related-1;request-0;offer-0;aid_related-0;medical_help-0;medical_products-0;search_and_rescue-0;security-0;military-0;child_alone-0;water-0;food-0;shelter-0;clothing-0;money-0;missing_people-0;refugees-0;death-0;other_aid-0;infrastructure_related-0;transport-0;buildings-0;electricity-0;tools-0;hospitals-0;shops-0;aid_centers-0;other_infrastructure-0;weather_related-1;floods-0;storm-0;fire-0;earthquake-0;cold-0;other_weather-1;direct_report-0     160\n",
      "related-1;request-1;offer-0;aid_related-1;medical_help-0;medical_products-0;search_and_rescue-0;security-0;military-0;child_alone-0;water-1;food-1;shelter-0;clothing-0;money-0;missing_people-0;refugees-0;death-0;other_aid-0;infrastructure_related-0;transport-0;buildings-0;electricity-0;tools-0;hospitals-0;shops-0;aid_centers-0;other_infrastructure-0;weather_related-0;floods-0;storm-0;fire-0;earthquake-0;cold-0;other_weather-0;direct_report-1     153\n",
      "related-1;request-0;offer-0;aid_related-1;medical_help-0;medical_products-0;search_and_rescue-0;security-0;military-0;child_alone-0;water-0;food-0;shelter-0;clothing-0;money-0;missing_people-0;refugees-0;death-0;other_aid-1;infrastructure_related-0;transport-0;buildings-0;electricity-0;tools-0;hospitals-0;shops-0;aid_centers-0;other_infrastructure-0;weather_related-1;floods-0;storm-0;fire-0;earthquake-1;cold-0;other_weather-0;direct_report-0     130\n",
      "related-1;request-0;offer-0;aid_related-0;medical_help-0;medical_products-0;search_and_rescue-0;security-0;military-0;child_alone-0;water-0;food-0;shelter-0;clothing-0;money-0;missing_people-0;refugees-0;death-0;other_aid-0;infrastructure_related-0;transport-0;buildings-0;electricity-0;tools-0;hospitals-0;shops-0;aid_centers-0;other_infrastructure-0;weather_related-1;floods-0;storm-0;fire-0;earthquake-1;cold-0;other_weather-0;direct_report-1     129\n",
      "related-1;request-0;offer-0;aid_related-0;medical_help-0;medical_products-0;search_and_rescue-0;security-0;military-0;child_alone-0;water-0;food-0;shelter-0;clothing-0;money-0;missing_people-0;refugees-0;death-0;other_aid-0;infrastructure_related-0;transport-0;buildings-0;electricity-0;tools-0;hospitals-0;shops-0;aid_centers-0;other_infrastructure-0;weather_related-1;floods-1;storm-1;fire-0;earthquake-0;cold-0;other_weather-0;direct_report-0     127\n",
      "related-1;request-0;offer-0;aid_related-1;medical_help-0;medical_products-0;search_and_rescue-0;security-0;military-0;child_alone-0;water-1;food-0;shelter-0;clothing-0;money-0;missing_people-0;refugees-0;death-0;other_aid-0;infrastructure_related-0;transport-0;buildings-0;electricity-0;tools-0;hospitals-0;shops-0;aid_centers-0;other_infrastructure-0;weather_related-0;floods-0;storm-0;fire-0;earthquake-0;cold-0;other_weather-0;direct_report-0     120\n",
      "related-1;request-1;offer-0;aid_related-0;medical_help-0;medical_products-0;search_and_rescue-0;security-0;military-0;child_alone-0;water-0;food-0;shelter-0;clothing-0;money-0;missing_people-0;refugees-0;death-0;other_aid-0;infrastructure_related-0;transport-0;buildings-0;electricity-0;tools-0;hospitals-0;shops-0;aid_centers-0;other_infrastructure-0;weather_related-0;floods-0;storm-0;fire-0;earthquake-0;cold-0;other_weather-0;direct_report-0     105\n",
      "related-1;request-0;offer-0;aid_related-1;medical_help-0;medical_products-0;search_and_rescue-0;security-0;military-0;child_alone-0;water-0;food-0;shelter-1;clothing-0;money-0;missing_people-0;refugees-0;death-0;other_aid-0;infrastructure_related-0;transport-0;buildings-0;electricity-0;tools-0;hospitals-0;shops-0;aid_centers-0;other_infrastructure-0;weather_related-0;floods-0;storm-0;fire-0;earthquake-0;cold-0;other_weather-0;direct_report-0      97\n",
      "related-1;request-1;offer-0;aid_related-1;medical_help-0;medical_products-0;search_and_rescue-0;security-0;military-0;child_alone-0;water-1;food-1;shelter-1;clothing-0;money-0;missing_people-0;refugees-0;death-0;other_aid-0;infrastructure_related-0;transport-0;buildings-0;electricity-0;tools-0;hospitals-0;shops-0;aid_centers-0;other_infrastructure-0;weather_related-0;floods-0;storm-0;fire-0;earthquake-0;cold-0;other_weather-0;direct_report-1      94\n",
      "related-1;request-0;offer-0;aid_related-1;medical_help-0;medical_products-0;search_and_rescue-0;security-0;military-0;child_alone-0;water-0;food-0;shelter-0;clothing-0;money-0;missing_people-0;refugees-0;death-1;other_aid-0;infrastructure_related-0;transport-0;buildings-0;electricity-0;tools-0;hospitals-0;shops-0;aid_centers-0;other_infrastructure-0;weather_related-0;floods-0;storm-0;fire-0;earthquake-0;cold-0;other_weather-0;direct_report-0      87\n",
      "related-1;request-1;offer-0;aid_related-1;medical_help-0;medical_products-0;search_and_rescue-0;security-0;military-0;child_alone-0;water-0;food-0;shelter-0;clothing-0;money-0;missing_people-0;refugees-0;death-0;other_aid-1;infrastructure_related-0;transport-0;buildings-0;electricity-0;tools-0;hospitals-0;shops-0;aid_centers-0;other_infrastructure-0;weather_related-0;floods-0;storm-0;fire-0;earthquake-0;cold-0;other_weather-0;direct_report-0      86\n",
      "related-1;request-1;offer-0;aid_related-1;medical_help-0;medical_products-0;search_and_rescue-0;security-0;military-0;child_alone-0;water-1;food-0;shelter-0;clothing-0;money-0;missing_people-0;refugees-0;death-0;other_aid-0;infrastructure_related-0;transport-0;buildings-0;electricity-0;tools-0;hospitals-0;shops-0;aid_centers-0;other_infrastructure-0;weather_related-0;floods-0;storm-0;fire-0;earthquake-0;cold-0;other_weather-0;direct_report-1      79\n",
      "related-1;request-1;offer-0;aid_related-1;medical_help-0;medical_products-0;search_and_rescue-0;security-0;military-0;child_alone-0;water-0;food-1;shelter-1;clothing-0;money-0;missing_people-0;refugees-0;death-0;other_aid-0;infrastructure_related-0;transport-0;buildings-0;electricity-0;tools-0;hospitals-0;shops-0;aid_centers-0;other_infrastructure-0;weather_related-0;floods-0;storm-0;fire-0;earthquake-0;cold-0;other_weather-0;direct_report-1      79\n",
      "related-1;request-1;offer-0;aid_related-1;medical_help-0;medical_products-0;search_and_rescue-0;security-0;military-0;child_alone-0;water-0;food-1;shelter-0;clothing-0;money-0;missing_people-0;refugees-0;death-0;other_aid-0;infrastructure_related-0;transport-0;buildings-0;electricity-0;tools-0;hospitals-0;shops-0;aid_centers-0;other_infrastructure-0;weather_related-0;floods-0;storm-0;fire-0;earthquake-0;cold-0;other_weather-0;direct_report-0      79\n",
      "                                                                                                                                                                                                                                                                                                                                                                                                                                                                 ... \n",
      "related-1;request-0;offer-0;aid_related-1;medical_help-0;medical_products-0;search_and_rescue-0;security-0;military-0;child_alone-0;water-1;food-0;shelter-0;clothing-0;money-0;missing_people-0;refugees-0;death-1;other_aid-0;infrastructure_related-1;transport-0;buildings-0;electricity-0;tools-0;hospitals-0;shops-1;aid_centers-0;other_infrastructure-0;weather_related-1;floods-0;storm-0;fire-0;earthquake-1;cold-1;other_weather-0;direct_report-1       1\n",
      "related-1;request-0;offer-0;aid_related-1;medical_help-1;medical_products-0;search_and_rescue-0;security-0;military-0;child_alone-0;water-0;food-0;shelter-0;clothing-0;money-0;missing_people-0;refugees-0;death-0;other_aid-0;infrastructure_related-1;transport-0;buildings-0;electricity-0;tools-0;hospitals-1;shops-0;aid_centers-0;other_infrastructure-0;weather_related-1;floods-0;storm-0;fire-0;earthquake-1;cold-0;other_weather-0;direct_report-0       1\n",
      "related-1;request-0;offer-0;aid_related-1;medical_help-1;medical_products-0;search_and_rescue-0;security-0;military-0;child_alone-0;water-0;food-0;shelter-0;clothing-0;money-0;missing_people-0;refugees-1;death-0;other_aid-0;infrastructure_related-0;transport-1;buildings-0;electricity-0;tools-0;hospitals-0;shops-0;aid_centers-0;other_infrastructure-0;weather_related-1;floods-0;storm-1;fire-0;earthquake-0;cold-0;other_weather-0;direct_report-1       1\n",
      "related-1;request-0;offer-0;aid_related-1;medical_help-0;medical_products-0;search_and_rescue-0;security-0;military-0;child_alone-0;water-0;food-0;shelter-0;clothing-0;money-0;missing_people-1;refugees-0;death-1;other_aid-0;infrastructure_related-0;transport-0;buildings-0;electricity-0;tools-1;hospitals-0;shops-0;aid_centers-0;other_infrastructure-0;weather_related-1;floods-0;storm-1;fire-0;earthquake-1;cold-0;other_weather-0;direct_report-0       1\n",
      "related-1;request-0;offer-0;aid_related-1;medical_help-0;medical_products-0;search_and_rescue-0;security-0;military-1;child_alone-0;water-0;food-0;shelter-0;clothing-0;money-0;missing_people-0;refugees-0;death-0;other_aid-1;infrastructure_related-0;transport-0;buildings-0;electricity-0;tools-0;hospitals-0;shops-0;aid_centers-0;other_infrastructure-0;weather_related-1;floods-0;storm-1;fire-0;earthquake-0;cold-0;other_weather-0;direct_report-0       1\n",
      "related-1;request-0;offer-0;aid_related-1;medical_help-0;medical_products-0;search_and_rescue-0;security-0;military-0;child_alone-0;water-1;food-0;shelter-1;clothing-0;money-0;missing_people-0;refugees-0;death-0;other_aid-0;infrastructure_related-1;transport-1;buildings-1;electricity-0;tools-0;hospitals-1;shops-0;aid_centers-1;other_infrastructure-1;weather_related-0;floods-0;storm-0;fire-0;earthquake-0;cold-0;other_weather-0;direct_report-0       1\n",
      "related-1;request-0;offer-0;aid_related-1;medical_help-0;medical_products-0;search_and_rescue-0;security-0;military-0;child_alone-0;water-1;food-1;shelter-1;clothing-0;money-1;missing_people-0;refugees-0;death-0;other_aid-1;infrastructure_related-1;transport-0;buildings-0;electricity-0;tools-0;hospitals-0;shops-0;aid_centers-0;other_infrastructure-1;weather_related-0;floods-0;storm-0;fire-0;earthquake-0;cold-0;other_weather-0;direct_report-1       1\n",
      "related-1;request-0;offer-0;aid_related-1;medical_help-0;medical_products-0;search_and_rescue-0;security-1;military-0;child_alone-0;water-0;food-0;shelter-1;clothing-0;money-0;missing_people-0;refugees-0;death-0;other_aid-0;infrastructure_related-1;transport-0;buildings-0;electricity-0;tools-0;hospitals-0;shops-0;aid_centers-1;other_infrastructure-0;weather_related-1;floods-0;storm-0;fire-0;earthquake-1;cold-0;other_weather-0;direct_report-0       1\n",
      "related-1;request-0;offer-0;aid_related-1;medical_help-1;medical_products-1;search_and_rescue-0;security-0;military-1;child_alone-0;water-0;food-0;shelter-0;clothing-0;money-0;missing_people-0;refugees-0;death-1;other_aid-0;infrastructure_related-0;transport-1;buildings-0;electricity-0;tools-0;hospitals-0;shops-0;aid_centers-0;other_infrastructure-0;weather_related-0;floods-0;storm-0;fire-0;earthquake-0;cold-0;other_weather-0;direct_report-0       1\n",
      "related-1;request-0;offer-0;aid_related-1;medical_help-0;medical_products-0;search_and_rescue-0;security-0;military-0;child_alone-0;water-0;food-0;shelter-1;clothing-0;money-0;missing_people-0;refugees-0;death-0;other_aid-0;infrastructure_related-1;transport-0;buildings-0;electricity-0;tools-0;hospitals-0;shops-0;aid_centers-1;other_infrastructure-0;weather_related-1;floods-0;storm-0;fire-0;earthquake-0;cold-0;other_weather-1;direct_report-0       1\n",
      "related-1;request-0;offer-0;aid_related-1;medical_help-0;medical_products-0;search_and_rescue-0;security-0;military-1;child_alone-0;water-0;food-0;shelter-0;clothing-0;money-0;missing_people-0;refugees-0;death-0;other_aid-1;infrastructure_related-1;transport-0;buildings-0;electricity-0;tools-0;hospitals-0;shops-0;aid_centers-0;other_infrastructure-1;weather_related-1;floods-0;storm-0;fire-0;earthquake-1;cold-0;other_weather-0;direct_report-0       1\n",
      "related-1;request-1;offer-0;aid_related-1;medical_help-0;medical_products-1;search_and_rescue-0;security-0;military-0;child_alone-0;water-1;food-1;shelter-1;clothing-0;money-0;missing_people-0;refugees-0;death-0;other_aid-0;infrastructure_related-0;transport-0;buildings-0;electricity-0;tools-0;hospitals-0;shops-0;aid_centers-0;other_infrastructure-0;weather_related-1;floods-1;storm-0;fire-0;earthquake-0;cold-0;other_weather-0;direct_report-0       1\n",
      "related-1;request-1;offer-0;aid_related-1;medical_help-0;medical_products-0;search_and_rescue-0;security-0;military-0;child_alone-0;water-0;food-0;shelter-0;clothing-0;money-0;missing_people-0;refugees-0;death-0;other_aid-1;infrastructure_related-1;transport-0;buildings-0;electricity-0;tools-0;hospitals-0;shops-0;aid_centers-0;other_infrastructure-1;weather_related-1;floods-0;storm-0;fire-0;earthquake-1;cold-0;other_weather-0;direct_report-1       1\n",
      "related-1;request-0;offer-0;aid_related-0;medical_help-0;medical_products-0;search_and_rescue-0;security-0;military-0;child_alone-0;water-0;food-0;shelter-0;clothing-0;money-0;missing_people-0;refugees-0;death-0;other_aid-0;infrastructure_related-0;transport-1;buildings-0;electricity-1;tools-0;hospitals-0;shops-0;aid_centers-0;other_infrastructure-0;weather_related-1;floods-1;storm-0;fire-0;earthquake-0;cold-0;other_weather-0;direct_report-0       1\n",
      "related-1;request-1;offer-0;aid_related-1;medical_help-0;medical_products-0;search_and_rescue-0;security-0;military-0;child_alone-0;water-0;food-0;shelter-0;clothing-0;money-0;missing_people-0;refugees-0;death-0;other_aid-1;infrastructure_related-1;transport-0;buildings-0;electricity-0;tools-0;hospitals-0;shops-0;aid_centers-0;other_infrastructure-1;weather_related-1;floods-0;storm-0;fire-0;earthquake-1;cold-0;other_weather-1;direct_report-0       1\n",
      "related-1;request-1;offer-0;aid_related-1;medical_help-0;medical_products-0;search_and_rescue-0;security-0;military-0;child_alone-0;water-0;food-0;shelter-1;clothing-0;money-0;missing_people-0;refugees-1;death-0;other_aid-0;infrastructure_related-0;transport-0;buildings-1;electricity-0;tools-0;hospitals-0;shops-0;aid_centers-0;other_infrastructure-0;weather_related-0;floods-0;storm-0;fire-0;earthquake-0;cold-0;other_weather-0;direct_report-1       1\n",
      "related-1;request-1;offer-0;aid_related-1;medical_help-0;medical_products-0;search_and_rescue-0;security-0;military-1;child_alone-0;water-0;food-0;shelter-0;clothing-0;money-0;missing_people-0;refugees-1;death-0;other_aid-0;infrastructure_related-0;transport-0;buildings-0;electricity-0;tools-0;hospitals-0;shops-0;aid_centers-0;other_infrastructure-0;weather_related-0;floods-0;storm-0;fire-0;earthquake-0;cold-0;other_weather-0;direct_report-1       1\n",
      "related-1;request-1;offer-0;aid_related-1;medical_help-0;medical_products-0;search_and_rescue-0;security-0;military-0;child_alone-0;water-0;food-0;shelter-0;clothing-0;money-0;missing_people-0;refugees-0;death-1;other_aid-1;infrastructure_related-1;transport-0;buildings-0;electricity-0;tools-0;hospitals-0;shops-0;aid_centers-0;other_infrastructure-1;weather_related-1;floods-0;storm-0;fire-1;earthquake-0;cold-0;other_weather-0;direct_report-0       1\n",
      "related-1;request-0;offer-0;aid_related-1;medical_help-0;medical_products-0;search_and_rescue-0;security-1;military-0;child_alone-0;water-0;food-0;shelter-0;clothing-0;money-0;missing_people-0;refugees-0;death-0;other_aid-0;infrastructure_related-0;transport-0;buildings-0;electricity-1;tools-0;hospitals-0;shops-0;aid_centers-0;other_infrastructure-0;weather_related-0;floods-0;storm-0;fire-0;earthquake-0;cold-0;other_weather-0;direct_report-1       1\n",
      "related-1;request-0;offer-0;aid_related-1;medical_help-0;medical_products-1;search_and_rescue-0;security-0;military-0;child_alone-0;water-0;food-0;shelter-0;clothing-0;money-0;missing_people-0;refugees-0;death-0;other_aid-0;infrastructure_related-0;transport-0;buildings-0;electricity-0;tools-1;hospitals-0;shops-0;aid_centers-0;other_infrastructure-0;weather_related-1;floods-1;storm-1;fire-0;earthquake-0;cold-0;other_weather-1;direct_report-1       1\n",
      "related-1;request-0;offer-0;aid_related-1;medical_help-0;medical_products-0;search_and_rescue-1;security-0;military-0;child_alone-0;water-0;food-0;shelter-0;clothing-0;money-0;missing_people-1;refugees-0;death-0;other_aid-0;infrastructure_related-0;transport-1;buildings-1;electricity-0;tools-0;hospitals-0;shops-0;aid_centers-0;other_infrastructure-0;weather_related-1;floods-0;storm-0;fire-0;earthquake-1;cold-0;other_weather-0;direct_report-0       1\n",
      "related-1;request-0;offer-0;aid_related-1;medical_help-0;medical_products-0;search_and_rescue-0;security-0;military-0;child_alone-0;water-0;food-0;shelter-0;clothing-0;money-0;missing_people-1;refugees-0;death-0;other_aid-0;infrastructure_related-0;transport-0;buildings-0;electricity-0;tools-0;hospitals-0;shops-0;aid_centers-0;other_infrastructure-0;weather_related-1;floods-0;storm-1;fire-0;earthquake-0;cold-0;other_weather-0;direct_report-0       1\n",
      "related-1;request-1;offer-0;aid_related-1;medical_help-0;medical_products-0;search_and_rescue-1;security-0;military-0;child_alone-0;water-0;food-0;shelter-0;clothing-0;money-0;missing_people-0;refugees-0;death-0;other_aid-0;infrastructure_related-1;transport-0;buildings-0;electricity-0;tools-0;hospitals-0;shops-0;aid_centers-0;other_infrastructure-1;weather_related-0;floods-0;storm-0;fire-0;earthquake-0;cold-0;other_weather-0;direct_report-0       1\n",
      "related-1;request-1;offer-0;aid_related-1;medical_help-1;medical_products-1;search_and_rescue-0;security-0;military-0;child_alone-0;water-0;food-0;shelter-0;clothing-0;money-0;missing_people-0;refugees-0;death-0;other_aid-0;infrastructure_related-0;transport-0;buildings-0;electricity-1;tools-0;hospitals-0;shops-0;aid_centers-0;other_infrastructure-0;weather_related-1;floods-0;storm-0;fire-0;earthquake-0;cold-1;other_weather-0;direct_report-1       1\n",
      "related-1;request-0;offer-0;aid_related-1;medical_help-0;medical_products-0;search_and_rescue-1;security-0;military-0;child_alone-0;water-0;food-0;shelter-0;clothing-0;money-0;missing_people-1;refugees-1;death-1;other_aid-0;infrastructure_related-0;transport-0;buildings-0;electricity-0;tools-0;hospitals-0;shops-0;aid_centers-0;other_infrastructure-0;weather_related-1;floods-0;storm-0;fire-0;earthquake-0;cold-0;other_weather-1;direct_report-0       1\n",
      "related-1;request-1;offer-0;aid_related-1;medical_help-0;medical_products-0;search_and_rescue-0;security-0;military-0;child_alone-0;water-1;food-0;shelter-0;clothing-0;money-0;missing_people-0;refugees-0;death-0;other_aid-1;infrastructure_related-0;transport-0;buildings-1;electricity-0;tools-0;hospitals-0;shops-0;aid_centers-0;other_infrastructure-0;weather_related-1;floods-0;storm-0;fire-0;earthquake-1;cold-0;other_weather-0;direct_report-1       1\n",
      "related-1;request-1;offer-0;aid_related-1;medical_help-0;medical_products-0;search_and_rescue-0;security-0;military-0;child_alone-0;water-0;food-0;shelter-0;clothing-0;money-0;missing_people-0;refugees-1;death-0;other_aid-1;infrastructure_related-1;transport-0;buildings-1;electricity-0;tools-0;hospitals-0;shops-0;aid_centers-0;other_infrastructure-1;weather_related-0;floods-0;storm-0;fire-0;earthquake-0;cold-0;other_weather-0;direct_report-1       1\n",
      "related-1;request-0;offer-0;aid_related-1;medical_help-0;medical_products-0;search_and_rescue-0;security-0;military-0;child_alone-0;water-1;food-0;shelter-0;clothing-0;money-0;missing_people-0;refugees-0;death-0;other_aid-1;infrastructure_related-0;transport-0;buildings-0;electricity-1;tools-0;hospitals-0;shops-0;aid_centers-0;other_infrastructure-0;weather_related-1;floods-0;storm-1;fire-0;earthquake-0;cold-0;other_weather-1;direct_report-0       1\n",
      "related-1;request-1;offer-0;aid_related-1;medical_help-1;medical_products-1;search_and_rescue-0;security-0;military-0;child_alone-0;water-1;food-1;shelter-1;clothing-0;money-0;missing_people-0;refugees-0;death-0;other_aid-0;infrastructure_related-0;transport-0;buildings-1;electricity-0;tools-0;hospitals-0;shops-0;aid_centers-0;other_infrastructure-0;weather_related-0;floods-0;storm-0;fire-0;earthquake-0;cold-0;other_weather-0;direct_report-1       1\n",
      "related-1;request-0;offer-0;aid_related-1;medical_help-0;medical_products-0;search_and_rescue-1;security-0;military-0;child_alone-0;water-0;food-1;shelter-1;clothing-0;money-0;missing_people-1;refugees-1;death-0;other_aid-0;infrastructure_related-1;transport-0;buildings-0;electricity-0;tools-0;hospitals-0;shops-0;aid_centers-0;other_infrastructure-1;weather_related-1;floods-0;storm-0;fire-0;earthquake-0;cold-0;other_weather-1;direct_report-0       1\n",
      "Name: categories, Length: 4003, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "# Count unique values for categories dataset\n",
    "for col in categories:\n",
    "    print(categories[col].value_counts())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 279,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "26180"
      ]
     },
     "execution_count": 279,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check similar records in both datasets\n",
    "len(set(list(categories['id'])).intersection(list(messages['id'])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Split `categories` into separate category columns.\n",
    "- Split the values in the `categories` column on the `;` character so that each value becomes a separate column. You'll find [this method](https://pandas.pydata.org/pandas-docs/version/0.23/generated/pandas.Series.str.split.html) very helpful! Make sure to set `expand=True`.\n",
    "- Use the first row of categories dataframe to create column names for the categories data.\n",
    "- Rename columns of `categories` with new column names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 280,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>26</th>\n",
       "      <th>27</th>\n",
       "      <th>28</th>\n",
       "      <th>29</th>\n",
       "      <th>30</th>\n",
       "      <th>31</th>\n",
       "      <th>32</th>\n",
       "      <th>33</th>\n",
       "      <th>34</th>\n",
       "      <th>35</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>related-1</td>\n",
       "      <td>request-0</td>\n",
       "      <td>offer-0</td>\n",
       "      <td>aid_related-0</td>\n",
       "      <td>medical_help-0</td>\n",
       "      <td>medical_products-0</td>\n",
       "      <td>search_and_rescue-0</td>\n",
       "      <td>security-0</td>\n",
       "      <td>military-0</td>\n",
       "      <td>child_alone-0</td>\n",
       "      <td>...</td>\n",
       "      <td>aid_centers-0</td>\n",
       "      <td>other_infrastructure-0</td>\n",
       "      <td>weather_related-0</td>\n",
       "      <td>floods-0</td>\n",
       "      <td>storm-0</td>\n",
       "      <td>fire-0</td>\n",
       "      <td>earthquake-0</td>\n",
       "      <td>cold-0</td>\n",
       "      <td>other_weather-0</td>\n",
       "      <td>direct_report-0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>related-1</td>\n",
       "      <td>request-0</td>\n",
       "      <td>offer-0</td>\n",
       "      <td>aid_related-1</td>\n",
       "      <td>medical_help-0</td>\n",
       "      <td>medical_products-0</td>\n",
       "      <td>search_and_rescue-0</td>\n",
       "      <td>security-0</td>\n",
       "      <td>military-0</td>\n",
       "      <td>child_alone-0</td>\n",
       "      <td>...</td>\n",
       "      <td>aid_centers-0</td>\n",
       "      <td>other_infrastructure-0</td>\n",
       "      <td>weather_related-1</td>\n",
       "      <td>floods-0</td>\n",
       "      <td>storm-1</td>\n",
       "      <td>fire-0</td>\n",
       "      <td>earthquake-0</td>\n",
       "      <td>cold-0</td>\n",
       "      <td>other_weather-0</td>\n",
       "      <td>direct_report-0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>related-1</td>\n",
       "      <td>request-0</td>\n",
       "      <td>offer-0</td>\n",
       "      <td>aid_related-0</td>\n",
       "      <td>medical_help-0</td>\n",
       "      <td>medical_products-0</td>\n",
       "      <td>search_and_rescue-0</td>\n",
       "      <td>security-0</td>\n",
       "      <td>military-0</td>\n",
       "      <td>child_alone-0</td>\n",
       "      <td>...</td>\n",
       "      <td>aid_centers-0</td>\n",
       "      <td>other_infrastructure-0</td>\n",
       "      <td>weather_related-0</td>\n",
       "      <td>floods-0</td>\n",
       "      <td>storm-0</td>\n",
       "      <td>fire-0</td>\n",
       "      <td>earthquake-0</td>\n",
       "      <td>cold-0</td>\n",
       "      <td>other_weather-0</td>\n",
       "      <td>direct_report-0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>related-1</td>\n",
       "      <td>request-1</td>\n",
       "      <td>offer-0</td>\n",
       "      <td>aid_related-1</td>\n",
       "      <td>medical_help-0</td>\n",
       "      <td>medical_products-1</td>\n",
       "      <td>search_and_rescue-0</td>\n",
       "      <td>security-0</td>\n",
       "      <td>military-0</td>\n",
       "      <td>child_alone-0</td>\n",
       "      <td>...</td>\n",
       "      <td>aid_centers-0</td>\n",
       "      <td>other_infrastructure-0</td>\n",
       "      <td>weather_related-0</td>\n",
       "      <td>floods-0</td>\n",
       "      <td>storm-0</td>\n",
       "      <td>fire-0</td>\n",
       "      <td>earthquake-0</td>\n",
       "      <td>cold-0</td>\n",
       "      <td>other_weather-0</td>\n",
       "      <td>direct_report-0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>related-1</td>\n",
       "      <td>request-0</td>\n",
       "      <td>offer-0</td>\n",
       "      <td>aid_related-0</td>\n",
       "      <td>medical_help-0</td>\n",
       "      <td>medical_products-0</td>\n",
       "      <td>search_and_rescue-0</td>\n",
       "      <td>security-0</td>\n",
       "      <td>military-0</td>\n",
       "      <td>child_alone-0</td>\n",
       "      <td>...</td>\n",
       "      <td>aid_centers-0</td>\n",
       "      <td>other_infrastructure-0</td>\n",
       "      <td>weather_related-0</td>\n",
       "      <td>floods-0</td>\n",
       "      <td>storm-0</td>\n",
       "      <td>fire-0</td>\n",
       "      <td>earthquake-0</td>\n",
       "      <td>cold-0</td>\n",
       "      <td>other_weather-0</td>\n",
       "      <td>direct_report-0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 36 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          0          1        2              3               4   \\\n",
       "0  related-1  request-0  offer-0  aid_related-0  medical_help-0   \n",
       "1  related-1  request-0  offer-0  aid_related-1  medical_help-0   \n",
       "2  related-1  request-0  offer-0  aid_related-0  medical_help-0   \n",
       "3  related-1  request-1  offer-0  aid_related-1  medical_help-0   \n",
       "4  related-1  request-0  offer-0  aid_related-0  medical_help-0   \n",
       "\n",
       "                   5                    6           7           8   \\\n",
       "0  medical_products-0  search_and_rescue-0  security-0  military-0   \n",
       "1  medical_products-0  search_and_rescue-0  security-0  military-0   \n",
       "2  medical_products-0  search_and_rescue-0  security-0  military-0   \n",
       "3  medical_products-1  search_and_rescue-0  security-0  military-0   \n",
       "4  medical_products-0  search_and_rescue-0  security-0  military-0   \n",
       "\n",
       "              9        ...                    26                      27  \\\n",
       "0  child_alone-0       ...         aid_centers-0  other_infrastructure-0   \n",
       "1  child_alone-0       ...         aid_centers-0  other_infrastructure-0   \n",
       "2  child_alone-0       ...         aid_centers-0  other_infrastructure-0   \n",
       "3  child_alone-0       ...         aid_centers-0  other_infrastructure-0   \n",
       "4  child_alone-0       ...         aid_centers-0  other_infrastructure-0   \n",
       "\n",
       "                  28        29       30      31            32      33  \\\n",
       "0  weather_related-0  floods-0  storm-0  fire-0  earthquake-0  cold-0   \n",
       "1  weather_related-1  floods-0  storm-1  fire-0  earthquake-0  cold-0   \n",
       "2  weather_related-0  floods-0  storm-0  fire-0  earthquake-0  cold-0   \n",
       "3  weather_related-0  floods-0  storm-0  fire-0  earthquake-0  cold-0   \n",
       "4  weather_related-0  floods-0  storm-0  fire-0  earthquake-0  cold-0   \n",
       "\n",
       "                34               35  \n",
       "0  other_weather-0  direct_report-0  \n",
       "1  other_weather-0  direct_report-0  \n",
       "2  other_weather-0  direct_report-0  \n",
       "3  other_weather-0  direct_report-0  \n",
       "4  other_weather-0  direct_report-0  \n",
       "\n",
       "[5 rows x 36 columns]"
      ]
     },
     "execution_count": 280,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# create a dataframe of the 36 individual category columns\n",
    "categories_split = categories['categories'].str.split(';', expand = True)\n",
    "categories_split.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 281,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0                    related\n",
      "1                    request\n",
      "2                      offer\n",
      "3                aid_related\n",
      "4               medical_help\n",
      "5           medical_products\n",
      "6          search_and_rescue\n",
      "7                   security\n",
      "8                   military\n",
      "9                child_alone\n",
      "10                     water\n",
      "11                      food\n",
      "12                   shelter\n",
      "13                  clothing\n",
      "14                     money\n",
      "15            missing_people\n",
      "16                  refugees\n",
      "17                     death\n",
      "18                 other_aid\n",
      "19    infrastructure_related\n",
      "20                 transport\n",
      "21                 buildings\n",
      "22               electricity\n",
      "23                     tools\n",
      "24                 hospitals\n",
      "25                     shops\n",
      "26               aid_centers\n",
      "27      other_infrastructure\n",
      "28           weather_related\n",
      "29                    floods\n",
      "30                     storm\n",
      "31                      fire\n",
      "32                earthquake\n",
      "33                      cold\n",
      "34             other_weather\n",
      "35             direct_report\n",
      "Name: 0, dtype: object\n"
     ]
    }
   ],
   "source": [
    "# select the first row of the categories dataframe\n",
    "row = categories_split.iloc[0]\n",
    "\n",
    "# use this row to extract a list of new column names for categories.\n",
    "# one way is to apply a lambda function that takes everything \n",
    "# up to the second to last character of each string with slicing\n",
    "category_colnames = row.apply(lambda x: x[:-2])\n",
    "print(category_colnames)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 282,
   "metadata": {},
   "outputs": [],
   "source": [
    "categories_split.columns = category_colnames"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Convert `categories_split` values to just numbers 0 or 1 and merge it with `categories` dataset\n",
    "- Iterate through the category columns in df to keep only the last character of each string (the 1 or 0). For example, `related-0` becomes `0`, `related-1` becomes `1`. Convert the string to a numeric value.\n",
    "- You can perform [normal string actions on Pandas Series](https://pandas.pydata.org/pandas-docs/stable/text.html#indexing-with-str), like indexing, by including `.str` after the Series. You may need to first convert the Series to be of type string, which you can do with `astype(str)`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 283,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>related</th>\n",
       "      <th>request</th>\n",
       "      <th>offer</th>\n",
       "      <th>aid_related</th>\n",
       "      <th>medical_help</th>\n",
       "      <th>medical_products</th>\n",
       "      <th>search_and_rescue</th>\n",
       "      <th>security</th>\n",
       "      <th>military</th>\n",
       "      <th>child_alone</th>\n",
       "      <th>...</th>\n",
       "      <th>aid_centers</th>\n",
       "      <th>other_infrastructure</th>\n",
       "      <th>weather_related</th>\n",
       "      <th>floods</th>\n",
       "      <th>storm</th>\n",
       "      <th>fire</th>\n",
       "      <th>earthquake</th>\n",
       "      <th>cold</th>\n",
       "      <th>other_weather</th>\n",
       "      <th>direct_report</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 36 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "0  related  request  offer  aid_related  medical_help  medical_products  \\\n",
       "0        1        0      0            0             0                 0   \n",
       "1        1        0      0            1             0                 0   \n",
       "2        1        0      0            0             0                 0   \n",
       "3        1        1      0            1             0                 1   \n",
       "4        1        0      0            0             0                 0   \n",
       "\n",
       "0  search_and_rescue  security  military  child_alone      ...        \\\n",
       "0                  0         0         0            0      ...         \n",
       "1                  0         0         0            0      ...         \n",
       "2                  0         0         0            0      ...         \n",
       "3                  0         0         0            0      ...         \n",
       "4                  0         0         0            0      ...         \n",
       "\n",
       "0  aid_centers  other_infrastructure  weather_related  floods  storm  fire  \\\n",
       "0            0                     0                0       0      0     0   \n",
       "1            0                     0                1       0      1     0   \n",
       "2            0                     0                0       0      0     0   \n",
       "3            0                     0                0       0      0     0   \n",
       "4            0                     0                0       0      0     0   \n",
       "\n",
       "0  earthquake  cold  other_weather  direct_report  \n",
       "0           0     0              0              0  \n",
       "1           0     0              0              0  \n",
       "2           0     0              0              0  \n",
       "3           0     0              0              0  \n",
       "4           0     0              0              0  \n",
       "\n",
       "[5 rows x 36 columns]"
      ]
     },
     "execution_count": 283,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "for column in categories_split:\n",
    "    # set each value to be the last character of the string\n",
    "    categories_split[column] = categories_split[column].apply(lambda x: x[-1])\n",
    "    \n",
    "    # convert column from string to numeric\n",
    "    categories_split[column] = categories_split[column].astype(np.int64)\n",
    "categories_split.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 284,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(26248, 2)"
      ]
     },
     "execution_count": 284,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "categories.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 285,
   "metadata": {},
   "outputs": [],
   "source": [
    "categories = categories.drop('categories', axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 286,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>related</th>\n",
       "      <th>request</th>\n",
       "      <th>offer</th>\n",
       "      <th>aid_related</th>\n",
       "      <th>medical_help</th>\n",
       "      <th>medical_products</th>\n",
       "      <th>search_and_rescue</th>\n",
       "      <th>security</th>\n",
       "      <th>military</th>\n",
       "      <th>...</th>\n",
       "      <th>aid_centers</th>\n",
       "      <th>other_infrastructure</th>\n",
       "      <th>weather_related</th>\n",
       "      <th>floods</th>\n",
       "      <th>storm</th>\n",
       "      <th>fire</th>\n",
       "      <th>earthquake</th>\n",
       "      <th>cold</th>\n",
       "      <th>other_weather</th>\n",
       "      <th>direct_report</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>7</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>9</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>12</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 37 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   id  related  request  offer  aid_related  medical_help  medical_products  \\\n",
       "0   2        1        0      0            0             0                 0   \n",
       "1   7        1        0      0            1             0                 0   \n",
       "2   8        1        0      0            0             0                 0   \n",
       "3   9        1        1      0            1             0                 1   \n",
       "4  12        1        0      0            0             0                 0   \n",
       "\n",
       "   search_and_rescue  security  military      ...        aid_centers  \\\n",
       "0                  0         0         0      ...                  0   \n",
       "1                  0         0         0      ...                  0   \n",
       "2                  0         0         0      ...                  0   \n",
       "3                  0         0         0      ...                  0   \n",
       "4                  0         0         0      ...                  0   \n",
       "\n",
       "   other_infrastructure  weather_related  floods  storm  fire  earthquake  \\\n",
       "0                     0                0       0      0     0           0   \n",
       "1                     0                1       0      1     0           0   \n",
       "2                     0                0       0      0     0           0   \n",
       "3                     0                0       0      0     0           0   \n",
       "4                     0                0       0      0     0           0   \n",
       "\n",
       "   cold  other_weather  direct_report  \n",
       "0     0              0              0  \n",
       "1     0              0              0  \n",
       "2     0              0              0  \n",
       "3     0              0              0  \n",
       "4     0              0              0  \n",
       "\n",
       "[5 rows x 37 columns]"
      ]
     },
     "execution_count": 286,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Merge the categories dataset with categories_split\n",
    "categories_clean = pd.merge(categories,categories_split,left_index=True,right_index=True)\n",
    "categories_clean.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Merge datasets.\n",
    "- Merge the messages and clean categories datasets using the common id\n",
    "- Assign this combined dataset to `df`, which will be cleaned in the following steps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 287,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>message</th>\n",
       "      <th>original</th>\n",
       "      <th>genre</th>\n",
       "      <th>related</th>\n",
       "      <th>request</th>\n",
       "      <th>offer</th>\n",
       "      <th>aid_related</th>\n",
       "      <th>medical_help</th>\n",
       "      <th>medical_products</th>\n",
       "      <th>...</th>\n",
       "      <th>aid_centers</th>\n",
       "      <th>other_infrastructure</th>\n",
       "      <th>weather_related</th>\n",
       "      <th>floods</th>\n",
       "      <th>storm</th>\n",
       "      <th>fire</th>\n",
       "      <th>earthquake</th>\n",
       "      <th>cold</th>\n",
       "      <th>other_weather</th>\n",
       "      <th>direct_report</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>Weather update - a cold front from Cuba that could pass over Haiti</td>\n",
       "      <td>Un front froid se retrouve sur Cuba ce matin. Il pourrait traverser Haiti demain. Des averses de...</td>\n",
       "      <td>direct</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>7</td>\n",
       "      <td>Is the Hurricane over or is it not over</td>\n",
       "      <td>Cyclone nan fini osinon li pa fini</td>\n",
       "      <td>direct</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8</td>\n",
       "      <td>Looking for someone but no name</td>\n",
       "      <td>Patnm, di Maryani relem pou li banm nouvel li ak timoun yo. Mesi se john jean depi Monben kwochi.</td>\n",
       "      <td>direct</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>9</td>\n",
       "      <td>UN reports Leogane 80-90 destroyed. Only Hospital St. Croix functioning. Needs supplies desperat...</td>\n",
       "      <td>UN reports Leogane 80-90 destroyed. Only Hospital St. Croix functioning. Needs supplies desperat...</td>\n",
       "      <td>direct</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>12</td>\n",
       "      <td>says: west side of Haiti, rest of the country today and tonight</td>\n",
       "      <td>facade ouest d Haiti et le reste du pays aujourd hui et ce soir</td>\n",
       "      <td>direct</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 40 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   id  \\\n",
       "0   2   \n",
       "1   7   \n",
       "2   8   \n",
       "3   9   \n",
       "4  12   \n",
       "\n",
       "                                                                                               message  \\\n",
       "0                                   Weather update - a cold front from Cuba that could pass over Haiti   \n",
       "1                                                              Is the Hurricane over or is it not over   \n",
       "2                                                                      Looking for someone but no name   \n",
       "3  UN reports Leogane 80-90 destroyed. Only Hospital St. Croix functioning. Needs supplies desperat...   \n",
       "4                                      says: west side of Haiti, rest of the country today and tonight   \n",
       "\n",
       "                                                                                              original  \\\n",
       "0  Un front froid se retrouve sur Cuba ce matin. Il pourrait traverser Haiti demain. Des averses de...   \n",
       "1                                                                   Cyclone nan fini osinon li pa fini   \n",
       "2    Patnm, di Maryani relem pou li banm nouvel li ak timoun yo. Mesi se john jean depi Monben kwochi.   \n",
       "3  UN reports Leogane 80-90 destroyed. Only Hospital St. Croix functioning. Needs supplies desperat...   \n",
       "4                                      facade ouest d Haiti et le reste du pays aujourd hui et ce soir   \n",
       "\n",
       "    genre  related  request  offer  aid_related  medical_help  \\\n",
       "0  direct        1        0      0            0             0   \n",
       "1  direct        1        0      0            1             0   \n",
       "2  direct        1        0      0            0             0   \n",
       "3  direct        1        1      0            1             0   \n",
       "4  direct        1        0      0            0             0   \n",
       "\n",
       "   medical_products      ...        aid_centers  other_infrastructure  \\\n",
       "0                 0      ...                  0                     0   \n",
       "1                 0      ...                  0                     0   \n",
       "2                 0      ...                  0                     0   \n",
       "3                 1      ...                  0                     0   \n",
       "4                 0      ...                  0                     0   \n",
       "\n",
       "   weather_related  floods  storm  fire  earthquake  cold  other_weather  \\\n",
       "0                0       0      0     0           0     0              0   \n",
       "1                1       0      1     0           0     0              0   \n",
       "2                0       0      0     0           0     0              0   \n",
       "3                0       0      0     0           0     0              0   \n",
       "4                0       0      0     0           0     0              0   \n",
       "\n",
       "   direct_report  \n",
       "0              0  \n",
       "1              0  \n",
       "2              0  \n",
       "3              0  \n",
       "4              0  \n",
       "\n",
       "[5 rows x 40 columns]"
      ]
     },
     "execution_count": 287,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# merge datasets\n",
    "df = pd.merge(messages,categories_clean,on = 'id', copy=False)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 288,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(26386, 40)"
      ]
     },
     "execution_count": 288,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 289,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "24779    9\n",
       "7747     4\n",
       "14246    4\n",
       "25512    4\n",
       "17553    4\n",
       "13914    4\n",
       "29119    4\n",
       "14135    4\n",
       "14592    4\n",
       "17919    4\n",
       "18313    4\n",
       "6573     4\n",
       "18925    4\n",
       "14073    4\n",
       "12589    4\n",
       "6327     4\n",
       "27768    4\n",
       "23002    4\n",
       "6492     4\n",
       "15938    4\n",
       "2446     4\n",
       "4956     4\n",
       "1652     4\n",
       "7892     4\n",
       "11503    4\n",
       "15576    4\n",
       "3250     4\n",
       "5776     4\n",
       "6515     4\n",
       "24247    4\n",
       "        ..\n",
       "27384    1\n",
       "25337    1\n",
       "29435    1\n",
       "19196    1\n",
       "8913     1\n",
       "10960    1\n",
       "21183    1\n",
       "23230    1\n",
       "14994    1\n",
       "12947    1\n",
       "2708     1\n",
       "661      1\n",
       "6806     1\n",
       "4759     1\n",
       "27288    1\n",
       "25241    1\n",
       "19100    1\n",
       "17053    1\n",
       "23198    1\n",
       "21151    1\n",
       "15026    1\n",
       "12979    1\n",
       "2740     1\n",
       "6838     1\n",
       "4791     1\n",
       "27320    1\n",
       "25273    1\n",
       "29371    1\n",
       "17085    1\n",
       "2049     1\n",
       "Name: id, Length: 26180, dtype: int64"
      ]
     },
     "execution_count": 289,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['id'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 290,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "26180"
      ]
     },
     "execution_count": 290,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(set(list(df['id'])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 291,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "id                            0\n",
       "message                       0\n",
       "original                  16140\n",
       "genre                         0\n",
       "related                       0\n",
       "request                       0\n",
       "offer                         0\n",
       "aid_related                   0\n",
       "medical_help                  0\n",
       "medical_products              0\n",
       "search_and_rescue             0\n",
       "security                      0\n",
       "military                      0\n",
       "child_alone                   0\n",
       "water                         0\n",
       "food                          0\n",
       "shelter                       0\n",
       "clothing                      0\n",
       "money                         0\n",
       "missing_people                0\n",
       "refugees                      0\n",
       "death                         0\n",
       "other_aid                     0\n",
       "infrastructure_related        0\n",
       "transport                     0\n",
       "buildings                     0\n",
       "electricity                   0\n",
       "tools                         0\n",
       "hospitals                     0\n",
       "shops                         0\n",
       "aid_centers                   0\n",
       "other_infrastructure          0\n",
       "weather_related               0\n",
       "floods                        0\n",
       "storm                         0\n",
       "fire                          0\n",
       "earthquake                    0\n",
       "cold                          0\n",
       "other_weather                 0\n",
       "direct_report                 0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 291,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Checking NA values after merging\n",
    "df.isna().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 292,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_count_unique = df['id'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 293,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "273"
      ]
     },
     "execution_count": 293,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# There are 273 duplicate IDs\n",
    "df_count_unique[df_count_unique>=2].sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 294,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>message</th>\n",
       "      <th>original</th>\n",
       "      <th>genre</th>\n",
       "      <th>related</th>\n",
       "      <th>request</th>\n",
       "      <th>offer</th>\n",
       "      <th>aid_related</th>\n",
       "      <th>medical_help</th>\n",
       "      <th>medical_products</th>\n",
       "      <th>search_and_rescue</th>\n",
       "      <th>...</th>\n",
       "      <th>aid_centers</th>\n",
       "      <th>other_infrastructure</th>\n",
       "      <th>weather_related</th>\n",
       "      <th>floods</th>\n",
       "      <th>storm</th>\n",
       "      <th>fire</th>\n",
       "      <th>earthquake</th>\n",
       "      <th>cold</th>\n",
       "      <th>other_weather</th>\n",
       "      <th>direct_report</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>24779</th>\n",
       "      <td>Shelter materials (thick polyesters) are being distributed to 18,000 households.</td>\n",
       "      <td>NaN</td>\n",
       "      <td>news</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24779</th>\n",
       "      <td>Shelter materials (thick polyesters) are being distributed to 18,000 households.</td>\n",
       "      <td>NaN</td>\n",
       "      <td>news</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24779</th>\n",
       "      <td>Shelter materials (thick polyesters) are being distributed to 18,000 households.</td>\n",
       "      <td>NaN</td>\n",
       "      <td>news</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24779</th>\n",
       "      <td>Shelter materials (thick polyesters) are being distributed to 18,000 households.</td>\n",
       "      <td>NaN</td>\n",
       "      <td>news</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24779</th>\n",
       "      <td>Shelter materials (thick polyesters) are being distributed to 18,000 households.</td>\n",
       "      <td>NaN</td>\n",
       "      <td>news</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24779</th>\n",
       "      <td>Shelter materials (thick polyesters) are being distributed to 18,000 households.</td>\n",
       "      <td>NaN</td>\n",
       "      <td>news</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24779</th>\n",
       "      <td>Shelter materials (thick polyesters) are being distributed to 18,000 households.</td>\n",
       "      <td>NaN</td>\n",
       "      <td>news</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24779</th>\n",
       "      <td>Shelter materials (thick polyesters) are being distributed to 18,000 households.</td>\n",
       "      <td>NaN</td>\n",
       "      <td>news</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24779</th>\n",
       "      <td>Shelter materials (thick polyesters) are being distributed to 18,000 households.</td>\n",
       "      <td>NaN</td>\n",
       "      <td>news</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7747</th>\n",
       "      <td>where we can paticipate in the law reorganizations?</td>\n",
       "      <td>ki kote nou ka patisipe nan dwa reyamenajman yo?</td>\n",
       "      <td>direct</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7747</th>\n",
       "      <td>where we can paticipate in the law reorganizations?</td>\n",
       "      <td>ki kote nou ka patisipe nan dwa reyamenajman yo?</td>\n",
       "      <td>direct</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7747</th>\n",
       "      <td>where we can paticipate in the law reorganizations?</td>\n",
       "      <td>ki kote nou ka patisipe nan dwa reyamenajman yo?</td>\n",
       "      <td>direct</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7747</th>\n",
       "      <td>where we can paticipate in the law reorganizations?</td>\n",
       "      <td>ki kote nou ka patisipe nan dwa reyamenajman yo?</td>\n",
       "      <td>direct</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14246</th>\n",
       "      <td>Mera ghar selab ki waja sy gir giya hy. aur dobara nahin bna sakta main shadeed mooshkil men hoo...</td>\n",
       "      <td>a.          a.                 a.       a.   a.</td>\n",
       "      <td>direct</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14246</th>\n",
       "      <td>Mera ghar selab ki waja sy gir giya hy. aur dobara nahin bna sakta main shadeed mooshkil men hoo...</td>\n",
       "      <td>a.          a.                 a.       a.   a.</td>\n",
       "      <td>direct</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14246</th>\n",
       "      <td>Mera ghar selab ki waja sy gir giya hy. aur dobara nahin bna sakta main shadeed mooshkil men hoo...</td>\n",
       "      <td>a.          a.                 a.       a.   a.</td>\n",
       "      <td>direct</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14246</th>\n",
       "      <td>Mera ghar selab ki waja sy gir giya hy. aur dobara nahin bna sakta main shadeed mooshkil men hoo...</td>\n",
       "      <td>a.          a.                 a.       a.   a.</td>\n",
       "      <td>direct</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25512</th>\n",
       "      <td>The flood situation has worsened with nearly 3,000 villages submerged forcing an estimated 150,0...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>news</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25512</th>\n",
       "      <td>The flood situation has worsened with nearly 3,000 villages submerged forcing an estimated 150,0...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>news</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25512</th>\n",
       "      <td>The flood situation has worsened with nearly 3,000 villages submerged forcing an estimated 150,0...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>news</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25512</th>\n",
       "      <td>The flood situation has worsened with nearly 3,000 villages submerged forcing an estimated 150,0...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>news</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17553</th>\n",
       "      <td>The last locust infestation in the Central Asian republic was in 1995 and is expected to reoccur...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>news</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17553</th>\n",
       "      <td>The last locust infestation in the Central Asian republic was in 1995 and is expected to reoccur...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>news</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17553</th>\n",
       "      <td>The last locust infestation in the Central Asian republic was in 1995 and is expected to reoccur...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>news</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17553</th>\n",
       "      <td>The last locust infestation in the Central Asian republic was in 1995 and is expected to reoccur...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>news</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13914</th>\n",
       "      <td>Falta tan tan poco.. ¬¨¬¥Simply Red - Holding back the years¬¨¬™. Arena Santiago, de all‚àö¬∞ so...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>social</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13914</th>\n",
       "      <td>Falta tan tan poco.. ¬¨¬¥Simply Red - Holding back the years¬¨¬™. Arena Santiago, de all‚àö¬∞ so...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>social</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13914</th>\n",
       "      <td>Falta tan tan poco.. ¬¨¬¥Simply Red - Holding back the years¬¨¬™. Arena Santiago, de all‚àö¬∞ so...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>social</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13914</th>\n",
       "      <td>Falta tan tan poco.. ¬¨¬¥Simply Red - Holding back the years¬¨¬™. Arena Santiago, de all‚àö¬∞ so...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>social</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29119</th>\n",
       "      <td>Most victims (90 per cent) show little or no symptoms of infection and therefore go a long time ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>news</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9131</th>\n",
       "      <td>NOTES: this sms is not important because this person is playing.</td>\n",
       "      <td>Wap kale kw pou di se moun serye ou konn pran se pa mw menm pou di sa .paske kenpt grapyay nan l...</td>\n",
       "      <td>direct</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9131</th>\n",
       "      <td>NOTES: this sms is not important because this person is playing.</td>\n",
       "      <td>Wap kale kw pou di se moun serye ou konn pran se pa mw menm pou di sa .paske kenpt grapyay nan l...</td>\n",
       "      <td>direct</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22059</th>\n",
       "      <td>Additionally, ointment to treat skin infections, cough medicine, iodine and bandages are needed ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>news</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22059</th>\n",
       "      <td>Additionally, ointment to treat skin infections, cough medicine, iodine and bandages are needed ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>news</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22059</th>\n",
       "      <td>Additionally, ointment to treat skin infections, cough medicine, iodine and bandages are needed ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>news</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22059</th>\n",
       "      <td>Additionally, ointment to treat skin infections, cough medicine, iodine and bandages are needed ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>news</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19003</th>\n",
       "      <td>Before we take your general questions, I'd like to make a few opening remarks concerning ISAF su...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>news</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19003</th>\n",
       "      <td>Before we take your general questions, I'd like to make a few opening remarks concerning ISAF su...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>news</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19003</th>\n",
       "      <td>Before we take your general questions, I'd like to make a few opening remarks concerning ISAF su...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>news</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19003</th>\n",
       "      <td>Before we take your general questions, I'd like to make a few opening remarks concerning ISAF su...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>news</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5643</th>\n",
       "      <td>Good evening. we are writing to ask you for a little help in the Sibert area. We are calling for...</td>\n",
       "      <td>Bonswa nou ekri'w se pou nou mande'w yon ti d pou zon sibert a sak rele bagay manje a nou poko g...</td>\n",
       "      <td>direct</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5643</th>\n",
       "      <td>Good evening. we are writing to ask you for a little help in the Sibert area. We are calling for...</td>\n",
       "      <td>Bonswa nou ekri'w se pou nou mande'w yon ti d pou zon sibert a sak rele bagay manje a nou poko g...</td>\n",
       "      <td>direct</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5643</th>\n",
       "      <td>Good evening. we are writing to ask you for a little help in the Sibert area. We are calling for...</td>\n",
       "      <td>Bonswa nou ekri'w se pou nou mande'w yon ti d pou zon sibert a sak rele bagay manje a nou poko g...</td>\n",
       "      <td>direct</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5643</th>\n",
       "      <td>Good evening. we are writing to ask you for a little help in the Sibert area. We are calling for...</td>\n",
       "      <td>Bonswa nou ekri'w se pou nou mande'w yon ti d pou zon sibert a sak rele bagay manje a nou poko g...</td>\n",
       "      <td>direct</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5573</th>\n",
       "      <td>If I would like to find a job in cepoz, what should I do?</td>\n",
       "      <td>Si mwen ta renmen jwen travay nan cepoz ki sa poum f.</td>\n",
       "      <td>direct</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5573</th>\n",
       "      <td>If I would like to find a job in cepoz, what should I do?</td>\n",
       "      <td>Si mwen ta renmen jwen travay nan cepoz ki sa poum f.</td>\n",
       "      <td>direct</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5573</th>\n",
       "      <td>If I would like to find a job in cepoz, what should I do?</td>\n",
       "      <td>Si mwen ta renmen jwen travay nan cepoz ki sa poum f.</td>\n",
       "      <td>direct</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5573</th>\n",
       "      <td>If I would like to find a job in cepoz, what should I do?</td>\n",
       "      <td>Si mwen ta renmen jwen travay nan cepoz ki sa poum f.</td>\n",
       "      <td>direct</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3217</th>\n",
       "      <td>Sir or madam, I'm hungry, I cannot stand because I'm very hungry, if you could share the food, i...</td>\n",
       "      <td>Mesye ou madam mwen grangou mwen pa menm kakanpe sitlman mwen grangou si nou tap vin separe manj...</td>\n",
       "      <td>direct</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3217</th>\n",
       "      <td>Sir or madam, I'm hungry, I cannot stand because I'm very hungry, if you could share the food, i...</td>\n",
       "      <td>Mesye ou madam mwen grangou mwen pa menm kakanpe sitlman mwen grangou si nou tap vin separe manj...</td>\n",
       "      <td>direct</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3217</th>\n",
       "      <td>Sir or madam, I'm hungry, I cannot stand because I'm very hungry, if you could share the food, i...</td>\n",
       "      <td>Mesye ou madam mwen grangou mwen pa menm kakanpe sitlman mwen grangou si nou tap vin separe manj...</td>\n",
       "      <td>direct</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3217</th>\n",
       "      <td>Sir or madam, I'm hungry, I cannot stand because I'm very hungry, if you could share the food, i...</td>\n",
       "      <td>Mesye ou madam mwen grangou mwen pa menm kakanpe sitlman mwen grangou si nou tap vin separe manj...</td>\n",
       "      <td>direct</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6687</th>\n",
       "      <td>Tell the President to give the nation over to God.</td>\n",
       "      <td>Di prezidan remt peyi a bay Bondye.</td>\n",
       "      <td>direct</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6687</th>\n",
       "      <td>Tell the President to give the nation over to God.</td>\n",
       "      <td>Di prezidan remt peyi a bay Bondye.</td>\n",
       "      <td>direct</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6687</th>\n",
       "      <td>Tell the President to give the nation over to God.</td>\n",
       "      <td>Di prezidan remt peyi a bay Bondye.</td>\n",
       "      <td>direct</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6687</th>\n",
       "      <td>Tell the President to give the nation over to God.</td>\n",
       "      <td>Di prezidan remt peyi a bay Bondye.</td>\n",
       "      <td>direct</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>202</th>\n",
       "      <td>?? port au prince ?? and food. they need goverment aid and international aid thak you. god bless...</td>\n",
       "      <td>p bay pap la syen ak manje. Yo bezwen ed gouvenman an ak d entenasyonal. Mesi, BonDye beni Ayiti.</td>\n",
       "      <td>direct</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>202</th>\n",
       "      <td>?? port au prince ?? and food. they need goverment aid and international aid thak you. god bless...</td>\n",
       "      <td>p bay pap la syen ak manje. Yo bezwen ed gouvenman an ak d entenasyonal. Mesi, BonDye beni Ayiti.</td>\n",
       "      <td>direct</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>202</th>\n",
       "      <td>?? port au prince ?? and food. they need goverment aid and international aid thak you. god bless...</td>\n",
       "      <td>p bay pap la syen ak manje. Yo bezwen ed gouvenman an ak d entenasyonal. Mesi, BonDye beni Ayiti.</td>\n",
       "      <td>direct</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>202</th>\n",
       "      <td>?? port au prince ?? and food. they need goverment aid and international aid thak you. god bless...</td>\n",
       "      <td>p bay pap la syen ak manje. Yo bezwen ed gouvenman an ak d entenasyonal. Mesi, BonDye beni Ayiti.</td>\n",
       "      <td>direct</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>273 rows × 39 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                   message  \\\n",
       "id                                                                                                           \n",
       "24779                     Shelter materials (thick polyesters) are being distributed to 18,000 households.   \n",
       "24779                     Shelter materials (thick polyesters) are being distributed to 18,000 households.   \n",
       "24779                     Shelter materials (thick polyesters) are being distributed to 18,000 households.   \n",
       "24779                     Shelter materials (thick polyesters) are being distributed to 18,000 households.   \n",
       "24779                     Shelter materials (thick polyesters) are being distributed to 18,000 households.   \n",
       "24779                     Shelter materials (thick polyesters) are being distributed to 18,000 households.   \n",
       "24779                     Shelter materials (thick polyesters) are being distributed to 18,000 households.   \n",
       "24779                     Shelter materials (thick polyesters) are being distributed to 18,000 households.   \n",
       "24779                     Shelter materials (thick polyesters) are being distributed to 18,000 households.   \n",
       "7747                                                  where we can paticipate in the law reorganizations?    \n",
       "7747                                                  where we can paticipate in the law reorganizations?    \n",
       "7747                                                  where we can paticipate in the law reorganizations?    \n",
       "7747                                                  where we can paticipate in the law reorganizations?    \n",
       "14246  Mera ghar selab ki waja sy gir giya hy. aur dobara nahin bna sakta main shadeed mooshkil men hoo...   \n",
       "14246  Mera ghar selab ki waja sy gir giya hy. aur dobara nahin bna sakta main shadeed mooshkil men hoo...   \n",
       "14246  Mera ghar selab ki waja sy gir giya hy. aur dobara nahin bna sakta main shadeed mooshkil men hoo...   \n",
       "14246  Mera ghar selab ki waja sy gir giya hy. aur dobara nahin bna sakta main shadeed mooshkil men hoo...   \n",
       "25512  The flood situation has worsened with nearly 3,000 villages submerged forcing an estimated 150,0...   \n",
       "25512  The flood situation has worsened with nearly 3,000 villages submerged forcing an estimated 150,0...   \n",
       "25512  The flood situation has worsened with nearly 3,000 villages submerged forcing an estimated 150,0...   \n",
       "25512  The flood situation has worsened with nearly 3,000 villages submerged forcing an estimated 150,0...   \n",
       "17553  The last locust infestation in the Central Asian republic was in 1995 and is expected to reoccur...   \n",
       "17553  The last locust infestation in the Central Asian republic was in 1995 and is expected to reoccur...   \n",
       "17553  The last locust infestation in the Central Asian republic was in 1995 and is expected to reoccur...   \n",
       "17553  The last locust infestation in the Central Asian republic was in 1995 and is expected to reoccur...   \n",
       "13914  Falta tan tan poco.. ¬¨¬¥Simply Red - Holding back the years¬¨¬™. Arena Santiago, de all‚àö¬∞ so...   \n",
       "13914  Falta tan tan poco.. ¬¨¬¥Simply Red - Holding back the years¬¨¬™. Arena Santiago, de all‚àö¬∞ so...   \n",
       "13914  Falta tan tan poco.. ¬¨¬¥Simply Red - Holding back the years¬¨¬™. Arena Santiago, de all‚àö¬∞ so...   \n",
       "13914  Falta tan tan poco.. ¬¨¬¥Simply Red - Holding back the years¬¨¬™. Arena Santiago, de all‚àö¬∞ so...   \n",
       "29119  Most victims (90 per cent) show little or no symptoms of infection and therefore go a long time ...   \n",
       "...                                                                                                    ...   \n",
       "9131                                      NOTES: this sms is not important because this person is playing.   \n",
       "9131                                      NOTES: this sms is not important because this person is playing.   \n",
       "22059  Additionally, ointment to treat skin infections, cough medicine, iodine and bandages are needed ...   \n",
       "22059  Additionally, ointment to treat skin infections, cough medicine, iodine and bandages are needed ...   \n",
       "22059  Additionally, ointment to treat skin infections, cough medicine, iodine and bandages are needed ...   \n",
       "22059  Additionally, ointment to treat skin infections, cough medicine, iodine and bandages are needed ...   \n",
       "19003  Before we take your general questions, I'd like to make a few opening remarks concerning ISAF su...   \n",
       "19003  Before we take your general questions, I'd like to make a few opening remarks concerning ISAF su...   \n",
       "19003  Before we take your general questions, I'd like to make a few opening remarks concerning ISAF su...   \n",
       "19003  Before we take your general questions, I'd like to make a few opening remarks concerning ISAF su...   \n",
       "5643   Good evening. we are writing to ask you for a little help in the Sibert area. We are calling for...   \n",
       "5643   Good evening. we are writing to ask you for a little help in the Sibert area. We are calling for...   \n",
       "5643   Good evening. we are writing to ask you for a little help in the Sibert area. We are calling for...   \n",
       "5643   Good evening. we are writing to ask you for a little help in the Sibert area. We are calling for...   \n",
       "5573                                            If I would like to find a job in cepoz, what should I do?    \n",
       "5573                                            If I would like to find a job in cepoz, what should I do?    \n",
       "5573                                            If I would like to find a job in cepoz, what should I do?    \n",
       "5573                                            If I would like to find a job in cepoz, what should I do?    \n",
       "3217   Sir or madam, I'm hungry, I cannot stand because I'm very hungry, if you could share the food, i...   \n",
       "3217   Sir or madam, I'm hungry, I cannot stand because I'm very hungry, if you could share the food, i...   \n",
       "3217   Sir or madam, I'm hungry, I cannot stand because I'm very hungry, if you could share the food, i...   \n",
       "3217   Sir or madam, I'm hungry, I cannot stand because I'm very hungry, if you could share the food, i...   \n",
       "6687                                                   Tell the President to give the nation over to God.    \n",
       "6687                                                   Tell the President to give the nation over to God.    \n",
       "6687                                                   Tell the President to give the nation over to God.    \n",
       "6687                                                   Tell the President to give the nation over to God.    \n",
       "202    ?? port au prince ?? and food. they need goverment aid and international aid thak you. god bless...   \n",
       "202    ?? port au prince ?? and food. they need goverment aid and international aid thak you. god bless...   \n",
       "202    ?? port au prince ?? and food. they need goverment aid and international aid thak you. god bless...   \n",
       "202    ?? port au prince ?? and food. they need goverment aid and international aid thak you. god bless...   \n",
       "\n",
       "                                                                                                  original  \\\n",
       "id                                                                                                           \n",
       "24779                                                                                                  NaN   \n",
       "24779                                                                                                  NaN   \n",
       "24779                                                                                                  NaN   \n",
       "24779                                                                                                  NaN   \n",
       "24779                                                                                                  NaN   \n",
       "24779                                                                                                  NaN   \n",
       "24779                                                                                                  NaN   \n",
       "24779                                                                                                  NaN   \n",
       "24779                                                                                                  NaN   \n",
       "7747                                                      ki kote nou ka patisipe nan dwa reyamenajman yo?   \n",
       "7747                                                      ki kote nou ka patisipe nan dwa reyamenajman yo?   \n",
       "7747                                                      ki kote nou ka patisipe nan dwa reyamenajman yo?   \n",
       "7747                                                      ki kote nou ka patisipe nan dwa reyamenajman yo?   \n",
       "14246                                                     a.          a.                 a.       a.   a.    \n",
       "14246                                                     a.          a.                 a.       a.   a.    \n",
       "14246                                                     a.          a.                 a.       a.   a.    \n",
       "14246                                                     a.          a.                 a.       a.   a.    \n",
       "25512                                                                                                  NaN   \n",
       "25512                                                                                                  NaN   \n",
       "25512                                                                                                  NaN   \n",
       "25512                                                                                                  NaN   \n",
       "17553                                                                                                  NaN   \n",
       "17553                                                                                                  NaN   \n",
       "17553                                                                                                  NaN   \n",
       "17553                                                                                                  NaN   \n",
       "13914                                                                                                  NaN   \n",
       "13914                                                                                                  NaN   \n",
       "13914                                                                                                  NaN   \n",
       "13914                                                                                                  NaN   \n",
       "29119                                                                                                  NaN   \n",
       "...                                                                                                    ...   \n",
       "9131   Wap kale kw pou di se moun serye ou konn pran se pa mw menm pou di sa .paske kenpt grapyay nan l...   \n",
       "9131   Wap kale kw pou di se moun serye ou konn pran se pa mw menm pou di sa .paske kenpt grapyay nan l...   \n",
       "22059                                                                                                  NaN   \n",
       "22059                                                                                                  NaN   \n",
       "22059                                                                                                  NaN   \n",
       "22059                                                                                                  NaN   \n",
       "19003                                                                                                  NaN   \n",
       "19003                                                                                                  NaN   \n",
       "19003                                                                                                  NaN   \n",
       "19003                                                                                                  NaN   \n",
       "5643   Bonswa nou ekri'w se pou nou mande'w yon ti d pou zon sibert a sak rele bagay manje a nou poko g...   \n",
       "5643   Bonswa nou ekri'w se pou nou mande'w yon ti d pou zon sibert a sak rele bagay manje a nou poko g...   \n",
       "5643   Bonswa nou ekri'w se pou nou mande'w yon ti d pou zon sibert a sak rele bagay manje a nou poko g...   \n",
       "5643   Bonswa nou ekri'w se pou nou mande'w yon ti d pou zon sibert a sak rele bagay manje a nou poko g...   \n",
       "5573                                                 Si mwen ta renmen jwen travay nan cepoz ki sa poum f.   \n",
       "5573                                                 Si mwen ta renmen jwen travay nan cepoz ki sa poum f.   \n",
       "5573                                                 Si mwen ta renmen jwen travay nan cepoz ki sa poum f.   \n",
       "5573                                                 Si mwen ta renmen jwen travay nan cepoz ki sa poum f.   \n",
       "3217   Mesye ou madam mwen grangou mwen pa menm kakanpe sitlman mwen grangou si nou tap vin separe manj...   \n",
       "3217   Mesye ou madam mwen grangou mwen pa menm kakanpe sitlman mwen grangou si nou tap vin separe manj...   \n",
       "3217   Mesye ou madam mwen grangou mwen pa menm kakanpe sitlman mwen grangou si nou tap vin separe manj...   \n",
       "3217   Mesye ou madam mwen grangou mwen pa menm kakanpe sitlman mwen grangou si nou tap vin separe manj...   \n",
       "6687                                                                   Di prezidan remt peyi a bay Bondye.   \n",
       "6687                                                                   Di prezidan remt peyi a bay Bondye.   \n",
       "6687                                                                   Di prezidan remt peyi a bay Bondye.   \n",
       "6687                                                                   Di prezidan remt peyi a bay Bondye.   \n",
       "202      p bay pap la syen ak manje. Yo bezwen ed gouvenman an ak d entenasyonal. Mesi, BonDye beni Ayiti.   \n",
       "202      p bay pap la syen ak manje. Yo bezwen ed gouvenman an ak d entenasyonal. Mesi, BonDye beni Ayiti.   \n",
       "202      p bay pap la syen ak manje. Yo bezwen ed gouvenman an ak d entenasyonal. Mesi, BonDye beni Ayiti.   \n",
       "202      p bay pap la syen ak manje. Yo bezwen ed gouvenman an ak d entenasyonal. Mesi, BonDye beni Ayiti.   \n",
       "\n",
       "        genre  related  request  offer  aid_related  medical_help  \\\n",
       "id                                                                  \n",
       "24779    news        1        0      0            1             0   \n",
       "24779    news        1        0      0            1             0   \n",
       "24779    news        1        0      0            1             0   \n",
       "24779    news        1        0      0            1             0   \n",
       "24779    news        1        0      0            1             0   \n",
       "24779    news        1        0      0            1             0   \n",
       "24779    news        1        0      0            1             0   \n",
       "24779    news        1        0      0            1             0   \n",
       "24779    news        1        0      0            1             0   \n",
       "7747   direct        1        0      0            0             0   \n",
       "7747   direct        0        0      0            0             0   \n",
       "7747   direct        1        0      0            0             0   \n",
       "7747   direct        0        0      0            0             0   \n",
       "14246  direct        2        0      0            0             0   \n",
       "14246  direct        2        0      0            0             0   \n",
       "14246  direct        2        0      0            0             0   \n",
       "14246  direct        2        0      0            0             0   \n",
       "25512    news        1        0      0            1             0   \n",
       "25512    news        1        0      0            1             0   \n",
       "25512    news        1        0      0            1             0   \n",
       "25512    news        1        0      0            1             0   \n",
       "17553    news        1        1      0            1             1   \n",
       "17553    news        1        0      0            0             0   \n",
       "17553    news        1        1      0            1             1   \n",
       "17553    news        1        0      0            0             0   \n",
       "13914  social        2        0      0            0             0   \n",
       "13914  social        1        0      0            0             0   \n",
       "13914  social        2        0      0            0             0   \n",
       "13914  social        1        0      0            0             0   \n",
       "29119    news        1        0      0            1             1   \n",
       "...       ...      ...      ...    ...          ...           ...   \n",
       "9131   direct        0        0      0            0             0   \n",
       "9131   direct        0        0      0            0             0   \n",
       "22059    news        1        1      0            1             1   \n",
       "22059    news        1        1      0            1             1   \n",
       "22059    news        1        1      0            1             1   \n",
       "22059    news        1        1      0            1             1   \n",
       "19003    news        1        1      0            1             0   \n",
       "19003    news        1        0      0            1             0   \n",
       "19003    news        1        1      0            1             0   \n",
       "19003    news        1        0      0            1             0   \n",
       "5643   direct        1        1      0            1             0   \n",
       "5643   direct        1        1      0            1             0   \n",
       "5643   direct        1        1      0            1             0   \n",
       "5643   direct        1        1      0            1             0   \n",
       "5573   direct        0        0      0            0             0   \n",
       "5573   direct        0        0      0            0             0   \n",
       "5573   direct        0        0      0            0             0   \n",
       "5573   direct        0        0      0            0             0   \n",
       "3217   direct        1        1      0            1             0   \n",
       "3217   direct        1        1      0            1             0   \n",
       "3217   direct        1        1      0            1             0   \n",
       "3217   direct        1        1      0            1             0   \n",
       "6687   direct        1        0      0            0             0   \n",
       "6687   direct        0        0      0            0             0   \n",
       "6687   direct        1        0      0            0             0   \n",
       "6687   direct        0        0      0            0             0   \n",
       "202    direct        1        1      0            1             0   \n",
       "202    direct        1        1      0            1             0   \n",
       "202    direct        1        1      0            1             0   \n",
       "202    direct        1        1      0            1             0   \n",
       "\n",
       "       medical_products  search_and_rescue      ...        aid_centers  \\\n",
       "id                                              ...                      \n",
       "24779                 0                  0      ...                  0   \n",
       "24779                 0                  0      ...                  0   \n",
       "24779                 0                  0      ...                  0   \n",
       "24779                 0                  0      ...                  0   \n",
       "24779                 0                  0      ...                  0   \n",
       "24779                 0                  0      ...                  0   \n",
       "24779                 0                  0      ...                  0   \n",
       "24779                 0                  0      ...                  0   \n",
       "24779                 0                  0      ...                  0   \n",
       "7747                  0                  0      ...                  0   \n",
       "7747                  0                  0      ...                  0   \n",
       "7747                  0                  0      ...                  0   \n",
       "7747                  0                  0      ...                  0   \n",
       "14246                 0                  0      ...                  0   \n",
       "14246                 0                  0      ...                  0   \n",
       "14246                 0                  0      ...                  0   \n",
       "14246                 0                  0      ...                  0   \n",
       "25512                 0                  0      ...                  0   \n",
       "25512                 0                  0      ...                  0   \n",
       "25512                 0                  0      ...                  0   \n",
       "25512                 0                  0      ...                  0   \n",
       "17553                 0                  0      ...                  0   \n",
       "17553                 0                  0      ...                  0   \n",
       "17553                 0                  0      ...                  0   \n",
       "17553                 0                  0      ...                  0   \n",
       "13914                 0                  0      ...                  0   \n",
       "13914                 0                  0      ...                  0   \n",
       "13914                 0                  0      ...                  0   \n",
       "13914                 0                  0      ...                  0   \n",
       "29119                 0                  0      ...                  0   \n",
       "...                 ...                ...      ...                ...   \n",
       "9131                  0                  0      ...                  0   \n",
       "9131                  0                  0      ...                  0   \n",
       "22059                 1                  0      ...                  0   \n",
       "22059                 1                  0      ...                  0   \n",
       "22059                 1                  0      ...                  0   \n",
       "22059                 1                  0      ...                  0   \n",
       "19003                 0                  0      ...                  0   \n",
       "19003                 0                  0      ...                  0   \n",
       "19003                 0                  0      ...                  0   \n",
       "19003                 0                  0      ...                  0   \n",
       "5643                  0                  0      ...                  0   \n",
       "5643                  0                  0      ...                  0   \n",
       "5643                  0                  0      ...                  0   \n",
       "5643                  0                  0      ...                  0   \n",
       "5573                  0                  0      ...                  0   \n",
       "5573                  0                  0      ...                  0   \n",
       "5573                  0                  0      ...                  0   \n",
       "5573                  0                  0      ...                  0   \n",
       "3217                  0                  0      ...                  0   \n",
       "3217                  0                  0      ...                  0   \n",
       "3217                  0                  0      ...                  0   \n",
       "3217                  0                  0      ...                  0   \n",
       "6687                  0                  0      ...                  0   \n",
       "6687                  0                  0      ...                  0   \n",
       "6687                  0                  0      ...                  0   \n",
       "6687                  0                  0      ...                  0   \n",
       "202                   0                  0      ...                  0   \n",
       "202                   0                  0      ...                  0   \n",
       "202                   0                  0      ...                  0   \n",
       "202                   0                  0      ...                  0   \n",
       "\n",
       "       other_infrastructure  weather_related  floods  storm  fire  earthquake  \\\n",
       "id                                                                              \n",
       "24779                     0                0       0      0     0           0   \n",
       "24779                     0                0       0      0     0           0   \n",
       "24779                     0                0       0      0     0           0   \n",
       "24779                     0                0       0      0     0           0   \n",
       "24779                     0                0       0      0     0           0   \n",
       "24779                     0                0       0      0     0           0   \n",
       "24779                     0                0       0      0     0           0   \n",
       "24779                     0                0       0      0     0           0   \n",
       "24779                     0                0       0      0     0           0   \n",
       "7747                      0                0       0      0     0           0   \n",
       "7747                      0                0       0      0     0           0   \n",
       "7747                      0                0       0      0     0           0   \n",
       "7747                      0                0       0      0     0           0   \n",
       "14246                     0                0       0      0     0           0   \n",
       "14246                     0                0       0      0     0           0   \n",
       "14246                     0                0       0      0     0           0   \n",
       "14246                     0                0       0      0     0           0   \n",
       "25512                     1                1       1      0     0           0   \n",
       "25512                     0                1       1      0     0           0   \n",
       "25512                     1                1       1      0     0           0   \n",
       "25512                     0                1       1      0     0           0   \n",
       "17553                     0                1       1      0     0           0   \n",
       "17553                     0                0       0      0     0           0   \n",
       "17553                     0                1       1      0     0           0   \n",
       "17553                     0                0       0      0     0           0   \n",
       "13914                     0                0       0      0     0           0   \n",
       "13914                     0                0       0      0     0           0   \n",
       "13914                     0                0       0      0     0           0   \n",
       "13914                     0                0       0      0     0           0   \n",
       "29119                     0                0       0      0     0           0   \n",
       "...                     ...              ...     ...    ...   ...         ...   \n",
       "9131                      0                0       0      0     0           0   \n",
       "9131                      0                0       0      0     0           0   \n",
       "22059                     0                0       0      0     0           0   \n",
       "22059                     0                0       0      0     0           0   \n",
       "22059                     0                0       0      0     0           0   \n",
       "22059                     0                0       0      0     0           0   \n",
       "19003                     0                1       1      0     0           1   \n",
       "19003                     0                1       1      0     0           1   \n",
       "19003                     0                1       1      0     0           1   \n",
       "19003                     0                1       1      0     0           1   \n",
       "5643                      0                0       0      0     0           0   \n",
       "5643                      0                0       0      0     0           0   \n",
       "5643                      0                0       0      0     0           0   \n",
       "5643                      0                0       0      0     0           0   \n",
       "5573                      0                0       0      0     0           0   \n",
       "5573                      0                0       0      0     0           0   \n",
       "5573                      0                0       0      0     0           0   \n",
       "5573                      0                0       0      0     0           0   \n",
       "3217                      0                0       0      0     0           0   \n",
       "3217                      0                0       0      0     0           0   \n",
       "3217                      0                0       0      0     0           0   \n",
       "3217                      0                0       0      0     0           0   \n",
       "6687                      0                0       0      0     0           0   \n",
       "6687                      0                0       0      0     0           0   \n",
       "6687                      0                0       0      0     0           0   \n",
       "6687                      0                0       0      0     0           0   \n",
       "202                       0                0       0      0     0           0   \n",
       "202                       0                0       0      0     0           0   \n",
       "202                       0                0       0      0     0           0   \n",
       "202                       0                0       0      0     0           0   \n",
       "\n",
       "       cold  other_weather  direct_report  \n",
       "id                                         \n",
       "24779     0              0              0  \n",
       "24779     0              0              0  \n",
       "24779     0              0              0  \n",
       "24779     0              0              0  \n",
       "24779     0              0              0  \n",
       "24779     0              0              0  \n",
       "24779     0              0              0  \n",
       "24779     0              0              0  \n",
       "24779     0              0              0  \n",
       "7747      0              0              0  \n",
       "7747      0              0              0  \n",
       "7747      0              0              0  \n",
       "7747      0              0              0  \n",
       "14246     0              0              0  \n",
       "14246     0              0              0  \n",
       "14246     0              0              0  \n",
       "14246     0              0              0  \n",
       "25512     0              0              0  \n",
       "25512     0              0              0  \n",
       "25512     0              0              0  \n",
       "25512     0              0              0  \n",
       "17553     0              1              1  \n",
       "17553     0              0              0  \n",
       "17553     0              1              1  \n",
       "17553     0              0              0  \n",
       "13914     0              0              0  \n",
       "13914     0              0              0  \n",
       "13914     0              0              0  \n",
       "13914     0              0              0  \n",
       "29119     0              0              0  \n",
       "...     ...            ...            ...  \n",
       "9131      0              0              0  \n",
       "9131      0              0              0  \n",
       "22059     0              0              0  \n",
       "22059     0              0              0  \n",
       "22059     0              0              0  \n",
       "22059     0              0              0  \n",
       "19003     0              0              0  \n",
       "19003     1              0              0  \n",
       "19003     0              0              0  \n",
       "19003     1              0              0  \n",
       "5643      0              0              1  \n",
       "5643      0              0              1  \n",
       "5643      0              0              1  \n",
       "5643      0              0              1  \n",
       "5573      0              0              0  \n",
       "5573      0              0              0  \n",
       "5573      0              0              0  \n",
       "5573      0              0              0  \n",
       "3217      0              0              1  \n",
       "3217      0              0              1  \n",
       "3217      0              0              1  \n",
       "3217      0              0              1  \n",
       "6687      0              0              0  \n",
       "6687      0              0              0  \n",
       "6687      0              0              0  \n",
       "6687      0              0              0  \n",
       "202       0              0              0  \n",
       "202       0              0              0  \n",
       "202       0              0              0  \n",
       "202       0              0              0  \n",
       "\n",
       "[273 rows x 39 columns]"
      ]
     },
     "execution_count": 294,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.set_index('id').loc[df_count_unique[df_count_unique>=2].index]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 295,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "206"
      ]
     },
     "execution_count": 295,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.duplicated(subset='id').sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Remove duplicates.\n",
    "- Check how many duplicates are in this dataset.\n",
    "- Drop the duplicates.\n",
    "- Confirm duplicates were removed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 296,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "206"
      ]
     },
     "execution_count": 296,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check number of duplicates\n",
    "len(df)-len(df.drop_duplicates(subset='id'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 297,
   "metadata": {},
   "outputs": [],
   "source": [
    "# drop duplicates\n",
    "df_clean = df.drop_duplicates(subset='id')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 298,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "id                            0\n",
       "message                       0\n",
       "original                  16027\n",
       "genre                         0\n",
       "related                       0\n",
       "request                       0\n",
       "offer                         0\n",
       "aid_related                   0\n",
       "medical_help                  0\n",
       "medical_products              0\n",
       "search_and_rescue             0\n",
       "security                      0\n",
       "military                      0\n",
       "child_alone                   0\n",
       "water                         0\n",
       "food                          0\n",
       "shelter                       0\n",
       "clothing                      0\n",
       "money                         0\n",
       "missing_people                0\n",
       "refugees                      0\n",
       "death                         0\n",
       "other_aid                     0\n",
       "infrastructure_related        0\n",
       "transport                     0\n",
       "buildings                     0\n",
       "electricity                   0\n",
       "tools                         0\n",
       "hospitals                     0\n",
       "shops                         0\n",
       "aid_centers                   0\n",
       "other_infrastructure          0\n",
       "weather_related               0\n",
       "floods                        0\n",
       "storm                         0\n",
       "fire                          0\n",
       "earthquake                    0\n",
       "cold                          0\n",
       "other_weather                 0\n",
       "direct_report                 0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 298,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Checking any NA values apart from Original\n",
    "df_clean.isna().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 302,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 302,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Confirm any duplicates\n",
    "df_clean.duplicated(subset='id').sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 300,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(26180, 40)"
      ]
     },
     "execution_count": 300,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_clean.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7. Save the clean dataset into an sqlite database.\n",
    "You can do this with pandas [`to_sql` method](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.to_sql.html) combined with the SQLAlchemy library. Remember to import SQLAlchemy's `create_engine` in the first cell of this notebook to use it below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 304,
   "metadata": {},
   "outputs": [],
   "source": [
    "engine = create_engine('sqlite:///DisasterResponse.db')\n",
    "df_clean.to_sql('Response', engine, index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8. Use this notebook to complete `etl_pipeline.py`\n",
    "Use the template file attached in the Resources folder to write a script that runs the steps above to create a database based on new datasets specified by the user. Alternatively, you can complete `etl_pipeline.py` in the classroom on the `Project Workspace IDE` coming later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}