Arabic_keyword_extracting_from_social_networks (1).ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "#Extracting Arabic keywords from social media posts"
      ],
      "metadata": {
        "id": "oKZvksAVwfZQ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "This notebook provides two examples of how to extract Arabic keywords from social media posts using the Python KeyBERT package. KeyBERT is a simple and easy-to-use keyword extraction technique that uses BERT embeddings to generate keywords and keyphrases that are most similar to a document. However, extracting keywords from social media posts is not as straightforward as extracting from documents, and existing tools do not work as well with Arabic as they do with English.\n",
        "\n"
      ],
      "metadata": {
        "id": "u6aOZSD5zLvT"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4wI4JR8xwZR6"
      },
      "outputs": [],
      "source": [
        "# install  packages\n",
        "\n",
        "!pip install keybert\n",
        "!pip install keyphrase-vectorizers\n",
        "!pip install pyarabic\n",
        "!pip install Arabic-stopwords\n",
        "!pip install camel-tools\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Data processing"
      ],
      "metadata": {
        "id": "wfjwRAHF0Qg-"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import re\n",
        "from pyarabic.araby import tokenize, is_arabicrange, strip_tashkeel\n",
        "import arabicstopwords.arabicstopwords as stpw\n",
        "\n",
        "\n",
        "def text_processing(text):\n",
        "  # remove non arabic data\n",
        "  #text=tokenize(text, conditions=is_arabicrange, morphs=strip_tashkeel)\n",
        "  text=tokenize(text,  morphs=strip_tashkeel)\n",
        "\n",
        "  #remove arabic stopwords\n",
        "  stp_words_list = open(\"./arabicstopword.txt\", encoding=\"utf-8\")\n",
        "  stp_words = stp_words_list.read()\n",
        "  stp_words = stp_words.split(\"\\n\")\n",
        "  from camel_tools.utils.normalize import(\n",
        "    normalize_alef_ar,\n",
        "    normalize_alef_maksura_ar,\n",
        "    normalize_teh_marbuta_ar,\n",
        "    )\n",
        "  added = []\n",
        "\n",
        "  for word in stp_words:\n",
        "    nar = normalize_alef_ar(word)\n",
        "    nam = normalize_alef_maksura_ar(nar)\n",
        "    res = normalize_teh_marbuta_ar(word)\n",
        "    added.append(res)\n",
        "\n",
        "  stp_words.append(added)\n",
        "  tokens = [w for w in text if not stpw.is_stop(w)]\n",
        "  tokens = [word for word in text if not word in stp_words]\n",
        "  return tokens\n"
      ],
      "metadata": {
        "id": "2ya1NiEXxDuq"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "I'm going to implement the extraction function with KeyBERT using two different classes:\n",
        "\n",
        "1.   CountVectorizer\n",
        "2.   KeyphraseCountVectorizer\n"
      ],
      "metadata": {
        "id": "EvdV-KMjxk1D"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "##1- using CounterVectorize"
      ],
      "metadata": {
        "id": "9-iJaeyDyx5E"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "The default model in KeyBERT (\"all-MiniLM-L6-v2\") works great for English documents. In contrast, for multi-lingual documents or any other language, \"paraphrase-multilingual-MiniLM-L12-v2\"\" has shown better performance.\n",
        "However, using (paraphrase-mpnet-base-v2) and paraphrase-multilingual-mpnet-base-v2 perofrm well but with higher complixity.\n",
        "\n"
      ],
      "metadata": {
        "id": "f7-IACt1yIPM"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#==================================================\n",
        "#1- using CounterVectorize\n",
        "#==================================================\n",
        "\n",
        "def extract_non_verb_keywords(sentence):\n",
        "  from transformers import pipeline\n",
        "  pos = pipeline('token-classification', model='CAMeL-Lab/bert-base-arabic-camelbert-mix-pos-msa')\n",
        "  pos=pos(sentence)\n",
        "  non_verb_kw=[]\n",
        "  for i in range (len(pos)):\n",
        "    kw=[]\n",
        "    for element in pos[i]:\n",
        "      if element.get('entity') not in ['verb', 'prep']:\n",
        "       w = element.get('word')\n",
        "       kw.append(w)\n",
        "       listToStr = ' '.join([str(elem) for elem in kw])\n",
        "       listToStr=listToStr.replace(\" ##\",\"\")\n",
        "    non_verb_kw.append(listToStr)\n",
        "  return non_verb_kw"
      ],
      "metadata": {
        "id": "VbAJ_IiWzDbr"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def arabic_keyword_extract_CV(text):\n",
        "  from keybert import KeyBERT\n",
        "  from sklearn.feature_extraction.text import CountVectorizer\n",
        "  #processing data\n",
        "  tokens=text_processing(text5)\n",
        "  text=\" \".join(tokens)\n",
        "  # with CountVectorizer, we should to define manually the keyword length using ngram_range\n",
        "  vectorizer= CountVectorizer(ngram_range=(1, 3))\n",
        "\n",
        "  kw_extractor = KeyBERT('paraphrase-multilingual-mpnet-base-v2')\n",
        "  keywords = kw_extractor.extract_keywords(text , vectorizer=vectorizer, stop_words=\"arabic\", use_mmr=True, diversity=0.7, top_n=5)\n",
        "  keywords=[item[0] for item in keywords]\n",
        "\n",
        "  #remove verbs\n",
        "  non_verb_keywords= extract_non_verb_keywords(keywords)\n",
        "  import re\n",
        "  # Define a regular expression pattern for Arabic letters\n",
        "  keywods= [keword for keword in non_verb_keywords if re.search('[\\u0600-\\u06FF\\u0750-\\u077F]+', keword)]\n",
        "  output = dict()\n",
        "  output[\"original_text\"] = text\n",
        "  output[\"keywords\"] = [','.join(keywods)]\n",
        "  import pandas as pd\n",
        "  df = pd.DataFrame(output)\n",
        "  df.to_csv(\"arabic_keywords_CV.csv\")\n",
        "  return output\n",
        "\n",
        "#testing\n",
        "text=\"لتكن عبرة للاخرين اصبحنا نرى انتشار واسع للتافهين بمواقع التواصل الاجتماعي واصبحو حديث الكل رغم انهم لم يقدمو اي قيمة مضافة للمجتمع لاكن للاسف فان النجوم الحقيقين الذين يستحقون ان يتصدرو الطوندونس نكاد لانسمع عنهم شيء رغم انهم يحتلونة مكانة رفيعة في اكبر المؤسسات العلمية الاروبية والعالمية .,ar\"\n",
        "text2=\" أخي الفلاح، أختي الفلاحة، القرض الفلاحي للمغرب كينخارط بجنبكم في المبادرة الوطنية للتأمين الاجباري عن المرض لضمان صحتكم وصحة عائلتكم. وهذا هو الوقت المناسب باش يكون عندكم حساب شخصي مستقل على حسابكم المهني\"\n",
        "text3=\"من 15 غشت إلى 18 سبتمبر 2022، ادفعوا لأول مرة بدون تماس باستخدام بطاقة Visa الخاصة بكم في المتاجر Maârif Culture , GO Sport Maroc , City Sport Maroc , PUMA Maroc Courir Maroc : وDaha Fazla استفيدوا من تعويض ٪20 😍 !للعلم أن جميع عمليات الدفع بدون تماس في هذه العلامات التجارية بواسطة بطاقة Visa تسمح لكم بالمشاركة تلقائيًا في السحب الكبير 🎉 لمحاولة الفوز بهدايا رائعة 🎁 بما في ذلك سيارة !المزيد من المعلومات عبر الرابط التالي : https://bit.ly/3pf9ARC \"\n",
        "text4=\"بمناسبة الذكرى 43 لاسترجاع وادي الذهب، يتقدم القرض الفلاحي للمغرب بأحر التهاني لكافة الشعب المغربي\"\n",
        "text5=\"بمناسبة اليوم الوطني للجالية المغربية بالخارج، القرض الفلاحي للمغرب يرحب بكم. ولمرافقتكم اليومية ، القرض الفلاحي للمغرب صمّم من اجلكم Air Pack MDM #RESPIRE عرض بنكي شامل يلائم متطلباتكم و يفتح أمامكم أبواب عالم من الامتيازات الجديدة . \"\n",
        "outputs = arabic_keyword_extract_CV (text)\n",
        "print (outputs)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "CXr-3jHszIj-",
        "outputId": "5cc07382-cc0b-46fd-c601-6617a7a1147c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Some weights of the model checkpoint at CAMeL-Lab/bert-base-arabic-camelbert-mix-pos-msa were not used when initializing BertForTokenClassification: ['bert.pooler.dense.bias', 'bert.pooler.dense.weight']\n",
            "- This IS expected if you are initializing BertForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing BertForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Asking to truncate to max_length but no maximum length is provided and the model has no predefined maximum length. Default to no truncation.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'original_text': 'بمناسبة الوطني للجالية المغربية بالخارج القرض الفلاحي للمغرب يرحب بكم . ولمرافقتكم اليومية القرض الفلاحي للمغرب صمم اجلكم Air Pack MDM # RESPIRE عرض بنكي شامل يلائم متطلباتكم يفتح أمامكم أبواب عالم الامتيازات الجديدة .', 'keywords': ['القرض الفلاحي للمغرب,القرض الفلاحي للمغرب,الوطني']}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        " ## 2- *KeyphraseCountVectorizer*"
      ],
      "metadata": {
        "id": "8NzXIUFey7S4"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "\n",
        "#==================================================\n",
        "# 2- using KeyphraseCountVectorizer\n",
        "#==================================================\n",
        "\n",
        "from keybert import KeyBERT\n",
        "from keyphrase_vectorizers import KeyphraseCountVectorizer\n",
        "\n",
        "def arabic_keyword_extract_KCV(text):\n",
        "\n",
        "  import re\n",
        "  text = re.sub('([@A-Za-z0-9_ـــــــــــــ]+)|[^\\w\\s]|#|http\\S+', '', text)\n",
        "  #pos_pattern regexp to ensure extracting keywords which start with nouns\n",
        "  vectorizer = KeyphraseCountVectorizer()\n",
        "  #try:\n",
        "  keywords = []\n",
        "  kw_extractor = KeyBERT(\"paraphrase-multilingual-mpnet-base-v2\")\n",
        "  keywords = kw_extractor.extract_keywords(text , keyphrase_ngram_range=(2,3), vectorizer=vectorizer, stop_words=\"arabic\", top_n=3, use_mmr=True, diversity=0.7)\n",
        "  keywords = [ x[0] for x in keywords]\n",
        "  output = dict()\n",
        "  output[\"original_text\"] = text\n",
        "  output[\"keywords\"] = [','.join(keywords)]\n",
        "  import pandas as pd\n",
        "  df = pd.DataFrame(output)\n",
        "  df.to_csv(\"arabic_keywords_KCV.csv\")\n",
        "  return output"
      ],
      "metadata": {
        "id": "OD0BtwdIxju2"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "\n",
        "#testing with 5 posts:\n",
        "\n",
        "outputs= arabic_keyword_extract_KCV(text)\n",
        "print(outputs)\n",
        "\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nfdwMqr1ytti",
        "outputId": "d8e615a1-450e-4c50-d4f3-094eb15eb2fa"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'original_text': 'لتكن عبرة للاخرين اصبحنا نرى انتشار واسع للتافهين بمواقع التواصل الاجتماعي واصبحو حديث الكل رغم انهم لم يقدمو اي قيمة مضافة للمجتمع لاكن للاسف فان النجوم الحقيقين الذين يستحقون ان يتصدرو الطوندونس نكاد لانسمع عنهم شيء رغم انهم يحتلونة مكانة رفيعة في اكبر المؤسسات العلمية الاروبية والعالمية ', 'keywords': ['التواصل الاجتماعي واصبحو حديث الكل رغم انهم,اصبحنا نرى انتشار واسع للتافهين,لتكن عبرة']}\n"
          ]
        }
      ]
    }
  ]
}