nlp-in-python-using-spacy-textacy.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/msivanes/miniconda3/envs/anlp/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 192 from C header, got 216 from PyObject\n",
      "  return f(*args, **kwds)\n"
     ]
    }
   ],
   "source": [
    "#conda install -c conda-forge spacy=2.1.3\n",
    "import spacy,textacy\n",
    "from spacy import displacy\n",
    "import re"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(spacy.__version__)\n",
    "print(textacy.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!python -m spacy download en_core_web_lg\n",
    "nlp = spacy.load('en_core_web_lg')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# with open('openshift-installation.json') as file:\n",
    "#     data=json.load(file)\n",
    "    \n",
    "# print(data)\n",
    "\n",
    "text = '''\n",
    "The kubelet exposes metrics that can be collected and stored in back-ends by Heapster.\n",
    "\n",
    "As an OpenShift Container Platform administrator, you can view a cluster’s metrics from all containers and components in one user interface. These metrics are also used by horizontal pod autoscalers in order to determine when and how to scale.\n",
    "\n",
    "This topic describes using Hawkular Metrics as a metrics engine which stores the data persistently in a Cassandra database. When this is configured, CPU, memory and network-based metrics are viewable from the OpenShift Container Platform web console and are available for use by horizontal pod autoscalers.\n",
    "\n",
    "Heapster retrieves a list of all nodes from the master server, then contacts each node individually through the /stats endpoint. From there, Heapster scrapes the metrics for CPU, memory and network usage, then exports them into Hawkular Metrics.\n",
    "\n",
    "The storage volume metrics available on the kubelet are not available through the /stats endpoint, but are available through the /metrics endpoint. See OpenShift Container Platform via Prometheus for detailed information.\n",
    "\n",
    "Browsing individual pods in the web console displays separate sparkline charts for memory and CPU. The time range displayed is selectable, and these charts automatically update every 30 seconds. If there are multiple containers on the pod, then you can select a specific container to display its metrics.\n",
    "\n",
    "If resource limits are defined for your project, then you can also see a donut chart for each pod. The donut chart displays usage against the resource limit. For example: 145 Available of 200 MiB, with the donut chart showing 55 MiB Used. \n",
    "'''\n",
    "text = re.sub('\\n', '', text) #removing new lines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc=nlp(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc_lower = nlp(text.lower())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Tokenization using spacy\n",
    "#Tokens that are not punctuation alone but preserve punctuation within a token\n",
    "tokens = [token for token in doc_lower if not token.is_punct] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'[the, kubelet, exposes, metrics, that, can, be, collected, and, stored, in, back, ends, by, heapster.as, an, openshift, container, platform, administrator, you, can, view, a, cluster, ’s, metrics, from, all, containers, and, components, in, one, user, interface, these, metrics, are, also, used, by, horizontal, pod, autoscalers, in, order, to, determine, when, and, how, to, scale.this, topic, describes, using, hawkular, metrics, as, a, metrics, engine, which, stores, the, data, persistently, in, a, cassandra, database, when, this, is, configured, cpu, memory, and, network, based, metrics, are, viewable, from, the, openshift, container, platform, web, console, and, are, available, for, use, by, horizontal, pod, autoscalers.heapster, retrieves, a, list, of, all, nodes, from, the, master, server, then, contacts, each, node, individually, through, the, /stats, endpoint, from, there, heapster, scrapes, the, metrics, for, cpu, memory, and, network, usage, then, exports, them, into, hawkular, metrics.the, storage, volume, metrics, available, on, the, kubelet, are, not, available, through, the, /stats, endpoint, but, are, available, through, the, /metrics, endpoint, see, openshift, container, platform, via, prometheus, for, detailed, information.browsing, individual, pods, in, the, web, console, displays, separate, sparkline, charts, for, memory, and, cpu, the, time, range, displayed, is, selectable, and, these, charts, automatically, update, every, 30, seconds, if, there, are, multiple, containers, on, the, pod, then, you, can, select, a, specific, container, to, display, its, metrics.if, resource, limits, are, defined, for, your, project, then, you, can, also, see, a, donut, chart, for, each, pod, the, donut, chart, displays, usage, against, the, resource, limit, for, example, 145, available, of, 200, mib, with, the, donut, chart, showing, 55, mib, used]'"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "str(tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Observations**\n",
    "Look how \"/stats\" and \"/metrics\" are not tokenized separately"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[The kubelet exposes metrics that can be collected and stored in back-ends by Heapster.,\n",
       " As an OpenShift Container Platform administrator, you can view a cluster’s metrics from all containers and components in one user interface.,\n",
       " These metrics are also used by horizontal pod autoscalers in order to determine when and how to scale.,\n",
       " This topic describes using Hawkular Metrics as a metrics engine which stores the data persistently in a Cassandra database.,\n",
       " When this is configured, CPU, memory and network-based metrics are viewable from the OpenShift Container Platform web console and are available for use by horizontal pod autoscalers.]"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Sentence segmentation\n",
    "sentences = [sent for sent in doc.sents]\n",
    "sentences[:5] #First 5 sentences"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stemming vs Lemmatization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Stemming usually refers to a crude heuristic process that chops off the ends of words in the hope of achieving this goal correctly most of the time, and often includes the removal of derivational affixes.\n",
    "\n",
    "Lemmatization usually refers to doing things properly with the use of a vocabulary and morphological analysis of words, normally aiming to remove inflectional endings only and to return the base or dictionary form of a word, which is known as the lemma.\n",
    "\n",
    "If confronted with the token saw, stemming might return just s, whereas lemmatization would attempt to return either see or saw, depending on whether the use of the token was as a verb or a noun. - Christopher Manning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(the, 'the', 'DET'), (kubelet, 'kubelet', 'NOUN'), (exposes, 'expose', 'VERB'), (metrics, 'metric', 'NOUN'), (that, 'that', 'ADJ'), (can, 'can', 'VERB'), (be, 'be', 'VERB'), (collected, 'collect', 'VERB'), (and, 'and', 'CCONJ'), (stored, 'store', 'VERB'), (in, 'in', 'ADP'), (back, 'back', 'NOUN'), (ends, 'end', 'NOUN'), (by, 'by', 'ADP'), (heapster.as, 'heapster.a', 'NOUN'), (an, 'an', 'DET'), (openshift, 'openshift', 'ADJ'), (container, 'container', 'NOUN'), (platform, 'platform', 'NOUN'), (administrator, 'administrator', 'NOUN')]\n"
     ]
    }
   ],
   "source": [
    "#Spacy only supports lemmatization. stemmers use is generally discouraged\n",
    "# lemma(accessed using lemma_ attribute) and pos(POS tagging using pos_ attribute) \n",
    "# is automatically created for us when we process text with nlp object\n",
    "print([(token, token.lemma_, token.pos_) for token in tokens][:20])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Named Entity Recognition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Heapster (PERSON)\n",
      "OpenShift Container Platform (ORG)\n",
      "Hawkular Metrics (ORG)\n",
      "Cassandra (PERSON)\n",
      "the OpenShift Container Platform (ORG)\n",
      "Heapster (PERSON)\n",
      "Heapster (PERSON)\n",
      "Hawkular Metrics (ORG)\n",
      "OpenShift Container Platform (ORG)\n",
      "Prometheus (PRODUCT)\n",
      "every 30 seconds (TIME)\n",
      "145 (CARDINAL)\n",
      "200 (CARDINAL)\n",
      "MiB (PRODUCT)\n",
      "55 (CARDINAL)\n",
      "MiB Used (PRODUCT)\n"
     ]
    }
   ],
   "source": [
    "# Prints the named entities detected\n",
    "for entity in doc.ents:\n",
    "    print(f\"{entity.text} ({entity.label_})\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We found all the entities Heapster, Hawkular Metrics, Cassandra, Prometheus (though the label itself does not look right).  Definitely labels with PERSON, ORG, PRODUCT can be used for further manual analysis.**[TODO]**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualizing Entities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div class=\"entities\" style=\"line-height: 2.5\">The kubelet exposes metrics that can be collected and stored in back-ends by \n",
       "<mark class=\"entity\" style=\"background: #aa9cfc; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
       "    Heapster\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">PERSON</span>\n",
       "</mark>\n",
       ". This topic describes using \n",
       "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
       "    Hawkular Metrics\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
       "</mark>\n",
       " as a metrics engine which stores the data persistently in a \n",
       "<mark class=\"entity\" style=\"background: #aa9cfc; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
       "    Cassandra\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">PERSON</span>\n",
       "</mark>\n",
       " database. When this is configured, CPU, memory and network-based metrics are viewable from \n",
       "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
       "    the OpenShift Container Platform\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
       "</mark>\n",
       " web console and are available for use by horizontal pod autoscalers.</div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "content = '''The kubelet exposes metrics that can be collected and stored in back-ends by Heapster. This topic describes using Hawkular Metrics as a metrics engine which stores the data persistently in a Cassandra database. When this is configured, CPU, memory and network-based metrics are viewable from the OpenShift Container Platform web console and are available for use by horizontal pod autoscalers.'''\n",
    "doc=nlp(content)\n",
    "displacy.render(doc, style='ent', jupyter=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part-of-speech tagging"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes we want to create an abstract for the text. We can quickly extract the relevant nouns from the text and show them. Noun chunks are noun phrases – not single words, but a short phrase which describes the noun."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sentence1 The kubelet\n",
      "sentence1 metrics\n",
      "sentence1 back-ends\n",
      "sentence1 Heapster\n",
      "sentence2 an OpenShift Container Platform administrator\n",
      "sentence2 you\n",
      "sentence2 a cluster’s metrics\n",
      "sentence2 all containers\n",
      "sentence2 components\n",
      "sentence2 one user interface\n",
      "sentence3 These metrics\n",
      "sentence3 horizontal pod autoscalers\n",
      "sentence3 order\n",
      "sentence4 This topic\n",
      "sentence4 Hawkular Metrics\n",
      "sentence4 a metrics engine\n",
      "sentence4 the data\n",
      "sentence4 a Cassandra database\n",
      "sentence5 CPU\n",
      "sentence5 memory\n",
      "sentence5 network-based metrics\n",
      "sentence5 the OpenShift Container Platform web console\n",
      "sentence5 use\n",
      "sentence5 horizontal pod autoscalers\n"
     ]
    }
   ],
   "source": [
    "for idx, sentence in enumerate(sentences[:5]):\n",
    "    for noun in sentence.noun_chunks: \n",
    "        print(f'sentence{idx+1}', noun)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the DET DT\n",
      "kubelet NOUN NN\n",
      "exposes VERB VBZ\n",
      "metrics NOUN NNS\n",
      "that ADJ WDT\n",
      "can VERB MD\n",
      "be VERB VB\n",
      "collected VERB VBN\n",
      "and CCONJ CC\n",
      "stored VERB VBN\n",
      "in ADP IN\n",
      "back NOUN NN\n",
      "ends NOUN NNS\n",
      "by ADP IN\n",
      "heapster.as NOUN NNS\n",
      "an DET DT\n",
      "openshift ADJ JJ\n",
      "container NOUN NN\n",
      "platform NOUN NN\n",
      "administrator NOUN NN\n"
     ]
    }
   ],
   "source": [
    "for token in tokens[:20]:\n",
    "    print(token, token.pos_, token.tag_) # text , POS tagging & its label"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generating Q&A from the above paragraph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "from textacy.spacier import utils as spacy_utils"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_sentence = \" This topic describes using Hawkular Metrics as a metrics engine which stores the data persistently in a Cassandra database.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[describes, using, stores]\n"
     ]
    }
   ],
   "source": [
    "#Get the main verbs in the above sentence\n",
    "doc = nlp(new_sentence)\n",
    "verbs = spacy_utils.get_main_verbs_of_sent(doc)\n",
    "print(verbs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "describes [topic]\n",
      "using []\n",
      "stores [which]\n"
     ]
    }
   ],
   "source": [
    "#Get the nominal subjects of the verbs\n",
    "for verb in verbs:\n",
    "    print(verb, spacy_utils.get_subjects_of_verb(verb))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "describes [using]\n",
      "using [Metrics]\n",
      "stores [data]\n"
     ]
    }
   ],
   "source": [
    "# Get objects from the sentence\n",
    "for verb in verbs:\n",
    "    print(verb, spacy_utils.get_objects_of_verb(verb))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "def paragraph_to_question(text):\n",
    "    doc = nlp(text)\n",
    "    results = []\n",
    "    for sentence in doc.sents:\n",
    "        root = sentence.root\n",
    "        ask_about = spacy_utils.get_subjects_of_verb(root)\n",
    "        answers = spacy_utils.get_objects_of_verb(root)\n",
    "        \n",
    "        if len(ask_about) > 0 and len(answers) > 0:\n",
    "            if root.lemma_ == 'be':\n",
    "                question = f'What {root} {ask_about[0]}'\n",
    "            else:\n",
    "                question = f'What does {ask_about[0]} {root.lemma_}?'\n",
    "            results.append({'question': question, 'answers': answers})\n",
    "        return results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'question': 'What is Bansoori', 'answers': [instrument]}]"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "paragraph_to_question('Bansoori is an Indian classical instrument.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'question': 'What does kubelet expose?', 'answers': [metrics]}]"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "paragraph_to_question('The kubelet exposes metrics that can be collected and stored in back-ends by Heapster.')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}