t81_558_class_11_02_word2vec.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_11_02_word2vec.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 11: Natural Language Processing and Speech Recognition**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 11 Material\n",
    "\n",
    "* Part 11.1: Getting Started with Spacy in Python [[Video]](https://www.youtube.com/watch?v=A5BtU9vXzu8&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_11_01_spacy.ipynb)\n",
    "* **Part 11.2: Word2Vec and Text Classification** [[Video]](https://www.youtube.com/watch?v=nWxtRlpObIs&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_11_02_word2vec.ipynb)\n",
    "* Part 11.3: What are Embedding Layers in Keras [[Video]](https://www.youtube.com/watch?v=OuNH5kT-aD0&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_11_03_embedding.ipynb)\n",
    "* Part 11.4: Natural Language Processing with Spacy and Keras [[Video]](https://www.youtube.com/watch?v=BKgwjhao5DU&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_11_04_text_nlp.ipynb)\n",
    "* Part 11.5: Learning English from Scratch with Keras and TensorFlow [[Video]](https://www.youtube.com/watch?v=Y1khuuSjZzc&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN&index=58) [[Notebook]](t81_558_class_11_05_english_scratch.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Google CoLab Instructions\n",
    "\n",
    "The following code ensures that Google CoLab is running the correct version of TensorFlow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: not using Google CoLab\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    %tensorflow_version 2.x\n",
    "    COLAB = True\n",
    "    print(\"Note: using Google CoLab\")\n",
    "except:\n",
    "    print(\"Note: not using Google CoLab\")\n",
    "    COLAB = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 11.2: Word2Vec and Text Classification\n",
    "\n",
    "Word2vec is a group of related models that are used to produce word embeddings. These models are shallow, two-layer neural networks that are trained to reconstruct linguistic contexts of words. Word2vec takes as its input a large corpus of text and produces a vector space, typically of several hundred dimensions, with each unique word in the corpus being assigned a corresponding vector in the space. Word vectors are positioned in the vector space such that words that share common contexts in the corpus are located in close proximity to one another in the space.\n",
    "\n",
    "Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). [Efficient estimation of word representations in vector space](https://arxiv.org/abs/1301.3781). arXiv preprint arXiv:1301.3781.\n",
    "\n",
    "![Word2Vec](https://pbs.twimg.com/media/C7jJxIjWkAA8E_s.jpg)\n",
    "[Trust Word2Vec](https://twitter.com/DanilBaibak/status/844647217885581312)\n",
    "\n",
    "### Suggested Software for Word2Vec\n",
    "\n",
    "* [GoogleNews Vectors](https://code.google.com/archive/p/word2vec/), [GitHub Mirror](https://github.com/mmihaltz/word2vec-GoogleNews-vectors)\n",
    "* [Python Gensim](https://radimrehurek.com/gensim/)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "C:\\Users\\jheaton\\.keras\\datasets\\GoogleNews-vectors-negative300.bin.gz\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras.utils import get_file\n",
    "\n",
    "try:\n",
    "    path = get_file('GoogleNews-vectors-negative300.bin.gz', origin='https://s3.amazonaws.com/dl4j-distribution/GoogleNews-vectors-negative300.bin.gz')\n",
    "except:\n",
    "    print('Error downloading')\n",
    "    raise\n",
    "    \n",
    "print(path)    \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gensim\n",
    "\n",
    "# Not that the path below refers to a location on my hard drive.\n",
    "# You should download GoogleNews Vectors (see suggested software above)\n",
    "model = gensim.models.KeyedVectors.load_word2vec_format(path, binary=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Word2vec makes each word a vector.  We are using the 300-number vector, which can be seen for the word \"hello\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "w = model['hello']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "300\n"
     ]
    }
   ],
   "source": [
    "print(len(w))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[-0.05419922  0.01708984 -0.00527954  0.33203125 -0.25       -0.01397705\n",
      " -0.15039062 -0.265625    0.01647949  0.3828125  -0.03295898 -0.09716797\n",
      " -0.16308594 -0.04443359  0.00946045  0.18457031  0.03637695  0.16601562\n",
      "  0.36328125 -0.25585938  0.375       0.171875    0.21386719 -0.19921875\n",
      "  0.13085938 -0.07275391 -0.02819824  0.11621094  0.15332031  0.09082031\n",
      "  0.06787109 -0.0300293  -0.16894531 -0.20800781 -0.03710938 -0.22753906\n",
      "  0.26367188  0.012146    0.18359375  0.31054688 -0.10791016 -0.19140625\n",
      "  0.21582031  0.13183594 -0.03515625  0.18554688 -0.30859375  0.04785156\n",
      " -0.10986328  0.14355469 -0.43554688 -0.0378418   0.10839844  0.140625\n",
      " -0.10595703  0.26171875 -0.17089844  0.39453125  0.12597656 -0.27734375\n",
      " -0.28125     0.14746094 -0.20996094  0.02355957  0.18457031  0.00445557\n",
      " -0.27929688 -0.03637695 -0.29296875  0.19628906  0.20703125  0.2890625\n",
      " -0.20507812  0.06787109 -0.43164062 -0.10986328 -0.2578125  -0.02331543\n",
      "  0.11328125  0.23144531 -0.04418945  0.10839844 -0.2890625  -0.09521484\n",
      " -0.10351562 -0.0324707   0.07763672 -0.13378906  0.22949219  0.06298828\n",
      "  0.08349609  0.02929688 -0.11474609  0.00534058 -0.12988281  0.02514648\n",
      "  0.08789062  0.24511719 -0.11474609 -0.296875   -0.59375    -0.29492188\n",
      " -0.13378906  0.27734375 -0.04174805  0.11621094  0.28320312  0.00241089\n",
      "  0.13867188 -0.00683594 -0.30078125  0.16210938  0.01171875 -0.13867188\n",
      "  0.48828125  0.02880859  0.02416992  0.04736328  0.05859375 -0.23828125\n",
      "  0.02758789  0.05981445 -0.03857422  0.06933594  0.14941406 -0.10888672\n",
      " -0.07324219  0.08789062  0.27148438  0.06591797 -0.37890625 -0.26171875\n",
      " -0.13183594  0.09570312 -0.3125      0.10205078  0.03063965  0.23632812\n",
      "  0.00582886  0.27734375  0.20507812 -0.17871094 -0.31445312 -0.01586914\n",
      "  0.13964844  0.13574219  0.0390625  -0.29296875  0.234375   -0.33984375\n",
      " -0.11816406  0.10644531 -0.18457031 -0.02099609  0.02563477  0.25390625\n",
      "  0.07275391  0.13574219 -0.00138092 -0.2578125  -0.2890625   0.10107422\n",
      "  0.19238281 -0.04882812  0.27929688 -0.3359375  -0.07373047  0.01879883\n",
      " -0.10986328 -0.04614258  0.15722656  0.06689453 -0.03417969  0.16308594\n",
      "  0.08642578  0.44726562  0.02026367 -0.01977539  0.07958984  0.17773438\n",
      " -0.04370117 -0.00952148  0.16503906  0.17285156  0.23144531 -0.04272461\n",
      "  0.02355957  0.18359375 -0.41601562 -0.01745605  0.16796875  0.04736328\n",
      "  0.14257812  0.08496094  0.33984375  0.1484375  -0.34375    -0.14160156\n",
      " -0.06835938 -0.14648438 -0.02844238  0.07421875 -0.07666016  0.12695312\n",
      "  0.05859375 -0.07568359 -0.03344727  0.23632812 -0.16308594  0.16503906\n",
      "  0.1484375  -0.2421875  -0.3515625  -0.30664062  0.00491333  0.17675781\n",
      "  0.46289062  0.14257812 -0.25       -0.25976562  0.04370117  0.34960938\n",
      "  0.05957031  0.07617188 -0.02868652 -0.09667969 -0.01281738  0.05859375\n",
      " -0.22949219 -0.1953125  -0.12207031  0.20117188 -0.42382812  0.06005859\n",
      "  0.50390625  0.20898438  0.11230469 -0.06054688  0.33203125  0.07421875\n",
      " -0.05786133  0.11083984 -0.06494141  0.05639648  0.01757812  0.08398438\n",
      "  0.13769531  0.2578125   0.16796875 -0.16894531  0.01794434  0.16015625\n",
      "  0.26171875  0.31640625 -0.24804688  0.05371094 -0.0859375   0.17089844\n",
      " -0.39453125 -0.00156403 -0.07324219 -0.04614258 -0.16210938 -0.15722656\n",
      "  0.21289062 -0.15820312  0.04394531  0.28515625  0.01196289 -0.26953125\n",
      " -0.04370117  0.37109375  0.04663086 -0.19726562  0.3046875  -0.36523438\n",
      " -0.23632812  0.08056641 -0.04248047 -0.14648438 -0.06225586 -0.0534668\n",
      " -0.05664062  0.18945312  0.37109375 -0.22070312  0.04638672  0.02612305\n",
      " -0.11474609  0.265625   -0.02453613  0.11083984 -0.02514648 -0.12060547\n",
      "  0.05297852  0.07128906  0.00063705 -0.36523438 -0.13769531 -0.12890625]\n"
     ]
    }
   ],
   "source": [
    "print(w)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code below shows the distance between two words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.4796925\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "w1 = model['king']\n",
    "w2 = model['queen']\n",
    "\n",
    "dist = np.linalg.norm(w1-w2)\n",
    "\n",
    "print(dist)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This shows the classic word2vec equation of **queen = (king - man) + female**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('queen', 0.7118192911148071),\n",
       " ('monarch', 0.6189674139022827),\n",
       " ('princess', 0.5902431607246399),\n",
       " ('crown_prince', 0.5499460697174072),\n",
       " ('prince', 0.5377321243286133),\n",
       " ('kings', 0.5236844420433044),\n",
       " ('Queen_Consort', 0.5235945582389832),\n",
       " ('queens', 0.5181134343147278),\n",
       " ('sultan', 0.5098593235015869),\n",
       " ('monarchy', 0.5087411999702454)]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.most_similar(positive=['woman', 'king'], negative=['man'])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code shows which item does not belong with the others."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\jheaton\\Miniconda3\\envs\\tensorflow\\lib\\site-packages\\gensim\\models\\keyedvectors.py:877: FutureWarning: arrays to stack must be passed as a \"sequence\" type such as list or tuple. Support for non-sequence iterables such as generators is deprecated as of NumPy 1.16 and will raise an error in the future.\n",
      "  vectors = vstack(self.word_vec(word, use_norm=True) for word in used_words).astype(REAL)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'dog'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.doesnt_match(\"house garage store dog\".split())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code shows the similarity between two words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5633577"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.similarity('iphone', 'android')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code shows which words are most similar to the given one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('dogs', 0.8680490255355835),\n",
       " ('puppy', 0.8106428384780884),\n",
       " ('pit_bull', 0.780396044254303),\n",
       " ('pooch', 0.7627376317977905),\n",
       " ('cat', 0.7609457969665527),\n",
       " ('golden_retriever', 0.7500901818275452),\n",
       " ('German_shepherd', 0.7465174198150635),\n",
       " ('Rottweiler', 0.7437615394592285),\n",
       " ('beagle', 0.7418621778488159),\n",
       " ('pup', 0.7406911253929138)]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.most_similar('dog')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.7 (tensorflow)",
   "language": "python",
   "name": "tensorflow"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}