tutorial_notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook introduces how to use the topicmodels module for implementing Latent Dirichlet Allocation using the collapsed Gibbs sampling algorithm of Griffiths and Steyvers (2004).  The module contains three classes: one for processing raw text, another for implementing LDA, and another for querying.  This tutorial will go through the main features of each, for full details see the documented source code.\n",
    "\n",
    "To illustrate LDA, the tutorial uses text data from State of the Union Addresses at the paragraph level.  These are available for download from http://www.presidency.ucsb.edu/sou.php.  They are contained in the tab-separated text file speech_data_extend.txt distributed with this tutorial.\n",
    "\n",
    "To interact with this data, we begin by importing some libraries that are not strictly speaking necessary for using topicmodels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To begin, we read in the data, specifying the encoding of the text data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data = pd.read_table(\"speech_data_extend.txt\",encoding=\"utf-8\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data object is called a pandas DataFrame, and is similar to a Data Frame in R.  One can see the data has three fields."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index([u'president', u'speech', u'year'], dtype='object')"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the tutorial we focus on State of the Union addresses made since the television era, which began in 1947."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9488"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = data[data.year >= 1947]\n",
    "len(data) # The number of documents (paragraphs of State of the Union Addresses) in the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cleaning Raw Text Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now import topicmodels, the module used in most of the analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import topicmodels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before implementing a topic model, it is important to pre-process the data.  The first class in the topicmodels module is called RawDocs and facilitates this pre-processing.  We are going to pass this class the text data contained in the DataFrame along with a list of stopwords specified by \"long\".  Stopwords are common words in English that tend to appear in all text, and so are not helpful in describing content.  There is no definitive list of stopwords, and another option we describe below is to let the data itself reveal which words are useful for discriminating among documents.  The list of words comes from http://snowball.tartarus.org/algorithms/english/stop.txt, but one need not use all of them.  In Hansen, McMahon, and Prat (2014), for example, we use just a subset of these, which you can use by specifying \"short\" instead of \"long\".  (You can view the stopwords by typing docsobj.stopwords into the interpreter)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "docsobj = topicmodels.RawDocs(data.speech, \"long\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Rather than passing the text as a DataFrame column, one can also pass a text file, in which case each new line will be read as a separate document.\n",
    "\n",
    "docsobj is now an object with several attributes.  The most important is its tokens attribute.  This is the outcome of taking each raw document, converting each contraction into its underlying words (e.g. \"don't\" into \"do not\"), coverting it into lowercase, and breaking it into its underlying linguistic elements (words, numbers, punctuation, etc.).  To illustrate, compare the fourth paragraph in the 1947 State of the Union Address as a raw document and after tokenization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I come also to welcome you as you take up your duties and to discuss with you the manner in which you and I should fulfill our obligations to the American people during the next 2 years. \n",
      "[u'i', u'come', u'also', u'to', u'welcome', u'you', u'as', u'you', u'take', u'up', u'your', u'duties', u'and', u'to', u'discuss', u'with', u'you', u'the', u'manner', u'in', u'which', u'you', u'and', u'i', u'should', u'fulfill', u'our', u'obligations', u'to', u'the', u'american', u'people', u'during', u'the', u'next', u'2', u'years', u'.']\n"
     ]
    }
   ],
   "source": [
    "print data.speech.values[3] # fourth paragraph (note that Python uses 0-indexing)\n",
    "print docsobj.tokens[3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point, all tokens are in the dataset, which can be useful for some purposes.  For example, one might want to count the number of question marks in each speech.  For implementing LDA, though, one generally wishes to focus on words.  docsobj has a method to clean tokens.  This will remove all non-alphanumeric tokens, and, by default, all numeric tokens as well (to keep numeric tokens, pass False as the second argument in parentheses).  The number passed as an argument removes all tokens whose length is less that - this can be useful if some symbols in the data like copyright signs trascribed as a single 'c' that the user would like to remove, in which case one would pass 1.  In this case, we pass 1 for illustration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "docsobj.token_clean(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[u'come', u'also', u'to', u'welcome', u'you', u'as', u'you', u'take', u'up', u'your', u'duties', u'and', u'to', u'discuss', u'with', u'you', u'the', u'manner', u'in', u'which', u'you', u'and', u'should', u'fulfill', u'our', u'obligations', u'to', u'the', u'american', u'people', u'during', u'the', u'next', u'years']\n"
     ]
    }
   ],
   "source": [
    "print docsobj.tokens[3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we remove the stopwords we passed when creating docsobj.  Here we pass \"tokens\" as an argument to specify that the stopwords should be removed from docsobj.tokens.  The other option would be to pass \"stems\", which we discuss below.  Notice by how much we have reduced the size of the document by removing stopwords!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[u'come', u'welcome', u'duties', u'discuss', u'manner', u'fulfill', u'obligations', u'american', u'people', u'next', u'years']\n"
     ]
    }
   ],
   "source": [
    "docsobj.stopword_remove(\"tokens\")\n",
    "print docsobj.tokens[3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next step is to attempt to group together words that are grammatically different but themeatically identical.  For example, the document above has the token \"obligations\" but another may have \"obligation\" and yet another \"oblige.\"  Ultimately these three words denote the same concept, and so we might want them to share the same symbol.  One way to achieve this is through stemming, a process whereby words are transformed through a deterministic algorithm to a base form.  One popular stemmer is the Porter stemmer, which docsobj applies (via its implementation in Python's Natural Language Toolkit).  This creates a new stems attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[u'come', u'welcom', u'duti', u'discuss', u'manner', u'fulfil', u'oblig', u'american', u'peopl', u'next', u'year']\n"
     ]
    }
   ],
   "source": [
    "docsobj.stem()\n",
    "print docsobj.stems[3]\n",
    "docsobj.stopword_remove(\"stems\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that the outcome of stemming need not be an English word.  These stems are the data on which we will run the topic model below.  We make an additional call to remove stopwords from stems, since the stemmed forms of tokens not in the stopword list may themselves be in the stopword list.\n",
    "\n",
    "The final step in pre-processing is to drop remaining words that are not useful for identifying content.  We have already dropped standard stopwords, but there may also be data-dependent common words.  For example, in data from Supreme Court proceedings, \"justice\" might be treated as a stopword.  Also, words that appear just once or twice in the collection are not informative of content either.  Ideally, one would like a measure of informativeness that both punishes common words in the data, and rare words.  One such option is to give each stem a tf-idf (term frequency - inverse document frequency) score.  This is standard in the language processing literature, so we omit details here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "docsobj.term_rank(\"stems\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This call produces two .csv files in the working directory, df_ranking.csv and tfidf_ranking.csv.  df_ranking.csv ranks each stem according to its document frequency, or the number of documents it appears in.  tfidf_ranking.csv ranks each stem accroding to the tf-idf measure, according to which highly informative words are those that appear frequently in the entire dataset, but in relatively few documents.  Stems with the highest scores include \"gun\", \"iraq\", and \"immigr\".\n",
    "\n",
    "At this stage, the user can decide how many stems to drop based on either the df or tf-idf scores.  The first argument to rank_remove specifies the ranking method to use (\"df\" or \"tfidf\"), the second whether to drop from \"tokens\" or \"stems\" (since we formed the rankings based on stems above, we should specify stems), and finally the value to use for the cutoff for dropping stems.  One might instead prefer to provide a number $n$ such that all stems with a tf-idf value less than or equal to the $n$th ranked stem are then dropped, which we illustrate below.  One can determine the cutoff from exploring the csv files, but here we plot the ranking in Python, which indicates a reasonable cutoff is 5000.  (When using df rather than tfidf, substitute docsobj.df_ranking)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x10e0bc610>]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAEACAYAAAC57G0KAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAE8VJREFUeJzt3X2wXGVhx/HvhiQECE2MQBKFEIoi+Mrb+AbKpaIFp1q1\n6mhbpVYZnKm1SisEa8u1dXxhxrHa1lrEOEAlilIz4hsvDrexpQIFAgEJGAyvJiEKIu9I8vSP59y5\ne292k5zcc/Y5z9nvZ2bn7p57d++P5eS3z33Os2dBkiRJkiRJkiRJkiRJkiRpKM0BrgZWAz8FPlVs\nXwBcDtwOXAbMT5JOkjQtexZfZwI/AY4FzgZOL7afAXw6QS5JUkX2BK4FXgCsBRYW2xcVtyVJmZlB\nnKJ5mDhyB3iw6/udKbclSZmZR5yiOZ5tC/2BwceRJG3PzBI/+xDwPeAoYBNxamYjsBi4v8fPrwMO\nnm5ASRoydwDPGcQv2oeJFTJ7AKuA1xCnas4oti+j90HWUHu6XTOaOkAPo6kD9DCaOkAPo6kD9DCa\nOkAfo6kD9DCaOkAPo6kD9FBZd+5oBL8YOI84Dz8DuAD4EXADcBHwXuBO4O1VBZIkVWNHBb8GOLLH\n9geAE6qPI0mqyozUARIYSx2gh7HUAXoYSx2gh7HUAXoYSx2gj7HUAXoYSx2gh7HUAXLV1Dl4SWqy\nyrqz5hF8mFXv40uS+ql7imZOzY8vSeqj7oJ3BC9JidRd8LNrfnxJUh8WvCS1lFM0ktRSjuAlqaUs\neElqKadoJKmlHMFLUktZ8JLUUnUX/NyaH1+S1EfdBb+g5seXJPVRd8HvU/PjS5L6qLvgF9b8+JKk\nPuou+OfW/PiSpAQChAchdFIHkaSM5PKBHwTgBTX/DklSD3UX/EXAH9X8OyRJAxYgHA5hMwTXw0vS\nzsni86yLkOEiCMvSRpGkbGRV8EdB+CWEJWnjSFIWcip4gHA2hG+7okaSdii7gn8mhLUQTk4XR5Ky\nkFvBQzFVcz+Et6SJI0lZyLHgoSj5zRD+HcLiwUeSpMbL5o1OU3SuA14CPATcAuEcCIcNNoMkabp2\n8CoU9oNwVjFtcymEv4Zw6GCiSVJj5TpF0/PH9obwVghfhLARwi0QvgbhbRB+p96IktQ4lRV8ncsW\nQ/nHD7OI5655GfEUB8cAm4Hbi8v1wHXAOug8WmFWSWqKXejO3hpW8Ns8xEzgQOAQ4FDgaOAIYCnw\nIHAjsLr4uh64B9gInSz+xJGkHoal4Ps+dIdY8i8Gjiy+HgAsAZ4AVgFXAWPArRa+pIwMe8H3/ZUd\n4kj/lcTpnRHiB3//ALgYuAI6jw02kySVkqA7y2vIqDn8brFC50oIv4bw3eL2EamTSVIPA+vOA4Ar\ngVuAm4EPFttHgXuBG4rLiT3u25CC7xYWQHgnhH+FcBeE2yF8FMLxnidHUkMMrDsXAYcX1+cCtwGH\nAWcBp+3gvg0s+G6hA+EVEJYXRb++GN2fCuFFFr6kRCrrzpk7+P7G4gLwCHAr8OziduYF2AnA/8ZL\nmEGcuz8CeAPwd8AjEC4B/hv4LnS2JIsqSTVbCtxFHMmfBdxJXJ74FWB+j59v+Ah+e8LM4rw5n4Sw\nGsIjEM6DMAJhdup0klpt4G90mktccvgJYCWwH/ENSAD/CCwG3jvlPgH4eNftseKSoXAA8BbgA8T/\nri8BK6CzIWksSW0wUlzGncUAZ0hmAZcCH+rz/aXAmh7bMx7B9xM6EN4I4VwIv4JwcZzHl6TKDKw7\nO8D5wOembO8+1e+HgQt73LeFBd8tLIFwJoSHIHzf8+ZIqsjAuvNYYCvxdADjSyJPIpb+TcQ5+JXA\nwh73bXnBjwsHFfPzd0L4uue5lzRNWXRnFiGrEXaDcCSE/yimbpYXZ8gc8Pn2JbVAFt2ZRchqhd0g\nvKx4p+w6CNdCeL9FL6mELLozi5D1CfMgnFy8Y/ZGCP8AYepKI0maKovuzCJk/cLuEP4MwiiEu4tP\nr3phfAGQpG1k0Z1ZhBysMBfCiuLUCL+A8LzUiSQ1ThbdmUXIdMIXITwK4fMQ/gbCXqkTSWqEtn5k\n3zAJHeCtwLOAdwAzgFOgc1PSWJJSy6I7HcHvtLBXMaJ/qDgg+77UiSQlk0V3ZhGyWcKBcaVNeBjC\n2+JJzyQNmSy6M4uQzRTeV5yf/rHigOyLUyeSNDBZdGcWIZsrzC5G9P8MYfyc9ZLaz4OswyPMAtYC\newOXA48Dy6FzVdJYkupSWXda8FkIHeAE4kndjgSeD51en4MrKX9ZdKdTNLUI8yA8AeEDEA4tzma5\nJHUqSZXJojuzCJmncDyEK4sDseuL5ZXvTp1KUiWy6M4sQrZDeAOEn6ROIakSHmRVt7AH8BjwbuB6\n4Gngt8XlXuj4YivlI4vutFQGKrwfwmoItxRr539enOvmTamTSSoli+7MImS7hS/Ek5lJykhl3emb\nZ9rtv4Dnpg4hKQ3n4FstHAjcClxdbNhKLP1vQOe2ZLEkbY9vdNLOCi8H9ihu7Au8GTgOeC50Hk0W\nS1I/WXSnc/CNFb4XP0ZQUgM5B69pWQUcnjqEpHp5vvHhdDVwGYRnd207FzqXpgokqXrOwQ+tcALw\njOLGMcSPDVwP/D10Lk8WS5IHWVWlMJs4ZfMeYAZ0Tk0cSBpmWXSnB1mzE14LYSx1CmnIZdGdWYRU\nt3AQhLtSp5CGnCcbUx3CHOA3xFU2W7suAP8EnctSJZOGiHPwqks4GphPXEI7fnkXccXVZ4CboPNU\nunxS62XRnU7RtEY4AsK1EDZDuAbC7qkTSS2WRXdmEVJlhNkQfgnh4NRJpBbznaxKofMUsAHYM3US\nSTtmwausx5k4eZmkBvMgq0oKY8AWYHOPb24FzoDOPQONJLVLZd25o3PRHACcD+xX/NJzgC8AC4Bv\nAAcCdwJvB35dRSA13oeAQ/t87zTgaMCClzKwiImzDs4FbgMOA84GTi+2nwF8usd9Pcg6dMKXISyD\nsCh1EiljybpzJXACsBZYWGxbVNyeyoIfOuFUCPdA2ALhlNRppEwl6c6lwF3A3sCDXds7U26Ps+CH\nVvgohHtTp5AyNfDunAtcB7ypuD210B/ocR8LfmiF/S14aZdV1p0784Efs4CLgQuIUzQAm4hTMxuB\nxcD9fe472nV9rLio/R4h/qUnacdGisvAdYiraD43ZfvZxIOrAMvwIKsmCTOLeXiXyUrlDexskscS\nzyx4U9cvPRO4BrgIWEL/ZZKugx9q4VfEdfHb21mfBl4FnTsGk0nKgmeTVNOFvdnxO16/CXzCjwiU\nJhnYG52kXdR5GHh4+z8TNgD7DiKNNIw8F41S2owFL9XGgldKm4APQ1gF4Wvx4KykHLiKRjsQ5kF4\nNYTjIQQIjualTLozi5BqinA3hANTp5AawA/8UOs8hueZlyplwaspHsdPipIqZcGrKR4D9kodQmoT\n3+ikhggriSe1u6/Ena6Fzr/UFEhKxXeyqm3CC4GjStzhQOAPoVPmPlIOsuhOV9GoRuH5EG5JnUKq\ngatoNPSewFU30nZZ8MrVE8Cc1CGkJrPglSsLXtoBC165copGSsiDrKpRmAFhK4SvQ3hj6jRShTzI\nqmHX2QqcBDxafJU0hQWvjHUuBa4Cdk+dRGoiC16582Cr1IcFr9w9iSN4qScLXrlzBC/1YcErd47g\npT78DEzl7glgr/jxf5X4LXQeq+ixpKQseOVuI3AIcHdFjzcHwjOh80hFjye1km90UobCJgiLUqfQ\nUPONTlJNnsA5fbWEBS9N5qoctYYFL03mqhy1hgUvTeYIXq1hwUuTOYJXa1jw0mSO4NUaFrw02ZNY\n8GoJC16azGWSag0LXprMEbxaw4KXJnMEr9aw4KXJngTeDOFjEPZNHUaajp0p+OXAJmBN17ZR4F7g\nhuJyYuXJpDTOB64H3gW8MnEWqXavAo5gcsGfBZy2g/t5sjFlLHwLwttSp9BQGujJxn4MPNhje6eq\nEFIDbQF2Sx1Cmo7pzMH/JXAj8BVgfjVxpMZ4Gj8vQZnb1YL/N+Ag4HBgA/DZyhJJzeAIXtnb1RHK\n/V3XzwUu6fNzo13Xx4qLlAMLXoMyUlySWcrkg6yLu65/GLiwx308yKqMhXMhnJI6hYZSZd25MyP4\nFcBxwD7APcQVNCPE6ZkArAdOrSqQ1BCO4JW9nSn4d/bYtrzqIFLDWPDKnqsEpN62ALMhNOnfSIDO\nltQhlI8617KHmh9fqlH4W+DjqVNM8SSwADpPpg6iWmXRnR5klSoVHoawd+oUqt1A38kqqRm24r9Z\nleDOIuXDglcp7ixSPix4leLOIuXDglcp7ixSPix4leLOIuXDglcp7ixSPrJYH63msOClfDiCVynu\nLFI+LHiV4s4i5cOCVynuLFI+LHiV4s4i5cOCVynuLFI+LHiV4s4i5SPgv1mV4M4i5WMrroNXCRa8\nlA+naFSKO4uUDwtepbizSPmw4FWKO4uUDwtepbizSPmw4FWKO4uUD5dJqhR3FikfLpNUKRa8lA+n\naFSKO4uUDwtepbizSPmw4FWKO4uUDwtepbizSPmw4FWKO4uUD5dJqhR3FikfLpNUKRa8lA+naFSK\nO4uUDwtepbizSPmw4FWKO4uUDwtepezMzrIc2ASs6dq2ALgcuB24DJhffTRJU1jwKmVndpavAidO\n2baMWPCHAD8qbkuql8skVYulTB7BrwUWFtcXFbenCjVnkoZMWAnhTalTqHaVdeeujgYWEqdtKL4u\n3M7PSqqGUzQqZWYFjxHo/4oz2nV9rLhI2jUWfDuNFJdklrLtFM2i4vpinKKRBiBcBOHtqVOodsmn\naL4DnFxcPxlYWU0cSdvhCF6VWwH8AngKuAd4D3GZ5BVsf5mkI3ipUuFCCH+cOoVqV1l37swc/Dv7\nbD+hqhCSdoojeJXiziLlI+DZJFWCBS/lwxG8SqlzNOBoQ6pUOAd4PfBA6iRDZA10/mTAv7Oy7rTg\npWyE+cCS1CmGzOPQ+dmAf2cW3ekqGkkqL/k6eElSw1nwktRSFrwktZQFL0ktZcFLUktZ8JLUUha8\nJLWUBS9JLWXBS1JLWfCS1FIWvCS1lAUvSS1lwUtSS1nwktRSFrwktZQFL0ktZcFLUktZ8JLUUha8\nJLWUBS9JLWXBS1JLWfCS1FIWvCS1lAUvSS1lwUtSS1nwktRSFrwktZQFL0ktZcFLUktZ8JLUUjOn\nef87gd8AW4DfAi+dbiBJUjOsBxb0+V4YZJASRlIH6GEkdYAeRlIH6GEkdYAeRlIH6GMkdYAeRlIH\n6GEkdYAeKuvOKqZoOhU8xiCNpA7Qw0jqAD2MpA7Qw0jqAD2MpA7Qx0jqAD2MpA7Qw0jqAHWabsEH\n4Arg/4BTph9HklSV6c7BHwNsAPYFLgfWAj+ebihJ0vRVOb1yFvAI8Nni9jrg4AofX5KGwR3Ac1KH\n2BPYu7i+F/A/wOvSxZEkdZvOFM1C4Ntdj/M14LJpJ5IkSZI0eCcSD7j+DDij5t+1HNgErOnatoB4\n0Pd24l8V87u+d2aRay2Tp5SOKh7jZ8Dnp5npAOBK4BbgZuCDDcg1B7gaWA38FPhUAzKN2w24Abik\nIZnuBG4qMl3TkEwUv/NbwK3E/4cvS5zrecTnaPzyEHFfT/1cnUn8t7cGuBDYvQGZ/qp4rJuL6zQg\n0y7ZjXiAdSkwi1goh9X4+14FHMHkgj8bOL24fgbw6eL684s8s4p865g40HwNE+/E/T7xRWpXLQIO\nL67PBW4jPgepc+1ZfJ0J/AQ4tgGZAE4jTvF9p7idOlOvN/ClzgRwHvDnxfWZwLyG5IK45HoDcXCT\nMtNS4OfEUgf4BnBy4kwvJPbTHGI/Xk5cgNKU/3elvAL4YdftZcWlTkuZXPBriccIIJbt2uL6mUz+\ni+KHwMuBxcRR0bh3AF+qMN9K4IQG5doTuBZ4QQMy7U98L8XxTIzgU2daDzxzyrbUmeYRi2uq1LnG\nvY6JJdIpMy0gDqieQXwRvAR4beJMbwXO7br9MWKx156pjpONPRu4p+v2vcW2QVpInLah+Dr+JD6r\nyDNuPNvU7fdRXealxL8wrm5ArhnEkcEmJqaQUmf6HPARYGvXttSZer2BL3Wmg4DNwFeB64EvE1ev\npc417h3AiuJ6ykwPEJdq3w38Avg1ccScMtPNxJmGBcTB1euJA5vaM9VR8E07B00gXaa5wMXEObeH\np3wvRa6txKmj/YFXE0fNKTP9AXA/cf6233syUjxPxxBflE8C/oL4jzN1ppnAkcAXi6+Psu1fxqn2\n9dnAG4Bv9vjeoDMdDHyIOLB6FvHf4J8mzrQW+Axxnv0HxEHWlkFkqqPg7yPOw407gMmvOoOwifgn\nD8Q/a+4vrk/Ntj8x233F9e7t900zwyxiuV9AnKJpSi6IB8O+RzxgkzLTK4E3EqdEVgC/R3y+Uj9P\nG4qvm4lLgV/agEz3Fpdri9vfIhb9xsS5IL4QXkd8viDtc3U0cBXwK+Bp4D+J08apn6flRbbjgAeJ\nB1ZT71O7ZCbxnVhLia/sdR9khW3n4M9mYg5rGdsevJhN/JP3DiZGjlcTVyV0mP7Biw5wPnH6oVvK\nXPswcZR+D2AV8JrEmbodx8QcfMpM/d7A14TnaRVwSHF9tMjUhFxfJx7IHJcy00uIUyJ7FI91HvGv\nsNTP037F1yXEefTxA+Sp/9/tkpOIBzrWEQ8Y1GkFca7tKeLc/3uIc11X0Hv50UeLXGuB3+/aPr78\naB3whWlmOpY4HbKaiSVkJybO9SLi3O1q4hLAjxTbUz9X445jYhVNykwHEZ+j1cSiGN9/m/A8vYQ4\ngr+RODKd14BcewG/ZOJFkQZkOp2JZZLnEf+aTp1pVZFpNRNTo6kzSZIkSZIkSZIkSZIkSZIkSZIk\nSVL9/h85qdb6doS1cQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10b952f90>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot([x[1] for x in docsobj.tfidf_ranking])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "number of unique stems = 4742\n",
      "number of total stems = 250000\n"
     ]
    }
   ],
   "source": [
    "docsobj.rank_remove(\"tfidf\",\"stems\",docsobj.tfidf_ranking[5000][1])\n",
    "all_stems = [s for d in docsobj.stems for s in d]\n",
    "print(\"number of unique stems = %d\" % len(set(all_stems)))\n",
    "print(\"number of total stems = %d\" % len(all_stems))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After pre-processing, we have 4742 unique stems, and 250000 total stems.  We now proceed to estimate a topic model on them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Estimating a Topic Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first step in estimation is to initialize a model via topicmodels' LDA class.  We will pass docsobj.stems as the set of documents, and we also need to decide on a number of topics.  Here we choose 30 topics. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ldaobj = topicmodels.LDA.LDAGibbs(docsobj.stems,30)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are three main parameters in LDA, the number of topics, and the two hyperparameters of the Dirichlet priors.  topicmodels.LDA follows the advice of Griffiths and Steyvers (2004) and sets the hyperparameter of the Dirichlet prior on topics to $200/V$, where $V$ is the number of unique vocabulary elements, and the hyperparameter of the Dirichlet prior on document-topic distributions to $50/K$, where $K$ is the number of topics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "30\n",
      "1.66666666667\n",
      "0.0421762969211\n"
     ]
    }
   ],
   "source": [
    "print ldaobj.K # number of topic, user defined.\n",
    "print ldaobj.alpha # hyperparameter for document-topic distribution, automatically defined\n",
    "print ldaobj.beta # hyperparameter for topics, automatically defined"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Should users wish to define their own priors, they can do so by calling ldaobj.set_prior(alpha,beta).  \n",
    "\n",
    "Another quantity set automatically by topicmodels.LDA is a random allocation of stems to topic assignments.  It is a 250000-dimensional vector of integers in $\\{0,\\ldots,29\\}$.  Should the user wish to define another seed, call ldaobj.set_seed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0  8  3 17 29 13  6  3 25  0]\n",
      "(250000,)\n"
     ]
    }
   ],
   "source": [
    "print ldaobj.topic_seed[:10]\n",
    "print ldaobj.topic_seed.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have initialized our topic model, we are ready to sample.  To sample, we pass three parameters.  The first is the number of iterations we want the chain to burn in before beginning to sample.  The second is a thinning interval, the number of iterations to let the chain run between samples.  Allowing for a thinning interval reduces autocorrelation between samples.  The third is the number of samples to take.  So, for example, if the user passes (1000,50,20) the following will happen.  First, the chain will run for 1,000 iterations.  Then 20 samples will be taken corresponding to the $\\{1050,1100\\ldots,1950,2000\\}$ iterations for a total of 2000 iterations overall.\n",
    "\n",
    "In order not to waste time in the tutorial, we start with a relatively short chain with no burnin, a thinning interval of 50, and 10 samples, for a total of 500 iterations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration 1 of (collapsed) Gibbs sampling\n",
      "Iteration 2 of (collapsed) Gibbs sampling\n",
      "Iteration 3 of (collapsed) Gibbs sampling\n",
      "Iteration 4 of (collapsed) Gibbs sampling\n",
      "Iteration 5 of (collapsed) Gibbs sampling\n",
      "Iteration 6 of (collapsed) Gibbs sampling\n",
      "Iteration 7 of (collapsed) Gibbs sampling\n",
      "Iteration 8 of (collapsed) Gibbs sampling\n",
      "Iteration 9 of (collapsed) Gibbs sampling\n",
      "Iteration 10 of (collapsed) Gibbs sampling\n",
      "Iteration 11 of (collapsed) Gibbs sampling\n",
      "Iteration 12 of (collapsed) Gibbs sampling\n",
      "Iteration 13 of (collapsed) Gibbs sampling\n",
      "Iteration 14 of (collapsed) Gibbs sampling\n",
      "Iteration 15 of (collapsed) Gibbs sampling\n",
      "Iteration 16 of (collapsed) Gibbs sampling\n",
      "Iteration 17 of (collapsed) Gibbs sampling\n",
      "Iteration 18 of (collapsed) Gibbs sampling\n",
      "Iteration 19 of (collapsed) Gibbs sampling\n",
      "Iteration 20 of (collapsed) Gibbs sampling\n",
      "Iteration 21 of (collapsed) Gibbs sampling\n",
      "Iteration 22 of (collapsed) Gibbs sampling\n",
      "Iteration 23 of (collapsed) Gibbs sampling\n",
      "Iteration 24 of (collapsed) Gibbs sampling\n",
      "Iteration 25 of (collapsed) Gibbs sampling\n",
      "Iteration 26 of (collapsed) Gibbs sampling\n",
      "Iteration 27 of (collapsed) Gibbs sampling\n",
      "Iteration 28 of (collapsed) Gibbs sampling\n",
      "Iteration 29 of (collapsed) Gibbs sampling\n",
      "Iteration 30 of (collapsed) Gibbs sampling\n",
      "Iteration 31 of (collapsed) Gibbs sampling\n",
      "Iteration 32 of (collapsed) Gibbs sampling\n",
      "Iteration 33 of (collapsed) Gibbs sampling\n",
      "Iteration 34 of (collapsed) Gibbs sampling\n",
      "Iteration 35 of (collapsed) Gibbs sampling\n",
      "Iteration 36 of (collapsed) Gibbs sampling\n",
      "Iteration 37 of (collapsed) Gibbs sampling\n",
      "Iteration 38 of (collapsed) Gibbs sampling\n",
      "Iteration 39 of (collapsed) Gibbs sampling\n",
      "Iteration 40 of (collapsed) Gibbs sampling\n",
      "Iteration 41 of (collapsed) Gibbs sampling\n",
      "Iteration 42 of (collapsed) Gibbs sampling\n",
      "Iteration 43 of (collapsed) Gibbs sampling\n",
      "Iteration 44 of (collapsed) Gibbs sampling\n",
      "Iteration 45 of (collapsed) Gibbs sampling\n",
      "Iteration 46 of (collapsed) Gibbs sampling\n",
      "Iteration 47 of (collapsed) Gibbs sampling\n",
      "Iteration 48 of (collapsed) Gibbs sampling\n",
      "Iteration 49 of (collapsed) Gibbs sampling\n",
      "Iteration 50 of (collapsed) Gibbs sampling\n",
      "Iteration 51 of (collapsed) Gibbs sampling\n",
      "Iteration 52 of (collapsed) Gibbs sampling\n",
      "Iteration 53 of (collapsed) Gibbs sampling\n",
      "Iteration 54 of (collapsed) Gibbs sampling\n",
      "Iteration 55 of (collapsed) Gibbs sampling\n",
      "Iteration 56 of (collapsed) Gibbs sampling\n",
      "Iteration 57 of (collapsed) Gibbs sampling\n",
      "Iteration 58 of (collapsed) Gibbs sampling\n",
      "Iteration 59 of (collapsed) Gibbs sampling\n",
      "Iteration 60 of (collapsed) Gibbs sampling\n",
      "Iteration 61 of (collapsed) Gibbs sampling\n",
      "Iteration 62 of (collapsed) Gibbs sampling\n",
      "Iteration 63 of (collapsed) Gibbs sampling\n",
      "Iteration 64 of (collapsed) Gibbs sampling\n",
      "Iteration 65 of (collapsed) Gibbs sampling\n",
      "Iteration 66 of (collapsed) Gibbs sampling\n",
      "Iteration 67 of (collapsed) Gibbs sampling\n",
      "Iteration 68 of (collapsed) Gibbs sampling\n",
      "Iteration 69 of (collapsed) Gibbs sampling\n",
      "Iteration 70 of (collapsed) Gibbs sampling\n",
      "Iteration 71 of (collapsed) Gibbs sampling\n",
      "Iteration 72 of (collapsed) Gibbs sampling\n",
      "Iteration 73 of (collapsed) Gibbs sampling\n",
      "Iteration 74 of (collapsed) Gibbs sampling\n",
      "Iteration 75 of (collapsed) Gibbs sampling\n",
      "Iteration 76 of (collapsed) Gibbs sampling\n",
      "Iteration 77 of (collapsed) Gibbs sampling\n",
      "Iteration 78 of (collapsed) Gibbs sampling\n",
      "Iteration 79 of (collapsed) Gibbs sampling\n",
      "Iteration 80 of (collapsed) Gibbs sampling\n",
      "Iteration 81 of (collapsed) Gibbs sampling\n",
      "Iteration 82 of (collapsed) Gibbs sampling\n",
      "Iteration 83 of (collapsed) Gibbs sampling\n",
      "Iteration 84 of (collapsed) Gibbs sampling\n",
      "Iteration 85 of (collapsed) Gibbs sampling\n",
      "Iteration 86 of (collapsed) Gibbs sampling\n",
      "Iteration 87 of (collapsed) Gibbs sampling\n",
      "Iteration 88 of (collapsed) Gibbs sampling\n",
      "Iteration 89 of (collapsed) Gibbs sampling\n",
      "Iteration 90 of (collapsed) Gibbs sampling\n",
      "Iteration 91 of (collapsed) Gibbs sampling\n",
      "Iteration 92 of (collapsed) Gibbs sampling\n",
      "Iteration 93 of (collapsed) Gibbs sampling\n",
      "Iteration 94 of (collapsed) Gibbs sampling\n",
      "Iteration 95 of (collapsed) Gibbs sampling\n",
      "Iteration 96 of (collapsed) Gibbs sampling\n",
      "Iteration 97 of (collapsed) Gibbs sampling\n",
      "Iteration 98 of (collapsed) Gibbs sampling\n",
      "Iteration 99 of (collapsed) Gibbs sampling\n",
      "Iteration 100 of (collapsed) Gibbs sampling\n",
      "Iteration 101 of (collapsed) Gibbs sampling\n",
      "Iteration 102 of (collapsed) Gibbs sampling\n",
      "Iteration 103 of (collapsed) Gibbs sampling\n",
      "Iteration 104 of (collapsed) Gibbs sampling\n",
      "Iteration 105 of (collapsed) Gibbs sampling\n",
      "Iteration 106 of (collapsed) Gibbs sampling\n",
      "Iteration 107 of (collapsed) Gibbs sampling\n",
      "Iteration 108 of (collapsed) Gibbs sampling\n",
      "Iteration 109 of (collapsed) Gibbs sampling\n",
      "Iteration 110 of (collapsed) Gibbs sampling\n",
      "Iteration 111 of (collapsed) Gibbs sampling\n",
      "Iteration 112 of (collapsed) Gibbs sampling\n",
      "Iteration 113 of (collapsed) Gibbs sampling\n",
      "Iteration 114 of (collapsed) Gibbs sampling\n",
      "Iteration 115 of (collapsed) Gibbs sampling\n",
      "Iteration 116 of (collapsed) Gibbs sampling\n",
      "Iteration 117 of (collapsed) Gibbs sampling\n",
      "Iteration 118 of (collapsed) Gibbs sampling\n",
      "Iteration 119 of (collapsed) Gibbs sampling\n",
      "Iteration 120 of (collapsed) Gibbs sampling\n",
      "Iteration 121 of (collapsed) Gibbs sampling\n",
      "Iteration 122 of (collapsed) Gibbs sampling\n",
      "Iteration 123 of (collapsed) Gibbs sampling\n",
      "Iteration 124 of (collapsed) Gibbs sampling\n",
      "Iteration 125 of (collapsed) Gibbs sampling\n",
      "Iteration 126 of (collapsed) Gibbs sampling\n",
      "Iteration 127 of (collapsed) Gibbs sampling\n",
      "Iteration 128 of (collapsed) Gibbs sampling\n",
      "Iteration 129 of (collapsed) Gibbs sampling\n",
      "Iteration 130 of (collapsed) Gibbs sampling\n",
      "Iteration 131 of (collapsed) Gibbs sampling\n",
      "Iteration 132 of (collapsed) Gibbs sampling\n",
      "Iteration 133 of (collapsed) Gibbs sampling\n",
      "Iteration 134 of (collapsed) Gibbs sampling\n",
      "Iteration 135 of (collapsed) Gibbs sampling\n",
      "Iteration 136 of (collapsed) Gibbs sampling\n",
      "Iteration 137 of (collapsed) Gibbs sampling\n",
      "Iteration 138 of (collapsed) Gibbs sampling\n",
      "Iteration 139 of (collapsed) Gibbs sampling\n",
      "Iteration 140 of (collapsed) Gibbs sampling\n",
      "Iteration 141 of (collapsed) Gibbs sampling\n",
      "Iteration 142 of (collapsed) Gibbs sampling\n",
      "Iteration 143 of (collapsed) Gibbs sampling\n",
      "Iteration 144 of (collapsed) Gibbs sampling\n",
      "Iteration 145 of (collapsed) Gibbs sampling\n",
      "Iteration 146 of (collapsed) Gibbs sampling\n",
      "Iteration 147 of (collapsed) Gibbs sampling\n",
      "Iteration 148 of (collapsed) Gibbs sampling\n",
      "Iteration 149 of (collapsed) Gibbs sampling\n",
      "Iteration 150 of (collapsed) Gibbs sampling\n",
      "Iteration 151 of (collapsed) Gibbs sampling\n",
      "Iteration 152 of (collapsed) Gibbs sampling\n",
      "Iteration 153 of (collapsed) Gibbs sampling\n",
      "Iteration 154 of (collapsed) Gibbs sampling\n",
      "Iteration 155 of (collapsed) Gibbs sampling\n",
      "Iteration 156 of (collapsed) Gibbs sampling\n",
      "Iteration 157 of (collapsed) Gibbs sampling\n",
      "Iteration 158 of (collapsed) Gibbs sampling\n",
      "Iteration 159 of (collapsed) Gibbs sampling\n",
      "Iteration 160 of (collapsed) Gibbs sampling\n",
      "Iteration 161 of (collapsed) Gibbs sampling\n",
      "Iteration 162 of (collapsed) Gibbs sampling\n",
      "Iteration 163 of (collapsed) Gibbs sampling\n",
      "Iteration 164 of (collapsed) Gibbs sampling\n",
      "Iteration 165 of (collapsed) Gibbs sampling\n",
      "Iteration 166 of (collapsed) Gibbs sampling\n",
      "Iteration 167 of (collapsed) Gibbs sampling\n",
      "Iteration 168 of (collapsed) Gibbs sampling\n",
      "Iteration 169 of (collapsed) Gibbs sampling\n",
      "Iteration 170 of (collapsed) Gibbs sampling\n",
      "Iteration 171 of (collapsed) Gibbs sampling\n",
      "Iteration 172 of (collapsed) Gibbs sampling\n",
      "Iteration 173 of (collapsed) Gibbs sampling\n",
      "Iteration 174 of (collapsed) Gibbs sampling\n",
      "Iteration 175 of (collapsed) Gibbs sampling\n",
      "Iteration 176 of (collapsed) Gibbs sampling\n",
      "Iteration 177 of (collapsed) Gibbs sampling\n",
      "Iteration 178 of (collapsed) Gibbs sampling\n",
      "Iteration 179 of (collapsed) Gibbs sampling\n",
      "Iteration 180 of (collapsed) Gibbs sampling\n",
      "Iteration 181 of (collapsed) Gibbs sampling\n",
      "Iteration 182 of (collapsed) Gibbs sampling\n",
      "Iteration 183 of (collapsed) Gibbs sampling\n",
      "Iteration 184 of (collapsed) Gibbs sampling\n",
      "Iteration 185 of (collapsed) Gibbs sampling\n",
      "Iteration 186 of (collapsed) Gibbs sampling\n",
      "Iteration 187 of (collapsed) Gibbs sampling\n",
      "Iteration 188 of (collapsed) Gibbs sampling\n",
      "Iteration 189 of (collapsed) Gibbs sampling\n",
      "Iteration 190 of (collapsed) Gibbs sampling\n",
      "Iteration 191 of (collapsed) Gibbs sampling\n",
      "Iteration 192 of (collapsed) Gibbs sampling\n",
      "Iteration 193 of (collapsed) Gibbs sampling\n",
      "Iteration 194 of (collapsed) Gibbs sampling\n",
      "Iteration 195 of (collapsed) Gibbs sampling\n",
      "Iteration 196 of (collapsed) Gibbs sampling\n",
      "Iteration 197 of (collapsed) Gibbs sampling\n",
      "Iteration 198 of (collapsed) Gibbs sampling\n",
      "Iteration 199 of (collapsed) Gibbs sampling\n",
      "Iteration 200 of (collapsed) Gibbs sampling\n",
      "Iteration 201 of (collapsed) Gibbs sampling\n",
      "Iteration 202 of (collapsed) Gibbs sampling\n",
      "Iteration 203 of (collapsed) Gibbs sampling\n",
      "Iteration 204 of (collapsed) Gibbs sampling\n",
      "Iteration 205 of (collapsed) Gibbs sampling\n",
      "Iteration 206 of (collapsed) Gibbs sampling\n",
      "Iteration 207 of (collapsed) Gibbs sampling\n",
      "Iteration 208 of (collapsed) Gibbs sampling\n",
      "Iteration 209 of (collapsed) Gibbs sampling\n",
      "Iteration 210 of (collapsed) Gibbs sampling\n",
      "Iteration 211 of (collapsed) Gibbs sampling\n",
      "Iteration 212 of (collapsed) Gibbs sampling\n",
      "Iteration 213 of (collapsed) Gibbs sampling\n",
      "Iteration 214 of (collapsed) Gibbs sampling\n",
      "Iteration 215 of (collapsed) Gibbs sampling\n",
      "Iteration 216 of (collapsed) Gibbs sampling\n",
      "Iteration 217 of (collapsed) Gibbs sampling\n",
      "Iteration 218 of (collapsed) Gibbs sampling\n",
      "Iteration 219 of (collapsed) Gibbs sampling\n",
      "Iteration 220 of (collapsed) Gibbs sampling\n",
      "Iteration 221 of (collapsed) Gibbs sampling\n",
      "Iteration 222 of (collapsed) Gibbs sampling\n",
      "Iteration 223 of (collapsed) Gibbs sampling\n",
      "Iteration 224 of (collapsed) Gibbs sampling\n",
      "Iteration 225 of (collapsed) Gibbs sampling\n",
      "Iteration 226 of (collapsed) Gibbs sampling\n",
      "Iteration 227 of (collapsed) Gibbs sampling\n",
      "Iteration 228 of (collapsed) Gibbs sampling\n",
      "Iteration 229 of (collapsed) Gibbs sampling\n",
      "Iteration 230 of (collapsed) Gibbs sampling\n",
      "Iteration 231 of (collapsed) Gibbs sampling\n",
      "Iteration 232 of (collapsed) Gibbs sampling\n",
      "Iteration 233 of (collapsed) Gibbs sampling\n",
      "Iteration 234 of (collapsed) Gibbs sampling\n",
      "Iteration 235 of (collapsed) Gibbs sampling\n",
      "Iteration 236 of (collapsed) Gibbs sampling\n",
      "Iteration 237 of (collapsed) Gibbs sampling\n",
      "Iteration 238 of (collapsed) Gibbs sampling\n",
      "Iteration 239 of (collapsed) Gibbs sampling\n",
      "Iteration 240 of (collapsed) Gibbs sampling\n",
      "Iteration 241 of (collapsed) Gibbs sampling\n",
      "Iteration 242 of (collapsed) Gibbs sampling\n",
      "Iteration 243 of (collapsed) Gibbs sampling\n",
      "Iteration 244 of (collapsed) Gibbs sampling\n",
      "Iteration 245 of (collapsed) Gibbs sampling\n",
      "Iteration 246 of (collapsed) Gibbs sampling\n",
      "Iteration 247 of (collapsed) Gibbs sampling\n",
      "Iteration 248 of (collapsed) Gibbs sampling\n",
      "Iteration 249 of (collapsed) Gibbs sampling\n",
      "Iteration 250 of (collapsed) Gibbs sampling\n",
      "Iteration 251 of (collapsed) Gibbs sampling\n",
      "Iteration 252 of (collapsed) Gibbs sampling\n",
      "Iteration 253 of (collapsed) Gibbs sampling\n",
      "Iteration 254 of (collapsed) Gibbs sampling\n",
      "Iteration 255 of (collapsed) Gibbs sampling\n",
      "Iteration 256 of (collapsed) Gibbs sampling\n",
      "Iteration 257 of (collapsed) Gibbs sampling\n",
      "Iteration 258 of (collapsed) Gibbs sampling\n",
      "Iteration 259 of (collapsed) Gibbs sampling\n",
      "Iteration 260 of (collapsed) Gibbs sampling\n",
      "Iteration 261 of (collapsed) Gibbs sampling\n",
      "Iteration 262 of (collapsed) Gibbs sampling\n",
      "Iteration 263 of (collapsed) Gibbs sampling\n",
      "Iteration 264 of (collapsed) Gibbs sampling\n",
      "Iteration 265 of (collapsed) Gibbs sampling\n",
      "Iteration 266 of (collapsed) Gibbs sampling\n",
      "Iteration 267 of (collapsed) Gibbs sampling\n",
      "Iteration 268 of (collapsed) Gibbs sampling\n",
      "Iteration 269 of (collapsed) Gibbs sampling\n",
      "Iteration 270 of (collapsed) Gibbs sampling\n",
      "Iteration 271 of (collapsed) Gibbs sampling\n",
      "Iteration 272 of (collapsed) Gibbs sampling\n",
      "Iteration 273 of (collapsed) Gibbs sampling\n",
      "Iteration 274 of (collapsed) Gibbs sampling\n",
      "Iteration 275 of (collapsed) Gibbs sampling\n",
      "Iteration 276 of (collapsed) Gibbs sampling\n",
      "Iteration 277 of (collapsed) Gibbs sampling\n",
      "Iteration 278 of (collapsed) Gibbs sampling\n",
      "Iteration 279 of (collapsed) Gibbs sampling\n",
      "Iteration 280 of (collapsed) Gibbs sampling\n",
      "Iteration 281 of (collapsed) Gibbs sampling\n",
      "Iteration 282 of (collapsed) Gibbs sampling\n",
      "Iteration 283 of (collapsed) Gibbs sampling\n",
      "Iteration 284 of (collapsed) Gibbs sampling\n",
      "Iteration 285 of (collapsed) Gibbs sampling\n",
      "Iteration 286 of (collapsed) Gibbs sampling\n",
      "Iteration 287 of (collapsed) Gibbs sampling\n",
      "Iteration 288 of (collapsed) Gibbs sampling\n",
      "Iteration 289 of (collapsed) Gibbs sampling\n",
      "Iteration 290 of (collapsed) Gibbs sampling\n",
      "Iteration 291 of (collapsed) Gibbs sampling\n",
      "Iteration 292 of (collapsed) Gibbs sampling\n",
      "Iteration 293 of (collapsed) Gibbs sampling\n",
      "Iteration 294 of (collapsed) Gibbs sampling\n",
      "Iteration 295 of (collapsed) Gibbs sampling\n",
      "Iteration 296 of (collapsed) Gibbs sampling\n",
      "Iteration 297 of (collapsed) Gibbs sampling\n",
      "Iteration 298 of (collapsed) Gibbs sampling\n",
      "Iteration 299 of (collapsed) Gibbs sampling\n",
      "Iteration 300 of (collapsed) Gibbs sampling\n",
      "Iteration 301 of (collapsed) Gibbs sampling\n",
      "Iteration 302 of (collapsed) Gibbs sampling\n",
      "Iteration 303 of (collapsed) Gibbs sampling\n",
      "Iteration 304 of (collapsed) Gibbs sampling\n",
      "Iteration 305 of (collapsed) Gibbs sampling\n",
      "Iteration 306 of (collapsed) Gibbs sampling\n",
      "Iteration 307 of (collapsed) Gibbs sampling\n",
      "Iteration 308 of (collapsed) Gibbs sampling\n",
      "Iteration 309 of (collapsed) Gibbs sampling\n",
      "Iteration 310 of (collapsed) Gibbs sampling\n",
      "Iteration 311 of (collapsed) Gibbs sampling\n",
      "Iteration 312 of (collapsed) Gibbs sampling\n",
      "Iteration 313 of (collapsed) Gibbs sampling\n",
      "Iteration 314 of (collapsed) Gibbs sampling\n",
      "Iteration 315 of (collapsed) Gibbs sampling\n",
      "Iteration 316 of (collapsed) Gibbs sampling\n",
      "Iteration 317 of (collapsed) Gibbs sampling\n",
      "Iteration 318 of (collapsed) Gibbs sampling\n",
      "Iteration 319 of (collapsed) Gibbs sampling\n",
      "Iteration 320 of (collapsed) Gibbs sampling\n",
      "Iteration 321 of (collapsed) Gibbs sampling\n",
      "Iteration 322 of (collapsed) Gibbs sampling\n",
      "Iteration 323 of (collapsed) Gibbs sampling\n",
      "Iteration 324 of (collapsed) Gibbs sampling\n",
      "Iteration 325 of (collapsed) Gibbs sampling\n",
      "Iteration 326 of (collapsed) Gibbs sampling\n",
      "Iteration 327 of (collapsed) Gibbs sampling\n",
      "Iteration 328 of (collapsed) Gibbs sampling\n",
      "Iteration 329 of (collapsed) Gibbs sampling\n",
      "Iteration 330 of (collapsed) Gibbs sampling\n",
      "Iteration 331 of (collapsed) Gibbs sampling\n",
      "Iteration 332 of (collapsed) Gibbs sampling\n",
      "Iteration 333 of (collapsed) Gibbs sampling\n",
      "Iteration 334 of (collapsed) Gibbs sampling\n",
      "Iteration 335 of (collapsed) Gibbs sampling\n",
      "Iteration 336 of (collapsed) Gibbs sampling\n",
      "Iteration 337 of (collapsed) Gibbs sampling\n",
      "Iteration 338 of (collapsed) Gibbs sampling\n",
      "Iteration 339 of (collapsed) Gibbs sampling\n",
      "Iteration 340 of (collapsed) Gibbs sampling\n",
      "Iteration 341 of (collapsed) Gibbs sampling\n",
      "Iteration 342 of (collapsed) Gibbs sampling\n",
      "Iteration 343 of (collapsed) Gibbs sampling\n",
      "Iteration 344 of (collapsed) Gibbs sampling\n",
      "Iteration 345 of (collapsed) Gibbs sampling\n",
      "Iteration 346 of (collapsed) Gibbs sampling\n",
      "Iteration 347 of (collapsed) Gibbs sampling\n",
      "Iteration 348 of (collapsed) Gibbs sampling\n",
      "Iteration 349 of (collapsed) Gibbs sampling\n",
      "Iteration 350 of (collapsed) Gibbs sampling\n",
      "Iteration 351 of (collapsed) Gibbs sampling\n",
      "Iteration 352 of (collapsed) Gibbs sampling\n",
      "Iteration 353 of (collapsed) Gibbs sampling\n",
      "Iteration 354 of (collapsed) Gibbs sampling\n",
      "Iteration 355 of (collapsed) Gibbs sampling\n",
      "Iteration 356 of (collapsed) Gibbs sampling\n",
      "Iteration 357 of (collapsed) Gibbs sampling\n",
      "Iteration 358 of (collapsed) Gibbs sampling\n",
      "Iteration 359 of (collapsed) Gibbs sampling\n",
      "Iteration 360 of (collapsed) Gibbs sampling\n",
      "Iteration 361 of (collapsed) Gibbs sampling\n",
      "Iteration 362 of (collapsed) Gibbs sampling\n",
      "Iteration 363 of (collapsed) Gibbs sampling\n",
      "Iteration 364 of (collapsed) Gibbs sampling\n",
      "Iteration 365 of (collapsed) Gibbs sampling\n",
      "Iteration 366 of (collapsed) Gibbs sampling\n",
      "Iteration 367 of (collapsed) Gibbs sampling\n",
      "Iteration 368 of (collapsed) Gibbs sampling\n",
      "Iteration 369 of (collapsed) Gibbs sampling\n",
      "Iteration 370 of (collapsed) Gibbs sampling\n",
      "Iteration 371 of (collapsed) Gibbs sampling\n",
      "Iteration 372 of (collapsed) Gibbs sampling\n",
      "Iteration 373 of (collapsed) Gibbs sampling\n",
      "Iteration 374 of (collapsed) Gibbs sampling\n",
      "Iteration 375 of (collapsed) Gibbs sampling\n",
      "Iteration 376 of (collapsed) Gibbs sampling\n",
      "Iteration 377 of (collapsed) Gibbs sampling\n",
      "Iteration 378 of (collapsed) Gibbs sampling\n",
      "Iteration 379 of (collapsed) Gibbs sampling\n",
      "Iteration 380 of (collapsed) Gibbs sampling\n",
      "Iteration 381 of (collapsed) Gibbs sampling\n",
      "Iteration 382 of (collapsed) Gibbs sampling\n",
      "Iteration 383 of (collapsed) Gibbs sampling\n",
      "Iteration 384 of (collapsed) Gibbs sampling\n",
      "Iteration 385 of (collapsed) Gibbs sampling\n",
      "Iteration 386 of (collapsed) Gibbs sampling\n",
      "Iteration 387 of (collapsed) Gibbs sampling\n",
      "Iteration 388 of (collapsed) Gibbs sampling\n",
      "Iteration 389 of (collapsed) Gibbs sampling\n",
      "Iteration 390 of (collapsed) Gibbs sampling\n",
      "Iteration 391 of (collapsed) Gibbs sampling\n",
      "Iteration 392 of (collapsed) Gibbs sampling\n",
      "Iteration 393 of (collapsed) Gibbs sampling\n",
      "Iteration 394 of (collapsed) Gibbs sampling\n",
      "Iteration 395 of (collapsed) Gibbs sampling\n",
      "Iteration 396 of (collapsed) Gibbs sampling\n",
      "Iteration 397 of (collapsed) Gibbs sampling\n",
      "Iteration 398 of (collapsed) Gibbs sampling\n",
      "Iteration 399 of (collapsed) Gibbs sampling\n",
      "Iteration 400 of (collapsed) Gibbs sampling\n",
      "Iteration 401 of (collapsed) Gibbs sampling\n",
      "Iteration 402 of (collapsed) Gibbs sampling\n",
      "Iteration 403 of (collapsed) Gibbs sampling\n",
      "Iteration 404 of (collapsed) Gibbs sampling\n",
      "Iteration 405 of (collapsed) Gibbs sampling\n",
      "Iteration 406 of (collapsed) Gibbs sampling\n",
      "Iteration 407 of (collapsed) Gibbs sampling\n",
      "Iteration 408 of (collapsed) Gibbs sampling\n",
      "Iteration 409 of (collapsed) Gibbs sampling\n",
      "Iteration 410 of (collapsed) Gibbs sampling\n",
      "Iteration 411 of (collapsed) Gibbs sampling\n",
      "Iteration 412 of (collapsed) Gibbs sampling\n",
      "Iteration 413 of (collapsed) Gibbs sampling\n",
      "Iteration 414 of (collapsed) Gibbs sampling\n",
      "Iteration 415 of (collapsed) Gibbs sampling\n",
      "Iteration 416 of (collapsed) Gibbs sampling\n",
      "Iteration 417 of (collapsed) Gibbs sampling\n",
      "Iteration 418 of (collapsed) Gibbs sampling\n",
      "Iteration 419 of (collapsed) Gibbs sampling\n",
      "Iteration 420 of (collapsed) Gibbs sampling\n",
      "Iteration 421 of (collapsed) Gibbs sampling\n",
      "Iteration 422 of (collapsed) Gibbs sampling\n",
      "Iteration 423 of (collapsed) Gibbs sampling\n",
      "Iteration 424 of (collapsed) Gibbs sampling\n",
      "Iteration 425 of (collapsed) Gibbs sampling\n",
      "Iteration 426 of (collapsed) Gibbs sampling\n",
      "Iteration 427 of (collapsed) Gibbs sampling\n",
      "Iteration 428 of (collapsed) Gibbs sampling\n",
      "Iteration 429 of (collapsed) Gibbs sampling\n",
      "Iteration 430 of (collapsed) Gibbs sampling\n",
      "Iteration 431 of (collapsed) Gibbs sampling\n",
      "Iteration 432 of (collapsed) Gibbs sampling\n",
      "Iteration 433 of (collapsed) Gibbs sampling\n",
      "Iteration 434 of (collapsed) Gibbs sampling\n",
      "Iteration 435 of (collapsed) Gibbs sampling\n",
      "Iteration 436 of (collapsed) Gibbs sampling\n",
      "Iteration 437 of (collapsed) Gibbs sampling\n",
      "Iteration 438 of (collapsed) Gibbs sampling\n",
      "Iteration 439 of (collapsed) Gibbs sampling\n",
      "Iteration 440 of (collapsed) Gibbs sampling\n",
      "Iteration 441 of (collapsed) Gibbs sampling\n",
      "Iteration 442 of (collapsed) Gibbs sampling\n",
      "Iteration 443 of (collapsed) Gibbs sampling\n",
      "Iteration 444 of (collapsed) Gibbs sampling\n",
      "Iteration 445 of (collapsed) Gibbs sampling\n",
      "Iteration 446 of (collapsed) Gibbs sampling\n",
      "Iteration 447 of (collapsed) Gibbs sampling\n",
      "Iteration 448 of (collapsed) Gibbs sampling\n",
      "Iteration 449 of (collapsed) Gibbs sampling\n",
      "Iteration 450 of (collapsed) Gibbs sampling\n",
      "Iteration 451 of (collapsed) Gibbs sampling\n",
      "Iteration 452 of (collapsed) Gibbs sampling\n",
      "Iteration 453 of (collapsed) Gibbs sampling\n",
      "Iteration 454 of (collapsed) Gibbs sampling\n",
      "Iteration 455 of (collapsed) Gibbs sampling\n",
      "Iteration 456 of (collapsed) Gibbs sampling\n",
      "Iteration 457 of (collapsed) Gibbs sampling\n",
      "Iteration 458 of (collapsed) Gibbs sampling\n",
      "Iteration 459 of (collapsed) Gibbs sampling\n",
      "Iteration 460 of (collapsed) Gibbs sampling\n",
      "Iteration 461 of (collapsed) Gibbs sampling\n",
      "Iteration 462 of (collapsed) Gibbs sampling\n",
      "Iteration 463 of (collapsed) Gibbs sampling\n",
      "Iteration 464 of (collapsed) Gibbs sampling\n",
      "Iteration 465 of (collapsed) Gibbs sampling\n",
      "Iteration 466 of (collapsed) Gibbs sampling\n",
      "Iteration 467 of (collapsed) Gibbs sampling\n",
      "Iteration 468 of (collapsed) Gibbs sampling\n",
      "Iteration 469 of (collapsed) Gibbs sampling\n",
      "Iteration 470 of (collapsed) Gibbs sampling\n",
      "Iteration 471 of (collapsed) Gibbs sampling\n",
      "Iteration 472 of (collapsed) Gibbs sampling\n",
      "Iteration 473 of (collapsed) Gibbs sampling\n",
      "Iteration 474 of (collapsed) Gibbs sampling\n",
      "Iteration 475 of (collapsed) Gibbs sampling\n",
      "Iteration 476 of (collapsed) Gibbs sampling\n",
      "Iteration 477 of (collapsed) Gibbs sampling\n",
      "Iteration 478 of (collapsed) Gibbs sampling\n",
      "Iteration 479 of (collapsed) Gibbs sampling\n",
      "Iteration 480 of (collapsed) Gibbs sampling\n",
      "Iteration 481 of (collapsed) Gibbs sampling\n",
      "Iteration 482 of (collapsed) Gibbs sampling\n",
      "Iteration 483 of (collapsed) Gibbs sampling\n",
      "Iteration 484 of (collapsed) Gibbs sampling\n",
      "Iteration 485 of (collapsed) Gibbs sampling\n",
      "Iteration 486 of (collapsed) Gibbs sampling\n",
      "Iteration 487 of (collapsed) Gibbs sampling\n",
      "Iteration 488 of (collapsed) Gibbs sampling\n",
      "Iteration 489 of (collapsed) Gibbs sampling\n",
      "Iteration 490 of (collapsed) Gibbs sampling\n",
      "Iteration 491 of (collapsed) Gibbs sampling\n",
      "Iteration 492 of (collapsed) Gibbs sampling\n",
      "Iteration 493 of (collapsed) Gibbs sampling\n",
      "Iteration 494 of (collapsed) Gibbs sampling\n",
      "Iteration 495 of (collapsed) Gibbs sampling\n",
      "Iteration 496 of (collapsed) Gibbs sampling\n",
      "Iteration 497 of (collapsed) Gibbs sampling\n",
      "Iteration 498 of (collapsed) Gibbs sampling\n",
      "Iteration 499 of (collapsed) Gibbs sampling\n",
      "Iteration 500 of (collapsed) Gibbs sampling\n"
     ]
    }
   ],
   "source": [
    "ldaobj.sample(0,50,10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because we allowed no burn in and started sampling straight away, one would imagine the initial draws were poor in terms of describing topics.  A formalization of this idea is to compute the perplexity of each of the samples.  Perplexity is a common goodness-of-fit meausure in natural language processing and information theory literature that describes how well a probability model explains data.  Lower values indicate better goodness-of-fit.  Calling ldaobj.perplexity() returns the perplexity of each sample."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 915.74059849,  871.91170051,  858.53324181,  851.5958892 ,\n",
       "        847.71522759,  845.05538051,  846.21565792,  844.11551862,\n",
       "        843.37388303,  843.68515761])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ldaobj.perplexity()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just as we suspected, the first sample has a much higher perplexity than the last.  Moreover, it might be that if we had kept sampling the chain, we could get even lower perplexity.  Once we call ldaobj.sample the first time, all further calls extend the existing chain by default rather than start from scratch.  So let's draw another ten samples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration 1 of (collapsed) Gibbs sampling\n",
      "Iteration 2 of (collapsed) Gibbs sampling\n",
      "Iteration 3 of (collapsed) Gibbs sampling\n",
      "Iteration 4 of (collapsed) Gibbs sampling\n",
      "Iteration 5 of (collapsed) Gibbs sampling\n",
      "Iteration 6 of (collapsed) Gibbs sampling\n",
      "Iteration 7 of (collapsed) Gibbs sampling\n",
      "Iteration 8 of (collapsed) Gibbs sampling\n",
      "Iteration 9 of (collapsed) Gibbs sampling\n",
      "Iteration 10 of (collapsed) Gibbs sampling\n",
      "Iteration 11 of (collapsed) Gibbs sampling\n",
      "Iteration 12 of (collapsed) Gibbs sampling\n",
      "Iteration 13 of (collapsed) Gibbs sampling\n",
      "Iteration 14 of (collapsed) Gibbs sampling\n",
      "Iteration 15 of (collapsed) Gibbs sampling\n",
      "Iteration 16 of (collapsed) Gibbs sampling\n",
      "Iteration 17 of (collapsed) Gibbs sampling\n",
      "Iteration 18 of (collapsed) Gibbs sampling\n",
      "Iteration 19 of (collapsed) Gibbs sampling\n",
      "Iteration 20 of (collapsed) Gibbs sampling\n",
      "Iteration 21 of (collapsed) Gibbs sampling\n",
      "Iteration 22 of (collapsed) Gibbs sampling\n",
      "Iteration 23 of (collapsed) Gibbs sampling\n",
      "Iteration 24 of (collapsed) Gibbs sampling\n",
      "Iteration 25 of (collapsed) Gibbs sampling\n",
      "Iteration 26 of (collapsed) Gibbs sampling\n",
      "Iteration 27 of (collapsed) Gibbs sampling\n",
      "Iteration 28 of (collapsed) Gibbs sampling\n",
      "Iteration 29 of (collapsed) Gibbs sampling\n",
      "Iteration 30 of (collapsed) Gibbs sampling\n",
      "Iteration 31 of (collapsed) Gibbs sampling\n",
      "Iteration 32 of (collapsed) Gibbs sampling\n",
      "Iteration 33 of (collapsed) Gibbs sampling\n",
      "Iteration 34 of (collapsed) Gibbs sampling\n",
      "Iteration 35 of (collapsed) Gibbs sampling\n",
      "Iteration 36 of (collapsed) Gibbs sampling\n",
      "Iteration 37 of (collapsed) Gibbs sampling\n",
      "Iteration 38 of (collapsed) Gibbs sampling\n",
      "Iteration 39 of (collapsed) Gibbs sampling\n",
      "Iteration 40 of (collapsed) Gibbs sampling\n",
      "Iteration 41 of (collapsed) Gibbs sampling\n",
      "Iteration 42 of (collapsed) Gibbs sampling\n",
      "Iteration 43 of (collapsed) Gibbs sampling\n",
      "Iteration 44 of (collapsed) Gibbs sampling\n",
      "Iteration 45 of (collapsed) Gibbs sampling\n",
      "Iteration 46 of (collapsed) Gibbs sampling\n",
      "Iteration 47 of (collapsed) Gibbs sampling\n",
      "Iteration 48 of (collapsed) Gibbs sampling\n",
      "Iteration 49 of (collapsed) Gibbs sampling\n",
      "Iteration 50 of (collapsed) Gibbs sampling\n",
      "Iteration 51 of (collapsed) Gibbs sampling\n",
      "Iteration 52 of (collapsed) Gibbs sampling\n",
      "Iteration 53 of (collapsed) Gibbs sampling\n",
      "Iteration 54 of (collapsed) Gibbs sampling\n",
      "Iteration 55 of (collapsed) Gibbs sampling\n",
      "Iteration 56 of (collapsed) Gibbs sampling\n",
      "Iteration 57 of (collapsed) Gibbs sampling\n",
      "Iteration 58 of (collapsed) Gibbs sampling\n",
      "Iteration 59 of (collapsed) Gibbs sampling\n",
      "Iteration 60 of (collapsed) Gibbs sampling\n",
      "Iteration 61 of (collapsed) Gibbs sampling\n",
      "Iteration 62 of (collapsed) Gibbs sampling\n",
      "Iteration 63 of (collapsed) Gibbs sampling\n",
      "Iteration 64 of (collapsed) Gibbs sampling\n",
      "Iteration 65 of (collapsed) Gibbs sampling\n",
      "Iteration 66 of (collapsed) Gibbs sampling\n",
      "Iteration 67 of (collapsed) Gibbs sampling\n",
      "Iteration 68 of (collapsed) Gibbs sampling\n",
      "Iteration 69 of (collapsed) Gibbs sampling\n",
      "Iteration 70 of (collapsed) Gibbs sampling\n",
      "Iteration 71 of (collapsed) Gibbs sampling\n",
      "Iteration 72 of (collapsed) Gibbs sampling\n",
      "Iteration 73 of (collapsed) Gibbs sampling\n",
      "Iteration 74 of (collapsed) Gibbs sampling\n",
      "Iteration 75 of (collapsed) Gibbs sampling\n",
      "Iteration 76 of (collapsed) Gibbs sampling\n",
      "Iteration 77 of (collapsed) Gibbs sampling\n",
      "Iteration 78 of (collapsed) Gibbs sampling\n",
      "Iteration 79 of (collapsed) Gibbs sampling\n",
      "Iteration 80 of (collapsed) Gibbs sampling\n",
      "Iteration 81 of (collapsed) Gibbs sampling\n",
      "Iteration 82 of (collapsed) Gibbs sampling\n",
      "Iteration 83 of (collapsed) Gibbs sampling\n",
      "Iteration 84 of (collapsed) Gibbs sampling\n",
      "Iteration 85 of (collapsed) Gibbs sampling\n",
      "Iteration 86 of (collapsed) Gibbs sampling\n",
      "Iteration 87 of (collapsed) Gibbs sampling\n",
      "Iteration 88 of (collapsed) Gibbs sampling\n",
      "Iteration 89 of (collapsed) Gibbs sampling\n",
      "Iteration 90 of (collapsed) Gibbs sampling\n",
      "Iteration 91 of (collapsed) Gibbs sampling\n",
      "Iteration 92 of (collapsed) Gibbs sampling\n",
      "Iteration 93 of (collapsed) Gibbs sampling\n",
      "Iteration 94 of (collapsed) Gibbs sampling\n",
      "Iteration 95 of (collapsed) Gibbs sampling\n",
      "Iteration 96 of (collapsed) Gibbs sampling\n",
      "Iteration 97 of (collapsed) Gibbs sampling\n",
      "Iteration 98 of (collapsed) Gibbs sampling\n",
      "Iteration 99 of (collapsed) Gibbs sampling\n",
      "Iteration 100 of (collapsed) Gibbs sampling\n",
      "Iteration 101 of (collapsed) Gibbs sampling\n",
      "Iteration 102 of (collapsed) Gibbs sampling\n",
      "Iteration 103 of (collapsed) Gibbs sampling\n",
      "Iteration 104 of (collapsed) Gibbs sampling\n",
      "Iteration 105 of (collapsed) Gibbs sampling\n",
      "Iteration 106 of (collapsed) Gibbs sampling\n",
      "Iteration 107 of (collapsed) Gibbs sampling\n",
      "Iteration 108 of (collapsed) Gibbs sampling\n",
      "Iteration 109 of (collapsed) Gibbs sampling\n",
      "Iteration 110 of (collapsed) Gibbs sampling\n",
      "Iteration 111 of (collapsed) Gibbs sampling\n",
      "Iteration 112 of (collapsed) Gibbs sampling\n",
      "Iteration 113 of (collapsed) Gibbs sampling\n",
      "Iteration 114 of (collapsed) Gibbs sampling\n",
      "Iteration 115 of (collapsed) Gibbs sampling\n",
      "Iteration 116 of (collapsed) Gibbs sampling\n",
      "Iteration 117 of (collapsed) Gibbs sampling\n",
      "Iteration 118 of (collapsed) Gibbs sampling\n",
      "Iteration 119 of (collapsed) Gibbs sampling\n",
      "Iteration 120 of (collapsed) Gibbs sampling\n",
      "Iteration 121 of (collapsed) Gibbs sampling\n",
      "Iteration 122 of (collapsed) Gibbs sampling\n",
      "Iteration 123 of (collapsed) Gibbs sampling\n",
      "Iteration 124 of (collapsed) Gibbs sampling\n",
      "Iteration 125 of (collapsed) Gibbs sampling\n",
      "Iteration 126 of (collapsed) Gibbs sampling\n",
      "Iteration 127 of (collapsed) Gibbs sampling\n",
      "Iteration 128 of (collapsed) Gibbs sampling\n",
      "Iteration 129 of (collapsed) Gibbs sampling\n",
      "Iteration 130 of (collapsed) Gibbs sampling\n",
      "Iteration 131 of (collapsed) Gibbs sampling\n",
      "Iteration 132 of (collapsed) Gibbs sampling\n",
      "Iteration 133 of (collapsed) Gibbs sampling\n",
      "Iteration 134 of (collapsed) Gibbs sampling\n",
      "Iteration 135 of (collapsed) Gibbs sampling\n",
      "Iteration 136 of (collapsed) Gibbs sampling\n",
      "Iteration 137 of (collapsed) Gibbs sampling\n",
      "Iteration 138 of (collapsed) Gibbs sampling\n",
      "Iteration 139 of (collapsed) Gibbs sampling\n",
      "Iteration 140 of (collapsed) Gibbs sampling\n",
      "Iteration 141 of (collapsed) Gibbs sampling\n",
      "Iteration 142 of (collapsed) Gibbs sampling\n",
      "Iteration 143 of (collapsed) Gibbs sampling\n",
      "Iteration 144 of (collapsed) Gibbs sampling\n",
      "Iteration 145 of (collapsed) Gibbs sampling\n",
      "Iteration 146 of (collapsed) Gibbs sampling\n",
      "Iteration 147 of (collapsed) Gibbs sampling\n",
      "Iteration 148 of (collapsed) Gibbs sampling\n",
      "Iteration 149 of (collapsed) Gibbs sampling\n",
      "Iteration 150 of (collapsed) Gibbs sampling\n",
      "Iteration 151 of (collapsed) Gibbs sampling\n",
      "Iteration 152 of (collapsed) Gibbs sampling\n",
      "Iteration 153 of (collapsed) Gibbs sampling\n",
      "Iteration 154 of (collapsed) Gibbs sampling\n",
      "Iteration 155 of (collapsed) Gibbs sampling\n",
      "Iteration 156 of (collapsed) Gibbs sampling\n",
      "Iteration 157 of (collapsed) Gibbs sampling\n",
      "Iteration 158 of (collapsed) Gibbs sampling\n",
      "Iteration 159 of (collapsed) Gibbs sampling\n",
      "Iteration 160 of (collapsed) Gibbs sampling\n",
      "Iteration 161 of (collapsed) Gibbs sampling\n",
      "Iteration 162 of (collapsed) Gibbs sampling\n",
      "Iteration 163 of (collapsed) Gibbs sampling\n",
      "Iteration 164 of (collapsed) Gibbs sampling\n",
      "Iteration 165 of (collapsed) Gibbs sampling\n",
      "Iteration 166 of (collapsed) Gibbs sampling\n",
      "Iteration 167 of (collapsed) Gibbs sampling\n",
      "Iteration 168 of (collapsed) Gibbs sampling\n",
      "Iteration 169 of (collapsed) Gibbs sampling\n",
      "Iteration 170 of (collapsed) Gibbs sampling\n",
      "Iteration 171 of (collapsed) Gibbs sampling\n",
      "Iteration 172 of (collapsed) Gibbs sampling\n",
      "Iteration 173 of (collapsed) Gibbs sampling\n",
      "Iteration 174 of (collapsed) Gibbs sampling\n",
      "Iteration 175 of (collapsed) Gibbs sampling\n",
      "Iteration 176 of (collapsed) Gibbs sampling\n",
      "Iteration 177 of (collapsed) Gibbs sampling\n",
      "Iteration 178 of (collapsed) Gibbs sampling\n",
      "Iteration 179 of (collapsed) Gibbs sampling\n",
      "Iteration 180 of (collapsed) Gibbs sampling\n",
      "Iteration 181 of (collapsed) Gibbs sampling\n",
      "Iteration 182 of (collapsed) Gibbs sampling\n",
      "Iteration 183 of (collapsed) Gibbs sampling\n",
      "Iteration 184 of (collapsed) Gibbs sampling\n",
      "Iteration 185 of (collapsed) Gibbs sampling\n",
      "Iteration 186 of (collapsed) Gibbs sampling\n",
      "Iteration 187 of (collapsed) Gibbs sampling\n",
      "Iteration 188 of (collapsed) Gibbs sampling\n",
      "Iteration 189 of (collapsed) Gibbs sampling\n",
      "Iteration 190 of (collapsed) Gibbs sampling\n",
      "Iteration 191 of (collapsed) Gibbs sampling\n",
      "Iteration 192 of (collapsed) Gibbs sampling\n",
      "Iteration 193 of (collapsed) Gibbs sampling\n",
      "Iteration 194 of (collapsed) Gibbs sampling\n",
      "Iteration 195 of (collapsed) Gibbs sampling\n",
      "Iteration 196 of (collapsed) Gibbs sampling\n",
      "Iteration 197 of (collapsed) Gibbs sampling\n",
      "Iteration 198 of (collapsed) Gibbs sampling\n",
      "Iteration 199 of (collapsed) Gibbs sampling\n",
      "Iteration 200 of (collapsed) Gibbs sampling\n",
      "Iteration 201 of (collapsed) Gibbs sampling\n",
      "Iteration 202 of (collapsed) Gibbs sampling\n",
      "Iteration 203 of (collapsed) Gibbs sampling\n",
      "Iteration 204 of (collapsed) Gibbs sampling\n",
      "Iteration 205 of (collapsed) Gibbs sampling\n",
      "Iteration 206 of (collapsed) Gibbs sampling\n",
      "Iteration 207 of (collapsed) Gibbs sampling\n",
      "Iteration 208 of (collapsed) Gibbs sampling\n",
      "Iteration 209 of (collapsed) Gibbs sampling\n",
      "Iteration 210 of (collapsed) Gibbs sampling\n",
      "Iteration 211 of (collapsed) Gibbs sampling\n",
      "Iteration 212 of (collapsed) Gibbs sampling\n",
      "Iteration 213 of (collapsed) Gibbs sampling\n",
      "Iteration 214 of (collapsed) Gibbs sampling\n",
      "Iteration 215 of (collapsed) Gibbs sampling\n",
      "Iteration 216 of (collapsed) Gibbs sampling\n",
      "Iteration 217 of (collapsed) Gibbs sampling\n",
      "Iteration 218 of (collapsed) Gibbs sampling\n",
      "Iteration 219 of (collapsed) Gibbs sampling\n",
      "Iteration 220 of (collapsed) Gibbs sampling\n",
      "Iteration 221 of (collapsed) Gibbs sampling\n",
      "Iteration 222 of (collapsed) Gibbs sampling\n",
      "Iteration 223 of (collapsed) Gibbs sampling\n",
      "Iteration 224 of (collapsed) Gibbs sampling\n",
      "Iteration 225 of (collapsed) Gibbs sampling\n",
      "Iteration 226 of (collapsed) Gibbs sampling\n",
      "Iteration 227 of (collapsed) Gibbs sampling\n",
      "Iteration 228 of (collapsed) Gibbs sampling\n",
      "Iteration 229 of (collapsed) Gibbs sampling\n",
      "Iteration 230 of (collapsed) Gibbs sampling\n",
      "Iteration 231 of (collapsed) Gibbs sampling\n",
      "Iteration 232 of (collapsed) Gibbs sampling\n",
      "Iteration 233 of (collapsed) Gibbs sampling\n",
      "Iteration 234 of (collapsed) Gibbs sampling\n",
      "Iteration 235 of (collapsed) Gibbs sampling\n",
      "Iteration 236 of (collapsed) Gibbs sampling\n",
      "Iteration 237 of (collapsed) Gibbs sampling\n",
      "Iteration 238 of (collapsed) Gibbs sampling\n",
      "Iteration 239 of (collapsed) Gibbs sampling\n",
      "Iteration 240 of (collapsed) Gibbs sampling\n",
      "Iteration 241 of (collapsed) Gibbs sampling\n",
      "Iteration 242 of (collapsed) Gibbs sampling\n",
      "Iteration 243 of (collapsed) Gibbs sampling\n",
      "Iteration 244 of (collapsed) Gibbs sampling\n",
      "Iteration 245 of (collapsed) Gibbs sampling\n",
      "Iteration 246 of (collapsed) Gibbs sampling\n",
      "Iteration 247 of (collapsed) Gibbs sampling\n",
      "Iteration 248 of (collapsed) Gibbs sampling\n",
      "Iteration 249 of (collapsed) Gibbs sampling\n",
      "Iteration 250 of (collapsed) Gibbs sampling\n",
      "Iteration 251 of (collapsed) Gibbs sampling\n",
      "Iteration 252 of (collapsed) Gibbs sampling\n",
      "Iteration 253 of (collapsed) Gibbs sampling\n",
      "Iteration 254 of (collapsed) Gibbs sampling\n",
      "Iteration 255 of (collapsed) Gibbs sampling\n",
      "Iteration 256 of (collapsed) Gibbs sampling\n",
      "Iteration 257 of (collapsed) Gibbs sampling\n",
      "Iteration 258 of (collapsed) Gibbs sampling\n",
      "Iteration 259 of (collapsed) Gibbs sampling\n",
      "Iteration 260 of (collapsed) Gibbs sampling\n",
      "Iteration 261 of (collapsed) Gibbs sampling\n",
      "Iteration 262 of (collapsed) Gibbs sampling\n",
      "Iteration 263 of (collapsed) Gibbs sampling\n",
      "Iteration 264 of (collapsed) Gibbs sampling\n",
      "Iteration 265 of (collapsed) Gibbs sampling\n",
      "Iteration 266 of (collapsed) Gibbs sampling\n",
      "Iteration 267 of (collapsed) Gibbs sampling\n",
      "Iteration 268 of (collapsed) Gibbs sampling\n",
      "Iteration 269 of (collapsed) Gibbs sampling\n",
      "Iteration 270 of (collapsed) Gibbs sampling\n",
      "Iteration 271 of (collapsed) Gibbs sampling\n",
      "Iteration 272 of (collapsed) Gibbs sampling\n",
      "Iteration 273 of (collapsed) Gibbs sampling\n",
      "Iteration 274 of (collapsed) Gibbs sampling\n",
      "Iteration 275 of (collapsed) Gibbs sampling\n",
      "Iteration 276 of (collapsed) Gibbs sampling\n",
      "Iteration 277 of (collapsed) Gibbs sampling\n",
      "Iteration 278 of (collapsed) Gibbs sampling\n",
      "Iteration 279 of (collapsed) Gibbs sampling\n",
      "Iteration 280 of (collapsed) Gibbs sampling\n",
      "Iteration 281 of (collapsed) Gibbs sampling\n",
      "Iteration 282 of (collapsed) Gibbs sampling\n",
      "Iteration 283 of (collapsed) Gibbs sampling\n",
      "Iteration 284 of (collapsed) Gibbs sampling\n",
      "Iteration 285 of (collapsed) Gibbs sampling\n",
      "Iteration 286 of (collapsed) Gibbs sampling\n",
      "Iteration 287 of (collapsed) Gibbs sampling\n",
      "Iteration 288 of (collapsed) Gibbs sampling\n",
      "Iteration 289 of (collapsed) Gibbs sampling\n",
      "Iteration 290 of (collapsed) Gibbs sampling\n",
      "Iteration 291 of (collapsed) Gibbs sampling\n",
      "Iteration 292 of (collapsed) Gibbs sampling\n",
      "Iteration 293 of (collapsed) Gibbs sampling\n",
      "Iteration 294 of (collapsed) Gibbs sampling\n",
      "Iteration 295 of (collapsed) Gibbs sampling\n",
      "Iteration 296 of (collapsed) Gibbs sampling\n",
      "Iteration 297 of (collapsed) Gibbs sampling\n",
      "Iteration 298 of (collapsed) Gibbs sampling\n",
      "Iteration 299 of (collapsed) Gibbs sampling\n",
      "Iteration 300 of (collapsed) Gibbs sampling\n",
      "Iteration 301 of (collapsed) Gibbs sampling\n",
      "Iteration 302 of (collapsed) Gibbs sampling\n",
      "Iteration 303 of (collapsed) Gibbs sampling\n",
      "Iteration 304 of (collapsed) Gibbs sampling\n",
      "Iteration 305 of (collapsed) Gibbs sampling\n",
      "Iteration 306 of (collapsed) Gibbs sampling\n",
      "Iteration 307 of (collapsed) Gibbs sampling\n",
      "Iteration 308 of (collapsed) Gibbs sampling\n",
      "Iteration 309 of (collapsed) Gibbs sampling\n",
      "Iteration 310 of (collapsed) Gibbs sampling\n",
      "Iteration 311 of (collapsed) Gibbs sampling\n",
      "Iteration 312 of (collapsed) Gibbs sampling\n",
      "Iteration 313 of (collapsed) Gibbs sampling\n",
      "Iteration 314 of (collapsed) Gibbs sampling\n",
      "Iteration 315 of (collapsed) Gibbs sampling\n",
      "Iteration 316 of (collapsed) Gibbs sampling\n",
      "Iteration 317 of (collapsed) Gibbs sampling\n",
      "Iteration 318 of (collapsed) Gibbs sampling\n",
      "Iteration 319 of (collapsed) Gibbs sampling\n",
      "Iteration 320 of (collapsed) Gibbs sampling\n",
      "Iteration 321 of (collapsed) Gibbs sampling\n",
      "Iteration 322 of (collapsed) Gibbs sampling\n",
      "Iteration 323 of (collapsed) Gibbs sampling\n",
      "Iteration 324 of (collapsed) Gibbs sampling\n",
      "Iteration 325 of (collapsed) Gibbs sampling\n",
      "Iteration 326 of (collapsed) Gibbs sampling\n",
      "Iteration 327 of (collapsed) Gibbs sampling\n",
      "Iteration 328 of (collapsed) Gibbs sampling\n",
      "Iteration 329 of (collapsed) Gibbs sampling\n",
      "Iteration 330 of (collapsed) Gibbs sampling\n",
      "Iteration 331 of (collapsed) Gibbs sampling\n",
      "Iteration 332 of (collapsed) Gibbs sampling\n",
      "Iteration 333 of (collapsed) Gibbs sampling\n",
      "Iteration 334 of (collapsed) Gibbs sampling\n",
      "Iteration 335 of (collapsed) Gibbs sampling\n",
      "Iteration 336 of (collapsed) Gibbs sampling\n",
      "Iteration 337 of (collapsed) Gibbs sampling\n",
      "Iteration 338 of (collapsed) Gibbs sampling\n",
      "Iteration 339 of (collapsed) Gibbs sampling\n",
      "Iteration 340 of (collapsed) Gibbs sampling\n",
      "Iteration 341 of (collapsed) Gibbs sampling\n",
      "Iteration 342 of (collapsed) Gibbs sampling\n",
      "Iteration 343 of (collapsed) Gibbs sampling\n",
      "Iteration 344 of (collapsed) Gibbs sampling\n",
      "Iteration 345 of (collapsed) Gibbs sampling\n",
      "Iteration 346 of (collapsed) Gibbs sampling\n",
      "Iteration 347 of (collapsed) Gibbs sampling\n",
      "Iteration 348 of (collapsed) Gibbs sampling\n",
      "Iteration 349 of (collapsed) Gibbs sampling\n",
      "Iteration 350 of (collapsed) Gibbs sampling\n",
      "Iteration 351 of (collapsed) Gibbs sampling\n",
      "Iteration 352 of (collapsed) Gibbs sampling\n",
      "Iteration 353 of (collapsed) Gibbs sampling\n",
      "Iteration 354 of (collapsed) Gibbs sampling\n",
      "Iteration 355 of (collapsed) Gibbs sampling\n",
      "Iteration 356 of (collapsed) Gibbs sampling\n",
      "Iteration 357 of (collapsed) Gibbs sampling\n",
      "Iteration 358 of (collapsed) Gibbs sampling\n",
      "Iteration 359 of (collapsed) Gibbs sampling\n",
      "Iteration 360 of (collapsed) Gibbs sampling\n",
      "Iteration 361 of (collapsed) Gibbs sampling\n",
      "Iteration 362 of (collapsed) Gibbs sampling\n",
      "Iteration 363 of (collapsed) Gibbs sampling\n",
      "Iteration 364 of (collapsed) Gibbs sampling\n",
      "Iteration 365 of (collapsed) Gibbs sampling\n",
      "Iteration 366 of (collapsed) Gibbs sampling\n",
      "Iteration 367 of (collapsed) Gibbs sampling\n",
      "Iteration 368 of (collapsed) Gibbs sampling\n",
      "Iteration 369 of (collapsed) Gibbs sampling\n",
      "Iteration 370 of (collapsed) Gibbs sampling\n",
      "Iteration 371 of (collapsed) Gibbs sampling\n",
      "Iteration 372 of (collapsed) Gibbs sampling\n",
      "Iteration 373 of (collapsed) Gibbs sampling\n",
      "Iteration 374 of (collapsed) Gibbs sampling\n",
      "Iteration 375 of (collapsed) Gibbs sampling\n",
      "Iteration 376 of (collapsed) Gibbs sampling\n",
      "Iteration 377 of (collapsed) Gibbs sampling\n",
      "Iteration 378 of (collapsed) Gibbs sampling\n",
      "Iteration 379 of (collapsed) Gibbs sampling\n",
      "Iteration 380 of (collapsed) Gibbs sampling\n",
      "Iteration 381 of (collapsed) Gibbs sampling\n",
      "Iteration 382 of (collapsed) Gibbs sampling\n",
      "Iteration 383 of (collapsed) Gibbs sampling\n",
      "Iteration 384 of (collapsed) Gibbs sampling\n",
      "Iteration 385 of (collapsed) Gibbs sampling\n",
      "Iteration 386 of (collapsed) Gibbs sampling\n",
      "Iteration 387 of (collapsed) Gibbs sampling\n",
      "Iteration 388 of (collapsed) Gibbs sampling\n",
      "Iteration 389 of (collapsed) Gibbs sampling\n",
      "Iteration 390 of (collapsed) Gibbs sampling\n",
      "Iteration 391 of (collapsed) Gibbs sampling\n",
      "Iteration 392 of (collapsed) Gibbs sampling\n",
      "Iteration 393 of (collapsed) Gibbs sampling\n",
      "Iteration 394 of (collapsed) Gibbs sampling\n",
      "Iteration 395 of (collapsed) Gibbs sampling\n",
      "Iteration 396 of (collapsed) Gibbs sampling\n",
      "Iteration 397 of (collapsed) Gibbs sampling\n",
      "Iteration 398 of (collapsed) Gibbs sampling\n",
      "Iteration 399 of (collapsed) Gibbs sampling\n",
      "Iteration 400 of (collapsed) Gibbs sampling\n",
      "Iteration 401 of (collapsed) Gibbs sampling\n",
      "Iteration 402 of (collapsed) Gibbs sampling\n",
      "Iteration 403 of (collapsed) Gibbs sampling\n",
      "Iteration 404 of (collapsed) Gibbs sampling\n",
      "Iteration 405 of (collapsed) Gibbs sampling\n",
      "Iteration 406 of (collapsed) Gibbs sampling\n",
      "Iteration 407 of (collapsed) Gibbs sampling\n",
      "Iteration 408 of (collapsed) Gibbs sampling\n",
      "Iteration 409 of (collapsed) Gibbs sampling\n",
      "Iteration 410 of (collapsed) Gibbs sampling\n",
      "Iteration 411 of (collapsed) Gibbs sampling\n",
      "Iteration 412 of (collapsed) Gibbs sampling\n",
      "Iteration 413 of (collapsed) Gibbs sampling\n",
      "Iteration 414 of (collapsed) Gibbs sampling\n",
      "Iteration 415 of (collapsed) Gibbs sampling\n",
      "Iteration 416 of (collapsed) Gibbs sampling\n",
      "Iteration 417 of (collapsed) Gibbs sampling\n",
      "Iteration 418 of (collapsed) Gibbs sampling\n",
      "Iteration 419 of (collapsed) Gibbs sampling\n",
      "Iteration 420 of (collapsed) Gibbs sampling\n",
      "Iteration 421 of (collapsed) Gibbs sampling\n",
      "Iteration 422 of (collapsed) Gibbs sampling\n",
      "Iteration 423 of (collapsed) Gibbs sampling\n",
      "Iteration 424 of (collapsed) Gibbs sampling\n",
      "Iteration 425 of (collapsed) Gibbs sampling\n",
      "Iteration 426 of (collapsed) Gibbs sampling\n",
      "Iteration 427 of (collapsed) Gibbs sampling\n",
      "Iteration 428 of (collapsed) Gibbs sampling\n",
      "Iteration 429 of (collapsed) Gibbs sampling\n",
      "Iteration 430 of (collapsed) Gibbs sampling\n",
      "Iteration 431 of (collapsed) Gibbs sampling\n",
      "Iteration 432 of (collapsed) Gibbs sampling\n",
      "Iteration 433 of (collapsed) Gibbs sampling\n",
      "Iteration 434 of (collapsed) Gibbs sampling\n",
      "Iteration 435 of (collapsed) Gibbs sampling\n",
      "Iteration 436 of (collapsed) Gibbs sampling\n",
      "Iteration 437 of (collapsed) Gibbs sampling\n",
      "Iteration 438 of (collapsed) Gibbs sampling\n",
      "Iteration 439 of (collapsed) Gibbs sampling\n",
      "Iteration 440 of (collapsed) Gibbs sampling\n",
      "Iteration 441 of (collapsed) Gibbs sampling\n",
      "Iteration 442 of (collapsed) Gibbs sampling\n",
      "Iteration 443 of (collapsed) Gibbs sampling\n",
      "Iteration 444 of (collapsed) Gibbs sampling\n",
      "Iteration 445 of (collapsed) Gibbs sampling\n",
      "Iteration 446 of (collapsed) Gibbs sampling\n",
      "Iteration 447 of (collapsed) Gibbs sampling\n",
      "Iteration 448 of (collapsed) Gibbs sampling\n",
      "Iteration 449 of (collapsed) Gibbs sampling\n",
      "Iteration 450 of (collapsed) Gibbs sampling\n",
      "Iteration 451 of (collapsed) Gibbs sampling\n",
      "Iteration 452 of (collapsed) Gibbs sampling\n",
      "Iteration 453 of (collapsed) Gibbs sampling\n",
      "Iteration 454 of (collapsed) Gibbs sampling\n",
      "Iteration 455 of (collapsed) Gibbs sampling\n",
      "Iteration 456 of (collapsed) Gibbs sampling\n",
      "Iteration 457 of (collapsed) Gibbs sampling\n",
      "Iteration 458 of (collapsed) Gibbs sampling\n",
      "Iteration 459 of (collapsed) Gibbs sampling\n",
      "Iteration 460 of (collapsed) Gibbs sampling\n",
      "Iteration 461 of (collapsed) Gibbs sampling\n",
      "Iteration 462 of (collapsed) Gibbs sampling\n",
      "Iteration 463 of (collapsed) Gibbs sampling\n",
      "Iteration 464 of (collapsed) Gibbs sampling\n",
      "Iteration 465 of (collapsed) Gibbs sampling\n",
      "Iteration 466 of (collapsed) Gibbs sampling\n",
      "Iteration 467 of (collapsed) Gibbs sampling\n",
      "Iteration 468 of (collapsed) Gibbs sampling\n",
      "Iteration 469 of (collapsed) Gibbs sampling\n",
      "Iteration 470 of (collapsed) Gibbs sampling\n",
      "Iteration 471 of (collapsed) Gibbs sampling\n",
      "Iteration 472 of (collapsed) Gibbs sampling\n",
      "Iteration 473 of (collapsed) Gibbs sampling\n",
      "Iteration 474 of (collapsed) Gibbs sampling\n",
      "Iteration 475 of (collapsed) Gibbs sampling\n",
      "Iteration 476 of (collapsed) Gibbs sampling\n",
      "Iteration 477 of (collapsed) Gibbs sampling\n",
      "Iteration 478 of (collapsed) Gibbs sampling\n",
      "Iteration 479 of (collapsed) Gibbs sampling\n",
      "Iteration 480 of (collapsed) Gibbs sampling\n",
      "Iteration 481 of (collapsed) Gibbs sampling\n",
      "Iteration 482 of (collapsed) Gibbs sampling\n",
      "Iteration 483 of (collapsed) Gibbs sampling\n",
      "Iteration 484 of (collapsed) Gibbs sampling\n",
      "Iteration 485 of (collapsed) Gibbs sampling\n",
      "Iteration 486 of (collapsed) Gibbs sampling\n",
      "Iteration 487 of (collapsed) Gibbs sampling\n",
      "Iteration 488 of (collapsed) Gibbs sampling\n",
      "Iteration 489 of (collapsed) Gibbs sampling\n",
      "Iteration 490 of (collapsed) Gibbs sampling\n",
      "Iteration 491 of (collapsed) Gibbs sampling\n",
      "Iteration 492 of (collapsed) Gibbs sampling\n",
      "Iteration 493 of (collapsed) Gibbs sampling\n",
      "Iteration 494 of (collapsed) Gibbs sampling\n",
      "Iteration 495 of (collapsed) Gibbs sampling\n",
      "Iteration 496 of (collapsed) Gibbs sampling\n",
      "Iteration 497 of (collapsed) Gibbs sampling\n",
      "Iteration 498 of (collapsed) Gibbs sampling\n",
      "Iteration 499 of (collapsed) Gibbs sampling\n",
      "Iteration 500 of (collapsed) Gibbs sampling\n"
     ]
    }
   ],
   "source": [
    "ldaobj.sample(0,50,10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 915.74059849,  871.91170051,  858.53324181,  851.5958892 ,\n",
       "        847.71522759,  845.05538051,  846.21565792,  844.11551862,\n",
       "        843.37388303,  843.68515761,  842.6811026 ,  844.49822166,\n",
       "        843.16216334,  841.69205381,  842.39886158,  841.40735716,\n",
       "        841.14476007,  842.36784153,  840.71664734,  842.10224441])"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ldaobj.perplexity()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Indeed, the topic model has improved, and may yet improve more with further sampling, but at this point we will stop to continue with the tutorial.  (In research applications, one would normally apply some convergence criterion to determine the stopping point.)  Ideally we'd like to throw away the initial samples and only keep the last ones.  ldaobj.samples_keep(n) keeps the last n samples of the chain (users can also pass a list of numbers corresponding to the indices they'd like to keep - remember that Python uses 0-indexing).  We will keep the last four samples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ldaobj.samples_keep(4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So far we have only been sampling topic assignments for all the words in the dataset.  What we really care about are the topics and the distribution of topics in each document.  ldaobj has been carrying these around for us while we have been sampling.  ldaobj.tt are the esitmated topics, and ldaobj.dt are the estimated document-topic distributions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(4742, 30, 4)\n",
      "(9488, 30, 4)\n"
     ]
    }
   ],
   "source": [
    "print ldaobj.tt.shape \n",
    "print ldaobj.dt.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The estimated topics are represented by $4742 \\times 30$ matrices whose columns sum to one, one for each sample, while the estimated distributions of topics within each document are represented by $9488 \\times 30$ matrices whose rows sum to one.  To get an idea of the topics that have been estimated, and whether they make sense, ldaobj.topic_content(n) produces topic_description.csv in the working directory.  Its rows contain the first n stems in each topic ranked according their probability, using the final stored sample.  It's a good idea to check the topics are \"reasonable\" before proceeding with any analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ldaobj.topic_content(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most economics researchers will probably be most interested initially in the distributions of topics within each document.  To generate these, one should average the matrices in ldaobj.dt.  Here we have only taken four samples for purposes of illustration, but in actual research one should ideally take as many as is computationally feasible.  ldaobj has a convenience method for doing this average, which will both return it as well as, by default, write it to dt.csv in the working directory (to disable printing, pass False to the method)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dt = ldaobj.dt_avg()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One might also be interested in the average topics themselves, in which case there is a similar convenience function available that writes tt.csv to the working directory by default.  Each unique stem in the data is associated to a number corresponding to rows of the topic matrices.  Therefore, in most cases one will probably want to print out this key too, available as ldaobj.dict_print."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "tt = ldaobj.tt_avg()\n",
    "ldaobj.dict_print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One might also want to replace the speech field in the original dataset with the estimated topics in order to have a ready-to-go dataset for regression or other econometric analysis.  The following code builds this dataset, and also writes it to file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data = data.drop('speech',1)\n",
    "for i in xrange(ldaobj.K): data['T' + str(i)] = dt[:,i]\n",
    "data.to_csv(\"final_output.csv\",index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If one wishes to analyze some function of the estimated document-topic distributions, this function should be computed for each separate sample and then averaged.  Since the relevant functions are context-specific, topicmodels.LDA does not provide them, but it can be easily extended to accomodate this."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Querying Using Estimated Topics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After estimating a topic model, one is often interested in estimating the distribution of topics for documents not included in estimation.  In this case, one option is to $\\textit{query}$ those documents by holding fixed the topics estimated from LDA, and only estimating the distribution of topics for the out-of-sample documents.  The topicmodels module also provides a class for such querying, which this section introduces.\n",
    "\n",
    "We will apply querying to the corpus of entire State of the Union Addresses since 1947 (recall that we estimated topics on the level of the paragraph within each speech).  In terms of estimating topics, the paragraph level is preferable to the speech level since individual paragraphs are more likely to be based around a single theme.  But, in terms of econometric work, the entire speech is a more natural unit of analysis.  At the same time, there is no general way of \"adding up\" probability distribution at the paragraph level in order to arrive at a speech-level distribution.  Hence the need for querying, which allows us to estimate the speech-level distributions.  (Extra credit: after the tutorial, estimate LDA on the entire speech level, and judge for yourself how the topics compare to those estimated at the paragraph level).\n",
    "\n",
    "The Query class is initialized in much the same way as LDA, but takes two additional objects: a 3-D array of estimated topics (number of tokens in the estimated topics $\\times$ number of estimated topics $\\times$ number of samples from the estimation); and a dictionary that maps tokens into an index.  We can just pass these directly from ldaobj, which contains data from the above estimated LDA model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data['speech'] = [' '.join(s) for s in docsobj.stems] # replace the speech field in the original data with its cleaned version from docsobj\n",
    "aggspeeches = data.groupby(['year','president'])['speech'].apply(lambda x: ' '.join(x)) # aggregate up to the speech level\n",
    "aggdocs = topicmodels.RawDocs(aggspeeches) # create new RawDocs object that contains entire speech stems in aggdocs.tokens\n",
    "queryobj = topicmodels.LDA.QueryGibbs(aggdocs.tokens,ldaobj.token_key,ldaobj.tt) # initialize query object with ldaobj attributes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before continuing, suppose that we instead wanted to query a document whose constitutent parts had not been included in estimation, for example a State of the Union Address from the 1930s.  How to proceed?  First, create a RawDocs object with the text to be queried (recall that RawDocs can take a basic text file, which each new line treated as a separate documents).  Second, perform the same cleaning steps as were done for the documents that went into the estimated model.  However, there is no need to do any stopword removal.  When you initialize a Query object, tokens in the documents to be queried that are not present in the estimated model are automatically stripped out.\n",
    "\n",
    "Since we don't need to estimate topics when querying, we can use far fewer iterations.  Let's start with 10."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sample 0 queried\n",
      "Sample 1 queried\n",
      "Sample 2 queried\n",
      "Sample 3 queried\n"
     ]
    }
   ],
   "source": [
    "queryobj.query(10) # query our four samples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To convince yourself that we don't need many iterations, let's look at the perplexity of the data at the entire speech level.  Notice that it is much higher than the perplexity of the data at the paragraph level.  This indicates that the topic model predicts content at the paragraph level much better."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 1232.02023518,  1231.76127481,  1232.21662346,  1231.15590423])"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "queryobj.perplexity()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's triple the number of iterations to 30 and again look at the perplexity.  (Unlike LDA's sampling, each call to query starts sampling from scratch)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sample 0 queried\n",
      "Sample 1 queried\n",
      "Sample 2 queried\n",
      "Sample 3 queried\n"
     ]
    }
   ],
   "source": [
    "queryobj.query(30) # query our four samples using more iterations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 1232.03494718,  1231.77821718,  1232.03326912,  1231.18207003])"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "queryobj.perplexity()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note these values are nearly exactly the same as for the 10-iteration querying.\n",
    "\n",
    "Finally, we follow similar steps as for LDA to output the estimated distribution of topics for entire speeches."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dt_query = queryobj.dt_avg()\n",
    "aggdata = pd.DataFrame(dt_query,index=aggspeeches.index,columns=['T' + str(i) for i in xrange(queryobj.K)])\n",
    "aggdata.to_csv(\"final_output_agg.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Assessing Output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point, you can use all of the csv files this tutorial has generated with your statistical software of choice (should this not be Python!) to analyze the topics.  Before finishing, though, we can perform an initial test of whether our output makes sense intuitively.  The following code determines each President's top topics, as measured in terms of deviations from the sample average."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def top_topics(x):\n",
    "\ttop = x.values.argsort()[-5:][::-1]\n",
    "\treturn(pd.Series(top,index=range(1,6)))\n",
    "\n",
    "temp = aggdata.reset_index()\n",
    "ranking = temp.set_index('president')\n",
    "ranking = ranking - ranking.mean()\n",
    "ranking = ranking.groupby(level='president').mean()\n",
    "ranking = ranking.sort('year')\n",
    "ranking = ranking.drop('year',1)\n",
    "ranking = ranking.apply(top_topics,axis=1)\n",
    "ranking.to_csv(\"president_top_topics.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this particular topic model, for example, George W. Bush's top topic contains words relating to military force, and Obama's employment and economic activity.  The topic model you estimate will of course vary, so I encourage you to open president_top_policy_topics.csv and topic_description.csv to have a look for yourself.  Note too that some topics probably relate to policy, while some others relate to pure rhetoric.  Depending on the nature of the analysis you want to do with the data, it may make sense to restrict attention to some subset of the estimated topics."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's all for now, I hope you enjoyed the tutorial, and begin to use topic modelling in your own work!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}